jQuery диалоговое окно UI: огромный DIV на знаниях в IE 8

I'm not sure I completely understand the question, but here is a stab at it:

jQuery UI's Dialog widget (in modal dialog mode) always puts a full page div over the content on the rest of the page by default. This is normal behavior. It probably does not appear in quirks mode because quirks mode is not supported by the jQueryUI team. If you do not want the dialog to have a content covering div, you should consider the other dialog options.

As far as scrollbars are concerned, normally these don't appear because of modal dialog being used. However, if your page is large enough to cause scrolling before the dialog appears, then browsers will react differently in all cases.