Yes it could affect the speed of your site.

One thing to consider as well is that you want the site to be semantically correct. That means writing it in such a manner that the information makes sense in the flow of the document. If proper semantics are used, the document will still make sense even if it doesn't have any style information associated with it.

If you're using a lot of divs to control the way your page is displayed, there's likely a better way to organize the content. With many positioned divs if for some reason the css couldn't be displayed, the page will likely not display in a nice flow.

Read the following for more information:

HTML Tutorial - Semantics
Coding Clean and Semantic Templates
Writing Semantically Correct HTML