#1
  1. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    37
    Rep Power
    14

    Question CSS keeping spiders away?


    Hi there:
    I wonder if there is a way to keep spiders away from certain sections of my web pages? For example, keep them away from the nav_bar, but index the rest of the html page?

    Someone told me it could be done by CSS but I've tried to search for it without any luck.

    Any suggestions would be greatly appreciated.

    Sincerely,
    Ponch
  2. #2
  3. Wiser? Not exactly.
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    May 2001
    Location
    Bonita Springs, FL
    Posts
    5,932
    Rep Power
    4033
    Not as far as I know. You can prevent them for indexing a page completely using a robots.txt file in your site, but it sounds like you are just wanting them to ignore certain parts of your pages? To do that, the only thing you could really do is use Javascript to include a *.js file which is nothing but a buch of document.write() commands to write out your page data. This is generally bad though as it will not only stop spiders from viewing those parts of your site, but it will also stop anyone using a browser that does not support Javascript, or anyone which has it disabled.
    Recycle your old CD's, don't just trash them



    If I helped you out, show some love with some reputation, or tip with Bitcoins to 1N645HfYf63UbcvxajLKiSKpYHAq2Zxud
  4. #3
  5. Perl Jedi
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2004
    Location
    New Hampshire
    Posts
    116
    Rep Power
    21

    Lightbulb


    The answer you are looking for is as follows:

    Create a robots.txt file in your website root directory with the following content:

    User-Agent: *
    Disallow: /

    That should do the trick.

    Here is the link where you can read the full thread I pulled this from:

    http://groups.google.com/groups?hl=e...dk%26rnum%3D11

    Regards,

    jlk
  6. #4
  7. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    37
    Rep Power
    14
    Thanks guys!
    JLK, it wasn't really what I was asking for. That will stop spiders to index the site all along. However, I just wanted certain sections on a page to not be indexed. Especially text like NAV_Buttons and Phone Numbers and Street Addresses, among others. Someone said one could do it with .css to keep the spiders away. But I haven't figured it out.

    Any suggestions out there?

    Peace,
    Ponch
  8. #5
  9. Cunning Linguist
    Devshed Supreme Being (6500+ posts)

    Join Date
    Jul 2003
    Location
    I used to live at home, now I stay at the house
    Posts
    10,180
    Rep Power
    2038
    All you gotta do is list the pages of your site that you don't want the spiders to see in the robots.txt file, and they'll look at everything but those pages...

    Here's a good tutorial... http://www.searchengineworld.com/rob...s_tutorial.htm
  10. #6
  11. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    37
    Rep Power
    14
    I don't know if I'm not explaining this correctly?!?

    I would like the spiders to crawl the entire site. Each and every page should be indexed. However, I would like the spiders to not be able to read certain sections of my pages. I'm not saying certain sections of the WEBSITE!

    For example, I want the spider to read the index.html page, but not the whole entire page. There are a few sections that would just take up nonsense and I only want the spider to capture valuable information that will make my ranking higher in the searches.

    I know how to use the robots.txt and have robots not being able to reach certain folders and sections within a whole web site, but I'm talking about keeping info away on a single page.

    Any help would be good. It's driving me nuts!

    Peace,
    Ponch
  12. #7
  13. No Profile Picture
    c0mpl33t n00b
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2004
    Posts
    13
    Rep Power
    0
    Originally Posted by ponch9
    I don't know if I'm not explaining this correctly?!?
    You're explaining it fine, the other two just aren't reading your posts properly. Or maybe they're just daft.

    Take kicken's advice and either do the 'bad' JS thing or leave it alone.

    OR take a different approach and realise that it can't be done... SO put all of your sensitive and/or garbage information on a different page (or set of different pages) and use the robots.txt file. If it's that big of a deal, use frames and tell the robots not to index the frame pages.

    So there is a way to emulate it, it just depends on how much design you're willing to sacrifice to get what you want. What's more important, the layout of your site or the sensitivity of the info?

IMN logo majestic logo threadwatch logo seochat tools logo