March 23rd, 2004, 05:19 PM
CSS keeping spiders away?
I wonder if there is a way to keep spiders away from certain sections of my web pages? For example, keep them away from the nav_bar, but index the rest of the html page?
Someone told me it could be done by CSS but I've tried to search for it without any luck.
Any suggestions would be greatly appreciated.
March 23rd, 2004, 06:07 PM
Recycle your old CD's, don't just trash them
If I helped you out, show some love with some reputation, or tip with Bitcoins to 1N645HfYf63UbcvxajLKiSKpYHAq2Zxud
March 23rd, 2004, 06:18 PM
The answer you are looking for is as follows:
Create a robots.txt file in your website root directory with the following content:
That should do the trick.
Here is the link where you can read the full thread I pulled this from:
April 1st, 2004, 07:39 AM
JLK, it wasn't really what I was asking for. That will stop spiders to index the site all along. However, I just wanted certain sections on a page to not be indexed. Especially text like NAV_Buttons and Phone Numbers and Street Addresses, among others. Someone said one could do it with .css to keep the spiders away. But I haven't figured it out.
Any suggestions out there?
April 1st, 2004, 08:02 AM
April 1st, 2004, 08:48 AM
I don't know if I'm not explaining this correctly?!?
I would like the spiders to crawl the entire site. Each and every page should be indexed. However, I would like the spiders to not be able to read certain sections of my pages. I'm not saying certain sections of the WEBSITE!
For example, I want the spider to read the index.html page, but not the whole entire page. There are a few sections that would just take up nonsense and I only want the spider to capture valuable information that will make my ranking higher in the searches.
I know how to use the robots.txt and have robots not being able to reach certain folders and sections within a whole web site, but I'm talking about keeping info away on a single page.
Any help would be good. It's driving me nuts!
April 1st, 2004, 09:42 AM
You're explaining it fine, the other two just aren't reading your posts properly. Or maybe they're just daft.
Originally Posted by ponch9
Take kicken's advice and either do the 'bad' JS thing or leave it alone.
OR take a different approach and realise that it can't be done... SO put all of your sensitive and/or garbage information on a different page (or set of different pages) and use the robots.txt file. If it's that big of a deal, use frames and tell the robots not to index the frame pages.
So there is a way to emulate it, it just depends on how much design you're willing to sacrifice to get what you want. What's more important, the layout of your site or the sensitivity of the info?