Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsWeb DesignSearch Engine Optimization

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old November 22nd, 2006, 08:33 AM
JavaReb JavaReb is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Dec 2002
Posts: 363 JavaReb User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 Days 9 h 29 m 24 sec
Reputation Power: 6
Send a message via AIM to JavaReb
Google not honoring robots.txt ?

I have a simple robots.txt in the root of my website with the following:
User-Agent: *
Disallow: /my_folder_name_not_to_index/



where "my_folder_name_not_to_index" is the name of the folder I do NOT want google (or any other bot) to index. However I can still search google and find items listed.

Help ? Once items are in Google's cache, can they ever be removed ?

Reply With Quote
  #2  
Old November 22nd, 2006, 09:19 AM
Hombre's Avatar
Hombre Hombre is offline
Pixel Cruncher
Dev Shed Novice (500 - 999 posts)
 
Join Date: Jan 2005
Location: UK
Posts: 647 Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)  Folding Points: 3232 Folding Title: Novice Folder
Time spent in forums: 2 Weeks 1 Day 9 h 1 m 56 sec
Reputation Power: 102
Hi,

I'd suggest using a no index meta tag as well as a robots.txt file to prevent caching. You can get rid of pages cached by Google, how quickly depends on how keen you are.
Take a look at this Google Help section that deals with both.

Reply With Quote
  #3  
Old December 4th, 2006, 09:53 AM
JavaReb JavaReb is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Dec 2002
Posts: 363 JavaReb User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 Days 9 h 29 m 24 sec
Reputation Power: 6
Send a message via AIM to JavaReb
ok, that worked.

one more thing, i also have the following in my robots.txt
Disallow: /*.pdf$
Disallow: /*.PDF$

However, I still see several links to pdf's that show up with a google search. I dont have the directories listed though.

Reply With Quote
Reply

Viewing: Dev Shed ForumsWeb DesignSearch Engine Optimization > Google not honoring robots.txt ?


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 3 hosted by Hostway
Stay green...Green IT