Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Google not honoring robots.txt ?

Receive the tools necessary to be the rock star of your field. Our 12-month program teaches you the evolving world of multi-channel marketing as well as the complex issues and opportunities found in the industry.

ASP Free and Iron Speed Designer are giving away $5,500+ in FREE licenses. Iron Speed's RAD CASE toolset can save up to 80% of your coding time. One free license per week, one perpetual license per month!
Download and Activate to enter!
Web development can be a daunting task, even for specialists. There is a lot of information to absorb and a lot of technologies to learn in order to manage a superior website. When trying to learn the ropes, developers need a reliable source to introduce new ideas that can be easily implemented. When working on large projects, even web veterans may run into a technology or an aspect of a technology that they are unfamiliar with.

Learn More!
Download to Enter | Contest Rules
Tutorials | Forums

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old November 22nd, 2006, 08:33 AM
JavaReb JavaReb is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Dec 2002
Posts: 390 JavaReb User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 Days 14 h 34 m 42 sec
Reputation Power: 10
Send a message via AIM to JavaReb
Google not honoring robots.txt ?
I have a simple robots.txt in the root of my website with the following:
User-Agent: *
Disallow: /my_folder_name_not_to_index/



where "my_folder_name_not_to_index" is the name of the folder I do NOT want google (or any other bot) to index. However I can still search google and find items listed.

Help ? Once items are in Google's cache, can they ever be removed ?
Reply With Quote
  #2  
Old November 22nd, 2006, 09:19 AM
Hombre's Avatar
Hombre Hombre is offline
Pixel Cruncher
Dev Shed Novice (500 - 999 posts)
  
Join Date: Jan 2005
Location: UK
Posts: 647 Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)Hombre User rank is Second Lieutenant (5000 - 10000 Reputation Level)  Folding Points: 3232 Folding Title: Novice Folder
Time spent in forums: 2 Weeks 1 Day 9 h 1 m 56 sec
Reputation Power: 106
Hi,

I'd suggest using a no index meta tag as well as a robots.txt file to prevent caching. You can get rid of pages cached by Google, how quickly depends on how keen you are.
Take a look at this Google Help section that deals with both.
Reply With Quote
  #3  
Old December 4th, 2006, 09:53 AM
JavaReb JavaReb is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Dec 2002
Posts: 390 JavaReb User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 Days 14 h 34 m 42 sec
Reputation Power: 10
Send a message via AIM to JavaReb
ok, that worked.

one more thing, i also have the following in my robots.txt
Disallow: /*.pdf$
Disallow: /*.PDF$

However, I still see several links to pdf's that show up with a google search. I dont have the directories listed though.
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Google not honoring robots.txt ?

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 


Powered by: vBulletin Version 3.0.5
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.

© 2003-2012 by Developer Shed. All rights reserved. DS Cluster 10 - Follow our Sitemap