Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Robots.txt

Discuss Robots.txt in the Search Engine Optimization forum on Dev Shed.
Robots.txt Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old May 5th, 2007, 01:48 AM
sir_drinxalot's Avatar
sir_drinxalot sir_drinxalot is offline
Known to taste like chicken
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Aug 2003
Location: In front of my computer
Posts: 228 sir_drinxalot User rank is Sergeant (500 - 2000 Reputation Level)sir_drinxalot User rank is Sergeant (500 - 2000 Reputation Level)sir_drinxalot User rank is Sergeant (500 - 2000 Reputation Level)sir_drinxalot User rank is Sergeant (500 - 2000 Reputation Level)sir_drinxalot User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 3 Days 13 h 59 m 40 sec
Reputation Power: 19
Send a message via MSN to sir_drinxalot
Facebook
Robots.txt
there is a file on my webserver named robots.txt, which is completely empty.

I'm guessing this file serves a purpose with robots/spiders/crawlers, which leads me to think maybe there should be some text in the file, maybe some file names, or instructions for the robot etc??

Anyone shed some light on what i'm meant to do with this particular file?

SD
__________________
"Operator, Give me the number for 911" - Homer J Simpson
JonnoWalmsley.com
vv Latest Image vv
Reply With Quote
  #2  
Old May 5th, 2007, 04:48 AM
pws1970's Avatar
pws1970 pws1970 is offline
SuperFlyGuy
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jul 2006
Posts: 135 pws1970 User rank is Second Lieutenant (5000 - 10000 Reputation Level)pws1970 User rank is Second Lieutenant (5000 - 10000 Reputation Level)pws1970 User rank is Second Lieutenant (5000 - 10000 Reputation Level)pws1970 User rank is Second Lieutenant (5000 - 10000 Reputation Level)pws1970 User rank is Second Lieutenant (5000 - 10000 Reputation Level)pws1970 User rank is Second Lieutenant (5000 - 10000 Reputation Level)pws1970 User rank is Second Lieutenant (5000 - 10000 Reputation Level)  Folding Points: 22216 Folding Title: Starter FolderFolding Points: 22216 Folding Title: Starter Folder
Time spent in forums: 2 Days 16 h 38 m 28 sec
Reputation Power: 74
Quote:
Originally Posted by sir_drinxalot
there is a file on my webserver named robots.txt, which is completely empty.

I'm guessing this file serves a purpose with robots/spiders/crawlers, which leads me to think maybe there should be some text in the file, maybe some file names, or instructions for the robot etc??

Anyone shed some light on what i'm meant to do with this particular file?

SD


The robots.txt file is used to help the visiting bots of what parts of your site to index or stay away from (like a folder where your admin section is or other sensitive data you don't want crawled) you can even ban certain robots from landing on your site completely - to read more on how best to set up your txt file read this.
Comments on this post
sir_drinxalot agrees: cheers mate!
Reply With Quote
  #3  
Old May 13th, 2007, 03:13 PM
ajin115 ajin115 is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Mar 2007
Posts: 8 ajin115 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 4 h 39 m 41 sec
Reputation Power: 0
Good information.
Reply With Quote
  #4  
Old May 13th, 2007, 10:31 PM
Noodles23's Avatar
Noodles23 Noodles23 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jun 2004
Location: Auckland, NZ
Posts: 426 Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)  Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2
Time spent in forums: 2 Days 23 h 23 m 29 sec
Reputation Power: 387
It can also be used by some search engines to find your sitemap file(s).

Code:
Sitemap: http://www.example.com/sitemap.xml.gz


People usually put a blank robots.txt in so it saves the search engines getting a 404 when requesting it.
Comments on this post
pws1970 agrees!
__________________
Cooking Forums
Reply With Quote
  #5  
Old May 30th, 2007, 04:01 PM
opensourcerules opensourcerules is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: May 2007
Posts: 10 opensourcerules User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 46 m 23 sec
Reputation Power: 0
You should ALWAYS have a robots.txt file that keeps robots out of folders you don't want indexed (admin; CGI etc.).
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Robots.txt

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 6 hosted by Hostway. 
Stay green...Green IT