Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
IBM developerWorks
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Told I have no robots.txt file but I do?

Discuss Told I have no robots.txt file but I do? in the Search Engine Optimization forum on Dev Shed.
Told I have no robots.txt file but I do? Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old January 1st, 2005, 04:37 AM
thall89553 thall89553 is offline
Contributing User
Dev Shed Beginner (1000 - 1499 posts)
  
Join Date: Jan 2004
Location: Budapest
Posts: 1,355 thall89553 User rank is Private First Class (20 - 50 Reputation Level)thall89553 User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 4 Days 7 h 37 m 56 sec
Reputation Power: 6
Send a message via Yahoo to thall89553
Told I have no robots.txt file but I do?
Can someone tell me why when I run my site's pages through www spidertest . com I get told I don't have a robot.txt file when I know I do? THe is

http:// design . tmhdesign . com /tasty_tuscany

Thanks so much.
__________________
Today the world, tomorrow the universe...
Reply With Quote
  #2  
Old January 1st, 2005, 01:22 PM
Scorpions4ever's Avatar
Scorpions4ever Scorpions4ever is offline
Banned ;)
Dev Shed God 5th Plane (7000 - 7499 posts)
  
Join Date: Nov 2001
Location: Glendale, Los Angeles County, California, USA
Posts: 7,402 Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level) 
Time spent in forums: 4 Weeks 1 Day 10 h 26 m 59 sec
Reputation Power: 715
robots.txt should be at the root URL of the domain (i.e.) http://design.tmhdesign.com/robots.txt

Putting it in a subdirectory is not the standard and won't work because robots don't look for it in a subdirectory,
__________________
Up the Irons
What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
"Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
Down with Sharon Osbourne
Puzzle of the Month solved by sizeablegrin, etienne141 and L7Sqr, superior C/C++ programmers of the month
Reply With Quote
  #3  
Old January 2nd, 2005, 06:25 PM
hawkins_786 hawkins_786 is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2004
Posts: 9 hawkins_786 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 0
hey

ive got a site that i recently made and i didnt know anything about a robots.txt file.

Is this why im not being ranked in google?
Reply With Quote
  #4  
Old January 3rd, 2005, 02:13 PM
Scorpions4ever's Avatar
Scorpions4ever Scorpions4ever is offline
Banned ;)
Dev Shed God 5th Plane (7000 - 7499 posts)
  
Join Date: Nov 2001
Location: Glendale, Los Angeles County, California, USA
Posts: 7,402 Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level)Scorpions4ever User rank is Major General (70000 - 90000 Reputation Level) 
Time spent in forums: 4 Weeks 1 Day 10 h 26 m 59 sec
Reputation Power: 715
Quote:
Originally Posted by hawkins_786
hey

ive got a site that i recently made and i didnt know anything about a robots.txt file.

Is this why im not being ranked in google?

Nope, I have websites that rank very well in google with no robots.txt file. The only reason for a robots.txt file is to prevent spiders from going into certain directories. Spiders don't have to respect the instructions in robots.txt either.

As for why those websites rank very well in Google, they actually have useful information that a people use. It is amazing how much traffic you get, once your site is regarded as an authority.
Reply With Quote
  #5  
Old January 4th, 2005, 04:07 PM
Crabby's Avatar
Crabby Crabby is offline
I do NOT have crabs...
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jan 2004
Location: England
Posts: 161 Crabby User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 56 m 43 sec
Reputation Power: 5
Heh, I've also found placing a robots.txt file is useful for keeping your 404 error logs clean
__________________
ArcadeXL.com - Action Games, Sports Games and more...
Disgusting Pictures
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Told I have no robots.txt file but I do?

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 5 hosted by Hostway.