Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Robots

Discuss Robots in the Search Engine Optimization forum on Dev Shed.
Robots Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Page 1 of 2 1 2 >
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old March 29th, 2005, 04:54 PM
scottishparis scottishparis is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Oct 2004
Location: Scotland
Posts: 16 scottishparis User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 33 m 2 sec
Reputation Power: 0
Send a message via AIM to scottishparis
Robots
Should I be bothered about having an robots.txt file on my server or is it just a complete waste of time?
Reply With Quote
  #2  
Old March 30th, 2005, 09:04 AM
dabomb_gent dabomb_gent is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Apr 2004
Posts: 195 dabomb_gent User rank is Lance Corporal (50 - 100 Reputation Level)dabomb_gent User rank is Lance Corporal (50 - 100 Reputation Level)dabomb_gent User rank is Lance Corporal (50 - 100 Reputation Level) 
Time spent in forums: 1 Day 20 h 7 m 57 sec
Reputation Power: 5
It won't hurt anyone to have it. I think there are only advantages in having one.
Reply With Quote
  #3  
Old March 30th, 2005, 09:34 AM
scottishparis scottishparis is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Oct 2004
Location: Scotland
Posts: 16 scottishparis User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 33 m 2 sec
Reputation Power: 0
Send a message via AIM to scottishparis
What are the advantages??
Reply With Quote
  #4  
Old March 30th, 2005, 09:37 AM
dabomb_gent dabomb_gent is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Apr 2004
Posts: 195 dabomb_gent User rank is Lance Corporal (50 - 100 Reputation Level)dabomb_gent User rank is Lance Corporal (50 - 100 Reputation Level)dabomb_gent User rank is Lance Corporal (50 - 100 Reputation Level) 
Time spent in forums: 1 Day 20 h 7 m 57 sec
Reputation Power: 5
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: admin.php
Disallow: /admin/

This is one thing you can do.
Reply With Quote
  #5  
Old April 4th, 2005, 06:17 PM
balford82 balford82 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Mar 2005
Posts: 108 balford82 User rank is Private First Class (20 - 50 Reputation Level)balford82 User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 19 h 32 m 42 sec
Reputation Power: 4
Does anyone know if robots count as "unique visitors" to a website? In other words, are they recognized as a visitor or will the website know that it is a robot? Or a combination of the two?
Reply With Quote
  #6  
Old April 5th, 2005, 06:25 AM
andymoo's Avatar
andymoo andymoo is offline
Timelord
Dev Shed Novice (500 - 999 posts)
  
Join Date: Oct 2003
Location: Loughborough, Leicestershire
Posts: 605 andymoo User rank is Second Lieutenant (5000 - 10000 Reputation Level)andymoo User rank is Second Lieutenant (5000 - 10000 Reputation Level)andymoo User rank is Second Lieutenant (5000 - 10000 Reputation Level)andymoo User rank is Second Lieutenant (5000 - 10000 Reputation Level)andymoo User rank is Second Lieutenant (5000 - 10000 Reputation Level)andymoo User rank is Second Lieutenant (5000 - 10000 Reputation Level)andymoo User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 4 Days 5 h 53 m 46 sec
Reputation Power: 63
Stats packages like Urchin will report spiders and robots seperate from standard browsers I think.
__________________
Andy Moore << oh no it's got a blog.....
Word Press WAP Plugin with Ad Mob Advertising revenue
PHP developer deploying ringtones, mp3 downloads and realtones
I'm a geek who's obsessed with stats and gadgets
Reply With Quote
  #7  
Old April 8th, 2005, 02:00 PM
hopstream's Avatar
hopstream hopstream is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Mar 2003
Posts: 187 hopstream User rank is Private First Class (20 - 50 Reputation Level)hopstream User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 6 h 9 m 30 sec
Reputation Power: 6
The purpose of robots.txt is to specify pages which shouldn't be seen by specied search engines. Examples would be a members area which you don't want spidered or anything like that.

Not having a robots.txt doesn't effect your ranking.
__________________

Saving money is easy with discount codes from under tag coupons. Coupon codes for online purchases.
Reply With Quote
  #8  
Old November 3rd, 2005, 12:05 AM
Blackhawk_996 Blackhawk_996 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2005
Posts: 61 Blackhawk_996 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 20 h 44 m
Reputation Power: 3
I notice this post is a bit dated but there are actually quite a few advantages. #1 being if your site is frequently spidered you can slow the crawl rate so your bandwidth isn't just hammered all the time. Also depending on your site content and needs you can use a robots.txt to not allow certain programs (like website copiers i.e. Offline Explorer)to access directories of your site. Not to mention most of the spiders standard requirements will be to include a robots.txt file in the not to distant future, if they don't already.

This is a copy of the robots.txt I have in use right now.
Attached Files
File Type: txt robots.txt (17.2 KB, 295 views)
Reply With Quote
  #9  
Old November 3rd, 2005, 03:18 PM
kaskudoo's Avatar
kaskudoo kaskudoo is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jun 2004
Location: east coast
Posts: 221 kaskudoo User rank is Private First Class (20 - 50 Reputation Level)kaskudoo User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 2 Days 13 h 25 m 39 sec
Reputation Power: 5
Send a message via AIM to kaskudoo Send a message via MSN to kaskudoo Send a message via Yahoo to kaskudoo
that sure is a long robots.txt file
mine is a little shorter:
Code:
User-agent: *
Disallow: /_private/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /800by600/
Disallow: /new_site_folder/
Disallow: /php_uploads/
Disallow: /stats/
Disallow: /old/
Reply With Quote
  #10  
Old November 3rd, 2005, 09:52 PM
Blackhawk_996 Blackhawk_996 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2005
Posts: 61 Blackhawk_996 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 20 h 44 m
Reputation Power: 3
Yeah I actually started using that one while running a gfx site/forum so most users couldn't just randomly slurp my site.
Reply With Quote
  #11  
Old November 10th, 2005, 12:26 PM
dasher1 dasher1 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Sep 2005
Location: Sydney Australia
Posts: 54 dasher1 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 15 h 30 m 2 sec
Reputation Power: 3
Amongst many other errors I am getting this sort of error relating to the forum on my site.

[client 207.46.98.49] File does not exist: /home/ranvetau/public_html/smf/Themes/default/help/index.php

If I use this in my robots txt file is it correctly constructed and will it resolve the error log

User-agent: *
Disallow: /_private/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /smf/
Reply With Quote
  #12  
Old November 10th, 2005, 05:31 PM
CreditFamily CreditFamily is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2005
Posts: 34 CreditFamily User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 14 h 56 m 33 sec
Reputation Power: 3
Talking
I added a robots.txt file just because I got tired to seeing 404 errors on my log analizer program. lol

-Mike
Credit Repair and Free Credit Report Info
http://www.CreditFamily.com
Reply With Quote
  #13  
Old November 11th, 2005, 12:55 PM
dasher1 dasher1 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Sep 2005
Location: Sydney Australia
Posts: 54 dasher1 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 15 h 30 m 2 sec
Reputation Power: 3
Thanks for your help smart arse. Credit Family? Must remember to recommend your organisation to my friends and family
Reply With Quote
  #14  
Old November 13th, 2005, 12:46 AM
Blackhawk_996 Blackhawk_996 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2005
Posts: 61 Blackhawk_996 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 20 h 44 m
Reputation Power: 3
Quote:
Originally Posted by dasher1
Amongst many other errors I am getting this sort of error relating to the forum on my site.

[client 207.46.98.49] File does not exist: /home/ranvetau/public_html/smf/Themes/default/help/index.php

If I use this in my robots txt file is it correctly constructed and will it resolve the error log

User-agent: *
Disallow: /_private/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /smf/


It's written correctly and should fix your error listed, although you might just actually wanna have a look at your forum install. From what I know of SMF your missing your main help page or it might be mapped incorrectly?
Reply With Quote
  #15  
Old November 13th, 2005, 12:56 AM
dasher1 dasher1 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Sep 2005
Location: Sydney Australia
Posts: 54 dasher1 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 15 h 30 m 2 sec
Reputation Power: 3
Thanks for your help. I will take you advice and try to determine the cause of the error.

My apologies to all except creditfamily for my response to his reply. For clarification I have responded to him in another thread as follows.

Is dasher1 on my back or what? lol
Sorry to dissapoint you dasher1. My advertising is done on google adwords and through reciprocal links.

Thanks for those of you who provided feedback!

-Mike

My reply

No I am not on your back. If anything I am trying to indicate to you that your reply to my question regarding a robots.txt file was beneath the dignity of reasonable person. In an effort to make you understand how rude you are, I am trying to play the game your way and finding it difficult to match you. I give up, you win.

You have asked questions that I am sure many with an advanced knowledge of the subject would find laughable. They, unlike you, have answered in polite and helpful terms. Your answer to my question did not receive the same consideration. In short mate, if you can't say anything helpful, keep your mouth shut and your fingers away from the keyboard. Only a fool stymies learning by ridiculing the pupil.
Reply With Quote
Reply
Page 1 of 2 1 2 >

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Robots

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox