Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me

The Shed is going Social! Join us on FaceBook and Twitter and chime in on the conversation.

Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Robots.txt on Wordpress


Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old March 4th, 2012, 11:17 AM
dominoman's Avatar
dominoman dominoman is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Oct 2004
Location: london
Posts: 469 dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 1 Day 15 h 59 m 28 sec
Reputation Power: 83
Send a message via MSN to dominoman
Robots.txt on Wordpress
I created a robots.txt file for my Wordpress blog, after reading up on how they work.

Could someone tell me whether this file looks OK?

http://www.linkbuildingtools.co.uk/robots.txt

When I validate it using an online checker it tells me that it is all in one block. For some reason the empty line doesnt get seen. Any idea?
__________________
Interested in online gaming? Access full reviews of all UK Online casinos here.

Roulette Strategy |William Hill | Learn about Wine | Jackpot joy
Last edited by dominoman : March 4th, 2012 at 11:50 AM.
Reply With Quote
  #2  
Old March 5th, 2012, 12:00 AM
wulftec0098 wulftec0098 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Sep 2011
Posts: 34 wulftec0098 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 8 h 48 m 3 sec
Reputation Power: 2
Test your robots.txt in GWT and if it get passed do not worry. Well I checked your post on Best Robots.txt file for WordPress, and checked your robots.txt file in the tool you mentioned to Validate Robots.txt file and it doesn't showed any error.
Reply With Quote
  #3  
Old March 5th, 2012, 02:57 AM
dominoman's Avatar
dominoman dominoman is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Oct 2004
Location: london
Posts: 469 dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level)dominoman User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 1 Day 15 h 59 m 28 sec
Reputation Power: 83
Send a message via MSN to dominoman
Quote:
Originally Posted by wulftec0098
Test your robots.txt in GWT and if it get passed do not worry. Well I checked your post on Best Robots.txt file for WordPress, and checked your robots.txt file in the tool you mentioned to Validate Robots.txt file and it doesn't showed any error.


Thanks. I found the error after that.

Putting crawl delay at the end of the block is poor syntax, and it means the validator doesnt recognise the next robots block of text.

I moved it and also updated the article (Best robots.txt file for Wordpress)

Now works perfectly.

I hope the set-up I developed helps other Wordpress bloggers. If I had known this before I would have saved myself a huge amount of hassle.
Reply With Quote
  #4  
Old March 5th, 2012, 06:49 AM
wulftec0098 wulftec0098 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Sep 2011
Posts: 34 wulftec0098 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 8 h 48 m 3 sec
Reputation Power: 2
Taking away what i said:
Quote:
I am still seeing Crawl Delay in your robots.txt and article. Please update so that others will get useful information regarding robots.txt for Wordpress.


Sorry, got it what you were saying. Compared the two versions of your blog posts-new and old through Google Cache and got what you were saying about crawl delay. Earlier you put the crawl delay at the end and now you have changed the position

I think, for Googlebot, we can control Crawl Rate through GWT.
Last edited by wulftec0098 : March 5th, 2012 at 07:02 AM.
Reply With Quote
  #5  
Old December 4th, 2012, 12:43 AM
Davis Joseph Davis Joseph is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jun 2012
Posts: 27 Davis Joseph User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 7 h 30 m 57 sec
Reputation Power: 0
Quote:
Originally Posted by dominoman

When I validate it using an online checker it tells me that it is all in one block. For some reason the empty line doesnt get seen. Any idea?


Don't use any online tool. Go with Google webmaster tool that authorized and reliable.

Your website robots.txt file is ok
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Robots.txt on Wordpress

« Previous Thread | Next Thread »

Developer Shed Advertisers and Affiliates



Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 


Powered by: vBulletin Version 3.0.5
Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.

© 2003-2013 by Developer Shed. All rights reserved. DS Cluster - Follow our Sitemap