March 4th, 2012, 11:17 AM
Robots.txt on Wordpress
I created a robots.txt file for my Wordpress blog, after reading up on how they work.
Could someone tell me whether this file looks OK?
When I validate it using an online checker it tells me that it is all in one block. For some reason the empty line doesnt get seen. Any idea?
Last edited by dominoman; March 4th, 2012 at 11:50 AM.
March 5th, 2012, 12:00 AM
Test your robots.txt in GWT and if it get passed do not worry. Well I checked your post on Best Robots.txt file for WordPress, and checked your robots.txt file in the tool you mentioned to Validate Robots.txt file and it doesn't showed any error.
March 5th, 2012, 02:57 AM
Thanks. I found the error after that.
Originally Posted by wulftec0098
Putting crawl delay at the end of the block is poor syntax, and it means the validator doesnt recognise the next robots block of text.
I moved it and also updated the article (Best robots.txt file for Wordpress)
Now works perfectly.
I hope the set-up I developed helps other Wordpress bloggers. If I had known this before I would have saved myself a huge amount of hassle.
March 5th, 2012, 06:49 AM
Taking away what i said:
Sorry, got it what you were saying. Compared the two versions of your blog posts-new and old through Google Cache and got what you were saying about crawl delay. Earlier you put the crawl delay at the end and now you have changed the position
I think, for Googlebot, we can control Crawl Rate through GWT.
Last edited by wulftec0098; March 5th, 2012 at 07:02 AM.
December 4th, 2012, 12:43 AM
Don't use any online tool. Go with Google webmaster tool that authorized and reliable.
Originally Posted by dominoman
Your website robots.txt file is ok