#1
  1. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2004
    Location
    london
    Posts
    469
    Rep Power
    84

    Robots.txt on Wordpress


    I created a robots.txt file for my Wordpress blog, after reading up on how they work.

    Could someone tell me whether this file looks OK?

    http://www.linkbuildingtools.co.uk/robots.txt

    When I validate it using an online checker it tells me that it is all in one block. For some reason the empty line doesnt get seen. Any idea?
    Last edited by dominoman; March 4th, 2012 at 11:50 AM.
    Interested in online gaming? Access full reviews of all UK Online casinos here.

    Roulette Strategy |William Hill | Learn about Wine | Jackpot joy
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2011
    Posts
    33
    Rep Power
    3
    Test your robots.txt in GWT and if it get passed do not worry. Well I checked your post on Best Robots.txt file for WordPress, and checked your robots.txt file in the tool you mentioned to Validate Robots.txt file and it doesn't showed any error.
  4. #3
  5. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2004
    Location
    london
    Posts
    469
    Rep Power
    84
    Originally Posted by wulftec0098
    Test your robots.txt in GWT and if it get passed do not worry. Well I checked your post on Best Robots.txt file for WordPress, and checked your robots.txt file in the tool you mentioned to Validate Robots.txt file and it doesn't showed any error.
    Thanks. I found the error after that.

    Putting crawl delay at the end of the block is poor syntax, and it means the validator doesnt recognise the next robots block of text.

    I moved it and also updated the article (Best robots.txt file for Wordpress)

    Now works perfectly.

    I hope the set-up I developed helps other Wordpress bloggers. If I had known this before I would have saved myself a huge amount of hassle.
    Interested in online gaming? Access full reviews of all UK Online casinos here.

    Roulette Strategy |William Hill | Learn about Wine | Jackpot joy
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2011
    Posts
    33
    Rep Power
    3
    Taking away what i said:
    I am still seeing Crawl Delay in your robots.txt and article. Please update so that others will get useful information regarding robots.txt for Wordpress.
    Sorry, got it what you were saying. Compared the two versions of your blog posts-new and old through Google Cache and got what you were saying about crawl delay. Earlier you put the crawl delay at the end and now you have changed the position

    I think, for Googlebot, we can control Crawl Rate through GWT.
    Last edited by wulftec0098; March 5th, 2012 at 07:02 AM.
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2012
    Posts
    24
    Rep Power
    0
    Originally Posted by dominoman

    When I validate it using an online checker it tells me that it is all in one block. For some reason the empty line doesnt get seen. Any idea?
    Don't use any online tool. Go with Google webmaster tool that authorized and reliable.

    Your website robots.txt file is ok

IMN logo majestic logo threadwatch logo seochat tools logo