#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    2
    Rep Power
    0

    Post How do recolve this Google crawler issue.


    Hi,

    I am not professional programmer. But i am running my asp classic based site since 4 years.

    Last year i move my site to Godaddy Windows Server due to Price and Hosting space was good. but after that my site traffic is goes down continously.

    Google start crawling my un-avaliable pages and directories.

    example:

    Google Crawling Like this...

    mysite.com/wallpaper/games/poetry/recipes/recipe_detail.asp?id=351&title=Faluda
    Actual URL is...

    mysite.com/recipes/recipe_detail.asp?id=351&title=Faluda
    These are sample URL, Google create so many like this.

    Please help me to indentify this issue.

    waiting.

    Khan
    Last edited by Big B; June 14th, 2013 at 09:59 AM. Reason: forget to write some text
  2. #2
  3. No Profile Picture
    Grumpier old Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Jun 2003
    Posts
    14,445
    Rep Power
    4539
    Maybe create a robots.txt file? There are web resources to help you with what to put in the file
    ======
    Doug G
    ======
    Bartender to Rene Descartes "have another beer?" Descartes: "I think not" and he vanished.
    --Alfred Bester
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    2
    Rep Power
    0
    Originally Posted by Doug G
    Maybe create a robots.txt file? There are web resources to help you with what to put in the file
    my robots.txt file as under.

    User-agent: *
    Disallow:

    Disallow: /admin/
    Disallow: /images/
    Disallow: /stats/
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    57
    Rep Power
    2
    Once you disallow in Robots.txt then it will stop crawling your web pages. And if problem persist again then add your website in Google webmaster tool and check if any error.
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    10
    Rep Power
    0
    Create robots.txt file and add it to your site. By using this you can allow or disallow the crawler to index which page of your site.

IMN logo majestic logo threadwatch logo seochat tools logo