#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    2
    Rep Power
    0

    Post How do recolve this Google crawler issue.


    Hi,

    I am not professional programmer. But i am running my asp classic based site since 4 years.

    Last year i move my site to Godaddy Windows Server due to Price and Hosting space was good. but after that my site traffic is goes down continously.

    Google start crawling my un-avaliable pages and directories.

    example:

    Google Crawling Like this...

    mysite.com/wallpaper/games/poetry/recipes/recipe_detail.asp?id=351&title=Faluda
    Actual URL is...

    mysite.com/recipes/recipe_detail.asp?id=351&title=Faluda
    These are sample URL, Google create so many like this.

    Please help me to indentify this issue.

    waiting.

    Khan
    Last edited by Big B; June 14th, 2013 at 09:59 AM. Reason: forget to write some text
  2. #2
  3. No Profile Picture
    Stumpier old Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Jun 2003
    Posts
    14,409
    Rep Power
    4538
    Maybe create a robots.txt file? There are web resources to help you with what to put in the file
    ======
    Doug G
    ======
    It is a truism of American politics that no man who can win an election deserves to. --Trevanian, from the novel Shibumi
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    2
    Rep Power
    0
    Originally Posted by Doug G
    Maybe create a robots.txt file? There are web resources to help you with what to put in the file
    my robots.txt file as under.

    User-agent: *
    Disallow:

    Disallow: /admin/
    Disallow: /images/
    Disallow: /stats/
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    61
    Rep Power
    1
    Once you disallow in Robots.txt then it will stop crawling your web pages. And if problem persist again then add your website in Google webmaster tool and check if any error.
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2013
    Posts
    11
    Rep Power
    0
    Create robots.txt file and add it to your site. By using this you can allow or disallow the crawler to index which page of your site.

IMN logo majestic logo threadwatch logo seochat tools logo