June 14th, 2013, 09:57 AM
How do recolve this Google crawler issue.
I am not professional programmer. But i am running my asp classic based site since 4 years.
Last year i move my site to Godaddy Windows Server due to Price and Hosting space was good. but after that my site traffic is goes down continously.
Google start crawling my un-avaliable pages and directories.
Google Crawling Like this...
Actual URL is...
These are sample URL, Google create so many like this.
Please help me to indentify this issue.
Last edited by Big B; June 14th, 2013 at 09:59 AM.
Reason: forget to write some text
June 16th, 2013, 10:28 AM
Maybe create a robots.txt file? There are web resources to help you with what to put in the file
It is a truism of American politics that no man who can win an election deserves to. --Trevanian, from the novel Shibumi
June 20th, 2013, 09:00 AM
my robots.txt file as under.
Originally Posted by Doug G
July 11th, 2013, 06:10 AM
Once you disallow in Robots.txt then it will stop crawling your web pages. And if problem persist again then add your website in Google webmaster tool and check if any error.
August 31st, 2013, 08:27 AM
Create robots.txt file and add it to your site. By using this you can allow or disallow the crawler to index which page of your site.