#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2004
    Posts
    303
    Rep Power
    17

    Only allow google to index my pages


    Hi,

    I have a problem with my website. I find that too many bots on the internet are requesting my pages.

    I want to make a .htaccess file. That only allows the google robots to index my pages. And dissalow all the other robots.

    Is there also a way to limit the amount of requests a bot can make to my website?

    Greets,
    Thomas
  2. #2
  3. No Profile Picture
    Lost in code
    Devshed Supreme Being (6500+ posts)

    Join Date
    Dec 2004
    Posts
    8,301
    Rep Power
    7170
    You can tell other bots that you don't want them to access your pages, but you cannot force them to not access your pages. You would do this using a robots.txt file.
    PHP FAQ

    Originally Posted by Spad
    Ah USB, the only rectangular connector where you have to make 3 attempts before you get it the right way around
  4. #3
  5. Come play with me!
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    13,749
    Rep Power
    9397
    So just Google, huh? Don't care about Bing or Yahoo or Yandex or MSN or Exabot or anybody else? Tsk tsk.

IMN logo majestic logo threadwatch logo seochat tools logo