Using mod_rewrite to handle search bots

Hi,

I have been toying with the idea of using the mod_rewrite module and the mod_setenvif module so that when a bot comes along to do it's business it gets a set of pages back that feed it's need for specific information but when a regular user agent clicks the link from the search results they get something different.

Has anyone done this before? Am I about to embark on the stupidest project ever? I would appreciate any help and feedback.

Cheers - Robert.

First off, I assume you are talking about dynamic PHP site. Can you do search-engine-friendly with PHP using mod_rewrite? If not, just start from the scratch and do a search in this forum, I have posted related stuffs dozen times. If you do, then start here -> http://forums.devshed.com/showthrea...5113&forumid=15 and get some ideas from it. Specifically, set something like RewriteCond %{HTTP_USER_AGENT} wap [OR] . That said, you don't even need to use mod_setenvif at all.
In addition, you need to take advantage of DirectoryIndex and implement internal redirection in mod_rewrite to launch index.php for regular users and index.robot.html for robots.

>> Has anyone done this before?

Not me.

>> Am I about to embark on the stupidest project ever?

Probably not. But you need to figure out if it's worth doing it for the extra resources consuming cost.

Anyhow, do a search first, my replies regarding to mod_rewrite in this forum is more than enough to get you going.

freebsd,

firstly, thanks for the slap on the wrist! i do use the forum search but not this time, i will in future

ok, PHP is not the big issue for me (i use SSI for small bits) but following some advice from phreq i am exploring a new way of structuring my site to incorporate product names in the URL.

however, i want the user to be able to click http://blah/product/ and get back to a frameset with a subframe containing the dynamic search results for product but the search bot to access the same URL and get a nice load of keywords and so on.

i am not a PHP expert (in fact in am probably not even a beginner) but i guess that PHP would allow me to build a dynamic page based on user_agent?

i will look into it so expect some dumb posts in the PHP forum in the coming days

thanks again,

robert.

Why don't you just set up a robots.txt file and exclude everything but a
single directory. In that directory put a index.php3 file and bam.
Your in business no.

It is more "bam" than "in business" I'm afraid - most major search engines will not index .php* pages - only .htm* and .shtm* pages and remamber that not all robots will follow the robots.txt standard

It may be better to do a lookup on the user agent and if it matches a known bot, serve content for that bot else serve the standard website

I've been thinking about bots for a while, but havn't had the time yet to do more reasearch.

Since Bots and spiders don't like searching through dynamic pages (.asp/.php...) what would happen if you do this:

Have your index.html page, with a meta tag to auto-redirect to your index.php. You publish your index.html page as the main page to serach engines. In the .html page you have all the info for all the different bots and spiders to catalogue, but the users will (hopefully) seamlessly go to the .php page and not be any the wiser.

Now, my question is: Will the bots actually index the .html page, or will they automatically be redirected to the .php page? I can't use any funky server side parsing using Apachie as I'm using a shared server at my ISP. (Maybe later, I'll get a dedicated server... but for the time being...)

Is this possible? or will the bot just be redirected as quick as a wink and then balk at the .php page?

Cheers,

Steve