Subscription Based Sites

One of the sites I am looking after is a subscription only based site and apart from the login page that will have info about what the site is about and the info it offers we don't want search engines to crawl the rest of the site.

Any ideas how we should best set this up? Would it be enough just to set the robots file/meta tag to disallow robots viewing anything other than the front page?

Thanks

Hi,

The robots.txt file is respected by a lot of the crawlers including some of the big guns. such as Google, MSN and so on.

There are some, however, who ignore the instructions and plough through.

I have used PHP access control instructions to restrict page access and have not noticed any being indexed. I assume that the crawlers see the page pretty much as a browser does so would not be able to 'view' the restricted areas. If they did and someone tried to view the page the viewer would fail the PHP access criteria and be redirected to the login area.

ATB

Thanks hombre, can't rep you right now but it's on its way.

So would I just put something like this in the robots.txt

User-agent: *
Disallow: /content_folders/

placing the intro page in the main folder and then all subsequent files into the content_folders area? Or is there another way you'd recommend?

I use a combination of the methods you mentioned to restrict the crawlers as follows...

User-agent: *
Disallow: /restricted_directory/
Disallow: /anypage.php

Most of my restricted pages are held in their own directory and that directory - and all pages contained within it - are covered by Disallow:/restricted_directory/

It was easier for me to have one or two pages contained within the root itself that I also didn't want crawled and those are named individually in the robots file: Disallow:/anypage.php

ATB