September 21st, 2012, 11:44 AM
I would like to not permit crawlers access to subdomain my with the exception of announcements, knowledgebase and downloads.
I need the crawlers to have access to the above three if they act as directory or .php. For instance the url could be my.option9.com/announcements.php or it could be my.option9.com/announcements/example.php
Did I place the asteric correctly for wildcard purposes? As well is the disallow statement after my allows going to interfere?
September 26th, 2012, 04:10 PM
Does anyone know the answer to the above asked questions which have been posted by Nullified on above post's stated date?