Preventing Search Engines from Accessing Certain Directories - robots.txt problem

Dev Shed Forums Sponsor:

Stop making mediocre tutorials.The best tutorials are video! Camtasia Studio makes it easy to create engaging, buzz-building screen videos at any size, in any popular format. Download the free trial!

February 1st, 2002, 03:28 PM

valeria_vi

Contributing User

Join Date: Mar 2001

Location: Chicago, IL

Posts: 63

Time spent in forums: < 1 sec
Reputation Power: 8

Preventing Search Engines from Accessing Certain Directories - robots.txt problem

we have a robots.txt file in the root directory of our nimlok.com web site to stop web searches from accessing sertain directories. I have checked with several references to make sure it is written correctly, and cannot find any problems with it.
Here it is:

Code:

User-agent: *
Disallow: /cgi-bin/
Disallow: /distributor/
Disallow: /ecommerce/
Disallow: /cad/
Disallow: /robots/
Disallow: /imanager/
Disallow: /images/
Disallow: /technical/
Disallow: /analog3.0/
Disallow: /new/

But if you go to Google and search for "Nimlok" a link to the distributor directory is listed as a sub-section of the first result. I am not sure if other directories also appear in search results - I did not see them on the first page, but chances are - they are displayed somewhere later in search results.
What's teh deal with it?

February 3rd, 2002, 06:39 AM

andrewl41

Coder Extraordinaire

Join Date: Jan 2002

Location: London

Posts: 33

Time spent in forums: < 1 sec
Reputation Power: 7

Did you register with google before you added the robots.txt file? If so it is probable that their robot grabbed these pages in the past and is hanging on to them

In my experience, search engines have a habit of holding on to information for a long time without checking its still valid!

If this is the case you'll have to contact them and have the informatiuon removed.

__________________
Andrew

"Chaos Theory is a theory invented by scientists panicked by the thought that the public were beginning to understand the old ones."

Viewing: Dev Shed Forums > Other > Dev Shed Lounge > Preventing Search Engines from Accessing Certain Directories - robots.txt problem

« Previous Thread | Next Thread »

Thread Tools	Search this Thread
Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off

View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox

Forum Jump

Free IT White Papers!

Accelerating Trading Partner Performance
One in five. That's how many partner transactions have at least one error. That is an amazing statistic, particularly given the extraordinary leaps in innovation across the global supply chain during the past two decades. Download this white paper to learn more.

Competing on Analytics
This Tech Analysis is designed to help identify characteristics shared by analytics competitors, and includes information about 32 organizations that have made a commitment to quantitative, fact-based analysis.

Cost Effective Scaling with Virtualization and Coyote Point Systems
An overview of the industry trend toward virtualization, how server consolidation has increased the importance of application uptime and the steps being taken to integrate load balancing technology with virtualized servers.

Five Checkpoints to Implementing IP Telephony
Implementation planning for IP PBX software and IP telephony has become vital as businesses replace discontinued legacy PBX phone systems. This informative whitepaper outlines five "checkpoints" for any implementation plan that will help make IP communications a successful proposition.

Hosted Email Security: Staying Ahead of New Threats
In the last two years, email has become a fierce battleground between the nefarious forces of spam and malware, and the heroes of messaging protection. The spam volumes increased alarmingly every month, bringing clever new forms of phishing and virus propagation attacks.

More Free IT White Papers!