Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Controlling Robots

Discuss Controlling Robots in the Search Engine Optimization forum on Dev Shed.
Controlling Robots Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old August 15th, 2007, 10:43 AM
Bailz Bailz is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Mar 2004
Location: Aberdeen, Scotland
Posts: 156 Bailz User rank is Corporal (100 - 500 Reputation Level)Bailz User rank is Corporal (100 - 500 Reputation Level)Bailz User rank is Corporal (100 - 500 Reputation Level)Bailz User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 1 Day 15 h 43 m 31 sec
Reputation Power: 7
Send a message via MSN to Bailz
Controlling Robots
My main index page pulls in pages from

/includes/content/*

For example:

index.php?section=section1

Pulls content from

/includes/content/section1

I'm guessing that doing

Disallow: /includes/content/section1

in my robots.txt will prevent the search engine from getting that content.

Is there any way to restrict access to these pages?
Reply With Quote
  #2  
Old August 15th, 2007, 09:01 PM
stymiee's Avatar
stymiee stymiee is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2003
Posts: 219 stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 20 h 15 m 53 sec
Reputation Power: 48
If you are using mod_rewrite you will have to block the virtual directories you want blocked as the robots will not be aware of thew hidden directories. They see what a user sees.
Reply With Quote
  #3  
Old August 16th, 2007, 04:52 AM
Bailz Bailz is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Mar 2004
Location: Aberdeen, Scotland
Posts: 156 Bailz User rank is Corporal (100 - 500 Reputation Level)Bailz User rank is Corporal (100 - 500 Reputation Level)Bailz User rank is Corporal (100 - 500 Reputation Level)Bailz User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 1 Day 15 h 43 m 31 sec
Reputation Power: 7
Send a message via MSN to Bailz
I'm not using mod_rewrite, just php includes. So if they request

index.php?section=section1

Then it'll include

includes/content/section1/index.php

as required.
Reply With Quote
  #4  
Old August 16th, 2007, 09:54 AM
stymiee's Avatar
stymiee stymiee is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2003
Posts: 219 stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 20 h 15 m 53 sec
Reputation Power: 48
Then you'll have to either dynamically add the meta noindex tag for those pages or list each page individually in your robots.txt. Google allows for a wildcard to be used which would make your life easier, but the other search engines don't so that's not an option.
Comments on this post
Bailz agrees: The noindex tag can be added easily! Thanks
Reply With Quote
  #5  
Old August 16th, 2007, 10:19 AM
Winters Winters is offline
Bad Coder
Dev Shed Intermediate (1500 - 1999 posts)
  
Join Date: Jul 2003
Posts: 1,806 Winters User rank is First Lieutenant (10000 - 20000 Reputation Level)Winters User rank is First Lieutenant (10000 - 20000 Reputation Level)Winters User rank is First Lieutenant (10000 - 20000 Reputation Level)Winters User rank is First Lieutenant (10000 - 20000 Reputation Level)Winters User rank is First Lieutenant (10000 - 20000 Reputation Level)Winters User rank is First Lieutenant (10000 - 20000 Reputation Level)Winters User rank is First Lieutenant (10000 - 20000 Reputation Level)Winters User rank is First Lieutenant (10000 - 20000 Reputation Level) 
Time spent in forums: 1 Week 6 Days 3 h 20 m 11 sec
Reputation Power: 186
Why not just make a .htaccess file, if you are using *nix that is, with:

Apache Code:
Original - Apache Code 
  1. <directory=/directory>
  2. NoIndex
  3. </directory>
  4.  
__________________
Formerly PHP-Newb
Under no circumstances click the button, because it is pointless. Thanks!
Reply With Quote
  #6  
Old August 16th, 2007, 10:47 PM
stymiee's Avatar
stymiee stymiee is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2003
Posts: 219 stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 20 h 15 m 53 sec
Reputation Power: 48
Because they are using PHP includes to add the content to the page. That means that directory isn't being accessed directly.
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Controlling Robots

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 4 hosted by Hostway.