Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsWeb DesignSearch Engine Optimization

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old November 17th, 2006, 12:23 AM
lyscer lyscer is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jul 2006
Posts: 93 lyscer User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 h 19 m 36 sec
Reputation Power: 3
Can a search BOT access this portion of my site?

can search engine bots hit https? I have about 30 pages of content that were originally written and saved in PDF format. The 30 pages of information is only available to paid members who have to log into our site. So my question is, if I convert all of this can a search engine bot get in and read this or will I have to make a link like <a href="link_to_content"></a> so that it is only visible through code? Will that method even work? Thanks again for the help.

Reply With Quote
  #2  
Old November 17th, 2006, 12:38 AM
Catacaustic's Avatar
Catacaustic Catacaustic is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Mar 2005
Location: A Land Down Under
Posts: 395 Catacaustic User rank is Second Lieutenant (5000 - 10000 Reputation Level)Catacaustic User rank is Second Lieutenant (5000 - 10000 Reputation Level)Catacaustic User rank is Second Lieutenant (5000 - 10000 Reputation Level)Catacaustic User rank is Second Lieutenant (5000 - 10000 Reputation Level)Catacaustic User rank is Second Lieutenant (5000 - 10000 Reputation Level)Catacaustic User rank is Second Lieutenant (5000 - 10000 Reputation Level)Catacaustic User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 5 Days 1 h 15 m 29 sec
Reputation Power: 54
In answer to your question, I think that some bots can read https:// documents, the same way that they can read and decypher PDF and Word documents.

But... If the content is only viewable by members after they have logged in, then it doesn't matter if it's http:// or https:// bots won't be able to get to it. Basically, if a person can't get to it without logging in to access the information, then a bot won't be able to either.
Comments on this post
KC Tan agrees: Bots can't read pages that need authentication

Reply With Quote
  #3  
Old November 17th, 2006, 08:35 AM
KC Tan's Avatar
KC Tan KC Tan is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Sep 2006
Location: Singapore
Posts: 78 KC Tan User rank is Sergeant (500 - 2000 Reputation Level)KC Tan User rank is Sergeant (500 - 2000 Reputation Level)KC Tan User rank is Sergeant (500 - 2000 Reputation Level)KC Tan User rank is Sergeant (500 - 2000 Reputation Level)KC Tan User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 11 h 14 m 51 sec
Reputation Power: 10
Send a message via MSN to KC Tan
Yes, as what has been mentioned by Catacaustic, pages that require log-ins will not be read by the bots. Adding a link to those pages will not help either if users are prompted for passwords upon keying in the urls directly.

Please don't mind me asking.. if your valuable content is only for your members, why make it available through search engines

Reply With Quote
  #4  
Old November 22nd, 2006, 02:33 AM
rudradatta rudradatta is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Feb 2006
Location: india
Posts: 73 rudradatta User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 16 h 22 m 36 sec
Reputation Power: 3
hi,

you have made pdf pages for paid members which is not meant for any free members. as you have created pages only for paid members and are restricted, the bot won't enter in to those pages.

if crawler visit those pages all informations will be opened. if you are looking all these things accrding to search engine point of view then at least you will have to make the information open for all. if access of visitors is prohibited then access of search engines may not be there.

if some thing is very personal it should not be disclosed to search engines.

Reply With Quote
  #5  
Old December 15th, 2006, 06:50 AM
rehash rehash is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Nov 2005
Posts: 164 rehash User rank is Sergeant (500 - 2000 Reputation Level)rehash User rank is Sergeant (500 - 2000 Reputation Level)rehash User rank is Sergeant (500 - 2000 Reputation Level)rehash User rank is Sergeant (500 - 2000 Reputation Level)rehash User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 11 h 25 m 35 sec
Reputation Power: 8
SE bots should not be able to crawl paid members areas
__________________
Content Rewrite Project | Busybits Web Directory | Names Dir

Reply With Quote
  #6  
Old December 26th, 2006, 07:18 AM
jbjacob's Avatar
jbjacob jbjacob is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Oct 2006
Location: United Kingdom
Posts: 169 jbjacob User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 2 Days 15 h 12 m 31 sec
Reputation Power: 2
Send a message via ICQ to jbjacob Send a message via MSN to jbjacob
Quote:
Originally Posted by rehash
SE bots should not be able to crawl paid members areas


Yes they do not.I had a long debate on this in a different forum.
If a normal guest can't access your webpage then even bots won't.

Reply With Quote
  #7  
Old January 6th, 2007, 07:47 PM
Dave A Dave A is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jan 2007
Posts: 5 Dave A User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 59 sec
Reputation Power: 0
Quote:
Originally Posted by rudradatta
hi,

you have made pdf pages for paid members which is not meant for any free members. as you have created pages only for paid members and are restricted, the bot won't enter in to those pages.

if crawler visit those pages all informations will be opened. if you are looking all these things accrding to search engine point of view then at least you will have to make the information open for all. if access of visitors is prohibited then access of search engines may not be there.

if some thing is very personal it should not be disclosed to search engines.


Well my search engine's webspider could read a pdf file but it will not venture across a https: area so in my case it wouldn't get indexed and if it is behind a passowrd protected area then it wouls also not go through it either.

Reply With Quote
Reply

Viewing: Dev Shed ForumsWeb DesignSearch Engine Optimization > Can a search BOT access this portion of my site?


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 1 hosted by Hostway
Stay green...Green IT