Discuss Can a search BOT access this portion of my site? in the Search Engine Optimization forum on Dev Shed. Can a search BOT access this portion of my site? Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.
Receive the tools necessary to be the rock star of your field. Our 12-month program teaches you the evolving world of multi-channel marketing as well as the complex issues and opportunities found in the industry.
ASP Free and Iron Speed Designer are giving away $5,500+ in FREE licenses. Iron Speed's RAD CASE toolset can save up to 80% of your coding time. One free license per week, one perpetual license per month! Download and Activate to enter!
Web development can be a daunting task, even for specialists. There is a lot of information to absorb and a lot of technologies to learn in order to manage a superior website. When trying to learn the ropes, developers need a reliable source to introduce new ideas that can be easily implemented. When working on large projects, even web veterans may run into a technology or an aspect of a technology that they are unfamiliar with.
Posts: 118
Time spent in forums: 17 h 5 m 51 sec
Reputation Power: 6
Can a search BOT access this portion of my site?
can search engine bots hit https? I have about 30 pages of content that were originally written and saved in PDF format. The 30 pages of information is only available to paid members who have to log into our site. So my question is, if I convert all of this can a search engine bot get in and read this or will I have to make a link like <a href="link_to_content"></a> so that it is only visible through code? Will that method even work? Thanks again for the help.
Posts: 1,665
Time spent in forums: 2 Weeks 2 Days 34 m 20 sec
Reputation Power: 1642
In answer to your question, I think that some bots can read https:// documents, the same way that they can read and decypher PDF and Word documents.
But... If the content is only viewable by members after they have logged in, then it doesn't matter if it's http:// or https:// bots won't be able to get to it. Basically, if a person can't get to it without logging in to access the information, then a bot won't be able to either.
Posts: 78
Time spent in forums: 11 h 14 m 51 sec
Reputation Power: 13
Yes, as what has been mentioned by Catacaustic, pages that require log-ins will not be read by the bots. Adding a link to those pages will not help either if users are prompted for passwords upon keying in the urls directly.
Please don't mind me asking.. if your valuable content is only for your members, why make it available through search engines
Posts: 73
Time spent in forums: 16 h 22 m 36 sec
Reputation Power: 7
hi,
you have made pdf pages for paid members which is not meant for any free members. as you have created pages only for paid members and are restricted, the bot won't enter in to those pages.
if crawler visit those pages all informations will be opened. if you are looking all these things accrding to search engine point of view then at least you will have to make the information open for all. if access of visitors is prohibited then access of search engines may not be there.
if some thing is very personal it should not be disclosed to search engines.
Posts: 5
Time spent in forums: 1 h 59 sec
Reputation Power: 0
Quote:
Originally Posted by rudradatta
hi,
you have made pdf pages for paid members which is not meant for any free members. as you have created pages only for paid members and are restricted, the bot won't enter in to those pages.
if crawler visit those pages all informations will be opened. if you are looking all these things accrding to search engine point of view then at least you will have to make the information open for all. if access of visitors is prohibited then access of search engines may not be there.
if some thing is very personal it should not be disclosed to search engines.
Well my search engine's webspider could read a pdf file but it will not venture across a https: area so in my case it wouldn't get indexed and if it is behind a passowrd protected area then it wouls also not go through it either.
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
Linear Mode
