How to automatically interact with a web page?

Hi;

This is sort of the opposite of web programming. In this case, I am not programming a web page, I want to write a program or record a script, whatever, to automatically interact with a web page.

This is not malicious -- I am trying to provide my users with a simplified interface to a (external) website, so they don't have to answer or even understand all the questions & buttons.

I suppose there are some simple tools for this, but I don't know them. Any suggestions?

To calm any fears I am trying to write an evil bot:

I am providing my clients with a preconfigured computer that performs a certain service -- video streaming. I want to make it as simple for them as possible to use, so automate almost everything. One of the things that has to be done is go on to the web page of my streaming service provider, log on, and enter the IP address of the computer that will be doing the streaming. I want to automate all that to spare the customer from the effort, plus I don't want to give him/her my account logon.

Is there any language or scripting or macro recording or anything tool that can help me do that?

Thanks,

Paul

Did you just describe a web browser? I think you did.

Define "interact." I am a bit lost on that part. What do you want it, and expect it, to do.



If they don't understand the questions, dumb it down for them. Anyone can understand buttons.

I think what he's looking for is to create a new (desktop application) interface to perform form POSTs or parse the results from passing certain url parameters, etc.

Which is fairly trivial in most languages.

A practical example I can think of for this would be to extract data from bls.gov. Go ahead and look around there :-) You can see how something that automatically navigates around there can be very handy ...

You'd be surprised at how many that can't understand buttons. I've ran into one person that literally needed to record me showing her how to do just about anything on her computer... opening up "My Computer" or even just copy and paste....

Had another person that had a corrupt IE install, so I installed Firefox and told her to use that... The problem with the IE install was causing her yahoo mail not to work... in FF, she still wasn't sure what to click to send mail (The Send button), so spent about 20 minutes on the phone with her trying to walk her through sending yahoo mail.

I would suggest that you have a look at Python and Beautiful Soup. After that, you can either give up or go to any other language you like. The principles are there.

One doesn't really interact with web pages. One responds. Some quibbles might be made with this, considering script, but the interaction is somewhat fixed and doesn't propagate too well.

If one responds appropriately, one might interact with the server, presuming it's designed for that purpose.

Well, medialint, you're right. I want to create a desktop application (or rather, a button on an application) that logs in to a web page, fills in some fields, then presses a button/link on the page.

This is not a web page I control; it is a web page of my streaming server provider. I don't want to ask my users to fill in an ip address, I don't want to give them my login, and frankly I don't want them to know who my streaming server provider is or that I use one -- anything that goes on under the hood.

I am not exactly describing a web browser per se, but rather a desktop application that acts like a web browser as far as the website is concerned, but not as far as the user is concerned.

To put it another way, I don't know exactly how a browser fills in a field or tells the webserver which link/button it clicked on.

I'll look at Python, but I suppose what you're saying is I should look at http protocol?

Thanks for any patience.

Paul

Just pop open the source of the page, find out the variables, then have your desktop application do a POST to the webserver. After that you have to handle the reply from the server. Sometimes, it's a normal text/html with a link to some other uri, sometimes, it's a application/octet stream that you'll have to handle yourself. The browser usually is able to either a) pop open the appropriate app to handle the incoming data, b) show whatever it got as a whole bunch of data.