Finding the URL to "Go" button?

Dev Shed Forums Sponsor:

October 1st, 2012, 02:30 AM

blnl

Registered User

Join Date: Sep 2012

Posts: 5

Time spent in forums: 2 h 11 m 39 sec
Reputation Power: 0

Finding the URL to "Go" button?

Dear all, I'm not a developer and I also can't read JavaScript.

I hope someone will be so kind to help me with the interpretation of the JavaScript code.

THE PROBLEM DESCRIPTION:
I would like to find a direct link for downloading a .csv file for use with wget command:

Code:

Original - Code
europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&formKey=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b1

europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&formKey=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b1

This is a harmless .csv file, which contains daily stock data from the NYSE Euronext web page. When clicking the "Go" button on this page, I'm asked to "Open" or "Save" the file.

I have scanned through the source code of the page in hope to be able to compile the direct link to the file, but I was not able to understand what this code does.

Please have a look at it. If someone at least can tell me if this is possible that would already help a lot.

Thanks in advance!

October 1st, 2012, 02:37 PM

gsharp

Registered User

Join Date: Apr 2012

Posts: 9

Time spent in forums: 6 h 6 m 10 sec
Reputation Power: 0

The direct link to the file is what will be in your URL bar if you select OPEN (assuming that your browser opens the file in the browser and not in an external application). For example, when I click GO in Chrome with the defaults selected, I am taken to europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&formKey=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b1

October 4th, 2012, 04:12 AM

blnl

Registered User

Join Date: Sep 2012

Posts: 5

Time spent in forums: 2 h 11 m 39 sec
Reputation Power: 0

Quote:

Originally Posted by gsharp

Thanks gsharp, but it is not that simple.
Yes wget will download something, but not what I expect to see. The result is a file containing the web page source code, rather than the stock price quotes.

Code:

wget "europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&form Key=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b1" --no-check-certificate -O TEST.csv

Unfortunately that does not work. The user choices must be included in the URL string too (something like typefile=csv&layout=vertical&typedate=dmy etc.).

I figured out how to compile an URL with the correct user choices for downloading historical data of the individual stocks. Here is an example:

Code:

wget "europeanequities.nyx.com/nyx_eu_listings/price_chart/download_historical?typefile=csv&layout=vertical&typedate=dmy&separator=point&mic=XAMS&isin=NL0006034001&name=ASML HOLDING&namefile=Price_Data_Historical& from=1283299200000&to=1349308800000&adjusted=1&base=0" --no-check-certificate -O ASML.csv

Although, I managed somehow to interpret the source code of the stock history applet, with right download choices and compile the straight forward URL download string, I'm not able to do the same for the daily stock quotes

.

Any suggestions?

October 14th, 2012, 08:00 AM

fv967

Registered User

Join Date: Apr 2010

Posts: 7

Time spent in forums: 2 h 1 m 59 sec
Reputation Power: 0

same issue

I'm exactly trying to find the same solution. The go button is triggering a javascript function which makes the request.
I'm not famliar with javascript. It should be possible to find the URL by tracing the outgoing traffic with some tool or programs. I'm trying to find such trace tool.

regards, Frank

October 15th, 2012, 11:23 AM

gsharp

Registered User

Join Date: Apr 2012

Posts: 9

Time spent in forums: 6 h 6 m 10 sec
Reputation Power: 0

Quote:

Originally Posted by fv967

If you're using Chrome, this functionality is built in. Just hit F12 and check the Network tab.

October 16th, 2012, 03:21 AM

fv967

Registered User

Join Date: Apr 2010

Posts: 7

Time spent in forums: 2 h 1 m 59 sec
Reputation Power: 0

Http Post

Here's what I found out:
The popup window is submitting an HTTP POST query. Before, it was an HTTP GET query.
I wrote a small java program which sends a POST message (with appropriate post data) and the requested information was coming back correctly.
The test I wrote was for retrieving stocks which are listed on each exchange (AM,BR,PA,LS).

So, the solution it to write a small program which launches an HTTP/POST.

reg, Frank

November 15th, 2012, 02:35 PM

blnl

Registered User

Join Date: Sep 2012

Posts: 5

Time spent in forums: 2 h 11 m 39 sec
Reputation Power: 0

Hello Frank,

Would you be so kind to share your code on this thread?

With kind regards,
Boris

November 28th, 2012, 04:17 AM

fv967

Registered User

Join Date: Apr 2010

Posts: 7

Time spent in forums: 2 h 1 m 59 sec
Reputation Power: 0

Code

Hi Boris,

There are two kinds of information you can download from euronext:
1. All listed stocks for an Exchange
2. Historic prices for a specific stock

Herewith the code for nr.1.

Quote:

private static String URL_EURONEXT_INFO_PARAMETERS = "format=3&layout=2&decimal_separator=1&date_format=1&op=Go&form_build_id=form-80ac117839ec70d62168953ce0b91657&form_id=nyx_download_form";

public static boolean postRequest(String urlString, StringBuffer strBufferHTML) throws IOException, ProtocolException, MalformedURLException {
java.net.URL url;
java.net.HttpURLConnection connection = null;
try {
url = new URL(urlString);
connection = (java.net.HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);

// Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(URL_EURONEXT_INFO_PARAMETERS);
wr.flush();
wr.close();

// Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
strBufferHTML.append(line);
strBufferHTML.append('\r');
}
rd.close();

} finally {
if (connection != null) {
connection.disconnect();
}
}
return true;
}

I've historic prices (10 years) from several exchanges. Maybe you don't need to program it yourself.

reg, Frank

November 28th, 2012, 04:19 AM

fv967

Registered User

Join Date: Apr 2010

Posts: 7

Time spent in forums: 2 h 1 m 59 sec
Reputation Power: 0

Code

Hi Boris,

There are two kinds of information you can download from euronext:
1. All listed stocks for an Exchange
2. Historic prices for a specific stock

Herewith the code for nr.1.

Quote:

I've historic prices (10 years) from several exchanges. Maybe you don't need to program it yourself. Just let me know.

reg, Frank

#10

November 28th, 2012, 10:03 AM

blnl

Registered User

Join Date: Sep 2012

Posts: 5

Time spent in forums: 2 h 11 m 39 sec
Reputation Power: 0

Quote:

Originally Posted by fv967

Hi Boris,

There are two kinds of information you can download from euronext:
1. All listed stocks for an Exchange
2. Historic prices for a specific stock

Herewith the code for nr.1.

I've historic prices (10 years) from several exchanges. Maybe you don't need to program it yourself. Just let me know.

reg, Frank

Hello Frank,

Thank you, for sharing your code here. I shall try it out as soon as I find some time (probably it has to wait until the Christmas holidays).

Concerning the historical prices, I already wrote a script for that one. Thanks for asking.

With kind regards,
Boris

#11

December 10th, 2012, 04:00 AM

blnl

Registered User

Join Date: Sep 2012

Posts: 5

Time spent in forums: 2 h 11 m 39 sec
Reputation Power: 0

Quote:

Originally Posted by fv967

Hello Frank,

I did not manage to run your program, but it's not your fault, it is my lack of know how.

However, I used your URL_EURONEXT_INFO_PARAMETERS string in wget and it gave me exactly what I needed. The file containing stock data is written on the disk.

Here is the full wget line:

Code:

wget --post-data "format=3&layout=2&decimal_separator=1&date_format=1&op=Go&form_build_id=form-80ac117839ec70d62168953ce0b91657&form_id=nyx_download_form" "europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&form Key=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b1" --no-check-certificate -O TEST.csv

Thank you so much Frank, you are a life saver.

With kind regards,
Boris

Finding the URL to "Go" button?

Developer Shed Advertisers and Affiliates