#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0

    Question Finding the URL to "Go" button?


    Dear all, I'm not a developer and I also can't read JavaScript.

    I hope someone will be so kind to help me with the interpretation of the JavaScript code.

    THE PROBLEM DESCRIPTION:
    I would like to find a direct link for downloading a .csv file for use with wget command:
    Code:
    europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&formKey=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b  1


    This is a harmless .csv file, which contains daily stock data from the NYSE Euronext web page. When clicking the "Go" button on this page, I'm asked to "Open" or "Save" the file.

    I have scanned through the source code of the page in hope to be able to compile the direct link to the file, but I was not able to understand what this code does.

    Please have a look at it. If someone at least can tell me if this is possible that would already help a lot.

    Thanks in advance!
  2. #2
  3. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2012
    Posts
    9
    Rep Power
    0
    The direct link to the file is what will be in your URL bar if you select OPEN (assuming that your browser opens the file in the browser and not in an external application). For example, when I click GO in Chrome with the defaults selected, I am taken to europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&formKey=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b 1
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0
    Originally Posted by gsharp
    The direct link to the file is what will be in your URL bar if you select OPEN (assuming that your browser opens the file in the browser and not in an external application). For example, when I click GO in Chrome with the defaults selected, I am taken to europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&formKey=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b 1
    Thanks gsharp, but it is not that simple.
    Yes wget will download something, but not what I expect to see. The result is a file containing the web page source code, rather than the stock price quotes.
    Code:
    wget "europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&form Key=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b1" --no-check-certificate -O TEST.csv
    Unfortunately that does not work. The user choices must be included in the URL string too (something like typefile=csv&layout=vertical&typedate=dmy etc.).

    I figured out how to compile an URL with the correct user choices for downloading historical data of the individual stocks. Here is an example:
    Code:
    wget "europeanequities.nyx.com/nyx_eu_listings/price_chart/download_historical?typefile=csv&layout=vertical&typedate=dmy&separator=point&mic=XAMS&isin=NL0006034001&name=ASML HOLDING&namefile=Price_Data_Historical& from=1283299200000&to=1349308800000&adjusted=1&base=0" --no-check-certificate -O ASML.csv
    Although, I managed somehow to interpret the source code of the stock history applet, with right download choices and compile the straight forward URL download string, I'm not able to do the same for the daily stock quotes .

    Any suggestions?
  6. #4
  7. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    0

    same issue


    I'm exactly trying to find the same solution. The go button is triggering a javascript function which makes the request.
    I'm not famliar with javascript. It should be possible to find the URL by tracing the outgoing traffic with some tool or programs. I'm trying to find such trace tool.

    regards, Frank
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2012
    Posts
    9
    Rep Power
    0
    Originally Posted by fv967
    I'm exactly trying to find the same solution. The go button is triggering a javascript function which makes the request.
    I'm not famliar with javascript. It should be possible to find the URL by tracing the outgoing traffic with some tool or programs. I'm trying to find such trace tool.

    regards, Frank
    If you're using Chrome, this functionality is built in. Just hit F12 and check the Network tab.
  10. #6
  11. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    0

    Http Post


    Here's what I found out:
    The popup window is submitting an HTTP POST query. Before, it was an HTTP GET query.
    I wrote a small java program which sends a POST message (with appropriate post data) and the requested information was coming back correctly.
    The test I wrote was for retrieving stocks which are listed on each exchange (AM,BR,PA,LS).

    So, the solution it to write a small program which launches an HTTP/POST.

    reg, Frank
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0

    Red face


    Hello Frank,

    Would you be so kind to share your code on this thread?

    With kind regards,
    Boris
  14. #8
  15. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    0

    Code


    Hi Boris,

    There are two kinds of information you can download from euronext:
    1. All listed stocks for an Exchange
    2. Historic prices for a specific stock

    Herewith the code for nr.1.
    private static String URL_EURONEXT_INFO_PARAMETERS = "format=3&layout=2&decimal_separator=1&date_format=1&op=Go&form_build_id=form-80ac117839ec70d62168953ce0b91657&form_id=nyx_download_form";

    public static boolean postRequest(String urlString, StringBuffer strBufferHTML) throws IOException, ProtocolException, MalformedURLException {
    java.net.URL url;
    java.net.HttpURLConnection connection = null;
    try {
    url = new URL(urlString);
    connection = (java.net.HttpURLConnection) url.openConnection();
    connection.setRequestMethod("POST");
    connection.setUseCaches(false);
    connection.setDoInput(true);
    connection.setDoOutput(true);

    // Send request
    DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
    wr.writeBytes(URL_EURONEXT_INFO_PARAMETERS);
    wr.flush();
    wr.close();

    // Get Response
    InputStream is = connection.getInputStream();
    BufferedReader rd = new BufferedReader(new InputStreamReader(is));
    String line;
    while ((line = rd.readLine()) != null) {
    strBufferHTML.append(line);
    strBufferHTML.append('\r');
    }
    rd.close();

    } finally {
    if (connection != null) {
    connection.disconnect();
    }
    }
    return true;
    }
    I've historic prices (10 years) from several exchanges. Maybe you don't need to program it yourself.

    reg, Frank
  16. #9
  17. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2010
    Posts
    7
    Rep Power
    0

    Code


    Hi Boris,

    There are two kinds of information you can download from euronext:
    1. All listed stocks for an Exchange
    2. Historic prices for a specific stock

    Herewith the code for nr.1.
    private static String URL_EURONEXT_INFO_PARAMETERS = "format=3&layout=2&decimal_separator=1&date_format=1&op=Go&form_build_id=form-80ac117839ec70d62168953ce0b91657&form_id=nyx_download_form";

    public static boolean postRequest(String urlString, StringBuffer strBufferHTML) throws IOException, ProtocolException, MalformedURLException {
    java.net.URL url;
    java.net.HttpURLConnection connection = null;
    try {
    url = new URL(urlString);
    connection = (java.net.HttpURLConnection) url.openConnection();
    connection.setRequestMethod("POST");
    connection.setUseCaches(false);
    connection.setDoInput(true);
    connection.setDoOutput(true);

    // Send request
    DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
    wr.writeBytes(URL_EURONEXT_INFO_PARAMETERS);
    wr.flush();
    wr.close();

    // Get Response
    InputStream is = connection.getInputStream();
    BufferedReader rd = new BufferedReader(new InputStreamReader(is));
    String line;
    while ((line = rd.readLine()) != null) {
    strBufferHTML.append(line);
    strBufferHTML.append('\r');
    }
    rd.close();

    } finally {
    if (connection != null) {
    connection.disconnect();
    }
    }
    return true;
    }
    I've historic prices (10 years) from several exchanges. Maybe you don't need to program it yourself. Just let me know.

    reg, Frank
  18. #10
  19. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0

    Thumbs up


    Originally Posted by fv967
    Hi Boris,

    There are two kinds of information you can download from euronext:
    1. All listed stocks for an Exchange
    2. Historic prices for a specific stock

    Herewith the code for nr.1.


    I've historic prices (10 years) from several exchanges. Maybe you don't need to program it yourself. Just let me know.

    reg, Frank

    Hello Frank,

    Thank you, for sharing your code here. I shall try it out as soon as I find some time (probably it has to wait until the Christmas holidays).

    Concerning the historical prices, I already wrote a script for that one. Thanks for asking.

    With kind regards,
    Boris
  20. #11
  21. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2012
    Posts
    5
    Rep Power
    0

    Thumbs up


    Originally Posted by fv967
    Hi Boris,

    There are two kinds of information you can download from euronext:
    1. All listed stocks for an Exchange
    2. Historic prices for a specific stock

    Herewith the code for nr.1.


    I've historic prices (10 years) from several exchanges. Maybe you don't need to program it yourself. Just let me know.

    reg, Frank
    Hello Frank,

    I did not manage to run your program, but it's not your fault, it is my lack of know how.

    However, I used your URL_EURONEXT_INFO_PARAMETERS string in wget and it gave me exactly what I needed. The file containing stock data is written on the disk.

    Here is the full wget line:
    Code:
    wget --post-data "format=3&layout=2&decimal_separator=1&date_format=1&op=Go&form_build_id=form-80ac117839ec70d62168953ce0b91657&form_id=nyx_download_form" "europeanequities.nyx.com/nl/popup/data/download?ml=nyx_pd_stocks&cmd=default&form Key=nyx_pd_filter_values%3A802a59dc69b9f3f1e2a04435d8b478b1" --no-check-certificate -O TEST.csv
    Thank you so much Frank, you are a life saver.

    With kind regards,
    Boris

IMN logo majestic logo threadwatch logo seochat tools logo