#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10

    Get a List of Files from a URL


    I was wondering how would I get a list of files from a url directory

    eg

    www.website.com/images/

    ?

    I can do this for a local directory with the following

    File folder = new File("C:/images/");
    File[] listOfFiles = folder.listFiles();

    for (int i = 0; i < listOfFiles.length; i++) {
    if (listOfFiles[i].isFile()) {
    System.out.println("File " + listOfFiles[i].getName());
    } else if (listOfFiles[i].isDirectory()) {
    System.out.println("Directory " + listOfFiles[i].getName());
    }
    }

    I'm just trying to get a list of images from a directory on my site and then download them the a local directory. I've worked out how to download a singular image from a url and save it to a local folder but the amount of images is uncertain.

    Any thoughts on how this could be done?

    Thanks
  2. #2
  3. Daniel Schildsky
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Mar 2004
    Location
    KL, Malaysia.
    Posts
    1,555
    Rep Power
    1621

    Local HD directory vs remote URL


    Originally Posted by doush.
    I was wondering how would I get a list of files from a url directory

    eg

    www.website.com/images/

    ?

    I can do this for a local directory with the following

    File folder = new File("C:/images/");
    File[] listOfFiles = folder.listFiles();

    for (int i = 0; i < listOfFiles.length; i++) {
    if (listOfFiles[i].isFile()) {
    System.out.println("File " + listOfFiles[i].getName());
    } else if (listOfFiles[i].isDirectory()) {
    System.out.println("Directory " + listOfFiles[i].getName());
    }
    }

    I'm just trying to get a list of images from a directory on my site and then download them the a local directory. I've worked out how to download a singular image from a url and save it to a local folder but the amount of images is uncertain.

    Any thoughts on how this could be done?

    Thanks
    The reason you could retrieve the list of files from the sample codes you provided is that you are retrieving from a local file system directory. As far as I know, it is not possible to retrieve the list of files from a remote url pointing to a site.

    Even if it is possible, it is subject to the server settings at the remote site, as I believe most of the servers do not allow their directories being listed for strangers to access randomly, unless it is a site meant for file download.
    When the programming world turns decent, the real world will turn upside down.
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10
    Yeah that's what I was thinking. It is my own site if that helps I can set the folder permissions etc. But I would rather not so others can't freely view my files. Hmmm I'll have to try something else.

    Thanks again
  6. #4
  7. Santosh Vaza
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2010
    Location
    JHome
    Posts
    356
    Rep Power
    13
    in most of browsers when you type the url which has bunch of images like http://localhost/Images/ it gives me list of folder contents, also if you type the full path http://localhost/Images/Test.jpg then it gets the image and displays it to browser, Try whether if possible to retrive the image using full path(what i suggest) else see what html is returned by server if possible you can obtain the full path by parsing the html.
    The thing done by browser can be done programatically !!, its how it runs.
    EDIT:
    I also agree with
    Originally Posted by tvc3mye
    Even if it is possible, it is subject to the server settings at the remote site, as I believe most of the servers do not allow their directories being listed for strangers to access randomly, unless it is a site meant for file download
    most of servers has default document configured for a specific directory , if you encounter such cases then its impossible to get all files in folder, depending on security settings configured on server.
    Last edited by codeJ; August 11th, 2010 at 09:42 AM.
    no one can become perfect by merely ceasing the act
  8. #5
  9. Daniel Schildsky
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Mar 2004
    Location
    KL, Malaysia.
    Posts
    1,555
    Rep Power
    1621

    Workaround - possible solution


    There are exceptions in all cases, including this one.

    Unless you are trying to retrieve your own site from a client end point of your own application, a work around is possible:

    Have your server deploy a servlet to retrieve a list of files from the folder specified by the request it receives. At your client end point, your application sends a request to the server by providing a path (virtual? relative ?) you intend to list. The servlet will return the list of files in the requested path, retrieved from the server's OS. Then, it serialises the file list to the client end point for further processing.

    This kind of application design is risky and error prone, as your application exposes the requested path to intruders, which may eavesdrop or intercept the request and manipulate the file path to retrieve files they were not supposed to access. One should seriously consider redesigning the entire application for a more robust approach.
    When the programming world turns decent, the real world will turn upside down.
  10. #6
  11. Santosh Vaza
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2010
    Location
    JHome
    Posts
    356
    Rep Power
    13
    Originally Posted by tvc3mye
    This kind of application design is risky and error prone, as your application exposes the requested path to intruders, which may eavesdrop or intercept the request and manipulate the file path to retrieve files they were not supposed to access. One should seriously consider redesigning the entire application for a more robust approach.
    you can use userid and password type of thing, For getting the list of files.
    authentication page -> list of files...
    no one can become perfect by merely ceasing the act
  12. #7
  13. Daniel Schildsky
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Mar 2004
    Location
    KL, Malaysia.
    Posts
    1,555
    Rep Power
    1621

    Security Risks


    Even if you password protect or better still, encrypt the data by transferring the data via HTTPS, it is still possible for an intruder to manipulate your data.

    Plus, as a good principle in application design, the application should not disclose any of its private or internal details to external systems. Any need to expose internal resource to outside world would be a program design flaw.
    When the programming world turns decent, the real world will turn upside down.
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10
    Originally Posted by tvc3mye
    Even if you password protect or better still, encrypt the data by transferring the data via HTTPS, it is still possible for an intruder to manipulate your data.

    Plus, as a good principle in application design, the application should not disclose any of its private or internal details to external systems. Any need to expose internal resource to outside world would be a program design flaw.
    Yeah I have to agree. Looks like back to the drawing board for me. I think I have another way worked out.

    Thanks everyone for your opinions, they all really helped.
  16. #9
  17. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2013
    Posts
    1
    Rep Power
    0
    did you get a workaround?
  18. #10
  19. Contributing User
    Devshed Expert (3500 - 3999 posts)

    Join Date
    Aug 2010
    Location
    Eastern Florida
    Posts
    3,719
    Rep Power
    348
    This is a 3 year old thread. You should start a new thread with your question/problem.

IMN logo majestic logo threadwatch logo seochat tools logo