#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    3
    Rep Power
    0

    SEC Edgar Mass Download Help


    The SEC has an FTP site for downloading the files from their EDGAR site. I need to download all files from ~4500 directories... I have the directory paths, and I am hoping there is a way to automate the download process.

    Ideally, I would like to copy each directory (there is 1 directory per filer) into a separate directory on my computer (as opposed to copying 4500 directories into 1 directory).

    I barely know any programming (a little VB) and I did my first FTP download (using Filezilla) 2 days ago, so this area is new to me.

    Question:

    Is there a (relatively) simple way to automate the copying of the directories using free windows-based tools (like Filezilla)?

    Thanks.
    Jim
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2004
    Location
    West Coxsackie, NY
    Posts
    37
    Rep Power
    11
    This is a tricky question. Anyway you do this the FTP program or script would have to download all files and place into root directory. Each one of those downloads is a command sent through FTP.
    It may be just as easy to just queue each file and let the ftp run.

    However the other option is to just download all the files and folders and manipulate the files on the local computer afterward.

    I would suggest a program called Flatfolder. http://flatfolder.sourceforge.net/
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    3
    Rep Power
    0

    Thanks


    Thanks - I was hoping to find a way to automate as 4500 manual transfers will take quite a while to load into the queue, but I'll take a look at flatfolder.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2004
    Location
    West Coxsackie, NY
    Posts
    37
    Rep Power
    11
    If you use a program like filezilla you should be able to just select all the files and queue them up automatically.
  8. #5
  9. Banned ;)
    Devshed Supreme Being (6500+ posts)

    Join Date
    Nov 2001
    Location
    Woodland Hills, Los Angeles County, California, USA
    Posts
    9,607
    Rep Power
    4247
    If you feel up to it, this is a very simple programming task. Some languages (e.g. python, ruby etc.) already come with a built-in FTP module and some others (such as perl, C, C++ etc.) have 3rd party libraries that make this a breeze. So it isn't that hard to program either and there are plenty of examples on the web on how to do so. For example:
    http://docs.python.org/2/library/ftplib.html

    Hell, python even comes with an example ftp mirror script, which does exactly what you want. All you have to do is install python on your computer and then look for ftpmirror.py under the tools/scripts sub-directory after installation. This example program downloads a tree from a remote server to your local computer, which is exactly what you want. No programming necessary, you can just use the provided example!

    If you want something a bit more robust (i.e. something that mirrors in both directions), there are also 3rd party programs such as this:
    http://code.activestate.com/recipes/...le-ftp-mirror/
    Up the Irons
    What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
    "Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
    Down with Sharon Osbourne

    "I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo
  10. #6
  11. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    3
    Rep Power
    0
    Thanks. I'll have a play with the code examples and see if I can get it to work.

IMN logo majestic logo threadwatch logo seochat tools logo