September 30th, 2013, 10:21 AM
SEC Edgar Mass Download Help
The SEC has an FTP site for downloading the files from their EDGAR site. I need to download all files from ~4500 directories... I have the directory paths, and I am hoping there is a way to automate the download process.
Ideally, I would like to copy each directory (there is 1 directory per filer) into a separate directory on my computer (as opposed to copying 4500 directories into 1 directory).
I barely know any programming (a little VB) and I did my first FTP download (using Filezilla) 2 days ago, so this area is new to me.
Is there a (relatively) simple way to automate the copying of the directories using free windows-based tools (like Filezilla)?
September 30th, 2013, 05:49 PM
This is a tricky question. Anyway you do this the FTP program or script would have to download all files and place into root directory. Each one of those downloads is a command sent through FTP.
It may be just as easy to just queue each file and let the ftp run.
However the other option is to just download all the files and folders and manipulate the files on the local computer afterward.
I would suggest a program called Flatfolder. http://flatfolder.sourceforge.net/
September 30th, 2013, 08:49 PM
Thanks - I was hoping to find a way to automate as 4500 manual transfers will take quite a while to load into the queue, but I'll take a look at flatfolder.
October 1st, 2013, 10:33 AM
If you use a program like filezilla you should be able to just select all the files and queue them up automatically.
October 1st, 2013, 12:35 PM
If you feel up to it, this is a very simple programming task. Some languages (e.g. python, ruby etc.) already come with a built-in FTP module and some others (such as perl, C, C++ etc.) have 3rd party libraries that make this a breeze. So it isn't that hard to program either and there are plenty of examples on the web on how to do so. For example:
Hell, python even comes with an example ftp mirror script, which does exactly what you want. All you have to do is install python on your computer and then look for ftpmirror.py under the tools/scripts sub-directory after installation. This example program downloads a tree from a remote server to your local computer, which is exactly what you want. No programming necessary, you can just use the provided example!
If you want something a bit more robust (i.e. something that mirrors in both directions), there are also 3rd party programs such as this:
Up the Irons
What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
"Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
Down with Sharon Osbourne
"I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo
October 1st, 2013, 09:21 PM
Thanks. I'll have a play with the code examples and see if I can get it to work.