#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    13
    Rep Power
    0

    Make python reads data from several text files in fast way


    hi, i need your help for what follows.
    i have a serie of files with data (ascii), they are results of a cfd simulation. each file contains results for the same x-axis, when y-axis changes.
    i must create a script, that loads all the results i have, id est all the files in my folder
    these files have the same name, with a different index (ie x1_p, x2_p, x3_p...).

    is there a way to make python loads the files in a fast way, so that i don't have to put 100 names by hand in the script?

    thanks a lot
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2012
    Posts
    39
    Rep Power
    2
    Use the os module to get a list of all the files and open them in turn.

    Example
    Code:
    import os
    for filename in os.listdir("c:\\my\\dir"):
        with open(filename) as f:
            # process file
    You may also want to use os.walk if you are going through subdirectories and the glob module if you want to do pattern matching on the file.
    Last edited by Quackajack; November 21st, 2012 at 10:14 AM. Reason: is should have been in
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    13
    Rep Power
    0
    thanks a lot, it seems useful!

    let me, please, better understand how does it work.
    first of all, i can put the script in the same folder (let's call it "sim") of mi data text files?

    the instruction "filename" is to use or it is a jolly name i have to substitute with something else?
  6. #4
  7. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,708
    Rep Power
    480
    Code:
    import os
    for filename in os.listdir("/tmp"):
        with open(filename) as f:
            print(f.readline())
    For the part in quotes substitute the name of a directory that makes sense on your system.

    Run this program, it will print the first line of every file. (or it will run into a special file and die dramatically, but oh well.)
    [code]Code tags[/code] are essential for python code and Makefiles!
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    13
    Rep Power
    0
    it gives:

    Code:
    Traceback (most recent call last):   File "<stdin>", line 2, in <module> IOError: [Errno 2] No such file or directory: 'x11_p.xy'
    and is not the first file in the directory. I don't understand this error, nothing else happens, anyway it had worked i think ...
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2012
    Posts
    39
    Rep Power
    2
    filename is the variable. You can call it anything you like. Breaking it down further, I can list all the files in the current directory (.) with the following

    Code:
    import os
    for filename in os.listdir("."):
        print filename
    You can choose what directory to list the files in by changing the parameter passed to listdir.

    Now that you have the filename you can check it matches a pattern so may want to you only want to open files that end in _p. There are other ways of doing this (see glob for example) but this is probably easier to understand to start with.

    As an example, the following checks each file in the current directory and if it is a text file, ends with .txt, prints out the file name followed by the first line of the text file.

    Code:
    import os
    for filename in os.listdir("."):
        if filename.endswith(".txt"):
            with open(filename) as datafile:
                print filename , ":" , datafile.readline()
    There is no error checking in the above so don't be surprised if it dies.
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    13
    Rep Power
    0
    excellent, it seems really what i need; i tried it and works well.

    thanks again a lot!
  14. #8
  15. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    13
    Rep Power
    0
    I notice that it takes correctly the files, but in a random order.
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2012
    Posts
    39
    Rep Power
    2
    If you refer to the online documentation for listdir it says the list is in arbitrary order.

    If you want it sorted alphabetically try replacing the for line with the following three:
    Code:
    myfiles = os.listdir(".")
    myfiles.sort()
    for filename in myfiles:
    This may still not give you the list in the order you want, depending upon the file naming convention. For example if the files are called x1_p to x11_p the sorted list will look like this:
    [ 'x10_p' , 'x11_p' , 'x1_p' , 'x2_p' , ... , 'x9_p']

    In this case if you can't change the naming convention you could look at using one of the optional arguments for sort to specify the comparison function or the key. The online documentation is your friend.
  18. #10
  19. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    13
    Rep Power
    0
    sure, thanks!

IMN logo majestic logo threadwatch logo seochat tools logo