Make python reads data from several text files in fast way

Dev Shed Forums Sponsor:

November 20th, 2012, 08:02 AM

giacomo84

Registered User

Join Date: Nov 2012

Posts: 13

Time spent in forums: 3 h 21 m 37 sec
Reputation Power: 0

Make python reads data from several text files in fast way

hi, i need your help for what follows.
i have a serie of files with data (ascii), they are results of a cfd simulation. each file contains results for the same x-axis, when y-axis changes.
i must create a script, that loads all the results i have, id est all the files in my folder
these files have the same name, with a different index (ie x1_p, x2_p, x3_p...).

is there a way to make python loads the files in a fast way, so that i don't have to put 100 names by hand in the script?

thanks a lot

November 20th, 2012, 08:51 AM

Quackajack

Contributing User

Join Date: Jul 2012

Posts: 35

Time spent in forums: 10 h 1 m 29 sec
Reputation Power: 1

Use the os module to get a list of all the files and open them in turn.

Example

Code:

import os
for filename in os.listdir("c:\\my\\dir"):
    with open(filename) as f:
        # process file

You may also want to use os.walk if you are going through subdirectories and the glob module if you want to do pattern matching on the file.

Last edited by Quackajack : November 21st, 2012 at 10:14 AM. Reason: is should have been in

November 21st, 2012, 07:47 AM

giacomo84

Registered User

Join Date: Nov 2012

Posts: 13

Time spent in forums: 3 h 21 m 37 sec
Reputation Power: 0

thanks a lot, it seems useful!

let me, please, better understand how does it work.
first of all, i can put the script in the same folder (let's call it "sim") of mi data text files?

the instruction "filename" is to use or it is a jolly name i have to substitute with something else?

November 21st, 2012, 10:02 AM

b49P23TIvg

Contributing User

Join Date: Aug 2011

Posts: 3,393

Time spent in forums: 1 Month 2 Weeks 3 Days 15 h 37 m 10 sec
Reputation Power: 383

Code:

import os
for filename in os.listdir("/tmp"):
    with open(filename) as f:
        print(f.readline())

For the part in quotes substitute the name of a directory that makes sense on your system.

Run this program, it will print the first line of every file. (or it will run into a special file and die dramatically, but oh well.)

__________________
[code]Code tags[/code] are essential for python code!

November 21st, 2012, 10:11 AM

giacomo84

Registered User

Join Date: Nov 2012

Posts: 13

Time spent in forums: 3 h 21 m 37 sec
Reputation Power: 0

it gives:

Code:

Traceback (most recent call last):   File "<stdin>", line 2, in <module> IOError: [Errno 2] No such file or directory: 'x11_p.xy'

and is not the first file in the directory. I don't understand this error, nothing else happens, anyway it had worked i think ...

November 21st, 2012, 10:37 AM

Quackajack

Contributing User

Join Date: Jul 2012

Posts: 35

Time spent in forums: 10 h 1 m 29 sec
Reputation Power: 1

filename is the variable. You can call it anything you like. Breaking it down further, I can list all the files in the current directory (.) with the following

Code:

import os
for filename in os.listdir("."):
    print filename

You can choose what directory to list the files in by changing the parameter passed to listdir.

Now that you have the filename you can check it matches a pattern so may want to you only want to open files that end in _p. There are other ways of doing this (see glob for example) but this is probably easier to understand to start with.

As an example, the following checks each file in the current directory and if it is a text file, ends with .txt, prints out the file name followed by the first line of the text file.

Code:

import os
for filename in os.listdir("."):
    if filename.endswith(".txt"):
        with open(filename) as datafile:
            print filename , ":" , datafile.readline()

There is no error checking in the above so don't be surprised if it dies.

November 22nd, 2012, 04:16 AM

giacomo84

Registered User

Join Date: Nov 2012

Posts: 13

Time spent in forums: 3 h 21 m 37 sec
Reputation Power: 0

excellent, it seems really what i need; i tried it and works well.

thanks again a lot!

November 22nd, 2012, 04:20 AM

giacomo84

Registered User

Join Date: Nov 2012

Posts: 13

Time spent in forums: 3 h 21 m 37 sec
Reputation Power: 0

I notice that it takes correctly the files, but in a random order.

November 22nd, 2012, 05:25 AM

Quackajack

Contributing User

Join Date: Jul 2012

Posts: 35

Time spent in forums: 10 h 1 m 29 sec
Reputation Power: 1

If you refer to the online documentation for listdir it says the list is in arbitrary order.

If you want it sorted alphabetically try replacing the for line with the following three:

Code:

myfiles = os.listdir(".")
myfiles.sort()
for filename in myfiles:

This may still not give you the list in the order you want, depending upon the file naming convention. For example if the files are called x1_p to x11_p the sorted list will look like this:
[ 'x10_p' , 'x11_p' , 'x1_p' , 'x2_p' , ... , 'x9_p']

In this case if you can't change the naming convention you could look at using one of the optional arguments for sort to specify the comparison function or the key. The online documentation is your friend.

#10

November 22nd, 2012, 10:01 AM

giacomo84

Registered User

Join Date: Nov 2012

Posts: 13

Time spent in forums: 3 h 21 m 37 sec
Reputation Power: 0

sure, thanks!

Make python reads data from several text files in fast way

Developer Shed Advertisers and Affiliates