Writing to a file within a for loop in python

Question

Below is part of a script that I wrote which takes each item in a list, passes it to a function, stores the output of the function as a variable with a line break at the end and then writes the variable to a new line in a file. It also prints the variable to the console.

for i in lane_list:
        out = count_genes(count, i, reader, total_genes) + '\n'
        count += 1
        outfile.write(out)
        print out

The script works but not in the way that I would expect. I can see the output being printed to the console as the script runs so I know the rate at which the script is running through the for loop and I assumed that python would write to the file at the same rate as it prints output to the console. What's odd is that for long periods of time nothing will be written to the file (I can see this by 'cating' the file in the console as the script's running) and then several hundred more lines will appear in the file at once. It's as if python is storing what it has to write to the file for a period of time and then writing all of it in one go. What was even more surprising is sometimes when I cat the file as the scripts running it will have written half of the last line but not all of it.

Can anyone explain to me why this is?

Read about flushing: stackoverflow.com/questions/3167494/…. Also it maybe helpful to use tail -f filename to get live preview of file (if you are working on linux). — Szymon P.
– Szymon P., Commented Mar 23, 2017 at 14:10

Holloway · Accepted Answer · 2017-03-23 14:13:14Z

1

As @szymon mentioned, the file is buffered as it is written. If you want to see it being written 'live', use open('/path/to/file'', 'w', 0) where the 0 means unbuffered.

For python3 use 1 for line buffered as unbuffered is only allowed for binary data.

answered Mar 23, 2017 at 14:13

Holloway

7,5471 gold badge29 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ben Jeffrey Over a year ago

Thanks all, is there any reason not to use this? Will it slow down for example?

Szymon P. Over a year ago

It can slow down - every time you save to the file it hits the disk (I/O operations). When buffer size is set, script hits the disk only when buffer is full (and flushes it).

Collectives™ on Stack Overflow

Writing to a file within a for loop in python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related