How do I handle errors when overwriting an existing file in python

Question

When I try to write to a file that already exists, but the write fails, the file gets overwritten with nothing (i.e. it gets cleared - contents of the existing file are deleted).

Example:

alaa@vernal:~/Test$ echo "sometext" > testfile
alaa@vernal:~/Test$ cat testfile 
sometext

In Python:

with open('testfile', 'w') as f:
    f.write(doesntexist)

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
NameError: name 'doesntexist' is not defined

The file is empty:

alaa@vernal:~/Test$ cat testfile 
alaa@vernal:~/Test$

I tried to handle the exception like this but it still empties the file:

import traceback

with open('testfile', 'w') as f:
    try:
        f.write(doesntexist)
        print('OK')
    except:
        print('NOT OK')
        traceback.print_exc()

NOT OK
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
NameError: name 'doesntexist' is not defined

I tried handling it in a different way, but it still empties the file:

import traceback

try:
    with open('testfile', 'w') as f:
        f.write(doesntexist)
        print('OK')
except:
    print('NOT OK')
    traceback.print_exc()

NOT OK
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
NameError: name 'doesntexist' is not defined

I feel like I'm missing something. Is there a straight forward way of handling this situation in Python to not clear the file if the write fails for any reason? Or do I have to resort to 'cleverer' code, for example: read the contents of the file first, and if the write fails, write it back again with the original contents, or maybe attempt to write the data to a temp file first, etc.?

You need to either define a variable doesntexist which holds the string to output, e.g. doesntexist=“text to write” or put quotes around doesntexist if you want that as the text to be output e.g. open(“myfile.txt”).write(“doesntexist”) — DisappointedByUnaccountableMod
– DisappointedByUnaccountableMod, Commented Jun 25, 2020 at 19:00
"When I try to write to a file that already exists, but the write fails, the file gets overwritten with nothing (i.e. it gets cleared - contents of the existing file are deleted)." What do you want to do instead? — Code-Apprentice
– Code-Apprentice, Commented Jun 25, 2020 at 19:08
I want it to not clear the file if my write to it fails. In my question, I show that the testfile has sometext in it, but after the failed write in Python, the file was cleared. I want sometext to still exist in it. — Alaa Ali
– Alaa Ali, Commented Jun 25, 2020 at 19:10
@barny I know that I have to define it. This was just an example. My question is how do I avoid clearing the contents of the file if it already if, for any reason, the write fails. For example if I change it to f.write(['some', 'array']) the write will fail with TypeError: expected a string or other character buffer object which is expected, but if the file I'm trying to write to existed and had content inside it, the contents of the file are cleared. My question is how can I avoid that? — Alaa Ali
– Alaa Ali, Commented Jun 25, 2020 at 19:14
It would not. I don't want to append to the file, I want to overwrite it, but I don't want to modify the contents of the existing file if the write fails for any reason. — Alaa Ali
– Alaa Ali, Commented Jun 25, 2020 at 19:17

chepner · Accepted Answer · 2020-06-25 19:34:55Z

open(..., 'w') immediately truncates the file, before you even have a chance to see if the subsequent writes succeeds.

Assuming you aren't required to modify the existing file in-place, it is far simpler and safer to create a new file that you will rename once the write has succeeded. You can wrap this in a context manager to simplify your business logic.

from tempfile import NamedTemporaryFile
from contextlib import contextmanager


def safe_write(fname, dir=None):
       
    if dir is None:
        dir = os.path.dirname(fname)
    try:
        try:
            f = None
            with NamedTemporaryFile('w', 
                                    dir=dir,
                                    delete=False) as f:
                yield f
        else:
            os.rename(f.name, fname)
    finally:
        try:
            if f is not None:
                os.remove(f.name)
        except FileNotFoundError:
            pass  # It doesn't exist if we successfully renamed it.


with safe_write('testfile') as f:
    f.write(doesntexist)

We create the temporary file in the same directory as the original to ensure they live on the same file system (but allow an explicit directory to be named if using the target directory isn't appropriate). delete=False ensures we still have a file to rename after we are done with the file.

The only thing this code tries to do with the original file is replace it atomically with the temporary file: the rename either succeeds, or leaves you with the original file still intact. Atomic renames are only possible within a file system, though. Moving a file to another file system requires actually copying the data in the file, rather than just updating its file-system entry.

Safely editing a file in-place is trickier and not something I'll attempt to discuss here.

Ondrej K. · Accepted Answer · 2020-06-25 19:37:02Z

2

The exception that you're handling happens already after the file has been open for writing and truncated.

open('testfile', 'w')

Does open the file for writing and truncates it. See docs:

mode is an optional string that specifies the mode in which the file is opened... 'w' for writing (truncating the file if it already exists)...

Then:

f.write(doesntexist)

raises NameError because you're trying to access a variable doesntexist which indeed doesn't exist. At this point (handling this exception) there isn't anything you could do to change behavior of open.

As to what to do, sort of depends on exact use case, there are other options.

For the simple example above (simply overwrite file with content of a str variable if it exists), you could do this for instance:

from pathlib import Path
try:
    Path("testfile").write_text(doesntexist)
    print("OK")
except NameError:
    print("Not OK")

This way the exception is raised when calling the method and before file gets opened and write is attempted.

You could open the file for appending with a mode, but then the writes are of course appended.

You can prepare content to write making sure you have all bits and only open and write when you know it's safe to do so.

...

In other words, it's not the write that fails... and write as such failing after the open succeeds is relatively less likely (you could run out of space on the device for instance ENOSPC - you'd see OSError).

The specific failure you're seeing is due to accessing non-existing variable, so it's not about what happens prior to your write code, you can also:

stuff_to_write = doesntexist
with open('testfile', 'w') as f:
    f.write(stuff_to_write)

Which would hit that NameError before you open and truncate.

Or:

try:
    stuff_to_write = doesntexist
    with open('testfile', 'w') as f:
        f.write(stuff_to_write)
except Exception:
    print('Oops')

Even though that's a little silly.

Actually the specific case of variable not existing can either happen by simple omission (and any linter or check or just running the script should discover that) or the variable only being assigned to inside a conditional block which I would consider the correct place to address that problem.

edited Jun 25, 2020 at 19:37

answered Jun 25, 2020 at 19:02

Ondrej K.

9,83911 gold badges31 silver badges46 bronze badges

4 Comments

DisappointedByUnaccountableMod Over a year ago

Please explain why exception NameError is raised?

chepner Over a year ago

@barny Because the name doesntexist doesn't exist.

chepner Over a year ago

There are other things that can go wrong besides a NameError being raised before write_text is even called,though, and write_text will truncate the file immediately just the same as open(..., 'w'). (In fact, write_text is little more than a wrapper around such a call).

Ondrej K. Over a year ago

Sure. I can hit EACCES (-> OSError) for instance. The question is... should I risk murking the answer with relatively clearly given example in the question? Would it be more or less helpful? But I sure should not condone or promote practice of naked except from the question. :)

cemysce · Accepted Answer · 2023-09-05 21:41:36Z

I don't have the reputation to comment on an existing post, and I'd rather not edit someone else's answer, so I'm just going to post this separate answer.

@chepner's answer is I think the right solution, but there's a couple of mistakes and opportunities for simplification:

safe_write is missing the @contextmanager decorator.
The inner try-else block is invalid, so it should be changed to one of the following:
- try-except-else (in which case the except should just do a raise to re-raise the existing exception)
- Consolidate the inner try block with the outer one. I believe there is no point to an else clause in a try block if you're not going to handle the exception, so the contents of the else block could just happen immediately after the with block.

Also, os.replace will clobber the destination on all POSIX and Windows, whereas os.rename will only do so on POSIX platforms.

Finally, and this is personal preference since I know in Python it's perfectly okay to rely on exceptions as a means of control flow, but I've tweaked the code to just use f as the indicator of whether the file was successfully renamed.

Going with the approach of consolidating the try blocks, the overall solution would be:

from tempfile import NamedTemporaryFile
from contextlib import contextmanager

@contextmanager
def safe_write(fname, dir=None):
    if dir is None:
        dir = os.path.dirname(fname)
    try:
        f = None
        with NamedTemporaryFile('w', 
                                dir=dir,
                                delete=False) as f:
            yield f
        os.replace(f.name, fname)
        f = None
    finally:
        if f is not None:
            os.remove(f.name)

with safe_write('testfile') as f:
    f.write(doesntexist)

Collectives™ on Stack Overflow

How do I handle errors when overwriting an existing file in python

3 Answers 3

Comments

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related