Safely create a file if and only if it does not exist with Python

Question

I wish to write to a file based on whether that file already exists or not, only writing if it doesn't already exist (in practice, I wish to keep trying files until I find one that doesn't exist).

The following code shows a way in which a potentially attacker could insert a symlink, as suggested in this post in between a test for the file and the file being written. If the code is run with high enough permissions, this could overwrite an arbitrary file.

Is there a way to solve this problem?

import os
import errno

file_to_be_attacked = 'important_file'

with open(file_to_be_attacked, 'w') as f:
    f.write('Some important content!\n')

test_file = 'testfile'

try:
    with open(test_file) as f: pass
except IOError, e:

    # Symlink created here
    os.symlink(file_to_be_attacked, test_file)

    if e.errno != errno.ENOENT:
        raise
    else:
        with open(test_file, 'w') as f:
            f.write('Hello, kthxbye!\n')

Check atomic writing with Python stackoverflow.com/questions/2333872/… — Mikko Ohtamaa
– Mikko Ohtamaa, Commented Jun 11, 2012 at 11:19
Ah ok. I understood what's the matter... you write ONLY if the file exist? — Mikko Ohtamaa
– Mikko Ohtamaa, Commented Jun 11, 2012 at 11:23
Could you write the file in a temporary location, then do a copy command without allowing overwrite? — Eric
– Eric, Commented Jun 11, 2012 at 11:33

Community · Accepted Answer · 2017-05-23 12:18:07Z

102

Edit: See also Dave Jones' answer: from Python 3.3, you can use the x flag to open() to provide this function.

Original answer below

Yes, but not using Python's standard open() call. You'll need to use os.open() instead, which allows you to specify flags to the underlying C code.

In particular, you want to use O_CREAT | O_EXCL. From the man page for open(2) under O_EXCL on my Unix system:

Ensure that this call creates the file: if this flag is specified in conjunction with O_CREAT, and pathname already exists, then open() will fail. The behavior of O_EXCL is undefined if O_CREAT is not specified.

When these two flags are specified, symbolic links are not followed: if pathname is a symbolic link, then open() fails regardless of where the symbolic link points to.

O_EXCL is only supported on NFS when using NFSv3 or later on kernel 2.6 or later. In environments where NFS O_EXCL support is not provided, programs that rely on it for performing locking tasks will contain a race condition.

So it's not perfect, but AFAIK it's the closest you can get to avoiding this race condition.

Edit: the other rules of using os.open() instead of open() still apply. In particular, if you want use the returned file descriptor for reading or writing, you'll need one of the O_RDONLY, O_WRONLY or O_RDWR flags as well.

All the O_* flags are in Python's os module, so you'll need to import os and use os.O_CREAT etc.

Example:

import os
import errno

flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY

try:
    file_handle = os.open('filename', flags)
except OSError as e:
    if e.errno == errno.EEXIST:  # Failed as the file already exists.
        pass
    else:  # Something unexpected went wrong so reraise the exception.
        raise
else:  # No exception, so the file must have been created successfully.
    with os.fdopen(file_handle, 'w') as file_obj:
        # Using `os.fdopen` converts the handle to an object that acts like a
        # regular Python file object, and the `with` context manager means the
        # file will be automatically closed when we're done with it.
        file_obj.write("Look, ma, I'm writing to a new file!")

edited May 23, 2017 at 12:18

CommunityBot

11 silver badge

answered Jun 11, 2012 at 11:53

me_and

15.8k8 gold badges65 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Mattie B Over a year ago

+1 for the obviously correct answer. I'm personally curious to know how many people actually have issues with the NFS caveat—I (perhaps recklessly) dismiss it as an obsolete environment my code should never be run on.

Fred Foo Over a year ago

@zigg: NFSv3 is from 1995, so it seems fair to regard older versions as obsolete.

me_and Over a year ago

I'd be more worried about the kernel version, personally. If you're running anything even vaguely resembling an up-to-date system, you should have no issue, but RHEL 3 (still in extended support phase) is running a 2.4 kernel, for example. Also, I've not investigated if they provide atomic writes on Windows on FAT or NTFS, which is a potentially major limitation.

Henry Gomersall Over a year ago

@me_and The python page on open flag constants suggests that this works fine with Windows. I'll be trying it shortly!

me_and Over a year ago

True, but I've not seen anywhere (including MSDN) that explicitly says these flags give atomic file creation. Possibly I'm being overly paranoid, but I'd want to see that "atomic" keyword before trusting this for anything that's security-critical.

|

Dave Jones · Accepted Answer · 2017-07-14 19:54:07Z

93

For reference, Python 3.3 implements a new 'x' mode in the open() function to cover this use-case (create only, fail if file exists). Note that the 'x' mode is specified on its own. Using 'wx' results in a ValueError as the 'w' is redundant (the only thing you can do if the call succeeds is write to the file anyway; it can't have existed if the call succeeds):

>>> f1 = open('new_binary_file', 'xb')
>>> f2 = open('new_text_file', 'x')

For Python 3.2 and below (including Python 2.x) please refer to the accepted answer.

edited Jul 14, 2017 at 19:54

answered Aug 27, 2013 at 20:26

Dave Jones

1,14011 silver badges11 bronze badges

4 Comments

Dan Lenski Over a year ago

Good suggestion. Unfortunately this appears to be POSIX-only (doesn't work on Windows): Python 3.2 (r32:88445, Feb 20 2011, 21:30:00) [MSC v.1500 64 bit (AMD64)] on win32 >>> open("c:/temp/foo.csv","wx") ValueError: invalid mode: 'wx'

Dave Jones Over a year ago

You're using python 3.2; the 'x' mode is in 3.3 and above but it is cross platform. Incidentally, you only use 'x' instead of 'wx' - the write mode is redundant as the only thing you could do with the file is write to it anyway

Dubslow Over a year ago

How are 'w' and 'x' duplicates? It's perfectly reasonable to open an existing file for writing (which overwrites it).

Dave Jones Over a year ago

It is reasonable to open an existing file for writing, but the entire point of the 'x' mode is to open the file if and only if it doesn't exist already, failing with an error when the file does exist. This is why it is redundant with the 'w' flag; if it succeeds the file is guaranteed to be empty (and hence there's very little point reading from it :).

Peter Mortensen · Accepted Answer · 2021-03-25 23:02:11Z

-3

This code will easily create a file if one does not exists.

import os
if not os.path.exists('file'):
    open('file', 'w').close()

edited Mar 25, 2021 at 23:02

Peter Mortensen

31.4k22 gold badges110 silver badges134 bronze badges

answered Mar 14, 2013 at 4:19

user2033758

1,9863 gold badges16 silver badges17 bronze badges

3 Comments

Henry Gomersall Over a year ago

Yes it will. The important point about the question was the safety aspect. The problem is that between identifying the presence of the file and using it or creating it, something might change that results in a bad outcome (as in the original question).

Rad Over a year ago

That's true. It's called TOCTOU!

Open AI - Opting Out Over a year ago

If another process creates and writes to the file after the if statement, this code will blank out the file.

Collectives™ on Stack Overflow

Safely create a file if and only if it does not exist with Python

3 Answers 3

Example:

11 Comments

4 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Example:

11 Comments

4 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related