Python unittest: directory and file paths

Question

I have the following file structure for a library I'm developing that exposes a keras model:

relevancy (repo)
    relevancy (package repo)
        data
            model.h5
            tokenizer.pickle
        test
            __init__.py
            test_model.py
        model.py
        __init__.py
    __init__.py
    setup.py

The library basically loads the pre-trained tokenizer.pickle and model.h5 and make predictions on input data.

Within model.py, I have a function with the following code that loads the tokenizer and model:

def load()
    with open("data/tokenizer.pickle", "rb") as f:
        tokenizer = pickle.load(f)
    model = keras.models.load_model("data/model.h5")
    return tokenizer, model

In test_model.py, I'm calling this function in my tests.

Then if I call python setup.py test under /relevancy (repo), I will get error saying that data/tokenizer.pickle is not found. Apparently, the relative is causing the problem.

How should I setup my directory or paths so that the tokenizer and model can always be loaded correctly?

larsks · Accepted Answer · 2018-10-31 19:49:29Z

2

If you need to access data files stored inside your package, consider using the pkg_resources module.

Then in model.py you can do something like this:

filename = pkg_resources.resource_filename(__name__, 'data/tokenizer.pickle')
with open(filename, 'rb') as f:
    ...

answered Oct 31, 2018 at 19:49

larsks

318k49 gold badges473 silver badges482 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Yilun Zhang Over a year ago

This works quite well! Thanks! I just have one more question: I set it like this and I ran python test_model.py from tests directory, it's showing that the data paths are from the top directory relevancy/data/tokenizer.pickle of the library, so it threw me no file or directory error. Is this normal? Is there a solution to make this work at the same time?

Collectives™ on Stack Overflow

Python unittest: directory and file paths

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related