0

I understand that in Python, relative imports are only allowed inside of packages. Personally, I find this functionality great; it makes it very easy to import stuff from other modules.

However, for files that are not inside a Package, relative imports are not allowed. A typical situation I encounter is in my small projects, which I often structure as follows :

my_project/
├── modules/
│   ├── __init__.py
│   ├── module1.py
│   └── module2.py
├── scripts/
    ├── script1.py
    └── script2.py

Now, from script1.py and script2.py, I'd want to import stuff from modules, but to do so I have to resort to convoluted ways, such as sys.path.append(Path(__file__).parent.parent.as_poxis()), which feel very weird.

It would feel very natural to just do from ..modules import whatever inside script1.py for example. I'm sure there is a good reason for why this isn't possible, but I am not sure why this is the case!

In summary, why are relative imports not allowed in Python scripts?

Improve this question
4
  • Python 3 allows relative imports, but there are some rules : docs.python.org/3/reference/… Commented Feb 10 at 15:43
  • the problem is that you should be installing your package (perhaps in editable mode). This is the problem with everyone being taught sys.append instead of how to actually package their projects. Commented Feb 10 at 16:03
  • Yes, I saw that this seems to be 'the way' to work with packages and imports. But if the script only makes sense in the context of my small project (e.g. a training script to train the models that are defined in modules), should I then actually add the scripts to the package itself ? If so, how do I run them as scripts now that they are modules ? (I am aware these are basic questions, i'd be happy with a reference link!) Commented Feb 10 at 17:09
  • 1
    @Frotaur you can do that! There are various ways to expose scrips as commands upon installation of a project (think fo many python libraries that do this, like the black formatter) But you don't have to, that's the point! If you actually package your project and install it (possibly in editable mode) then your scrips can jsut import the package anywhere Commented Feb 10 at 22:03

3 Answers 3

Reset to default
4

This is a general design decision. Full stop. I am not a core Python developper, so I can just try to guess the rationale behind it.

It is indeed frustrating... until you understand that everything works smoothly as soon as you start to correctly package your projects. The rule is that something requires more than one file (and what you have shown is already complex enough...) it should be packaged according to the PyPA rules.

Not only you immediately gain relative imports because the whole project is a package, but installing the project on a new system and have it immediately accessible becomes a piece of cake. You just declare any script as command line entry points, and they will end in the path immediately after installation. If you use high level tools to manage it (Pycharm or hatch or...) you also gain easy testing, including against a range of Python versions at no cost.

That is the reason why I suspect that the underlying reason for not allowing relative imports outside packages is to encourage developpers to always package any project requiring more than one single script file.

In order to be more explicit with the problems of script within a packaged project, I can provide you a direct link to the page about the initial configuration of the pyproject.toml file.

Any function can be turned into an executable by declaring it in the project.scripts table.

Your example could become:

my_project/
├── pyproject.toml
├── my_project/
    ├── __init__.py
    ├── module1.py
    ├── module2.py
    ├── script1.py
    └── script2.py

Then in your pyproject.toml you could add:

[project.scripts]
script1 = "my_project.script1:main"
script2 = "my_project.script2:main"

Installation would automatically generate the commands script1 and script2 that would automatically call the function main in respectively script1.py and script2.py.

Learning how to package a project does require some work (even if a tool like hatch can give you a default project skeleton that you only have to edit) but in the long term, you will realize that it really was worth it.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks for the answer, I will look into how to deal with scripts properly inside packages. But when developing, wouldn't that prevent me from executing the scripts ? (since they will technically be modules) ? I will still need a script which is not inside the package, or am I wrong ? (I am aware these are basic questions, i'd be happy with a reference link!)
@Frotaur: just see my edit...
It's too bad doing this kind of thing right involves getting used to a lot of new concepts and boilerplate all at once. The learning curve is awkwardly steep compared to other parts of learning to work in Python.
@Frotaur: thanks to the high level tools, the learning curve is not that steep. If you use hatch or Pycharm to build your project skeleton, you will have a pyproject.toml file populated with acceptable defaults. Said differently it will just work for the simplest things. For the others, just do what I still do: open an editor window on pyproject.toml, and a browser window on the Python Packaging User Guide, and optionally another one on the configuration of hatch and hatchling...
Thanks a lot for the pointers! I used to bother with all this only if I thought that my package will be reused in other projects; From what I understand, it is actually standard practice to do this even if the package will only be used within the project itself. Thanks!
3

Relative imports are not a directory traversal mechanism. from ..modules import whatever doesn't mean "go up a directory, find a modules directory there, and run the whatever.py file in that directory". It means "import whatever from the modules subpackage (or submodule) of the parent package of the current package".

This is only meaningful if there is a current package.

And yes, a normal relative import can load modules from somewhere that isn't where you'd expect if you treated it like a directory traversal mechanism. Namespace packages are the simplest case of that.

Consider what the implications would be if relative imports did work the way you want. You do from ..modules import whatever. Now you're importing something from somewhere completely outside the import path.

How are any imports inside whatever.py supposed to work? It's still completely outside the import path. If whatever.py does import modules.thing, Python isn't going to find modules, because that's not on the import path.

What name is this thing you're importing going to be registered as, under sys.modules? It can't be whatever, because that'd hide an actual whatever module that was actually on the import path. Same with modules.whatever. There isn't a clearly sensible choice, but it's gotta be registered in sys.modules somehow, for if it gets imported again.

Improve this answer

7 Comments

Thanks for the answer, although I feel that doesn't really answer my question. While they are not directory traversal mechanism explicitly, aren't they effectively doing exactly that ? Or are there occasions where the parent package isn't literally in the parent directory?
@Frotaur: There are in fact occasions where from ..modules import whatever will load something from a directory other than the parent directory. Namespace packages are the simplest case.
I see, thanks for the additional comments. It is still unclear to me why in other languages (ok, I don't know that many, but js) doing relative imports everywhere is not a problem, whereas it seems to be very bad in Python.
@Frotaur: What languages are you thinking of? Off the top of my head, I don't actually recall any where relative imports work like you're thinking.
I was thinking about javascript imports such as import { Component } from '../Component'; where IIRC the ../ is relative to the current file. I might be wrong, I'm not very good at javascript.
|
0

When writing programs, simply import the package (or components that are needed from it). When developing a library, I start out by creating a proper package including __init__.py, __all__ and a /tests directory. The library code is run from the latter and as soon as something works properly I convert the code to a pytest compatible test fixture. When done, there is no more code, only tests that can be run by typing pytest. I usually ship the tests along with the package so that if something doesn't work on a different platform people can send me test output.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.