How can I add paths to the $PYTHONPATH or $PATH variables when a script within my python project is executed?

Question

I have a python project with multiple scripts (scriptA, scriptB, scriptC) that must be able to find packages located in subpath of the project that is not a python package or module. The organization is like so:

|_project
  |_scriptA.py
  |_scriptB.py
  |_scriptC.py
  |_thrift_packages
    |_gen-py
        |_thriftA
          |__init__.py
          |_thriftA.py

On a per script basis I am adding the absolute path to this directory to sys.path

Is there a way that I can alter the PYTHONPATH or sys.path every time a script is executed within the project so that I do not have to append the path to this directory to sys.path on a per-script basis?

"I have a python project with multiple scripts (scriptA, scriptB, scriptC) that must be able to find packages located in subpath of the project that is not a python package or module" Why, though? Instead use venv to make a virtualenv, make a setup.py and/or setup.cfg and install your package in editable mode. I know this is not straightforward but it's the right way to do it. — Iguananaut
– Iguananaut, Commented Oct 29, 2019 at 20:34
I've written guides to this over the years. Most recently this one (see the "packaging" section) — Iguananaut
– Iguananaut, Commented Oct 29, 2019 at 20:34
@Iguananaut your guide on github returns a page with: "action 'view' is not supported for filetype 'ipynb'" — Dan MacNeil
– Dan MacNeil, Commented Sep 24, 2022 at 20:56

Iguananaut · Accepted Answer · 2019-10-29 21:08:30Z

4

You have an XY problem, albeit an understandable one since the "proper" way to develop a Python project is not obvious, and there aren't a lot of good guides to getting started (there are also differing opinions on this, especially in the details, though I think what I'll describe here is fairly commonplace).

First, at the root of your project, you can create a setup.py. These days this file can just be a stub; eventually the need for it should go away entirely, but some tools still require it:

$ cat setup.py
#!/usr/bin/env python
from setuptools import setup
setup()

Then create a setup.cfg. For most Python projects this should be sufficient--you only need to put additional code in setup.py for special cases. Here's a template for a more sophisticated setup.cfg, but for your project you need at a minimum:

$ cat setup.cfg
[metadata]
name = project
version = 0.1.0

[options]
package_dir =
    =thrift_packages/gen-py
packages = find:

Now create and activate a virtual environment for your project (going in-depth into virtual environments will be out of scope for this answer but I'm happy to answer follow-up questions).

$ mkdir -p ~/.virtualenvs
$ python3 -m venv ~/.virtualenvs/thrift
$ source ~/.virtualenvs/thrift/bin/activate

Now you should have a prompt that look something like (thrift) $ indicating that you're in an isolated virtualenv. Here you can install any dependencies for your package, as well as the package itself, without interfering with your main Python installation.

Now install your package in "editable" mode, meaning that the path to the sources you're actively developing on will automatically be added to sys.path when you run Python (including your top-level scripts):

$ pip install -e .

If you then start Python and import your package you can see, for example, something like:

$ python -c 'import thriftA'; print(thriftA)
<module 'thriftA' from '/path/to/your/source/code/project/thrift_packages/gen-py/thriftA/__init__.py'>

If this seems like too much trouble, trust me, it isn't. Once you get the hang of this (and there are several project templates, e.g. made with cookie-cutter to take the thinking out of it) you'll see that it makes things like paths less trouble). This is how I start any non-trivial project (anything more than a single file); if you set everything up properly you'll never have to worry about fussing with sys.path or $PYTHONPATH manually.

In this guide, although the first part is a bit application-specific, if you ignore the specific purpose of this code a lot of it is actually pretty generic, especially the section titled "Packaging our package", which repeats some of this advice in more detail.

As an aside, if you're already using conda you don't need to create a virtualenv, as conda environments are just fancy virtualenvs.

An advantage to doing this the "right" way, is that when it comes time to install your package, whether by yourself, or by users, if your setup.cfg and setup.py are set up properly, then all users have to do is run pip install . (without the -e) and it should work the same way.

edited Oct 29, 2019 at 21:08

answered Oct 29, 2019 at 20:54

Iguananaut

23.8k6 gold badges54 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

sinoroc Over a year ago

Missing the part about the entry points for the console scripts.

Iguananaut Over a year ago

That would be good too but isn't necessarily a requirement here; we don't know. Sometimes there are scripts that are useful to run in development or as tests, but that don't need to be installed. If OP mentions script installation then I'll add it.

Anthony O Over a year ago

Thank you, this is exactly what I was looking for. What do you mean by "These days this file can just be a stub; eventually the need for it should go away entirely"?

Iguananaut Over a year ago

Great, I'm glad this was helpful. Like I wrote I know it's a mouthful but it should ease development and use of your packages. What I meant was that it used to be that you coded all your package metadata in Python directly in the setup.py file. Now most any static metadata can go directly in the setup.cfg file so there is no need, in that case, for an explicit setup.py file, but many tools including pip currently still expect it.

Iguananaut Over a year ago

The Python packaging infrastructure and tooling has evolved a great deal over the years and has often not been as stable as it is in some other programming language communities, which is why it can be confusing and difficult to find good information.

|

Florian Bernard · Accepted Answer · 2019-10-29 20:24:38Z

0

You should add __init__.py in each package, and then properly call all your script.

|_project 
    |_scriptA.py 
    |_scriptB.py 
    |_scriptC.py 
    |__init__.py <====== Here
    |_thrift_packages 
        |__init__.py <====== Here
        |_gen-py 
        |_thriftA 
            |__init__.py 
            |_thriftA.py

Assuming that, project is in your pythonpath, you can do:

from project.thrift_packages.thriftA import thriftA

answered Oct 29, 2019 at 20:24

Florian Bernard

2,5891 gold badge11 silver badges24 bronze badges

Comments

gelonida · Accepted Answer · 2019-11-09 00:47:19Z

yes you can.

However I think the other answers are more adapted to your current issue. Here I just explain, how to call subprocesses with another environment, another current working directory. This can come in handy in other situations.

You can get the current environment as a dict (os.environ) copy it to another dict, modify it and pass it to a supbprocess call. (subprocess functions have an env parameter)

import os
import subprocess
new_env = dict(os.environ)
new_env["PYTHONPATH"] = whateveryouwanthere  # add here the pythonpath that you want
cmd = ["mycmd.py", "myarg1", "myarg2"]  # replace with your command
subprocess.call(cmd, env=new_env)  #
# or alternatively if you also want to change the current directory
# subprocess.call(cmd, env=new_env, cwd=path_of_your_choice)  #

Collectives™ on Stack Overflow

How can I add paths to the $PYTHONPATH or $PATH variables when a script within my python project is executed?

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related