0

For my Uni thesis project, I need firstly to setup automatically multiple github repositories within python script. Problems I am facing is that:

  • each repository has its own requirements and some of them conflict with other repository's requirements. The most prominent is 'diffusers', for one method I am using huggingface's diffusers for other method I am using ShivamShrirao's one. but when importing, it it is only import diffusers, so I think when I 'pip install' one of them it is overwritting the other one.
  • I need to call functions from these repositories inside the code, so I think I need to add the cloned repositories to PATH.

so for that I created a folder structure as follows:

main
├── method_1_folder
│   ├── method_1_github_repository_cloned_folder
│   └── venv
├── method_2_folder
│   ├── method_2_github_repository_cloned_folder
│   └── venv
└── running_python_file.py

and tried to create a separate environment for each method so I would call a prep_work function that would clone the github, create an environment, activate it, then add its path to python path then install all requirements. makes sense right? but when I print the path of the active environment after all the actions above, I find out that I am still working on the running_python_file.py environment. would you please help to correct whatever I am doing wrong?

def prep_work(filename, repo_file_name, repo_url, characters, trainingstep=None):
    environment_not_created = True

    # cloning respository
    path = os.path.join(os.getcwd(), filename, repo_file_name)
    if not os.path.exists(path):
        Repo.clone_from(repo_url, path)
        subprocess.run(["python", "-m", "venv", f"{os.path.join(os.getcwd(), filename, 'venv')}"], check=True)
        environment_not_created = False

    os.chdir(os.path.join(os.getcwd(), filename))
    os.system(f'source {os.path.join(os.getcwd(), "venv", "bin", "activate")}')
    sys.path.append(path)
    print("Active environment:", sys.prefix)
14
  • I'm not sure exactly what you need to do in running_python_file but you if you could switch the logic in it to a bash script, then your issue is fixed (in theory) Commented Mar 23, 2024 at 10:34
  • @EdoAkse I would like to do it within python script not in bash script. as I explained above 'running_python_file' has many method (including prep_work) 7 of them start by cloning a github repository. Commented Mar 23, 2024 at 11:54
  • 1
    Each virtual environment directory has a bin/python executable. Use that executable in the sub-processes. Commented Mar 23, 2024 at 15:56
  • that is what I am doing..after creating a folder titled 'filenmae', I am moving to it with chdir then activating the bin folder in the subprocesses. do you mean I need to write 'os.system(f'source {os.path.join(os.getcwd(), "venv", "bin/python", "activate")}')' Commented Mar 24, 2024 at 10:04
  • 1
    Do not try to activate. Use the python binary directly: subprocess.check_call(['path/to/venv/bin/python', 'some_script.py']). Commented Mar 30, 2024 at 21:41

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.