I'm following the guide Using Python.NET with Virtual Environments and unable to get Python.NET to use a virtual environment I've created.
I've built a simple Main that initializes Python.net, loads a script that enumerates the installed python packages, then configure the appropriate paths and variables to enter a pre-defined virtual environment, and re-run that same script. The output shows the same for both enumerations.
listpip.py:
from importlib import metadata
def go():
for dist in metadata.distributions():
print(f"{dist.name}=={dist.version}")
program.cs:
namespace PythonLoader
{
using System;
using System.Runtime.InteropServices;
using Python.Runtime;
public class ProgramPipList
{
public static void Main(string[] args)
{
string script = "listpip";
Runtime.PythonDLL = @"C:\Program Files\Python312\python312.dll";
PythonEngine.Initialize();
PythonEngine.BeginAllowThreads();
dynamic app;
using (Py.GIL())
{
using (PyModule scope = Py.CreateScope())
{
#region Before
// execute
Console.WriteLine("");
Console.WriteLine("Before");
Console.WriteLine("------");
app = Py.Import(script);
app.go();
#endregion
#region After
string pathSeparator = ";";
if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) pathSeparator = ":";
string venvDir = Path.GetFullPath("./venv");
string scriptsDir = Path.GetFullPath("./venv/scripts");
string libDir = Path.GetFullPath("./venv/lib");
string sitePackagesDir = Path.GetFullPath("./venv/lib/site-packages");
string fullPath = scriptsDir + pathSeparator + libDir + pathSeparator + sitePackagesDir;
Console.WriteLine("");
Console.WriteLine("Settings new python home and path");
Environment.SetEnvironmentVariable("PATH", Environment.GetEnvironmentVariable("PATH") + pathSeparator + venvDir, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", venvDir, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONPATH", fullPath, EnvironmentVariableTarget.Process);
PythonEngine.PythonHome = venvDir;
PythonEngine.PythonPath = fullPath;
Console.WriteLine("");
Console.WriteLine("After");
Console.WriteLine("-----");
app = Py.Import(script);
app.go();
Console.WriteLine("");
#endregion
}
}
}
}
}
Output:
C:\Code\Misc\PythonLoader\src\PythonLoader\bin\Debug\net8.0>pythonloader
Before
------
click==8.1.7
colorama==0.4.6
find_libpython==0.4.0
h11==0.14.0
aiohttp==3.9.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
dataclasses-json==0.6.6
fastapi==0.110.0
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.5.0
greenlet==3.0.3
huggingface-hub==0.23.2
idna==3.7
intel-openmp==2021.4.0
Jinja2==3.1.4
joblib==1.4.2
jsonpatch==1.33
jsonpointer==2.4
langchain-community==0.0.29
langchain-core==0.1.52
langsmith==0.1.63
MarkupSafe==2.1.5
marshmallow==3.21.2
mkl==2021.4.0
mpmath==1.3.0
multidict==6.0.5
mypy-extensions==1.0.0
networkx==3.3
nltk==3.8.1
numpy==1.26.4
orjson==3.10.3
packaging==23.2
pillow==10.3.0
pip==24.0
pydantic==2.6.4
pydantic_core==2.16.3
PyYAML==6.0.1
regex==2024.5.15
requests==2.32.3
safetensors==0.4.3
scikit-learn==1.5.0
scipy==1.13.1
sentencepiece==0.2.0
sentence-transformers==2.3.1
sniffio==1.3.1
SQLAlchemy==2.0.30
starlette==0.36.3
sympy==1.12.1
tbb==2021.12.0
tenacity==8.3.0
threadpoolctl==3.5.0
tokenizers==0.19.1
torch==2.3.0
tqdm==4.66.4
transformers==4.41.1
typing_extensions==4.12.0
typing-inspect==0.9.0
urllib3==2.2.1
uvicorn==0.21.1
yarl==1.9.4
Settings new python home and path
After
-----
click==8.1.7
colorama==0.4.6
find_libpython==0.4.0
h11==0.14.0
aiohttp==3.9.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
dataclasses-json==0.6.6
fastapi==0.110.0
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.5.0
greenlet==3.0.3
huggingface-hub==0.23.2
idna==3.7
intel-openmp==2021.4.0
Jinja2==3.1.4
joblib==1.4.2
jsonpatch==1.33
jsonpointer==2.4
langchain-community==0.0.29
langchain-core==0.1.52
langsmith==0.1.63
MarkupSafe==2.1.5
marshmallow==3.21.2
mkl==2021.4.0
mpmath==1.3.0
multidict==6.0.5
mypy-extensions==1.0.0
networkx==3.3
nltk==3.8.1
numpy==1.26.4
orjson==3.10.3
packaging==23.2
pillow==10.3.0
pip==24.0
pydantic==2.6.4
pydantic_core==2.16.3
PyYAML==6.0.1
regex==2024.5.15
requests==2.32.3
safetensors==0.4.3
scikit-learn==1.5.0
scipy==1.13.1
sentencepiece==0.2.0
sentence-transformers==2.3.1
sniffio==1.3.1
SQLAlchemy==2.0.30
starlette==0.36.3
sympy==1.12.1
tbb==2021.12.0
tenacity==8.3.0
threadpoolctl==3.5.0
tokenizers==0.19.1
torch==2.3.0
tqdm==4.66.4
transformers==4.41.1
typing_extensions==4.12.0
typing-inspect==0.9.0
urllib3==2.2.1
uvicorn==0.21.1
yarl==1.9.4
The virtual environment is present and defined:
C:\Code\Misc\PythonLoader\src\PythonLoader\bin\Debug\net8.0>dir venv
Volume in drive C has no label.
Volume Serial Number is EA70-9732
Directory of C:\Code\Misc\PythonLoader\src\PythonLoader\bin\Debug\net8.0\venv
05/31/2024 04:16 PM <DIR> .
06/10/2024 08:31 AM <DIR> ..
05/31/2024 04:16 PM <DIR> Include
05/31/2024 04:16 PM <DIR> Lib
06/10/2024 07:47 AM 265 pyvenv.cfg
05/31/2024 04:16 PM <DIR> Scripts
1 File(s) 265 bytes
5 Dir(s) 593,590,071,296 bytes free
C:\Code\Misc\PythonLoader\src\PythonLoader\bin\Debug\net8.0>venv\scripts\activate
(venv) C:\Code\Misc\PythonLoader\src\PythonLoader\bin\Debug\net8.0>pip list
Package Version
------------------ --------
certifi 2024.2.2
charset-normalizer 3.3.2
debugpy 1.8.1
idna 3.7
pip 24.0
requests 2.32.3
urllib3 2.2.1
What changes would I need to make in order to have the second call to listpip.py use the virtual environment?
Based on the discussions on the pythonnet repo in Github it appears that the environment variables and configuration have to be set before initializing pythonnet if you're using a virtual environment. Given that it's a static class, it makes it difficult to be able to have multiple instances within the same process using different virtual environments.
I've gotten around this by writing my own wrapper that uses the shell and encapsulation of user scripts into a wrapper script. If anyone is interested, please feel free to use it. I've tested it on Windows, Ubuntu, and Mac, and it's working nicely. Granted, it uses the shell, and has limitations in terms of input data types (JSON), output data types (JSON), and enforces a restriction on how the encapsulated (user) script is written. But, it serves its purpose, and each script invocation is a separate process (e.g. spawning cmd.exe or /bin/bash).
Full transparency - I would fully advocate that people use pythonnet for anything related to integration of Python code into a C# environment - but if you need support for multiple concurrent virtual environments, it seems an alternate solution like the one I published may be required today.