Search code examples
javabatch-fileenvironment-variablespython-importgdal

ImportError for module osgeo (embd. py) while invoking portable binaries of GDAL from java, after using a bat script to update env. path variables

Edit 1: Forgot to mention that python, I'm using is embedded python.

I have downloaded windows GDAL binaries from GIS Internals. Premise:

Directory structure: 

Parent
 |gnuplot
 |mysoft.exe
 |gdal
 |python-embedded
 |jvm
 |gsl2
 |mybinaries
 |mysoftlaunch.bat

The bat file is provided by the GDAL binaries and I have made relevant changes to the file and the directory I put it in.

@echo off

set SDK_ROOT=%~dp0gdal\
set SDK_ROOT=%SDK_ROOT:\\=\%
set GNUPLOT_ROOT=%~dp0gnuplot\
set PYTHON_ROOT=%~dp0python-3.7.3-embed-amd64\
@echo %SDK_ROOT%
@echo %GNUPLOT_ROOT%

goto setenv
if "%1" == "setenv" goto setenv

%comspec% /k "%SDK_ROOT%SDKShell.bat" setenv %1
goto exit

:setenv
@echo Setting environment for using the GDAL and MapServer tools.

if "%2"=="hideoci" goto hideoci

set ocipath=0
set _path="%PATH:;=" "%"
for %%p in (%_path%) do if not "%%~p"=="" if exist %%~p\oci.dll set ocipath=1

if "%ocipath%"=="0" goto hideoci
@echo WARNING: If you encounter problems with missing oci libraries then type:
@echo   SDKShell hideoci
goto setenv2

...
...

:setenv2
@echo at set env 2
SET "PATH=%SDK_ROOT%bin;%SDK_ROOT%bin\gdal\python\osgeo;%SDK_ROOT%bin\proj6\apps;%SDK_ROOT%bin\gdal\apps;%SDK_ROOT%bin\ms\apps;%SDK_ROOT%bin\gdal\csharp;%SDK_ROOT%bin\ms\csharp;%SDK_ROOT%bin\curl;%GNUPLOT_ROOT%bin\;%SDK_ROOT%gsl2\bin\;%PATH%"
SET "GDAL_DATA=%SDK_ROOT%bin\gdal-data"
SET "GDAL_DRIVER_PATH=%SDK_ROOT%bin\gdal\plugins"
SET "PYTHONPATH=%SDK_ROOT%bin\gdal\python\osgeo;%SDK_ROOT%bin\gdal\python;%SDK_ROOT%bin\ms\python;%PYTHON_ROOT%"
SET "PROJ_LIB=%SDK_ROOT%bin\proj6\SHARE"
SET "LD_LIB_PATH=%SDK_ROOT%gsl2\lib"
@echo done setting variables
start "MYSOFT" cmd /c java -jar "%~dp0mysoft.exe%"
@echo started mysoftware
exit
:exit

A few things here:

  1. I have used launch4j to provide path to the jvm for my jar file and made it an exe which is why there is a mysoft.exe
  2. I have specified relative paths in my software to access required executable or scripts.
  3. I have used the python-embedded and given the relative path of the python script which I'd invoke like gdal_merge.py.
  4. I didn't want to do the setx once in the bat file and run the software directly because if the users change the directory of the software and run the script again, the path size limit in windows will be exhausted and will be a disaster. (why I'm launching the software from the bat file.)

The script was supposed to set the environmental variables for all the child processes launched.

I can confirm that the gdal_translate worked fine. [I did specify the relative path to it not invoke it from the environment.]

Work flow:

  1. I set the environmental variables in bat file and launch the java program.
  2. I have a Java Program as a user interface. The Java Program invokes some executable or script, which includes gdal_merge.py.
  3. I specify the full path to gdal_merge.py in my code and invoke it using ProcessBuilder

Error:

from osgeo import gdal
ModuleNotFoundError: No module named 'osgeo'.

Solution expected: After setting path from a bat file (to embedded-python, jvm, gdal{all required like gdal_data,gdal_driver, path for binaries etc.}), launch a Java software which inherits those environmental variables which launches a gdal_merge.py file.

In the below fashion:

String path = System.getProperty("user.dir");
ProcessBuilder pb = new ProcessBuilder();
pb.inheritIO();
pb.command(Paths.get(path+"/python-embedded/python.exe").toString(),Paths.get(path+"/gdal/....../python/scripts/gdal_merge.py").toString(),...{other parameters required for gdal_merge.py});

PS:
Paths.get(x+"abc/def/").toString() - I'm using to convert *NIX style paths to Windows style paths.

Solution

  • According to the Issue28245 Python bugs, this is a deliberate feature. Embedded python is isolated completely from your system environment.

    To include path where this embedded python looks for packages, you need to edit pythonxx._pth file.

    Just add a path in each line. No need of variables. You can add relative path also.

    Examples of a ._pth file:

    pythonxx.zip # default value
. # default value
..\gdal\bin\gdal\python\
..\gdal\bin\gdal\python\osgeo\
..\gdal\bin\gdal\python\scripts\
..\gdal\bin\ms\python\

    This works like a charm.