I am putting together a Python package that uses a Cygwin executable without having Cygwin installed. This means that I have an executable (.exe) file and a library (.dll) file inside my package. I am doing this so that the tool can be used on Windows by people who only use Windows and Python. I am new to Python packages, so I appreciate any help.
How do I make my package point to the executable file inside my package? This would be used, for example, instead of the executable referenced by PATH. The answer might be out there, but all I've found in my search is a bunch of info on how to create an executable from a Python script, which is NOT what I want.
The tool is sclite, a tool to score speech recognition output. For information about how I get the tool available on Windows, have a look at the How to Install SCLITE (for reproduce-ability) section, below.
Here's a small, "toy" example of how I have my package set up.
$ tree package_holder_dir/
package_holder_dir/
├── bbd_package
│ ├── __init__.py
│ ├── non_py_exe_dll
│ │ ├── cygwin1.dll
│ │ └── sclite.exe
│ ├── score
│ │ ├── __init__.py
│ │ └── run_sclite.py
│ └── score_transcript.py
├── MANIFEST.in
├── README.txt
└── setup.py
3 directories, 9 files
My first guess was that the sclite.exe should go in the MANIFEST.in file.
# @file MANIFEST.in
include ./README.txt
include ./bbd_package/non_py_exe_dll/sclite.exe
include ./bbd_package/non_py_exe_dll/cygwin1.dll
I also tried putting them in setup.py
My setup.py is as follows
#!/usr/bin/env/python3
# -*- coding: utf-8 -*-
# @file setup.py
import setuptools
from distutils.core import setup
with open ("README.txt", "r") as fh:
long_description = fh.read()
setup(
name='bbd_package',
url='[email protected]',
author='bballdave025',
author_email='[email protected]',
packages=setuptools.find_packages(),
#data_files=[('lib', ['./non_py_exe_dll/cygwin1.dll']),
# './non_py_exe_dll/sclite.exe'],
version='0.0.1',
description="Example for SO",
long_description=long_description,
include_package_data=True
) ##endof: setup
Note this source (archived) for my use of data_files for the location of the DLL. However, I could not find the files when I installed the distribution elsewhere. That's why they are commented out here.
Using MANIFEST.in seemed to work, but then I had to use a relative path to access the executable. That won't work when trying to import bbd_package in another directory.
Let me try to illustrate with my two Python files:
score_transcript.py simply calls run_sclite.py.
#!/usr/env/bin python3
# -*- coding: utf-8 -*-
# @file run_sclite.py
import os, sys, subprocess
def run_sclite(hyp, ref):
subprocess.call(['../non_py_exe_dll/sclite.exe', '-h', hyp, '-r', ref, '-i', 'rm', \
'-o', 'all snt'])
I can install it on my system:
C:\toy_executable_example\package_holder_dir>pip install .
Then if I happen to be in the directory with run_sclite.py
C:\toy_executable_example\package_holder_dir\bbd_package\score>python
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bbd_package.score.run_sclite
>>> bbd_package.score.run_sclite.run_sclite('a.hyp', 'a.ref')
sclite Version: 2.10, SCTK Version: 1.3
...output that shows it works...
>>>
However, from any other directory, no dice.
C:\Users\me>python
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bbd_package.score.run_sclite
>>> bbd_package.score.run_sclite.run_sclite('a.hyp', 'a.ref')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\site-packages\bbd_package\score\run_sclite.py", line 9, in run_sclite
'-o', 'all snt'])
File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 267, in call
with Popen(*popenargs, **kwargs) as p:
File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 997, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
>>>
How can I tell Python to look for the executable inside my package?
This information is taken from running systeminfo from Window's Command Prompt.
OS Name: Microsoft Windows 10 Enterprise
OS Version: 10.0.15063 N/A Build 15063
OS Manufacturer: Microsoft Corporation
OS Configuration: Member Workstation
OS Build Type: Multiprocessor Free
I'm including a link to the installation instructions (using Cygwin) here, look for my comment. Here are the commands with no explanation
$ cd
$ git clone https://github.com/kaldi-asr/kaldi.git
$ cd kaldi/tools
$ extras/check_dependencies.sh
$ make -j $(nproc --all)
$ cp -R sctk-2.4.10 ~/
$ cd
$ rm -rf kaldi
$ cd sctk-2.4.10/
$ cp $HOME/.bashrc "${HOME}/.bashrc.$(date +%Y%m%d-%H%M%S).bak"
$ echo -e "\n\n## Allow access to sclite, rfilter, etc" >> $HOME/.bashrc
$ echo 'export PATH='"$(pwd)/bin"':$PATH' >> $HOME/.bashrc
$ source ~/.bashrc
I'm also including a link (archived) to "quicker" instructions and the information about the EXE and DLL files. This is all thanks to @benreaves
Easier (but even uglier) workaround, which I gave to an intern who did some Word Error Rate calculations for me:
On my laptop with Cygwin installed, I create
sclite.exeusing mycp src/*/*.c . ; make allworkaround from my previous message
I create a zip file containing
sclite.exeandcygwin1.dllfrom my laptop'sc:/cygwin/bin/folderEmail that zip file to her, and she copies both files in a single new folder on her laptop
andset PATH=.;%PATH%
I don't set the PATH, because I want this to be distributible.
This worked just fine on her laptop running Windows 7, and she didn't need Cygwin or any other NIST software on her laptop.
-Ben
I finally got this to work. The sources which I patched together are below. The basic answer is that I used the __file__ variable to find the directory from which the packaged module was called, then used relative paths to get to my executable. I'm not really satisfied with this solution (here {archived} are some situations in which it won't work), but it gets the job done for now.
The MANIFEST.in file stayed the same:
# @file MANIFEST.in
# @author bballdave025
include ./README.txt
include ./bbd_package/non_py_exe_dll/sclite.exe
include ./bbd_package/non_py_exe_dll/cygwin1.dll
However, I needed to add the following to the exe-running code. This is in run_sclite.py
from pathlib import Path
#...
path_to_here = os.path.dirname(os.path.abspath(__file__))
path_to_pkg_root = Path(path_to_here).resolve().parents[1]
path_to_exe = os.path.join(path_to_pkg_root, 'non_py_exe_dll')
#...
Note that Python version >= 3.4 is necessary for pathlib
Here is the whole (run_sclite.py) file:
#!/usr/env/bin python3
# -*- coding: utf-8 -*-
# @file run_sclite.py
# @author bballdave025
import os, sys, subprocess
from pathlib import Path
def run_sclite(hyp, ref):
path_to_here = os.path.dirname(os.path.abspath(__file__))
path_to_pkg_root = Path(path_to_here).resolve().parents[1]
path_to_exe = os.path.join(path_to_pkg_root, 'non_py_exe_dll')
subprocess.call([os.path.join(path_to_exe, 'sclite.exe'), '-h', hyp, '-r', ref, \
'-i', 'rm', '-o', 'all snt'])
##endof: run_sclite(hyp, ref)
Mouse vs. Python, with clarification on when this solution won't work and other alternatives. (archived)
Getting to package root directory (SO) (archived ; Look for "Using os.path").
What finally got me to the answer: Python Packaging Tutorial . Look at the 'Note:'. (archived)