The question is if there another way that I'm just not thinking of to solve those problems below, or is it really the answer to build a python flavor with our tools? I have a proposed solution that solves the problems, but that doesn't mean it's the right answer.
I work in a support organization where we're developing tools for support on our main product. This product has it's own flavor of OS that it runs on. We have three distinct problems to solve in packaging o
The flavor of OS has it's original python binaries we can install to, but this limits us to the OS version of python which will change as a separate team that manages the OS. This is expected to change over the next 2 years between releases 3.4, 3.5, and 3.6 of python - which have changes that affect libraries we use consistently, for better and worse.
The tools that we build are also used as standalone tools at sites with no connectivity externally for analysis. We're currently limited to "hope the python that's on wherever they're using it plays nice".
There are significant performance improvements in later versions of python that we would like to take advantage of, and can't without upgrading to 3.6, but that removes our ability to use the older versions because of some significant differences in the python lib that break things in one vs. the other.
My original approach was to try and make a standalone virtualenv that was relocatable, but the more I looked at that code, the more I found that it's just editing PYTHONHOME and PATH, and if you wanted it to be relocatable, you have to copy all the binaries anyway, or it's only usable if the host has the version of python you're built for. This also had the downside of needing a lot of modification to the virtualenv scripting so that the shebangs were modified, the paths were updated, etc - and would need to be updated on every move, or have dynamic shebangs.
Right now I'm looking at creating our own "flavor" of python - but that feels like bringing an axe to a dinner party to cut your carrots. It solves all of the problems of multiple locations that it could be used all having a consistent up to date version of python with the tools we've installed, so all the users would need to do is run a script that updates the PATH with the /bin where these things are installed.
So back to the question:
Is there another way that I'm just not thinking of to solve those problems, or is it really the answer to build a python flavor with our tools? Am I making this feel wrong because of inexperience, or is this a valid answer I should be considering?
I actually ended up blending the two - I created my own "activate" file that dynamically allocates path (To whatever the activate file has been called fromhas some additional functionality, and put that into a python binary. When sourced, it will update the shebangs.
activate will now set the path dynamically, using it's own directory that it resides in as the new PATH shim. This assumes, as with most venvs, that the activate script resides in bin.
activate now has a helper script and a wrapper that calls it for all files in bin that have a shebang. It updates those files shebang to the path of the current python executable (Which should have been updated by the virtualenv activate script at this point). It will only do this if the shebangs are different.
The script changes for reference:
function update_shebang() {
# Call like: update_shebang check_sas_cabling
full_path=${1}
# Back up our scripts, making them hidden by default.
file_dir=$(dirname $full_path)
file_name=$(basename $full_path)
cp ${full_path} "${file_dir}/.${file_name}.bak"
# Find old shebang - will look something like this:
# /workdir/pure_python/bin/python3
oldbang=$(grep '#!' ${full_path})
newbang_path=$(readlink -f `which python`)
# We have to escape the !, but then it keeps the backslash, so we have to
# get rid of it.
newbang=$(echo "#\!$newbang_path" | tr -d '\\')
# Just using sed in place
# We dont' want to spam people every time they activate - so only modify
# them if they aren't the same.
if [ "$(echo $oldbang | tr -d "/")" != "$(echo $newbang | tr -d "/")" ]; then
# Don't modify binary matches.
if [ ! "$(echo $oldbang | grep 'Binary')" ]; then
echo "Updated shebang for ${file_name} from ${oldbang} to ${newbang}"
sed -i "s|${oldbang}|${newbang}|" ${full_path}
fi
fi
}
function update_all_shebangs() {
# Wrapper around update_shebang to update all the
# shebangs in ${NEW_PATH}/bin.
given_path=$1
myfiles=$(ls $given_path | grep -ve "^activate$")
for file in $myfiles; do
fullpath=$given_path/$file
if grep -q "#!" $fullpath; then
update_shebang $fullpath || exit 1
fi
done
}
And the new dynamic path portion:
NEW_PATH="$(dirname $(dirname $(readlink -f -n $BASH_SOURCE)))"
I also incorporated creating this in a docker image that ran all the pip installs for my required stuff and then bundled it with all of this - so it takes a little bit longer, but is truly relocatable in all senses of the word right out of the box.
Will put up the skeleton repo that includes this once I script the docker installation that's required. :D