How to Run a Pyomo Optimization Problem in a Django-Heroku App?

I have been trying to deploy a Django app that use the python optimization package 'pyomo' https://pyomo.readthedocs.io/en/stable/index.html

It works locally, but not when deployed to heroku. In production, it is not able to find the uploaded executable for the solver. From the logs (the app is called 'number_generator' at the moment):

2023-05-30T19:49:36.256914+00:00 app[web.1]: RuntimeError: Attempting to use an unavailable solver.
2023-05-30T19:49:36.256915+00:00 app[web.1]:
2023-05-30T19:49:36.256915+00:00 app[web.1]: The SolverFactory was unable to create the solver "HIGHS"
2023-05-30T19:49:36.256915+00:00 app[web.1]: and returned an UnknownSolver object.  This error is raised at the point
2023-05-30T19:49:36.256915+00:00 app[web.1]: where the UnknownSolver object was used as if it were valid (by calling
2023-05-30T19:49:36.256915+00:00 app[web.1]: method "solve").
2023-05-30T19:49:36.256915+00:00 app[web.1]:
2023-05-30T19:49:36.256916+00:00 app[web.1]: The original solver was created with the following parameters:
2023-05-30T19:49:36.256916+00:00 app[web.1]: executable: /app/number_generator/executables/highs
2023-05-30T19:49:36.256916+00:00 app[web.1]: type: HIGHS
2023-05-30T19:49:36.256916+00:00 app[web.1]: _args: ()
2023-05-30T19:49:36.256916+00:00 app[web.1]: options: {}
2023-05-30T19:49:36.257340+00:00 app[web.1]: 10.1.20.80 - - [30/May/2023:19:49:36 +0000] "POST /opttool/ HTTP/1.1" 500 145 "https://simple-esa.herokuapp.com/opttool/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
2023-05-30T19:49:36.257469+00:00 heroku[router]: at=info method=POST path="/opttool/" host=simple-esa.herokuapp.com request_id=73e964ec-dfca-487b-8b8c-319d01ce55f0 fwd="2.106.20.24" dyno=web.1 connect=0ms service=11075ms status=500 bytes=445 protocol=https

I have used the 'highs' solver and the following folder configurations, with corresponding view-functions to load the executable:

One using the static folder:

> ..
> number_generator
>    static
>      highs
>        highs.exe
> ..

import pyomo.environ as pyo
from django.contrib.staticfiles import finders
...

def opttool(...):
    ...
    p = finders.find('highs/highs.exe')
    p = p.replace('.exe', '')
    opt = pyo.SolverFactory('HIGHS', executable=p)
    ...

...and one at the same level of the static folder:

> ..
> number_generator
>   static
>   executables
>     highs.exe
> ..

import pyomo.environ as pyo
import os
...

def opttool(...):
    ...
    p = os.path.join(os.path.dirname(__file__), 'executables', 'highs.exe')
    p = p.replace('.exe', '')
    opt = pyo.SolverFactory('HIGHS', executable=p)
    ...

As mentioned, both work locally, but not when deployed to heroku. Any ideas?

UPDATE:

AirSquid made a great suggestion (30/5/2023) to try and navigate on the CLI and test if the .exe file works. This was new to me (and seems extremely useful!), that it is possible to do this with the command:

heroku run bash --app your-app-name

..and AirSquid was right; it does not:

~/number_generator/executables $ ls 
docs  highs.exe
~/number_generator/executables $ highs --version
bash: highs: command not found

...while on the desktop:

C:\GitRepos\Basic-ESA\number_generator\executables>highs
usage: highs [options] stub [-AMPL] [<assignment> ...]

Options:
        --  end of options
        -=  show solver options and exit
        -?  show usage and exit
        -a  show solver options (ASL style, 1st synonyms if provided) and exit
        -c  show constraint descriptions and exit
        -e  suppress echoing of assignments
        -s  write .sol file (without -AMPL)
        -v  show version and exit

Could the problem be that .exe files are not executable on Linux? ..and that the Heroku interface is running on Linux..? I tried downloading and installing glpk on the heroku CLI (and of course changed to 'glpk' in pyomo): pyo.SolverFactory('glpk') ...i assumed that this would work, since I locally (on windows) can run gurobi after installing this solver, without specifying the executable. But my heroku app still does not locate glpk after installation. This is maybe just due to heroku seeming not to save this permanently, but in the session only: I accessed the CLI after some minutes and glpk was lost:

~ $ ls
db.sqlite3  manage.py  number_generator  Procfile  __pycache__  random_number_generator  ReadMe  requirements.txt  runtime.txt  staticfiles

...so I guess:

A stand-alone solver binary for linux, that can be uploaded to staticfiles could be the solution. I tried CLP from https://www.coin-or.org/download/binary/Clp/, but the Clp-1.7.4-linux-x86 version did not seem to work in production (locally it resulted in an ApplicationError, which makes sense to me since I'm running windows):

~/number_generator/executables/CLP/bin $ ls
clp
~/number_generator/executables/CLP/bin $ clp --version
bash: clp: command not found

Installing a solver on/during heroku deployment could be needed, and then locating the binary. I might have to learn to use this 'buildpack' tool? https://github.com/heroku/heroku-buildpack-apt
...A simpler solution could exist?

Solution

SOLUTION!

My guess nr. 2 above worked! I found a GLPK ubuntu package (there were other solvers as well): https://packages.ubuntu.com/kinetic/glpk-utils

I then added the buildpack for apt-get to Heroku https://github.com/heroku/heroku-buildpack-apt (see https://devcenter.heroku.com/articles/buildpacks)

This successfully installed GLPK on deployment. I could then find the path for the glpsol binary using which glpsol in the heroku CLI, meaning the call through pyomo worked. In my case it was in '.apt/usr/bin/glpsol':

pyo.SolverFactory('glpk', executable='.apt/usr/bin/glpsol')