I am working on an application that will allow users to submit their python scripts. These python scripts will contain Language Models (LMs) that will be used by the app to calculate certain metrics.
I was planning on running these scripts inside Docker containers, for scalability and security concerns. They would function as "black-boxes" that would accept an input and return an output without the app needing to know what is going on inside the container.
For now, I simply need to get a proof-of-concept working, and I assumed that Docker allowed users to not only create containers manually, but also automatically. After hours of searching, I believe I was proven wrong. I have read about something called Kubernetes, but I am unsure if this is what I need.
So my question is simple: Is it possible to use only Docker for this, or do I need to learn other tools like Kubernetes to do this?
I've thought about (for just the proof-of-concept) to use a python program that calls the submitted code, but I have no idea how it would install the necessary packages from the submitted code's imports. I also don't know how I would keep the code running, as LMs need to stay loaded to run, and if script1.py calls script2.py, it won't keep executing until script2.py is done running, meaning I would have to wait for the LM to load every time I need to call its functions. Also, I already have a docker-compose.yml file that automatically installs all the dependencies and containerizes the submitted python scripts, but it must be run manually.
If you would like to look at some of the code for the containerization:
This creates a container with the script that acts as a middleman between the LM and the server, and automatically installs the required dependencies of the LM.
FROM python:3.9
WORKDIR /usr/app/src
COPY communicator.py ./
COPY lm_submission.py ./
COPY requirements.txt ./
RUN pip3 install -r requirements.txt
This docker-compose file manually creates the "server" (the thing that gives inputs to the LMs and waits for outputs) and the communicator (in theory, there would be lots of communicators running multiple LMs and 1 server. Also the chatGPT_roberta_model parameter would be a variable name that changes depending on the name of the LM being run).
version: '3.9'
services:
communicator:
build: .
command: sh -c "sleep 2s; python3 ./communicator.py chatGPT_roberta_model"
environment:
LISTEN_HOST: server
LISTEN_PORT: 5555
ports:
- '5556:5555'
depends_on:
- server
server:
build: ./server/
command: sh -c "python3 ./server.py"
environment:
SEND_HOST: server
SEND_PORT: 5555
ports:
- '5555:5555'
If you use the right tools, it takes relatively little time to prepare your POC.
I suggest you start exploring minikube, which is a lite and portable version of Kubernetes perfect for this type of use-case.
Take a look here: https://minikube.sigs.k8s.io/docs/start/
Obviously, the Companies have Cloud Providers or large Kubernetes Clusters of tests available for these needs; minikube is more of something for the laboratory.
(I speak of Kubernetes because, as a container orchestrator, it is certainly the best choice and closest to a possible production use-case)
Then find out about the Kubernetes Deployment and Service resources, in order to release your "black-box" application. The Deployment will allow you to release 1 or more Pods (instances of your application + OS + Network) and the Service will allow you to make the application reachable inside the K8s Cluster and/or outside.
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
https://kubernetes.io/docs/concepts/services-networking/service/
It will then be possible to ensure that these application instances (Pods) remain up&running and always available for the end-user by following a few simple steps --> Multi container pod with command sleep k8
Finally, I suggest not to create containers with active root users (if possible) and to prevent users from accessing these Pods via SSH; surely a better solution could be to develop a front-end.
PS: If you don't know how to generate Kubernetes manifests, you can convert Docker Compose manifests simply --> https://kubernetes.io/docs/tasks/configure-pod-container/translate-compose-kubernetes/