Search code examples
pythondockerelasticsearchflaskdocker-compose

Docker compose flask-app fails to connect to ES exit with code 1


I am new to docker-compose and I was following a tutorial the aim is to make ES work with a flask app when I launch the two containers and create the network to isolate them manually everything works fine but when I tried to use docker-compose I have this strange error which looks more of a warning than the root cause of the absence of connection between ES and my flask web app :

docker-compose up
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "foodtrucks_default" with the default driver
Creating es ... done
Creating foodtrucks_web_1 ... done
Attaching to es, foodtrucks_web_1
es     | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
es     | [2021-02-25T09:26:46,268][INFO ][o.e.n.Node               ] [] initializing ...
es     | [2021-02-25T09:26:46,480][INFO ][o.e.e.NodeEnvironment    ] [XZz4TuM] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [921.1gb], net total_space [982.3gb], types [ext4]
es     | [2021-02-25T09:26:46,481][INFO ][o.e.e.NodeEnvironment    ] [XZz4TuM] heap size [990.7mb], compressed ordinary object pointers [true]
es     | [2021-02-25T09:26:46,486][INFO ][o.e.n.Node               ] [XZz4TuM] node name derived from node ID [XZz4TuMGRQmISPJ-VpaDOA]; set [node.name] to override
es     | [2021-02-25T09:26:46,487][INFO ][o.e.n.Node               ] [XZz4TuM] version[6.3.2], pid[1], build[default/tar/053779d/2018-07-20T05:20:23.451332Z], OS[Linux/5.4.0-65-generic/amd64], JVM["Oracle Corporation"/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
es     | [2021-02-25T09:26:46,489][INFO ][o.e.n.Node               ] [XZz4TuM] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.yFMQabHq, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
es     | [2021-02-25T09:26:51,001][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [aggs-matrix-stats]
es     | [2021-02-25T09:26:51,002][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [analysis-common]
es     | [2021-02-25T09:26:51,002][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [ingest-common]
es     | [2021-02-25T09:26:51,002][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [lang-expression]
es     | [2021-02-25T09:26:51,003][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [lang-mustache]
es     | [2021-02-25T09:26:51,003][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [lang-painless]
es     | [2021-02-25T09:26:51,003][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [mapper-extras]
es     | [2021-02-25T09:26:51,004][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [parent-join]
es     | [2021-02-25T09:26:51,004][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [percolator]
es     | [2021-02-25T09:26:51,005][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [rank-eval]
es     | [2021-02-25T09:26:51,005][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [reindex]
es     | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [repository-url]
es     | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [transport-netty4]
es     | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [tribe]
es     | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-core]
es     | [2021-02-25T09:26:51,007][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-deprecation]
es     | [2021-02-25T09:26:51,007][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-graph]
es     | [2021-02-25T09:26:51,008][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-logstash]
es     | [2021-02-25T09:26:51,008][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-ml]
es     | [2021-02-25T09:26:51,009][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-monitoring]
es     | [2021-02-25T09:26:51,009][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-rollup]
es     | [2021-02-25T09:26:51,009][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-security]
es     | [2021-02-25T09:26:51,010][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-sql]
es     | [2021-02-25T09:26:51,010][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-upgrade]
es     | [2021-02-25T09:26:51,011][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded module [x-pack-watcher]
es     | [2021-02-25T09:26:51,012][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded plugin [ingest-geoip]
es     | [2021-02-25T09:26:51,012][INFO ][o.e.p.PluginsService     ] [XZz4TuM] loaded plugin [ingest-user-agent]
web_1  | /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.3) or chardet (3.0.4) doesn't match a supported version!
web_1  |   RequestsDependencyWarning)
web_1  | Unable to connect to ES. Retrying in 5 secs...
web_1  | Unable to connect to ES. Retrying in 5 secs...
web_1  | Unable to connect to ES. Retrying in 5 secs...
web_1  | Out of retries. Bailing out...
foodtrucks_web_1 exited with code 1

the content of the flask-app container :

cat flask-app/app.py
from elasticsearch import Elasticsearch, exceptions
import os
import time
from flask import Flask, jsonify, request, render_template
import sys
import requests

es = Elasticsearch(host='es')

app = Flask(__name__)

def load_data_in_es():
    """ creates an index in elasticsearch """
    url = "http://data.sfgov.org/resource/rqzj-sfat.json"
    r = requests.get(url)
    data = r.json()
    print("Loading data in elasticsearch ...")
    for id, truck in enumerate(data):
        res = es.index(index="sfdata", doc_type="truck", id=id, body=truck)
    print("Total trucks loaded: ", len(data))

def safe_check_index(index, retry=3):
    """ connect to ES with retry """
    if not retry:
        print("Out of retries. Bailing out...")
        sys.exit(1)
    try:
        status = es.indices.exists(index)
        return status
    except exceptions.ConnectionError as e:
        print("Unable to connect to ES. Retrying in 5 secs...")
        time.sleep(5)
        safe_check_index(index, retry-1)

def format_fooditems(string):
    items = [x.strip().lower() for x in string.split(":")]
    return items[1:] if items[0].find("cold truck") > -1 else items

def check_and_load_index():
    """ checks if index exits and loads the data accordingly """
    if not safe_check_index('sfdata'):
        print("Index not found...")
        load_data_in_es()

###########
### APP ###
###########
@app.route('/')
def index():
    return render_template('index.html')

@app.route('/debug')
def test_es():
    resp = {}
    try:
        msg = es.cat.indices()
        resp["msg"] = msg
        resp["status"] = "success"
    except:
        resp["status"] = "failure"
        resp["msg"] = "Unable to reach ES"
    return jsonify(resp)

@app.route('/search')
def search():
    key = request.args.get('q')
    if not key:
        return jsonify({
            "status": "failure",
            "msg": "Please provide a query"
        })
    try:
        res = es.search(
                index="sfdata",
                body={
                    "query": {"match": {"fooditems": key}},
                    "size": 750 # max document size
              })
    except Exception as e:
        return jsonify({
            "status": "failure",
            "msg": "error in reaching elasticsearch"
        })
    # filtering results
    vendors = set([x["_source"]["applicant"] for x in res["hits"]["hits"]])
    temp = {v: [] for v in vendors}
    fooditems = {v: "" for v in vendors}
    for r in res["hits"]["hits"]:
        applicant = r["_source"]["applicant"]
        if "location" in r["_source"]:
            truck = {
                "hours"    : r["_source"].get("dayshours", "NA"),
                "schedule" : r["_source"].get("schedule", "NA"),
                "address"  : r["_source"].get("address", "NA"),
                "location" : r["_source"]["location"]
            }
            fooditems[applicant] = r["_source"]["fooditems"]
            temp[applicant].append(truck)

    # building up results
    results = {"trucks": []}
    for v in temp:
        results["trucks"].append({
            "name": v,
            "fooditems": format_fooditems(fooditems[v]),
            "branches": temp[v],
            "drinks": fooditems[v].find("COLD TRUCK") > -1
        })
    hits = len(results["trucks"])
    locations = sum([len(r["branches"]) for r in results["trucks"]])

    return jsonify({
        "trucks": results["trucks"],
        "hits": hits,
        "locations": locations,
        "status": "success"
    })

if __name__ == "__main__":
    ENVIRONMENT_DEBUG = os.environ.get("DEBUG", False)
    check_and_load_index()
    app.run(host='0.0.0.0', port=5000, debug=ENVIRONMENT_DEBUG)

the content of the docker compose

# cat docker-compose.yml 
version: "3"
services:
  es:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
    container_name: es
    environment:
      - discovery.type=single-node
    ports:
      - 9200:9200
    volumes:
      - esdata1:/usr/share/elasticsearch/data
  web:
    image: ducker1989/foodtrucks-web
    command: python3 app.py
    depends_on:
      - es
    ports:
      - 5000:5000
    volumes:
      - ./flask-app:/opt/flask-app
volumes:
    esdata1:
      driver: local

my docker-compose version is :

# docker-compose --version
docker-compose version 1.28.2, build 67630359

What am I missing any advice, indication is greatly welcomed


Solution

  • Well finally I found what is wrong it is the time.sleep(5) ES takes sometime to launch which makes the web up quit before all the services of ES are up which results in the exit of the app changing it to 15 did the trick :

    except exceptions.ConnectionError as e:
        print("Unable to connect to ES. Retrying in 15 secs...")
        time.sleep(15)
        safe_check_index(index, retry-1)