I am new to docker-compose and I was following a tutorial the aim is to make ES work with a flask app when I launch the two containers and create the network to isolate them manually everything works fine but when I tried to use docker-compose I have this strange error which looks more of a warning than the root cause of the absence of connection between ES and my flask web app :
docker-compose up
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "foodtrucks_default" with the default driver
Creating es ... done
Creating foodtrucks_web_1 ... done
Attaching to es, foodtrucks_web_1
es | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
es | [2021-02-25T09:26:46,268][INFO ][o.e.n.Node ] [] initializing ...
es | [2021-02-25T09:26:46,480][INFO ][o.e.e.NodeEnvironment ] [XZz4TuM] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [921.1gb], net total_space [982.3gb], types [ext4]
es | [2021-02-25T09:26:46,481][INFO ][o.e.e.NodeEnvironment ] [XZz4TuM] heap size [990.7mb], compressed ordinary object pointers [true]
es | [2021-02-25T09:26:46,486][INFO ][o.e.n.Node ] [XZz4TuM] node name derived from node ID [XZz4TuMGRQmISPJ-VpaDOA]; set [node.name] to override
es | [2021-02-25T09:26:46,487][INFO ][o.e.n.Node ] [XZz4TuM] version[6.3.2], pid[1], build[default/tar/053779d/2018-07-20T05:20:23.451332Z], OS[Linux/5.4.0-65-generic/amd64], JVM["Oracle Corporation"/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13]
es | [2021-02-25T09:26:46,489][INFO ][o.e.n.Node ] [XZz4TuM] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.yFMQabHq, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
es | [2021-02-25T09:26:51,001][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [aggs-matrix-stats]
es | [2021-02-25T09:26:51,002][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [analysis-common]
es | [2021-02-25T09:26:51,002][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [ingest-common]
es | [2021-02-25T09:26:51,002][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [lang-expression]
es | [2021-02-25T09:26:51,003][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [lang-mustache]
es | [2021-02-25T09:26:51,003][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [lang-painless]
es | [2021-02-25T09:26:51,003][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [mapper-extras]
es | [2021-02-25T09:26:51,004][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [parent-join]
es | [2021-02-25T09:26:51,004][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [percolator]
es | [2021-02-25T09:26:51,005][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [rank-eval]
es | [2021-02-25T09:26:51,005][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [reindex]
es | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [repository-url]
es | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [transport-netty4]
es | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [tribe]
es | [2021-02-25T09:26:51,006][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-core]
es | [2021-02-25T09:26:51,007][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-deprecation]
es | [2021-02-25T09:26:51,007][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-graph]
es | [2021-02-25T09:26:51,008][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-logstash]
es | [2021-02-25T09:26:51,008][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-ml]
es | [2021-02-25T09:26:51,009][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-monitoring]
es | [2021-02-25T09:26:51,009][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-rollup]
es | [2021-02-25T09:26:51,009][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-security]
es | [2021-02-25T09:26:51,010][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-sql]
es | [2021-02-25T09:26:51,010][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-upgrade]
es | [2021-02-25T09:26:51,011][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded module [x-pack-watcher]
es | [2021-02-25T09:26:51,012][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded plugin [ingest-geoip]
es | [2021-02-25T09:26:51,012][INFO ][o.e.p.PluginsService ] [XZz4TuM] loaded plugin [ingest-user-agent]
web_1 | /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.3) or chardet (3.0.4) doesn't match a supported version!
web_1 | RequestsDependencyWarning)
web_1 | Unable to connect to ES. Retrying in 5 secs...
web_1 | Unable to connect to ES. Retrying in 5 secs...
web_1 | Unable to connect to ES. Retrying in 5 secs...
web_1 | Out of retries. Bailing out...
foodtrucks_web_1 exited with code 1
the content of the flask-app container :
cat flask-app/app.py
from elasticsearch import Elasticsearch, exceptions
import os
import time
from flask import Flask, jsonify, request, render_template
import sys
import requests
es = Elasticsearch(host='es')
app = Flask(__name__)
def load_data_in_es():
""" creates an index in elasticsearch """
url = "http://data.sfgov.org/resource/rqzj-sfat.json"
r = requests.get(url)
data = r.json()
print("Loading data in elasticsearch ...")
for id, truck in enumerate(data):
res = es.index(index="sfdata", doc_type="truck", id=id, body=truck)
print("Total trucks loaded: ", len(data))
def safe_check_index(index, retry=3):
""" connect to ES with retry """
if not retry:
print("Out of retries. Bailing out...")
sys.exit(1)
try:
status = es.indices.exists(index)
return status
except exceptions.ConnectionError as e:
print("Unable to connect to ES. Retrying in 5 secs...")
time.sleep(5)
safe_check_index(index, retry-1)
def format_fooditems(string):
items = [x.strip().lower() for x in string.split(":")]
return items[1:] if items[0].find("cold truck") > -1 else items
def check_and_load_index():
""" checks if index exits and loads the data accordingly """
if not safe_check_index('sfdata'):
print("Index not found...")
load_data_in_es()
###########
### APP ###
###########
@app.route('/')
def index():
return render_template('index.html')
@app.route('/debug')
def test_es():
resp = {}
try:
msg = es.cat.indices()
resp["msg"] = msg
resp["status"] = "success"
except:
resp["status"] = "failure"
resp["msg"] = "Unable to reach ES"
return jsonify(resp)
@app.route('/search')
def search():
key = request.args.get('q')
if not key:
return jsonify({
"status": "failure",
"msg": "Please provide a query"
})
try:
res = es.search(
index="sfdata",
body={
"query": {"match": {"fooditems": key}},
"size": 750 # max document size
})
except Exception as e:
return jsonify({
"status": "failure",
"msg": "error in reaching elasticsearch"
})
# filtering results
vendors = set([x["_source"]["applicant"] for x in res["hits"]["hits"]])
temp = {v: [] for v in vendors}
fooditems = {v: "" for v in vendors}
for r in res["hits"]["hits"]:
applicant = r["_source"]["applicant"]
if "location" in r["_source"]:
truck = {
"hours" : r["_source"].get("dayshours", "NA"),
"schedule" : r["_source"].get("schedule", "NA"),
"address" : r["_source"].get("address", "NA"),
"location" : r["_source"]["location"]
}
fooditems[applicant] = r["_source"]["fooditems"]
temp[applicant].append(truck)
# building up results
results = {"trucks": []}
for v in temp:
results["trucks"].append({
"name": v,
"fooditems": format_fooditems(fooditems[v]),
"branches": temp[v],
"drinks": fooditems[v].find("COLD TRUCK") > -1
})
hits = len(results["trucks"])
locations = sum([len(r["branches"]) for r in results["trucks"]])
return jsonify({
"trucks": results["trucks"],
"hits": hits,
"locations": locations,
"status": "success"
})
if __name__ == "__main__":
ENVIRONMENT_DEBUG = os.environ.get("DEBUG", False)
check_and_load_index()
app.run(host='0.0.0.0', port=5000, debug=ENVIRONMENT_DEBUG)
the content of the docker compose
# cat docker-compose.yml
version: "3"
services:
es:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: es
environment:
- discovery.type=single-node
ports:
- 9200:9200
volumes:
- esdata1:/usr/share/elasticsearch/data
web:
image: ducker1989/foodtrucks-web
command: python3 app.py
depends_on:
- es
ports:
- 5000:5000
volumes:
- ./flask-app:/opt/flask-app
volumes:
esdata1:
driver: local
my docker-compose version is :
# docker-compose --version
docker-compose version 1.28.2, build 67630359
What am I missing any advice, indication is greatly welcomed
Well finally I found what is wrong it is the time.sleep(5) ES takes sometime to launch which makes the web up quit before all the services of ES are up which results in the exit of the app changing it to 15 did the trick :
except exceptions.ConnectionError as e:
print("Unable to connect to ES. Retrying in 15 secs...")
time.sleep(15)
safe_check_index(index, retry-1)