I am trying to have my cluster running on AWS' ECS with the discovery-ec2 plugin for clustering. After starting Elasticsearch it seems to hang and nothing is happening happening anymore after the discovery-ec2 plugin is loaded in, logs (at the end of this post) don't seem to provide much help. Following the various configuration files used and logs:
Dockerfile:
FROM elasticsearch:8.5.3
ENV CLUSTER_NAME default
ENV EC2_INSTANCE_ID undefined
COPY elasticsearch.yml /usr/share/elasticsearch/config/
COPY --chown=elasticsearch:elasticsearch entrypoint.sh /usr/local/bin/new-entrypoint.sh
RUN chmod +x /usr/local/bin/new-entrypoint.sh
RUN echo "y" |bin/elasticsearch-plugin install discovery-ec2
ENTRYPOINT ["/bin/bash", "/usr/local/bin/new-entrypoint.sh"]
new-entrypoint.sh:
#!/bin/bash
export EC2_INSTANCE_ID=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id)
if [[ ! -e "$B64_TLS_CERTIFICATE" ]]; then
echo -n "$B64_TLS_CERTIFICATE" |base64 -d > /usr/share/elasticsearch/config/elastic-certificates.p12
fi
# Resume to base image entrypoint
/bin/tini -s -- /usr/local/bin/docker-entrypoint.sh "$@"
elasticsearch.yaml:
cluster.name: ${CLUSTER_NAME}
node.name: ${EC2_INSTANCE_ID}
network.host: _ec2_
discovery.seed_providers: ec2
discovery.ec2.host_type: private_ip
discovery.ec2.any_group: true
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com
cluster.initial_master_nodes: [i-0595b8xxxx, i-07fa13axxxx, i-0a5c21xxxx]
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
http.port: 9200
transport.tcp.port: 9300
node.data: true
node.master: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
logger.org.elasticsearch.discovery: DEBUG
Logs:
Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
{"@timestamp":"2022-12-30T10:05:36.484Z", "log.level": "INFO", "message":"version[8.5.3], pid[175], build[docker/4ed5ee9afac63de92ec98f404ccbed7d3ba9584e/2022-12-05T18:22:22.226119656Z], OS[Linux/4.14.301-224.520.amzn2.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:36.490Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:36.491Z", "log.level": "INFO", "message":"JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,-Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-13108333766483741156, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1750m, -Xmx1750m, -XX:MaxDirectMemorySize=917504000, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:39.010Z", "log.level": "INFO", "message":"Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.062Z", "log.level":"DEBUG", "message":"Using either environment variables, system properties or instance profile credentials", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.ec2.Ec2ClientSettings","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.144Z", "log.level": "INFO", "message":"loaded module [aggs-matrix-stats]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.144Z", "log.level": "INFO", "message":"loaded module [analysis-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.145Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.145Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [frozen-indices]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [ingest-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.148Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.148Z", "log.level": "INFO", "message":"loaded module [ingest-user-agent]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.148Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.149Z", "log.level": "INFO", "message":"loaded module [lang-mustache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.149Z", "log.level": "INFO", "message":"loaded module [lang-painless]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.154Z", "log.level": "INFO", "message":"loaded module [legacy-geo]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.154Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.155Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.155Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.156Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.156Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.156Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [repository-encrypted]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.158Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.158Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.158Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.160Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.160Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.160Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.161Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.161Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.161Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.162Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.162Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.163Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.163Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.163Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.164Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.164Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.164Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.165Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.165Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.165Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.167Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.167Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.167Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.168Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.168Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.169Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.170Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.171Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.171Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.171Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.172Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.172Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.172Z", "log.level": "INFO", "message":"loaded plugin [discovery-ec2]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.215Z", "log.level":"DEBUG", "message":"obtaining ec2 [placement/availability-zone] from ec2 meta-data url http://169.254.169.254/latest/meta-data/placement/availability-zone", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
Thanks :)
A bit late but posting what I went with in case anybody else is interested in a solution.
I have been doing some back and forth with Elastic on a Github issue, and it appears the plugin does not support IMDSv2 which means it would require containers to have access to the instance role which I view as a breach of the rule of least privilege (the container shall only have access to its task role). But even after running with IMDSv1, it seems like the plugin would not work in a containerized environment, and according to the person who replied on the issue, is by design and not intended to be supported.
To work around this, I made my Elasticsearch tasks to use awsvpc networking mode in order to have a specific ENI attached to it (and de-facto a VPC IP) with Cloud Map configured for the service, ensuring that a Route53 domain maintains an updated list of Elasticsearch addresses even during scaling. You would then have to use the DNS record with Elasticsearch's configuration discovery.seed_hosts for discovery of other nodes.
Following the full configuration:
cluster.name: ${CLUSTER_NAME}
node.name: ${EC2_INSTANCE_ID}
network.host: 0.0.0.0
discovery.seed_hosts: ["${DISCOVERY_DNS_HOST_ADDRESS}"]
cluster.initial_master_nodes: ${CLUSTER_INITIAL_MASTER_NODES} # Shall be removed after the first run
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.monitoring.collection.enabled: true
logger.org.elasticsearch.discovery: ${LOG_LEVEL:INFO}
You should also note that ECS may not be the most appropriate way of running Elasticsearch mostly because ECS does not seem suitable for running stateful applications. Furthermore, running the whole Elastic stack (ES, Kibana, Fleet, Agent, Heartbeat, Custom Alerting) was a lot of frustration, and still is, due to some Elastic products being unstable, random features completely breaking after upgrade or downgrade. If one have the possibility and the resources to, I would recommend going through a fully managed alternative rather than setting it all up.