configurationcluster-computingeventstoredb
Correct EventStore config for a 3 node cluster?
So I had EventStore 5.0.7 installed as a 3 node cluster, working just fine.
I tried to upgrade to EventStore 21.10.1. The config for EventStore has changed substantially since the move from 5.x to 20.x and 21.x, and despite multiple readings of all kinds of documentation, I'm still doing something wrong.
What we see is 6 nodes appearing - each server twice - and the gossip failing, and nothing working, ie, cannot insert events.
What am I doing wrong?
EventStore 5.0.7
EventStore 21.10.1
Config for EventStore 21.10.1
---
# Paths
Db: /var/lib/eventstore
Index: /var/lib/eventstore/index
Log: /var/log/eventstore
# Run in insecure mode
Insecure: true
DisableInternalTcpTls: true
DisableExternalTcpTls: true
# Network configuration
IntIp: 172.31.47.243
ExtIp: 0.0.0.0
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: false
# Projections configuration
RunProjections: System
ClusterSize: 3
LogLevel: Verbose
LogHttpRequests: true
LogFailedAuthenticationAttempts: true
LogConfig: /etc/eventstore/logconfig.json
HttpPortAdvertiseAs: 2114
ExtHostAdvertiseAs: 54.209.234.141
IntTcpHeartbeatTimeout: 2000
ExtTcpHeartbeatTimeout: 2000
IntTcpHeartbeatInterval: 5000
ExtTcpHeartbeatInterval: 5000
GossipTimeoutMs: 5000
GossipIntervalMs: 2000
StatsPeriodSec: 900
DiscoverViaDns: false
GossipSeed: 172.31.45.192:2113,172.31.41.141:2113
Config for EventStore 21.10.1 (as seen at startup)
MODIFIED OPTIONS:
STATS PERIOD SEC: 900 (Yaml)
LOG HTTP REQUESTS: true (Yaml)
LOG FAILED AUTHENTICATION ATTEMPTS: true (Yaml)
INSECURE: true (Yaml)
LOG: /var/log/eventstore (Yaml)
LOG CONFIG: /etc/eventstore/logconfig.json (Yaml)
LOG LEVEL: Verbose (Yaml)
CLUSTER SIZE: 3 (Yaml)
DISCOVER VIA DNS: false (Yaml)
GOSSIP SEED: 172.31.46.96:2113,172.31.40.110:2113 (Yaml)
GOSSIP INTERVAL MS: 2000 (Yaml)
GOSSIP TIMEOUT MS: 5000 (Yaml)
DB: /var/lib/eventstore (Yaml)
INDEX: /var/lib/eventstore/index (Yaml)
INT IP: 172.31.35.133 (Yaml)
EXT IP: 0.0.0.0 (Yaml)
HTTP PORT: 2113 (Yaml)
ENABLE EXTERNAL TCP: true (Yaml)
INT TCP PORT: 1112 (Yaml)
EXT TCP PORT: 1113 (Yaml)
EXT HOST ADVERTISE AS: 3.82.200.231 (Yaml)
HTTP PORT ADVERTISE AS: 2114 (Yaml)
INT TCP HEARTBEAT TIMEOUT: 2000 (Yaml)
EXT TCP HEARTBEAT TIMEOUT: 2000 (Yaml)
INT TCP HEARTBEAT INTERVAL: 5000 (Yaml)
EXT TCP HEARTBEAT INTERVAL: 5000 (Yaml)
DISABLE INTERNAL TCP TLS: true (Yaml)
DISABLE EXTERNAL TCP TLS: true (Yaml)
ENABLE ATOM PUB OVER HTTP: false (Yaml)
RUN PROJECTIONS: System (Yaml)
DEFAULT OPTIONS:
HELP: False (<DEFAULT>)
VERSION: False (<DEFAULT>)
CONFIG: /etc/eventstore/eventstore.conf (<DEFAULT>)
WHAT IF: False (<DEFAULT>)
START STANDARD PROJECTIONS: False (<DEFAULT>)
DISABLE HTTP CACHING: False (<DEFAULT>)
WORKER THREADS: 0 (<DEFAULT>)
ENABLE HISTOGRAMS: False (<DEFAULT>)
SKIP INDEX SCAN ON READS: False (<DEFAULT>)
MAX APPEND SIZE: 1048576 (<DEFAULT>)
LOG CONSOLE FORMAT: Plain (<DEFAULT>)
LOG FILE SIZE: 1073741824 (<DEFAULT>)
LOG FILE INTERVAL: Day (<DEFAULT>)
LOG FILE RETENTION COUNT: 31 (<DEFAULT>)
DISABLE LOG FILE: False (<DEFAULT>)
AUTHORIZATION TYPE: internal (<DEFAULT>)
AUTHORIZATION CONFIG: <empty> (<DEFAULT>)
AUTHENTICATION TYPE: internal (<DEFAULT>)
AUTHENTICATION CONFIG: <empty> (<DEFAULT>)
DISABLE FIRST LEVEL HTTP AUTHORIZATION: False (<DEFAULT>)
TRUSTED ROOT CERTIFICATES PATH: <empty> (<DEFAULT>)
CERTIFICATE RESERVED NODE COMMON NAME: eventstoredb-node (<DEFAULT>)
CERTIFICATE FILE: <empty> (<DEFAULT>)
CERTIFICATE PRIVATE KEY FILE: <empty> (<DEFAULT>)
CERTIFICATE PASSWORD: <empty> (<DEFAULT>)
CERTIFICATE STORE LOCATION: <empty> (<DEFAULT>)
CERTIFICATE STORE NAME: <empty> (<DEFAULT>)
CERTIFICATE SUBJECT NAME: <empty> (<DEFAULT>)
CERTIFICATE THUMBPRINT: <empty> (<DEFAULT>)
STREAM INFO CACHE CAPACITY: 0 (<DEFAULT>)
NODE PRIORITY: 0 (<DEFAULT>)
COMMIT COUNT: -1 (<DEFAULT>)
PREPARE COUNT: -1 (<DEFAULT>)
CLUSTER DNS: fake.dns (<DEFAULT>)
CLUSTER GOSSIP PORT: 2113 (<DEFAULT>)
GOSSIP ALLOWED DIFFERENCE MS: 60000 (<DEFAULT>)
READ ONLY REPLICA: False (<DEFAULT>)
UNSAFE ALLOW SURPLUS NODES: False (<DEFAULT>)
DEAD MEMBER REMOVAL PERIOD SEC: 1800 (<DEFAULT>)
LEADER ELECTION TIMEOUT MS: 1000 (<DEFAULT>)
QUORUM SIZE: 1 (<DEFAULT>)
PREPARE ACK COUNT: 1 (<DEFAULT>)
COMMIT ACK COUNT: 1 (<DEFAULT>)
MIN FLUSH DELAY MS: 2 (<DEFAULT>)
DISABLE SCAVENGE MERGING: False (<DEFAULT>)
SCAVENGE HISTORY MAX AGE: 30 (<DEFAULT>)
CACHED CHUNKS: -1 (<DEFAULT>)
CHUNKS CACHE SIZE: 536871424 (<DEFAULT>)
MAX MEM TABLE SIZE: 1000000 (<DEFAULT>)
HASH COLLISION READ LIMIT: 100 (<DEFAULT>)
MEM DB: False (<DEFAULT>)
USE INDEX BLOOM FILTERS: True (<DEFAULT>)
INDEX CACHE SIZE: 0 (<DEFAULT>)
SKIP DB VERIFY: False (<DEFAULT>)
WRITE THROUGH: False (<DEFAULT>)
UNBUFFERED: False (<DEFAULT>)
CHUNK INITIAL READER COUNT: 5 (<DEFAULT>)
PREPARE TIMEOUT MS: 2000 (<DEFAULT>)
COMMIT TIMEOUT MS: 2000 (<DEFAULT>)
WRITE TIMEOUT MS: 2000 (<DEFAULT>)
UNSAFE DISABLE FLUSH TO DISK: False (<DEFAULT>)
UNSAFE IGNORE HARD DELETE: False (<DEFAULT>)
SKIP INDEX VERIFY: False (<DEFAULT>)
INDEX CACHE DEPTH: 16 (<DEFAULT>)
OPTIMIZE INDEX MERGE: False (<DEFAULT>)
ALWAYS KEEP SCAVENGED: False (<DEFAULT>)
REDUCE FILE CACHE PRESSURE: False (<DEFAULT>)
INITIALIZATION THREADS: 1 (<DEFAULT>)
READER THREADS COUNT: 0 (<DEFAULT>)
MAX AUTO MERGE INDEX LEVEL: 2147483647 (<DEFAULT>)
WRITE STATS TO DB: False (<DEFAULT>)
MAX TRUNCATION: 268435456 (<DEFAULT>)
CHUNK SIZE: 268435456 (<DEFAULT>)
STATS STORAGE: File (<DEFAULT>)
DB LOG FORMAT: V2 (<DEFAULT>)
STREAM EXISTENCE FILTER SIZE: 256000000 (<DEFAULT>)
KEEP ALIVE INTERVAL: 10000 (<DEFAULT>)
KEEP ALIVE TIMEOUT: 10000 (<DEFAULT>)
INT HOST ADVERTISE AS: <empty> (<DEFAULT>)
ADVERTISE HOST TO CLIENT AS: <empty> (<DEFAULT>)
ADVERTISE HTTP PORT TO CLIENT AS: 0 (<DEFAULT>)
ADVERTISE TCP PORT TO CLIENT AS: 0 (<DEFAULT>)
EXT TCP PORT ADVERTISE AS: 0 (<DEFAULT>)
INT TCP PORT ADVERTISE AS: 0 (<DEFAULT>)
GOSSIP ON SINGLE NODE: <empty> (<DEFAULT>)
CONNECTION PENDING SEND BYTES THRESHOLD: 10485760 (<DEFAULT>)
CONNECTION QUEUE SIZE THRESHOLD: 50000 (<DEFAULT>)
DISABLE ADMIN UI: False (<DEFAULT>)
DISABLE STATS ON HTTP: False (<DEFAULT>)
DISABLE GOSSIP ON HTTP: False (<DEFAULT>)
ENABLE TRUSTED AUTH: False (<DEFAULT>)
PROJECTION THREADS: 3 (<DEFAULT>)
PROJECTIONS QUERY EXPIRY: 5 (<DEFAULT>)
FAULT OUT OF ORDER PROJECTIONS: False (<DEFAULT>)
PROJECTION COMPILATION TIMEOUT: 500 (<DEFAULT>)
PROJECTION EXECUTION TIMEOUT: 250 (<DEFAULT>)
Gossip for EventStore 21.10.1
{
"members": [
{
"instanceId": "ed2ee047-eb59-4b11-86fd-a5b366edd0ce",
"timeStamp": "2022-01-12T23:17:42.539034Z",
"state": "Unknown",
"isAlive": true,
"internalTcpIp": "172.31.46.231",
"internalTcpPort": 1112,
"internalSecureTcpPort": 0,
"externalTcpIp": "52.91.48.59",
"externalTcpPort": 1113,
"externalSecureTcpPort": 0,
"httpEndPointIp": "52.91.48.59",
"httpEndPointPort": 2114,
"lastCommitPosition": -1,
"writerCheckpoint": 0,
"chaserCheckpoint": 0,
"epochPosition": -1,
"epochNumber": -1,
"epochId": "00000000-0000-0000-0000-000000000000",
"nodePriority": 0,
"isReadOnlyReplica": false
},
{
"instanceId": "dfcc4139-2966-454c-8cee-71261cedafba",
"timeStamp": "2022-01-12T23:17:40.0803168Z",
"state": "Unknown",
"isAlive": false,
"internalTcpIp": "172.31.46.43",
"internalTcpPort": 1112,
"internalSecureTcpPort": 0,
"externalTcpIp": "44.201.237.180",
"externalTcpPort": 1113,
"externalSecureTcpPort": 0,
"httpEndPointIp": "44.201.237.180",
"httpEndPointPort": 2114,
"lastCommitPosition": -1,
"writerCheckpoint": 0,
"chaserCheckpoint": 0,
"epochPosition": -1,
"epochNumber": -1,
"epochId": "00000000-0000-0000-0000-000000000000",
"nodePriority": 0,
"isReadOnlyReplica": false
},
{
"instanceId": "2a47929c-afd6-496f-b87b-d85904eeed18",
"timeStamp": "2022-01-12T23:17:40.539795Z",
"state": "Unknown",
"isAlive": true,
"internalTcpIp": "172.31.38.246",
"internalTcpPort": 1112,
"internalSecureTcpPort": 0,
"externalTcpIp": "3.93.17.39",
"externalTcpPort": 1113,
"externalSecureTcpPort": 0,
"httpEndPointIp": "3.93.17.39",
"httpEndPointPort": 2114,
"lastCommitPosition": -1,
"writerCheckpoint": 0,
"chaserCheckpoint": 0,
"epochPosition": -1,
"epochNumber": -1,
"epochId": "00000000-0000-0000-0000-000000000000",
"nodePriority": 0,
"isReadOnlyReplica": false
},
{
"instanceId": "00000000-0000-0000-0000-000000000000",
"timeStamp": "2022-01-12T22:39:46.4047071Z",
"state": "Manager",
"isAlive": true,
"internalTcpIp": "172.31.46.43",
"internalTcpPort": 2113,
"internalSecureTcpPort": 0,
"externalTcpIp": "172.31.46.43",
"externalTcpPort": 2113,
"externalSecureTcpPort": 0,
"httpEndPointIp": "172.31.46.43",
"httpEndPointPort": 2113,
"lastCommitPosition": -1,
"writerCheckpoint": -1,
"chaserCheckpoint": -1,
"epochPosition": -1,
"epochNumber": -1,
"epochId": "00000000-0000-0000-0000-000000000000",
"nodePriority": 0,
"isReadOnlyReplica": false
},
{
"instanceId": "00000000-0000-0000-0000-000000000000",
"timeStamp": "2022-01-12T22:53:47.9621597Z",
"state": "Manager",
"isAlive": true,
"internalTcpIp": "172.31.46.231",
"internalTcpPort": 2113,
"internalSecureTcpPort": 0,
"externalTcpIp": "172.31.46.231",
"externalTcpPort": 2113,
"externalSecureTcpPort": 0,
"httpEndPointIp": "172.31.46.231",
"httpEndPointPort": 2113,
"lastCommitPosition": -1,
"writerCheckpoint": -1,
"chaserCheckpoint": -1,
"epochPosition": -1,
"epochNumber": -1,
"epochId": "00000000-0000-0000-0000-000000000000",
"nodePriority": 0,
"isReadOnlyReplica": false
},
{
"instanceId": "00000000-0000-0000-0000-000000000000",
"timeStamp": "2022-01-12T22:53:47.9621597Z",
"state": "Manager",
"isAlive": true,
"internalTcpIp": "172.31.38.246",
"internalTcpPort": 2113,
"internalSecureTcpPort": 0,
"externalTcpIp": "172.31.38.246",
"externalTcpPort": 2113,
"externalSecureTcpPort": 0,
"httpEndPointIp": "172.31.38.246",
"httpEndPointPort": 2113,
"lastCommitPosition": -1,
"writerCheckpoint": -1,
"chaserCheckpoint": -1,
"epochPosition": -1,
"epochNumber": -1,
"epochId": "00000000-0000-0000-0000-000000000000",
"nodePriority": 0,
"isReadOnlyReplica": false
}
],
"serverIp": "52.91.48.59",
"serverPort": 2114
}
Solution
This online tool : https://configurator.eventstore.com/ should help you setup the configuration correctly
- ER_NOT_SUPPORTED_AUTH_MODE - MySQL server
- How to see SET/GET commands in Redis
- nginx the "ssl" directive is deprecated, use the "listen ... ssl"
- Klaro configuration, there is a way to disable the 'auto script execution' after giving the consent?
- Using System.Text.Json to Serialize an IConfiguration back to Json
- git fatal error: Unsupported SSL backend 'schannel'
- VSCode change debug console name automatically
- How do I read configuration settings from Symfony2 config.yml?
- Apache Apisix state serialization
- Spring Boot 3 - property binding into Map "duplicates" keys
- where should I locate a common ESAPI.properties file used by multiple wars inside an ear?
- Using RedisTagAwareAdapter in Symfony 4.4
- How to properly install and configure JSF libraries via Maven?
- pgadmin4 : postgresql application server could not be contacted
- Copying text outside of Vim with set mouse=a enabled
- Database (database/database.sqlite) does not exist. Database works from artisan tinker
- Why doesn't strongly typed Configuration work with AOT
- Deploying Jakarta EE Application on Jetty 12: from jetty-ee10-maven-plugin to deployment on Jetty, ClassNotFoundException and CDIProvider Issues
- TensorFlow libdevice not found. Why is it not found in the searched path?
- Turning off eslint rule for a specific file
- Rancher Desktop: How to set insecure-repositories on Windows, as set in /etc/docker/daemon.json file
- What is the best method for setting up a config file in Python
- How to use Nix as a Package Manager for Neovim to manage plugins and configurations?
- Datanode process not running in Hadoop
- Python - Can't find pip.ini or pip.conf in Windows
- Why does IOptions<T> work with implicit configuration binding in one ASP.NET Core example but not in another?
- Rider - I can't start a project if there are uploaded projects
- Configuring Elasticsearch and Kibana to work with each other without requiring enrollment token
- Spring boot application properties are not reading in AWS environment
- Making Sphinx detect changes to auxiliary configuration/template files