Search code examples
ubuntudocker-composegraph-databasestriplestoreanzograph

AnzoGraph failing to start up using docker-compose in Travis CI build with Ubuntu Focal VM


I'm trying to start up an AnzoGraph container as part of an experimental Travis CI job. When I test locally (on MacOS) everything works fine. The experimental script starts AnzoGraph using the following docker-compose:

version: '3'

services:

  anzograph:
    image: cambridgesemantics/anzograph:2.5.15
    ports:
      - '8080:8080'
      - '7070:7070'
      - '8443:8443'
    volumes:
      - ./data:/opt/anzograph/persistence/
    shm_size: 8g

The issue is that when I try to run this at a Travis CI build it fails to start AnzoGraph. Here is the OS version of the VM running the Travis CI build:

Operating System Details
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:    20.04
Codename:   focal

And here is the log I could capture through docker-compose logs:

Pulling anzograph (cambridgesemantics/anzograph:2.5.15)...
Creating anzograph_anzograph_1 ... 
 cambridgesemantics/anzograph\nDigest: sha256:6596a6c064bf6b332d5c984f7ddee489f4405e354b45676346f60af37a14af04\nStatus: Downloaded newer image for cambridgesemantics/anzograph:2.5.15\n'
b'Attaching to anzograph_anzograph_1\n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0001-anzograph-pre-requisites.sh\n\x1b[36manzograph_1  |\x1b[0m INFO: detected 31392516 Bytes of total memory\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0002-rsync-data.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Synced data for AnzoGraph, activation done\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Synced data for AnzoGraph, activation done\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Synced data for AnzoGraph, activation done\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0003-anzograph-license-injection.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: No prior license activation found, Continue setup\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: no BYOL\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0004-anzograph-eula-accept.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: CSI Product EULA accepted\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0005-anzograph-set-cluster-ipaddrs.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0006-anzograph-settings-configuration.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Configuring AnzoGraph database settings.\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph settings profile standalone\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: applying AnzoGraph settings defined by profile standalone\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0007-anzograph-persistent-dir-config.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: No prior persistence directory activation found, Continue setup\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Persistence directory not given, taking default: /opt/anzograph/persistence\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0010-anzograph-load-ui-credentials.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activate "admin" password: ok \n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activate "query" password: ok \n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0011-anzograph-load-grpc-credentials.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0015-check-metadata-reachability.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Probing for metadata service at 169.254.169.254:80\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Metadata service not reachable at 169.254.169.254:80\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0016-anzograph-udx-logger-config.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0020-anzograph-ctrl-plane-activation.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph control plane is not started\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Starting AnzoGraph control plane\\n\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph control plane startup triggered\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Wait for AnzoGraph control plane to come up\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: port 5600 at localhost is up\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0030-anzograph-db-activation.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph database is not started\n\x1b[36manzograph_1  |\x1b[0m System error. Contact Cambridge Semantics Support. Reference: 0 == madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Starting AnzoGraph databaseStarting AnzoGraph...\n\x1b[36manzograph_1  |\x1b[0m Error on Node 0 (IP:127.0.0.1) - detected startup failed: Database did not start\n\x1b[36manzograph_1  |\x1b[0m System error. Contact Cambridge Semantics Support. Reference: startup: Process did not succeed - System error. Contact Cambridge Semantics Support. Reference: 0 == madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Logs from /tmp/azgmgrd.log:\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.964072Z: [65:65] Reading /opt/anzograph/config/ip_addrs.conf\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.964929Z: [65:65] AzgState: NotStarted \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.965028Z: [65:65] =============== AnzoGraph 2.5.15.r202303241755 #b74ec7f67be4251a9ec7881b12b94e0e2cc40fef ===============\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.965040Z: [65:65] System manager startup\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.966113Z: [66:67] Starting DB monitoring ....\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.966205Z: [66:67] g_scan_db_pids_file: false\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.968243Z: [66:66] Preparing for secure connections\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.973356Z: [66:66] Sysmgrd 0 listening on 0.0.0.0:5600\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.039447Z: [66:71] StartGqe request on node 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.039486Z: [66:71] waitfor_sibling_mgrs\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.039506Z: [66:71] GetStatus request on node 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041430Z: [66:71] AzgState: Starting \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041493Z: [66:71] Crash DIR name: 735360204\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041824Z: [66:71] Last sysmrd start time: 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041835Z: [66:71] Time /opt/anzograph/config/settings.conf modified: 735360200\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041848Z: [66:71] Sending settings.conf to /opt/anzograph/config on 0 compute nodes\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.042096Z: [66:86] Launched /opt/anzograph/bin/azgd -node 0 -time 735360204 -errpipe /tmp/sbxbooterr.np0 -internal /opt/anzograph/internal -pidfile /tmp/azgpids.log  \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.096817Z: [66:71] \n\x1b[36manzograph_1  |\x1b[0m   -----------------------------------------------\n\x1b[36manzograph_1  |\x1b[0m   error:     Database did not start\n\x1b[36manzograph_1  |\x1b[0m              System error. Contact Cambridge Semantics Support. Reference: startup: Process did not succeed - System error. Contact Cambridge Semantics Support. Reference: 0 == madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument\n\x1b[36manzograph_1  |\x1b[0m   code:      1030\n\x1b[36manzograph_1  |\x1b[0m   entity:    detected startup failed\n\x1b[36manzograph_1  |\x1b[0m   location:  quiet_dbstart() in gsysmgrd.cpp:1019\n\x1b[36manzograph_1  |\x1b[0m   time:      735360204096420 2023-04-21T02:43:24.096420Z\n\x1b[36manzograph_1  |\x1b[0m   pid:       66\n\x1b[36manzograph_1  |\x1b[0m   thread:    86\n\x1b[36manzograph_1  |\x1b[0m   -----------------------------------------------\n\x1b[36manzograph_1  |\x1b[0m ERROR /docker-entrypoint.sh[12]: Failed to start AnzoGraph database\n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0001-anzograph-pre-requisites.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Directory exists\n\x1b[36manzograph_1  |\x1b[0m INFO: detected 31392516 Bytes of total memory\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0002-rsync-data.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activation of etc not required\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activation of config not required\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activation of lib/udx not required\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0003-anzograph-license-injection.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: License activation is already done\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0004-anzograph-eula-accept.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: CSI Product EULA accepted\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0005-anzograph-set-cluster-ipaddrs.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0006-anzograph-settings-configuration.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph database settings are already configured\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0007-anzograph-persistent-dir-config.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0010-anzograph-load-ui-credentials.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Persistent directory activation is already done\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activate "admin" password: ok \n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activate "query" password: ok \n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0011-anzograph-load-grpc-credentials.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0015-check-metadata-reachability.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Check for metadata service skipped\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0016-anzograph-udx-logger-config.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0020-anzograph-ctrl-plane-activation.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph control plane is not started\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Starting AnzoGraph control plane\\n\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph control plane startup triggered\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Wait for AnzoGraph control plane to come up\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: port 5600 at localhost is up\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0030-anzograph-db-activation.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph database is not started\n\x1b[36manzograph_1  |\x1b[0m System error. Contact Cambridge Semantics Support. Reference: 0 == madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Starting AnzoGraph databaseStarting AnzoGraph...\n\x1b[36manzograph_1  |\x1b[0m Error on Node 0 (IP:127.0.0.1) - detected startup failed: Database did not start\n\x1b[36manzograph_1  |\x1b[0m System error. Contact Cambridge Semantics Support. Reference: startup: Process did not succeed - System error. Contact Cambridge Semantics Support. Reference: 0 == madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Logs from /tmp/azgmgrd.log:\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.964072Z: [65:65] Reading /opt/anzograph/config/ip_addrs.conf\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.964929Z: [65:65] AzgState: NotStarted \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.965028Z: [65:65] =============== AnzoGraph 2.5.15.r202303241755 #b74ec7f67be4251a9ec7881b12b94e0e2cc40fef ===============\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.965040Z: [65:65] System manager startup\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.966113Z: [66:67] Starting DB monitoring ....\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.966205Z: [66:67] g_scan_db_pids_file: false\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.968243Z: [66:66] Preparing for secure connections\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:23.973356Z: [66:66] Sysmgrd 0 listening on 0.0.0.0:5600\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.039447Z: [66:71] StartGqe request on node 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.039486Z: [66:71] waitfor_sibling_mgrs\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.039506Z: [66:71] GetStatus request on node 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041430Z: [66:71] AzgState: Starting \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041493Z: [66:71] Crash DIR name: 735360204\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041824Z: [66:71] Last sysmrd start time: 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041835Z: [66:71] Time /opt/anzograph/config/settings.conf modified: 735360200\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.041848Z: [66:71] Sending settings.conf to /opt/anzograph/config on 0 compute nodes\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.042096Z: [66:86] Launched /opt/anzograph/bin/azgd -node 0 -time 735360204 -errpipe /tmp/sbxbooterr.np0 -internal /opt/anzograph/internal -pidfile /tmp/azgpids.log  \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.096817Z: [66:71] \n\x1b[36manzograph_1  |\x1b[0m   -----------------------------------------------\n\x1b[36manzograph_1  |\x1b[0m   error:     Database did not start\n\x1b[36manzograph_1  |\x1b[0m              System error. Contact Cambridge Semantics Support. Reference: startup: Process did not succeed - System error. Contact Cambridge Semantics Support. Reference: 0 == madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument\n\x1b[36manzograph_1  |\x1b[0m ERROR /docker-entrypoint.sh[12]: Failed to start AnzoGraph database\n\x1b[36manzograph_1  |\x1b[0m   code:      1030\n\x1b[36manzograph_1  |\x1b[0m   entity:    detected startup failed\n\x1b[36manzograph_1  |\x1b[0m   location:  quiet_dbstart() in gsysmgrd.cpp:1019\n\x1b[36manzograph_1  |\x1b[0m   time:      735360204096420 2023-04-21T02:43:24.096420Z\n\x1b[36manzograph_1  |\x1b[0m   pid:       66\n\x1b[36manzograph_1  |\x1b[0m   thread:    86\n\x1b[36manzograph_1  |\x1b[0m   -----------------------------------------------\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.937637Z: [37:37] Reading /opt/anzograph/config/ip_addrs.conf\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.938671Z: [37:37] AzgState: NotStarted \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.938746Z: [37:37] =============== AnzoGraph 2.5.15.r202303241755 #b74ec7f67be4251a9ec7881b12b94e0e2cc40fef ===============\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.938761Z: [37:37] System manager startup\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.940064Z: [38:39] Starting DB monitoring ....\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.940150Z: [38:39] g_scan_db_pids_file: false\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.941606Z: [38:38] Preparing for secure connections\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:24.945270Z: [38:38] Sysmgrd 0 listening on 0.0.0.0:5600\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.013970Z: [38:43] StartGqe request on node 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.014009Z: [38:43] waitfor_sibling_mgrs\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.014022Z: [38:43] GetStatus request on node 0\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.015945Z: [38:43] AzgState: Starting \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.016050Z: [38:43] Crash DIR name: 735360205\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.027485Z: [38:43] Last sysmrd start time: 735360204\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.027515Z: [38:43] Time /opt/anzograph/config/settings.conf modified: 735360200\n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.027920Z: [38:63] Launched /opt/anzograph/bin/azgd -node 0 -time 735360205 -errpipe /tmp/sbxbooterr.np0 -internal /opt/anzograph/internal -pidfile /tmp/azgpids.log  \n\x1b[36manzograph_1  |\x1b[0m 2023-04-21 02:43:25.085417Z: [38:43] \n\x1b[36manzograph_1  |\x1b[0m   -----------------------------------------------\n\x1b[36manzograph_1  |\x1b[0m   error:     Database did not start\n\x1b[36manzograph_1  |\x1b[0m              System error. Contact Cambridge Semantics Support. Reference: startup: Process did not succeed - System error. Contact Cambridge Semantics Support. Reference: 0 == madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument\n\x1b[36manzograph_1  |\x1b[0m   code:      1030\n\x1b[36manzograph_1  |\x1b[0m   entity:    detected startup failed\n\x1b[36manzograph_1  |\x1b[0m   location:  quiet_dbstart() in gsysmgrd.cpp:1019\n\x1b[36manzograph_1  |\x1b[0m   time:      735360205085085 2023-04-21T02:43:25.085085Z\n\x1b[36manzograph_1  |\x1b[0m   pid:       38\n\x1b[36manzograph_1  |\x1b[0m   thread:    63\n\x1b[36manzograph_1  |\x1b[0m   -----------------------------------------------\n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0001-anzograph-pre-requisites.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Directory exists\n\x1b[36manzograph_1  |\x1b[0m INFO: detected 31392516 Bytes of total memory\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0002-rsync-data.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activation of etc not required\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activation of config not required\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activation of lib/udx not required\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0003-anzograph-license-injection.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: License activation is already done\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0004-anzograph-eula-accept.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: CSI Product EULA accepted\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0005-anzograph-set-cluster-ipaddrs.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0006-anzograph-settings-configuration.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph database settings are already configured\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0007-anzograph-persistent-dir-config.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Persistent directory activation is already done\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0010-anzograph-load-ui-credentials.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activate "admin" password: ok \n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Activate "query" password: ok \n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0011-anzograph-load-grpc-credentials.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0015-check-metadata-reachability.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Check for metadata service skipped\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0016-anzograph-udx-logger-config.sh\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0020-anzograph-ctrl-plane-activation.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph control plane is not started\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Starting AnzoGraph control plane\\n\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph control plane startup triggered\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: Wait for AnzoGraph control plane to come up\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: port 5600 at localhost is up\n\x1b[36manzograph_1  |\x1b[0m \n\x1b[36manzograph_1  |\x1b[0m /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0030-anzograph-db-activation.sh\n\x1b[36manzograph_1  |\x1b[0m INFO /docker-entrypoint.sh: AnzoGraph database is not started\n'

As I've been able to run this experiment successfully with other triplestores (Jena Fuseki, AllegroGraph and BlazeGraph), I'm wondering what specific characteristic of AnzoGraph (or the way I'm configuring it) is preventing me from doing the same with it.

I've tried using sudo, modifying the docker-compose spec to privileged, use root user and even trying to apply any solution I could find to the error "madvise(base, acquireSize_, MADV_ACQUIRE): Invalid argument", but none of them worked.

Something that might be the cause to it, although I could not be sure of that is that Travis CI's VM does not have the transparent_hugepages directory, as I've checked using the following commands:

sudo ls /sys
...
sudo ls /sys/kernel/
...
sudo ls /sys/kernel/mm
hugepages  swap
sudo ls /sys/kernel/mm/transparent_hugepage
ls: cannot access '/sys/kernel/mm/transparent_hugepage': No such file or directory

Interestingly, I found a closed issue o minikube github (https://github.com/kubernetes/minikube/issues/5765) with the same error I'm facing, but the solution applied to it did not seem to make sense for my case (although I've tried it as well).


Solution

  • I just noticed your post.

    Anzograph DB is a native MPP graphstore, that uses a specific kernel feature that does not seem to be supported by the kernel "your" Travis CI is using. You located the issue (Minikube issue 5765), I filed some years ago against the coreos kernel that minikube had been using at that time, that used to be compiled in a way that did not offer support for this required feature.

    The kernel feature that AnzoGraph expects to be customizable is called "THP" (Transparent HugePage Support).

    As you mentioned your Ubuntu 20.04.5 LTS (x86_64) VM, I just installed "Ubuntu 20.04.6 LTS (Focal Fossa)" as a VM.

    This Ubuntu install is using linux-image-5.4.0-147-generic/focal-updates,focal-security,now 5.4.0-147.164 amd 64 as kernel.

    I see full THP support offered:

    ls /sys/kernel/mm/transparent_hugepage defrag enabled hpage_pmd_size khugepaged shmem_enabled use_zero_page

    I do not fully understand your setup in regards to Travis CI, but the key issue is the docker host kernel that your Travis CI is using, is lacking the required THP support.