Search code examples
kubernetesenvoyproxyconfigmap

dynamic envoy configuration from k8s configmap

I tried to configure envoy in my kubernetes cluster by following this example: https://www.envoyproxy.io/docs/envoy/latest/start/quick-start/configuration-dynamic-filesystem

My static envoy config:

    node:
      cluster: test-cluster
      id: test-id

    dynamic_resources:
      cds_config:
        path: /var/lib/envoy/cds.yaml
      lds_config:
        path: /var/lib/envoy/lds.yaml

    admin:
      access_log_path: "/dev/null"
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 19000

The dynamic config from configmap is mounted to and contains the files .

I used a configmap to mount the config files (cds.yaml and lds.yaml) into to envoy pod (to /var/lib/envoy/) but unfortunately the envoy configuration doesn't change when I change the config in the configmap. The mounted config files are updated as expected.

I can see from the logs, that envoy watches the config files:

[2021-03-01 17:50:21.063][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:47] added watch for directory: '/var/lib/envoy' file: 'cds.yaml' fd: 1
[2021-03-01 17:50:21.063][1][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:140] maybe finish initialize state: 1
[2021-03-01 17:50:21.063][1][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:149] maybe finish initialize primary init clusters empty: true
[2021-03-01 17:50:21.063][1][info][config] [source/server/configuration_impl.cc:95] loading 0 listener(s)
[2021-03-01 17:50:21.063][1][info][config] [source/server/configuration_impl.cc:107] loading stats configuration
[2021-03-01 17:50:21.063][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:47] added watch for directory: '/var/lib/envoy' file: 'lds.yaml' fd: 1

and once I update the configmap I also get the logs that something changed:

[2021-03-01 17:51:50.881][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 80 file: ..data
[2021-03-01 17:51:50.881][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 80 file: ..data

but envoy doesn't reload the config.

It seems that kubernetes updates the config files by changing a directory and envoy doesn't recognise that the config files are changed.

Is there an easy way to fix that? I don't want to run and xDS server for my tests but hot config reload would be great for my testing 😇

Thanks!

Solution

  • I think the answer to your issue is that the filesystem events that Envoy uses to reload its xDS config are not triggered by configmap volumes. See more explanation in the README for the crossover utility.