Search code examples
cebpfxdp-bpf

bpf_map_update_elem() not updating all CPUs for BPF_MAP_TYPE_LRU_PERCPU_HASH

In my XDP program I observe that bpf_map_update_elem() does not update all CPUs for a BPF_MAP_TYPE_LRU_PERCPU_HASH. Here's my map:

struct {
    __uint(type, BPF_MAP_TYPE_LRU_PERCPU_HASH);
    __uint(key_size, sizeof(__u32));
    __uint(value_size, sizeof(__u64));
    __uint(max_entries, 1<<22);
} ips_state SEC(".maps");

Here's how I insert into the map:

__u64 value = bpf_ktime_get_ns();
long status = bpf_map_update_elem(&ips_state, &iph->saddr, &value, BPF_ANY);

When I dump the table using bpftool, I see that only one of the CPUs have a value:

key:
0a 02 01 66
value (CPU 00): 00 00 00 00 00 00 00 00
value (CPU 01): 00 00 00 00 00 00 00 00
...
value (CPU 24): 4f f1 14 a6 1e 6a 5c 00
value (CPU 25): 00 00 00 00 00 00 00 00
...

When I attempt to read from the map in the XDP program, the program reads from CPU other than 24, and I get a value of 0.

How can I update all the CPUs in XDP? I don't see an API specific to updating all the CPUs. Or does this have to be done in user space?

Solution

  • The behavior you are describing is actually a feature of the PERCPU variant of the map. Giving each CPU its own piece of memory allows all CPUs to change values in the map without the need for locking or atomics which increases performance for write-heavy use cases.

    If you want to share data across multiple CPUs you will need to use the non-PERCPU variant and use spinlocks or atomics to avoid race conditions.

    Lastly, in v5.19 the bpf_map_lookup_percpu_elem helper was added to allow programs to read the map elements of PERCPU maps for other CPUs.