Why does the gold linker cause dl_iterate_phdr() not to return my custom note section?

On Linux, I would like to store some structures in a custom .note.foobar section and discover them at runtime.

I compile and link the program below once with gold and once without:

$ gcc -o test-ld test.c
$ gcc -o test-gold -fuse-ld=gold test.c

You can see that the ld-linked version finds the section while the gold-linked version does not:

$ ./test-ld
note section at vaddr: 2c4
note section at vaddr: 2f0
found f00dface
note section at vaddr: 324
note section at vaddr: 7a8
note section at vaddr: 270
note section at vaddr: 1c8
$ ./test-gold
note section at vaddr: 254
note section at vaddr: 7a8
note section at vaddr: 270
note section at vaddr: 1c8

However, the section does exist in both binaries:

$ readelf -x .note.foobar test-ld

Hex dump of section '.note.foobar':
  0x000002f0 04000000 14000000 67452301 666f6f00 ........gE#.foo.
  0x00000300 cefa0df0 00000000 00000000 00000000 ................
  0x00000310 04000000 14000000 67452301 666f6f00 ........gE#.foo.
  0x00000320 efbeadde                            ....

$ readelf -x .note.foobar test-gold 

Hex dump of section '.note.foobar':
  0x00000280 04000000 14000000 67452301 666f6f00 ........gE#.foo.
  0x00000290 cefa0df0 00000000 00000000 00000000 ................
  0x000002a0 04000000 14000000 67452301 666f6f00 ........gE#.foo.
  0x000002b0 efbeadde                            ....

So you would expect the test-gold program to report a section at vaddr 280, but it does not.

Why can dl_iterate_phdr not find this section, while readelf can, and what is gold doing differently to cause this?

#define _GNU_SOURCE
#include <link.h>
#include <stdlib.h>
#include <stdio.h>

typedef struct {
  unsigned int elf_namesize;
  unsigned int elf_datasize;
  unsigned int elf_type;
  unsigned int elf_name;
  unsigned int bar;
} foo_t;

const foo_t __attribute__((used,section(".note.foobar,\"a\"#"))) foo1 = {
  4,
  20,
  0x01234567,
  0x6f6f66,
  0xf00dface,
};
const foo_t __attribute__((used,section(".note.foobar,\"a\"#"))) foo2 = {
  4,
  20,
  0x01234567,
  0x6f6f66,
  0xdeadbeef,
};

static int
callback(struct dl_phdr_info *info, size_t size, void *data)
{
  for (int i = 0; i < info->dlpi_phnum; i++) {
    const ElfW(Phdr)* phdr = &info->dlpi_phdr[i];
    if (phdr->p_type == PT_NOTE) {
      foo_t *payload = (foo_t*)(info->dlpi_addr + phdr->p_vaddr);
      printf("note section at vaddr: %lx\n", phdr->p_vaddr);
      if (phdr->p_memsz >= sizeof(foo_t) && payload->elf_type == 0x01234567 && payload->elf_name == 0x6f6f66) {
        printf("found %x\n", payload->bar);
      }
    }
  }

  return 0;
}

int
main(int argc, char *argv[])
{
  dl_iterate_phdr(callback, NULL);
  return 0;
}

Solution

This code:

foo_t *payload = (foo_t*)(info->dlpi_addr + phdr->p_vaddr);

assumes that your .note.foobar is the very first Elf...Note in the PT_NOTE segment, but you can't make that assumption -- the order of notes in PT_NOTE is not guaranteed; you need to iterate over all of them.

You can verify that there are multiple notes with readelf -n test-{ld,gold}.

It appears that GNU-ld emits a separate PT_NOTE for each .note* section, while Gold merges them all into a single PT_NOTE segment. Either behavior is perfectly fine as far as ELF standard is concerned, though GNU-ld is wasteful (there is no need to emit extra PT_NOTE program headers).

Here is what I get for your test program:

readelf -l test-ld | grep NOTE
  NOTE           0x00000000000002c4 0x00000000004002c4 0x00000000004002c4
  NOTE           0x00000000000002f0 0x00000000004002f0 0x00000000004002f0
  NOTE           0x0000000000000324 0x0000000000400324 0x0000000000400324

readelf -l test-gold | grep NOTE
  NOTE           0x0000000000000254 0x0000000000400254 0x0000000000400254

P.S.

Why does the gold linker cause dl_iterate_phdr() not to return my custom note section?

The direct answer is that dl_iterate_phdr doesn't deal with (or care) about sections. It iterates over segments, and assignment of sections to segments is up for linkers to perform as they see fit.