Search code examples
variableslinux-kerneloperating-systemkernel

Where are Kernel global variables stored?


As i know global variables(in userspace programs) are stored in coressponding DATA (and BSS) segment. But how and where are they stored in case of kernel code?


Solution

  • For an example let's look at one of Linux's global variable. Basically, Linux is an ELF image compressed with gzip or other compression tools. As stated on Wikipedia (https://en.wikipedia.org/wiki/Vmlinux):

    Traditionally, when creating a bootable kernel image, the kernel is also compressed using gzip, or, since Linux 2.6.30,[3] using LZMA or bzip2, which requires a very small decompression stub to be included in the resulting image. The stub decompresses the kernel code, on some systems printing dots to the console to indicate progress, and then continues the boot process. Support for LZO,[4] xz[5] and LZ4[6] compression was added later.

    On Ubuntu, there is a special script called extract-vmlinux which comes with your version of the Linux kernel (https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/). The Linux ELF file is /boot/vmlinuz. To decompress it, you must use the extract-vmlinux script which can be installed with

    sudo apt-get install linux-headers-$(uname -r)
    

    The script is safe to use because it is a standard script which is found in the official repository of packages for your distribution (most packages there are open source and safe). Now use

    sudo /usr/src/kernels/$(uname -r)/scripts/extract-vmlinux /boot/vmlinuz-$(uname -r) > vmlinux
    

    to extract the Linux kernel's ELF file from the compressed vmlinux image to your home repository. Now we can look at the ELF file of the Linux kernel. Use readelf -a vmlinux to have a summary of the content. The output for me is this:

    user@user-System-Product-Name:~$ readelf -a vmlinux
    ELF Header:
      Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
      Class:                             ELF64
      Data:                              2's complement, little endian
      Version:                           1 (current)
      OS/ABI:                            UNIX - System V
      ABI Version:                       0
      Type:                              EXEC (Executable file)
      Machine:                           Advanced Micro Devices X86-64
      Version:                           0x1
      Entry point address:               0x1000000
      Start of program headers:          64 (bytes into file)
      Start of section headers:          46137728 (bytes into file)
      Flags:                             0x0
      Size of this header:               64 (bytes)
      Size of program headers:           56 (bytes)
      Number of program headers:         5
      Size of section headers:           64 (bytes)
      Number of section headers:         35
      Section header string table index: 34
    
    Section Headers:
      [Nr] Name              Type             Address           Offset
           Size              EntSize          Flags  Link  Info  Align
      [ 0]                   NULL             0000000000000000  00000000
           0000000000000000  0000000000000000           0     0     0
      [ 1] .text             PROGBITS         ffffffff81000000  00200000
           0000000000e025f7  0000000000000000  AX       0     0     4096
      [ 2] .rodata           PROGBITS         ffffffff82000000  01200000
           00000000004e46f2  0000000000000000  WA       0     0     4096
      [ 3] .pci_fixup        PROGBITS         ffffffff824e4700  016e4700
           00000000000032b0  0000000000000000   A       0     0     16
      [ 4] .tracedata        PROGBITS         ffffffff824e79b0  016e79b0
           0000000000000078  0000000000000000   A       0     0     1
      [ 5] __ksymtab         PROGBITS         ffffffff824e7a28  016e7a28
           0000000000010a34  0000000000000000   A       0     0     4
      [ 6] __ksymtab_gpl     PROGBITS         ffffffff824f845c  016f845c
           00000000000121ec  0000000000000000   A       0     0     4
      [ 7] __kcrctab         PROGBITS         ffffffff8250a648  0170a648
           00000000000058bc  0000000000000000   A       0     0     4
      [ 8] __kcrctab_gpl     PROGBITS         ffffffff8250ff04  0170ff04
           00000000000060a4  0000000000000000   A       0     0     4
      [ 9] __ksymtab_strings PROGBITS         ffffffff82515fa8  01715fa8
           0000000000037bea  0000000000000001 AMS       0     0     1
      [10] __init_rodata     PROGBITS         ffffffff8254dba0  0174dba0
           00000000000002a8  0000000000000000   A       0     0     32
      [11] __param           PROGBITS         ffffffff8254de48  0174de48
           00000000000038e0  0000000000000000   A       0     0     8
      [12] __modver          PROGBITS         ffffffff82551728  01751728
           00000000000005a0  0000000000000000  WA       0     0     8
      [13] __ex_table        PROGBITS         ffffffff82551cd0  01751cd0
           0000000000001af4  0000000000000000   A       0     0     4
      [14] .notes            NOTE             ffffffff825537c4  017537c4
           00000000000001ec  0000000000000000   A       0     0     4
      [15] .data             PROGBITS         ffffffff82600000  01800000
           00000000003657c0  0000000000000000  WA       0     0     8192
      [16] __bug_table       PROGBITS         ffffffff829657c0  01b657c0
           00000000000178d4  0000000000000000  WA       0     0     1
      [17] .vvar             PROGBITS         ffffffff8297e000  01b7e000
           0000000000001000  0000000000000000  WA       0     0     16
      [18] .data..percpu     PROGBITS         0000000000000000  01c00000
           000000000002f000  0000000000000000  WA       0     0     4096
      [19] .init.text        PROGBITS         ffffffff829ae000  01dae000
           00000000000772e5  0000000000000000  AX       0     0     16
      [20] .altinstr_aux     PROGBITS         ffffffff82a252e5  01e252e5
           0000000000002d9b  0000000000000000  AX       0     0     1
      [21] .init.data        PROGBITS         ffffffff82a2a000  01e2a000
           00000000001c74f0  0000000000000000  WA       0     0     8192
      [22] .x86_cpu_dev.init PROGBITS         ffffffff82bf14f0  01ff14f0
           0000000000000028  0000000000000000   A       0     0     8
      [23] .parainstructions PROGBITS         ffffffff82bf1518  01ff1518
           0000000000022b7c  0000000000000000   A       0     0     8
      [24] .altinstructions  PROGBITS         ffffffff82c14098  02014098
           0000000000006660  0000000000000000   A       0     0     1
      [25] .altinstr_replace PROGBITS         ffffffff82c1a6f8  0201a6f8
           0000000000001789  0000000000000000  AX       0     0     1
      [26] .iommu_table      PROGBITS         ffffffff82c1be88  0201be88
           00000000000000f0  0000000000000000   A       0     0     8
      [27] .apicdrivers      PROGBITS         ffffffff82c1bf78  0201bf78
           0000000000000040  0000000000000000  WA       0     0     8
      [28] .exit.text        PROGBITS         ffffffff82c1bfb8  0201bfb8
           0000000000001f4b  0000000000000000  AX       0     0     1
      [29] .smp_locks        PROGBITS         ffffffff82c1e000  0201e000
           000000000000a000  0000000000000000   A       0     0     4
      [30] .data_nosave      PROGBITS         ffffffff82c28000  02028000
           0000000000001000  0000000000000000  WA       0     0     4
      [31] .bss              NOBITS           ffffffff82c29000  02029000
           00000000005d7000  0000000000000000  WA       0     0     4096
      [32] .brk              NOBITS           ffffffff83200000  02029000
           000000000002c000  0000000000000000  WA       0     0     1
      [33] .init.scratch     PROGBITS         ffffffff83400000  02800000
           0000000000400000  0000000000000000  WA       0     0     32
      [34] .shstrtab         STRTAB           0000000000000000  02c00000
           0000000000000180  0000000000000000           0     0     1
    Key to Flags:
      W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
      L (link order), O (extra OS processing required), G (group), T (TLS),
      C (compressed), x (unknown), o (OS specific), E (exclude),
      l (large), p (processor specific)
    
    There are no section groups in this file.
    
    Program Headers:
      Type           Offset             VirtAddr           PhysAddr
                     FileSiz            MemSiz              Flags  Align
      LOAD           0x0000000000200000 0xffffffff81000000 0x0000000001000000
                     0x00000000015539b0 0x00000000015539b0  R E    0x200000
      LOAD           0x0000000001800000 0xffffffff82600000 0x0000000002600000
                     0x000000000037f000 0x000000000037f000  RW     0x200000
      LOAD           0x0000000001c00000 0x0000000000000000 0x000000000297f000
                     0x000000000002f000 0x000000000002f000  RW     0x200000
      LOAD           0x0000000001dae000 0xffffffff829ae000 0x00000000029ae000
                     0x0000000000e52000 0x0000000000e52000  RWE    0x200000
      NOTE           0x00000000017537c4 0xffffffff825537c4 0x00000000025537c4
                     0x00000000000001ec 0x00000000000001ec         0x4
    
     Section to Segment mapping:
      Segment Sections...
       00     .text .rodata .pci_fixup .tracedata __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __init_rodata __param __modver __ex_table .notes 
       01     .data __bug_table .vvar 
       02     .data..percpu 
       03     .init.text .altinstr_aux .init.data .x86_cpu_dev.init .parainstructions .altinstructions .altinstr_replacement .iommu_table .apicdrivers .exit.text .smp_locks .data_nosave .bss .brk .init.scratch 
       04     .notes 
    
    There is no dynamic section in this file.
    
    There are no relocations in this file.
    
    The decoding of unwind sections for machine type Advanced Micro Devices X86-64 is not currently supported.
    
    No version information found in this file.
    
    Displaying notes found in: .notes
      Owner                Data size    Description
      Xen                  0x00000006   Unknown note type: (0x00000006)
       description data: 6c 69 6e 75 78 00 
      Xen                  0x00000004   Unknown note type: (0x00000007)
       description data: 32 2e 36 00 
      Xen                  0x00000008   Unknown note type: (0x00000005)
       description data: 78 65 6e 2d 33 2e 30 00 
      Xen                  0x00000008   Unknown note type: (0x00000003)
       description data: 00 00 00 80 ff ff ff ff 
      Xen                  0x00000008   Unknown note type: (0x0000000f)
       description data: 00 00 00 00 80 00 00 00 
      Xen                  0x00000008   NT_VERSION (version)
       description data: c0 e1 9a 82 ff ff ff ff 
      Xen                  0x00000008   NT_ARCH (architecture)
       description data: 00 20 00 81 ff ff ff ff 
      Xen                  0x00000029   Unknown note type: (0x0000000a)
       description data: 21 77 72 69 74 61 62 6c 65 5f 70 61 67 65 5f 74 61 62 6c 65 73 7c 70 61 65 5f 70 67 64 69 72 5f 61 62 6f 76 65 5f 34 67 62 
      Xen                  0x00000004   Unknown note type: (0x00000011)
       description data: 01 88 00 00 
      Xen                  0x00000004   Unknown note type: (0x00000009)
       description data: 79 65 73 00 
      Xen                  0x00000008   Unknown note type: (0x00000008)
       description data: 67 65 6e 65 72 69 63 00 
      Xen                  0x00000010   Unknown note type: (0x0000000d)
       description data: 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 
      Xen                  0x00000004   Unknown note type: (0x0000000e)
       description data: 01 00 00 00 
      Xen                  0x00000004   Unknown note type: (0x00000010)
       description data: 01 00 00 00 
      Xen                  0x00000008   Unknown note type: (0x0000000c)
       description data: 00 00 00 00 00 80 ff ff 
      Xen                  0x00000008   Unknown note type: (0x00000004)
       description data: 00 00 00 00 00 00 00 00 
      GNU                  0x00000014   NT_GNU_BUILD_ID (unique build ID bitstring)
        Build ID: c0263a3075bc0a9388365ddf35ab5422da3356a9
      Linux                0x00000001   OPEN
       description data: 00 
      Xen                  0x00000008   Unknown note type: (0x00000012)
       description data: 40 07 00 01 00 00 00 00
    

    The important section for the question is the Program headers section:

    Program Headers:
      Type           Offset             VirtAddr           PhysAddr
                     FileSiz            MemSiz              Flags  Align
      LOAD           0x0000000000200000 0xffffffff81000000 0x0000000001000000
                     0x00000000015539b0 0x00000000015539b0  R E    0x200000
      LOAD           0x0000000001800000 0xffffffff82600000 0x0000000002600000
                     0x000000000037f000 0x000000000037f000  RW     0x200000
      LOAD           0x0000000001c00000 0x0000000000000000 0x000000000297f000
                     0x000000000002f000 0x000000000002f000  RW     0x200000
      LOAD           0x0000000001dae000 0xffffffff829ae000 0x00000000029ae000
                     0x0000000000e52000 0x0000000000e52000  RWE    0x200000
      NOTE           0x00000000017537c4 0xffffffff825537c4 0x00000000025537c4
                     0x00000000000001ec 0x00000000000001ec         0x4
    

    I would guess that the RW sections are the data/bss segments of the static and freestanding kernel image. There isn't much bss data (uninitialized data) in this kernel image. You can see that by looking at the filesize versus the memsize of the different segments. In bss data, only the size is written in the executable. The loader will then assign zeroes to these segments. Since we have only memsizes which are equal to the filesizes, then we don't have much bss here.

    Also, we can look at the different virtual addresses at which the kernel will load itself after decompression. This will be useful to find the segment in which a certain global variable resides.

    Now, to determine where a global variable is loaded one can look at the System.map file in the /boot directory. For example, we can take a random global variable in kernel/sched/core.c called sysctl_sched_rt_period and find its address using

    user@user-System-Product-Name:~$ sudo grep "sysctl_sched_rt_period"  /boot/System.map-$(uname -r)
    
    ffffffff8266eb24 D sysctl_sched_rt_period
    

    The address is thus ffffffff8266eb24. This address resides in the RW segment loaded at ffffffff82600000 because the size of this segment is 37f000 and because ffffffff82600000 + 37f000 > ffffffff8266eb24. You can clearly see that this global variable is stored in the data segment of the kernel's executable.