Search code examples
concurrencyiolinux-kernelnfs

Do kernel NFS module have concurrent limit?

Background: I was testing a nfs-server with fio. And I find that no matter how much "iodepth" is set to fio. The nfs-server can only have "64 Inflight". So I just suspect that somewhere around "nfs protocol" limits the max concurrent(max io in flight).

fio command is 

fio -numjobs=1 -iodepth=128 -direct=1 -ioengine=libaio -sync=1 -rw=write -bs=4k -size=500M -time_based -runtime=90 -name=Fiow -directory=/75

My nfs-server is based on ganesha, and got conclusion "64 Inflight" by using ganesha_stats.py.

So I have two options for now:

  1. Study the calling-graph and read code to find the problem

    1. I download linux kernel code, but struggle to . Which function/source file should I begin , maybe vfs.c:nfsd_write?
    2. Tried to use 'perf' to trace calling-graph to speed up my code reading tour for linux kernel, but failed. Because 'perf report' shows shared library symbol without function name.

  2. Learn the nfs protocol/mount cmd to seek the limit.

Can somebody can help me with this? :)

Solution

  • Assuming you're using NFSv4.1 (RFC 5661):

    In NFSv4.1, the number of outstanding requests is bounded by the size of the slot table [...].

    And in Linux:

    #define NFS4_DEF_SLOT_TABLE_SIZE (64U)

    It is the default for this module param:

    module_param(max_session_slots, ushort, 0644);
MODULE_PARM_DESC(max_session_slots, "Maximum number of outstanding NFSv4.1 "
        "requests the client will negotiate");

    IIUC the total maximum for this is:

    #define NFS4_MAX_SLOT_TABLE (1024U)