Search code examples
distributed-systemraft

Out-of-order AppendEntries in Raft


The Raft paper states that:

AppendEntries:

If an existing entry conflicts with a new one (same index but different terms), delete the existing entry and all that follow it (§5.3)

From the perspective of the leader, it sends an AppendEntries request with entries [1, 2] to a follower and later on sends another AppendEntries request with entries [1, 2, 3].

From the follower's perspective, it initially receives the second AppendEntries request with entries [1, 2, 3], and subsequently receives the first AppendEntries request with entries [1, 2]. According to the paper, upon receiving the AppendEntries with entries [1, 2], the follower must delete the log entry [3] (assuming that [1, 2, 3] were successfully appended).

The question arises as to how to handle AppendEntries that arrive out of order. It seems the paper does not provide a solution. How do industrial implementations of Raft handle this issue?


Solution

  • Out of order messages from the same leader do not affect follower's log. The follower would simply acknowledge AppendEntriesRPC call.

    Details:

    Entires in raft carry both index and term (and an actual value). Let's replace your log of [1,2,3] with [[1,1,a][2,1,b][3,1,c]] to avoid confusion on what is the term and what is the index. In these tuples the first item is the index, the second item is the term and the third one is the value of the log entry.

    You original question would be:

    • a follower's log is [[1,1,a][2,1,b][3,1,c]]
    • a late message from a leader comes with [[1,1,a][2,1,b]]

    In this case, the follower checks entries in the message one by one. Since both [1,1,a] and [2,1,b] are already in the follower's log, the follower does nothing and just acknowledges the AppendEntriesRPC.

    A conflict would happen if a leader sends a message with entries like [[1,1,a][2,2,d]]. After receiving the request, the follower would:

    • match the first entry to the existing entry in the follower's log
    • detect a conflicting entry at index 2
    • according to raft protocol, the follower would truncate its log at the conflicting entry
    • the follower's log would become [[1,1,a]] - the conflicting entry and all next entries are removed
    • the follower would accept the [2,2,d] and the follower's log would be [[1,1,a][2,2,d]]

    References:

    • https://raft.github.io/raft.pdf page 4
    • rule 3: a conflict happens if a new term arises and not-yet-committed entries may be in conflict and those to be removed
    • rule 4: only missing entries are added, if entries already in the log (same index and term), no extra steps are required