● Stage 05 • Distributed Systems

Distributed Consensus Engine (Raft)

Implement a 3-node fault-tolerant replicated state machine. Handle randomized election timeouts, leader transitions, heartbeat broadcasts, log replication, and partition split-brain recovery.

First steps

Start three local processes. Send a message between each pair.
Implement elections and heartbeats. Use Figure 2 of the Raft paper for the state rules.
Stop the leader. Check that a replacement is elected before adding client writes.

Project Goal

Understand distributed consensus algorithms and linearizable state machine replication.
Implement randomized election timeouts to ensure single-leader election without split votes.
Replicate log entries across nodes and commit writes only when acknowledged by a majority quorum ($N/2 + 1$).
Handle node crashes, network drops, and split-brain scenarios safely without data divergence.

Skills Acquired

State Machine Replication Leader Election Protocols Quorum Arithmetic ($N/2 + 1$) Raft RPC Wire Framing Log Invariant Matching Split-Brain Partition Recovery

Technical Specification

Cluster Setup: 3 node processes listening on ports 9001, 9002, and 9003.
Election Timeout: Uniform random timeout between 150ms and 300ms.
Heartbeat Frequency: Leader broadcasts empty AppendEntries every 50ms.
Commit Rule: Entry committed only after write acknowledgment from $\ge 2$ nodes.

Milestones

05.1

Node Network Mesh & State Scaffolding

Implement state variables (currentTerm, votedFor, log[], commitIndex). Bind listening ports for 3 nodes and configure RPC transport.

05.2

Leader Election with Quorum Voting

When the election timer elapses without a heartbeat, transition to Candidate and request votes. Become Leader upon receiving $\ge 2$ positive votes.

05.3

Log Replication & State Machine Commit

Accept client write commands on the leader. Broadcast AppendEntries to peers. When a quorum acknowledges, advance commitIndex and reply to client.

05.4

Network Partition & Split-Brain Simulation

Simulate network partition isolating Node 1. Verify Nodes 2 and 3 elect a new leader and accept writes. Reconnect Node 1 and verify it discovers the higher term and steps down.

Verification Commands

Build raft_node first. Elections are random: check your logs for the actual leader before sending writes. The example below assumes Node 1 leads initially, then Node 2; adjust the port and captured PID if a different node wins. A follower read must wait until it has applied the committed entry.

# 1. Start 3-node cluster and capture each process ID
./raft_node --id=1 --port=9001 --peers=9002,9003 &
NODE1_PID=$!
./raft_node --id=2 --port=9002 --peers=9001,9003 &
NODE2_PID=$!
./raft_node --id=3 --port=9003 --peers=9001,9002 &
NODE3_PID=$!

# 2. Issue write to leader
curl -X POST http://localhost:9001/set -d '{"key":"k1","val":"v1"}'

# 3. Simulate failure: kill current leader
kill -9 "$NODE1_PID"

# 4. Verify surviving majority elects new leader and accepts writes
sleep 1
curl -X POST http://localhost:9002/set -d '{"key":"k2","val":"v2"}'
curl -X GET  'http://localhost:9003/get?key=k2' # Returns "v2"

# 5. Stop the surviving test nodes when finished
kill "$NODE2_PID" "$NODE3_PID"

Acceptance Criteria

The cluster must maintain consistency during the crash of any single node. When a network partition divides the cluster, the majority partition must continue processing writes while the minority partition rejects writes. Upon network reconnection, all nodes must converge to the authoritative log.

← Stage 04 Return to Track Index →