"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson
In brief
Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.