FDB Talk: Self healing from Gray Failures

In July 2025, I gave a presentation at Adobe HQ in San Jose on detecting and self healing from gray failures in FoundationDB. The talk is embedded below:

Talk summary: Gray failures in FoundationDB are subtle performance degradations that, while not full crashes, impact system health and burden operators with manual intervention. This talk presents a simple yet effective self-healing mechanism designed to address such issues. We’ll demonstrate how we reused FoundationDB’s core architectural building blocks—such as unbundled roles, central Cluster Controller, transport layer latency metrics, and fast recovery—to implement a “complaining” algorithm where processes report slow peers, enabling automated detection and remediation, thereby significantly improving FoundationDB’s resilience and reducing operational load.