Ironically, one of the most challenging aspects of distributed systems is configuration management. Consensus, fault tolerance, leader election, and other concepts are complex but relatively straightforward. Configuration management is challenging because it’s about the convergence of the internal system state, a declarative API, and tooling that glues together that API with other adjacent systems (CI/CD, developer tools, DevOps, etc.). There’s no algorithm like Raft or Paxos to guide the implementation. And so many different concerns end up with an API that requires the knowledge of multiple roles (operators and developers).