Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
We’re hitting a classic distributed systems wall and I’m looking for war stories or "least worst" practices. The Context: We maintain a distributed stateful engine (think search/analytics). The architecture is standard: a Control Plane (Coordinator) assigns data segments to Worker Nodes. The workload involves heavy use of mmap and lazy loading for large datasets. The Incident: We had a cascading failure where the Coordinator got stuck in a loop, DDOS-ing a specific node. The Signal: Coordinator sees Node A has significantly fewer rows (logical count) than the cluster average. It flags Node A as "underutilized." The Action: Coordinator attempts to rebalance/load new segments onto Node A. The Reality: Node A is actually sitting at 197GB RAM usage (near OOM). The data on it happens to be extremely wide (fat rows, huge blobs), so its logical row count is low, bu
Why it failed
We’re hitting a classic distributed systems wall and I’m looking for war stories or "least worst" practices. The Context: We maintain a distributed stateful engine (think search/analytics). The architecture is standard: a Control Plane (Coordinator) assigns data se
- Kind
- startup
- Status
- Shut down
- Inventor
- leo_e
- Ended
- 2025
- Source
- hackernews