Docker stack fails to allocate IP on an overlay network, and gets stuck in `NEW` current state
**Description** I do have a docker swarm cluster with 5 managers and 4 working nodes. A while ago, it was 3 managers and 4 working nodes. We use immutable infrastructure for the hosts. EDIT: I managed to get reproducible tests in https://github.com/moby/moby/issues/37338#issuecomment-437558916 Majority of our containers connect to a specific overlay network, with CIDR /24. my_network: driver: overlay driver_opts: encrypted: "" ipam: driver: default config: - subnet: 10.100.2.0/24 Docker stack deployments happen all the time in this specific cluster. Occasionally (and I cannot understand what causes it), the swarm is unable to allocate an IP to that task. "Failed allocation for service <service>" error="could not find an available IP while allocating VIP" So I assumed we ran out of IPs for the CIDR. But when I count the number of tasks currently attached to the network, there was less than 40 running tasks. I also went and counted all the stopped/historical tasks, and counted all the IPs on that network; still, there were less than 120 IPs. A lot less than the 200-and-something I'd expect. I tried to restrict the task history size, but that by itself didn't make any difference. I deleted almost all stacks, and some containers were able to get a new IP. But the problem manifested itself as far as all things got redeployed. I actually looked for the NetworkDB stats when the problem was happening, and it was all lines like: Net
Posted by cintiadr on Jun 25, 2018, 1:10 AM
Tags
Related issues
- [UPDATE NOW] PHP RCE Vulnerability (CVE-2026-109618 and CVE-2026-109619)
- Server requests and latency increased after upgrading from Next.js 15 to 16
- Leak of Geedge Networks internal documents (100,000+ from Jira, Confluence, GitLab)
- Can't download videos from YouTube on cobalt.tools
- `google_service_networking_connection` produce an error when trying to destroy the resource
- WebGPU-based renderer for the editor
Raise real-world problems, find people facing the same issue, and discuss solutions.4,000 issues0 comments
github (Network)
[UPDATE NOW] PHP RCE Vulnerability (CVE-2026-109618 and CVE-2026-109619)
Server requests and latency increased after upgrading from Next.js 15 to 16
Leak of Geedge Networks internal documents (100,000+ from Jira, Confluence, GitLab)
Can't download videos from YouTube on cobalt.tools
`google_service_networking_connection` produce an error when trying to destroy the resource
WebGPU-based renderer for the editor
Zed downloads NodeJS binary and npm packages from Internet without user’s consent
FR: Allow custom domains for Tailscale Funnel
Docker stack fails to allocate IP on an overlay network, and gets stuck in `NEW` current state
**Description** I do have a docker swarm cluster with 5 managers and 4 working nodes. A while ago, it was 3 managers and 4 working nodes. We use immutable infrastructure for the hosts. EDIT: I managed to get reproducible tests in https://github.com/moby/moby/issues/37338#issuecomment-437558916 Majority of our containers connect to a specific overlay network, with CIDR /24. my_network: driver: overlay driver_opts: encrypted: "" ipam: driver: default config: - subnet: 10.100.2.0/24 Docker stack deployments happen all the time in this specific cluster. Occasionally (and I cannot understand what causes it), the swarm is unable to allocate an IP to that task. "Failed allocation for service <service>" error="could not find an available IP while allocating VIP" So I assumed we ran out of IPs for the CIDR. But when I count the number of tasks currently attached to the network, there was less than 40 running tasks. I also went and counted all the stopped/historical tasks, and counted all the IPs on that network; still, there were less than 120 IPs. A lot less than the 200-and-something I'd expect. I tried to restrict the task history size, but that by itself didn't make any difference. I deleted almost all stacks, and some containers were able to get a new IP. But the problem manifested itself as far as all things got redeployed. I actually looked for the NetworkDB stats when the problem was happening, and it was all lines like: Net
Comments (0)
No comments yet. Start the discussion.