Consider expanding the cases where Vector retries requests
## Community Note * Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request * If you are interested in working on this issue or have submitted a pull request, please leave a comment ## Current behavior Vector's retry behavior varies by sink, but typically we retry requests that are viewed to be temporary failures that we expect to recover. This includes things like HTTP 503s and 429s. This does not include failures that are viewed as non-temporary like HTTP 403s and 404s. Events in these requests are dropped and Vector continues processing. ## Possible issue The above means that Vector drops events under circumstances like: * Misconfiguration in Vector like using an invalid Datadog API key * Misconfiguration in upstreams such as a deploy of an internal Elasticsearch instance causing it to start returning 403s where previously the credentials were valid or provider outages like when a [GCP outage caused customer load balancers to start returning 404s](https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh) Under these circumstances, I think it is reasonable for users to not want Vector to drop events, but just retry and block processing until a human can intervene and fix the issue. ## Idea One idea would be to retry _all_ failures up until a maximum number of retries after which failed batches
Posted by jszwedko on Jan 14, 2022, 8:57 PM
Tags
Related issues
- ⚠️ PHP CVEs. Update required!
- I think RTS could continue to work if someone made a massive leap of faith and put all the modern infra around it that a AAA extraction s...
- RFC: Pluggable SessionDB Provider — PostgreSQL, MySQL, and Beyond
- v2.1.113+ broken on Termux/Android: native binary requires glibc, no JS fallback
- I’m firmly anti-Tahoe and haven’t updated, and I’ve started to find Apple despicable.
- [FEATURE] Link a source control repo as the source for organization skills
Raise real-world problems, find people facing the same issue, and discuss solutions.3,000 issues0 comments
reliability (Sysadmin)
⚠️ PHP CVEs. Update required!
I think RTS could continue to work if someone made a massive leap of faith and put all the modern infra around it that a AAA extraction s...
RFC: Pluggable SessionDB Provider — PostgreSQL, MySQL, and Beyond
v2.1.113+ broken on Termux/Android: native binary requires glibc, no JS fallback
I’m firmly anti-Tahoe and haven’t updated, and I’ve started to find Apple despicable.
[FEATURE] Link a source control repo as the source for organization skills
Feature Request: Option to disable paste text collapse in input field
We went down the WASM sandboxing rabbit hole at Gobii when building our agent infra.
Consider expanding the cases where Vector retries requests
## Community Note * Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request * If you are interested in working on this issue or have submitted a pull request, please leave a comment ## Current behavior Vector's retry behavior varies by sink, but typically we retry requests that are viewed to be temporary failures that we expect to recover. This includes things like HTTP 503s and 429s. This does not include failures that are viewed as non-temporary like HTTP 403s and 404s. Events in these requests are dropped and Vector continues processing. ## Possible issue The above means that Vector drops events under circumstances like: * Misconfiguration in Vector like using an invalid Datadog API key * Misconfiguration in upstreams such as a deploy of an internal Elasticsearch instance causing it to start returning 403s where previously the credentials were valid or provider outages like when a [GCP outage caused customer load balancers to start returning 404s](https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh) Under these circumstances, I think it is reasonable for users to not want Vector to drop events, but just retry and block processing until a human can intervene and fix the issue. ## Idea One idea would be to retry _all_ failures up until a maximum number of retries after which failed batches
Comments (0)
No comments yet. Start the discussion.