cmd/go: add support for dealing with flaky tests
## Background First off, flaky tests (a test that usually passes but sometimes fails) are the worst and you should not write them and fix them immediately. That said, the larger teams & projects & tests get, the more likely they get introduced. And sometimes it's technically and/or politically hard to get them fixed. (e.g. the person who wrote it originally left the company or works on a new team) You're then left deciding whether to skip/delete the entire test (which might be otherwise super useful), or teach your team to become immune to and start ignoring test failures, which is the worst, when teams start submitting when CI's red, becoming blind to real test failures. Google doesn't have this problem internally because Google's build system supports detecting & annotating flaky tests: https://bazel.build/reference/be/common-definitions Tailscale has its [own test wrapper](https://github.com/tailscale/tailscale/blob/main/cmd/testwrapper/testwrapper.go) that retries tests [annotated with `flakytest.Mark`](https://github.com/tailscale/tailscale/blob/main/cmd/testwrapper/flakytest/flakytest.go) as flaky. We can't use `go test -exec=...` unfortunately, as that prevents caching (https://github.com/golang/go/issues/27207). So instead we need a separate tool wraps `cmd/go` (kinda awkwardly, as it turns out). Go internally also uses these flaky test markers 79 times: * https://pkg.go.dev/internal/testenv#SkipFlaky * https://pkg.go.dev/internal/testenv#SkipFlakyNet So, clear
Posted by bradfitz on Aug 23, 2023, 4:12 PM
Tags
Related issues
- This is the biggest problem with contracts, there's simply no way to actually say "please always execute this check".
- I don't disagree with you on this, I worked my whole life in corporate and haven't worked as a OSS maintainer before, though I will _and ...
- the new VSCode UI, looks cute, i don't like it.
- I've been working on my own private harness for the past 8 months, and I've been collecting ideas from such repos I've stumbled upon.
- The way AI is able to interact with outside resources is pretty impressive, but the quality of code it produces to me is still questionab...
- Add PR-style review comments in Git Diff UI that feed into the active agent thread
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
ux (Software)
This is the biggest problem with contracts, there's simply no way to actually say "please always execute this check".
I don't disagree with you on this, I worked my whole life in corporate and haven't worked as a OSS maintainer before, though I will _and ...
the new VSCode UI, looks cute, i don't like it.
I've been working on my own private harness for the past 8 months, and I've been collecting ideas from such repos I've stumbled upon.
The way AI is able to interact with outside resources is pretty impressive, but the quality of code it produces to me is still questionab...
Add PR-style review comments in Git Diff UI that feed into the active agent thread
Thanks for indulging me so far, by the way, I really appreciate this discussion, it's very stimulating.
Migrate GDScript to the GDExtension mechanism and overhaul the implementation
cmd/go: add support for dealing with flaky tests
## Background First off, flaky tests (a test that usually passes but sometimes fails) are the worst and you should not write them and fix them immediately. That said, the larger teams & projects & tests get, the more likely they get introduced. And sometimes it's technically and/or politically hard to get them fixed. (e.g. the person who wrote it originally left the company or works on a new team) You're then left deciding whether to skip/delete the entire test (which might be otherwise super useful), or teach your team to become immune to and start ignoring test failures, which is the worst, when teams start submitting when CI's red, becoming blind to real test failures. Google doesn't have this problem internally because Google's build system supports detecting & annotating flaky tests: https://bazel.build/reference/be/common-definitions Tailscale has its [own test wrapper](https://github.com/tailscale/tailscale/blob/main/cmd/testwrapper/testwrapper.go) that retries tests [annotated with `flakytest.Mark`](https://github.com/tailscale/tailscale/blob/main/cmd/testwrapper/flakytest/flakytest.go) as flaky. We can't use `go test -exec=...` unfortunately, as that prevents caching (https://github.com/golang/go/issues/27207). So instead we need a separate tool wraps `cmd/go` (kinda awkwardly, as it turns out). Go internally also uses these flaky test markers 79 times: * https://pkg.go.dev/internal/testenv#SkipFlaky * https://pkg.go.dev/internal/testenv#SkipFlakyNet So, clear
Comments (0)
No comments yet. Start the discussion.