IssueRaiser

Benchmarks are too expensive for ordinary users to run, but it would be useful if they could publish their benchmarks using prod over tim...

Benchmarks are too expensive for ordinary users to run, but it would be useful if they could publish their benchmarks using prod over time, that would expose degradations in a more objective manner. Of course there’s always the problem of teaching to the test and out of test degradations, but presumably bugs would be independent of that.

Posted by data-ottawa on Sep 9, 2025, 4:47 PM

Tags

Related issues

Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments

pricing (EdTech)

Benchmarks are too expensive for ordinary users to run, but it would be useful if they could publish their benchmarks using prod over tim...

data-ottawaSep 9, 2025, 4:47 PM

Benchmarks are too expensive for ordinary users to run, but it would be useful if they could publish their benchmarks using prod over time, that would expose degradations in a more objective manner. Of course there’s always the problem of teaching to the test and out of test degradations, but presumably bugs would be independent of that.

Comments (0)

No comments yet. Start the discussion.

Comment on this issue