There are some problems reasoning about it.
There are some problems reasoning about it. In the thinking output, it isn't really discrete. Therefore I can't apply series-system reliability formulas. But those are a very good metaphor. I can see in the thinking output [1 + 1 = 2], [10 + 1 = 11] , [20 + 1 = 21], ... [N]. So the metaphor is what is the probability of each being correct is the probability of the agent solving the equation. If each bracket is right 95% of the time, a 10-step chain finishes correctly 0.95^10 ≈ 60% of the time. So I started a climb at 5 digits X 5 digits, 6 digits X 6 digits, ..... N digits X N digits. There is clean decrease in reliability that the agent will get the correct answer with a cliff where it will always fail. Model Last pass First fail ------ --------- ---------- Haiku 10 digits 12 digits Sonnet 30 digits 33 digits Opus 50 digits 52 digits I have an agent that reverse engineers any website creating an API that is optimized to use the least amount of resources to interact with the website. The agent also writes its self every iteration -- a recursive agent. The agent will update its own instructions and will run an evaluation (I hate that they stole the word harness because it is a test harness) against 5 different extremely difficult to reverse engineer websites like Ticketmaster, Youtube, Twitch, ect.. Each evaluation whether it passes finding all the endpoints including streaming, graphql, and websockets, the number of tokens is tracked and the amount of time. There is nothing deterministic about it but it IS PROBABILISTIC meaning with the same prompt the chance of passing and with how many tokens is a distribution. I trying very hard to build a mental model of how this all works.
Posted by dataviz1000 on Apr 18, 2026, 9:37 PM
Tags
Related issues
- The issue is really the extreme difficulty of programming them, especially for anything sequential (rather than combinatorial).
- I probably could have beared with it if it was only limited to programming, but it showing up in creative fields has made me feel all sor...
- I hate that I'm in deep enough to know what you mean.
- The biggest problem with “AI” is that big corps are shoving it down our throats.
- in the middel of a coding session with 5.6 sol.
- [Hero Run] 535B-A23B on 18T tokens
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
reliability (Other)
The issue is really the extreme difficulty of programming them, especially for anything sequential (rather than combinatorial).
I probably could have beared with it if it was only limited to programming, but it showing up in creative fields has made me feel all sor...
I hate that I'm in deep enough to know what you mean.
The biggest problem with “AI” is that big corps are shoving it down our throats.
in the middel of a coding session with 5.6 sol.
[Hero Run] 535B-A23B on 18T tokens
I hate that this is how AI is being used..
inexperienced protobuf user here -- why do non-googlers generally use protobuf?
There are some problems reasoning about it.
There are some problems reasoning about it. In the thinking output, it isn't really discrete. Therefore I can't apply series-system reliability formulas. But those are a very good metaphor. I can see in the thinking output [1 + 1 = 2], [10 + 1 = 11] , [20 + 1 = 21], ... [N]. So the metaphor is what is the probability of each being correct is the probability of the agent solving the equation. If each bracket is right 95% of the time, a 10-step chain finishes correctly 0.95^10 ≈ 60% of the time. So I started a climb at 5 digits X 5 digits, 6 digits X 6 digits, ..... N digits X N digits. There is clean decrease in reliability that the agent will get the correct answer with a cliff where it will always fail. Model Last pass First fail ------ --------- ---------- Haiku 10 digits 12 digits Sonnet 30 digits 33 digits Opus 50 digits 52 digits I have an agent that reverse engineers any website creating an API that is optimized to use the least amount of resources to interact with the website. The agent also writes its self every iteration -- a recursive agent. The agent will update its own instructions and will run an evaluation (I hate that they stole the word harness because it is a test harness) against 5 different extremely difficult to reverse engineer websites like Ticketmaster, Youtube, Twitch, ect.. Each evaluation whether it passes finding all the endpoints including streaming, graphql, and websockets, the number of tokens is tracked and the amount of time. There is nothing deterministic about it but it IS PROBABILISTIC meaning with the same prompt the chance of passing and with how many tokens is a distribution. I trying very hard to build a mental model of how this all works.
Comments (0)
No comments yet. Start the discussion.