[Hero Run] 535B-A23B on 18T tokens
Issue is to describe the Hero Run scaling ladder, config, and track its performance. Tracking: https://wandb.ai/marin-community/marin_moe/reports/535B-A23B-18T-Token-Hero-Run-Scaling-Ladder--VmlldzoxNzc2MDM5Ng Details on the full model spec is in the comment below. ### Why run a scaling ladder? 1. This lets us project the model performance and compare it to the scaling projections of our prior recipes. If our projection looks meaningfully worse, then we know we need to go back to the drawing board on some aspect of the model or data. This step can catch a lot of bugs! 2. We can look at how training dynamics change as we scale, in particular the gradient norms and token dropping. In our last run, the scaling ladder caught gradient norm growth to over 4 as we scaled out token horizon, which caused us to find a fix with logit z loss. Later ablations showed that without this fix, runs under certain conditions like high batch size would completely blow up mid-run. 3. We can project our evals across the entire trajectory of the run. This means that across the entire ~100 day run, we know exactly how we are doing compared to the projection. If at any point we fail to match the projection, it can prompt early investigation into training dynamics. 4. If something strange start to happen in the Hero run, we can reference the smaller scales runs to see if its actually normal. For instance, in prior scaling ladders we saw the grad norm grow heavily up to 40% of the run, then decrease
Posted by ClassicLarry on Aug 18, 2026, 11:48 PM
Tags
Related issues
- The issue is really the extreme difficulty of programming them, especially for anything sequential (rather than combinatorial).
- I probably could have beared with it if it was only limited to programming, but it showing up in creative fields has made me feel all sor...
- I hate that I'm in deep enough to know what you mean.
- The biggest problem with “AI” is that big corps are shoving it down our throats.
- in the middel of a coding session with 5.6 sol.
- I hate that this is how AI is being used..
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
reliability (Other)
The issue is really the extreme difficulty of programming them, especially for anything sequential (rather than combinatorial).
I probably could have beared with it if it was only limited to programming, but it showing up in creative fields has made me feel all sor...
I hate that I'm in deep enough to know what you mean.
The biggest problem with “AI” is that big corps are shoving it down our throats.
in the middel of a coding session with 5.6 sol.
[Hero Run] 535B-A23B on 18T tokens
I hate that this is how AI is being used..
inexperienced protobuf user here -- why do non-googlers generally use protobuf?
[Hero Run] 535B-A23B on 18T tokens
Issue is to describe the Hero Run scaling ladder, config, and track its performance. Tracking: https://wandb.ai/marin-community/marin_moe/reports/535B-A23B-18T-Token-Hero-Run-Scaling-Ladder--VmlldzoxNzc2MDM5Ng Details on the full model spec is in the comment below. ### Why run a scaling ladder? 1. This lets us project the model performance and compare it to the scaling projections of our prior recipes. If our projection looks meaningfully worse, then we know we need to go back to the drawing board on some aspect of the model or data. This step can catch a lot of bugs! 2. We can look at how training dynamics change as we scale, in particular the gradient norms and token dropping. In our last run, the scaling ladder caught gradient norm growth to over 4 as we scaled out token horizon, which caused us to find a fix with logit z loss. Later ablations showed that without this fix, runs under certain conditions like high batch size would completely blow up mid-run. 3. We can project our evals across the entire trajectory of the run. This means that across the entire ~100 day run, we know exactly how we are doing compared to the projection. If at any point we fail to match the projection, it can prompt early investigation into training dynamics. 4. If something strange start to happen in the Hero run, we can reference the smaller scales runs to see if its actually normal. For instance, in prior scaling ladders we saw the grad norm grow heavily up to 40% of the run, then decrease
Comments (0)
No comments yet. Start the discussion.