Guide

Decision forks: helping AI agents choose the right direction

By Atlas AI Resources · Published October 5, 2026 · How we make these guides

On a long job, an agent makes a handful of choices that decide how the whole run goes: which fix to build on, which lead to follow, which approach to try first. This guide explains what new research found about those choices and gives you simple habits that help your bots pick better directions, or hand the hard picks to you.

1. What a decision fork is

A decision fork is a point in a task where two or more reasonable next steps are open and one of them will turn out better. The wrong choice usually looks fine at the moment. You only find out later, after the agent has spent much of its time and budget going the wrong way.

A September 2026 paper, “The Tasteful Agent,” calls the skill of choosing well at these points an agent’s taste, and builds a benchmark for it called Taste-Bench. The authors mined forks from real agent runs on software engineering tasks and on AI research tasks, then asked models to pick the better next step without seeing what happened after the fork. Of 4,657 forks they mined, 502 survived filtering into the released benchmark.

2. What the researchers found

These are the paper’s own measurements on its setup, not a guarantee for your bots, but the pattern is useful.

• Picking the right direction is hard. A question only counts as correct if the model gets it right with the options in both orders, so random guessing scores 25%. The best model the authors tested scored 59.7%.
• Late evidence is the problem. When the clue that settles the choice is already visible before the fork, the average across 14 models was 62.3%. When the answer only shows up after more work, the average fell to 21.0%, about the same as guessing.
• Thinking longer didn’t help. Raising the reasoning setting on two models moved their scores by −0.2 and +2.2 points. The models spent the most reasoning on the hardest forks and still got them wrong, which suggests the missing piece was evidence, not effort.
• Good advice helps a lot. On 41 held-out coding tasks, an agent given a short note at each fork (the situation, the option to avoid, the option to take) went from 14.6% success with no notes to 39.0% with correct notes. Notes from a smaller model trained on past outcomes reached 33.7%.
• Choosing well is its own skill. Scores on this benchmark were only partly in line with a well-known end-to-end coding benchmark (correlation 0.63), so a model that finishes tasks well isn’t automatically good at picking directions.

3. Habits that help your bots at forks

You don’t need the benchmark to use the lesson. These habits work with any agent, including a Grok Bot.

• Ask the bot to name the fork. In your instructions for long jobs, ask it to stop at a real choice and write down the options and what it would need to know to pick. That turns a silent guess into something you or it can check.
• Buy evidence, not more thinking. When the answer depends on what happens later, have the bot run a small, cheap check first: a quick test, one sample draft, a single test run of a routine. A retry is only worth it when it brings new information.
• Let stakes and certainty set the path. Low-stakes, clear-cut choices can go ahead. Unclear ones should gather one more piece of evidence. High-stakes or still-unclear ones come to you. A bot sounding sure of itself is not the same as your approval.
• Turn lessons into fork notes. When a run goes wrong at a fork, write the lesson in the same shape the paper used (situation, avoid, take) and have the bot add it to a skill. One of SpaceXSI’s own Grok Bot guides describes the same habit: keep a skill you edit over time, and have the bot add to it every time you steer it.
• Judge the finished task. Count how often the whole job ends well and what that costs, rather than how many steps or calls a run took.
• Test on your own work. If you’re choosing a model or setup for long jobs, a leaderboard is a starting point at most. Try a few real tasks of your own and compare the results.

4. Trying the benchmark yourself

The code is open (MIT license) and the dataset is on Hugging Face. Access to the dataset is gated to reduce the chance of it leaking into model training, so you request access first. The project README says a full run is 1,004 requests and about 8 million input tokens, so try a small limited run before a full one. It’s a research tool for comparing models, not something your bots need day to day.

Useful public resources

Related: Skills before routines · Parallel agent campaigns · Agent memory for AI fleets · Grok Bot for engineering