How to Debug an AI Output That Is Almost Right

The hardest AI output to fix is the one that is 80% right and 20% wrong. A 5-step debugging protocol for the almost-right output, with worked examples.

Why almost-right is harder than wrongAn AI output that is completely wrong is easy to deal with. You throw it away, re-write the prompt, and try again. The cost of the wrong output is one re-write, and the re-write is fast because the prompt was clearly bad.An AI output that is almost right is the hard case. The output is 80% usable. The other 20% is wrong, in a way that is specific to your use case. You cannot throw the output away, because most of it is good. You cannot use the output as-is, because the 20% is wrong. You are stuck in the middle, and the middle is where most of the time goes.The almost-right output happens because the prompt is mostly right and the model is mostly right, but one of the two is wrong in a way you did not specify. The fix is to find the specific wrong part, isolate it, and re-prompt for just that part. That is the debugging protocol.The 5-step protocolStep 1: identify the 20% that is wrongRead the output and mark the parts that are wrong. Not "could be better" or "feels off." Wrong. The word in the output is the wrong word. The fact in the output is the wrong fact. The format in the output is the wrong format. Be specific. "The third bullet point is factually wrong" is specific. "The output is not great" is not.If you cannot identify the 20%, the output is not almost-right, it is wrong. Treat it as a wrong output and re-write the prompt from scratch. The debugging protocol is for outputs where you can name the wrong part.Step 2: name the failure modeOnce you have identified the 20%, name the failure mode. The most common failure modes are:Wrong word. The model picked a synonym that does not match your meaning. ("Use case" instead of "scenario." "Lever" instead of "button.")Wrong fact. The model made something up. (A date, a number, a name, a citation that does not exist.)Wrong format. The model produced a list when you wanted prose, or prose when you wanted a list, or 800 words when you wanted 600.Wrong tone. The model is too formal, too casual, too friendly, too direct, or otherwise off from the tone you wanted.Wrong audience. The model wrote for a different reader than the one in the prompt. (Generic instead of "marketing managers at small companies.")Wrong constraint. The model violated one of the constraints in the prompt. (Used an em-dash you told it not to use, used a bullet list you told it not to use, used jargon you told it to avoid.)Naming the failure mode is the part most people skip, and the part that takes the longest to skip. The next step depends on the failure mode. A "wrong word" failure needs a different fix than a "wrong fact" failure, and the fix is not the same for any of them.Step 3: re-prompt for just the wrong partTake the 80% that is right and the 20% that is wrong, and re-prompt the model with both. The new prompt should be:

    Here is an output from a previous prompt. 80% of it is right. 20%
    of it is wrong. The wrong part is [SPECIFIC FAILURE + LOCATION,
    e.g. "the third bullet point, which claims the user can do X;
    in fact, the user cannot do X because Y"].

    Rewrite the wrong part to be correct. Keep the rest of the output
    unchanged. Do not rewrite the parts that are right.

    Output: [PASTE THE FULL ORIGINAL OUTPUT HERE]

That is the re-prompt. It is 4 lines. The lines do the work because they tell the model exactly what to fix and what to leave alone. The model gets a clear scope (just the wrong part) and a clear constraint (do not change the rest). The output of the re-prompt is usually correct on the first try, because the prompt is now precise enough to make the model succeed.Step 4: verify the unchanged parts stayed unchangedThis is the part most people skip, and the part that catches the most silent regressions. When you ask the model to fix part of an output, the model may "improve" the parts you told it to leave alone. The improvement is sometimes an improvement, but it is sometimes a regression that you do not notice until the final draft is in front of you.The fix is to diff the new output against the original. If the diff is limited to the part you asked the model to fix, the re-prompt worked. If the diff includes other parts, the re-prompt over-reached, and you need to either re-prompt with stronger "leave the rest alone" instructions, or manually copy back the unchanged parts from the original.Step 5: keep the 20% in a "common failures" listOnce you have debugged an almost-right output, the failure mode you identified is data. It is a thing the model does, in response to the prompts you give it, that you did not intend. The next time you write a prompt, you can preempt this specific failure by adding a constraint that addresses it.For example, if you debugged a "wrong word" failure where the model kept using "leverage" as a verb, the next prompt you write should include "do not use 'leverage' as a verb" in the constraints. The constraint is the thing that prevents the failure from happening on the first try, instead of the second or third.The list of common failures is a small file in your prompt library. It is the audit trail of every almost-right output you have debugged. It is also the most useful artifact in the prompt library, because it converts debugging-time (expensive) into prompt-time (cheap).A worked exampleHere is a real one. The prompt: "Write a 600-word blog post about why my prompts kept failing at work. Audience: marketing managers. Format: three sections, each with a copy-paste example, no bullet lists."The output: 580 words, three sections, mostly on-target, with a bullet list in section 2 that the prompt explicitly forbade.Step 1: identify the 20% wrong. Section 2 has a 4-item bullet list. The prompt said no bullet lists.Step 2: name the failure mode. Wrong constraint. The model ignored the "no bullet lists" constraint.Step 3: re-prompt for just the wrong part. "Here is the output. The 4-item bullet list in section 2 is wrong. The prompt said no bullet lists. Rewrite section 2 to keep the same content but use prose instead of a bullet list. Keep sections 1 and 3 unchanged."Step 4: verify. Diff the new output against the original. Sections 1 and 3 are byte-for-byte identical. Section 2 is now prose, with the same 4 points but no bullet markers. The re-prompt worked.Step 5: update the common failures list. Add: "The model sometimes uses bullet lists even when forbidden, especially when the content has clear discrete points. Counter: explicitly say 'use prose, even for discrete points.' Or: pre-format the section as a numbered list, then ask the model to convert to prose."That is the worked example. The debugging took 4 minutes. The next time the same prompt would have produced a bullet list, the constraint is in the prompt, and the debugging does not have to happen again.When to skip the protocolSkip the protocol if the output is wrong in more than 30% of the parts. The protocol is for almost-right outputs. If the output is more wrong than that, the prompt is the bug, not the output. Re-write the prompt from scratch and try again. Debugging an output that is 50% wrong is faster by re-prompting than by repairing.Skip the protocol if you cannot name the failure mode. The protocol requires a specific failure mode (wrong word, wrong fact, wrong format, wrong tone, wrong audience, wrong constraint). If the failure is "feels off" or "not quite right," the protocol will not help, because there is no specific thing to fix. Re-prompt with a stronger voice anchor or a more specific audience, and the new output will be more on-target.The honest partThe 5-step protocol is the thing I wish someone had told me about 12 months ago. Before I had the protocol, I was treating almost-right outputs as failures and re-writing the prompt from scratch every time. Each re-write took 10-15 minutes and produced an output that was 60-70% right, with a different 30-40% wrong. The pattern repeated. I never got to an output I was happy with, because each re-write introduced new failures.The protocol changed the pattern. Now I re-prompt for the specific failure, keep the parts that are right, and update the common-failures list so the same failure does not happen again. The outputs are still almost-right sometimes. The time to fix them is 4 minutes instead of 15. The next prompt is better because of the last failure. The protocol compounds.If you are spending more time on AI outputs than you expected, the protocol is probably the missing piece. The model is not the bottleneck. The debugging is the bottleneck. The protocol is the fix.Want the audit-trail feature baked into the prompt pack?The Blog Writing Factory includes a "common failures" prompt that captures the patterns from each run, so the next run starts with the failures of the last run already in the constraints. The protocol is the manual version. The pack is the system that does it automatically.Buy on Gumroad. $14

Send me the rough edges if you try it. I read every message.