Judgment Points

Notes on applied AI, written while learning it

Most workflows need one or two model calls. The rest should be code and a person who signs off.

I'm learning the forward-deployed engineering craft in public. This site is my notebook: how to audit a real workflow, decide which steps deserve a model, and prove the result is safe to ship.

One workflow, allocated step by step Illustrative: meeting transcript to approved follow-up tasks
Model

Extract action items

Why it sits here
Pulling commitments out of spoken language is the fuzzy step that earns a model call.
When it goes wrong
Item with no clear owner: flag it, never assign it.
7 steps are plain code, 2 are model calls and 1 is a person approving. If a map shows a model at every step, the map is wrong. Select any step to see why it sits where it does.

Why I work this way

The models are the same for everyone. The context is not.

Everyone can buy the same models.

Frontier intelligence is sold on the same terms to every company. Prompting it a little better than the next team is not a lasting advantage.

Few have written down how the work runs.

The real process lives in people's heads, in spreadsheets built to get around the system, and in exceptions nobody documented. That context is what makes a model useful on one specific account.

The edge is where it gets applied.

Put intelligence only at the steps that need judgment. Keep everything around those steps deterministic, logged, and approved by a person before it commits.

The method

Six stages, run as a loop.

  1. 1

    Audit

    Map the workflow as it really runs, exceptions included, and measure real time.

  2. 2

    Judgment

    Decide whether it needs changing at all, then which steps need a model.

  3. 3

    Build

    Scripts first, then an API, then a front end. Prove one workflow before building shared parts.

  4. 4

    Eval

    Grade against real past cases. Escalating to a person counts as a pass.

  5. 5

    Deploy

    Shadow mode before autonomy. Every run leaves a trace.

  6. 6

    Observe

    Watch edit rate and bypass rate. Whatever breaks next names the next workflow.

Each pass reveals the next workflow, because bottlenecks are connected.

From the notebook

What I'm writing up first.

All notes