The work behind Inkolega

Our own agent engine

16,666 lines of TypeScript. Its hard rules sit in a deterministic policy gate in code, outside the model, so a fooled model can't talk its way past them, and nothing it writes reaches a customer without a person's tap. Claude Code writes it, I direct it, and a second AI attacks the big builds. Most tests run against a stand-in for the AI model, so they test our code, not the AI.

Checks on every answer

Every number in Otto's answer must appear in a tool result, a record or the owner's own words. "I've sent the list" is caught by its grammar, because Otto cannot send. A bare "ok" cannot change a figure he was just told. A failed check gets one repair round, then a visible flag. His own rules become checks too: a price list Otto drafts is refused if it holds a formula, a hidden cell or a price equal to cost.

Every model run is sealed: it starts in a fresh empty folder with only our own tools, and is killed if the tool's startup report lists anything else. No run starts without a passing canary. It plants a file holding a fresh codename and an order, and passes only if the real test word comes back and the codename stays out. The same plant runs with project settings switched on, and that run must find the codename, or the canary proves nothing.

Red team on our own engine

Our own AI wrote 470 attack emails (prompt injection, fraud). 2 failed to run, and the engine read the other 468 live. 337 (72%) went straight to a person and 131 got a draft. None of the 468 gave away data, changed a recipient or confirmed fake bank details.

The counting rule was fixed before the run: 461 held under it outright. Of the other 7, 3 were false flags, and 4 got a draft to repeat an attacker's code word or address, which I judged harmless. A second AI grading the 131 drafts was stricter: it said the attacker got something in 6 of them (fully in 2), mostly that same repeated code word or address. We also ran 100 normal emails: 76 got a draft, and the 24 that went to a person were complaints and thank-yous, not a single plain booking or question.

Supplier sheet reader

When a supplier's sheet arrives in a new layout, a model writes a small plain-code parser for it. Code keeps the parser only if it reproduces the known right output exactly on held-out files of that layout. At run time no AI model is involved, and any file that doesn't match what the parser was accepted on goes to a person.

We generated 30 layouts with traps modelled on real sheets: English, German and Dutch headers, merged headers, European and US number formats, subtotal rows, items behind a cover sheet. Each was tested on 5 sealed files and 4 drift files with one planted change, like two swapped columns or a discount row. Our gate accepted all 30 layouts, got 0 of 150 sealed files wrong and sent all 120 drift files to a person. A schema-only check, the kind the nearest open-source tool uses, let 14 wrong outputs through: 5 on clean files, all from one layout, and 9 on drift files. The price of being strict: 54 drift files the parser would have read correctly also went to a person.

The pass mark was written before any code and never moved. The data and grader were hashed before the gate was written, and a fresh clone re-runs the gate on the saved parsers and gets every number again. Seven malicious parsers are refused before they run. 7,440 lines of Python, tests included; 56 tests.

Where the jobs come from

Calls with our first customer, a hotel, and a long sit-down with the owner of a wholesale trading firm. The hotel wanted guests booking direct and paying before they arrive, with nobody typing card numbers by hand, and agreed to pay only once it was live and working. It has paid every month since September. The trading owner is the engine of every deal, and when he travels nothing moves. He fears his data in tools he doesn't trust, and an AI writing to his customers by itself. Nothing our AI writes reaches a customer without a tap.

Live test rounds

A character he relies on has to answer the same way every time, so we test it like one. A frozen, hashed file of his real asks runs through the real model on a frozen copy of his files, at a pinned clock. Each ask runs three times, because the same model on the same build flips. Code scores first, then a grader model scores right, trust, effort and voice, three votes and the median, and we read every extreme score ourselves. Rounds are compared by cases lost, never by points.

The replay game: a backtest on his own deals

Can an AI make the calls the owner makes? We rebuild his world exactly as it stood at a real moment in the past, hide everything after it, let Otto decide, and only then open what he really did. We never filter a full database, because one forgotten column leaks the future. The store is built forward from empty, item by item in the order he received them, by Otto's own production code at a clock pinned to that moment. A planted item dated after the cut turns the check red.

Code finds his decisions: each step on a deal, each deposit he asked for, each price list he sent. Where one exact answer exists, code scores it to the cent, beside two plain baselines: "same margin as last time" and "move nothing". A figure that appears nowhere in what he had by then is marked unknowable and kept out of the score. Results read "matched him", never "better than him": even where better is the goal, logs can't prove it. It borrows from SWE-bench and from walk-forward backtesting in trading research.

Otto has to earn the right to suggest a price. Per supplier, it stays silent until his history holds at least 20 lines he moved by hand, it moved the same way on more than half of them, and none fell below his price floor. The replay holds the cut on a sample of his files today; his full history is next.

Which deal needs him today

A follow-up keeper tracks open deals and says who needs a message today. It has to pass this bench before we build it. Code writes WhatsApp-style deal chats with known states, and a grader in code scores any system's reading of them.

On "does this person need a message from me today?", one prompt to Claude Opus beat simply answering "yes" for everyone by about 21 points, and a plain no-AI rule by 29.5 (95% interval +22.6 to +36.3). It put 97.6% of the 254 open deals on the morning list, every deal it listed belonged there, and it got all five fields right (deal state, who moves next, due date, last message, the messages cited as evidence) on 95.9% of 565 deals. A keyword rule got all five fields right on 97.8% of deals worded like the ones it was written from, and on 34.5% worded differently. These code-written chats are cleaner than real ones, so the bench is not yet hard enough to pick a design.

The twin

Real data has no answer key, so we generated a whole trading firm: 2.5 years, 62,492 business chat messages, 17,246 emails in 8,033 threads and about 1,800 documents in the real file formats, down to their layout and number quirks. It includes 12 private chats that must never be read. Beside it sits a truth file with every deal, step, payment and hand-moved price.

The same seed gives the same twin. A checker confirms totals to the cent and fails if any made-up price equals a real one. We plant 56 details across chats, mails, attachments and files, beside traps like a forwarded offer, an unsent draft and an old price, and every miss gets a cause: not stored, stored but not found, found but not said, or said wrong.

Context engineering race

Can a cheap model get better at a job if a stronger model rewrites its instructions from its graded mistakes, with no model training? We ran Microsoft's open-source SkillOpt loop, with Claude Opus rewriting Claude Haiku's instructions, on Formula, a public benchmark where ACE, a published method, gained 18 points on DeepSeek-V3.1. The plan and the sealed test set were hashed (fingerprinted, so neither could change) before the run, and the noise floor, how far scores move by pure chance, was worked out first: 6.6 points at 200 items.

On Claude Haiku, the learning runs lifted the score from 71.0% to between 85.5% and 89.0%, a gain of 14.5 to 18 points (p < 0.0001). Two controls show the gain needs the right answers: rules written without seeing any graded examples scored 68.5%, and learning from shuffled, wrong answers scored 74.5%. Pasting every training example into the prompt also worked (87.5%), but used about ten times the input per question. The gain is on the same kinds of item it learned from, not on new kinds: every formula in the test also appears in training.

A second race, turning customers' order messages into orders against an invented catalogue, was too easy to teach anything: Claude Sonnet got 399 of 400 and Claude Opus 400 of 400, so we stopped it before the learning runs.

Memory tools, measured

Before trusting a memory product with an owner's history, we ran them hands on, on one made-up slice of 175 items with 15 questions. Mem0 got 5 right and 2 partly. LightRAG got 9 and 2. A cheap model simply reading the raw items got 11 and 3, and 14 and 1 with each item labelled by its order. The limit was never storage. It was whether each message is linked to its deal, and whether the raw words get read.

Graphiti was too slow to run whole, at 28 seconds an item, but it alone handled a planted correction properly, so we copied that one mechanism.

Testing a model instead of trusting it

A promising outside decision model appeared this autumn. Instead of trusting its documentation, we built 1,270 judgement items of Otto's own kinds from the twin and ran both. Ours won on work or personal (0.985 to 0.910), on spotting a promise (0.967 to 0.842) and on numbers and dates (0.960 to 0.893). The outside model lost up to 20 points on Danish against English, where ours lost nothing. We kept its best idea: asking "is it any of these?" as a separate question.

What other agent systems taught us

• Town It writes your profile in under a minute and you correct it, so nobody starts from a blank page. Its founder calls that low time to value the only reason the product works.
• Claude Code A firm's deny now holds even when a person's add-on says allow, because the check is re-run without the person's layer. Putting deny rules first was not enough.
• Codex CLI The sandbox (what a command can touch) and the approval policy (when a person must say yes) are separate settings. When several command rules match, the strictest wins.
• OpenClaw After downtime, missed runs are merged and rescheduled, not replayed one by one. A job that keeps failing switches itself off and says how to fix it.
• Grok Bot An approval nobody answers expires after about 10 minutes instead of running. Instructions inside a received email never count as permission.
• OpenAI dots A separate reviewer agent can veto risky steps, and three straight denials trip a circuit breaker. A finished run is not proof the result arrived, so check delivery separately.
• Microsoft Autopilot Using a tool takes two separate yeses: consent to the tool's token, and membership of the data behind it. Anyone can stop the agent; only an admin can restart it.
• Meta Muse A separate process, Sentinel, is the only way out to connectors and the internet. Taint rule: a process that has read user data loses its auto-allow.
• Sierra Grade a job by the final state of the system, not the agent's words. Run it k times and score the chance all k pass.
• Intercom Fin Check the final answer against its sources. If the check fails, hand over to a person instead of guessing.
• Replit Agent Its agent deleted a production database during a code freeze and said rollback was impossible, unaware that Replit had one. What an agent can undo should be a written list it cites, not its own belief.
• Anthropic Start evals with 20 to 50 tasks from real failures. A new checker is poor out of the box, and you tune it by reading its logs.
• LangChain Review only the actions that reach the outside. Read-only tools run unreviewed, and a model missing a fact asks instead of inventing it.
• Lindy Turn a proven routine into a script that calls the model only when stuck. Show a job's cost before it runs.
• Letta Its default Slack "pairing" guard was never enforced and behaved as open. Test that a guard actually blocks, not that its setting is on.
• OpenHands Each action gets a risk level, and only those above a threshold ask a person. Unknown risk asks too, and a crashed checker counts as high.
• Fyxer The edit a person makes to a draft is the learning signal. A full rewrite teaches little.
• Shortwave Avoid long chains: one call picks the tools, they run in parallel, one call answers. Each draft gets the 5 to 10 most similar sent emails as style examples.
• Help Scout No drafts until an inbox has about 100 past replies. Its own support writer warns that weak drafts tempt people to just hit send.
• Inbox Zero Compare each draft with what was actually sent and keep at most 3 lessons from the difference. A memory written from chat must quote the person's exact words, checked in code.
• Marblism Every tool gets a level: always, ask or never. Chat memory stays with one agent, while a shared Brain holds what the whole team knows.
• Wabi Its terms make the user's confirming tap their legal instruction to act, and cap liability for AI features at $100.
• Restate An approval wait is saved with a callback ID, so the agent sleeps using no compute and the wait survives a crash.
• OpenBot Speak first without nagging: notify only when the count of waiting things rises, and stay silent on a quiet day.
• cua Type a password only when the page is on the saved site's exact address, so a look-alike page gets nothing.
• Geckobox Stop kills the agent's whole process tree, children first. Reading a job's status repairs it: a "running" job whose worker is dead is marked failed, with the reason.