Every few weeks another “agent SDK” launches, and the pitch is always the same: install this package, hand it a prompt, and you have an AI agent.
That’s true. You do. You have an agent the way a go-kart is a vehicle. It moves, it steers, and you should not merge onto the highway with it.
We looked hard at the popular ones. We didn’t ship on any of them. Not because they’re bad. Some of them are genuinely excellent. It’s that an SDK solves the easiest 10% of the problem and leaves you holding the other 90% at exactly the moment you have real customers.
Here’s what’s actually under our hood, and why it’s a different category of thing.
What an agent SDK actually gives you
Strip away the marketing and every agent SDK is the same four things:
- A loop. Call the model. If it asks for a tool, run the tool, feed the result back, call the model again. Repeat until it stops asking.
- Some built-in tools. Usually read a file, write a file, run a shell command, search the web.
- Context management. When the conversation gets too long, summarize the old part and throw the original away.
- A vendor. Almost always exactly one.
That loop is maybe 200 lines of code. It’s not the hard part. It was never the hard part.
The hard part is everything that happens when the loop is running on someone else’s money, in someone else’s account, for forty minutes, on a server that might restart.
The four questions an SDK can’t answer
What happens when the process dies?
An SDK loop lives in one process. Kill it at minute 19 of a 40-minute job and the work is gone. Not paused. Gone. There’s no record of which step it was on, no way to resume, and if step 12 already sent an email, nobody knows whether to send it again.
Whose data is this?
An SDK has no concept of a customer. Its memory is a folder or a variable. Ask it to only let this agent see documents this user is allowed to see, and there’s nowhere to put that rule, because the SDK has no tenants. You do.
Where does this tool run?
An SDK runs its tools wherever the SDK is running. But in a real product, “send an SMS” has to run on your servers with your carrier credentials, “call our webhook” has to run on the customer’s infrastructure, and “install this Python package and run this script” absolutely must not run anywhere near either of those.
What did it forget?
Every SDK compacts: summarize the old turns, drop the originals. Ask it what you decided three hours ago and it either remembers, or it confidently doesn’t. You can’t tell which from the outside. Neither can it.
We didn’t want better answers to these questions. We wanted an architecture where they mostly stop being questions.
The conveyor belt
Start with memory, because it’s the one everybody gets wrong and the one customers feel first.
Most systems treat context as a bucket. Fill it up, and when it overflows, boil it down to a smaller bucket and pour out the rest. That’s compaction. It’s universal, and it’s why your AI assistant is sharp for twenty minutes and vague by lunch.
We treat context as a conveyor belt.
FRONT of belt (in the AI's active mind) BACK (remembered, not resident)
┌───────────────┬──────────────────┬───────────────────┬──────────────────────┐
│ raw turns │ recent summary │ older summaries │ archived raw turns │
│ last N │ anchored, │ folded into │ full fidelity in │
│ verbatim │ structured │ each other │ the knowledge graph │
└───────────────┴──────────────────┴───────────────────┴──────────────────────┘
▲ ▲ │
│ └── a new message arrives, everything shifts right ──────┤
│ │
└──────────── recall pulls it back to the front ─────────────┘
(and pins it there, so it can't
immediately slide away again)
Messages sit on the belt in order. Every new turn nudges the older ones toward the back. Things at the back get compressed, first into a running summary, then into a summary of summaries.
But nothing falls off the end. It leaves the belt, not the system. Every original turn stays in full fidelity in the knowledge graph, tagged with its thread, its session, and its position in the conversation.
So when you ask about something from three weeks ago, the AI doesn’t consult a vague summary and guess. It goes and gets the actual turn, brings it back to the front of the belt, and answers from the real thing.
That’s not a metaphor we invented to sound clever. It’s how human memory actually behaves. Whatever you discussed most recently with this person about this thing is top of mind. Everything else isn’t gone. It’s one cue away.
Which is exactly why the belt is scoped to a thread, not a session. If you talk to your AI Teammate in Slack on Monday, again on Wednesday, and again on Friday, that’s one continuous memory, not three strangers wearing the same name badge. Sessions are visits. The thread is the relationship.
Why “just compact better” isn’t the fix
The team at Factory.ai published something genuinely useful. Instead of scoring summaries with text-similarity metrics, they tested whether an agent could actually keep working after compaction. Could it recall the original error? Name the files it changed? Explain what it decided and why?
They benchmarked three approaches. The best scored 3.70 out of 5. The others came in at 3.44 and 3.35.
Read that again. The winner, a serious team’s best attempt, purpose-built and carefully tuned, loses roughly a quarter of what mattered.
And one category collapsed across every single approach. Ask any of them which files were modified, and scores fell to 2.19 to 2.45 out of 5. Universally. Their conclusion was blunt: this doesn’t get fixed with better summary writing. It needs a separate index. Real structured tracking, not prose.
Two lessons we took straight into our design.
First: agents run out of memory. Summarizing forgets the details. An index list doesn’t. We keep an artifact index, a plain structured list of every file read, every file written, every page fetched, with the turn it happened on. It is never summarized, because it isn’t writing. It’s a ledger.
what a summary remembers what the artifact index knows
─────────────────────────── ─────────────────────────────────────────
"we worked on the config read src/config.ts turn 12 4,021 B
file and fixed the bug" write src/config.ts turn 19 4,300 B
fetch docs.stripe.com turn 23 8 results
Second, and this is the real unlock: Factory found that the most aggressive compression, squeezing context down by 99.3%, ended up using more total tokens, because the agent kept having to go re-fetch what it lost. Their line is the one to remember: the right target is not tokens per request, it’s tokens per task.
Once you can reliably retrieve what you compressed, compression stops being dangerous. A bad summary costs you a lookup instead of costing you the fact.
That’s the whole point of the belt. It doesn’t make compaction lossless. It makes it reversible, and reversible changes what kind of problem it is. Losing information is a correctness bug. Looking something up is a latency cost.
Reference, not value
The same idea applies to what tools hand back.
Watch a typical agent read a large file. It dumps all 200 kilobytes into the conversation. That single read might consume a fifth of the entire context window. And now it’s stuck there, crowding out everything else, until it gets compacted down into “we looked at a file.”
We don’t put the file in the conversation. We put a receipt:
the tool runs → the full result is stored and indexed
→ the conversation gets a short digest + an id
→ the AI can ask for any exact piece of it, any time
200 KB file read → about 400 tokens sitting in context
→ "read_artifact(id, lines 40-80)" returns the real bytes
Nothing is thrown away. It just may not be “top of mind”. Natural recall is built in. The AI keeps a working memory of what it has seen and can pull up any exact passage on demand, which, again, is how you actually work. You don’t hold an entire contract in your head. You remember which contract, and roughly where the clause was.
The harness: our loop doesn’t live in a process
Here’s the structural difference, and it’s the one that’s hardest to retrofit later.
Our agent loop doesn’t run in a script. It runs as a durable workflow, on infrastructure built for work that has to survive the machine it started on.
AN SDK LOOP A DURABLE HARNESS
┌─────────────────┐ ┌──────────────────────────────┐
│ one process │ │ workflow (survives restarts)│
│ │ │ │
│ step 1 ✓ │ │ step 1 ✓ recorded │
│ step 2 ✓ │ │ step 2 ✓ recorded │
│ step 3 ✗ CRASH │ │ step 3 ✗ retried ✓ │
│ │ │ step 4 ✓ recorded │
│ ☠ all gone │ │ step 5 … continues │
└─────────────────┘ └──────────────────────────────┘
In practice this means:
- Every external call is a step that can be retried on its own. The model call, the SMS, the file upload: each one retries independently instead of restarting the whole job.
- A pod can die and the work continues. Not from the beginning. From where it was.
- Long-running is normal. A job can wait four hours for a human approval without holding a connection open or burning a dollar of compute.
- Nothing irreversible happens twice. Before an agent does something it can’t undo, like sending a real email or completing a real payment, we write it down first. If a resume ever finds an unfinished entry, the system checks whether it already happened instead of blindly doing it again.
That last one is a customer-trust feature disguised as an engineering detail. The scariest failure mode for a business AI isn’t forgetting. It’s doing something twice.
Tools have addresses
Most harnesses have exactly one place tools run: wherever the harness is.
Ours knows that where a tool runs is part of what the tool is. Every tool declares its venue:
TOOL VENUE RUNS
─────────────────────────────────────────────────────────────────────
search the knowledge base builtin our servers, with your permissions
look up a contact builtin our servers, with your permissions
send an SMS / place a call builtin our servers, our carrier credentials
your custom integration http your infrastructure, your rules
something the UI does client in the user's browser
install a package, run it workspace a disposable machine, far from both
This is why we can give an AI Teammate a real computer, a live Linux desktop where it can install software, run code, and browse, without that being terrifying.
The AI’s mind never goes to that machine. The machine gets four commands: run this, read that, write this, list those. It holds no credentials, has no database access, and can see none of your customer data beyond the specific files we handed it. The reasoning, the permissions, the approval gates, and the audit log all stay on our side of the wall.
Compare that with the usual approach of installing an agent framework onto the machine and hoping the sandbox holds.
And because “where does this run” is a property of the tool rather than a special case buried in the code, adding a computer to an AI Teammate didn’t require a new system. It required one new venue.
We are not married to a model
Most agent SDKs come from a model vendor. That isn’t an accident and it isn’t malicious. It’s just what they’re for. Use their SDK, use their model.
Our runtime treats the model as a swappable part. Claude and GPT sit behind the same interface today, and the pieces people assume are vendor features (the loop, the tool system, context management, streaming) are ours.
The nice consequence is that vendor-specific capabilities become provider capabilities rather than platform assumptions. If a model offers built-in web search, a conversation running on that model gets it. A conversation running on a different model doesn’t miss a beat. Nothing in the platform branches on a vendor’s name.
That isn’t ideology. It’s insurance. Models change monthly. Pricing changes monthly. Being able to move without a rewrite is worth more than any single model’s convenience features.
The part nobody else can copy quickly
Everything above is architecture, and architecture can be copied.
This part can’t, easily.
Our memory doesn’t live in a folder. It lives in a typed knowledge graph with semantic search on top and permissions on every node. It’s the substrate we wrote about in Most are doing AI memory all wrong. Every conversation turn, every document, every call transcript, every artifact an agent produced becomes a node that knows where it came from and who is allowed to see it.
So when an AI Teammate recalls something, three things happen that a folder-plus-summary system structurally cannot do:
- It can cite. Page 3 of that contract. The 4-minute mark of that call. Not “I believe you mentioned.”
- It respects permissions. Recall runs inside the same access rules as the rest of your data. An agent cannot surface a document to someone who was never allowed to open it, not because we filter the output afterward, but because the record was never retrievable for that user in the first place.
- It’s scoped to a workspace. One customer’s memory is not adjacent to another’s. There is no shared index and no code path that forgets which tenant it’s serving.
A generic agent SDK cannot do this at any price, because it has no tenants, no provenance, and no permission model. Those aren’t features you bolt on later. They’re assumptions you either build on or you don’t.
Putting it together
┌──────────────────────────────────────────────────────────────────┐
│ YOUR APP ── open protocol, live streaming ──────────────────► │
├──────────────────────────────────────────────────────────────────┤
│ DURABLE HARNESS survives crashes · retries steps · resumes │
├──────────────────────────────────────────────────────────────────┤
│ MODEL-AGNOSTIC RUNTIME the loop · tools · the conveyor belt │
├──────────────────────────────────────────────────────────────────┤
│ TOOL VENUES our servers │ your webhook │ browser │ a computer │
├──────────────────────────────────────────────────────────────────┤
│ MEMORY typed graph · semantic recall · provenance · per-tenant│
└──────────────────────────────────────────────────────────────────┘
An SDK gives you the third row. Half of it.
The other four rows are the ones your customers actually experience: that it didn’t lose their work, that it remembers last month, that it cites its sources, that it can’t see the neighbor’s files, and that when it did something in the real world it did it exactly once.
The bottom line
An agent SDK is a great way to find out whether an idea works. It is not a way to run a business on one.
The loop was never the hard part. The hard part is durability, tenancy, memory that survives compression, tools that know where they’re allowed to run, and an audit trail that holds up when someone asks what happened.
We built a harness instead of importing a loop. It’s more work up front, and it’s the reason we can hand an AI Teammate a real computer, a real phone line, and real access to your company’s knowledge without any of those being a leap of faith.
Most systems make you choose between an AI that remembers and an AI that’s safe. That’s a false choice. It only looks like one if memory is a text file and safety is a prompt.
Put your context on a conveyor belt. Put your memory in a graph that knows who owns it. Put your loop somewhere it can survive the night.