Most are doing AI memory all wrong

A lot of “AI Influencers” shipping “agent memory” right now are doing the same thing: they write notes to a markdown file, let the context window fill up, and then compact, summarizing the transcript down to a smaller blob so they can keep going. Harnesses like OpenClaw and the Hermes agent harness treat a folder of .md files plus periodic compaction as if that were memory.

It isn’t. It’s quicksand. It looks solid on day one and swallows the whole thing by month three.

I want to lay out a different foundation, the one we designed in our working-memory and ingestion specs, and make the case for why a layered memory substrate (semantic vector search sitting on top of a large, typed, self-weighting knowledge graph) is the only thing that actually scales into what I’ll call a Company Brain: a platform that learns.

Why markdown + compaction is quicksand

A markdown memory file has four fatal properties:

1. Compaction is lossy by design. Every summarization pass throws away detail to save tokens. You cannot get it back. The thing you needed in October got averaged out of existence in August. Memory that silently rots is worse than no memory, because you trust it.
2. There is no provenance. A line in notes.md doesn’t know it came from page 3 of a contract, or the 4-minute mark of a sales call spoken by Ada. So recall can never cite. It can only vibe.
3. Retrieval is grep. You either stuff the whole file into the prompt (and pay for it, and blow the window) or you keyword-match against prose. There is no notion of “things related to this,” only “text that contains this.”
4. It doesn’t compound. Two files never link. Yesterday’s lesson and today’s task are strangers. The system never gets smarter; it just gets bigger and slower until someone deletes it.

Bolting a database under those markdown files doesn’t fix it. You’ve just given the quicksand a nicer bucket. The problem was never storage. The problem is the shape of what you store.

   MARKDOWN + COMPACTION  (quicksand)          LAYERED MEMORY  (bedrock)

     memory.md  ┐                               ┌───────────────────────────┐
     notes.md   ├─ stuff into the prompt        │   semantic vector layer   │ ← meaning
     summary.md ┘        │                      ├───────────────────────────┤
         │  window fills up                     │   typed knowledge graph   │ ← structure
         ▼                                      │   ◦──edge──◦──edge──◦     │
     ✂  COMPACT (lossy)                         └───────────────────────────┘
         │                                              │ provenance on every node
         ▼                                              ▼
     memory silently rots                    cite: report.pdf p.3 · call @ 00:04:12 · "Ada"

The foundation: typed nodes + typed edges, with provenance

Instead of writing prose to a file, we turn every piece of incoming information, whether a document, a call transcript or a CRM record, into typed memory nodes. Heavy extraction happens upstream (parsing, OCR, transcription); the memory layer only ever does one job: materialize nodes and the edges between them.

Two things make this a foundation and not just a fancier note:

• Every node carries provenance as first-class, queryable properties. A document chunk knows its page_number, file_name, and source_id. A transcript segment knows its start_ms, end_ms, and the speaker. Only the text is embedded for semantic search; the provenance rides along as structured scalars. That’s what lets recall cite the exact page and the exact second.
• Nodes have deterministic identity. A node keyed on its real-world identity (say a CRM id) produces the same node every time it’s ingested. Re-syncing a record updates the existing node instead of spawning a duplicate. The graph stays clean instead of bloating, which is the exact failure mode of append-only markdown.

Then we connect them. Edges are typed and weighted, never free-form: relationship_type comes from a small controlled vocabulary (PRODUCED, LEARNED, INFORMED_BY, …), and each edge carries weights = {confidence, importance, strength} plus a short natural-language edge_text reason that is itself embeddable. A controlled vocabulary is the “don’t make a mess” rule: it’s the difference between a knowledge graph and a hairball.

  raw source (doc · call · record)
        │  extract → provenance-tagged segments
        ▼
  ┌────────────────┐   PRODUCED · {conf, imp, strength}   ┌────────────────┐
  │ NODE           │ ───────────────────────────────────► │ NODE           │
  │ text  (vector) │                                      │ text  (vector) │
  │ page: 3        │ ◄──── same id ⇒ upsert, no dupes ─── │ speaker: Ada   │
  │ source: S-1    │        edge_text: "answers the ask"  │ t: 00:04:12    │
  └────────────────┘                                      └────────────────┘

Recall that is granular, not vague

Because meaning lives in the vector layer and structure lives in the graph, recall is a two-move operation: match on meaning, then walk the structure. You ask a question; the system finds the semantically closest nodes, follows their typed edges to the surrounding context, and returns a ranked list of the actual segments, each one carrying its provenance back with it.

The answer isn’t “here’s a paragraph I found.” It’s:

“…the flux-capacitor supply chain is the single largest operational hazard.” Source: report.pdf, p.3 · reinforced by call-42 @ 00:04:12 (Ada)

That’s the thing a markdown file physically cannot do. You get citations down to the page, the timestamp, and the speaker. You also get related nodes you didn’t literally ask for, because the graph knew they were connected.

  query ──► [ semantic match ] ──► [ walk typed edges ] ──► ranked, CITED answer
                 meaning                 structure               │
                                                                 ▼
                     "…"  · report.pdf p.3 · call-42 @00:04:12 (Ada) · deal "Acme"

The part nobody builds: a memory that learns

Storage is table stakes. The reason this is a foundation is the learn loop.

Every node and every edge carries a feedback_weight, and retrieval projects that weight into ranking. So when a recall answer comes back, the user (or an automated verifier) can react (👍, 👎, or a correction), and that feedback is tied to the exact nodes and edges that produced the answer. An improve pass then propagates it: good paths get boosted, bad paths get downranked, and corrections can be distilled into new knowledge.

We go one step further at the top of the loop. When a task finishes, the one place we spend a real reasoning pass is to distill a Learning: a small, reusable outcome like “for enterprise deals, lead with SSO.” That Learning becomes its own node, and future tasks find it through an INFORMED_BY edge. Yesterday’s lesson literally wires itself into tomorrow’s work.

   recall answer ──► 👍 / 👎 (+ correction)
        ▲                        │
        │                        ▼
   better NEXT recall     reweight the exact nodes & edges
        ▲                 that produced this answer  ── improve ──┐
        │                                                         │
        └──────────  distilled Learning ◄── INFORMED_BY ──── new task

This is the difference in one sentence: markdown memory gets bigger; graph memory gets smarter. A compaction pass makes yesterday blurrier. An improve pass makes tomorrow sharper.

The payoff: a Company Brain

Now zoom out. If every interaction (every document ingested, every call transcribed, every deal touched, every task run and rated) writes back into one linked, self-weighting graph with provenance, you don’t have “agent memory” anymore. You have a Company Brain: a single substrate where people, deals, documents, conversations, tasks, and outcomes are all connected and all continuously re-weighted by what actually worked.

Ask it “what have we learned about enterprise deals?” and it doesn’t grep a file. It returns distilled Learning nodes, ranked by how often they led to good outcomes, each traceable to the tasks and calls they came from. Every employee, human or AI, reads from and writes to the same brain, and the brain gets better every day it’s used.

        ┌───────────────────  THE COMPANY BRAIN  ───────────────────┐
        │   people · deals · documents · calls · tasks · outcomes   │
        │        one linked, self-weighting knowledge graph         │
        │        (semantic recall + provenance on every node)       │
        └───────────────────────────────────────────────────────────┘
              ▲  every interaction writes back           │
              └───────────────  it learns  ──────────────┘

The bottom line

Markdown files plus compaction, even with a database bolted underneath, will not scale and will never give you robust recall. It is append-only prose that rots a little every time you summarize it, can’t cite where anything came from, and never connects one fact to another.

A layered memory substrate, meaning semantic vector search over a large, typed, provenance-rich knowledge graph with a feedback loop that re-weights the exact nodes and edges behind every answer, is a foundation you can build a company on.

One is quicksand. The other is bedrock. Stop pouring your memory into the quicksand.