Agent Memory That Beats Grep: How I Built Heimdall

Originally published on my blog. Repo: github.com/ArihantDeva/heimdall

The problem: agents redo work because their memory is search, not retrieval

Every agent session starts cold. That’s the hidden tax of working with AI coding agents: no matter how many times an agent has solved a problem, the next session begins with zero knowledge of it.

I’ve been building with agents for a while now, and the pattern that cost me the most wasn’t slow loops or bad prompts — it was rework. The same pipeline, the same fix, the same architecture decision, rebuilt three times in three different project directories, because the agent starting each session had no idea the work already existed.

Grep can’t fix it. The answer to “did I already solve X?” lives in a different project’s file, described in prose, under a path you don’t know. A single LLM query is a coin flip — it hallucinates, or it returns one flat hit with no indication of whether the file is even still on disk.

The realization that changed my approach: agents needed a memory layer that returned ranked, verified, reusable work — not text matches.

So I built one. It’s called Heimdall, it’s open source (MIT), and the part that makes it work is the part nobody builds: the trust verification layer.

The build: a memory layer, not a search box

Three pieces, each solving a distinct failure mode.

1. A persistent graph store

Every completed fix, pattern, or decision is recorded as a node — title, body, keywords, and edges to related work — in a local semantic memory daemon.

I did not reimplement the vector engine. The store is Graft, a local-first semantic memory daemon (SQLite + local embeddings + graph edges, Apache 2.0). It provides hybrid lexical + vector ranked retrieval, keyword dedup, and a local embedding model (bge-m3) that runs entirely on-device.

Proper attribution up front: Graft is the storage and ranking engine; Heimdall is the orchestration layer on top — watching agent sessions, keeping the graph fresh, and making results trustworthy. That split is the honest story, and it’s the right architecture: don’t rebuild a vector store, build the layer that makes one usable by an agent.

2. Ranked retrieval + graph walk

Search returns top-k candidates scored by a hybrid of lexical and semantic similarity, then walks the graph to surface related work you didn’t know you had.

Ask about a resume-tailoring pipeline and you get the pipeline, the scoring script you forgot about, and the gotcha notes — edges are part of every search, not an afterthought. The graph walk is the difference between searching and remembering: it surfaces the work your future self needs, not just the exact string your past self wrote.

3. The verification layer — the actual novel part

Every hit gets a trust verdict before it’s shown:

• STRONG — the anchor path exists on disk and has strong lexical coverage
• WEAK — semantic match, path exists
• STALE — path gone → auto-rehome (find where the file moved) or auto-remove
• REBUILT — stale node found its new home, rebuilt in place

An agent acting on a dead path is worse than no answer. The verdict is what makes the graph actable: my agent trusts a STRONG hit enough to reuse it without re-verifying, and STALE hits get cleaned up instead of poisoning future searches.

Why it beats the alternatives

	grep	single LLM query	Heimdall
finds prose knowledge	✗	~	✓
ranked candidates	✗	one flat hit	✓ top-k
path verified on disk	✗	✗	✓ verdicts
discovers related work	✗	✗	✓ graph walk
survives file reorgs	✗	✗	✓ self-healing

How it fits with the tools you already know

If you’ve seen code-graph tools, the natural question is: isn’t this what graphify or Graft already do? Fair — here’s the honest split.

Graphify builds a per-repo AST+semantic graph (graphify-out/graph.json) from your source, LLM-extracts nodes, and answers questions by BFS-walking that graph with a token budget. It answers codebase questions — “where does this function live, what calls it” — with precise, token-cheap context. What it doesn’t do: persist across projects, or tell you whether a node’s file still exists. It’s a code graph, not a memory.

Graft (Apache 2.0) is the semantic memory backend: a local daemon with SQLite + local embeddings + graph edges, hybrid lexical/vector retrieval, and verified semantic caching. It’s the storage and ranking engine — the hardest part to build well. But a backend isn’t a system: it needs something to watch sessions, keep the graph fresh as files move, and label hits so an agent can trust them.

Heimdall is the layer between. It watches agent sessions and auto-syncs the graph (edits, moves, deletes, bulk renames), verifies every hit with a trust verdict, guards against grep-habit, and orients agents at session start.

	graphify	Graft	Heimdall
scope	one repo	one daemon	all projects
content	code AST+semantic	notes/facts	notes + code + session edits
query	BFS over graph.json	hybrid ranked	ranked + graph walk
trust verdicts	✗	partial	✓ STRONG/WEAK/STALE
session auto-sync	✗	✗	✓
reorg self-healing	✗	✗	✓
drop-in	✗ (code-only)	✗ (backend)	✓ (orchestrates Graft)

All three are useful; they’re just different layers. Graphify gives you a code map, Graft gives you a store, Heimdall is the agent-integration layer that makes retrieval trustworthy.

The lesson: memory is a systems problem

The insight that generalizes: agent memory fails not from lack of storage but from lack of retrieval with trust.

Any agent harness — Claude Code, Cursor, Copilot, a custom agent — could ship this. The components are small: a graph store, a session watcher, a verifier, a guard. None of it is hard. The hard part is noticing that this is the missing layer — and that grep-shaped thinking is why agents keep re-doing work.

In practice it paid for itself: I stopped rebuilding pipelines that already existed, and sessions now start with relevant prior work injected instead of from zero. A concrete example: I needed an ATS-scoring check for a resume workflow. A year ago that was a fresh build — grep for “ats” returns a dozen unrelated files, a single query returns one guess. With Heimdall, one search returned the scoring script I’d forgotten, its calibration notes, and a gotcha note about a parser quirk — ranked, verified, reusable in seconds. That’s the difference between an agent that redoes work and one that compounds it.

Open source

The repo is at github.com/ArihantDeva/heimdall — MIT licensed, with:

• extensions/ — kb-tools.ts, kb-orient.ts, kb-autosync.ts, kb-search-guard.ts + lib/kb-guard-core.mjs (pure JS, zero deps)
• bin/ — kb-search.sh, kb_search_verify.py (the trust verifier), sync-edits.sh, kb-health.sh, kb-stale-scan.py, telemetry.sh, seed-graft.sh, kb-rebuild.sh, kb-rehome.sh
• config/ + launchd/ templates — plug into any semantic-memory backend with a retrieve/insert/explore CLI

The Graft binary is not bundled — Heimdall detects it and degrades gracefully. Install Graft, point bin/ at it, done.

If you run agents across multiple projects, this is the missing layer.