I benchmarked 8 local LLMs to extract memory from my AI-agent sessions. The 16GB model beat the 70B - and a JSON schema beat all of them

TL;DR

I run a self-hosted memory layer for my coding agents. One job in it uses a local LLM to read each finished session and pull out durable “facts” - decisions, preferences, configs, and step-by-step procedures - so the next session starts already knowing them. Sessions contain personal and financial data, so extraction has to stay on my machine. The question was: which local model?

I stopped guessing and measured 8 of them on the same 6 sessions, same prompt, on an M4 Pro / 48GB. Four things surprised me:

1. The output schema mattered more than the model. Ollama’s plain format:"json" made several models return a single fact instead of a list - 1 fact per session. Forcing a structured-output schema lifted every model, including the incumbent, from 1 to 31 facts over the same 6 sessions. Biggest single win of the whole exercise, and it’s a config line.
2. The biggest model never finished. llama3.3:70b (34GB) OOM’d three sessions in. A 16GB dense model beat it on both quality and the ability to actually run.
3. More extractions ≠ better. One model produced 120 facts (4× the winner) - and they were garbage fragments. A naive “more is better” metric would have crowned the worst model.
4. The winner was the cheapest one that fit comfortably in RAM. qwen3.5:27b-int4, 16GB - smaller than the model it replaced.

Context: why a local model at all

The project is agent-memory-hub - a self-hosted store (Postgres + pgvector) that captures my agent sessions and injects the relevant slice back into future ones. The facts layer is the interesting part: after a session ends, an LLM reads the transcript and emits typed, durable statements:

• decision - “don’t withdraw from the pension plan to pay off the loan”
• preference - “only invest after the emergency reserve is fully funded”
• config - “the pension is a PGBL on the progressive tax table”
• fact - “the consigned loan runs Jul/26-May/27 at ~2.49%/mo”
• procedure - “to close a cash gap: renegotiate supplier terms, then liquidate crypto/USD, then sell BR equities”

Those transcripts are full of financial and personal detail. Sending them to a hosted API was a non-starter, so extraction runs on a local model via Ollama. That’s the whole reason this benchmark exists: I don’t get to pick “just use the best cloud model.” I have to find the best model that runs on the hardware I own.

Method (small on purpose, reproducible)

• Corpus: 6 real sessions (a financial-planning thread - dense, procedure-heavy - which is the hardest case for this extractor).
• Same prompt as the production extractor, same structured-output schema once I found the fix (more below).
• Machine: MacBook M4 Pro, 48GB unified memory. Empirically it leans on swap well before the nominal 48GB is full, which turns out to matter a lot for the larger models.
• Signals I recorded per session: number of facts, number of procedure facts specifically, whether the JSON parsed, wall-clock seconds - plus I read the actual output of every model. Count is a proxy; the manual read is the tiebreaker.

This is not MMLU. It’s a scrappy, task-specific benchmark on my data - which is exactly the point. The “best model” for reading my agent sessions is not a leaderboard question.

How the run actually went

The first pass almost sent me down the wrong road. Every model except the Qwen 27B dense returned exactly one fact per session - Gemma, Mistral, even my incumbent MoE. Same financial session, run through Gemma 3 27B: one trivial fact, and that was it. My honest first read was “these models are too weak for extraction.”

Then it clicked: it wasn’t the models. Ollama’s format:"json" lets a model return a single JSON object when the prompt asks for an array - and Gemma and Mistral did exactly that. My production extractor used that same format:"json", which meant the first eval was quietly unfair: it rewarded the one model that happened to emit arrays and punished the rest for a formatting quirk. The fix wasn’t a stronger model - it was a schema. A structured-output schema that forces {facts:[...]} gives every model the same shot (and it’s a real improvement, so it went straight into production).

Re-running with the fair schema, the benchmark started killing itself. The eval process OOM’d again and again. Two mistakes on my side: I was loading each new model on top of the previous one still resident, and every model ran with Ollama’s default context - which inflates the KV-cache toward 32k when extraction needs ~4k (the extractor truncates input at ~12k chars). On a 48GB machine that’s a guaranteed crash. I rewrote the harness with three safeguards: num_ctx=8192, unload the previous model before loading the next, and checkpoint per model so an OOM on model 7 didn’t cost me the results of models 1-6. Only then did the run finish end to end.

And with the fair schema, the ranking flipped twice. The “weak” baseline MoE jumped to 31 facts / 9 procedures - never weak, just mis-measured. Then the raw counts lied in the other direction: command-r exploded to 120 facts, one session alone producing 68. That’s not a model reading well; that’s a model fragmenting the transcript. Which is where I stopped trusting counts and started reading output. The findings below are what survived that.

Finding 1 - the schema beat every model

That first surprise is the single most important result here, so it’s worth stating as a rule rather than a war story. The fix the unfairness demanded - a structured-output schema - is this:

FACTS_SCHEMA = {
    "type": "object",
    "properties": {
        "facts": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "kind": {"type": "string",
                             "enum": ["preference", "decision", "config", "fact", "procedure"]},
                    "text": {"type": "string"},
                },
                "required": ["kind", "text"],
            },
        }
    },
    "required": ["facts"],
}

body = {"model": model, "prompt": prompt, "format": FACTS_SCHEMA,
        "stream": False, "options": {"temperature": 0.2, "num_ctx": 8192}}

Same models, same prompt, now constrained to {facts:[...]} with a kind enum. The incumbent MoE went from 1 → 31 facts over the same 6 sessions. Every model improved. The enum also killed almost all the “wrong category” noise for free.

The lesson generalizes past this project: when a local model “can’t do structured extraction,” check your output contract before you swap the model. The harness was the bottleneck, not the weights.

Finding 2 - the biggest model never finished

Conventional wisdom says throw the largest model you can at a reasoning-ish task. So I tried llama3.3:70b-q3 (34GB on disk).

It OOM’d on the third session and never recovered. On a 48GB machine that leans on swap well before that 48GB is nominally full, a 34GB model plus its KV-cache simply doesn’t have room to work. Its two completed sessions were mediocre anyway (5 and 3 facts, one with zero procedures).

gemma3:27b-it-q8_0 (29GB) did the same thing - OOM on session 6.

Meanwhile a 16GB dense model ran all six comfortably and produced the best output. On this task, “bigger” lost twice: it didn’t fit, and where it did run, it wasn’t better.

Finding 3 - the KV-cache was the real memory hog

Here’s the subtler part of the OOM story. The model weights weren’t the whole problem - the KV-cache was. Ollama’s default context window (up to 32k) inflates the cache enormously, and extraction only needs ~4k tokens of input.

Pinning num_ctx=8192 was the difference between “27B-class models crash mid-run” and “27B-class models are stable.” So part of “the big model doesn’t fit” was actually a configuration bug on my side. Worth checking before you conclude your hardware is too small: a right-sized context window can move a model from OOM to comfortable.

Finding 4 - count is a trap; quality is the metric

command-r:35b produced 120 facts - nearly 4× the eventual winner. If I’d ranked by raw count, it would have won in a landslide.

Then I read them:

• “Seu problema é de fluxo.” (“Your problem is cash flow.”)
• “Cartão virou crédito, não meio de pagamento.” (“The card became credit, not a payment method.”)
• “Sacar a previdência custa ~R$ 6k a mais.”

These aren’t durable memory. They’re the transcript chopped into motivational fragments - over-extraction that would pollute recall, not help it. One session alone produced 68 of them. Count went up; signal went down.

The inverse failure showed up too. gemma3:27b-it-qat produced a healthy 38 facts - tied with the winner on volume - but only 1 procedure across all 6 sessions. For a system whose whole point is turning procedures into reusable agent skills, a model that can’t spot procedures is useless no matter how many facts it emits. If you care about a specific extraction subtype, measure that subtype, not the aggregate.

Finding 5 - the winner

qwen3.5:27b-int4 (dense, 16GB) took it:

• 38 facts, 10 procedures across the 6 sessions - the most procedures of any model.
• Self-contained statements (each readable without the transcript), correct language, cleanly categorized.
• 16GB footprint - smaller than the 23GB MoE it replaced. It didn’t just win on quality; it relieved memory pressure instead of adding it.

The honest tradeoff: it’s slower. The MoE (3B active params) averaged ~16s/session; the dense 27B averaged ~60s. For an interactive path that would matter. But this runs as a nightly batch over finished sessions, so I happily traded ~44 seconds per session for better facts and more procedures. The int8 variant of the same model was marginally different in quality, ~86s/session, and 29GB on disk - right at the ceiling. Not worth it.

The full table

All sizes below are exact on-disk sizes from ollama list (what the model actually occupies on the SSD). Runtime RAM is higher - weights plus KV-cache - which is exactly why the ceiling bites (see Finding 3).

What didn’t work / the limits

I want to be straight about how far this generalizes:

• N is small. Six sessions, single run each. It’s enough to separate “great / fine / broken,” not enough for a 2-point quality gap. I treated close calls (MoE vs dense qwen) with the manual read, not the counts.
• No gold-labeled set. “Quality” here is fact count + procedure count + me reading the output. A curated gold set would make the ranking sharper; that’s future work.
• Mostly one language, one domain. The corpus is Portuguese, finance-heavy. A model that nails this might lag on English code sessions. If your data differs, your winner might differ - which is the whole thesis.
• One machine. Every OOM verdict is relative to 48GB unified memory. On a 24GB CUDA card the ceilings move; on a 128GB box the 70B might finish (and might still lose on quality).
• Speed vs quality is a real fork. The MoE is ~4× faster. If this were interactive, I might have chosen it. It’s a batch job, so I didn’t.

Takeaway

Before you download a 70B or buy a bigger GPU for a local extraction task:

1. Fix your output contract first. A structured-output schema beat every model swap I could make. If a model “can’t do structured extraction,” suspect the harness before the weights.
2. Define quality; don’t count. The highest-volume model was the worst. Decide what a good extraction is, then measure that - including the specific subtype you actually care about.
3. Right-size the context window. The KV-cache, not the weights, caused half my OOMs. num_ctx matched to the task can turn “doesn’t fit” into “comfortable.”
4. Measure on your own data. No leaderboard tells you which model reads your sessions best. Six sessions and an afternoon did.

The model reading my memory today is a 16GB int4 that a “bigger is better” instinct would have skipped - and it beat a 70B that couldn’t even finish the run.

The whole thing is open-source and self-hosted; the extractor, the schema, and the eval script are in the repo. Link below.

Repository: github.com/carloshpdoc/agent-memory-hub