A Domain Graph, Measured Against What It Forgot
Could I turn my AI chat history into compact knowledge graphs without losing what I return to those conversations for? I built a harness to find out. Compression worked; reconstruction depended on what I asked for. The harness is the point; the graph is what it produced.
The premise
I built a custom Claude skill called domain. It reads a conversation and
extracts a warranted event graph: who established something, changed it, named
it, or asked for it to be made, with evidence back to the conversation.
Partway through, the question got larger. Could I run my entire AI chat history — Claude, ChatGPT, and Codex — through this and end up with something smaller that still answers the questions I reopen those chats to ask?
Smaller is easy. Useful after getting smaller is the part that needs a test. The intended harness compares transcript and graph on file size, token count, query latency, and reconstruction accuracy, conversation by conversation. This first pass establishes compression, investigates what survives, and tests a cheap way to predict where the graph will need help. It does not establish a query-latency improvement or a production pipeline across the whole archive.
The harness is the point; the graph is what it produced. A testing harness isn't overhead you add after a feature. It's how you find out which feature was worth building.
The spine: extract an event, keep its warrant
The skill's unit is an origin event. Its basic shape is
from -[operation]-> target, with a separate to for whom the act was for,
plus a warrant and a source span. The distinction I want to preserve is
between a meaning edge — why something matters — and an execution edge — how
something gets done.
The ledger lives in domain.toon. TOON means token-oriented notation: a
compact table representation suited to a uniform edge array. The important
compression, though, happens before serialization. The extractor chooses
which events deserve a row.
That choice is visible in the artifacts. The branding graph retains naming, redefinition, correction, and handoff events. It holds back an abandoned name and a thematic association that didn't become a durable project-state change. The technical graph retains the intake strategy and the request for a YAML write-up; individual source entries remain content inside that artifact, rather than becoming separate origin events.
The graph preserves a record of changes. It doesn't promise to preserve everything those changes produced. The harness has to score both kinds of question, or compression can look successful simply because the test never asks for what was discarded.
Compression worked. Reconstruction split by question.
I started with two hand-extracted conversations: Domain table structure and updates, about REBUS/BURSE and the relationship between human principles and work; and Neara Corus Demo Refinement, a technical conversation dominated by source lists, stack choices, and an intake specification.
| conversation | transcript bytes | graph bytes | estimated transcript → graph tokens | character compression |
|---|---|---|---|---|
| branding / philosophy | 33,671 | 4,239 | ~8,313 → ~1,058 | 7.9× |
| technical / specification | 74,016 | 3,315 | ~18,351 → ~828 | 22.2× |
Those are file measurements and character-count-divided-by-4 token estimates,
not tokenizer results. tiktoken was unavailable. The compression column is
the ratio of character counts, consistent with those estimates; bytes are
listed separately so the units don't quietly change underneath the claim.
The reconstruction exercise answered questions from the graph alone and graded against the transcript. The experiment brief reports the same ordering in both conversations: decision, state, provenance, and terminology recall held up better than verbatim content recall. The underlying question-and-answer grading sheet isn't among the retained artifacts, so I am not presenting its exact scores as independently auditable results here. The brief itself warns that its round percentages come from a tiny, hand-graded question set. Trust the ordering as a working signal, not the apparent precision.
The files make the mechanism inspectable:
| question asks for | what the graph keeps | the lesson |
|---|---|---|
| a naming or framing decision | actor, operation, target, warrant | The event is the useful compression unit. |
| why the technical conversation changed direction | a short record of the new framing and intake strategy | A decision can survive without the surrounding exposition. |
| the full source registry or YAML | the event that requested the artifact | Recording that content was produced does not reconstruct its contents. |
The technical conversation has 11 extracted edges despite its much larger payload; the branding conversation has 18. Those are extraction judgments, not objective counts of everything meaningful that happened. But they explain why the technical file compresses harder: much of its size is content attached to comparatively few selected events.
The resulting product decision is conditional. A graph alone looks promising for decision questions. Content-heavy conversations need the graph and working pointers back to the source spans. The higher compression ratio comes partly from dropping exactly the material I might reopen the chat to get.
Watching one correction survive
In the branding conversation, the assistant tries to reinterpret the relationship between Discipline and Ascending Sound. I correct it:
Discipline is in the right place table. Ascending sound is the most accurate row within the BURSE table.
The graph retains that sentence as the warrant on this edge:
from: jeremy
to: assistant
op: correct
target: as_row
Follow the information through the pipeline. The transcript contains the assistant's proposed reinterpretation, the tables, my correction, and the assistant's acknowledgment. The graph selects the correction, identifies who made it and to whom, and attaches it to the Ascending Sound row. A future reader can recover the intervention without reading the whole exchange.
Now change the question to ask for the complete table. The correction edge doesn't contain it. The source transcript does. The same extraction succeeds or fails depending on what the reader needs reconstructed.
There is another limit visible in the technical example. Its graph condenses the RAG discussion into a framing decision. The transcript is more qualified: RAG may still help with source intake, even though it is the wrong primary product frame. That qualification matters. A short edge label cannot silently replace the fuller answer.
Likewise, the demo's model input is synthetic; public grid and vegetation layers are reference data and proxies, not Neara's private operational model. The source conversation explicitly withholds a claim to exact crew costs. Retaining the demo decision must not turn those boundaries into facts the demo has supposedly proved.
This is why the warrant and pointer belong in the design. They let a compact claim lead back to the passage that can qualify it. The retained ledgers have span labels; this pass does not establish that those labels form a validated, automatic retrieval path.
The cheap signal: structure beats vocabulary
Before extracting the archive, I wanted a cheap estimate of where graph-only reconstruction might be enough. My first candidate was edges per KB: how many durable events a conversation contains relative to its size.
That is circular. Counting extracted edges requires doing the extraction I am trying to decide whether to do. It can confirm a result afterward. It can't be the cheap gate before it.
So I compared signals computable directly from the transcript:
| signal | branding | technical | separation | lesson |
|---|---|---|---|---|
| extracted edges/KB | 0.547 | 0.152 | 3.6× | Useful afterward; requires extraction. |
| turns/KB | 2.25 | 0.221 | 10.2× | Strong separation here, no semantic reasoning. |
| average characters/turn | 449 | 4,588 | roughly 10×, inverted | Long payloads show up in turn length. |
| fenced-code share of lines | 3% | 70% | strongly separated | Useful, but dependent on formatting. |
| type-token ratio | 0.166 | 0.149 | 1.1× | Barely distinguishes these files. |
| gzip ratio | 0.307 | 0.291 | 1.05× | My compressibility hunch didn't work. |
signals.py counts ### USER and ### ASSISTANT markers and divides by
UTF-8 bytes expressed in KiB, although its output calls the unit KB. Code share
is the fraction of lines inside fences, not a fraction of tokens. Type-token
ratio measures unique words relative to word occurrences; gzip ratio measures
compressed bytes relative to original bytes.
On these files, structural signals separated the conversations; lexical diversity and byte compressibility barely moved. That is a result about this pair, not a general verdict on either technique.
The broader scan was restricted to themed web conversations and excluded
short texts. Its reported shape was smooth rather than bimodal: a continuum
between conversational back-and-forth and long content dumps. Branding and
philosophy titles appeared toward the high-cadence end; specifications,
research, and filing dumps toward the low end. That is face validity, not
independent labeling. The retained scan.py shows the selection and counting
method; its full result table isn't part of the referenced artifacts, so I
am keeping the distribution claim qualitative.
The implication matters more than a convenient label: decision-heavy and content-heavy are positions on a spectrum. Any gate that turns that spectrum into a choice is a provisional operating rule.
Calibration: predict, confirm, log the miss
The next pass compared cadence with hand-counted edges across five
conversations. This is the calibration retained in stats.py:
| conversation | turns/KB | hand-counted edges | recorded KB | edges/KB |
|---|---|---|---|---|
| branding | 2.25 | 18 | 32.9 | 0.547 |
| decision-ledger | 1.38 | 5 | 7.2 | 0.694 |
| uncertainty-arch | 0.90 | 6 | 21.1 | 0.284 |
| RLM vs Corus | 0.60 | 2 | 8.4 | 0.238 |
| technical | 0.22 | 11 | 74.0 | 0.149 |
There is a measurement wrinkle worth leaving visible: the calibration script uses the manually entered 74.0 KB for the technical conversation, whereas the signal script computes 72.3 KiB from its bytes. That is why its edge density appears as 0.149 here and 0.152 above. These are the recorded calculations, not a perfectly normalized benchmark.
The retained calibration gives Pearson r = 0.80 and Spearman rho = 0.90, with N = 5. Enough to motivate another experiment. Nowhere near enough to treat the relationship as settled.
I set provisional gates: predict decision-heavy at turns/KB ≥ 0.70; confirm that label at edges/KB ≥ 0.40. The feedback ledger is simple:
| conversation | cheap prediction | extraction label | result |
|---|---|---|---|
| branding | decision | decision | agrees |
| decision-ledger | decision | decision | agrees |
| uncertainty-arch | decision | content | miss |
| RLM vs Corus | content | content | agrees |
| technical | content | content | agrees |
That is 4/5 agreement, or 80%, on the calibration examples under hand-set thresholds. It is not held-out accuracy, and the confirmation label is itself derived from subjective edge counts. My and Claude's judgment can wobble by a couple of edges; the apparent precision of a density doesn't remove that.
The proposed loop is the useful architecture: a cheap gate predicts, the extraction I do anyway supplies a confirmation signal, and the ledger records prediction against outcome. Disagreements show where to inspect and retune. That is a feedback design demonstrated on a small calibration table, not an already self-tuning service.
What the anomalies actually teach
Read the tables together and the lessons are blunt:
- Compression needs a reconstruction question attached. The technical file wins on size partly by shedding payload. That is useful for recovering decisions and costly for recovering their complete contents.
- A pre-filter cannot depend on the expensive result. Edges/KB belongs after extraction. Turns/KB earns its place before extraction because it only needs transcript structure.
- Cadence is not substance. Decision-ledger has higher edge density than branding despite lower turn density. A few dense turns can carry more decisions than a faster exchange.
- A miss is a specification for the next experiment. Uncertainty-arch clears the cadence gate but falls below the event-density gate. The brief's diagnosis is analytical assistant exposition with relatively few discrete human decisions. Turns tell me an exchange happened, not what it settled.
The suggested next move was to combine turns/KB with average characters per turn or fenced-code share. The scripts add a useful qualification: whole-file average characters per turn largely inverts turns per byte, aside from encoding differences. It should not be assumed to supply independent information. Fenced-code share is the more distinct candidate, though it may only help on conversations whose payload is fenced.
Where it stands
This is a first phased pass: two complete hand extractions, a tiny subjective reconstruction exercise, and a five-point calibration. Token counts are estimates. Labels are judgments. Thresholds are hand-set. The broader scan is selected by topic and length. Query latency and end-to-end span retrieval remain unestablished. The directions are useful enough to guide the next test; the exact numbers are not a performance guarantee.
The harness has already changed what I would build. Graph-only memory is conditional on the question. Source pointers are part of useful reconstruction. The gate needs a cheap structural signal, and its mistakes need to remain visible after extraction.
That is a better place to start than a master plan for converting the entire archive. The instrument found where the attractive feature stops being sufficient, while the experiment was still small enough to inspect.
Next: add fenced-code share as the second gate feature, keep logging prediction against extraction, and test whether it improves the misses without breaking the cases cadence already gets right.