Data engineer. I optimize pipelines and processes, and write about what I learn doing it. Ideas start the argument, intuition shortens it, data ends it.
From One-Off Prompts to a Spec-Driven Platform: An Agent Is a File, Not a Program
At the end of the last post I said the preamble was the first spec I ever wrote: the paragraph I kept re-typing into a browser tab to explain PRESTAGE, STAGE and the difference between a business key and a hash key before I could ask my actual question. I hadn’t written it down yet. Writing it down was the easy part. The hard part was deciding what kind of thing the written-down version should be. ...
Measuring What SELECT * Pulls: Five of Seven CTEs Used Under Half of It
An earlier audit gave me a satisfying story about SELECT * in one warehouse, and the story turned out to be too small. In the staging and business-vault layers, models joined sibling tables that carry the same metadata columns (RECORD_SOURCE, LOAD_DATETIME, EFFECTIVE_FROM), and the duplicates ended up materialized under suffixed names: RECORD_SOURCE_2, LOAD_DATETIME_2. One regex over INFORMATION_SCHEMA.COLUMNS found a single table of 165,591,873 rows carrying both. Row count times value width puts the duplicate RECORD_SOURCE at 3.3 GB and the duplicate LOAD_DATETIME at 1.3 GB uncompressed. Snowflake does not report storage per column, so their compressed cost is not measured here. A join of that shape no longer builds: as of September 2026, five variants of it, as views and as tables, from a bare SELECT * to an explicit duplicate list, all fail with duplicate column name. How the suffixes got there is in the earlier post. ...
Is RAG Even the Right Tool? Measure the Corpus Before You Build One
The last post ended on a corpus problem: five of nine real questions had no answer in the documentation at all. This one takes the corpus as given and asks the other half — how you get an answer back out of it — and argues that the architecture everybody reaches for first is the wrong one, for reasons that have nothing to do with how good the embeddings are. Before you build either one, four measurements tell you which fits, and none of them needs a vector store: what it costs to list every scope, how many scopes fit whole inside a context window, how old the pages are, and whether the catalogue says what each scope is for. The figures below are what those four returned on one corpus. Yours will differ; you can have them before you commit to an architecture. ...
When the LLM First Became Useful: Two Wins, and Why Neither Was an Answer
I was a skeptic about letting a language model near serious data engineering. A chat window in a browser tab (feels like decades back now) has never seen your data, can’t read your repo, forgets everything the moment you close it, and will cheerfully invent a column that doesn’t exist. In a Data Vault warehouse, where no bug is ever local and every wrong answer costs you a reload, that seemed like exactly the wrong tool. ...
Per-Brand Hash Keys in Data Vault 2.0: Make PSA_HK Encode Identity, Not Attributes
In standard automate_dv, a satellite’s parent key is a hash of all the candidate key columns, and its hashdiff is a hash of all the payload. That’s the right default, and it holds as long as every row of a source means the same thing. It stops holding the moment one logical entity is fed by several sources that don’t agree on what makes a row unique. The ITEM_LIFETIME entity here is exactly that: one table, fed by seven source systems, each with its own notion of a row’s identity. If you take the standard route and hash the union of every possible key column into PSA_HK, a source that doesn’t populate one of those columns hashes a NULL, or a zero-key default, into its key. Then that column gets a value one day, because a normal attribute update finally filled it in. The hash flips. And a flipped PSA_HK doesn’t look like an update to the pipeline — it looks like a new entity. The satellite opens a fresh row. The prior history stays technically valid under the old key, but the business row’s timeline is now split across two keys. ...
Retrieval Is Not Enough: Test the Corpus Before You Build the RAG
Before tuning a retrieval system, ask a cheaper question: does the corpus actually contain the answers people need? Every discussion about internal search seems to become a discussion about the retrieval stack: chunk size, embedding model, re-ranker, hybrid search. I took a step back and ran a small test. The question: are the answers in the corpus at all? It needs no infrastructure and no procurement, and what comes back can end the project before anyone has picked a vector store. ...
Finding the Minimal PSA_HK: Verify Data Vault Granularity with Data, Not Documentation
PSA_HK is the hash that says two rows are the same row. It’s computed once when data lands in the Persistent Staging Area, and from that moment every satellite version, every join, every reload is keyed off it. Get it wrong and you don’t get an error — you get a warehouse that quietly disagrees with itself, and a fix that means reloading every downstream model that ever touched the key. ...
Two Staging Models Beat One: PRESTAGE, STAGE, and the Real Cost of a Wildcard
There is a version of this argument in every data team. One side says a staging model should do one thing, so you split it: one model to clean and type the raw columns, a second to declare identity. The other side says that is two files to open, two models to build, and one extra hop for anyone tracing a column, all to enforce something a code review could enforce for free. ...