Operate a data product
Run a Dataset day to day: freshness per Source, match rate, disagreements between Sources, extraction through free lookups, and the fix for a wrong value.
Operating a Dataset means keeping every Source fresh, every value traceable, and free lookups from giving the Dataset away. The Data Engineer seat owns it.
What to watch
| Number | Read from | Act when |
|---|---|---|
| Age per Source | Each record's refreshedAt | A Source falls behind its refresh schedule |
| Match rate | Billed hits against free misses | Misses rise in one region or key shape: a coverage gap |
| Disagreements | Values the selection rule set aside | One Source loses more often than the others |
| Distinct keys per caller | The request log | A free key walks the key space, which reads as extraction |
| Corrections per thousand served | Corrected values | Any rise: read the Source before the next refresh |
A rising miss rate in one place is the cheapest coverage signal a Dataset gets. Read the misses before buying a new Source.
Keep it fresh
Each Source is read on the Dataset's refresh schedule, and each record keeps the time it was last read. Refreshing one Source now, or backfilling a past range, is Dataset refresh, with status planned. A wrapped Source's Integration that needs re-approval pauses its reads, and the records already read keep their refreshedAt.
Guard the free lookups
Every Dataset API starts with the caps a data business needs. They bound lookups with no key per day and free lookups per key per month, and test keys read synthetic records. List calls are paged and priced per row, so a free key cannot page through the whole Dataset.
Fix a wrong value
A value that was wrong when served is fixed with its reason and its evidence, and every Customer who read it hears about it. That is Corrections, with status planned. A value that changed in the world is a refresh, not a correction.
Nothing is purged. A removed record leaves every list and keeps its history, and Thing.restore brings it back.