Sources and refresh
Where a Dataset's records come from: owned, wrapped or recruited Sources, each record a Resource in one of three modes, read on a schedule.
A Source is what a Dataset draws on but did not build. Each record it brings is a Resource, and the Dataset reads every Source again on its refresh schedule.
Three kinds of Source
| Kind | What it is | Ratebook's example |
|---|---|---|
| Owned | The Startup collects or builds it | District boundaries it draws from ordinances |
| Wrapped | Another party's feed or Product, under its terms | A licensed provider's address-to-district file |
| Recruited | Each Person paid to supply it, under a Task's terms | Reviewers who read new county ordinances |
A wrapped Source is reached through an Integration, which holds its credentials at the Startup's scope. Its terms decide what the Dataset may resell: the right to read is not the right to resell.
Three modes of a Resource
mode | Who leads | Reads |
|---|---|---|
upstream | The Source | Go through to the Source, and a mirror is kept |
mirrored | The Source, read-only to the Startup | Come from the copy, set at mirroredAt |
owned | db.sb | Come from the Startup's own record |
An upstream Resource is the freshest and the most dependent on its Source. A mirrored one is as fresh as its last read. Ratebook keeps state rates mirrored from each bulletin and its own district boundaries owned.
When Sources disagree
Two Sources can report different values for one key. A declared selection rule picks the answer, such as "the authority's own bulletin wins", and keeps the losing value. Every served value names the Source it came from, so a buyer reads which one won.
On a schedule
refresh sets how often every Source is read again: daily for Ratebook. Each read sets the record's refreshedAt, and a changed value keeps the one it replaced. Reading one Source now, or backfilling a past range, is Dataset refresh, with status planned.
Enrichment
Enrich adds fields to an existing Thing the same way a Source does: identity stays, and each new value names its origin. An enriched field is a value from a Source like any other.