- Every dbt source freshness invocation writes a sources.json artifact containing max_loaded_at for each source.
- When you compare two sources.json files (previous vs. current), dbt can tell which sources have newer max_loaded_at timestamps.
- The special selector source_status:fresher+ asks dbt to:
- pick those fresher sources and every downstream model (+)
- skip everything else.
- pick those fresher sources and every downstream model (+)
1. dbt Core (CLI)
Prerequisites
- dbt v1.3+ (where state selection is stable).
- Your sources: blocks already include a freshness: section.
Commands to Run:
# Generate freshness state
dbt source freshness # writes target/sources.json
# Run only models whose sources are fresher than last prod artifacts
dbt build \
--select "source_status:fresher+" \
--state path/to/last_prod_artifacts–state tells dbt where the previous sources.json (and manifest) live; many teams copy the whole target/ folder from the last production job into an S3/GCS “artifacts” bucket.
Automating it
- End your prod job with
aws s3 cp target/ s3://my-bucket/artifacts/prod/$GIT_SHA/ –recursive - In dev/CI, set DBT_STATE=s3://my-bucket/artifacts/prod/latest/. dbt will pick it up automatically.
2. dbt Cloud (Jobs UI)
What’s different?
- dbt Cloud automatically wires the –state flag to the artifacts from the job’s last successful run, so you don’t have to manage paths yourself.
- You control whether a failing freshness check blocks downstream steps via the “Run source freshness” checkbox vs. adding dbt source freshness as an explicit step.
Job steps
| Step | Command | Notes |
| 1 | Run source freshness (checkbox on) | Produces the current sources.json; if it fails, job still proceeds. |
| 2 | dbt build –select “source_status:fresher+” | Cloud injects –state for you; only fresher sources + children will build. |
| 3 | In Advanced section, enable the “Compare changes against”and select “This Job”. | That way dbt cloud will automatically provide the sources.json file from the previous run to the state command. |
Tip: If you do want the job to stop when freshness is outside SLA, add dbt source freshness as a step 1 command, not via the checkbox – a non-zero exit code will halt the job.
Edge cases & best-practice tips
- No sources changed? source_status:fresher+ can yield an empty selection; dbt will exit 0 quickly, saving warehouse credits.
- Combining with Slim CI: You can OR selectors, e.g.
dbt build –select “state:modified+ source_status:fresher+” –state … to run code changes and fresher data together. - Materialization matters: downstream views will always query fresh data; tables or incremental models need re-builds, so using the selector prevents stale tables.
- Version control: Commit sources.json into your artifact store, not your repo. It changes every run and bloats PRs.
- Testing only: Swap dbt build for dbt test if you merely want to assert downstream model tests on fresher data.
Quick copy-paste reference
# dbt Core
dbt source freshness
dbt build --select "source_status:fresher+" --state path/to/last_prod_artifacts
# dbt Cloud job steps
# 1. (checkbox) Run source freshness
# 2. dbt build --select "source_status:fresher+"
# 3. enable the “Compare changes against - This Job”This configuration ensures models run only when new data arrives, avoiding unnecessary executions.
