Why “fresher” selection works

1. Every dbt source freshness invocation writes a sources.json artifact containing max_loaded_at for each source.
   
2. When you compare two sources.json files (previous vs. current), dbt can tell which sources have newer max_loaded_at timestamps.
   
3. The special selector source_status:fresher+ asks dbt to:
   
   • pick those fresher sources and every downstream model (+)
     
   • skip everything else.
     

1. dbt Core (CLI)

Prerequisites

• dbt v1.3+ (where state selection is stable).
  
• Your sources: blocks already include a freshness: section.
  

Commands to Run:

# Generate freshness state

dbt source freshness                    # writes target/sources.json

# Run only models whose sources are fresher than last prod artifacts

dbt build \

  --select "source_status:fresher+" \

  --state path/to/last_prod_artifacts

–state tells dbt where the previous sources.json (and manifest) live; many teams copy the whole target/ folder from the last production job into an S3/GCS “artifacts” bucket.

Automating it

1. End your prod job with
   aws s3 cp target/ s3://my-bucket/artifacts/prod/$GIT_SHA/ –recursive
   
2. In dev/CI, set DBT_STATE=s3://my-bucket/artifacts/prod/latest/. dbt will pick it up automatically.
   

2. dbt Cloud (Jobs UI)

What’s different?

• dbt Cloud automatically wires the –state flag to the artifacts from the job’s last successful run, so you don’t have to manage paths yourself.
  
• You control whether a failing freshness check blocks downstream steps via the “Run source freshness” checkbox vs. adding dbt source freshness as an explicit step.
  

Job steps

Tip: If you do want the job to stop when freshness is outside SLA, add dbt source freshness as a step 1 command, not via the checkbox – a non-zero exit code will halt the job.

Edge cases & best-practice tips

• No sources changed? source_status:fresher+ can yield an empty selection; dbt will exit 0 quickly, saving warehouse credits.
  
• Combining with Slim CI: You can OR selectors, e.g.
  dbt build –select “state:modified+ source_status:fresher+” –state … to run code changes and fresher data together.
  
• Materialization matters: downstream views will always query fresh data; tables or incremental models need re-builds, so using the selector prevents stale tables.
  
• Version control: Commit sources.json into your artifact store, not your repo. It changes every run and bloats PRs.
  
• Testing only: Swap dbt build for dbt test if you merely want to assert downstream model tests on fresher data.
  

Quick copy-paste reference

# dbt Core

dbt source freshness

dbt build --select "source_status:fresher+" --state path/to/last_prod_artifacts

# dbt Cloud job steps

# 1. (checkbox) Run source freshness

# 2. dbt build --select "source_status:fresher+"

# 3. enable the “Compare changes against - This Job”

This configuration ensures models run only when new data arrives, avoiding unnecessary executions.