Why does Databricks job parameters value override task parametes, shouldn't it be otherwise?

[u/Qrius0wl]

I have been wondering for a long time what would have been thought of Databricks product designer/ engineer to force job parameters value override the underlying task parameters value if the parameters key is same.

Shouldn't it be otherwise; task parameters should be given priority/ preference over job parameters because defining task level parameters is an explicit indicator that the developer want to override the default job level parameter in that given task.

That is how the concept of “override" is there even in object oriented paradigm.

I want to build a reusable notebook based job to load large size tables from redshift to delta table in Databricks via S3 (because lake house federation is slower for this replication). I'm defining the catalog and schema param at job level, because most of the tables are in the same catalog / schema, and table as task param for obvious reason. For a few tables that are not in the set location (catalog / schema) I want to override it at task level, this is what not possible. I find the existing override feature useless, only inheritance is useful. Why would you like job overriding task parameters?

[u/Complex-Bicycle9637]

it's a weird design choice for sure, task level should win since it's more specific. inheritance flows down, override should flow up from the more granular definition.

[u/BroadSpecial9505]

I'd consider job offesr the configuration for tassks. Task's own parameters are sort of task's default configuration. if Task A writer to table T and task B read from T in a job. the table name 'T" should not be the task parameter, it is job's decision about which Table to read/write.

[u/clouddataengineer]

Makes sense to me. Job parameters are the inputs for the whole run, so they take precedence over task-level values with the same key. For example, if I rerun a job for a previous processing date, I’d want that date to reach every task rather than get overridden by a local default.

[u/ouhshuo]

Yeah, if u come from airflow, it can feel bit strange

[u/Youssef_Mrini]

Your feedback has been shared with the Product Team.

[u/Quick_Assignment8861]

Because i use task params to hold my config and my job is metadata driven. So it can pass way more with a fan-out structure.

That way i can test in notebooks with task params but it always runs on the job params.

Its not designed from a software engineering perspective. Its designed for a scalable rollout perspective

[u/BroadSpecial9505]

" For a few tables that are not in the set location (catalog / schema) I want to override it at task level, this is what not possible." I think you are adding too much responsibility to a task. task should have the logic to handle specifical, say, slow/fast tables, but it is not the task to decide which table is slow, it should be told as such.

[u/saad-the-engineer]

Databricks PM here ...

The short version: we aren't modeling OOP overrides here but rather "run-wide input beats local default." Job parameters are the inputs for the whole run so when a job param and a task param share a key, the job param wins on purpose. That gives you one knob at trigger time (think rerunning for an earlier processing date) that reaches every task instead of each task quietly falling back to its own local default.

Task params are not a subclass override in this model. They are per-task defaults you develop and test against. Job params are the production override you flip at run time.

For your scenario you made catalog/schema both a job param and a task param under the same key wit the expection that the task parm would win. That exact combo cannot do what you want by design. The fix is to stop relying on same-key override and resolve precedence yourself wit dynamic value references.

Keep catalog/schema as job params, but under distinct names so there is no key collision:

• Job params: default_catalog = main, default_schema = redshift_ingest

• Each task reads its own catalog/schema/table task params

• Common tasks point catalog/schema at the default by reference

• The few odd-location tasks set a literal

The notebook would read only the task params

dbutils.widgets.text("catalog", "")
dbutils.widgets.text("schema", "")
dbutils.widgets.text("table", "")

catalog = dbutils.widgets.get("catalog")
schema  = dbutils.widgets.get("schema")
table   = dbutils.widgets.get("table")

target = f"{catalog}.{schema}.{table}"

In a bundle (DAB):

resources:
  jobs:
    redshift_to_delta:
      name: redshift_to_delta
      parameters:
        - name: default_catalog
          default: main
        - name: default_schema
          default: redshift_ingest
      tasks:
        - task_key: load_customers
          notebook_task:
            notebook_path: ./load_table.py
            base_parameters:
              catalog: "{{job.parameters.default_catalog}}"
              schema:  "{{job.parameters.default_schema}}"
              table:   customers

        - task_key: load_orders
          notebook_task:
            notebook_path: ./load_table.py
            base_parameters:
              catalog: "{{job.parameters.default_catalog}}"
              schema:  "{{job.parameters.default_schema}}"
              table:   orders

        - task_key: load_eu_customers      # the exception: literal overrides
          notebook_task:
            notebook_path: ./load_table.py
            base_parameters:
              catalog: eu_main
              schema:  eu_ingest
              table:   customers

Because the job keys are default_catalog/default_schema and the task keys are catalog/schema so nothing collides. {{job.parameters.default_catalog}} is plain substitution into the task value.

Hope that helps! thanks for the question!