Partizionamento pianificato, job idempotenti, schemi imposti dove entrano i dati.
Constitution — Spark data platform
Version 1.0.0 · Ratified 2026-11-10
These principles hold for every spec, plan and task of the platform. A spec may add rules; it never weakens these.
-
Partitioning is planned. Each table's partition columns are chosen in the plan from its query patterns, with a target file size (for example 128 to 512 MB). No table is partitioned by a high-cardinality column, and small files are compacted on a schedule.
-
Idempotent jobs. A job run for a given date or batch overwrites exactly its partitions (dynamic partition overwrite or a
MERGEon a key), so rerunning it never duplicates rows. -
Schemas enforced where data enters. Ingestion reads with an explicit schema, never
inferSchema, and writes to Delta or Iceberg tables with schema enforcement on. Records that do not match go to a quarantine table with the reason. -
Shuffles are deliberate. Joins with small tables use broadcast, skewed keys are handled with salting or adaptive query execution, and every job's plan is checked with
explain()before its first production run. -
Jobs are tested locally. Transformations are plain functions over DataFrames, tested with a local
SparkSessionand small fixtures in CI. A job is not scheduled until its tests pass. -
Jobs are observable and sized. Each job reports rows read and written, duration and cost to the platform metrics, and runs with explicit executor sizes. A run that reads or writes far outside its usual row count fails instead of publishing.