When using Auto Loader (`cloudFiles`) to ingest files incrementally, you must specify a checkpoint location. This location stores the metadata of files that have already been processed. If this is omitted, Auto Loader has no way to remember processed files and will scan the entire source directory from scratch on each pipeline run, leading to full reprocessing.

The silver layer focuses on data quality and reliability. Applying `NOT NULL` and `CHECK` constraints ensures that the data meets business rules and integrity requirements before it’s consumed by the gold layer or other users. While optimization (A & D) is important, it’s often more relevant in the gold/consumption layer. Change Data Feed (C) is useful for specific incremental processing patterns but is not the primary tool for enforcing core data quality in the silver layer.

Z-ordering (or multi-dimensional clustering) co-locates related data (rows with similar `customer_id` values) within the physical files of each partition. When a query filters on `customer_id`, Databricks can skip reading entire files that don’t contain the relevant values, dramatically reducing I/O. A Bloom filter (C) can also help but is generally less efficient for high-cardinality columns like IDs and is not as commonly the first-choice optimization for this specific scenario compared to Z-ordering.

`OPTIMIZE` is the correct command to rewrite small files into larger, more efficient ones, which is the root cause of the slow dashboard queries. Scheduling it during off-peak hours minimizes impact. `VACUUM` (C) removes old files but does not compact small active files. Auto-optimization (D) works on *writes*, but for existing tables with already-fragmented data, a manual `OPTIMIZE` is needed. Increasing compute (B) treats the symptom, not the cause.

Deduplication in streaming requires storing data to identify duplicates. If you compare each batch against *all* previously seen data (e.g., the entire table), the state grows indefinitely. The solution is to define a watermark on an event-time column, which allows Spark to automatically drop old state data, limiting state size and preventing unbounded growth. This is a classic streaming state management issue.

Change Data Feed (CDF) records row-level changes (inserts, updates, deletes) made to a Delta table. When enabled, downstream jobs can read these change data feeds using `table_changes` syntax to incrementally process only the new or modified data since their last run, which is far more efficient than reprocessing the entire table.

Delta Lake’s `RESTORE TABLE` command is designed for this exact scenario. It allows you to quickly revert a table to an earlier version (or timestamp) using the transaction log, effectively undoing the erroneous `DELETE`. This is atomic, reliable, and leverages Delta’s built-in versioning, making it superior to manual `INSERT` or `MERGE` operations which would be complex and error-prone.

A key characteristic of `IDENTITY` columns in Delta Lake is that they are generated automatically and are unique, but they are *not* guaranteed to be contiguous. Gaps can appear due to transaction rollbacks, failed operations, or caching mechanisms. Applications relying on strictly contiguous sequences should not use `IDENTITY` columns. The other options are incorrect: you can use `OVERRIDING SYSTEM VALUE` (C), and you can alter sequence parameters (D).

A stream-static join is the standard and efficient pattern for joining a streaming DataFrame (facts) with a static DataFrame (dimension table). Spark can cache the static dimension data on each executor, allowing for fast, per-event lookups. A broadcast join (D) is a manual implementation of this idea but less direct. Stream-stream joins (A, C) are for joining two streams and are unnecessary and more complex for a static dimension.

When using Auto Loader (`cloudFiles`) to ingest files incrementally, you must specify a checkpoint location. This location stores the metadata of files that have already been processed. If this is omitted, Auto Loader has no way to remember processed files and will scan the entire source directory from scratch on each pipeline run, leading to full reprocessing.0