Apache Spark 4.2 vs 4.1: CDC, Geospatial and Arrow-by-Default Risks

Last Updated: September 23, 2026

Apache Spark 4.2.0 shipped on 14 July 2026. The headline features are real: a SQL CHANGES clause for reading row-level changes, native GEOMETRY and GEOGRAPHY types, and Auto CDC in Spark Declarative Pipelines. But the change most likely to break your jobs is not on that list. Spark 4.2 turns on Arrow for every regular Python UDF and for all PySpark-to-JVM data exchange, and it raises the PyArrow floor to 18.0.0. So spark 4.2 vs 4.1 is really a Python-runtime migration with some new SQL features attached. It also quietly changes how shuffle retries behave and how some SQL names resolve.

This post covers both halves. Part one explains how the new CDC and geospatial features work inside the engine, including where they stop short. Part two is an upgrade gate: every documented behaviour change, the symptom it causes, and the legacy flag that rolls it back.

What this covers: the CHANGES clause and its netChanges post-processing, geospatial types and SRID rules, what Arrow-by-default does to UDF data types, shuffle checksum rollback, an industrial IoT (IIoT) telemetry example, and a tested upgrade checklist.

Context and Background

Spark 4.2 is the third release of the 4.x line. According to the official release notes, it resolved more than 1,700 Jira tickets from more than 250 contributors. Maintenance releases followed within days: 4.1.3 and 4.0.4 on 15 July and 3.5.9 on 16 July. So the 4.1 line is still supported, and teams have no reason to rush.

The baseline, Spark 4.1.0, was released on 16 December 2025. It was a large release. It added Spark Declarative Pipelines (SDP), a framework where you declare datasets and Spark handles the dependency graph, checkpoints and retries. It shipped Real-Time Mode (RTM) for Structured Streaming, with stateless Scala queries at sub-second latency. It also made SQL scripting and the VARIANT type generally available. On the Python side, 4.1 dropped Python 3.9 and raised the minimums to pandas 2.2.0 and PyArrow 15.0.0.

Spark 4.2 builds directly on those pieces. Auto CDC is a new flow type inside Declarative Pipelines. RTM gains a PySpark trigger for stateless queries without Python UDFs. The biggest structural change, though, is in Data Source V2 (DSv2), the connector API that lakehouse table formats plug into. DSv2 now has a standard change-data interface, a transaction API foundation and schema evolution on INSERT.

That matters because change feeds have always been format-specific. Delta Lake has its Change Data Feed, Iceberg has a create_changelog_view procedure, and Hudi has incremental queries. Each has its own syntax and its own quirks around copy-on-write rewrites. If you have read our breakdown of log-based change data capture with Debezium, you know that source-database CDC is a mature problem. Table-level CDC on a lakehouse is not, and Spark 4.2 is the first attempt to standardise it at the engine layer.

For query-engine context, our Trino vs Presto vs Spark comparison covers where Spark sits among lakehouse engines. This post stays inside Spark and focuses on what changes between two adjacent minor versions.

What Actually Changed in Spark 4.2 vs 4.1

Spark 4.2 vs 4.1 comes down to four things. There are two new engine features: a DSv2 CDC API with a SQL CHANGES clause, and native GEOMETRY/GEOGRAPHY types. Python now takes the Arrow path by default. And four SQL behaviours changed: shuffle checksum rollback, system namespace resolution, empty grouping sets and case-insensitive CTE names.

The table below sets out the changes that matter for production pipelines. It is not a full changelog, but every row can change what your code does or what it can do.

Area	Spark 4.1	Spark 4.2
Row-level change reads	Format-specific (Delta CDF, Iceberg changelog view)	SQL `CHANGES` clause, `spark.read.changes()`, `readStream.changes()` over DSv2 (SPARK-55948/55949/55950)
Declarative CDC	Hand-written `MERGE` in SDP flows	Auto CDC, SCD Type 1 only (SPARK-56249)
Spatial data	Binary/strings plus external libraries	`GEOMETRY(srid)` and `GEOGRAPHY(srid)` types, enabled by default (SPARK-51658, SPARK-56771)
Python UDF execution	Pickled row-at-a-time by default	Arrow-optimized by default (SPARK-54555)
PySpark to JVM exchange	Row-based unless Arrow enabled	Arrow by default (`spark.sql.execution.arrow.pyspark.enabled=true`)
Minimum PyArrow	15.0.0	18.0.0
PyPy	Supported	No longer officially supported
Shuffle retry safety	Order-independent checksum configs exist	Enabled by default, with full rollback of succeeding stages on mismatch
SQL name resolution	Current catalog first for `builtin.x` / `session.x`	`system.builtin` and `system.session` tried first
SQL PATH	Not available	`SET PATH`, `current_path()`, opt-in via `spark.sql.path.enabled`
New SQL	Recursive CTEs, KLL/Theta sketches	`QUALIFY`, `time_bucket`, `NEAREST BY`, metric views, cursors, vector functions
Streaming	RTM for Scala stateless	RTM trigger in PySpark; stable source names via `IDENTIFIED BY`; state store row checksums
JVM	Existing JDK support	Adds Java 25; K8s image base moves to `25-jre`
Bundled jars	Includes gcs-connector	gcs-connector removed from the distribution

Figure 1: The Spark 4.2 upgrade gate. Each decision maps to a documented behaviour change and its rollback flag.

Read the gate top to bottom. The first three checks are about the Python runtime and fail loudly at import or first UDF call. The fourth, shuffle checksums, fails only under executor loss, so it is the one most likely to reach production untested. The last check covers SQL name resolution, where the danger is a changed result with no error at all.

The feature additions are opt-in; the defaults are not

Everything new in 4.2 is additive. CHANGES only works on connectors that implement it. Geospatial types only appear if you declare them. QUALIFY, time_bucket and metric views are new syntax that old queries never use.

The risk sits entirely in changed defaults. Several configuration defaults flipped (three Arrow flags and the two shuffle-checksum flags), and several analyzer rules now resolve names differently. None of those require code changes to trigger. A job that ran on 4.1.3 on Monday can behave differently on 4.2.0 on Tuesday with the same jar and the same Python wheel set. So the useful question is not what you can adopt. It is what changed underneath you, and whether you noticed.

Why the Python change dominates

For most shops, Python UDFs are the widest surface in the Spark estate. A mid-sized IIoT platform might have hundreds of small UDFs that parse vendor-specific payloads, apply calibration curves or decode bit-packed status words. In 4.1 those ran through pickle serialisation, one row at a time. In 4.2 they run through Arrow record batches by default.

Arrow is columnar and generally faster, and the Databricks engineering post describes it as a path existing UDFs can use “without a code rewrite”. That is true for well-typed UDFs. It is not true for UDFs that return loosely typed values, rely on pickle-specific Python types, or run under PyPy. Those are the UDFs that tend to live in telemetry decoders.

Why the SQL changes are sneakier

The SQL migration items are individually small. Taken together, they share a pattern: most change results rather than raise errors. An empty GROUPING SETS (()) over empty input now returns one grand-total row instead of none. A NATURAL JOIN can pick different join columns when names differ only in case. A SQL UDF parameter named current_date now resolves to the built-in function. Each can shift a dashboard number without a stack trace, which is why the gate ends in a row-count and aggregate diff rather than a simple pass/fail run.

How the CHANGES Clause and netChanges Work

The CDC feature is the most architecturally interesting part of 4.2. It is also the most often misdescribed. It is not a CDC engine that tails a database log. It is a read API. A connector exposes raw change rows, and Spark applies standard post-processing on top.

The connector contract

A catalog opts in by overriding TableCatalog.loadChangelog(ident, context, options). The default implementation throws an UnsupportedOperationException saying the catalog does not support CDC. The returned Changelog object must include three metadata columns:

_change_type: one of insert, delete, update_preimage or update_postimage.
_commit_version: a LONG or STRING whose natural order matches commit order.
_commit_timestamp: the commit’s timestamp, shared by every row in that commit.

The connector also declares three capability flags. Spark reads these to decide which post-processing passes to run:

containsCarryoverRows(): the raw data may contain identical insert/delete pairs from copy-on-write file rewrites.
representsUpdateAsDeleteAndInsert(): updates arrive as delete+insert pairs rather than materialised pre/post images.
containsIntermediateChanges(): the same row can change several times within the requested range.

A connector that sets any of these flags to true must also implement rowId(), and one that declares carry-overs or delete+insert updates must implement rowVersion() as well. Those name the columns that identify a row and record the commit that last changed its content.

Figure 2: The CDC read path in Spark 4.2. The parser builds a ChangelogContext, the catalog returns a Changelog with capability flags, and Spark runs only the post-processing passes the flags require.

The sequence shows why this design is sound. The connector does the cheap, format-specific part: finding which files changed between two versions. Spark does the error-prone, format-neutral part: deciding which rows are real changes. Before 4.2, every format reimplemented that second step, and each got the edge cases slightly differently.

The three deduplication modes

The query controls post-processing through two options. deduplicationMode takes one of three values from ChangelogContext.DeduplicationMode:

none: raw rows as the connector produced them.
dropCarryovers: remove identical insert/delete pairs created by copy-on-write rewrites. This is the default.
netChanges: collapse all changes to a row identity across the range into one net effect.

The second option, computeUpdates, defaults to false. When it is true and the connector represents updates as delete+insert pairs, Spark relabels matching pairs as update_preimage and update_postimage.

The ResolveChangelogTable analyzer rule turns those options into a plan. For carry-over removal, Spark adds a window over (rowId, _commit_version) that counts deletes and inserts and tracks the minimum and maximum rowVersion. Equal version bounds within a delete/insert pair mean the row was rewritten, not changed, so the pair is filtered out. Update detection is a relabelling projection over the same window.

One combination is rejected outright. If you ask for computeUpdates with deduplicationMode = none on a connector that surfaces carry-overs, analysis fails. Otherwise every untouched row in a rewritten file would be reported as an update. That is exactly the bug that makes hand-rolled Delta CDF consumers noisy after an OPTIMIZE.

Syntax, as documented

The grammar at the v4.2.0 tag accepts version or timestamp bounds, each optionally INCLUSIVE or EXCLUSIVE, with options in a WITH clause:

-- Batch: net effect of versions 10..20 on a CDC-capable DSv2 table
SELECT asset_id, _change_type, _commit_version
FROM plant.asset_registry
  CHANGES FROM VERSION 10 TO VERSION 20
  WITH ('deduplicationMode' = 'netChanges', 'computeUpdates' = 'true') AS c;

-- Streaming: open-ended, starting point optional
CREATE STREAMING TABLE registry_changes AS
SELECT * FROM STREAM plant.asset_registry CHANGES FROM VERSION 0;

The PySpark equivalent uses reader options. The documented examples use startingVersion and endingVersion:

# PySpark 4.2 — only for DSv2 tables whose catalog implements loadChangelog()
batch = (spark.read
         .option("startingVersion", "10")
         .option("endingVersion", "20")
         .option("deduplicationMode", "netChanges")   # check your connector
         .changes("plant.asset_registry"))

stream = (spark.readStream
          .option("startingVersion", "10")
          .changes("plant.asset_registry"))

The table names here are illustrative. The option key deduplicationMode matches the name the v4.2.0 parser reads, but confirm it against your connector’s documentation. Some connectors may add their own options on top.

Streaming semantics you must know

Batch CDC is straightforward. Streaming CDC with post-processing has three behaviours that the Javadoc spells out and most blog summaries skip.

First, Spark uses _commit_timestamp as event time with a zero-delay watermark. A commit’s rows are buffered in state and emitted only in a later micro-batch, when the watermark moves past that commit. The last commit’s output is emitted when the source terminates. So a streaming CHANGES read with post-processing always runs at least one micro-batch behind.

Second, streaming netChanges only merges changes that are buffered together. If each row identity appears in at most one commit per buffered window, the output equals computeUpdates output. For a true full-range collapse, the documentation says to use a batch read.

Third, the connector contract is strict. All rows of one commit must land in the same micro-batch, and each micro-batch must have strictly later timestamps than every earlier one. Rows that arrive late are silently dropped by the watermark filter. A null _commit_timestamp raises CHANGELOG_CONTRACT_VIOLATION.NULL_COMMIT_TIMESTAMP. Atomic-commit formats that stamp wall-clock commit time meet this contract naturally. A custom connector over an eventually consistent store may not.

Pushdown is deliberately limited

When any post-processing pass applies, Spark only pushes down predicates on _commit_version, _commit_timestamp and the rowId() columns. A filter on _change_type or an ordinary data column stays above the scan. The reason is correctness: pushing it down could drop one half of a delete/insert pair and break carry-over detection.

The practical consequence is cost. WHERE site = 'PUNE' on a CDC read will not prune files the way it would on a normal scan. If you need selective change reads, make the selective column part of the row identity, or filter on commit bounds first.

Which formats actually support it

This is where most coverage overreaches, so here is exactly what the connector projects’ own sources say as of this writing.

Delta Lake 4.3.0 (June 2026) is built on Spark 4.1.0 and 4.0.1, not 4.2. Its experimental Kernel-based DSv2 connector adds catalog-driven batch CDC using SELECT … CHANGES FROM VERSION/TIMESTAMP, behind the flag spark.databricks.delta.changelogV2.enabled. That is batch-only, experimental and flag-gated.

Apache Iceberg has an open feature request, issue #17949, to integrate with Spark’s CDC API. Nothing has shipped yet. Iceberg’s own create_changelog_view procedure is still the supported path. If you are planning around Iceberg, our Iceberg v3 spec and upgrade guide covers what the format itself offers today.

So the honest position is this: the API is in Spark 4.2, and the ecosystem is catching up. Treat CHANGES as something to design towards, not something to migrate onto this quarter.

Auto CDC in Declarative Pipelines

Auto CDC is the declarative consumer of a change feed. It applies changes to a target table as SCD Type 1, meaning the target always holds the latest value per key with no history. SCD Type 2 is not in 4.2. The Python signature at the v4.2.0 tag accepts only 1 for stored_as_scd_type.

from pyspark import pipelines as dp

dp.create_streaming_table("asset_dim")

dp.create_auto_cdc_flow(
    target="asset_dim",
    source="registry_changes",          # a CDC source defined in the pipeline
    keys=["asset_id"],
    sequence_by="_commit_version",
    apply_as_deletes="_change_type = 'delete'",
    except_column_list=["_change_type", "_commit_timestamp"],
    stored_as_scd_type=1,
)

This example is illustrative, but every parameter name comes from the 4.2.0 API. The sequence_by expression orders events so that out-of-order changes resolve correctly. apply_as_deletes marks which events remove a key. Under the hood the flow writes through a new streaming sink, Scd1MergeStreamingWrite (SPARK-56957).

Figure 3: An asset-registry change stream feeding an SCD Type 1 dimension through Auto CDC. The watermark buffering step is where streaming CDC latency comes from.

Two constraints from the docstring deserve emphasis. First, keys must be unqualified column identifiers. Second, the key set is part of the flow’s persisted state. Renaming, adding, removing or retyping a key column across runs is unsupported and “will produce undefined behavior”. The only safe way to change keys is a full refresh of the target. For an asset registry keyed on asset_id, that is fine. For a table where the business key is still under debate, wait.

Geospatial Types and SRID Handling

Native geospatial support arrived through SPIP SPARK-51658 and is enabled by default. Spark follows the OGC Simple Feature Access model. At runtime every value is Well-Known Binary (WKB) plus a Spatial Reference Identifier (SRID), the code that names its coordinate reference system.

Two types, two coordinate models

GEOMETRY is planar. It suits projected or local coordinates, such as a plant floor grid in metres, UTM zones or Web Mercator (SRID 3857). It accepts any SRID in Spark’s registry, including SRID 0 for “unspecified”.

GEOGRAPHY is spherical. It uses longitude/latitude in degrees, edge interpolation is always spherical, and only geographic SRIDs are accepted. The common case is 4326 (WGS 84), which Spark maps to OGC:CRS84 with longitude-first axis order. Spark will cast GEOGRAPHY to GEOMETRY (SPARK-55539); the release notes list only that direction, so treat the reverse as an explicit decision about coordinates.

The documentation gives a concrete example of getting this wrong. On a sphere, the shortest path from London to New York crosses Canada. A planar geometry suggests a path that does not. For a haul truck on a 2 km mine site the difference is negligible. For a maritime fleet it is not.

SRIDs are part of the column type

In SQL you cannot declare a bare GEOMETRY column. You must write either a fixed SRID or ANY:

CREATE TABLE fleet.asset_positions (
  asset_id   STRING,
  ts         TIMESTAMP,
  pos        GEOGRAPHY(4326),     -- fixed SRID, persistable
  floor_xy   GEOMETRY(0)          -- local plant grid, SRID 0
) USING parquet;

The constructor functions have asymmetric defaults, and that asymmetry is the most common trap. ST_GeogFromWKB(wkb) always returns SRID 4326. ST_GeomFromWKB(wkb) returns SRID 0 unless you pass the second argument (SPARK-55295). Insert an SRID-0 geometry into a GEOMETRY(4326) column and Spark raises GEO_ENCODER_SRID_MISMATCH_ERROR. Use ST_SetSrid to relabel it, but only when you are sure the coordinates really are in that system. ST_SetSrid changes the label, not the numbers.

Mixed-SRID columns (GEOMETRY(ANY), GEOGRAPHY(ANY)) exist for in-memory work. Parquet, Delta and Iceberg all require one SRID per column, so the ANY variants cannot be persisted. An unknown SRID, or a projected SRID on a GEOGRAPHY column, fails with ST_INVALID_SRID_VALUE.

The SRID registry is pinned, not live

Spark ships a prebuilt registry built from the PROJ database, with OGC overrides for 4326, 4267 and 4269. It is pinned per release and never synced with external databases. The 4.2.0 documentation lists PROJ 9.8.1 for Spark 4.2.0, while the Jira title for SPARK-55790 refers to PROJ 9.7.1 data. Either way, if your plant uses a very recent national grid definition, check that its code exists before you standardise on it.

WKB parsing is strict

Spark validates WKB on parse. Line strings and polygon rings must have finite coordinates. Polygon rings must be closed and have at least four points. Infinity is always rejected, and NaN is allowed only to represent an empty point. When parsing as GEOGRAPHY, longitude must lie in [-180, 180] and latitude in [-90, 90].

For IIoT data this is a feature, not a nuisance. GPS modules produce garbage fixes on cold start, in tunnels and under crane booms. Previously those passed through as opaque binary. In 4.2 they fail at parse time, so you need a quarantine path rather than a job failure.

What is missing: spatial predicates

This is the gap most release write-ups miss. The 4.2.0 built-in function reference lists five geospatial functions: ST_AsBinary, ST_GeomFromWKB, ST_GeogFromWKB, ST_Srid and ST_SetSrid. There is no built-in ST_Contains, ST_Intersects or ST_Distance. WKT writing, Parquet read/write and Thrift server result sets are supported, but relationship and measurement functions are not.

So Spark 4.2 gives you a correct, portable storage type with SRID discipline. It does not replace Apache Sedona or an equivalent library for geofencing and proximity. The realistic 4.2 architecture stores positions as native types and runs predicates through Sedona or a UDF until the function surface grows.

An IIoT Telemetry Example: Mobile Assets on a Plant Site

Consider an illustrative site with 400 tracked mobile assets: forklifts, automated guided vehicles and yard tractors. Each gateway sends a position every five seconds as a WKB point, plus battery and load telemetry. That is about 80 position events per second, or roughly 6.9 million per day. The volume is modest, which makes it a clean test of correctness rather than scale.

Figure 4: An IIoT position pipeline on Spark 4.2. Native types handle parsing and SRID enforcement; spatial predicates still route through Sedona or a UDF.

Outdoor assets report WGS 84 coordinates and go through ST_GeogFromWKB, which stamps SRID 4326 automatically. Indoor AGVs report positions on the plant’s local grid in metres. Those go through ST_GeomFromWKB(wkb, srid), or stay at SRID 0 in a separate GEOMETRY(0) column. The SRID check in Figure 4 is where Spark 4.2 earns its keep. A vendor firmware update that silently switches coordinate systems now fails at write time instead of corrupting a month of heat maps.

Rollups use the new time_bucket(bucketSize, ts[, origin]) function (SPARK-56594):

-- Illustrative: 5-minute activity per asset
SELECT asset_id,
       time_bucket(INTERVAL 5 MINUTES, ts) AS bucket,
       count(*)                            AS fixes,
       max(ts)                             AS last_seen
FROM fleet.asset_positions
GROUP BY asset_id, time_bucket(INTERVAL 5 MINUTES, ts)
QUALIFY row_number() OVER (PARTITION BY asset_id ORDER BY max(ts) DESC) <= 12;

Note one subtlety from the function reference. For TIMESTAMP values, year-month buckets and calendar-day parts of day-time buckets align to the session time zone. For TIMESTAMP_NTZ, bucketing is done in UTC. A multi-site operator running one Spark session across plants in different time zones should bucket on TIMESTAMP_NTZ or set the session zone explicitly.

The asset registry is a different data shape: slow-changing master data. It holds which forklift belongs to which cell, its rated load and its maintenance state. That is the Auto CDC case from Figure 3. Once your catalog implements loadChangelog, the registry’s CHANGES stream feeds an SCD Type 1 dimension, and the position stream joins against it. Until then, the same design runs on your format’s native change feed, and you swap the source later.

The digital twin then renders current positions, five-minute activity and geofence breaches. Only the last of those needs a spatial predicate, and it is the part that still sits outside core Spark. For streaming latency, the trade-offs in our Flink vs Spark Streaming vs Kafka Streams comparison still apply. RTM in PySpark helps only for stateless queries without Python UDFs, which rules out most decoders.

What Breaks When You Upgrade: The Spark 4.2 Migration Guide, Decoded

The Spark 4.2 migration guide lists each change in a sentence or two. The table below adds the symptom you will actually see and the rollback. Every flag here comes from the official SQL and PySpark migration guides.

Change in 4.2	Symptom on upgrade	Fix or legacy flag
PyArrow minimum 15.0.0 → 18.0.0	Import or session start fails on older images	Rebuild images with PyArrow ≥ 18.0.0
Arrow PySpark exchange on by default	`toPandas()`/`createDataFrame()` dtype or schema differences	`spark.sql.execution.arrow.pyspark.enabled=false`
Regular Python UDFs Arrow-optimized	Return-type coercion differs; wrong-typed returns can error instead of becoming null	`spark.sql.execution.pythonUDF.arrow.enabled=false`
Python UDTFs Arrow-optimized	Same coercion differences for table functions	`spark.sql.execution.pythonUDTF.arrow.enabled=false`
PyPy no longer supported	Unsupported runtime	Move to CPython
`createDataFrame` from NumPy needs PyArrow	Schema now follows Arrow type mapping	Review inferred schema or pass one explicitly
Pandas UDF nullable ints arrive as `Int8`–`Int64`	`float64` assumptions break: `np.isnan`, float arithmetic, `astype(int)`	Update UDF code; no legacy flag documented
pandas-on-Spark `drop` raises `KeyError` on any missing label	Jobs that tolerated partial drops fail	Pass `errors="ignore"` or filter labels
Python Data Source type mismatch	`DATA_SOURCE_RETURN_SCHEMA_MISMATCH`	Return data matching the declared schema
`SimpleDataSourceStreamReader` offset must advance	`SIMPLE_STREAM_READER_OFFSET_DID_NOT_ADVANCE`	Advance the end offset past the last record
Order-independent shuffle checksums on	Stage rollbacks, longer runtimes, or job failure when rollback is impossible	`spark.sql.shuffle.orderIndependentChecksum.enabled=false` and `...enableFullRetryOnMismatch=false`
`system.builtin` / `system.session` namespaces	`builtin.f()` or `session.f()` no longer reaches a schema of that name when the system namespace has a same-named object	`spark.sql.legacy.persistentCatalogFirst=true`, or qualify the catalog
Duplicate CTE names checked case-insensitively	`DUPLICATED_CTE_NAMES` parse error	Rename CTEs
`NATURAL JOIN` honours `spark.sql.caseSensitive`	Different join columns, different results	Use explicit `USING` or `ON`
SQL UDF parameter shadowing parameterless built-ins	Parameter named `current_date` etc. resolves to built-in	Rename, or `spark.sql.legacy.allowUdfParameterToShadowParameterlessFunction=true`
`SET CATALOG name` checks session variables first	Wrong catalog if a variable shares the name	Use `SET CATALOG 'name'`
`Observation.get` raises underlying error	Exceptions where code expected an empty result	Add error handling
Empty grouping set returns grand total	One row instead of zero on empty input	`spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled=false`
`CustomTaskMetric.mergeWith` sums by default	Wrong UI metrics for non-additive connector metrics	Connector authors override `mergeWith`
Derby JDBC data source deprecated	Deprecation warnings	Plan migration off Derby

Two more changes are not in the guide but surface in operations. The gcs-connector is gone from the bundled jars, so Google Cloud Storage jobs that relied on it being present need it added explicitly. And the Kubernetes image base moves to Java 25, which matters if you layer native agents or JVM flags onto the stock image.

Arrow-by-default: what happens to your UDF data types

Arrow’s type system is stricter than pickle’s, and that causes most Python surprises. With pickled UDFs, Python returned whatever it liked. If the value did not match the declared return type, Spark usually produced a null. On the Arrow path, return values are converted into a typed Arrow array. Since 4.1, spark.sql.execution.pandas.convertToArrowArraySafely defaults to true, so unsafe conversions such as integer overflow or float truncation raise errors instead of wrapping silently.

The practical effect on a telemetry decoder is direct. Say a UDF is declared to return IntegerType but sometimes returns 42.0 from a scaling step. Under pickle that value might have become null, and nobody noticed. Under Arrow it may be coerced or raise, depending on the conversion. Either outcome differs from 4.1. The 4.1 migration guide already warned that coercion changes when output does not match the schema. 4.2 simply makes that path the default for everyone.

Pandas UDFs have a separate, better-documented change. Pandas UDFs always used Arrow. What changed in 4.2 is the pandas dtype for nullable integer columns that contain nulls in a batch. They used to arrive as float64, with NaN for nulls. They now arrive as pandas extension dtypes Int8, Int16, Int32 or Int64, with pd.NA for nulls.

# Illustrative decoder that breaks on 4.2
@pandas_udf("double")
def load_pct(raw_load: pd.Series) -> pd.Series:
    # 4.1: raw_load is float64 with NaN for missing readings
    # 4.2: raw_load is Int32 with pd.NA; np.isnan() yields <NA>, so np.where raises TypeError
    return pd.Series(np.where(np.isnan(raw_load), 0.0, raw_load / 250.0))

# 4.2-safe version
@pandas_udf("double")
def load_pct(raw_load: pd.Series) -> pd.Series:
    return (raw_load.astype("Float64") / 250.0).fillna(0.0).astype("float64")

The failure is data-dependent, which makes it dangerous. The dtype changes only when a batch actually contains a null. A test fixture with clean data passes. Production fails on the first shift where a sensor drops out. Build test fixtures with nulls in every nullable integer column.

Shuffle checksum rollback, in plain terms

A shuffle stage is indeterminate when rerunning it can produce different output for the same partition. Round-robin repartitioning, rand()-based logic and some non-deterministic UDFs are typical causes. If an executor dies after downstream stages have already read part of the shuffle, Spark reruns the lost map tasks. If the rerun differs, downstream stages have mixed old and new data. That is a silent correctness bug.

The order-independent checksum is computed per map output over the set of rows, so row order does not matter. It is separate from the older spark.shuffle.checksum.enabled, which detects file corruption and is order-sensitive. When a retry produces a different checksum, Spark 4.2 rolls back and re-executes every succeeding stage that depends on that output (SPARK-54556). If rollback is not possible for some stage, the job fails.

The configuration keys were introduced in 4.1.0. The migration guide identifies 4.2 as the release where they are on by default. The operational impact is threefold. There is a small, always-on cost to compute checksums. Retries are more expensive, because a mismatch triggers wider re-execution. And some jobs that previously “succeeded” will now fail. That last group is the point: those jobs may have been producing inconsistent results all along.

For IIoT platforms on spot or preemptible nodes, executor loss is routine, so this matters more than average. The recommendation is to keep checksums on and watch stage retry counts and job failure rates on the canary. Then fix the source of indeterminism, for example by replacing round-robin repartition(n) with hash partitioning on a stable key. Turning checksums off restores 4.1 behaviour, including its risk.

Name resolution changes that alter results quietly

The system.builtin and system.session namespaces are a security and clarity improvement. system.builtin.lower(x) gives an unambiguous way to reach the built-in lower, whatever user functions exist. The side effect lands on anyone with a persistent schema literally named builtin or session. Two-part references now try the system namespace first. The fix is to reach such schemas with an explicit catalog prefix, for example spark_catalog.session.x.

SET PATH is opt-in, and spark.sql.path.enabled defaults to false. With it off, unqualified names resolve against a fixed default path, and SET PATH is rejected with UNSUPPORTED_FEATURE.SET_PATH_WHEN_DISABLED. Nothing changes unless you turn it on. Once you do, views and SQL functions persist the path they were created with. That makes resolution predictable, but it also makes path changes part of your deployment surface.

Trade-offs, Gotchas, and What Goes Wrong

CDC is an API, not a feature you can use today on every format. The engine-side post-processing is solid and well specified. But with Delta’s support experimental and flag-gated, and Iceberg’s still an open issue, most production estates cannot run CHANGES against their main tables yet. Design new consumers around the _change_type / _commit_version / _commit_timestamp contract, so you can switch sources later without rewriting consumers.

Streaming netChanges is not batch netChanges. Streaming only collapses changes buffered together within the watermark window. A consumer that expects exactly one row per key per run will see duplicates. If you need a guaranteed full-range collapse, schedule a batch read.

Streaming CDC adds at least one micro-batch of latency. Because output is held until the watermark passes the commit, a low-latency dashboard fed by STREAM … CHANGES will lag by one trigger interval or more. That is fine for a registry dimension and wrong for alarms.

Filters on CDC reads are expensive. With post-processing on, predicates on ordinary data columns are not pushed down. A narrow change query can therefore scan every changed file in the range.

Geospatial types without predicates. You gain SRID safety and portable Parquet storage. You still need Sedona or UDFs for containment and distance. And a Python UDF over geometry now runs on the Arrow path by default, so test its type conversions too.

ST_SetSrid is a label, not a transform. Relabelling local-grid metres as 4326 produces points in the ocean. Spark has no built-in reprojection in 4.2.

Legacy flags are a bridge, not a destination. Each flag you set to restore 4.1 behaviour is a divergence from upstream defaults. Future releases may remove them. Track them in a single configuration file with an owner and an expiry date.

Auto CDC key changes are one-way. Changing the key set of an Auto CDC flow needs a full refresh. On a large target, plan that refresh window before you pick keys.

Removed jars fail late. A missing gcs-connector fails at first GCS access, not at submit time. Batch jobs that touch GCS only in a final export step can run for an hour before failing.

Practical Recommendations

Treat 4.2 as two projects on different schedules. The Python runtime migration is mandatory, touches every PySpark job and should be done first on a canary cluster. Feature adoption, meaning CDC, geospatial types and the new SQL, is optional and depends on connector support you do not control.

Do not blanket-disable the new defaults. Setting all three Arrow flags to false, plus the checksum flags, makes 4.2 behave like 4.1 and hides every problem the upgrade was meant to surface. Instead, run the canary with defaults on, collect failures, fix the code, and use legacy flags only per job, with a ticket attached.

For IIoT teams, the geospatial types are worth adopting early for storage even without predicates. Enforcing SRIDs at the column level catches firmware and integration errors that otherwise surface weeks later as wrong heat maps. Keep Sedona for predicates, and plan to retire it only when the built-in function list grows.

Upgrade checklist

Inventory every image and wheel set; confirm PyArrow ≥ 18.0.0, CPython only, pandas ≥ 2.2.0.
Grep pandas UDFs for np.isnan, float arithmetic on integer inputs and astype(int); add null-bearing integer fixtures.
Run all Python UDF tests with Arrow on; list any that need spark.sql.execution.pythonUDF.arrow.enabled=false and file fixes.
Search SQL for schemas named builtin or session, duplicate-case CTE names, NATURAL JOIN, empty grouping sets and SQL UDF parameters named after parameterless built-ins.
Add gcs-connector explicitly if you use GCS.
Canary the heaviest indeterminate-shuffle jobs on spot capacity; record stage retries and failures with checksums on.
Diff row counts and key aggregates between 4.1.3 and 4.2.0 runs on the same inputs.
Keep a single, owned list of legacy flags with removal dates.
For CDC and geospatial types, prototype on a non-critical table, and confirm your connector version supports what you need.

Frequently Asked Questions

What is the main difference between Spark 4.2 and 4.1?

Spark 4.2 adds a standard change data capture API with a SQL CHANGES clause and native GEOMETRY/GEOGRAPHY types. It also brings Auto CDC for SCD Type 1 in Declarative Pipelines, plus new SQL such as QUALIFY and time_bucket. The biggest operational difference is in the defaults. Python UDFs and PySpark data exchange use Arrow by default, PyArrow 18.0.0 is the new minimum, and order-independent shuffle checksums are on. Spark 4.1 introduced Declarative Pipelines, Real-Time Mode, and GA for VARIANT and SQL scripting.

Does the Spark 4.2 CHANGES clause work with Delta Lake and Iceberg?

Only partly, and not by default. CHANGES works on DSv2 tables whose catalog implements TableCatalog.loadChangelog(). Delta Lake 4.3.0 is built on Spark 4.1 and 4.0. Its experimental Kernel-based DSv2 connector supports batch CHANGES behind spark.databricks.delta.changelogV2.enabled. Iceberg has an open feature request to integrate with Spark’s CDC API, and its create_changelog_view procedure remains the supported route. Check your connector’s release notes before you rely on it.

What does netChanges do in Spark CDC?

netChanges is a deduplication mode that collapses all changes to one row identity within the requested range into one net effect. For example, an insert followed by two updates becomes a single insert with the final values. Spark does this only when the connector declares that intermediate changes may exist and implements rowId(). In batch reads it covers the whole range. In streaming reads it only merges changes that are buffered together before the watermark advances, so it may not fully collapse.

Will my PySpark UDFs break when upgrading to Spark 4.2?

Many will run unchanged, but three groups are at risk. Regular Python UDFs now run on the Arrow path, so return values that do not match the declared type may be coerced or raise instead of becoming null. Pandas UDFs receive nullable integer columns as Int8–Int64 extension dtypes instead of float64 when a batch contains nulls. And PyPy is no longer supported. Rollback flags exist for the Arrow defaults, but the pandas dtype change needs code fixes.

How do SRIDs work with Spark GEOMETRY and GEOGRAPHY types?

Every value carries an SRID, and SQL columns must declare a fixed SRID or ANY. GEOMETRY accepts any SRID in Spark’s PROJ-based registry, including 0 for unspecified. GEOGRAPHY accepts only geographic SRIDs, usually 4326. ST_GeomFromWKB(wkb) defaults to SRID 0, while ST_GeogFromWKB always uses 4326. Inserting a mismatched SRID raises GEO_ENCODER_SRID_MISMATCH_ERROR. Parquet, Delta and Iceberg need one SRID per column, so ANY columns cannot be persisted.

Should I disable shuffle checksums after upgrading to Spark 4.2?

Usually not. The checksum detects when retrying an indeterminate shuffle stage produces different data, which would otherwise silently corrupt results. On a mismatch, Spark rolls back and reruns dependent stages, and fails the job if it cannot. That can make some jobs slower or fail. But those jobs were exposed to inconsistent output on 4.1. Fix the source of indeterminism instead. Use both spark.sql.shuffle.orderIndependentChecksum.* flags only as a short-term bridge.

References

Apache Spark, Spark Release 4.2.0 and Spark 4.2.0 released.
Apache Spark, Migration Guide: SQL, Datasets and DataFrame, section “Upgrading from Spark SQL 4.1 to 4.2”.
Apache Spark, Upgrading PySpark, section “Upgrading from PySpark 4.1 to 4.2”.
Apache Spark, Geospatial (Geometry/Geography) Types.
Apache Spark Javadoc, Changelog interface.
Apache Spark, Spark Release 4.1.0.
Delta Lake, Release 4.3.0.
Apache Iceberg, Issue #17949: Iceberg integration with Spark’s CDC.
Databricks, Introducing Apache Spark 4.2.

By Riju — about