Standard
Apache Arrow is an open-source columnar in-memory data format and toolkit for fast analytics, enabling efficient data interchange across languages and systems.
A Spark benchmark's "Arrow" UDF row ran the pickle path, so its 48.9x against 48.5x tie says nothing about Arrow transport. The same benchmark's useArrow=True UDFs ran about 2x faster than the plain ones.
Reality
Spark 4.2 makes the Arrow paths PySpark's defaults, according to a dev.to history of how the format reached each JVM-to-Python boundary. Teams upgrading get that serialization without opting in, so jobs with UDFs and pandas conversions need a test run first.
Reality
Iceberg 1.11.0 ships a File Format API that sends Spark, Flink, Hive and plain-Java reads and writes through one registry of format models. A dev.to walkthrough says the file format list stays fixed, so Vortex and Lance still need a change inside Iceberg.
Reality
Cloudflare made Logpush Transformers generally available and opened Logpush to all self-serve plans. The SQL query runs inside a delivery service that Cloudflare says sends more than 110 million records per second to customer destinations.
Reality
Dell is adding a semantic layer and knowledge graph to its AI Data Platform for AI agents, with both due in the first half of 2027. Dell says agents now burn tokens on every query piecing together answers that should already exist.
Perspective Coverage
3 publishersReality
Ritchie Vink is selling 2.0 as a boring upgrade. Mostly it is, but engine="auto" now resolves to streaming by default, so joins and group_bys stop carrying incidental row order, and pipelines that leaned on it will drift quietly.
Reality
AWS cut Glue pricing 30% and shipped full Iceberg v3, but the rate follows the version string, and its own two announcement posts do not agree on which Python you will be running.
Reality
Two decisions made on public mailing lists this week change what a safe lakehouse upgrade plan looks like. Streaming writers and anyone queued behind Parquet 1.18.0 need to re-read their notes.
Reality
Interlace drops templating from its dbt-style models. Its argument: parse the SQL, and the dependency graph you were asked to declare twice is already sitting in the FROM clause.
Reality