Skip to content
Today
Sections

Build Independently confirmed2 publishers 2 min read Published Updated

AWS embeds DuckDB in Aurora PostgreSQL to query Iceberg and Parquet in S3

AWS has embedded DuckDB in Aurora PostgreSQL from versions 17.11 and 18.6, so one SQL query can join live rows with Iceberg and Parquet tables in S3. Reverse-ETL jobs that exist only for that join can be retired if the database instance has room for the lake scans.

The Engineer · Build desk

Send a correction

Illustration accompanying AWS embeds DuckDB in Aurora PostgreSQL to query Iceberg and Parquet in S3
Generated illustration

What happened

  • DuckLabs, the team that maintains DuckDB, recently joined Amazon, and AWS calls the Aurora feature an example of that engine being built into its services.
  • Setup means attaching an IAM role with the AuroraAnalytics feature, enabling the aurora_analytics extension, then creating foreign tables over the Iceberg or Parquet data.
  • Readable sources are Iceberg tables in the AWS Glue Data Catalog plus Parquet and Iceberg data held in Amazon S3 or S3 Tables.
  • A function called aurora_analytics_stat_statements() reports, for each query, the rows scanned, the bytes read from S3 and the cache hits.

Why it matters

  • What's now possible capability A check against archived lake history can run inside the transaction that is writing the new row, before that row commits.
  • What it limits constraint A team whose Iceberg tables sit in a non-AWS REST catalog has to add a Glue federation registration before Aurora can query them.
  • What it limits constraint Clusters on major versions before 17, or on 17 below 17.11 and 18 below 18.6, need an upgrade before the extension is available to them.
  • What it sets up precedent AWS says future DuckDB improvements can reach Aurora and other AWS services, so the same embedded engine is likely to appear in more of its products.

AWS says query processing "stays within Aurora, with no additional network hops" [5]. Frequently accessed lake data is cached in the Aurora instance, so repeat queries against it return faster [13]. One query can also read live operational data "including uncommitted writes" next to the lake tables [4]. For that to work, the embedded engine has to see the PostgreSQL session's own transaction state. I think it is the best-engineered part of the design.

The documented route to an Iceberg REST catalog runs through AWS Glue Data Catalog federation. The external catalog is registered with Glue once. Its tables then get foreign tables the same way Glue-native tables do [7]. From there, one query can join Aurora rows with Iceberg tables registered across several catalogs [8].

AWS describes the pipelines this is meant to replace as duplicating data, raising infrastructure costs and needing ongoing engineering effort to stay synchronized [9]. It also pitches the feature at AI agents, arguing that it is impractical to predict and pre-replicate every dataset an agent might need [15]. The argument works just as well for a human analyst with an unplanned question, though agents make better launch copy.

The efficiency case rests on predicate pushdown and column pruning. AWS says they limit reads to relevant data and keep queries efficient "even as the underlying data grows" [12]. Its demonstration is a psql session that runs `CREATE EXTENSION aurora_analytics;` and then builds a simple financial scenario [16]. That is a small example on AWS's side. For the efficiency claim to hold in production, the join a pipeline feeds today has to read a small slice of the lake per call, or keep landing in the cache. Where the join pulls most of its bytes from S3 on every request, I would keep the local copy. The per-query statistics function shows which of those cases applies before anything is switched off [14].

What to watch

  • Whether AWS adds a direct Iceberg REST catalog connection to Aurora that skips Glue federation.
  • Pricing and instance-sizing guidance for aurora_analytics lake scans once teams run them against production primaries.
  • Which other AWS services AWS names next as carrying the embedded DuckDB engine.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence62
Adoption
Not scored
Hype gap+20
Incentives72
Confidence70
Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

  1. [1]

    Amazon Aurora PostgreSQL can now directly query operational data together with data stored in a data lake in Apache Iceberg and Apache Parquet formats, using existing PostgreSQL applications and tools.

    Reported SupportedView cited source
  2. [2]

    The capability is supported on Aurora PostgreSQL 17 starting with 17.11 and 18 starting with 18.6.

    Reported SupportedView cited source
  3. [3]

    DuckDB is now embedded directly within Aurora PostgreSQL.

    Reported SupportedView cited source
  4. [4]

    Live operational data, including uncommitted writes, can be queried alongside the data lake in a single query.

    Reported SupportedView cited source
  5. [5]

    Query processing stays within Aurora, with no additional network hops and no ETL pipelines that duplicate data.

    Reported SupportedView cited source
  6. [6]

    Aurora can query Apache Iceberg tables managed through the AWS Glue Data Catalog, as well as Parquet and Iceberg data stored in Amazon S3 and S3 Tables.

    Reported SupportedView cited source
  7. [7]

    External Iceberg REST Catalog-compatible catalogs are queried through AWS Glue Data Catalog federation: the external catalog is registered once with Glue, then foreign tables are created the same way as for Glue-native tables.

    Reported SupportedView cited source
  8. [8]

    A single query can join data stored in Aurora with Iceberg tables registered across multiple catalogs.

    Reported SupportedView cited source
  9. [9]

    Previously, a common approach to combining Aurora transactional data with historical records in S3 was reverse ETL pipelines that duplicated data, increased infrastructure costs, and required ongoing engineering effort to keep synchronized.

    Reported SupportedView cited source
  10. [10]

    DuckLabs, the team that maintains the DuckDB project, recently joined Amazon, and AWS calls this capability an example of DuckDB being integrated into its services.

    Reported SupportedView cited source
  11. [11]

    To use the feature, you create an Aurora PostgreSQL cluster, attach an IAM role with the AuroraAnalytics feature (which gives Aurora access to S3 and the AWS Glue Data Catalog), enable the aurora_analytics extension, and create foreign tables pointing to Iceberg or Parquet data.

    Reported SupportedView cited source
  12. [12]

    Aurora applies predicate pushdown and column pruning so only relevant data is read, which AWS says keeps queries efficient even as the underlying data grows.

    Reported SupportedView cited source
  13. [13]

    Frequently accessed data is cached in the Aurora instance so subsequent queries against the same data return faster.

    Reported SupportedView cited source
  14. [14]

    aurora_analytics_stat_statements() reports per-query metrics such as rows scanned, bytes read from Amazon S3, and cache hits.

    Reported SupportedView cited source
  15. [15]

    AWS says that with AI agents embedded in applications it is impractical to predict and pre-replicate every dataset an agent might need.

    Reported SupportedView cited source
  16. [16]

    AWS's walkthrough connected to Aurora PostgreSQL with psql, ran CREATE EXTENSION aurora_analytics; and set up a simple financial scenario.

    Reported SupportedView cited source
  17. [17]

    AWS says building the capability around DuckDB means future improvements to the open source engine can bring performance and functionality gains to Aurora and other AWS services.

    Reported InsufficientView cited source

Sources

2 independent publishers whose own reporting we read for this story.

  1. aws.amazon.com

    2 articles · September 30, 2026

    Aurora PostgreSQL now supports querying of Apache Iceberg and Parquet data
  2. siliconangle.com

    1 article · September 30, 2026

    AWS embeds DuckDB in its PostgreSQL DBMS to speed queries of live and historical data

Topics and entities

Topics

Entities