Key Takeaways

  • Craig Kerstiens and engineers from Crunchy Data and Snowflake open-sourced pg_lake in November, bundling roughly 12 separate extensions into a single release.
  • The extension embeds an Apache Iceberg catalog directly into Postgres, keeping table metadata organized natively without third-party catalog services.
  • DuckDB runs in-process inside Postgres as an analytical query engine, reading columnar Parquet files from Amazon S3 at high speeds.
  • Adding a simple USING iceberg clause to standard CREATE TABLE statements offloads storage to S3, bypassing traditional row-oriented table bloat.

One Database Instead of a Fragile Pipeline

Craig Kerstiens spent years watching engineering teams build the same brittle architecture. A startup writes its operational data into Postgres. Then the team builds an extraction pipeline, dumps logs and transactional tables into S3, and spins up a separate analytical warehouse just to run aggregations. Every moving part adds a failure point, another secret to rotate, and another vendor bill.

In November, Crunchy Data and Snowflake took a different approach by releasing pg_lake. Kerstiens describes the project as an ambitious technical feat: “We open sourced pg_lake back in November, which, oh, man, to me it's like a work of art of like it's actually like 12 extensions in one.”

The project began with a narrow focus: managing Iceberg metadata from inside Postgres. “So pg_lake is like the overarching extension, but like we started with putting an Iceberg catalog inside Postgres,” Kerstiens said. “Well, it's just the catalog. But once we have the catalog there, well, okay, great. Well, we need something to query this, so we embedded DuckDB inside Postgres.”

Postgres handles transactional rows well, but it struggles with massive scans across cold data. DuckDB solves that exact problem. As Kerstiens put it, “DuckDB is like SQLite, but for analytics, right? So, you've got SQLite as your lightweight, embedded transactional database, right? It's great for local stuff, right? I want to ship a small database around, I I want to embed something.” By dropping DuckDB straight into the Postgres runtime, pg_lake removes the need for an external analytical sidecar.

Storing Parquet with a Single Clause

The developer workflow inside pg_lake does not require new client libraries or custom query syntax. Developers interact with familiar SQL schemas. The database routes the actual bytes to object storage behind the scenes.

“So now I can go inside Postgres and go, 'Create table, orders,' and all I have to say is, 'using Iceberg.' And now, it's automatically going to store the data in these Parquet files in S3,” Kerstiens explained.

This design flips the standard data platform pattern on its head. Instead of writing rows to transactional disk storage and later copying them to Parquet files via an overnight batch job, the database writes Parquet files directly to S3. When queries come in, DuckDB reads those Parquet files using columnar execution paths designed specifically for analytics.

For technical teams, this collapses two separate layers into one interface. You keep Postgres as the front door for your application, your connection poolers, and your developer tooling. Your analytical queries run against cheap object storage using a vectorized execution engine, without you ever configuring an external lakehouse platform.

What to Do With This

Audit your current reporting and analytics setup. If you are running an external pipeline solely to copy Postgres tables into S3 for warehouse queries, spin up a staging Postgres instance with pg_lake installed. Create a sample table using the USING iceberg syntax, point it at a test S3 bucket, and benchmark your heaviest aggregation queries directly inside Postgres.