Demystifying Apache Iceberg: Why Cloudflare R2 Data Catalog Changes the Open Lakehouse Game

While browsing Cloudflare’s platform offerings recently, a relatively new product caught my attention: R2 Data Catalog.

Cloudflare’s summary was concise:

“Managed Apache Iceberg catalog with automated table maintenance — Turn your R2 bucket into a managed data lake.”

If you have spent any time working with analytics, data engineering, or cloud architecture, that one sentence packs a tremendous punch.

Cloudflare initially shook up the cloud storage market when they launched R2—an S3-compatible object store featuring zero egress fees. But seeing them roll out a managed Apache Iceberg catalog with built-in table maintenance signaled something much bigger: the open data lakehouse architecture is going mainstream, and the days of cloud vendor data-moats are coming to an end.

To understand why this is such a big deal, we need to understand what Apache Iceberg is, why legacy data lakes were so painful to run, and how pairing Iceberg with R2 solves some of the most frustrating bottlenecks in data architecture.

---

The “Data Lake” Era (and Why Hive Tables Broke Down)

For years, the standard recipe for building an analytical “data lake” looked like this:

1. Dump gigabytes or terabytes of raw logs and event streams into an S3 bucket.
2. Convert those files into columnar formats like Apache Parquet or ORC.
3. Partition the files into directory hierarchies based on date or category:

   s3://my-lakehouse/events/year=2026/month=09/day=11/part-0001.parquet
   s3://my-lakehouse/events/year=2026/month=09/day=11/part-0002.parquet

4. Register the directory in an Apache Hive Metastore (HMS) or AWS Glue Data Catalog so engines like Presto, Trino, Spark, or Athena could query it using SQL.

This architecture, originally popularized by Apache Hive in the early Hadoop days, served the industry for over a decade. But as datasets grew to petabyte scales and write frequencies increased from daily batch jobs to near-real-time streaming, the cracks in the Hive model became impossible to ignore.

1. The S3 LIST Operation Bottleneck

Hive-style tables don’t track files explicitly; they track directories. When a query engine needed to plan a query over a partition, it had to issue an object storage LIST request across the bucket.

On object stores like S3, LIST operations are notoriously slow (paginated at 1,000 keys per call) and computationally expensive. If a partition had tens of thousands of files, query planning alone could take several minutes before a single row of data was actually read.

2. No ACID Guarantees (The Broken Ingestion Problem)

Object stores are key-value systems; they do not have transactional multi-file commits. If an ETL pipeline was halfway through writing 50 Parquet files to a partition and crashed:

• The destination partition was left in a corrupted, half-written state.
• Readers querying the table concurrently saw partial, inconsistent data.
• Cleaning up required manual, risky intervention.

3. Schema Evolution Headaches

Need to drop a column, rename one, or reorder fields? In Hive tables, schema was tied to column positions or names inside individual Parquet files. Renaming or reordering columns often broke historical files or produced silent nulls in downstream dashboards.

4. Partitioning Was Rigid and Fragile

If you initially partitioned by day and later wanted to repartition by hour (or by customer_id), you had to rewrite every single existing file in your lakehouse. Furthermore, users had to remember the explicit physical partitioning scheme in every query predicate (WHERE year=2026 AND month=09 AND day=11), or risk triggering a full bucket scan.

---

Enter Apache Iceberg: The Open Table Format

To fix these problems, Netflix created Apache Iceberg (later open-sourced through the Apache Software Foundation).

The single most important concept to grasp about Apache Iceberg is this:

Apache Iceberg is not a database engine, nor is it a file format. It is an open table format specification.

Where Parquet defines how bytes are laid out inside a single file, Iceberg defines how hundreds or millions of files are organized and tracked as a single, consistent table.

The Core Shift: State in Metadata, Not Directories

Instead of inferring table state by scanning directory paths, Iceberg tracks every single data file explicitly in a hierarchical tree of immutable metadata files.

               +----------------------+
               |   Catalog Pointer    |  (e.g., Cloudflare R2 Data Catalog)
               +----------------------+
                          │
                          ▼
               +----------------------+
               |     Table Metadata   |  (v3.metadata.json)
               | - Schema & History   |
               | - Snapshot Pointer   |
               +----------------------+
                          │
                          ▼
               +----------------------+
               |    Manifest List     |  (snap-1092834.avro)
               | - List of manifests  |
               +----------------------+
                      │        │
           ┌──────────┘        └──────────┐
           ▼                              ▼
+----------------------+      +----------------------+
|    Manifest File     |      |    Manifest File     |  (m1.avro, m2.avro)
| - Data file paths    |      | - Data file paths    |
| - Min/Max stats      |      | - Min/Max stats      |
| - Null counts        |      | - Null counts        |
+----------------------+      +----------------------+
     │          │                  │          │
     ▼          ▼                  ▼          ▼
┌─────────┐┌─────────┐        ┌─────────┐┌─────────┐
│ f1.pq   ││ f2.pq   │        │ f3.pq   ││ f4.pq   │  (Parquet Data Files)
└─────────┘└─────────┘        └─────────┘└─────────┘

This tree structure provides several game-changing capabilities:

1. Snapshot Isolation and True ACID Transactions

Every write, append, update, or delete in Iceberg creates a new Snapshot. Reads always query a specific snapshot. If a write fails midway, the new snapshot is never committed—readers only see clean, validated snapshots. Writers commit via an atomic compare-and-swap (CAS) operation on the catalog pointer.

2. High-Performance Metadata Pruning

Each Manifest File stores column-level statistics (min/max values, null counts) for every Parquet data file it tracks.

When you run a query like:

SELECT * FROM events 
WHERE user_id = 42 AND event_time >= '2026-09-01';

The query engine doesn’t even need to touch object storage data files or list directories. It simply inspects the manifest metadata, prunes out 99% of data files that cannot possibly contain matching values, and only downloads the exact Parquet files needed.

3. Safe Schema Evolution

Iceberg assigns every column a permanent, unique integer ID that never changes. You can add, rename, reorder, or drop columns freely. Older Parquet files written with previous column names continue to read seamlessly without rewriting data.

4. Hidden Partitioning

Iceberg abstracts partition transforms (such as month(event_time) or bucket(16, user_id)). Users query the logical column directly (WHERE event_time >= '2026-09-01'), and Iceberg automatically derives which partitions to scan. If you change the partition scheme later, Iceberg supports Partition Evolution—new data is written with the new spec while historical data remains untouched.

5. Time Travel and Rollbacks

Because snapshots are immutable, you can query historical versions of your table out of the box:

SELECT * FROM events FOR SYSTEM_TIME AS OF '2026-09-01 12:00:00 UTC';

If a faulty deployment inserts corrupted rows, rolling back is as simple as repointing the table to the previous snapshot.

---

The Catch: Catalogs & “Day-2” Table Maintenance

While Iceberg solved the table format dilemma, adopting it in production historically introduced two operational burdens:

1. You Need a Catalog

To make atomic commits, multiple engines (Spark, Trino, DuckDB) need a shared central authority that holds the current metadata.json pointer.

While AWS Glue or Hive Metastore were commonly used, the community increasingly rallied around the Apache Iceberg REST Catalog Specification—an open standard allowing any HTTP service to act as a catalog. Still, self-hosting a REST catalog (like Apache Polaris or Project Nessie) means provisioning servers, managing databases, and securing credentials.

2. The Maintenance Overhead (Compaction & Cleanup)

When you write streaming data or micro-batches into Iceberg, you inevitably create the small file problem—thousands of small Parquet files and manifest entries.

To keep query performance fast and storage costs low, someone has to run table maintenance jobs:

• Compaction: Combining hundreds of small Parquet files into optimal 128 MB or 512 MB files.
• Snapshot Expiration: Deleting old snapshots and manifests that are past retention limits.
• Orphan File Cleanup: Purging abandoned files from failed writes.

In most organizations, this meant setting up scheduled Apache Spark clusters just to run housekeeping jobs like rewrite_data_files().

---

Why Cloudflare R2 Data Catalog is a Game Changer

This brings us right back to Cloudflare’s announcement:

“Managed Apache Iceberg catalog with automated table maintenance - Turn your R2 bucket into a managed data lake.”

Cloudflare isn’t trying to build another proprietary query engine that locks your data inside their walled garden. Instead, they are providing the ideal storage and catalog foundation:

1. Native Iceberg REST Catalog

Cloudflare’s Data Catalog implements the standard Apache Iceberg REST Catalog specification. This means it is instantly compatible with any engine that speaks Iceberg REST:

• DuckDB
• PyIceberg
• Apache Spark
• Trino
• ClickHouse
• Snowflake
• Databricks

There is zero proprietary lock-in. Your data is plain Parquet in your R2 bucket, and your metadata adheres to the open Apache Iceberg specification.

2. Automated Table Maintenance

This is one of the highest friction points for small-to-medium teams adopting Iceberg. Cloudflare handles background compaction and snapshot expiration directly within R2. You can stream events from Cloudflare Workers or ingestion pipelines without worrying about your table degrading into millions of tiny, uncompacted files.

3. The Big One: Zero Egress Fees

In AWS, storing data in S3 is cheap, but querying it from outside AWS (or across regions/clouds) incurs punishing data egress fees. This “data gravity” often forces teams to consolidate all their compute and tooling inside a single cloud provider.

With Cloudflare R2, there are $0 data egress charges.

You can store your Iceberg data lakehouse on R2, and simultaneously:

• Run fast ad-hoc local queries from your laptop via DuckDB.
• Train machine learning models in a GPU cluster on another cloud.
• Power enterprise dashboards using Snowflake or Trino.

All of these engines can read from the exact same R2 Iceberg catalog without you worrying about an unexpected $5,000 data egress bill at the end of the month.

---

Hands-On: How It Looks in Practice

Getting started with R2 Data Catalog is straightforward.

1. Enabling the Catalog via Wrangler

You can enable the Iceberg catalog directly on an existing R2 bucket using the Cloudflare wrangler CLI:

# Enable the Iceberg catalog on your R2 bucket
npx wrangler r2 bucket catalog enable my-lakehouse-bucket

This returns your Catalog URI (https://catalog.cloudflarestorage.com/<ACCOUNT_ID>/my-lakehouse-bucket) and Warehouse Name (<ACCOUNT_ID>_my-lakehouse-bucket).

[!NOTE] Connecting requires two sets of credentials:

1. Cloudflare API Token (Bearer token with Workers R2 Storage or R2 Data Catalog permissions) to communicate with the Iceberg REST Catalog.
2. R2 S3 Access Keys (Access Key ID and Secret Access Key) to read and write underlying Parquet data files via R2’s S3-compatible API.

2. Querying with PyIceberg

Because Cloudflare exposes the open REST catalog protocol, you can interact with your lakehouse using Python and pyiceberg (installed with pyiceberg[pyarrow,s3fs]):

import os
from pyiceberg.catalog import load_catalog

account_id = os.getenv("CF_ACCOUNT_ID")
bucket_name = "my-lakehouse-bucket"

# Initialize the Iceberg REST Catalog pointing to Cloudflare R2
catalog = load_catalog(
    "cloudflare_r2",
    **{
        "type": "rest",
        "uri": f"https://catalog.cloudflarestorage.com/{account_id}/{bucket_name}",
        "warehouse": f"{account_id}_{bucket_name}",
        "token": os.getenv("CF_API_TOKEN"),
        "s3.endpoint": f"https://{account_id}.r2.cloudflarestorage.com",
        "s3.access-key-id": os.getenv("R2_ACCESS_KEY_ID"),
        "s3.secret-access-key": os.getenv("R2_SECRET_ACCESS_KEY"),
        "s3.region": "auto",
        "py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
    }
)

# List namespaces and load the table
print("Namespaces:", catalog.list_namespaces())
table = catalog.load_table("analytics.user_events")

# Scan the table using predicate pushdown
df = table.scan(
    row_filter="event_type == 'purchase' and amount > 30.0",
    selected_fields=("event_id", "user_id", "event_type", "amount", "event_timestamp")
).to_arrow().to_pandas()

print(df.head())

3. Querying with DuckDB

Prefer lightweight, lightning-fast SQL on your local machine? DuckDB has first-class native support for the Iceberg REST catalog:

-- Install and load DuckDB extensions
INSTALL iceberg;
INSTALL httpfs;
LOAD iceberg;
LOAD httpfs;

-- Configure Cloudflare API token for the Iceberg REST catalog
CREATE SECRET r2_catalog (
    TYPE ICEBERG,
    TOKEN '<CF_API_TOKEN>'
);

-- Attach the Cloudflare Iceberg REST catalog directly
ATTACH '<ACCOUNT_ID>_my-lakehouse-bucket' AS lakehouse (
    TYPE ICEBERG,
    ENDPOINT 'https://catalog.cloudflarestorage.com/<ACCOUNT_ID>/my-lakehouse-bucket'
);

-- Query using standard SQL with predicate pushdown
SELECT event_id, user_id, event_type, amount, event_timestamp
FROM lakehouse.analytics.user_events
WHERE amount > 30.0
LIMIT 10;

---

Final Thoughts: The Decoupled Future of Data

The evolution of data architecture over the last decade can be summarized as the steady decoupling of the data stack:

1. Decoupling Compute from Storage: (Snowflake, BigQuery, S3).
2. Decoupling Storage from Proprietary Formats: (Parquet, ORC).
3. Decoupling Table Management from Proprietary Engines: (Apache Iceberg).
4. Decoupling Catalogs and Clouds from Egress Taxes: (Iceberg REST Catalog + Cloudflare R2).

By combining an open, battle-tested standard like Apache Iceberg with an affordable, zero-egress storage layer like Cloudflare R2, the friction of building a robust data lakehouse is dropping significantly.

You no longer need an army of data platform engineers just to maintain an open data lake. Whether you are building an edge-ingestion pipeline with Cloudflare Workers or setting up a multi-cloud analytics hub, Apache Iceberg on R2 is an architecture well worth exploring.