For years the data lake was a polite fiction: a pile of Parquet files in object storage, with every query engine pretending the directory listing was a table. It mostly worked — until a job failed halfway through a write, or two writers collided, or someone renamed a partition column and broke every downstream query.
Apache Iceberg replaces that fiction with a real table format: a metadata layer over open files that gives you atomic commits, schema evolution, and snapshots. The strategic shift is that Snowflake, Databricks, AWS, and Google have all adopted it — which means, for the first time, your storage layer can outlive your choice of engine.
What the metadata layer buys you
ACID on object storage: every write produces a new snapshot that becomes visible atomically, so readers never see a half-written table and concurrent writers resolve conflicts instead of corrupting each other. The 'directory full of files' failure modes simply disappear.
Time travel falls out of the same design — query the table as of any retained snapshot, which turns 'the dashboard looked wrong on Tuesday' from an archaeology project into a SQL clause, and makes rolling back a bad load a metadata operation rather than a restore.
Hidden partitioning fixes the oldest lake footgun. You declare a transform — day(event_ts), bucket(user_id, 32) — and Iceberg maintains the partition values and prunes files automatically. Analysts write WHERE event_ts > ... like a normal predicate; nobody needs to know the magic partition column, and you can change the partition scheme later without rewriting the table or the queries.
Migrating without downtime
The good news: because Iceberg's data files are ordinary Parquet, migration is mostly a metadata exercise. Snapshot procedures can register existing files under Iceberg metadata without copying data, so the pattern is straightforward — snapshot the existing table, dual-write during validation, flip readers table by table, keep the old path until the last consumer moves. No big-bang weekend.
Two decisions deserve care up front. First, the catalog — the source of truth for table state. A REST-based catalog keeps you engine-neutral, which is the whole point. Second, maintenance: streaming writes produce small files, so schedule compaction and snapshot expiry from day one, or watch query performance decay slowly and mysteriously.
The lock-in question
The quiet win is negotiating leverage. When your tables are Iceberg in your own buckets, the warehouse becomes a replaceable compute layer — Spark for heavy transforms, Trino for ad-hoc SQL, Snowflake or DuckDB where they shine, all reading the same tables. Vendors know this, which is exactly why they all raced to support it. Openness is the rare feature that keeps paying after the migration is done.