You’re staring into your inbox at a sea of requests from your data team asking you to spin up another warehouse for the marketing analytics project, the finance team pushing back on your Snowflake bill – again – and your engineering team needs to share data with the ML team running Spark… but your current table format doesn’t play nice with external engines.
These problems are exactly what Apache Iceberg tables were made to solve. Originally developed by Netflix to handle massive-scale data challenges, Iceberg has become the open table format that redefines how enterprises manage data lakes. It’s not just another file format – it’s a complete rethinking of how tables should work in modern data architectures.
If your team is managing data infrastructure at scale, Iceberg tables offer something your current setup likely can’t: true interoperability without sacrificing performance.
But like any architectural decision, there are trade-offs, implementation challenges, and cost implications. We’ll cover everything you need to know before making the switch.
What Is an Apache Iceberg Table?
Apache Iceberg is an open table format designed specifically for huge analytical datasets in data lakes. Where a traditional data lake table will treat your data as a collection of files scattered across object storage, Iceberg introduces a layer of abstraction that makes those files act like actual database tables – all while maintaining the features you’d expect from a transactional database.
In other words: traditional data lakes give you the storage capacity you need, but without the reliability and features of a proper database. Iceberg, in comparison, brings those database-like capabilities to your data lake thanks to a sophisticated metadata layer.
Apache Iceberg Architecture
How exactly does this all work though? Iceberg uses a three-tier architecture that separates concerns from data storage, metadata management, and catalog operations.
Let’s break this down:
- Catalog layer: This layer is the entry point to Iceberg tables, tracking current metadata pointers for each table and ensuring atomic operations when metadata changes. You can use various catalogs – Snowflake’s built-in catalog, AWS Glue, a REST catalog, or even DynamoDB.
- Metadata layer: This is where Iceberg’s real power lives because every time you modify a table, Iceberg will write a new metadata file in JSON format. These metadata files are immutable snapshots that describe the complete state of your table at a specific point in time, including schema, partitioning specification, and pointers to manifest files.
- Manifest and data files: Manifest files are Avro-formatted files that contain lists of data files, partition information, and column-level statistics.
NOTE: The catalog layer doesn’t store your actual data – or even most of your metadata – it just points to the current state of the table.
This layered approach means Iceberg can update table state by writing small metadata files instead of rewriting entire datasets. So when you delete a single row, Iceberg doesn’t copy the entire 500GB table. Instead, it just writes a new metadata file pointing to the same data files with updated deletion markers.
Why Apache Iceberg Matters for Your Data Stack
So, you understand what Apache Iceberg tables are and how they work, but how does that information inform decisions for your data stack?
Here are the things you’ll want to keep in mind when choosing if you want to use Apache Iceberg or not:
- Cross-engine compatibility without data duplication
- ACID transactions in your data lake
- Time travel and schema evolution
Cross-Engine Compatibility Without Data Duplication
Everyone on your team wants to work with either Spark, Snowflake, Dremio – whatever tool speaks to them that week. Traditionally, this means maintaining separate copies of your data in different formats, with each optimized for a specific tool.
Iceberg completely eliminates that problem because it’s an open standard supported by Snowflake, Databricks, AWS service, Dremio, Spark, Flink, Trino, and more. That means you can store your data once – and query it from everywhere.
This isn’t just convenient. It’s a fundamental shift in data architecture that:
- Reduces storage costs
- Eliminates complex ETL pipelines
- Prevents the data consistency nightmares caused by maintaining multiple copies
How does this work? When Snowflake queries an Iceberg table stored in S3 – and Spark simultaneously runs a transformation on the same data. Both engines see a consistent view of the table. No conflicts, no overwrites, no manual coordination needed.
ACID Transactions in Your Data Lake
Traditional data lakes are eventually consistent at best. If two jobs try to write the same table simultaneously, you might get stuck with:
- Corrupted data
- Lost writes
- Inconsistent results
Iceberg brings ACID (Atomicity, Consistency, Isolation, Durability) transactions to object storage.
What does this mean practically?
- Your nightly ETL job can safely run alongside your real-time streaming updates
- Your analysts can query tables while data engineers are actively updating them
- Multiple teams can work on the same datasets without stepping on each other’s toes
Time Travel and Schema Evolution
Too often does bad deployment corrupt your production data. But with Iceberg, every change creates a new snapshot so you can query historical versions of your table or roll back to a previous state without restoring from backups.
Schema evolution works the same way. Need to add a new column or change a column type? Iceberg handles it without requiring you to rewrite billions of rows. The new schema is recorded in metadata while your existing data files remain untouched.
Apache Iceberg Across the Cloud Ecosystem
Iceberg’s true power comes from its universality. Here’s how the major cloud providers and data platforms have embraced it:
Apache Iceberg on AWS
AWS has made massive investments in Iceberg support across its analytics portfolio, introducing tools like Amazon S3 Tables, Amazon Athena, Amazon EMR, and AWS Glue.
- Amazon S3 Tables: Provides optimized storage for Iceberg tables with automatic compactions, snapshot management, and performance optimizations.
- Amazon Athena: Supports Iceberg natively for read, write, DDL, and time travel queries.
- Amazon EMR: Added Iceberg support starting with version 6.5.0, which allows for Spark, Hive, Flink, and Trino to work with Iceberg tables.
- AWS Glue: Acts as a managed Iceberg catalog and can automatically maintain Iceberg table metadata. This is especially powerful for multi-engine workflows.
Apache Iceberg on Snowflake
Snowflake’s Iceberg implementation is where things really shine. Snowflake supports two types of Iceberg tables:
- Snowflake-Managed Iceberg Tables: Snowflake acts as the catalog and manages all metadata operations. These tables deliver performance nearly identical to native Snowflake tables while storing data in your external cloud stage. So you get the full Snowflake feature set – clustering, search optimization, time travel, and dynamic tables – while maintaining data in an engine-agnostic format.
- Externally-Managed Iceberg Tables: Your Iceberg catalog lives elsewhere – like a AWS Glue or REST catalog – and Snowflake connects to it. These are typically read-only in Snowflake but allow you to query data managed by other systems. Performance remains stronger than traditional external tables because Snowflake can cache data locally and leverage Iceberg’s metadata for partition pruning.
Dremio and Apache Iceberg
Dremio has positioned itself as a lakehouse platform built specifically around Iceberg. Dremio provides a semantic layer that sits on top of your Iceberg table so you can:
- Query data from multiple Iceberg catalogs
- Add governance policies
- Create reflections (Dremio’s version of materialized views) for performance optimization
NOTE: Dremio is especially interesting because it is in support of Project Polaris (formerly Project Nessie) which brings Git-like branching and merging to data. This enables isolated development environments, catalog-level rollback, and multi-table transactions.
The Hidden Costs of Apache Iceberg Tables
Apache Iceberg tables aren’t all pros. They can get expensive fast if you’re not careful. The main things you need to look out for are:
- Metadata management overhead
- Small file proliferation
- Storage costs that creep up
- Compute spikes for external engines
Let’s take a closer look at each of these so you can best understand how to use Apache Iceberg so it won’t devour your monthly bill.
Metadata Management Overhead
Every transaction in Iceberg writes new metadata files. Adding 100 rows? New metadata files. Updating a single record? New metadata file. Run a streaming job that writes micro-batches every few seconds? That’s thousands of new metadata files. Each hour.
These small files add up quickly. While each might be tiny (usually only a few KB), the sheer volume creates two problems:
- Cloud storage costs add up. S3 charges per request, and listing thousands of small files generates millions of API calls. These costs fade into the background with your Snowflake bill, but show up as increased AWS charges.
- Query planning slows down. Before Iceberg can execute a query, it needs to read the metadata chain to understand the current state of the table. With thousands of snapshots, this metadata traversal often becomes a bottleneck.
The only way to manage this is regular maintenance. You’ll want to automate it so you can run cleanup and optimization tasks consistently and identify infrastructure weaknesses early.
Small File Proliferation
Iceberg doesn’t automatically consolidate small data files. So if you’re running streaming ingestion or frequent small updates, you’ll end up with thousands of tiny Parquet files. And since each file requires a separate read operation, this destroys query performance.
The fix here is compaction. You’ll want to rewrite small files into larger, optimized files.
But there’s a catch: compaction is compute-intensive. Running compaction on a 10TB table can consume hours of warehouse time. You’ll want to balance the performance benefits of well-compacted data against the compute costs of regular compactions.
This is where Snowflake warehouse optimization becomes critical. You don’t want your compaction jobs running on oversize warehouses – and you also can’t let them drag on forever on undersized ones.
Storage Costs that Creep Up
Iceberg tables never delete data files – at least, not automatically. When you delete rows or update records, Iceberg writes new files and marks old ones as deleted metadata. But those old files remain in S3 until someone goes back to clean them up.
Time travel is great – until you realize you’re paying to store 30 days of table history across hundreds of tables. That math gets ugly fast. You’re left with significant cloud storage costs just for maintaining historical versions.
Compute Spikes for External Engines
When you query Iceberg tables from Snowflake that are managed by an external catalog (like AWS Glue), Snowflake needs to fetch metadata from that catalog. This adds latency to every query. For infrequently run queries, this is fine. But if you’re running hundreds of queries per hour against externally-managed tables, those metadata fetches add up to real compute time.
The fix here is to refresh metadata proactively instead of on-demand.
But auto-refresh itself consumes computes. You’re trading query latency for background processing costs.
Taking Control of Snowflake Costs with Apache Iceberg
Iceberg adoption is accelerating – and Snowflake’s roadmap indicates deeper Iceberg integration. Apache Iceberg represents a powerful architectural choice for modern data platforms. They enable true data lakehouse architectures, eliminate data silos, and provide flexibility that native table formats simply can’t match.
But Apache Iceberg also introduces new cost and performance challenges:
- Metadata management, small file proliferation, and storage growth require constant attention.
- Query patterns that work well on native tables can become expensive on Iceberg.
- Warehouse sizing becomes trickier when some queries need metadata-heavy planning before execution.
If you’re running Iceberg tables in Snowflake and find cost management challenging, you’re not alone. Snowflake’s consumption-based pricing and Iceberg’s metadata overhead create a difficulty to optimize for overhead.
This is exactly the problem Yuki was made for. Our platform automatically optimizes warehouse sizing, query routing, and resource allocation for Snowflake workloads – including the unique challenges of Iceberg tables.
Yuki will handle:
- Complexity of right-sizing compute for metadata-heavy operations
- Automatically route queries to appropriately-sized warehouses
- Provide visibility into the true cost drivers of your Iceberg implementation
Organizations using Yuki see 30% reductions in Snowflake costs without sacrificing performance. The platform requires no code changes, integrates in under an hour, and starts optimizing immediately. For teams managing Iceberg at scale, this automation isn’t just convenient. It’s essential.
Want to see how much you can save on your Snowflake bill? Get your personalized Yuki demo and see real optimization opportunities in your environment.


