The dataset is only 200 GB. Why does opening it take longer than reading it?

The same bytes are not the same workload

Imagine two layouts with identical content. In the first, every record batch creates another tiny object. In the second, many batches are combined into a modest number of larger columnar files.

The first layout forces the engine to discover, plan, open, and coordinate many more units of work. File metadata must be inspected. There may be object-store requests for manifests, headers or Parquet footers. The scheduler has more tasks or must group many files into fewer input splits. Connection setup and request latency can dominate the transfer of a tiny object.

A file does not necessarily become one task. Engines can combine small file ranges into a larger input split, and they can process multiple files in one task. Even then, the underlying metadata and open operations still exist. Saying "200,000 files means exactly 200,000 tasks" would be inaccurate.

The simplified arithmetic gives us a sense of the change. If 200,000 one-MiB files were rearranged into idealized 256-MiB files without changing the total data volume, we would need about 782 data files rather than 200,000. Real results depend on table partitions, compression, row group boundaries and the available data in each partition. File sizing is a target, not a promise that every file reaches exactly 256 MiB.

Why not just increase the target file size?

The writer determines the actual file boundaries. A target property is not an instruction to concatenate data from unrelated tasks after the job finishes.

Iceberg's Spark write guidance points out a crucial distinction: a file cannot be bigger than the task that creates it, and a file does not cross a table partition boundary. Compressed columnar output may be much smaller than the in-memory input data assigned to the task. Changing a target to 512 MB will not create 512 MB files if each task produces only a few megabytes. Iceberg Spark writes

I would look at how the job partitions output. Did it create one partition per minute and customer, leaving only a few rows in most combinations? Does each microbatch produce a separate file in every active partition? Are tasks too small? Is the file count driven by partition cardinality or by the scheduler's write parallelism?

Sometimes the right answer is to choose a coarser partitioning strategy and rely on file statistics to prune reads. Other times, adjusting the write distribution and output task size is enough. There is no universal best file size. Smaller files can reduce rewrite cost for frequently changing data, while larger files tend to improve sequential scans and reduce overhead. Parquet also has row group sizing considerations that are separate from whole-file sizing. Parquet file configurations

Compaction is a data rewrite, not a rename

For a table with snapshot metadata, I would schedule compaction to rewrite groups of small files into fewer larger ones. The rewritten files contain the same logical rows and preserve the table's required delete semantics. The table then commits metadata that points new readers to the replacement files.

I would not simply delete original objects immediately after uploading replacements. A reader may still be using an earlier snapshot, and the compaction commit might fail. Existing files remain authoritative until the table's commit protocol establishes that the new snapshot is current. Old files must remain as long as retained snapshots or active readers require them.

A compactor also needs a rate budget. Rewriting 200 GB reads that data and writes it again, competing with ordinary queries and ingestion. A busy table can accumulate small files faster than a slow background compactor removes them. Track new files per hour, the distribution of file sizes, metadata planning time and data rewrite throughput, rather than treating "compaction succeeded" as the only metric.

What makes cleanup dangerous?

Suppose a writer has uploaded fresh data files but has not completed its metadata commit. A cleanup job scans object storage and sees files not referenced by the current table snapshot. If it deletes them immediately, that writer might later commit metadata pointing at missing data.

Iceberg explicitly warns that orphan-file cleanup must use a safe retention interval because in-progress writes can appear orphaned. Snapshot expiration and orphan removal also solve different problems. One removes old history and files no longer required by retained snapshots. The other looks for files outside known table references. Iceberg maintenance

My immediate fix would combine query and writer metrics, target the worst partitions first, and run a controlled compaction experiment while verifying row counts and snapshot consistency. My durable fix would change the writer's partitioning and file-size behavior so the problem does not return tomorrow.

Now a question about the last line: What makes it safe to tell readers that hundreds of new files are one complete table version rather than half of a failed upload? That is the subject of the next lesson.