Key Features
Tech Stack
- On a 4.01 billion-row dataset, OLake sustained around 319,562 rows/sec (full-load) from Postgres into Iceberg.
- The next-best ingestion tool we tested on the same dataset managed about 46,000 rows/sec, making OLake roughly 6.8× faster for full loads.
- For CDC workloads, OLake ingested change batches at around 41,390 rows/sec, compared to ~26,900 rows/sec for the closest alternative.
- Average memory usage was about 44 GB, peaking at ~59 GB on a 64-vCPU / 128 GB RAM VM.
- Parquet file output stabilized at ~300–400 MB per file (after compression), improving performance downstream and avoiding “small file” fragmentation.
How we got these improvements:
1. Rewrote the writer architecture: Parsing, schema evolution, buffering, and batch management now happen in Go. Only the final Parquet and Iceberg write path uses Java. This cut down huge amounts of serialization and JVM churn.
2. Introduced a new batching and buffering model: Instead of producing many small Parquet files, we buffer data in memory per thread and commit large chunks (roughly 4 GB before compression). This keeps throughput high and files uniform.
3. Optimized Iceberg metadata operations: Commits remain atomic even with large batches, and schema evolution happens fully in Go before any write, reducing cross-system coordination.
4. Improved operational stability: CPU, memory, and disk behaviour remained predictable even at multi-billion-row scales.
Benchmark setup:
- Dataset: ~4.01 billion rows from the NYC Taxi + FHV trips parquet sets (row width ~120–144 bytes).
- Test machine: Azure Standard D64ls v5 (64 vCPUs, 128 GB RAM).
Iceberg stored on local NVMe for the benchmark, same architecture works with S3/GCS/HDFS.
The full benchmark results, methodology, and configs are here: https://olake.io/docs/benchmarks/
And the deep-dive into how we got the ~7× speedup is here: https://olake.io/blog/how-olake-becomes-7x-faster/
I’d love feedback from the HN community, specifically around tuning (batch sizes, commit frequency, partitioning strategies), Iceberg best practices, and real-world constraints you’ve seen in high-volume pipelines.
Happy to answer questions or share configs. Thanks for taking a look!
Not affiliated with Hacker News or Y Combinator. We simply enrich the public API with analytics.