Columnar Formats and Other Big Data Formats

• Vortex Benchmarks
• Is Parquet becoming the bottleneck? Why new storage formats are emerging in 2025 (Lance, Vortex, and more)

Lance

• [2504.15247v1] Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings
• Lance v2: A New Columnar Container Format

Parquet

Vortex

GitHub - vortex-data/vortex: An extensible, state of the art columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

Nimble

GitHub - facebookincubator/nimble: New file format for storage of large columnar datasets.

F3

• dl.acm.org/doi/pdf/10.1145/3749163