For years, Pandas was the default choice for data manipulation in Python. It was (and still is) incredibly productive for small-to-medium datasets and exploratory work.
But in 2026, when you're working with real production data lakehouses — especially those built on Apache Iceberg, AWS Glue, and S3 — Pandas starts showing its age very quickly.
The Spark Setup That Felt Like a Step Backwards
On a recent project, I inherited a single-machine Spark setup that was being used to process data sitting in an Iceberg lakehouse on S3, cataloged in AWS Glue.
The configuration was heavy:
- Long
spark-submitcommands - Manual JAR management
- Explicit S3 paths everywhere
- Lots of boilerplate just to read a table
Even after getting everything wired up correctly, the jobs were slow. Not "occasionally slow" — consistently slow for what should have been straightforward transformations.
After spending far too much time fighting the setup instead of solving the actual data problem, I started looking for something better.
Enter Polars + PyIceberg
This is when I discovered how powerful the combination of Polars and PyIceberg has become.
Polars is a modern DataFrame library written in Rust. It brings several advantages that matter a lot in 2026 lakehouse environments:
- Native lazy evaluation (similar to Spark's logical plans, but with almost zero overhead)
- True multi-threaded execution that automatically uses all available CPU cores
- Excellent Parquet + Iceberg support via PyIceberg
- Direct integration with AWS Glue Catalog — no more manually constructing S3 paths
- Dramatically lower memory usage compared to Pandas or Spark on a single machine
The same workload that was taking a long time in the Spark setup completed in less than half the time using Polars in lazy mode.
The code was also dramatically simpler.
When Polars + PyIceberg Shines
This stack is particularly strong when:
- You're working on a single machine or small cluster (up to tens of GBs comfortably, sometimes more)
- You want maximum performance per core with minimal operational overhead
- You're doing transformations, feature engineering, or analytics workloads
- You want to avoid managing Spark clusters just for medium-sized jobs
The developer experience is night and day compared to traditional Spark on a laptop or small EC2 instance.
Where Spark / EMR Still Makes Sense
Of course, no single tool wins in every scenario.
Even with native
sink_iceberg() support, there are still cases where Spark (or EMR) is the stronger choice:- Extremely large data volumes (hundreds of GB to multiple TB)
- Heavy concurrent Iceberg writes combined with maintenance operations (compaction, snapshot expiration, branching at scale)
- Workloads that benefit from Spark’s mature distributed execution and write optimizations
In these situations, Spark remains a very capable and reliable tool.
Current Recommended Approach (with Native sink_iceberg)
Good news — native
sink_iceberg() support has now landed in Polars. This significantly improves the developer experience for Iceberg lakehouse workloads.The recommended pattern in 2026 is now much cleaner:
- Perform all transformations, joins, and feature engineering using Polars in lazy mode with PyIceberg.
- Use the new
.sink_iceberg()method to write directly to Iceberg tables (with full Glue Catalog integration). - This removes the need for intermediate Parquet staging in most cases.
This approach delivers excellent performance on a single machine or small cluster while keeping the code simple and fully lazy.
When You Might Still Want Spark
Even with native sinking available, there are still scenarios where Spark (or EMR) remains the better choice:
- Extremely large-scale concurrent writes (multi-terabyte)
- Heavy Iceberg maintenance operations (compaction, expire snapshots, branching at scale)
- Workloads that benefit from Spark’s mature distributed write optimizations and fault tolerance
Practical Recommendation
A pragmatic approach many teams follow today:
- Use Polars + PyIceberg (with
sink_iceberg) for the vast majority of transformation and write workloads. - Only bring in PySpark when you hit the limits of single-machine / small-cluster execution on very large write-heavy jobs.
This hybrid strategy is now much lighter than before, thanks to native Iceberg sinking support in Polars.
Final Thoughts
Pandas isn't going anywhere for quick analysis and small scripts. But if you're building or working on modern data lakehouses in 2026, especially with Iceberg on S3 + Glue, it's worth seriously evaluating Polars + PyIceberg.
In many single-machine and medium-scale scenarios, it is simply the better tool: faster, lighter, and more pleasant to work with.
With native Iceberg sinking now available, the era of reaching for Spark by default for every data job has shifted significantly — especially for single-machine and medium-scale lakehouse workloads.
Image: Performance comparison between Polars and Spark on the same workload.
