Polars LazyFrame and PySpark: The Syntax Cheatsheet

01 — The shared idea

Two libraries, one habit

Polars LazyFrame and a PySpark DataFrame do not run your code line by line. They record a plan, then run it once when you ask for a result.

If you already write one of them, the other is mostly a change of spelling. A filter is still a filter. A group-by is still a group-by. A window is still "compute this inside each group, and keep every row."

This post uses one tiny orders table the whole way through, and these two imports:

import polars as pl
import pyspark.sql.functions as sp

pl is the Polars module. sp is PySpark's function module. DataFrame methods such as .select, .filter, .join, and .groupBy live on the DataFrame, not inside sp. Window specs are the one extra import you will need later:

from pyspark.sql.window import Window

When the plan actually runs

Polars waits for .collect(), .fetch(), or a .sink_*() write. Spark waits for an action: .show(), .count(), .collect(), or .write. Until then you are only describing work.

02 — The table we keep reusing

Orders and customers

Six orders. One of them has no amount. One customer in the customer table never ordered anything.

order_id	customer_id	region	status	amount	quantity	order_date	note
1	C1	APAC	paid	120.0	2	2026-01-15	gift rush
2	C2	EMEA	pending	45.5	1	2026-01-16	standard
3	C1	APAC	paid	200.0	4	2026-02-02	gift
4	C3	AMER	refunded	null	1	2026-02-10	null
5	C2	EMEA	paid	80.0	3	2026-03-01	rush
6	C4	APAC	paid	15.0	1	2026-03-12	standard gift

customer_id	name	tier
C1	Ada	gold
C2	Lin	silver
C3	Mo	bronze
C5	Priya	gold

Build them once. In Polars, .lazy() turns an in-memory frame into a LazyFrame. In Spark, a DataFrame is already lazy.

from datetime import date
 
orders = pl.DataFrame({
    "order_id": [1, 2, 3, 4, 5, 6],
    "customer_id": ["C1", "C2", "C1", "C3", "C2", "C4"],
    "region": ["APAC", "EMEA", "APAC", "AMER", "EMEA", "APAC"],
    "status": ["paid", "pending", "paid", "refunded", "paid", "paid"],
    "amount": [120.0, 45.5, 200.0, None, 80.0, 15.0],
    "quantity": [2, 1, 4, 1, 3, 1],
    "order_date": [
        date(2026, 1, 15), date(2026, 1, 16), date(2026, 2, 2),
        date(2026, 2, 10), date(2026, 3, 1), date(2026, 3, 12),
    ],
    "note": ["gift rush", "standard", "gift", None, "rush", "standard gift"],
}).lazy()
 
customers = pl.DataFrame({
    "customer_id": ["C1", "C2", "C3", "C5"],
    "name": ["Ada", "Lin", "Mo", "Priya"],
    "tier": ["gold", "silver", "bronze", "gold"],
}).lazy()

orders = spark.createDataFrame([
    (1, "C1", "APAC", "paid", 120.0, 2, "2026-01-15", "gift rush"),
    (2, "C2", "EMEA", "pending", 45.5, 1, "2026-01-16", "standard"),
    (3, "C1", "APAC", "paid", 200.0, 4, "2026-02-02", "gift"),
    (4, "C3", "AMER", "refunded", None, 1, "2026-02-10", None),
    (5, "C2", "EMEA", "paid", 80.0, 3, "2026-03-01", "rush"),
    (6, "C4", "APAC", "paid", 15.0, 1, "2026-03-12", "standard gift"),
], ["order_id", "customer_id", "region", "status", "amount", "quantity", "order_date", "note"])
 
orders = orders.withColumn("order_date", sp.to_date("order_date"))
 
customers = spark.createDataFrame([
    ("C1", "Ada", "gold"),
    ("C2", "Lin", "silver"),
    ("C3", "Mo", "bronze"),
    ("C5", "Priya", "gold"),
], ["customer_id", "name", "tier"])

Reading files is the same idea with a different verb. Polars scan functions stay lazy. Spark's reader stays lazy too.

Goal	Polars LazyFrame	PySpark
CSV	`pl.scan_csv("orders.csv")`	`spark.read.csv("orders.csv", header=True, inferSchema=True)`
Parquet	`pl.scan_parquet("orders.parquet")`	`spark.read.parquet("orders.parquet")`
See the plan	`lf.explain()`	`df.explain()`
See the schema	`lf.collect_schema()`	`df.printSchema()`
Materialise	`lf.collect()`	`df.show()` or `df.collect()`
First n rows	`lf.head(5).collect()`	`df.limit(5).show()`
Write Parquet	`lf.sink_parquet("out.parquet")`	`df.write.parquet("out.parquet")`

.collect() on Spark pulls every row to the driver. Prefer .show() while you are learning, and a write when the result is large. Polars .collect() is the normal way to finish a single-machine plan.

03 — Columns are expressions

`pl.col` and `sp.col`

Almost every transformation is "make an expression, then hand it to the frame."

pl.col("amount")          # the amount column
pl.lit(0)                 # a constant column
pl.col("amount") * 1.1    # a new expression, not yet a column name
(pl.col("amount") * 1.1).alias("amount_taxed")

sp.col("amount")
sp.lit(0)
sp.col("amount") * 1.1
(sp.col("amount") * 1.1).alias("amount_taxed")

.alias() is how both libraries name a computed column. If you forget it inside .select(), Polars names the column after the expression and Spark often names it after the expression text. Name it yourself.

You can also drop into a SQL snippet when the expression is awkward:

pl.sql_expr("round(amount * 1.1, 2)")
sp.expr("round(amount * 1.1, 2)")

Parentheses are not optional

& and | bind more tightly than > and ==. Write (pl.col("region") == "APAC") & (pl.col("amount") > 50). The Spark version is identical, with sp.col. Without the parentheses, Python throws a TypeError before either engine sees the plan.

04 — Shaping columns

Select, add, rename, drop

Select throws the other columns away.

orders.select("order_id", "region", pl.col("amount"))

orders.select("order_id", "region", sp.col("amount"))

order_id	region	amount
1	APAC	120.0
2	EMEA	45.5
3	APAC	200.0
4	AMER	null
5	EMEA	80.0
6	APAC	15.0

With-columns keeps everything and adds or replaces.

orders.with_columns(
    (pl.col("amount") * 1.1).round(2).alias("amount_taxed"),
    pl.col("note").str.to_uppercase().alias("note_upper"),
)

orders.withColumns({
    "amount_taxed": sp.round(sp.col("amount") * 1.1, 2),
    "note_upper": sp.upper(sp.col("note")),
})

withColumn (singular) is the one-column form: .withColumn("amount_taxed", sp.round(sp.col("amount") * 1.1, 2)). Polars has no singular version. One call can add as many expressions as you want, and they all see the columns from before that call. A new column cannot be used by its sibling in the same with_columns or withColumns. Chain a second call.

order_id	amount	amount_taxed	note_upper
1	120.0	132.0	GIFT RUSH
2	45.5	50.05	STANDARD
3	200.0	220.0	GIFT
4	null	null	null
5	80.0	88.0	RUSH
6	15.0	16.5	STANDARD GIFT

Null times 1.1 is still null. Filling comes later.

Goal	Polars	PySpark
Rename one	`.rename({"customer_id": "cust"})`	`.withColumnRenamed("customer_id", "cust")`
Drop columns	`.drop("note", "quantity")`	`.drop("note", "quantity")`
Cast	`pl.col("amount").cast(pl.Int64)`	`sp.col("amount").cast("long")`
Keep all except	`pl.all().exclude("note")`	`.drop("note")`
Round	`pl.col("amount").round(2)`	`sp.round(sp.col("amount"), 2)`
Absolute value	`pl.col("amount").abs()`	`sp.abs(sp.col("amount"))`

Common Spark type names: "string", "int", "long", "double", "boolean", "date", "timestamp". Polars uses pl.String, pl.Int64, pl.Float64, pl.Boolean, pl.Date, pl.Datetime.

05 — Rows

Filter, sort, distinct, limit

orders.filter(pl.col("status") == "paid")
orders.filter(
    (pl.col("region") == "APAC") & (pl.col("amount") > 50)
)
orders.filter(pl.col("status").is_in(["paid", "pending"]))
orders.filter(pl.col("amount").is_between(50, 150))
orders.filter(pl.col("amount").is_not_null())

orders.filter(sp.col("status") == "paid")
orders.filter(
    (sp.col("region") == "APAC") & (sp.col("amount") > 50)
)
orders.filter(sp.col("status").isin(["paid", "pending"]))
orders.filter(sp.col("amount").between(50, 150))
orders.filter(sp.col("amount").isNotNull())

Paid orders are 1, 3, 5, and 6. APAC and amount above 50 leaves orders 1 and 3. Order 4 disappears from every numeric comparison because null is not greater than 50, not less than 50, and not equal to 50.

.where() on a Spark DataFrame is the same method as .filter().

Goal	Polars	PySpark
Sort ascending	`.sort("amount")`	`.orderBy("amount")`
Sort descending, nulls last	`.sort("amount", descending=True, nulls_last=True)`	`.orderBy(sp.col("amount").desc_nulls_last())`
Several keys	`.sort("region", "order_date")`	`.orderBy("region", "order_date")`
Distinct rows	`.unique()`	`.distinct()`
Distinct on keys	`.unique(subset=["customer_id"])`	`.dropDuplicates(["customer_id"])`
First n	`.limit(3)`	`.limit(3)`
Drop rows with nulls	`.drop_nulls(subset=["amount"])`	`.dropna(subset=["amount"])`

Descending with nulls last yields amounts 200, 120, 80, 45.5, 15, then null. If you do not say where nulls go, the two engines will not agree.

06 — Decisions and missing values

When, otherwise, and null

The shape of when is the biggest syntax difference in daily code.

orders.with_columns(
    pl.when(pl.col("amount").is_null()).then(pl.lit("missing"))
      .when(pl.col("amount") >= 100).then(pl.lit("large"))
      .otherwise(pl.lit("small"))
      .alias("size")
)

orders.withColumn(
    "size",
    sp.when(sp.col("amount").isNull(), "missing")
      .when(sp.col("amount") >= 100, "large")
      .otherwise("small"),
)

The value sits in the second argument of when. There is no .then(). Branches run in order, and a null condition does not match, so amount >= 100 falls through when amount is missing and the next isNull branch can still catch it. Order 4 is "missing", orders 1 and 3 are "large", and the rest are "small".

Filling nulls:

pl.col("amount").fill_null(0)
pl.col("note").fill_null("none")

sp.coalesce(sp.col("amount"), sp.lit(0.0))
sp.coalesce(sp.col("note"), sp.lit("none"))

coalesce walks its arguments and returns the first one that is not null. fill_null is the one-column version of that idea. After filling amount, order 4 becomes 0 and the other amounts stay as they were.

Two different 'counts'

pl.len() and sp.count("*") count rows. pl.col("amount").count() and sp.count("amount") count non-null amounts. On this table that is 6 versus 5. Mixing them up is the usual reason a "number of orders" looks one short.

07 — Grouping

One row per group

summary = (
    orders
    .group_by("region")
    .agg(
        pl.len().alias("n_orders"),
        pl.col("amount").sum().alias("revenue"),
        pl.col("amount").mean().round(2).alias("avg_amount"),
        pl.col("customer_id").n_unique().alias("n_customers"),
    )
    .sort("region")
)

summary = (
    orders
    .groupBy("region")
    .agg(
        sp.count("*").alias("n_orders"),
        sp.sum("amount").alias("revenue"),
        sp.round(sp.avg("amount"), 2).alias("avg_amount"),
        sp.countDistinct("customer_id").alias("n_customers"),
    )
    .orderBy("region")
)

region	n_orders	Polars revenue	Spark revenue	avg_amount	n_customers
AMER	1	0.0	null	null	1
APAC	3	335.0	335.0	111.67	2
EMEA	2	125.5	125.5	62.75	1

AMER is the lesson. Its only amount is null.

Both means are null. An average of no numbers is missing.
Spark's sum of no numbers is null.
Polars' sum() inside agg returns 0.

If a later step divides by revenue or filters revenue > 0, those two results take different paths. Fill nulls before the aggregation when you want them to match, or use a conditional sum and accept the documented difference.

More aggregations you will actually use:

Question	Polars inside `.agg()`	PySpark inside `.agg()`
Row count	`pl.len()`	`sp.count("*")`
Non-null count	`pl.col("amount").count()`	`sp.count("amount")`
Sum	`pl.col("amount").sum()`	`sp.sum("amount")`
Mean	`pl.col("amount").mean()`	`sp.avg("amount")`
Min / max	`.min()` / `.max()`	`sp.min("amount")` / `sp.max("amount")`
Median	`pl.col("amount").median()`	`sp.percentile_approx("amount", 0.5)`
Std dev	`pl.col("amount").std()`	`sp.stddev("amount")`
First / last	`.first()` / `.last()`	`sp.first("note")` / `sp.last("note")`
Distinct count	`pl.col("customer_id").n_unique()`	`sp.countDistinct("customer_id")`
List of values	`pl.col("order_id")`	`sp.collect_list("order_id")`

n_unique() counts null as its own distinct value. countDistinct ignores nulls. On a column that contains a null, Polars reports one extra.

Filter before you group when the filter does not depend on the aggregate ("paid orders only"). Filter after you group when it does ("regions with revenue above 100"). Spark calls the second one .filter() on the grouped result as well. The SQL name HAVING is not a method you need.

08 — Joins

Same `how`, one spelling change

orders.join(customers, on="customer_id", how="left")

orders.join(customers, on="customer_id", how="left")

That left join:

order_id	customer_id	name	tier	amount
1	C1	Ada	gold	120.0
2	C2	Lin	silver	45.5
3	C1	Ada	gold	200.0
4	C3	Mo	bronze	null
5	C2	Lin	silver	80.0
6	C4	null	null	15.0

C4 has an order and no customer row, so name and tier are null. Priya (C5) has a customer row and no orders, so a left join from orders drops her. An inner join would also drop order 6. A right join, kept from the customer side, would keep Priya and drop C4.

Different key names:

orders.join(customers, left_on="customer_id", right_on="id", how="left")

orders.join(customers, orders.customer_id == customers.id, how="left")

Intent	Polars `how`	PySpark `how`
Matching keys only	`"inner"`	`"inner"`
Keep every left row	`"left"`	`"left"`
Keep every right row	`"right"`	`"right"`
Keep every row	`"full"`	`"full"`
Left rows that match, no extra columns	`"semi"`	`"left_semi"`
Left rows that do not match	`"anti"`	`"left_anti"`
Every combination	`"cross"`	use `.crossJoin(other)`

Semi join against gold customers keeps orders 1 and 3 (Ada) and adds no name column. Anti join against the whole customer table keeps only order 6 (C4).

gold = customers.filter(pl.col("tier") == "gold")
orders.join(gold, on="customer_id", how="semi")
orders.join(customers, on="customer_id", how="anti")

gold = customers.filter(sp.col("tier") == "gold")
orders.join(gold, on="customer_id", how="left_semi")
orders.join(customers, on="customer_id", how="left_anti")

Stacking frames vertically is pl.concat([a, b], how="vertical") or pl.concat([a, b], how="diagonal_relaxed") when columns differ. Spark's match is a.unionByName(b) and a.unionByName(b, allowMissingColumns=True).

09 — Strings and dates

A namespace in Polars, a function in Spark

Polars parks this work on the expression: .str and .dt. Spark parks it in sp.

orders.select(
    pl.col("note").str.to_uppercase().alias("upper"),
    pl.col("note").str.to_lowercase().alias("lower"),
    pl.col("note").str.len_chars().alias("n_chars"),
    pl.col("note").str.contains("gift").alias("has_gift"),
    pl.col("note").str.starts_with("gift").alias("starts_gift"),
    pl.col("note").str.slice(0, 4).alias("prefix"),
    pl.col("note").str.replace("gift", "present", literal=True).alias("edited"),
    pl.col("note").str.split(" ").alias("tokens"),
    pl.col("note").str.strip_chars().alias("trimmed"),
)

orders.select(
    sp.upper(sp.col("note")).alias("upper"),
    sp.lower(sp.col("note")).alias("lower"),
    sp.length(sp.col("note")).alias("n_chars"),
    sp.col("note").contains("gift").alias("has_gift"),
    sp.col("note").startswith("gift").alias("starts_gift"),
    sp.substring(sp.col("note"), 1, 4).alias("prefix"),
    sp.regexp_replace(sp.col("note"), "gift", "present").alias("edited"),
    sp.split(sp.col("note"), " ").alias("tokens"),
    sp.trim(sp.col("note")).alias("trimmed"),
)

order_id	n_chars	has_gift	tokens
1	9	true	gift, rush
2	8	false	standard
3	4	true	gift
4	null	null	null
5	4	false	rush
6	13	true	standard, gift

Indexes start at different numbers

Polars str.slice(0, 4) takes four characters from the start. Spark substring(col, 1, 4) does the same job, and the start position is 1, not 0. The same trap shows up on arrays: Polars list.get(0) is the first element, Spark getItem(0) is also the first, but element_at is 1-based.

Glue columns together:

pl.concat_str([pl.col("region"), pl.col("status")], separator="-")
sp.concat_ws("-", sp.col("region"), sp.col("status"))

Order 1 becomes APAC-paid.

Dates:

orders.select(
    pl.col("order_date").dt.year().alias("year"),
    pl.col("order_date").dt.month().alias("month"),
    pl.col("order_date").dt.day().alias("day"),
    pl.col("order_date").dt.truncate("1mo").alias("month_start"),
)

orders.select(
    sp.year(sp.col("order_date")).alias("year"),
    sp.month(sp.col("order_date")).alias("month"),
    sp.dayofmonth(sp.col("order_date")).alias("day"),
    sp.date_trunc("month", sp.col("order_date")).alias("month_start"),
)

Every row in the sample is year 2026. Months are 1, 1, 2, 2, 3, 3. Parse a string column with pl.col("order_date").str.to_date("%Y-%m-%d") or sp.to_date("order_date"). Spark's default pattern already accepts yyyy-MM-dd.

10 — Windows

Aggregate without collapsing the rows

A group-by turns six orders into three regions. A window keeps six orders and stamps a group answer onto each one.

(
    orders
    .sort("order_date")
    .with_columns(
        pl.col("amount").sum().over("region").alias("region_revenue"),
        pl.col("amount")
          .rank(method="dense", descending=True)
          .over("region")
          .alias("rank_in_region"),
        pl.col("amount").shift(1).over("customer_id").alias("prev_amount"),
    )
)

region = Window.partitionBy("region")
region_ordered = Window.partitionBy("region").orderBy(sp.col("amount").desc_nulls_last())
by_customer = Window.partitionBy("customer_id").orderBy("order_date")
 
(
    orders
    .withColumn("region_revenue", sp.sum("amount").over(region))
    .withColumn("rank_in_region", sp.dense_rank().over(region_ordered))
    .withColumn("prev_amount", sp.lag("amount", 1).over(by_customer))
)

Sorted by order date:

order_id	customer	region	amount	region_revenue	Polars rank	prev_amount
1	C1	APAC	120	335	2	null
2	C2	EMEA	45.5	125.5	2	null
3	C1	APAC	200	335	1	120
4	C3	AMER	null	null	null	null
5	C2	EMEA	80	125.5	1	45.5
6	C4	APAC	15	335	3	null

prev_amount follows the customer through time, which is why C1's second order sees 120 and C2's second order sees 45.5. The first order for each customer has no previous row.

Rank and nulls

Polars rank() leaves a null amount unranked, so order 4 stays null. Spark dense_rank() always assigns a number. desc_nulls_last() puts that null last in AMER, at rank 1, because it is the only row. Do not compare rank columns across the two engines until you have decided where nulls sit.

Running totals and "previous row" need an order. A plain sum over the region does not.

Goal	Polars	PySpark
Sum inside a group, keep rows	`pl.col("amount").sum().over("region")`	`sp.sum("amount").over(Window.partitionBy("region"))`
Previous row	`pl.col("amount").shift(1).over("customer_id")`	`sp.lag("amount", 1).over(w)`
Next row	`pl.col("amount").shift(-1).over(...)`	`sp.lead("amount", 1).over(w)`
Row number	`pl.int_range(1, pl.len() + 1).over("region")` after a sort, or `pl.col("amount").rank("ordinal")`	`sp.row_number().over(w)`
Dense rank	`.rank(method="dense", descending=True)`	`sp.dense_rank().over(w)`
Rank with gaps	`.rank(method="min", descending=True)`	`sp.rank().over(w)`
Share of group	`pl.col("amount") / pl.col("amount").sum().over("region")`	`sp.col("amount") / sp.sum("amount").over(w)`

A running sum in Spark is a window with a frame, not a bare sum().over(partition):

running = (
    Window.partitionBy("customer_id")
    .orderBy("order_date")
    .rowsBetween(Window.unboundedPreceding, Window.currentRow)
)
sp.sum("amount").over(running)

Polars expresses that as pl.col("amount").cum_sum().over("customer_id") once the frame is sorted inside the group. Sort first, then call cum_sum, or the running order is whatever order the rows already have.

11 — Lists and a few extras

After you split a string

split produces a list. Polars calls it list. Spark calls it an array.

(
    orders
    .with_columns(pl.col("note").str.split(" ").alias("tokens"))
    .with_columns(
        pl.col("tokens").list.len().alias("n_tokens"),
        pl.col("tokens").list.get(0).alias("first_token"),
        pl.col("tokens").list.contains("gift").alias("has_gift"),
    )
)

(
    orders
    .withColumn("tokens", sp.split(sp.col("note"), " "))
    .withColumn("n_tokens", sp.size(sp.col("tokens")))
    .withColumn("first_token", sp.col("tokens").getItem(0))
    .withColumn("has_gift", sp.array_contains(sp.col("tokens"), "gift"))
)

One row per token:

orders.with_columns(pl.col("note").str.split(" ").alias("token")).explode("token")

orders.withColumn("token", sp.explode(sp.split(sp.col("note"), " ")))

Order 1 becomes two rows, gift and rush. A null note disappears in Spark's explode and stays as a null token if you use explode_outer. Polars explode keeps a null list as a null row.

12 — A full pipeline

The shape you will write every week

Paid orders only. Treat a missing amount as zero. Add 10% tax. Attach the customer. Call an unmatched customer "unknown". Sum the taxed amount by region and tier.

(
    orders
    .filter(pl.col("status") == "paid")
    .with_columns(pl.col("amount").fill_null(0))
    .with_columns((pl.col("amount") * 1.1).round(2).alias("amount_taxed"))
    .join(customers, on="customer_id", how="left")
    .with_columns(pl.col("tier").fill_null("unknown"))
    .group_by("region", "tier")
    .agg(
        pl.len().alias("n_orders"),
        pl.col("amount_taxed").sum().round(2).alias("taxed_revenue"),
    )
    .sort("region", "tier")
    .collect()
)

(
    orders
    .filter(sp.col("status") == "paid")
    .withColumn("amount", sp.coalesce(sp.col("amount"), sp.lit(0.0)))
    .withColumn("amount_taxed", sp.round(sp.col("amount") * 1.1, 2))
    .join(customers, on="customer_id", how="left")
    .withColumn("tier", sp.coalesce(sp.col("tier"), sp.lit("unknown")))
    .groupBy("region", "tier")
    .agg(
        sp.count("*").alias("n_orders"),
        sp.round(sp.sum("amount_taxed"), 2).alias("taxed_revenue"),
    )
    .orderBy("region", "tier")
    .show()
)

region	tier	n_orders	taxed_revenue
APAC	gold	2	352.0
APAC	unknown	1	16.5
EMEA	silver	1	88.0

Ada's two paid orders are 120 and 200, times 1.1, which is 352. C4 is paid, has no customer, and contributes 16.5 in APAC under "unknown". The pending and refunded orders never enter. Read it top to bottom: each line is one verb from this post.

13 — Mistakes that look like bugs

Check this list before you blame the engine

The plan never ran

You built a LazyFrame or a Spark DataFrame and printed the object. Call .collect() or .show().

& without parentheses

Both libraries need (col > 1) & (col < 10).

A new column used in the same call

amount_taxed does not exist for the other expressions in that with_columns or withColumns. Add another call.

Null comparisons

amount > 0 drops nulls. It does not label them. Use is_null / isNull or fill first.

Sum of an empty group

Polars agg sum is 0. Spark sum is null. Means are null in both.

Distinct count of nulls

n_unique() counts null. countDistinct does not.

Spark substring starts at 1

Polars slice offsets start at 0.

Semi join names

Polars how="semi" and how="anti". Spark how="left_semi" and how="left_anti".

Window with no order

lag, lead, row_number, and running sums need .orderBy inside the Spark window, and a sort before shift or cum_sum in Polars.

collect on a large Spark job

.show() and .write stay distributed. .collect() ships the whole result to one machine.

14 — Pocket map

Keep this next to the editor

You want	Polars LazyFrame	PySpark
A column	`pl.col("amount")`	`sp.col("amount")`
A constant	`pl.lit(0)`	`sp.lit(0)`
Rename expression	`.alias("revenue")`	`.alias("revenue")`
Keep some columns	`.select(...)`	`.select(...)`
Add columns	`.with_columns(...)`	`.withColumn` / `.withColumns`
Keep rows	`.filter(...)`	`.filter(...)` or `.where(...)`
In a set	`.is_in([...])`	`.isin([...])`
Between	`.is_between(0, 10)`	`.between(0, 10)`
Is null	`.is_null()`	`.isNull()`
If / else	`pl.when(c).then(a).otherwise(b)`	`sp.when(c, a).otherwise(b)`
Fill null	`.fill_null(0)`	`sp.coalesce(col, sp.lit(0))`
Group	`.group_by(...).agg(...)`	`.groupBy(...).agg(...)`
Row count	`pl.len()`	`sp.count("*")`
Join	`.join(other, on="id", how="left")`	`.join(other, on="id", how="left")`
Sort	`.sort("amount", descending=True)`	`.orderBy(sp.col("amount").desc())`
Distinct keys	`.unique(subset=["id"])`	`.dropDuplicates(["id"])`
Upper case	`.str.to_uppercase()`	`sp.upper(col)`
Contains	`.str.contains("gift")`	`col.contains("gift")`
Year	`.dt.year()`	`sp.year(col)`
Window	`expr.over("region")`	`expr.over(Window.partitionBy("region"))`
Previous row	`.shift(1).over("id")`	`sp.lag("amount", 1).over(w)`
Run it	`.collect()`	`.show()`
Inspect	`.explain()`	`.explain()`

PolarsPySparkLazy plansExpressions

How to practise

Rebuild the six-row table in both libraries. Turn one step of the pipeline widget off, predict the table, then look. When your prediction matches for filter, with-columns, join, and group-by, you can read either codebase. The rest of the map is spelling.

Two libraries, one habit#

Orders and customers#

pl.col and sp.col#

Select, add, rename, drop#

Filter, sort, distinct, limit#

When, otherwise, and null#

One row per group#

Same how, one spelling change#

A namespace in Polars, a function in Spark#

Aggregate without collapsing the rows#

After you split a string#

The shape you will write every week#

Check this list before you blame the engine#