SNOW-3472759: support for to_polars() for snowpark -> polars interoperability by sfc-gh-mayliu · Pull Request #4288 · snowflakedb/snowpark-python

sfc-gh-mayliu · 2026-07-20T21:08:36Z

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

Fixes SNOW-3472759
Fill out the following pre-review checklist:
- I am adding a new automated test(s) to verify correctness of my new code
  - If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
- I am adding new logging messages
- I am adding a new telemetry message
- I am adding new credentials
- I am adding a new dependency
- If this is a new feature/behavior, I'm adding the Local Testing parity changes.
- I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
- If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines
Please describe how your code solves the related issue.

Please write a short description of how your code change solves the related issue.

This PR adds DataFrame.to_polars() to enable direct conversion from a Snowpark DataFrame to a Polars DataFrame/LazyFrame

Finalized benchmark evaluation: https://docs.google.com/document/d/1jABNlY-zNEMvQZKGg-wKdwu4r7-XpxIGnWE50Vjxiyg/edit?usp=sharing
Complementary Design doc: https://docs.google.com/document/d/1FKeSB5K9HIHG-aQwcP_2oPRmT7Pd4hcj03l8tictEPM/edit?usp=sharing
Following stacked PR for AST support: SNOW-3472759: to_polars() AST support #4290
Tests passed on sproc: https://buildkite.com/snowflake/snowflake--snowpark-python-sp-integ-tests/builds/160/tests?jid=019faa00-cf3e-4199-950e-0521dfec5c9b&tab=output

codecov-commenter · 2026-07-20T21:45:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.47%. Comparing base (2b9291a) to head (5ad5fd8).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4288      +/-   ##
==========================================
+ Coverage   95.26%   95.47%   +0.20%     
==========================================
  Files         171      172       +1     
  Lines       44720    44783      +63     
  Branches     7676     7685       +9     
==========================================
+ Hits        42603    42757     +154     
+ Misses       1300     1253      -47     
+ Partials      817      773      -44

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…to-polars

…lelize file open and read for parquet, to_arrow for plans with 1 query

…row_batches; 3. add usage notes in docstring

snowflake-security-bot

Snowflake Security Review

Security grade: A — Passed ✅

No security findings after adjudication. This PR passes the Snowflake Security Review.

📊 5 of 6 files (1 test skipped) · 11,645 lines reviewed · 4 candidates → 0 kept · retrieval: on

sfc-gh-yuwang · 2026-07-29T19:00:03Z

+        )
+    # The stream objects are owned by the returned LazyFrame; Polars closes them
+    # when the scan is materialized (or if the LazyFrame is discarded).
+    streams = _open_stage_files_parallel(


i have a question about this ingestion path.
Since this path is ingesting with a stream of parquet file,
if I call collect() two times, will the second collect() still be able to fetch anything since the stream is already consumed at that time?

Both session.file.get_stream and SnowflakeFile.open eagerly downloads/buffers, so the multiple Polars .collect() calls are safe to get the predicate/projection pruned data on CPU without needing to re-download. However, it's a Snowflake infra-limitation that network I/O pruning costs can't be saved due to the lack of HTTP Range GET support in session.file and SnowflakeFile

Tested with 1M rows: first .collect() takes 0.04s, second .collect() takes 0.03s — effectively the same cpu-decode cost with no additional network I/O.

this sounds good!

sfc-gh-yuwang · 2026-07-29T19:02:08Z

+    rid = uuid.uuid4().hex[:12]
+    stage = df._session.get_session_stage().rstrip("/")
+    sub = f"{stage}/{sub_prefix}/{rid}/"
+    df.write.copy_into_location(


it appears like the uploaded parquet file is never removed, do we need a garbage cleaning mechanism to make sure the stage is not filled when user invoke this ingestion route multiple times?

get_session_stage() returns a temporary stage that Snowflake auto-drops on session close, so no need to recycle here

I am thinking about a scenario in which user have a long running pipeline that call to_polars() in this ingestion route, in that scenario, the stage will keep accumulating uploaded parquet files. Do we need to worry about this scenario or is this not a expected usage?

Good question, discussed offline. TLDR: long-lived sessions exist but are rare, and there's no storage limit on temp stages. sproc also rejects REMOVE {file} calls, so a clean up would result in inconsistency between local and server environments. We will rely on temp stages' property to clean up after itself, similar to existing Snowpark patterns

sfc-gh-yuwang · 2026-07-29T19:11:48Z

+        overwrite=True,
+        statement_params=statement_params,
+    )
+    rows = df._session.sql(f"LIST '{sub}'").collect(statement_params=statement_params)


I remember that List would not return fully qualified stage name, should we make it fully qualified before return?

Good point, LS returns stage-relative paths and we should make it fully qualified. We already have the fully-qualified sub prefix at hand, so the fix is to extract just the filename from the LIST result and prepend sub instead. Fix incoming.

snowflake-security-bot

Snowflake Security Review

Security grade: A — Passed ✅

No security findings after adjudication. This PR passes the Snowflake Security Review.

📊 5 of 6 files (1 test skipped) · 11,646 lines reviewed · 4 candidates → 0 kept · retrieval: on

…to-polars

snowflake-security-bot

Snowflake Security Review

Security grade: A — Passed ✅

No security findings after adjudication. This PR passes the Snowflake Security Review.

📊 5 of 6 files (1 test skipped) · 11,623 lines reviewed · 2 candidates → 0 kept · retrieval: on

snowflake-security-bot

Snowflake Security Review

Security grade: A — Passed ✅

No security findings after adjudication. This PR passes the Snowflake Security Review.

📊 5 of 6 files (1 test skipped) · 11,623 lines reviewed · 2 candidates → 1 kept · retrieval: on

add support for to_polars() interop

d4596e7

sfc-gh-mayliu requested review from a team as code owners July 20, 2026 21:08

sfc-gh-mayliu requested review from sfc-gh-aling, sfc-gh-bkogan and sfc-gh-yixie July 20, 2026 21:08

test coverage

b293da5

sfc-gh-mayliu requested a review from a team as a code owner July 20, 2026 23:20

sfc-gh-mayliu requested review from sfc-gh-helmeleegy and sfc-gh-jkew July 20, 2026 23:20

graphite-app Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread src/snowflake/snowpark/dataframe.py Outdated

sfc-gh-mayliu requested a review from a team July 21, 2026 18:35

sfc-gh-mayliu mentioned this pull request Jul 21, 2026

SNOW-3472759: to_polars() AST support #4290

Open

9 tasks

sfc-gh-mayliu commented Jul 21, 2026

View reviewed changes

Comment thread src/snowflake/snowpark/dataframe.py Outdated

sfc-gh-mayliu added 5 commits July 22, 2026 11:27

align limit(0)

42d9671

use pl.filter for data correctness; predicate pushdown tbd

9d4a8b5

Merge remote-tracking branch 'origin/main' into SNOW-3472759-support-…

32d6c21

…to-polars

Provide both arrow and parquet-based to_polars interop choices; paral…

84b5f37

…lelize file open and read for parquet, to_arrow for plans with 1 query

1. expose max_workers as param; 2. consolidate eager arrow with to_ar…

6a5ed0a

…row_batches; 3. add usage notes in docstring

snowflake-security-bot Bot reviewed Jul 28, 2026

View reviewed changes

sfc-gh-yuwang reviewed Jul 29, 2026

View reviewed changes

return fully qualified stage path

54bd11c

snowflake-security-bot Bot reviewed Jul 29, 2026

View reviewed changes

sfc-gh-yuwang approved these changes Jul 29, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into SNOW-3472759-support-…

0256fe0

…to-polars

sfc-gh-aling reviewed Jul 29, 2026

View reviewed changes

Comment thread src/snowflake/snowpark/_internal/polars_backend.py Outdated

Comment thread src/snowflake/snowpark/dataframe.py Outdated

Comment thread src/snowflake/snowpark/dataframe.py

use 1.55.0

4390645

sfc-gh-mayliu added 2 commits July 30, 2026 10:39

styling/maintainability clean up

5921695

Merge remote-tracking branch 'origin/main' into SNOW-3472759-support-…

5ad5fd8

…to-polars

sfc-gh-mayliu force-pushed the SNOW-3472759-support-to-polars branch from 13c8011 to 5ad5fd8 Compare July 30, 2026 17:44

sfc-gh-mayliu requested review from a team and removed request for sfc-gh-helmeleegy July 30, 2026 17:46

snowflake-security-bot Bot reviewed Jul 30, 2026

View reviewed changes

sfc-gh-aling approved these changes Jul 30, 2026

View reviewed changes

sfc-gh-mayliu merged commit cd6d20b into main Jul 30, 2026
51 of 54 checks passed

sfc-gh-mayliu deleted the SNOW-3472759-support-to-polars branch July 30, 2026 21:12

github-actions Bot locked and limited conversation to collaborators Jul 30, 2026

Uh oh!

Conversation

sfc-gh-mayliu commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

snowflake-security-bot Bot left a comment

Choose a reason for hiding this comment

Snowflake Security Review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snowflake-security-bot Bot left a comment

Choose a reason for hiding this comment

Snowflake Security Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snowflake-security-bot Bot left a comment

Choose a reason for hiding this comment

Snowflake Security Review

Uh oh!

snowflake-security-bot Bot left a comment

Choose a reason for hiding this comment

Snowflake Security Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sfc-gh-mayliu commented Jul 20, 2026 •

edited

Loading

codecov-commenter commented Jul 20, 2026 •

edited

Loading