DataFrame API serialize columns to a JSON string column (#26185)

[cj-zhukov]

Which issue does this PR close?

• Closes #DataFrame API: serialize columns to a JSON string column #26185.

Rationale for this change

There is no way to serialize a struct, or a set of columns, into a JSON string column without collect()-ing the DataFrame and rebuilding it. Spark has to_json(struct) and Polars has struct.json_encode(). This PR adds a Spark-compatible to_json to datafusion-spark so the conversion stays a lazy projection and runs batch by batch.

use datafusion_spark::expr_fn::to_json;

df.with_column("json", to_json(&["foo", "bar"]))?;   // {"foo":1,"bar":"bak"}
df.with_column("json", to_json(col("event")))?;      // existing struct column

SELECT to_json(named_struct('foo', foo, 'bar', bar)) AS json FROM t;

What changes are included in this PR?

• datafusion/spark/src/function/json/to_json.rs: ToJson scalar UDF (Struct -> Utf8). Serializes each batch with the arrow-json row encoder; NULL fields are omitted (Spark ignoreNullFields default), a NULL struct yields NULL, nested structs/lists are handled recursively. Non-struct input is rejected at plan time.
• IntoJsonStruct trait + expr_fn::to_json(input): accepts either a struct Expr or a list of column names, which are packed with named_struct keyed by their unqualified names. The result is a plain Expr, so with_column needs no changes.
• datafusion/spark/src/function/json/mod.rs: registration (functions(), expr_fn).
• datafusion/core/Cargo.toml: datafusion-spark as a dev-dependency for the DataFrame tests.

What is the testing strategy for this PR?

• Unit tests in to_json.rs (array/scalar input, nulls, nested types, escaping, invalid input, name packing incl. qualified names).
• datafusion/core/tests/dataframe/mod.rs: with_column_to_json (add, overwrite, col(..) on a struct column, SQL equivalence, plan-time rejection) and with_column_to_json_edge_cases (null handling, constant input, nested struct + list, escaping/UTF-8, qualified names, empty input, empty column list).

Are there any user-facing changes?

Yes: new to_json function in datafusion-spark (SQL via with_spark_features() / register_udf, DataFrame via datafusion_spark::expr_fn::to_json). No changes to existing APIs.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.16188% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.79%. Comparing base (4e6e564) to head (7b547bc).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/spark/src/function/json/to_json.rs	86.01%	9 Missing and 44 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26187      +/-   ##
==========================================
+ Coverage   82.77%   82.79%   +0.01%     
==========================================
  Files        1148     1150       +2     
  Lines      451075   452027     +952     
  Branches   451075   452027     +952     
==========================================
+ Hits       373397   374274     +877     
- Misses      54930    54933       +3     
- Partials    22748    22820      +72     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.