Add Arrow IPC format options

[pepijnve]

Which issue does this PR close?

• Closes Support Configuring arrow_ipc::FileWriter via ArrowWriterOptions #8635

Rationale for this change

It can sometimes be useful to not spend CPU cycles compressing and decompressing data. The Arrow IPC write support currently only supports LZ4 compression even though the writer supports no compression and zstd. This PR adds config options to make the compression settings user configurable.

What changes are included in this PR?

• Add Arrow config options for compression type, compression level, and alignment
• Add glue code to pass config options to the file format implementation

What is the testing strategy for this PR?

Added SLTs to cover new options

Are there any user-facing changes?

Yes, additional options become available, but the default behaviour remains the same.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v55.1.0 (current)
       Built [  58.003s] (current)
     Parsing datafusion v55.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion v55.1.0 (baseline)
       Built [  55.668s] (baseline)
     Parsing datafusion v55.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.608s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 115.850s] datafusion
    Building datafusion-common v55.1.0 (current)
       Built [  32.635s] (current)
     Parsing datafusion-common v55.1.0 (current)
      Parsed [   0.063s] (current)
    Building datafusion-common v55.1.0 (baseline)
       Built [  32.790s] (baseline)
     Parsing datafusion-common v55.1.0 (baseline)
      Parsed [   0.063s] (baseline)
    Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.761s] 223 checks: 220 pass, 3 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field TableOptions.arrow in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:2971

--- failure enum_unit_variant_changed_kind: An enum unit variant changed kind ---

Description:
A public enum's exhaustive unit variant has changed to a different kind of enum variant, breaking possible instantiations and patterns.
        ref: https://doc.rust-lang.org/reference/items/enumerations.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_unit_variant_changed_kind.ron

Failed in:
  variant OutputFormat::ARROW in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:4225

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant ConfigFileType:ARROW in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:2950

     Summary semver requires new major version: 3 major and 0 minor checks failed
    Finished [  67.296s] datafusion-common
    Building datafusion-datasource-arrow v55.1.0 (current)
       Built [  40.400s] (current)
     Parsing datafusion-datasource-arrow v55.1.0 (current)
      Parsed [   0.007s] (current)
    Building datafusion-datasource-arrow v55.1.0 (baseline)
       Built [  40.135s] (baseline)
     Parsing datafusion-datasource-arrow v55.1.0 (baseline)
      Parsed [   0.008s] (baseline)
    Checking datafusion-datasource-arrow v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.079s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure unit_struct_changed_kind: unit struct changed kind ---

Description:
A public unit struct has been changed to a normal (curly-braces) struct, which cannot be constructed using the same struct literal syntax.
        ref: https://github.com/rust-lang/cargo/pull/10871
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/unit_struct_changed_kind.ron

Failed in:
  struct ArrowFormat in /home/runner/work/datafusion/datafusion/datafusion/datasource-arrow/src/file_format.rs:138
  struct ArrowFormat in /home/runner/work/datafusion/datafusion/datafusion/datasource-arrow/src/file_format.rs:138
  struct ArrowFormatFactory in /home/runner/work/datafusion/datafusion/datafusion/datasource-arrow/src/file_format.rs:80
  struct ArrowFormatFactory in /home/runner/work/datafusion/datafusion/datafusion/datasource-arrow/src/file_format.rs:80

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  81.542s] datafusion-datasource-arrow
    Building datafusion-ffi v55.1.0 (current)
       Built [  52.638s] (current)
     Parsing datafusion-ffi v55.1.0 (current)
      Parsed [   0.060s] (current)
    Building datafusion-ffi v55.1.0 (baseline)
       Built [  53.729s] (baseline)
     Parsing datafusion-ffi v55.1.0 (baseline)
      Parsed [   0.062s] (baseline)
    Checking datafusion-ffi v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.245s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 107.954s] datafusion-ffi
    Building datafusion-proto v55.1.0 (current)
       Built [  51.467s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.013s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  51.948s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.014s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.121s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 104.625s] datafusion-proto
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [  93.420s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.015s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [  93.000s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.016s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.105s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 188.894s] datafusion-sqllogictest

[codecov-commenter]

Codecov Report

❌ Patch coverage is 73.61111% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.77%. Comparing base (bc693cc) to head (4c91fe3).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/datasource-arrow/src/file_format.rs	58.06%	13 Missing ⚠️
datafusion/common/src/config.rs	88.23%	4 Missing ⚠️
datafusion/ffi/src/session/mod.rs	60.00%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #26155    +/-   ##
========================================
  Coverage   82.77%   82.77%            
========================================
  Files        1147     1147            
  Lines      450580   450968   +388     
  Branches   450580   450968   +388     
========================================
+ Hits       372946   373287   +341     
- Misses      54931    54956    +25     
- Partials    22703    22725    +22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[alamb]

Looks good to me -- thank you @pepijnve