Skip to content

[EPIC] Full lambda support #21172

Description

@gstvg

Some features haven't been implemented and others pulled out from #18921 #21679 to keep it from getting too large (>5k LOC) and should be implemented in follow on PR's:

Major features:

Minor features:

  • Named parameters like ScalarUDF already supports: func(a => 3)
  • Expr simplifier optimizations for LambdaVariable
  • ListView support in remove_list_null_values and HigherOrderUDF::clean_null_values
  • Optional ScalarUDFImpl methods not present on HigherOrderUDF(some are unlikely to be used and could be added on demand only):
    • simplify
    • with_updated_config
    • preimage
    • placement
    • evaluate_bounds
    • propagate_constraints
    • output_ordering
    • preserves_lex_ordering

Activity

  1. LiaCastaneda commented on Apr 13, 2026

    @LiaCastaneda
    Contributor

    @gstvg I think we should also include as a folloup adding HigherOrderUDF to the public user guide docs https://datafusion.apache.org/library-user-guide/functions/adding-udfs.html

  2. gstvg commented on Apr 14, 2026

    @gstvg
    ContributorAuthor

    @LiaCastaneda added, thanks

  3. LiaCastaneda commented on Apr 28, 2026

    @LiaCastaneda
    Contributor

    Opened a PR to implement any_match - #21903
    @ologlogn also opened a PR for array_filter - #21895

  4. LiaCastaneda commented on Apr 28, 2026

    @LiaCastaneda
    Contributor

    @gstvg if you don't have a PR/branch already I can open a PR for #21679 (comment), since I think it would simplify the code a bit as well

  5. gstvg commented on Apr 29, 2026

    @gstvg
    ContributorAuthor

    @LiaCastaneda Feel free to do it, yes, I haven't done nothing on that

  6. changed the title [-][DRAFT, EPIC] Full lambda support[/-] [+][EPIC] Full lambda support[/+] on Apr 29, 2026
  7. LiaCastaneda commented on May 18, 2026

    @LiaCastaneda
    Contributor

    @gstvg I opened a small PR #22326 to add the exact signature

  8. LiaCastaneda commented on May 19, 2026

    @LiaCastaneda
    Contributor

    @gstvg are you working on adding protobuf support? I can open a PR for it this week otherwise.

  9. gstvg commented on May 19, 2026

    @gstvg
    ContributorAuthor

    @LiaCastaneda I was working on exactly that today 😅 #22362 . Hope that it make into the current release 🤞

  10. LiaCastaneda commented on May 19, 2026

    @LiaCastaneda
    Contributor

    nice!

  11. Adam-Alani commented on Jun 1, 2026

    @Adam-Alani

    Hey, opened up a pr to add map_transform support, similar to array_transform here: #22689

  12. gstvg commented on Jun 2, 2026

    @gstvg
    ContributorAuthor

    thanks @Adam-Alani, added to the list

  13. LiaCastaneda commented on Sep 7, 2026

    @LiaCastaneda
    Contributor

    👋 I noticed structs as lambda parameters are not supported #25017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions