Repository navigation
Add support for lambda/higher order functions #14205
Description
Is your feature request related to a problem or challenge?
Some engines, like DuckDB and Clickhouse, supports lambda functions, like:
SELECT list_filter(numbers, x -> x > 40) as greater_than_40s FROM relationThere's already support for the syntax in sqlparser-rs
From #12206 (comment) by @jayzhan211
Activity
rkrishn7 commentedon Jan 27, 2025
Hello! I'm interested in working on this.
Agreed that tweaking the existing
ScalarUDFcode would be ideal as it would be least breaking. However, after poking around a bit, I have some concerns:A more prohibitive interface certainly seems like it would help implementation here, albeit at the expense of consumer flexibility.
Any thoughts/feedback would be much appreciated. Thanks!
gstvg commentedon Jan 29, 2025
Hi @rkrishn7, have you already started any work?
Since creating this issue, I’ve been working on a PoC to refine the proposal, and I managed to get it working. I’d like to continue with it if you’re OK with that—I probably should have assigned the issue to myself earlier, my bad 🤦🏻♂️! Anyway, many thanks for your interest and for your questionings, and I would really appreciate your input in futures questions (I expect many to arise!)
I added a new ScalarUDF method invoke_with_lambda_args that receives ScalarFunctionArgs instead of ScalarFunctionArgs, and has a default implementation calling invoke_with_args, returning an error if any arg is a lambda
This should preserve ordering of any arg.
But maybe we could add an
lambda_argumentstoScalarFunctionArgssimiltar to here, what do you think ?The lambda implementation should build a RecordBatch, likely using some other concrete argument, and pass it to the lambda PhysicalExpr evaluate method
That said, if you have any thoughts on the current interface, I’d love to hear your feedback. It's a
list_map([1,2], x -> x*2)udf, you can see the executed output here. Thanks! 🙏gstvg commentedon Jan 29, 2025
Lambda syntax only works with databricks dialect, proposed fix here apache/datafusion-sqlparser-rs#1686
rkrishn7 commentedon Jan 30, 2025
Hi @gstvg! No worries at all, please continue working on it since you've started!
Happy to take a look at the work you've done already and give feedback where it makes sense 👍🏾
gstvg commentedon Feb 8, 2025
take
alexanderbianchi commentedon Oct 9, 2025
@gstvg Is there an update on the status here?
gstvg commentedon Oct 11, 2025
Hi @alexanderbianchi. The code is already functional, mostly finished, but it ended up being a bigger change than I expected, ~3k LOC, some of it in core parts of DF. After I got stuck documenting and splitting it into smaller PRs, I stopped for a while.. but it's already been 6 months (!!). I'll start working on it again this weekend and hopefully we can make it into the 51.0 release 🙏
ahmed-mez commentedon Nov 12, 2025
Hi @gstvg, thanks a lot for working on this! Is there an update on the progress so far? I'm also interested in this feature.
gstvg commentedon Nov 12, 2025
Hi @ahmed-mez, I'm finishing the text that explains and justifies the path I took, as well as showing alternative approaches. I found out today that it cant make into the 51 release anymore, sorry about that. Last week I found a flaw in my code that would break existing code but I already fixed it, so now all that's left is the text
gstvg commentedon Nov 25, 2025
Hi @rkrishn7, my bad for taking too long. Based on your questions, I assume you thought a bit about the implementation details of this. What's your opinion on my approach and/or the alternatives? Thanks
@ahmed-mez @alexanderbianchi I opened a draft in #18921, if any of you have an opinion on it, if please leave a comment, thanks
derrley commentedon Feb 10, 2026
Looking forward to this!
LiaCastaneda commentedon Mar 19, 2026
@gstvg since we don't have the Proposal doc in the original PR anymore, do you think it's worth moving it here? so it doesn't get lost and others can review as well.
gstvg commentedon Mar 26, 2026
@LiaCastaneda I set it as the PR description there. I think that's better than here because it's very specific to that particular implementation, and greatly differs from other lambda support attempts, for example. WDYT?
But your suggestion made me note that the proposal here is very outdated, so at least I removed that part. Thanks
LiaCastaneda commentedon Mar 26, 2026
Ah, I was not aware it was in the PR description (while reviewing I guided myself mostly on the doc but PR description makes sense as well) thanks!
gstvg commentedon Apr 29, 2026
I'm closing this as #21679 got merged. Many thanks for everyone who showed interest on the feature ❤️
ahmed-mez commentedon Apr 29, 2026
Congrats & kudos to @gstvg and @LiaCastaneda for the massive work you've done to get this shipped !