Repository navigation
Add vector distance, array math, and array aggregate functions #21536
Description
Activity
- added a commit that references this issue
on Apr 10, 2026 - added 2 commits that reference this issue
on Apr 17, 2026 - added a commit that references this issue
on Apr 23, 2026 Given the DataFusion core is pretty large already, i wonder if we would be better off making this package of functions as a datafusion-contrib repo
Perhaps something like https://github.com/datafusion-contrib/datafusion-functions-json 🤔
At least until we have a larger community of poeple / committers who are willing to help wiht maintaining a new function library
I realize this is a bit late as we just merged #21542
But I am thinking that
datafusion-contrib/array_mathmight make a very nice function package 🤔Given the DataFusion core is pretty large already, i wonder if we would be better off making this package of functions as a datafusion-contrib repo
Perhaps something like https://github.com/datafusion-contrib/datafusion-functions-json 🤔
At least until we have a larger community of poeple / committers who are willing to help wiht maintaining a new function library
I realize this is a bit late as we just merged #21542
But I am thinking that
datafusion-contrib/array_mathmight make a very nice function package 🤔I feel the functions suggested here are simple enough that maintenance burden isn't too much a concern
Reacted by Andrew Lamb- added a commit that references this issue
on May 3, 2026 - added a commit that references this issue
on May 20, 2026 @crm26 I would like to help on this.
Hey @SubhamSinghal — thanks for reaching out and happy to collaborate however works.
Six functions left:
array_add,array_subtract,array_scale,array_sum,array_product,array_avg. The three already merged (#21542, #21861, #22013) lock in the per-function pattern Jefffrey's been reviewing.Heads up: I've been going serially (one PR open at a time) since alamb's original split request, so we don't pile work on reviewers. Three of my older PRs (#21363, #21353, #21367) are also still stale. Worth keeping the in-flight count modest between us — maybe 2 open at a time across both contributors?
How would you like to approach it?
Reacted by Subham Singhal@crm26 sound fine to me. You can specify which one you are working on. I can work on rest.
- added 3 commits that reference this issue
on May 22, 2026 Hey @SubhamSinghal — saw #22459, thanks for the fast turnaround.
array_subtractis essentially a copy ofarray_add(just operator swap), so happy for you to take that next too. I'll claimarray_scale(different signature — list + scalar — so it covers new territory and we don't collide). The three aggregates (array_sum,array_product,array_avg) are still open for whoever has bandwidth.Will leave any comments on #22459 directly if I spot anything as I read through.
7 remaining items
- added 2 commits that reference this issue
on May 23, 2026 - added a commit that references this issue
on May 26, 2026 - added a commit that references this issue
on May 26, 2026 - added a commit that references this issue
on Jun 1, 2026 - added a commit that references this issue
on Jun 4, 2026 - added a commit that references this issue
on Jun 8, 2026 - added a commit that references this issue
on Jun 8, 2026 - added a commit that references this issue
on Jun 11, 2026 Claiming
array_avg— the last unclaimed split function from this issue. Will template from the mergedarray_sum(#22542) using the same SQL aggregate NULL convention: skip NULL elements, empty array → NULL, all-NULL row → NULL. Result typeFloat64. Aliaslist_avgper thelist_sum/list_productpattern.- added a commit that references this issue
on Jun 27, 2026
Summary
This issue tracks adding vector math and array aggregate scalar functions to DataFusion. These close gaps versus DuckDB and LanceDB for vector search and array analytics workloads.
Replaces #21371 and #21376, which were requested to be split into function-per-PR submissions (per @alamb's review).
Functions
Vector math (with shared
vector_math.rsprimitives)cosine_distance(array, array) → float64array_cosine_similarityinner_product(array, array) → float64array_inner_productarray_normalize(array) → arrayArray element-wise math
array_add(array, array) → arrayarray_subtract(array, array) → arrayarray_scale(array, scalar) → arrayArray aggregate scalars
array_sum/list_sum(array) → numericlist_sumarray_product/list_product(array) → numericlist_productarray_avg/list_avg(array) → float64list_avgAlias fix
list_minArrayMin(parity with existinglist_maxonArrayMax)Submission plan
One PR per function, submitted serially. Each PR will reference this issue.
References