Function IDs

[hannes]

I tried mapping DuckDB query plans to substrait and it seems like scalar and aggregate functions need to be in the extension definition and are then referred to by an ID. Is this correct?

From the documentation it also seems like those function IDs are supposed to be system-specific, is that correct, too?

[jacques-n]

Arg, totally missed this question, sorry about that. I guess it doesn't show up in the slack alerts.

Function ids themselves are just local to a particular plan and are a way of normalizing the actual function mapping which maps to a canonical location (e.g. uri) and key (just been updated to the name of the function plus the input argument types, used to be an integer offset).

The goal is for the project itself to declare a common set of frequently functions signatures but then people should be able to also define their own function uris that hold additional custom functions signatures. For example, substrait has an aggregate function avg(i8) => i64?. Some systems prefer to use a floating point type for average and may create another version which is avg(i8) => fp64. The specification allows these alternative implementations. A substrait producer would potentially identify the available set of maps that a consumer has by probing it and finding out which subset of functions it implements, including this alternative version of average.

[hannes]

Thanks Jacques for your extensive clarification on this. CCing @cpcloud for further offline discussion.

[hannes]

As an added note, the function question is absolutely a problem Substrait creates. By insisting on precise definitions, you create a problem that you have no clear path on how to solve yet.

[cpcloud]

I think there's a bit of a disconnect here on what's needed "right now" versus what we may need in the future. Function specification is a challenging topic, clearly.

Since there are producers and consumers in active development with needs met by a much simpler function specification, how about we simplify the specification until there are producers/consumers that actually meet the criteria for needing something with the current level of complexity.

I believe that we should onboard as many systems as reasonably possible and get people building things ASAP to validate the specification. Currently, the people building those things are finding it rather challenging to handle function calls in a compliant way.

For example, I'm not sure we have a producer under active development that is distributing partial aggregate plans to N possibly different consumers, while we do have multiple producer/consumer pairs that will not need this functionality for quite awhile (possibly ever).

Since Substrait is a new effort with a lot of value to add, I think we should focus on getting developers on-boarded to Substrait as quickly as possible.

Without doing this, I think that:

1. developers will simply ignore the function spec because they don't need it for their use case. This might be fine for the use case, but certainly limits the utility of Substrait and leads to a bunch of non-spec-compliant code.
2. developers will not use Substrait, because while it may solve a problem they have, it's too difficult to adhere to the function spec with their existing system. @jacques-n as you said, there are already many existing systems that add a lot of value and behave in ways that cannot be reasonably changed.

I think we have to make it easier for existing systems to adhere to substrait, which I believe means we need to simplify the function spec and add depth as the need arises.

[westonpace]

I think I am confused on the perceived burden here.

how about we simplify the specification until there are producers/consumers that actually meet the criteria for needing something with the current level of complexity.

Given the simplification we have already done there is no need for a producer or consumer to read the functions yaml so I'm not sure what could be simplified further.

As an added note, the function question is absolutely a problem Substrait creates. By insisting on precise definitions, you create a problem that you have no clear path on how to solve yet.

If a system doesn't want to solve that problem then there is an alternative. Specify all functions as belonging to your own extension. There is no need to actually write a YAML for it. Just make up some URL and then as long as your producers and consumers internally agree on the meaning of the names you should be fine.

When you want interoperability with other systems then it seems that there would need to be some level of agreement on the meaning behind those names.

developers will simply ignore the function spec because they don't need it for their use case. This might be fine for the use case, but certainly limits the utility of Substrait and leads to a bunch of non-spec-compliant code.

I guess I fall into this camp. I think it's fine. You said it yourself that many people don't need this yet. People will not want to do the work for interoperability until there is a need for it. Once there is a need they can do the work to be compliant. The functions.yaml isn't really going to lead to spec-compliant code by itself anyways. You would need a suite of integration tests to truly get to something you can trust.

I think we have to make it easier for existing systems to adhere to substrait, which I believe means we need to simplify the function spec and add depth as the need arises.

That said, I do think it is possible to at least agree on function names, arity, and perhaps even argument and return types.

Isn't that exactly what the function spec is currently doing? What could be simplified out?

[jacques-n]

I think there is some ambiguity around the current specification. This is probably largely due to weak documentation and bad naming. As such, I've tried to address both in #76. By and large, the capability you appear to seek already exists in the specification. To make it clearer, I've posted a patch to provides additional documentation as well as an example unknown.yaml extension file that exposes how one could express a simple set of functions that are type opaque.

I've also added specific clarifications in the function name mapping specification that clarifies that any function that is a single-impl function can be referenced using the simple name (without any argument type additions):

If there is only one impl, the name may be defined at the top-level definition of the object. If there are multiple impls, the name is a composite string that includes the top-level name (e.g. add) followed by a compound string that describes the types of the arguments (e.g. add:i8_i8). In the case there is one impl, the multiple impl naming scheme is also supported.

[westonpace]

Note that missing functionality can often be supplied by plan transformations (sometimes at a cost to performance) as well. For example, system A might support "add with saturating overflows" and system B may only support "add with null overflows". If the producer requests "add with saturating overflows" then a transformation library could write a plan for system B that does a coalesce after the add.

There's also been some discussion of "optional behavior". For example, a producer might say "add and I don't care how overflow is handled" and both system A and system B could consume the plan and the producer has acknowledged and clearly communicated the behavior may depend on the system.

An analogue might be something like instruction sets. x86 is a well known instruction set that many processors support. There are standard extensions like SSE4.2 that are multi-vendor but only provided by newer or more extensive processors. And there are extensions like FMA4 which are, at the moment, single-vendor only.

[cpcloud]

I do think the instruction sets analogy could work, but personally it would make me sad if this is how Substrait evolved.

The problem with instruction sets is that effectively a single vendor owned the specification. I think we're explicitly trying to avoid such a scenario and get more "vendors" (keeping the terminology from the analogy) involved so that one voice doesn't dominate the outcome.

That said, I do think it is possible to at least agree on function names, arity, and perhaps even argument and return types.

[jacques-n]

The documentation was updated in #76 and people will hopefully find answers in the extensions docs as well as the binary serialization docs.