Define custom resource request schema

[buchanae]

There's no place to request some types of resources:

• GPUs (architecture, brand, model, memory, number of cores?)
• Disk type
• CPU architecture
• AWS Spot Market bidding strategy
• Cloud/cluster-specific resources such as specialized hardware (e.g. Tensor Processing Unit)

Also, some existing concepts in the Resources schema are not clear

• zones: not a consistent concept across clusters, e.g. in GCE there are multiple global regions, each with multiple zones.
• preemptible does not fully capture the concept in AWS

A proposed solution is to remove some concepts from the schema, such as preemptible and zones, and add a new field of type map[string]string to allow more flexible, implementation-specific resource requests.

[pditommaso]

My feeling is that this kind of requests (eg. CPU architecture, spot instance, bidding strategy, etc) should be delegated to a queue concept which could be (optionally) specified by a task.

[geoffjentry]

Something we've run into over time is that there's a tension between a desire for generic key/values that an execution backend can feel free to interpret and people's desire for something to work the same no matter where they go. This is less bad for concepts which don't map cleanly at all but once you start getting to things which are semi-shared it becomes problematic.

If I had a good solution to this problem I'd have solved it already, but worth pointing out.

[buchanae]

My feeling is that this kind of requests (eg. CPU architecture, spot instance, bidding strategy, etc) should be delegated to a queue concept which could be (optionally) specified by a task.

If using queue names with a large workflow, would each step need to define its queue? If you moved that workflow to another cluster, would you have to edit all those queue names?

I guess, alternatively, one might argue that it's the workflow engine's responsibility to map workflow task resource requests into TES queue names. That has some tradeoffs:

• requires administrators to manage a set of queues for all types of tasks.
• requires the workflow engine to know details about the TES service's resources, which makes it more difficult to point a workflow engine at an arbitrary TES service.
• adds an extra indirection for understanding what a task actually requested, which adds a little more work to, for example, auditing a history of tasks to understand what resources were used.
• not all tasks are driven by an intelligent workflow engine. Ad hoc clients would need to know how to use queues effectively too.

Something we've run into over time is that there's a tension between a desire for generic key/values that an execution backend can feel free to interpret and people's desire for something to work the same no matter where they go. This is less bad for concepts which don't map cleanly at all but once you start getting to things which are semi-shared it becomes problematic.

This all reminds me a bit of XML namespaces, where you can reference the vocabulary you'll be using for a set of terms. I wonder if we can learn something from that.

[pditommaso]

If using queue names with a large workflow, would each step need to define its queue? If you moved that workflow to another cluster, would you have to edit all those queue names?

Won't be the same problem having to define for each task the CPU architecture, spot/on-demand, bidding strategy, etc, only much uglier?

Also the queue is a concept available in any batch scheduler, including the new generation, I mean for example AWS Batch.

To avoid to specify the same queue for each task, it could be possible to define a default queue at workflow or executor level. Then only a subset of tasks would need to specify a different one.

It's realistic to expect that some class of resources to only be available in some cluster architecture, for example spot instances are only available in a cloud cluster, while do not apply to legacy batch schedulers. For this reason, I think having such level of indirection would make the task definition, and begin so also the workflow, more portable.

[buchanae]

@pditommaso good points. Would the queue field be a single string? Would the Resources type still exist?

[pditommaso]

One possibility is to add queue as a string field in the Resources type and remove preemptible and zones.

[kellrott]

Introducing the idea of queues seems a little problematic. Assuming a user is on a AWS batch backed system, but they are not the sys-admin, if there was no GPU based queue set up by their admin, there is no way to create a GPU machine queue (at least not defined in the common API) and issue jobs on it.

By providing a vendor tag field, there is at least a possible mechanism to issue directives that the system then uses when it tries to identify resources.

Basically, I feel like the idea of queues essentially takes the problem of custom resource requests and shoves it into a non-standardized API. Having a custom tag field in the TES schema may be non-standard, but its a matter of figuring out which vendor strings you want to append vs figuring out which additional API endpoint you would have to call out to in order to create a new queue.

[pditommaso]

By providing a vendor tag field, there is at least a possible mechanism to issue directives that the system then uses when it tries to identify resources.

I'm not against a tag field but I'm trying to see it in practice. For example, let's say there's a tag instanceType that allow to specify a cloud VM type. How that would work in practice? Each cloud provider defines it's own VM type identifier, hence there should be a instance type tag for each provider, shouldn't be?

[buchanae]

Tags have values, so it might look like:

{
  "tags": {
    "gce-instance-type": "n1-standard-1"
  }
}

[buchanae]

Oh, sorry, I misunderstood your question. You're asking whether we should have gce-instance-type and aws-instance-type vs just instance-type, right?

It's more difficult to find a common ground between all the providers, so I'd lean towards have provider-specific terms.

[pditommaso]

Yes, exactly there should gce-instance-type, aws-instance-type, azure-instance-type .. etc.

Not only for each cloud platform but also for different services. To remain in the AWS Batch example, a Batch job would not be able to use a aws-instance-typetag but it would require a specific tag eg. aws-batch-queue, etc.

Also it would the client create a request including all the tags, independently the target platform/service?

[geoffjentry]

I might be misunderstanding but if we're backing e.g. AWS aware fields into TES doesn't that kinda limit who can be implementing it?

[buchanae]

Here's what I propose:

1. Remove the zones field: it's not clear what it means, and is platform-specific.
2. Remove the preemptible field: it's not a universal concept.
3. Add Resources.tags field for resources which are implementation-specific.

There was some discussion about whether to reuse the Task.tags field for generic resource requests. The thought is that it's better not to mix concerns. Task.tags are for user defined tags, Resources.tags are for implementation-defined resources.

There's an open question about the type of this field.

1. map<string, string> is the easy choice. Implementations would be required to interpret and validate the string value.

2. protobuf's Any type is interesting. It allows the value to include a URL naming its type, allowing this field to use well-defined schemas which are defined externally. However, given our experience with other complex protobuf types, we (the Funnel team) worry that we might run into issues with limited or unfriendly support in protobuf libraries.

We'd also like to collect some use cases about how people are actually using resources. @geoffjentry and @pditommaso can you share any details about the types of resources your users are requesting?

[pditommaso]

The most common ones with legacy batch schedulers are:

• cpus
• queue(s)
• memory needed
• storage (disk) needed
• parallel environment
• run time

The problem that each schedulers implement all kind of perversions with the parameters, so it's very difficult to have an exhaustive list. For example generally the request for each of the above resources can be soft (better to fulfil but not mandatory) or hard (the scheduler enforce it). Also for the memory you can specify to check the rss memory or the free memory or the virtual memory. With the cpus you can bind the execution to some specify cores (cpu affinity concept), etc.

What we do with NF is to provide an abstraction to the basic ones (cpus, memory, storage, time and queue) and leave the user to provide cluster specific settings via a free form option that is passed as it is to the underlying scheduler.