Pants: add `python_distribution` BUILD metadata for runners by cognifloyd · Pull Request #5907 · StackStorm/st2

cognifloyd · 2023-02-16T22:59:18Z

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

This PR adds the st2_runner_python_distribution() macro to the BUILD metadata for all of our runners.

Thanks to the macro, we only have to define 3 (or 4) parameters for each runner: runner_name, description, entry_points, and (for the python runner) zip_safe=False. I copied each of these from the setup.py file, which will become obsolete once pants can generate the setup.py file for us. For the description, I tried to keep the spacing fairly consistent, but there was enough variation in the wording that I did not delegate creating the description to the macro.

Under entry_points, I have "tagged" the st2common.runners.runner and the orquesta.expressions.functions categories as stevedore_namespaces. Once we switch to 2.16 and enable the pants.backend.experimental.python.framework.stevedore plugin, the stevedore plugin will, for tests, use this "tag" to look up which wheels implement a plugin in that stevedore namespace.

To facilitate comparison, I will add a comment to each of the files that shows the current setup.py entry_points.

cognifloyd · 2023-02-16T23:00:44Z

contrib/runners/action_chain_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "action-chain": "action_chain_runner.action_chain_runner",
+        },


st2/contrib/runners/action_chain_runner/setup.py

Lines 54 to 56 in 9d311c5

"st2common.runners.runner": [

"action-chain = action_chain_runner.action_chain_runner",

],

cognifloyd · 2023-02-16T23:01:17Z

contrib/runners/announcement_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "announcement": "announcement_runner.announcement_runner",
+        },


st2/contrib/runners/announcement_runner/setup.py

Lines 53 to 55 in 9d311c5

"st2common.runners.runner": [

"announcement = announcement_runner.announcement_runner",

],

cognifloyd · 2023-02-16T23:01:55Z

contrib/runners/http_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "http-request": "http_runner.http_runner",
+        },


st2/contrib/runners/http_runner/setup.py

Lines 53 to 55 in 9d311c5

"st2common.runners.runner": [

"http-request = http_runner.http_runner",

],

cognifloyd · 2023-02-16T23:02:17Z

contrib/runners/inquirer_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "inquirer": "inquirer_runner.inquirer_runner",
+        },


st2/contrib/runners/inquirer_runner/setup.py

Lines 53 to 55 in 9d311c5

"st2common.runners.runner": [

"inquirer = inquirer_runner.inquirer_runner",

],

cognifloyd · 2023-02-16T23:02:41Z

contrib/runners/local_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "local-shell-cmd": "local_runner.local_shell_command_runner",
+            "local-shell-script": "local_runner.local_shell_script_runner",
+        },


st2/contrib/runners/local_runner/setup.py

Lines 54 to 57 in 9d311c5

"st2common.runners.runner": [

"local-shell-cmd = local_runner.local_shell_command_runner",

"local-shell-script = local_runner.local_shell_script_runner",

],

cognifloyd · 2023-02-16T23:03:06Z

contrib/runners/noop_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "noop": "noop_runner.noop_runner",
+        },


st2/contrib/runners/noop_runner/setup.py

Lines 51 to 53 in 9d311c5

"st2common.runners.runner": [

"noop = noop_runner.noop_runner",

],

cognifloyd · 2023-02-16T23:04:08Z

contrib/runners/orquesta_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "orquesta": "orquesta_runner.orquesta_runner",
+        },
+        stevedore_namespace("orquesta.expressions.functions"): {
+            "st2kv": "orquesta_functions.st2kv:st2kv_",
+            "task": "orquesta_functions.runtime:task",
+            "basename": "st2common.expressions.functions.path:basename",
+            "dirname": "st2common.expressions.functions.path:dirname",
+            "from_json_string": "st2common.expressions.functions.data:from_json_string",
+            "from_yaml_string": "st2common.expressions.functions.data:from_yaml_string",
+            "json_dump": "st2common.expressions.functions.data:to_json_string",
+            "json_parse": "st2common.expressions.functions.data:from_json_string",
+            "json_escape": "st2common.expressions.functions.data:json_escape",
+            "jsonpath_query": "st2common.expressions.functions.data:jsonpath_query",
+            "regex_match": "st2common.expressions.functions.regex:regex_match",
+            "regex_replace": "st2common.expressions.functions.regex:regex_replace",
+            "regex_search": "st2common.expressions.functions.regex:regex_search",
+            "regex_substring": "st2common.expressions.functions.regex:regex_substring",
+            "to_human_time_from_seconds": "st2common.expressions.functions.time:to_human_time_from_seconds",
+            "to_json_string": "st2common.expressions.functions.data:to_json_string",
+            "to_yaml_string": "st2common.expressions.functions.data:to_yaml_string",
+            "use_none": "st2common.expressions.functions.data:use_none",
+            "version_compare": "st2common.expressions.functions.version:version_compare",
+            "version_more_than": "st2common.expressions.functions.version:version_more_than",
+            "version_less_than": "st2common.expressions.functions.version:version_less_than",
+            "version_equal": "st2common.expressions.functions.version:version_equal",
+            "version_match": "st2common.expressions.functions.version:version_match",
+            "version_bump_major": "st2common.expressions.functions.version:version_bump_major",
+            "version_bump_minor": "st2common.expressions.functions.version:version_bump_minor",
+            "version_bump_patch": "st2common.expressions.functions.version:version_bump_patch",
+            "version_strip_patch": "st2common.expressions.functions.version:version_strip_patch",
+            "yaml_dump": "st2common.expressions.functions.data:to_yaml_string",
+            "yaml_parse": "st2common.expressions.functions.data:from_yaml_string",
+        },


st2/contrib/runners/orquesta_runner/setup.py

Lines 51 to 87 in 9d311c5

"st2common.runners.runner": [

"orquesta = orquesta_runner.orquesta_runner",

],

"orquesta.expressions.functions": [

"st2kv = orquesta_functions.st2kv:st2kv_",

"task = orquesta_functions.runtime:task",

"basename = st2common.expressions.functions.path:basename",

"dirname = st2common.expressions.functions.path:dirname",

"from_json_string = st2common.expressions.functions.data:from_json_string",

"from_yaml_string = st2common.expressions.functions.data:from_yaml_string",

"json_dump = st2common.expressions.functions.data:to_json_string",

"json_parse = st2common.expressions.functions.data:from_json_string",

"json_escape = st2common.expressions.functions.data:json_escape",

"jsonpath_query = st2common.expressions.functions.data:jsonpath_query",

"regex_match = st2common.expressions.functions.regex:regex_match",

"regex_replace = st2common.expressions.functions.regex:regex_replace",

"regex_search = st2common.expressions.functions.regex:regex_search",

"regex_substring = st2common.expressions.functions.regex:regex_substring",

(

"to_human_time_from_seconds = "

"st2common.expressions.functions.time:to_human_time_from_seconds"

),

"to_json_string = st2common.expressions.functions.data:to_json_string",

"to_yaml_string = st2common.expressions.functions.data:to_yaml_string",

"use_none = st2common.expressions.functions.data:use_none",

"version_compare = st2common.expressions.functions.version:version_compare",

"version_more_than = st2common.expressions.functions.version:version_more_than",

"version_less_than = st2common.expressions.functions.version:version_less_than",

"version_equal = st2common.expressions.functions.version:version_equal",

"version_match = st2common.expressions.functions.version:version_match",

"version_bump_major = st2common.expressions.functions.version:version_bump_major",

"version_bump_minor = st2common.expressions.functions.version:version_bump_minor",

"version_bump_patch = st2common.expressions.functions.version:version_bump_patch",

"version_strip_patch = st2common.expressions.functions.version:version_strip_patch",

"yaml_dump = st2common.expressions.functions.data:to_yaml_string",

"yaml_parse = st2common.expressions.functions.data:from_yaml_string",

],

cognifloyd · 2023-02-16T23:04:37Z

contrib/runners/python_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "python-script": "python_runner.python_runner",
+        },


st2/contrib/runners/python_runner/setup.py

Lines 51 to 53 in 9d311c5

"st2common.runners.runner": [

"python-script = python_runner.python_runner",

],

cognifloyd · 2023-02-16T23:05:00Z

contrib/runners/remote_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "remote-shell-cmd": "remote_runner.remote_command_runner",
+            "remote-shell-script": "remote_runner.remote_script_runner",
+        },


st2/contrib/runners/remote_runner/setup.py

Lines 54 to 57 in 9d311c5

"st2common.runners.runner": [

"remote-shell-cmd = remote_runner.remote_command_runner",

"remote-shell-script = remote_runner.remote_script_runner",

],

cognifloyd · 2023-02-16T23:05:26Z

contrib/runners/winrm_runner/BUILD

+        stevedore_namespace("st2common.runners.runner"): {
+            "winrm-cmd": "winrm_runner.winrm_command_runner",
+            "winrm-ps-cmd": "winrm_runner.winrm_ps_command_runner",
+            "winrm-ps-script": "winrm_runner.winrm_ps_script_runner",
+        },


st2/contrib/runners/winrm_runner/setup.py

Lines 54 to 58 in 9d311c5

"st2common.runners.runner": [

"winrm-cmd = winrm_runner.winrm_command_runner",

"winrm-ps-cmd = winrm_runner.winrm_ps_command_runner",

"winrm-ps-script = winrm_runner.winrm_ps_script_runner",

],

contrib/runners/winrm_runner/BUILD

amanda11

Minor suggested on description as the winrm is 3 runners...

tag with stevedore_namespace so that once we enable the pants-plugin for stevedore, pants can install things appropriately for tests to access the setuptools metadata.

…metdata Co-authored-by: Amanda McGuinness <amanda.mcguinness@intive.com>

cognifloyd added runners pantsbuild labels Feb 16, 2023

cognifloyd added this to the pants milestone Feb 16, 2023

cognifloyd self-assigned this Feb 16, 2023

pull-request-size bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Feb 16, 2023

cognifloyd commented Feb 16, 2023

View reviewed changes

cognifloyd force-pushed the pants-python_distributions-runners branch from 3166126 to 46ceea5 Compare February 16, 2023 23:09

cognifloyd marked this pull request as ready for review February 16, 2023 23:10

cognifloyd requested review from a team, amanda11, arm4b, nzlosh, rush-skills and winem February 16, 2023 23:11

This comment was marked as resolved.

Sign in to view

cognifloyd force-pushed the pants-python_distributions-runners branch from db7f769 to 14fb366 Compare February 24, 2023 17:45

amanda11 reviewed Feb 27, 2023

View reviewed changes

contrib/runners/winrm_runner/BUILD Outdated Show resolved Hide resolved

amanda11 reviewed Feb 27, 2023

View reviewed changes

cognifloyd force-pushed the pants-python_distributions-runners branch from a1b35f1 to cc74cc5 Compare February 27, 2023 17:51

cognifloyd enabled auto-merge February 27, 2023 17:52

amanda11 approved these changes Feb 27, 2023

View reviewed changes

cognifloyd requested a review from a team February 27, 2023 23:44

arm4b approved these changes Mar 2, 2023

View reviewed changes

cognifloyd and others added 5 commits March 2, 2023 11:49

runner python_distributions for building wheel/sdist with pants

c039ab2

clarify where zip_app is needed for wheels

ff61cf1

record entry_points for runner wheels

1c370c4

tag with stevedore_namespace so that once we enable the pants-plugin for stevedore, pants can install things appropriately for tests to access the setuptools metadata.

Note that the winrm_runner includes 3 runners in python_distribution …

5ba91a8

…metdata Co-authored-by: Amanda McGuinness <amanda.mcguinness@intive.com>

update changelog entry

69c32d7

cognifloyd force-pushed the pants-python_distributions-runners branch from cc74cc5 to 69c32d7 Compare March 2, 2023 17:50

cognifloyd merged commit cd8bb4c into master Mar 2, 2023

cognifloyd deleted the pants-python_distributions-runners branch March 2, 2023 20:47

cognifloyd mentioned this pull request Mar 24, 2023

Testing Wheels generated by Pantsbuild StackStorm/community#119

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pants: add `python_distribution` BUILD metadata for runners#5907

Pants: add `python_distribution` BUILD metadata for runners#5907
cognifloyd merged 5 commits intomasterfrom
pants-python_distributions-runners

cognifloyd commented Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

cognifloyd Feb 16, 2023

Uh oh!

This comment was marked as resolved.

Uh oh!

amanda11 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	"st2common.runners.runner": [
	"action-chain = action_chain_runner.action_chain_runner",
	],

	"st2common.runners.runner": [
	"announcement = announcement_runner.announcement_runner",
	],

	"st2common.runners.runner": [
	"http-request = http_runner.http_runner",
	],

	"st2common.runners.runner": [
	"inquirer = inquirer_runner.inquirer_runner",
	],

	"st2common.runners.runner": [
	"local-shell-cmd = local_runner.local_shell_command_runner",
	"local-shell-script = local_runner.local_shell_script_runner",
	],

	"st2common.runners.runner": [
	"noop = noop_runner.noop_runner",
	],

	"st2common.runners.runner": [
	"orquesta = orquesta_runner.orquesta_runner",
	],
	"orquesta.expressions.functions": [
	"st2kv = orquesta_functions.st2kv:st2kv_",
	"task = orquesta_functions.runtime:task",
	"basename = st2common.expressions.functions.path:basename",
	"dirname = st2common.expressions.functions.path:dirname",
	"from_json_string = st2common.expressions.functions.data:from_json_string",
	"from_yaml_string = st2common.expressions.functions.data:from_yaml_string",
	"json_dump = st2common.expressions.functions.data:to_json_string",
	"json_parse = st2common.expressions.functions.data:from_json_string",
	"json_escape = st2common.expressions.functions.data:json_escape",
	"jsonpath_query = st2common.expressions.functions.data:jsonpath_query",
	"regex_match = st2common.expressions.functions.regex:regex_match",
	"regex_replace = st2common.expressions.functions.regex:regex_replace",
	"regex_search = st2common.expressions.functions.regex:regex_search",
	"regex_substring = st2common.expressions.functions.regex:regex_substring",
	(
	"to_human_time_from_seconds = "
	"st2common.expressions.functions.time:to_human_time_from_seconds"
	),
	"to_json_string = st2common.expressions.functions.data:to_json_string",
	"to_yaml_string = st2common.expressions.functions.data:to_yaml_string",
	"use_none = st2common.expressions.functions.data:use_none",
	"version_compare = st2common.expressions.functions.version:version_compare",
	"version_more_than = st2common.expressions.functions.version:version_more_than",
	"version_less_than = st2common.expressions.functions.version:version_less_than",
	"version_equal = st2common.expressions.functions.version:version_equal",
	"version_match = st2common.expressions.functions.version:version_match",
	"version_bump_major = st2common.expressions.functions.version:version_bump_major",
	"version_bump_minor = st2common.expressions.functions.version:version_bump_minor",
	"version_bump_patch = st2common.expressions.functions.version:version_bump_patch",
	"version_strip_patch = st2common.expressions.functions.version:version_strip_patch",
	"yaml_dump = st2common.expressions.functions.data:to_yaml_string",
	"yaml_parse = st2common.expressions.functions.data:from_yaml_string",
	],

	"st2common.runners.runner": [
	"python-script = python_runner.python_runner",
	],

	"st2common.runners.runner": [
	"remote-shell-cmd = remote_runner.remote_command_runner",
	"remote-shell-script = remote_runner.remote_script_runner",
	],

	"st2common.runners.runner": [
	"winrm-cmd = winrm_runner.winrm_command_runner",
	"winrm-ps-cmd = winrm_runner.winrm_ps_command_runner",
	"winrm-ps-script = winrm_runner.winrm_ps_script_runner",
	],

Uh oh!

Conversation

cognifloyd commented Feb 16, 2023

Background

Overview of this PR

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

amanda11 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants