Skip to content

Add support for multi deployment using workflow cwl#968

Open
Nazim-crim wants to merge 40 commits into
masterfrom
multipart-deployment
Open

Add support for multi deployment using workflow cwl#968
Nazim-crim wants to merge 40 commits into
masterfrom
multipart-deployment

Conversation

@Nazim-crim

@Nazim-crim Nazim-crim commented May 22, 2026

Copy link
Copy Markdown
Contributor

@Nazim-crim Nazim-crim self-assigned this May 22, 2026
@github-actions github-actions Bot added ci/tests Tests of the package and features ci/doc Issue related to documentation of the package process/wps3 Issue related to WPS 3.x (REST-JSON) processes support feature/oas Issues related to OpenAPI specifications. labels May 22, 2026
@codecov

codecov Bot commented May 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.55446% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.30%. Comparing base (1b9edbb) to head (1ddf15b).
⚠️ Report is 6 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
weaver/processes/utils.py 94.47% 2 Missing and 9 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #968      +/-   ##
==========================================
+ Coverage   88.22%   88.30%   +0.07%     
==========================================
  Files          88       88              
  Lines       20569    20755     +186     
  Branches     2702     2744      +42     
==========================================
+ Hits        18148    18327     +179     
- Misses       1733     1735       +2     
- Partials      688      693       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions github-actions Bot added the feature/cli Issues or features related to CLI operations. label May 29, 2026
@Nazim-crim Nazim-crim requested a review from fmigneault June 2, 2026 17:52

@fmigneault fmigneault left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review still in progress. Posting pending comments to stage in the meantime.

Comment thread CHANGES.rst Outdated
Comment thread CHANGES.rst Outdated
Comment thread weaver/cli.py Outdated
Comment thread weaver/cli.py Outdated
Comment thread weaver/cli.py Outdated
Comment thread tests/wps_restapi/test_processes.py Outdated
Comment thread tests/wps_restapi/test_processes.py Outdated
Comment thread weaver/processes/utils.py Outdated
Comment thread weaver/processes/utils.py Outdated
Comment thread weaver/cli.py Outdated
Comment thread tests/processes/test_utils.py Outdated
Comment thread tests/functional/test_cli.py
Comment thread tests/processes/test_utils.py Outdated
Comment thread weaver/processes/utils.py Outdated
Comment thread weaver/processes/utils.py Outdated
Comment thread weaver/processes/utils.py Outdated
Comment thread weaver/processes/utils.py Outdated
Comment on lines +685 to +710
# Reorder if root workflow specified and validate it
if root_workflow_cid and root_workflow_cid in parts_by_cid:
root_pkg = parts_by_cid[root_workflow_cid]
# Validate that the root is actually a Workflow (per RFC 5621 and multipart/related requirements)
root_class = root_pkg.get("class", "")
if root_class != "Workflow":
raise HTTPBadRequest(json={
"title": "Invalid root workflow reference",
"description": (
f"The 'start' parameter references a CWL with class '{root_class}', "
"but only 'Workflow' is permitted as root document in multipart/related."
),
"cause": {"Content-ID": root_workflow_cid, "class": root_class}
})
cwl_packages = [pkg for pkg in cwl_packages if pkg is not root_pkg]
cwl_packages.append(root_pkg)
elif not root_workflow_cid and cwl_packages:
# No explicit start parameter: validate first element is a Workflow (RFC 5621 §7 default)
first_pkg = cwl_packages[0]
first_class = first_pkg.get("class", "")
if first_class and first_class != "Workflow":
LOGGER.warning(
"No 'start' parameter provided in multipart/related. First element has class '%s' "
"but 'Workflow' is recommended for root document. Proceeding with deployment.",
first_class
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this in a separate function

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to _validate_and_reorder_multipart_workflow , should it split into two or its fine like this

Comment thread weaver/processes/utils.py
Comment on lines +564 to +565
def parse_multipart_deploy(content, content_type, request=None):
# type: (Union[str, bytes], str, Optional[AnyRequestType]) -> Tuple[List[CWL], Optional[JSON]]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please refactor in smaller chunks.

In the long run, I plan to support multipart "Deploy+Execute" (like this #834 (comment)).

Therefore, I will need to inject other intermediate/diverging steps, and would like to reuse these functions such as _classify_multipart_part. I would need to have the steps better split out:

  1. parse multipart => [parts]
  2. for part => interpret part
    • for now, you consider only CWL / process description as you currently do
    • I can latch on these functions filter "Execute" later
  3. filter / sort parts in relevant way

Comment thread tests/wps_restapi/test_processes.py Outdated
Comment thread tests/wps_restapi/test_processes.py Outdated
@Nazim-crim Nazim-crim marked this pull request as ready for review June 12, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/doc Issue related to documentation of the package ci/tests Tests of the package and features feature/cli Issues or features related to CLI operations. feature/oas Issues related to OpenAPI specifications. process/wps3 Issue related to WPS 3.x (REST-JSON) processes support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support multi-CWL file deployment Support Multipart CWL Deployment

2 participants