Discuss workflow macro design and lifecycle

[carloea2]

Feature Summary

Workflow macros are an important feature for Texera, but the current discussion in #5179 mixes several design choices with implementation review. Based on the feedback from @Yicong-Huang and @chenlica, we should move the macro design discussion into an issue, hold a focused live discussion, and record the decisions here before continuing.

References:

• feat: add workflow macro UI support #5179: frontend-oriented workflow macro implementation
• Add workflow macros for importing, grouping, and collapsing reusable sub-workflows #5178: original feature proposal for importing/grouping/collapsing reusable sub-workflows
• [Hackathon] feat: AI-augmented macro operators #5115: hackathon macro-operator prototype with a different UX
• Added operator grouping feature #754: previous operator grouping/collapse behavior

Proposed Solution or Design

Use this issue to align on the macro design before reviewing code in detail. The discussion should decide:

1. Macro creation UX

   • Should users create a macro by selecting existing operators and choosing Create Macro from the context menu?
   • Should users create a macro by importing an existing workflow?
   • Should both flows be supported?

2. Macro representation and reuse

   • Should a macro look and behave like a normal logical operator?
   • Should macro input/output ports be derived from the internal sub-DAG?
   • Should reusable macros appear directly in the operator menu, for example under My Macros?
   • Should the long-term design also be able to represent operators such as Hash Join as macros?

3. Macro internal view and editing

   • Should users edit macro internals in-place on the current canvas, or jump into a separate/drill-down workflow view?
   • Are edits local to one workflow snapshot, or shared across all uses of the macro?
   • Should macro internals expose explicit input/output ports instead of dummy source/sink operators?

4. Runtime and statistics behavior

   • Should macro expansion remain frontend-only, with the backend executing the flattened operator graph?
   • Should the backend be unaware of macros?
   • When collapsed, should a macro show aggregated statistics, internal leaf-operator statistics, or allow users to choose?

5. Macro lifecycle and persistence

   • How should macro definitions and macro instances be stored?
   • How should users remove, rename, disable, search, share, or version macros?
   • What documentation, tutorials, and tests are needed before enabling the feature by default?

Impact / Priority

?

Affected Area

• Workflow UI
• Workflow Engine (Amber)
• Storage / Metadata
• Documentation / Tutorials

[carloea2]

@chenlica @Yicong-Huang @Xiao-zhen-Liu @aglinxinyuan

[Yicong-Huang]

Let me convert this to a Discussion post. sorry if that's not clear. for uncertain topics, we use disucussions. once a consensus is reached, we open an issue (or feel free to use umbrella issue + multiple sub-issues) to track decided todos. Discussions can have back and forth ideas. issues should be clear on what exactly to do.

[Yicong-Huang]

I shared my preferences in #5179 already. If you are having an offline discussion, please report back here.

[carloea2]

In that case, we can just wait for @chenlica and an internal meeting to decide and report the decision here.

[chenlica]

@carloea2 Let's do a meeting to discuss the details and report the results here.

[Xiao-zhen-Liu]

We discussed this feature offline. We agreed that the design in #5115 is more general in that it has explicit encapsulation of the input and output ports, and that the user chooses which ports of the macro workflow is exposed.

[chenlica]

Do we have any plan to finish this important feature?

[zuozhiw]

I saw the demo video and I would have absolute no problem and will be very happy if we can ship #5115 as-is

[chenlica]

Maybe @carloea2 can take the lead to finish it? Notice that @Xiao-zhen-Liu's availability in the summer is limited. It will be good to have a discussion before that, and report the plan here.

[carloea2]

I think we can wait for @Xiao-zhen-Liu to come back since he raised the PR first, and his design was chosen, I could take his code and raise the PRs but I feel it is not right

[aglinxinyuan]

I don't think we're on the same page. Let's have a group discussion next Monday. If we cannot finish, we can have one more discussion on Wednesday.

[chenlica]

We had an offline discussion about this topic. @carloea2 : please summarize the results.

[carloea2]

In summary:

• We will keep the first implementation simple.
• First version will not allow reusability of a macro or sharing.
• The main changes will be frontend related according to the experience Xiaozhen designed, by adding some properties to the workflow json.

[Yicong-Huang]

Hi @carloea2 could you please elaborate more? Thanks.

[carloea2]

There are no more details at the moment, we just agreed on the user experience.

Design Video for reference #5115

[Yicong-Huang]

Please involve the community in the discussion, instead of "make an offline decision and inform the community".

If you had an offline discussion, please post more details so we can continue the discussion here.

Cc @chenlica @aglinxinyuan @Xiao-zhen-Liu please share more details.

[carloea2]

I did not request an offline discussion.

[chenlica]

@carloea2 Please provide more details as suggested.

[carloea2]

Ok.

1. Frontend will do the expansion, If we have workflow 1 with macro A, and macro A has macro B, then the whole hierarchical expansion will be done on frontend, and the backend does not need to be aware of this.
2. We agreed to show the macro UI as in Xiaozhen demo and if I remember correctly and introduce buttons to add/remove inputs/outputs.
3. We have not defined details of what the frontend should show about statistics.

A general question: if the goal is to meaningfully involve the broader community, shouldn’t the whole design and decision-making discussions happen in a public, async, and searchable place like GitHub?

[chenlica]

@carloea2 Thanks for the summary. Not sure if I fully understand your question. I believe we are allowed to do offline discussions for efficiency reasons, but we need to publish the discussed proposals to the community to maintain the transparency.

[yangzhang75]

Hi, I was suggested to fold my recent proposal (#6066, now closed) into this discussion.

I've been prototyping something adjacent to macros: a "Blocks" gallery for saving and reusing workflow pieces across workflows. Reading through this thread, my design is different in direction (drop-in reuse vs. in-place collapsing). But a few pieces from the prototype might be useful for the macros work:

• Built-in Sample macros (a small read-only starter set in the panel, for onboarding)
• Mini-map preview of each macro (SVG rendering with real operator icons)
• Snip-region save (box-select a sub-region of the canvas, save only that part as a macro instead of the whole workflow)
• Apply-to-canvas logic (fresh UUIDs, remap links, place on empty area)

Demo of the prototype: https://drive.google.com/file/d/12pPeBUvbLlsrO5t5pp0C9G2xeEordTD2/view?usp=sharing

Would any of these be useful to fold into the macros direction? Thank you!

[chenlica]

@carloea2 Please chime in. If needed, we can have a discussion and report the results here.

[carloea2]

@yangzhang75 do you want to have a zoom meeting so we can understand your proposal and design in comparison with the one presented here?

[yangzhang75]

Sure. What times work for you this week?

[chenlica]

I suggest we do not do this type "scheduling meeting" messages on GitHub.

@Yicong-Huang : what's a good practice for such messages?

[yangzhang75]

After meeting with @carloea2, here's a summary of what we discussed.

The base macro feature (the #5115 direction — explicit input/output port encapsulation and in-place collapse) will be built first. Once it's merged, I can open separate PRs adding these extra features on top:

• Live preview for a macro — a zoomable mini-map of a collapsed macro's internal sub-DAG (with real operator icons), so you can see what's inside before expanding it.
• More flexible selection when creating a macro — a box/region ("snip") select to turn part of the canvas into a macro, in addition to selecting operators one by one.
• Macro gallery for browsing and reusing saved macros — this fits later, alongside the reuse/sharing phase (v1 doesn't include reuse).

[carloea2]

We have been exploring a revised approach to macros and would like to discuss it here before settling the design.

Some ideas we are considering:

1. • Workflow and macro relationship: At the execution level, workflows and macros are both operator graphs with the same engine capabilities. One possible design choice is to reserve standalone execution for workflows, even though a macro’s DAG may technically be executable independently.
2. • Composition: Under this model, a workflow would act as the entry point, similar to main in a program. It could contain macros, and macros could contain other macros, similar to functions calling other functions.
3. • Macro representation: A macro could represent a sub-DAG as a single operator, with exposed input/output ports connected to its internal graph.
4. • UI experience: Reusable macros could appear in the operator panel alongside regular operators. Xiaozhen’s UI could serve as the starting point for editing and navigating macro internals. https://www.youtube.com/watch?v=ZYQIoPiEIfo&feature=youtu.be
5. • Independent versioning: We are considering versioning workflows and macros independently, with macro instances referencing their definitions through pointers. For example, workflow A version 8 could reference macro B version 5.
6. • Execution: An initial approach could be to recursively expand macros in the frontend before submitting the graph for execution. The saved workflow would retain its hierarchy and references, while the execution engine would receive a flat operator graph. This should avoid engine changes, although persistence and APIs may still need work.

This would broaden the earlier proposal that deferred reuse. Whether these capabilities should be included initially or introduced in stages is also open for discussion.

Some questions we would like feedback on:

1. Overall model: Does the distinction between an executable workflow and a reusable macro make sense? Are there use cases where allowing standalone macro execution would be valuable?
2. Export/import: How should a workflow carry its macro dependencies when imported into another deployment? Would bundling the referenced definitions and versions be a suitable approach?
3. Form view: How should exposed properties appear across nested macros? Which values should belong to the shared definition, and which should each instance be able to override?
4. Versioning: Should references point to specific versions? How should users adopt updates, especially when ports or properties change?

These are starting points for discussion. Suggestions for alternative models, missing use cases, or a simpler initial scope would be welcome.

[chenlica]

Thanks for the summary. It's based on an offline discussion by @carloea2 @aglinxinyuan @Xiao-zhen-Liu @mengw15 @parshimers @sshiv012 and @chenlica .

For "idea 1," a macro is viewed as a workflow that cannot run by itself, similar to a non-main function in programming languages.

For "idea 4," please add a link to "Xiaozhen’s UI."

[parshimers]

I'm glad this got revisited. I didn't follow the earlier discussion. I'm glad it led to form views, but I think macros are a separate and useful thing.

I agree with the idea of treating macros as a kind of special workflow, which cannot be run on its own. I think the simple inlining of those workflows into whatever workflow they are used in, as a special kind of operator, is a very common and natural thing. AsterixDB's SQL++ UDFs are implemented in exactly the same way; they are simply queries which get written into some other query where the function appears in the main query.
I think a workflow that references a macro should always refer to it with a reference, and only resolve that reference at compile time. It really can only be the compiler that can take ownership of this part.
The dependencies are indeed a tricky problem. I think at least for "pure" operators it's fine, but for UDFs it can get complicated. I think it might be easier to constrain those out of the design for now if it complicates it.
I don't think a property within a macro should be able to live outside of the context of the macro. It's effectively an operator, after all, and having other workflow side effects seem kind of strange to me.

P.S. (Mentor hat): I noticed this thread kind of devolved into a series of offline discussions, and while the latest post did an OK job of trying to share the state and loop everyone else in, the rest of the thread is really an antipattern. The point of discussing it here is to make the discussion entirely open, and not to agree on when closed discussions should or shouldn't take place. It can't be the case that discussions end up as a dumping ground for designs which have already been agreed to a priori by a clique of contributors before ever being brought out into the open, basically as an afterthought.

[sshiv012]

I support treating macros as reusable subgraphs with explicit ports and independently pinned versions.

I think having the below might be nice,

• Allow macros to be tested with sample inputs, even if only workflows are independently executable
• Keep defaults in the definition and overrides in each instance; expose nested properties explicitly.
• Frontend expansion seems reasonable initially, but preserve the hierarchy mapping for errors and statistics, and maybe leave a path towards server-side expansion?