[Notebook Migration] Keeping the notebook mapping and Jupyter panel for Python file input

[zyratlo]

Issue #8007 adds Python files as an input to the notebook migration tool. A notebook arrives already segmented into cells, and those cells are what the tool keys its operator mapping off, which in turn drives the Jupyter panel and the click to highlight behavior in the workspace. A script has no cells, so we have to decide what that input gets. There are two options:

A. Workflow only
Send the whole script to the LLM, get a workflow back, stop there. No mapping or Jupyter panel. Smallest change, and no Jupyter dependency for script users at all.

B. Have the LLM segment the script
The mapping call already asks the model which parts of the input became which UDF. For a script we ask for line ranges instead of cell IDs, then build cells from the ranges it returns. Everything downstream is unchanged: same mapping shape, same storage, same panel, same highlighting.

We are leaning toward B.

The reasoning is that the mapping is the only way a user can check the conversion. Migration output is generated, so "which part of my code became this operator" is the first question anyone asks when the workflow looks wrong. Scripts are usually longer and less structured than notebooks, so that question is harder to answer by eye, not easier. Option A would also mean two inputs with visibly different support sitting in the same modal.

B is also cheaper than it sounds, it does not add a round trip, since the mapping call already exists. The new work is reconciling ranges that overlap or leave gaps, plus a script variant of the conversion prompt so existing notebook migrations see byte identical text and are unaffected.

Pushback or any suggestions are welcome, but if there are no objections we will continue with B.

[sshiv012]

I like B as well, AI should get better and inexpensive in doing such non-deterministic search problems, the mapping helps users understand and verify the migration. My main concern is treating LLM-generated line ranges as the source of truth. Could we preserve the original script and validate the ranges against it before building cells? Overlaps may be legitimate when the same code contributes to multiple operators, while gaps should remain visible as unmapped code rather than disappear.
It would also be nice to distinguish “this operator came from these lines” from “these lines can run independently as a notebook cell,” since imports and shared state may cross those boundaries.

[zyratlo]

Thanks for taking a look! Just a heads up that this ended up merged in #8544, so what's below is how it works in main now.

On the source of truth point, I think we're mostly on the same page. The LLM only supplies the cut points; the text of every cell is sliced straight from the uploaded script, so the model can't add or change any code that shows up in the panel. The ranges also get checked against the script before any cells are built: reversed ones are flipped, ones that run past the file are clamped, and malformed ones are dropped. The part we can't do is semantically check that the lines actually match what the UDF does, so a well formed but wrong range would still give misleading highlighting (but we are willing to accept that as LLMs can't be guaranteed reliable).

Overlaps and gaps work pretty much the way you describe. Lines that two operators both claim become one shared cell that highlights for both, and a gap becomes its own cell that no operator claims, so it stays visible and just doesn't highlight anything. The one exception is a gap made up entirely of blank lines, which gets dropped so Jupyter doesn't show empty cells.

The runnable vs. "came from these lines" distinction is a really good one. If I'm understanding it right though, it doesn't cause problems today, since the panel is read only and never runs anything. The menu and toolbar are hidden, cells can't be edited, and the run shortcuts are disabled. So the cells are really just a display unit for highlighting, and a cell that couldn't run on its own isn't an issue yet. If we ever let the panel execute code, though, that would definitely need revisiting.

[sshiv012]

Ahh, that makes sense. yes we can revisit in future if needed. Thank you for clarifying. And happy to know it got merged :)