Keep track of provenance in the metadata viewer, and allow user to override values - #2083
Draft
jdbocarsly wants to merge 9 commits into
Draft
jdbocarsly wants to merge 9 commits into
jdbocarsly wants to merge 9 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2083 +/- ##
==========================================
+ Coverage 80.50% 80.70% +0.20%
==========================================
Files 84 85 +1
Lines 7909 8003 +94
==========================================
+ Hits 6367 6459 +92
- Misses 1542 1544 +2
🚀 New features to boost your workflow:
|
datalab
 |
||||||||||||||||||||||||||||
| Project |
datalab
|
| Branch Review |
jdb/metadata-provenance
|
| Run status |
|
| Run duration | 18m 08s |
| Commit |
|
| Committer | Josh Bocarsly |
| View all properties for this run ↗︎ | |
| Test results | |
|---|---|
 |
0
|
 |
0
|
 |
0
|
 |
0
|
 |
188
|
| View all changes introduced in this branch ↗︎ | |
A block's metadata is rarely all from one place: a sample mass written by the instrument, a molar mass derived from the sample's formula, either of them corrected by hand. Which of those a value came from is worth knowing, and it decides what should happen to that value later. So a field is bound to a source, and it is the binding that persists rather than the value. Bound to a source, the value is re-read on every render, and follows the file if the file is replaced or the sample if the sample is edited. Bound to the user, it is what they gave and nothing recomputes it. Bound to nothing, the sources are tried in order -- a guess, so it is made again each time and may change its mind when a better source turns up. The rule underneath: the block's own choices stay open, the user's are kept. Clearing a field by hand is a choice, and says there is no good value for this one, so it survives the next render rather than being filled in again from the file. That is why it cannot be represented as an absent value, and why a binding has to be a thing in its own right. A block supplies `metadata_sources`; everything else is here. Every source is read whether or not it wins, so the interface can say where a number came from and offer the alternatives, and one event covers binding any field to any of them. The item schemas move because DataBlockResponse is embedded in them.
A metadata field the block tracks now renders as its value plus where that value came from, and right-clicking it offers what can be done: override it by hand, take it from one of the other sources, set it to empty, or hand it back to the block to decide. The menu only offers what the block says it has, so a source with nothing for this field never appears, and "choose automatically" only shows up once there is a choice to undo. A tracked field is listed even when it is empty, since an empty one is exactly the one somebody needs to reach. The provenance sits beside the value rather than replacing it -- quiet enough to read past, with the explanation on hover. A value somebody typed is the one worth being able to pick out at a glance, so that is the one that is marked. An emptied box sends a value of the user's rather than no binding at all, which is what makes "there is no good value for this" survive the next render.
"edited", in yellow, is how the rest of datalab says a change has not been saved yet. On a metadata value it meant something else entirely -- that the number came from a person rather than from the file -- and read as a warning about work in progress. It now says "user supplied", in a blue that carries no such implication.
A right-click was a strange thing to require of the one control in the panel that does anything; a plain click opens the menu now, and a right-click still does too for anyone who reaches for one. An edit gains a save and a cancel beside the box, so backing out is something you can see rather than something you have to know. Clicking away cancels as well, on the grounds that abandoning an edit should never be the thing that writes a value. Both of those needed the same fix, which the tests found: a click on a menu option, or on the save or cancel button, was bubbling back to the field and reopening the menu the click had just closed.
"Read from the file" is only half of what somebody wants to know, and the half they already assumed. A block can now say what to call each of its sources, so the note reads "Supplied by NiCl2btd_MT_ZFC_210707_MPMS-XL.rso.dat", and falls back to the source's own name where a block has nothing better to offer. A value of the user's says what was done rather than what will happen to it -- overwritten, or set to blank -- since the consequence is already in the menu. The help cursor goes: a note that explains itself on hover does not need to advertise that it will.
A binding is somebody deciding that a value should not be worked out the usual way. Six months later that is exactly the thing nobody can remember, so the choice now carries who made it and when, and the note on the value says so: "Value overwritten by Ada Lovelace on 5 Sep 2026". Choosing which source to take a value from is attributed the same way, since it is the same kind of decision -- somebody looked at a number and said use that one. There is not always anyone to name. A binding made by a script, or before this was recorded, keeps the time or says nothing at all rather than failing; the interface says only what it knows.
`to_web` rebuilds the block's errors from what its plot functions report, which threw away anything an event had recorded on the way there. A control that failed therefore did nothing and said nothing about why -- the worst of both, and a slow thing to debug from the outside. Event errors are now kept on the instance for the life of the request and reported alongside the plots', while the stored errors are still cleared by a clean render, so nothing sticks around from last time. Also says in `metadata_sources` what its callers already required: it is asked what the sources are when a binding is set, which happens before anything has been rendered.
Emptying a field made it jump to the bottom of the panel and filling it in brought it back, because the list was built from the values first and the empty tracked fields appended afterwards. A field that moves when you change it is hard to work with and easy to lose. The order now comes from `metadata_fields`, which the block writes out in the order its metadata model declares them and which carries every field whether or not it has a value -- `metadata` cannot serve, since a value of None is dropped from it altogether. Blocks that track nothing keep the order they had.
jdbocarsly
force-pushed
the
jdb/metadata-provenance
branch
from
September 8, 2026 15:37
40baaba to
e5f26ce
Compare
A code review found several holes in the provenance, the worst of which made it worthless: `metadata_bindings` was loadable from the web, so anyone who could update a block could POST a binding claiming a colleague had set a value, and the note on the field would say so. It is server-authoritative now, like the other three metadata slots -- only `set_metadata_source` writes it, and only that path stamps who and when. A binding that is not a mapping is ignored rather than taking the whole resolution down with it. Values are coerced by validating them rather than by assigning them, since a model only checks assignments when it asks to. Without this, a metadata model that did not happen to enable `validate_assignment` got no checking at all: a file offering "heavy" for a mass had it stored verbatim, and a user typing "99" had it kept as a string. Setting a source now resolves before returning. `to_web` only runs the plot functions, so a block whose plots do not resolve -- or which has no plots -- answered the request with the value and the source the field had beforehand, and the click appeared to do nothing. A source may no longer be called "user" or "auto". Those are the two answers the event gives itself, so a block naming one of them could never bind to it, and asking for it would silently blank the field. Three smaller ones: clicking a field no longer swallows the click, so opening one menu closes any other; a binding that records a time but nobody to name now says "by user" rather than crediting the timestamp; and the derived provenance is stripped before a block is sent, since the server rebuilds it and refuses to load it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The metadata viewer currently nicely displays the metadata available on a block, which is generally set during file read. This PR adds the ability for the block developer to set how the metadata should be determined (e.g., from the data file vs. inherited from the sample vs. supplied by the user). The user can also override any values by clicking on it in the metadata viewer, and that metadata is tracked.
This PR will also probably hide metadata by default, and fix an issue where hiding/showing the metadata causes the screen to jump positions (not complete yet).
Some screenshots of the new behavior (demonstrated with the private SQUID block):
