feat: serve schemas based on which tags exist on gh

[jo-mueller]

PR following #564 that solves the problem a bit more elegantly (I hope):

This PR changes the build process of the page so that the following happens on build:

• All tags on ome/ngff-spec are automatically retrieved with git ls-remote --tags https://github.com/ome/ngff-spec
• This gives a list of tagged versions on that repo (currently not listing 0.1 ... 0.5)
• Copy all the schemas over here and serve them under the respective tag.

This way, we should be able to access the schemas of all past (we'd need to tag them) and present through the ngff page, including the dev versions. This also centralizes the tagging over at ngff-spec as a core mechanic to make schemas available through the ngff page.

cc @jni

[github-actions]

Automated Review URLs

• Readthedocs

[jo-mueller]

@lubianat do you have ideas how to make this nicer? I tried a few things with redirects, but maybe the RTD redirects supersede any redirects to outside pages (i.e. via an htaccess file) I am defining here, which is annoying.

To have the browser render the schemas as text instead of downloading the files instantly, we would need to rename the files into .json, I think.

[lubianat]

@jo-mueller should we go ahead and add the .json for the next releases (per #462)?

I don't have a suggestion on how to make it nicer, though

[jo-mueller]

We could serve them as both *.schema and *.json? For the time being, this could also happen here until we make the change at ngff-spec?

[lubianat]

We could serve them as both *.schema and *.json? For the time being, this could also happen here until we make the change at ngff-spec?

I don't have a good perspective on the implications. I guess we can do it at least transitionally, while tools move towards .json or .schema.json

[jni]

including the dev versions

I expect that we would want those to be hidden under a toggle/details/whatever.

[jni]

How do I review this? I don't know what you mean by "nicer", and looking at the preview build I only see 0.6rc0, no dev versions...

[jo-mueller]

@jni I think there's two aspects to this:

1. displaying the spec texts of all tagged versions
2. delivering the schemas for all tagged versions

This PR only addresses the second one. I hadn't thought about doing the same with all the tagged versions, but I think that'll be a bit more complex (but not much). I guess the approach to automatically getting all the tagged spec versions here, would be something like:

• remove the submodules from this repo entirely
• on build, get all the tagged versions (as this PR does it)...
• ....and clone all of them (which this PR doesn't yet do).
• The toctree that organizes the content of the left sidebar on https://ngff.openmicroscopy.org/specifications/index.html would then need to be built automatically as part of that CI.

On a plus side, that would get rid of all submodule shenanigans here, which I think would be a big plus ^^" On the downside, that would potentially be quite a bit of cloning/updating stuff as part of a normal build command?

[jo-mueller]

decision point: We serve the schemas as both .schema and .schema.json, in the future (on the 0.9dev branch) we rename the schemas to .json or .schema.json

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ngff-weekly-dev-update-thread/110810/140

[jni]

@jo-mueller So, on the schemas side: where are the schemas actually served from this PR? Can I get a RTD build URL? Also, can you re-trigger build maybe with an empty commit so we can see the 0.9.dev1 schemas get served?

I finally grokked the .schema vs .schema.json issue. I suggest serving both for now. 0.9.dev1 links to the historical .schema files, so let's punt on fully renaming to at least .dev2.

[jni]

Ok, found it:

https://ngff--581.org.readthedocs.build/0.6.dev1/schemas/label.schema

[jni]

Since currently the 0.9.dev1 schemas are not being published, I consider this a strict improvement and would advocate for it getting merged ASAP.

[jo-mueller]

@jni small thing that was missing here: I modified the code a bit so that schemas are copied to the _html_extra directory for

• all tagged versions of the ngff-spec repo (0.6.dev1...0.9.dev1)
• all submoduled versions (0.1...0.5)

Without this change, we would stop serving schemas for 0.1...0.5 :)

Also to keep in mind that once ome/ngff-spec#187 is resolved, we can do a bit less special-casing here and split the build process in two functions that

• function 1: clone/fetch all the schemas according to the tag
• function 2: Build and render the desired example documents for a subset of versions we want to display (0.1...0.6 + dev)

Edit: Also, the failing lint check points to some broken cross-links in the 0.9.dev1

For the Links > Internal check, the following failures were found:

• At ./_build/html/_temp_ngff_spec/index.html:626:

  internally linking to #spec:example:coordinate_transformation; the file exists, but the hash 'spec:example:coordinate_transformation' does not

• At ./_build/html/_temp_ngff_spec/index.html:655:

  internally linking to #orientation-values; the file exists, but the hash 'orientation-values' does not

[jo-mueller]

@jni ok, all the schemas are available on the RTD build:

• https://ngff--581.org.readthedocs.build/0.1/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.2/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.3/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.4/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.5/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev1/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev2/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev3/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev4/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6rc0/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.9.dev1/schemas/image.schema

Edit And as a little extra: All the schemas are now also available with a .schema.json extension (e.g., https://ngff--581.org.readthedocs.build/0.9.dev1/schemas/image.schema.json) so the browser shows you the schema instead of downloading it.