Links to node source analyses should be automated and/or validated

[mabijkerk]

Context

Note that I had this issue written by AI and then made some minor changes. config/nodes_download_source.yml maps node keys to GitHub links for the NSA spreadsheets in etdataset-public. It is read by Node::ALL and regenerated by hand with rake fetch_nodes_download_source, which we do not run consistently.

Shortcomings

The main problem is that nothing keeps this file in sync. The task is not automated and nothing validates the result, so the file drifts from etdataset-public between runs and we only find out when a user hits a dead link. Nodes end up with no link, or with a link to a file that has since moved or been removed:

• 110 of the 333 nodes referenced by related_node: in config/interface/input_elements/ have no link at all.
• 23 of 266 links point at files that no longer exist in etdataset-public (I checked the basenames against the full upstream tree; none of them moved, they are gone).
• 16 entries are for nodes that nothing in etmodel references anymore.

Running the task does not resolve this on its own. It only writes a key when it finds a working URL and never removes one, so a link that breaks upstream stays in the file.

The task is also slow. It builds 8 candidate URLs per node (4 file endings x 2 types) and does a serial HTTP HEAD on each, so roughly 2700 requests for a full run. A single call to the GitHub trees API returns all 339 NSA paths at once.

Worth noting for any automation: the task reads InputElement.all, so it needs a seeded database and will not run on a clean CI checkout as is.

Steps to reproduce

1. Run rake fetch_nodes_download_source.
2. Open a node download link for e.g. energy_heat_burner_network_gas or energy_chp_ultra_supercritical_coal and note the 404.
3. Compare the keys in config/nodes_download_source.yml against the related_node: values in config/interface/input_elements/ to see the missing and stale entries.

Version: master.

Proposed solution

We should see if we can structurally improve this, so that it is validated and/or kept up to date automatically. Some suggestions from AI:

• Run the task on a schedule so the file does not depend on someone remembering.
• Add validation that reports both broken links and referenced nodes without a link, and fails when either appears.
• Make the task remove links whose file is gone upstream, instead of only adding.
• Use the GitHub trees API to fetch the file list in one request rather than probing each URL.
• Consider dropping the stored YAML and working out the link from the node key when it is needed, since the URL is derivable.
• Report entries for nodes that etmodel no longer references so they can be cleaned up.