Handle Eurostat async ZIP responses (footer URL polling)

[aborruso]

Problem

Eurostat's SDMX API sometimes returns an empty data message with a download URL in the response footer instead of the actual data. This happens for large datasets that are served asynchronously as ZIP archives.

Currently opensdmx does not handle this case: the user receives an empty response with no explanation.

Expected behaviour

When Eurostat returns a footer URL, opensdmx should:

1. Detect the footer URL in the response
2. Poll the URL until the file is ready (with configurable retry delay and max attempts)
3. Download and extract the ZIP
4. Return the data as if it had been a normal synchronous response

Reference

khaeru/sdmx handles this in sdmx/source/estat.py via a finish_message() hook that polls the footer URL with configurable retry delay and max attempts.

Notes

• Affects large Eurostat datasets only — standard requests are unaffected
• The polling logic should be transparent to the caller
• Retry parameters (delay, max attempts) could be exposed as provider-level config in portals.json

[aborruso]

Closing for now — not a priority.

Investigation note for whoever picks this up later: the khaeru/sdmx reference handles the async case as an SDMX-ML <footer:Footer> carrying a download URL. But opensdmx requests data as SDMX-CSV (data_format_param: SDMX-CSV), and a <footer:Footer> is an SDMX-ML construct that cannot appear in a CSV response. Eurostat's current docs also describe a different async path (SOAP queued id + /async/status + /async/data), which belongs to the disschain SOAP service, not the /api/dissemination/sdmx/2.1 endpoint opensdmx uses.

So before implementing any polling, the prerequisite is to capture the raw response of a large filtered SDMX-CSV query (status + Content-Type + body) on a query estimated in the 500k–5M cell async window. That single capture decides whether the fix is footer-URL polling, a better error message, or a 413 fallback — none of which is settled yet. Reopen when prioritized.