Skip to content

perf(storage): redirect TOS downloads to signed URLs - #171

Merged
shark0F0497 merged 1 commit into
main-v2from
fix/cloud-mcap-direct-download
Aug 6, 2026
Merged

perf(storage): redirect TOS downloads to signed URLs#171
shark0F0497 merged 1 commit into
main-v2from
fix/cloud-mcap-direct-download

Conversation

@shark0F0497

Copy link
Copy Markdown
Collaborator

Pull Request Checklist

  • Code follows the style guidelines
  • Tests pass locally
  • Code is formatted
  • Documentation updated if needed
  • Commit messages follow conventional commits
  • PR description is complete and clear

Summary

Redirect full TOS-backed MCAP downloads to short-lived signed TOS URLs instead of proxying the entire file through Keystone. Keep ranged MCAP preview reads on the existing same-origin proxy path.

Motivation

Cloud MCAP downloads are slow because Keystone currently relays the complete object, adding an extra network hop and making Keystone bandwidth and connection lifetime part of the download path.

Changes

  • Return a 307 redirect for authenticated, non-Range TOS downloads using a download token.
  • Generate browser-accessible TOS signed URLs from STS credentials and convert private ivolces.com endpoints to public volces.com endpoints.
  • Cap signed URL lifetime to the remaining download-token lifetime and STS credential expiration.
  • Preserve the existing proxy behavior for Range requests and all MinIO downloads.
  • Add tests for full-download redirects and ranged-response proxying.

Type of Change

  • Performance improvement
  • Test changes

Impact Analysis

Breaking Changes

None.

Backward Compatibility

Fully backward compatible. The existing storage endpoint, download-token flow, Range preview behavior, and MinIO path remain available.

Testing

Test Environment

  • Go 1.24-compatible repository toolchain
  • Linux

Test Cases

  • Full TOS download returns a temporary redirect to a signed public TOS URL.
  • TOS Range request remains proxied and returns partial content.
  • Download-token validation and expiration tests pass.
  • Full Go test suite passes with race detection.
  • golangci-lint reports zero issues.

Commands run:

go test -cover -race ./...
golangci-lint run ./...

Test Coverage

  • New tests added
  • Existing tests updated
  • Coverage maintained

Performance Impact

  • Memory usage: Decreased for full downloads on Keystone
  • CPU usage: Decreased for full downloads on Keystone
  • Throughput: Improved by downloading directly from TOS
  • Lock contention: No change

Documentation

  • No documentation changes needed

Additional Notes

  • No frontend or CORS changes are required for the download flow.
  • Range reads intentionally remain on the Keystone proxy for MCAP preview compatibility.

@shark0F0497
shark0F0497 merged commit 70fac6f into main-v2 Aug 6, 2026
4 checks passed
@shark0F0497
shark0F0497 deleted the fix/cloud-mcap-direct-download branch August 6, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant