Skip to content

download a stream of bytes, not single byte - #123

Open
kosta wants to merge 1 commit into
ThouCheese:masterfrom
kosta:streaming-bytes
Open

kosta wants to merge 1 commit into
ThouCheese:masterfrom
kosta:streaming-bytes

Conversation

@kosta

@kosta kosta commented Oct 7, 2022

Copy link
Copy Markdown

Hi!

I noticed that fact that your download_streaming APIs return an impl Stream<Item = crate::Result<u8>> which is very inefficient. create::Result<u8> has size 72 on my machine (64bit arm), so for every byte copied, we need to pass 72 bytes around. But even if it was impl Stream<Item = u8>, it would still be inefficient. Such streams always work on slices of bytes (e.g. see the Read or AsyncRead traits) or here, it's the easiest and most idiomatic to just return a stream of Bytes.

So I deprecated download_streamed and added download_bytes_stream.

Hope you like it :)

Cheers,
Kosta

Comment thread src/client/object.rs
} else {
Ok(resp.error_for_status()?.bytes().await?.to_vec())
Ok(resp.error_for_status()?)
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created a non-pub helper function download_response to unify the common download code paths. I also noticed checking for StatusCode::NOT_FOUND was only done for download, not download_streamed. I think it makes more sense to have that unified here, but it might be a breaking change?

Up for you to decide, I'm happy to move that error handling around.

Usability note: Often the google error responses include helpful content that is being discarded here. I think on http status errors, we should actually try to convert it into an Error::Google(GoogleErrorResponse). Maybe in a separate MR?

Comment thread src/client/object.rs
.bytes()
.await?
.into())
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual methods are now very straightfoward

@kosta

kosta commented Oct 21, 2022

Copy link
Copy Markdown
Author

Actually, I noticed that I not only need the stream of bytes but also at least the content-length for my use case (which is provided in a header). I will update my MR accordingly.

@kosta

kosta commented Jan 2, 2023

Copy link
Copy Markdown
Author

Nevermind my last comment, please review this and merge if you feel this is appropriate.

@kosta

kosta commented Feb 2, 2023

Copy link
Copy Markdown
Author

Hi! This PR is open since October and is +96 −32 across 3 files and I think it's pretty straightforward. I would appreciate you taking the time to review it, but of course I understand if you cannot make the time to do it. If you think this PR does not improve your codebase, please close it. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant