Repository navigation
Support custom host and port, add streaming response support #54
Description
Thanks for creating this crate, it's a time saver! We needed a way to query Snowflake and convert to arrow/parquet, so this is perfect for us. There are some tweaks I would need in order to use this for tests. I'm happy to submit a PR, but wanted to see if you were accepting them?
Currently, the host is hard-coded to go snowflakecomputing.com. There is a way to test snowflake now on LocalStack, but it requires a custom host and port (host: snowflake.localhost.localstack.cloud, port: 4566).
I think this can be added to the builder without affecting the current API in lib.rs (just the signature to request() in connection.rs):
pub struct SnowflakeApi {
connection: Arc<Connection>,
session: Session,
account_identifier: String,
host: Option<String>, // this is new
}
pub fn with_host(mut self, uri: &str) {
self.host = Some(uri);
}Then, in the request() function in connection.rs:
pub async fn request<R: serde::de::DeserializeOwned>(
&self,
query_type: QueryType,
account_identifier: &str,
extra_get_params: &[(&str, &str)],
auth: Option<&str>,
body: impl serde::Serialize,
host: Option<&str>, // this is new
)
...
let url = host.unwrap_or_else(|| format!(
"https://{}.snowflakecomputing.com/{}",
&account_identifier, context.path
));REFS:
Hard coding:
snowflake-rs/snowflake-api/src/connection.rs
Lines 147 to 149 in 57705c5
LocalStack on Snowflake docs: https://snowflake.localstack.cloud/introduction/
Activity
ddimaria commentedon Sep 13, 2024
I have a working fork here: https://github.com/quadratichq/snowflake-rs, though it still needs some polish. Sadly, hitting LocalStack Snowflake gave an empty body on the session login. After contacting LocalStack, it seems like their product is in public preview, and may be a paid feature in the future.
My goal is to be able to test the snowflake interactions without hitting the production snowflake server. HTTP Mock is relatively new, which offers the ability to record live requests/responses into saved fixtures to use in tests. They require a custom host injection to act as a proxy, so the code in the fork may still be useful. I'll update this issue with my findings.
ddimaria commentedon Sep 24, 2024
In that same fork, I implemented a stream response. The original full response is available, but this stream helps avoid taking down a server for overly-large data sets.
[-]Support custom host and port[/-][+]Support custom host and port, add streaming response support[/+]sgrebnov commentedon Sep 24, 2024
Note, for streaming support there is also the following PR available:
#44
The code has been battle tested for several months as part of the following project
https://github.com/spiceai/spiceai