Repository navigation
v0.9.0/dataset-scanner - #269
Merged
Merged
Conversation
thanos
changed the title
Open directories, globs, or path lists via FileSystem, parse typed Hive keys from paths, and resolve schema from the first fragment footer.
Lazy scanner compiles Expression filters into partition pruning, Parquet pushdown, and residual Compute.filter; to_stream yields an Agent-backed :dataset stream with exact stats and scan telemetry. - closed #296 - closed #297 - closed #298 - closed #299 - closed #230 - closed #231 - closed #232 - closed #233
Check in a PyArrow hive fixture with exact Scanner prune stats, expand Expression tests, and document Dataset/Scanner via guide, README, notebook, and a labeled pushdown-ladder benchmark. - closed #304 - closed #305 - closed #306 - closed #307 - closed #308 - closed #309 - closed #310 - closed #311 - closed #312 - closed #313 - closed #314 - closed #315 - closed #316
Document struct fields, function parameters and options, and typical usage examples; add doctests for Expression builders and Memory/Local filesystem.
Guard Int→Float64 residual filters, compare fragment schemas by name and
type, return {:error,_} from Scanner.stats after close, and remove O(n²)
fragment list patterns plus related test and doc gaps.
Bump mix/crate/Livebook pins to 0.9.0, add CHANGELOG and release notes, refresh README/docs, and clear cargo clippy so the quality gate is green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M0: upgrade arrow-rs to 59.3.0 (tonic 0.14, ADBC 0.24)
arrow-rsmajor; read upgrade notes for parquet, arrow-flight (Flight SQL prepared-statement APIs require >= 56), and the FFI/CDI surface. #264arrow-*,parquet,arrow-flight; run full suite + cargoclippy --all-targets; regenerate `checksum-Elixir.ExArrow.Native.exs. #265tonic 0.14,adbc_core / adbc_driver_manager 0.24 (0.22 capped arrow at <58 and dual-versioned the tree; 0.24 accepts >=58, <60)#26656→59: Flight SQL client errors areFlightError(route through existingflight_error_to_term); #267