Repository navigation

Add a memory bound FileStatisticsCache for the Listing Table - #20047

Merged

alamb merged 92 commits into

apache:mainfrom

mkleen:file-stats-cache

May 18, 2026

Commits on May 14, 2026
- Add a default FileStatisticsCache implementation for the ListingTable
  mkleen
  committed
  537c95d
- fixup! Add a default FileStatisticsCache implementation for the ListingTable
  mkleen
  committed
  19f1bb2
- Adapt memory usage when removing entries
  mkleen
  committed
  a0df5c3
- Adapt heapsize for &str
  mkleen
  committed
  6416ace
- Fix formatting
  mkleen
  committed
  728db11
- Adapt heapsize for &str and add another scalarvalue
  mkleen
  committed
  8471d62
- Add better error message
  mkleen
  committed
  8541e70
- Add todo to add heapsize for ordering in CachedFileMetadata
  mkleen
  committed
  c4acbbc
- Fix comment/docs on DefaultFileStatisticsCache
  mkleen
  committed
  3202fc7
- Simplify test data generation
  mkleen
  committed
  c1b9df6
- Remove potential stale entry, if entry is too large
  mkleen
  committed
  71fe3d9
- Fix typo in sql logic test comment
  mkleen
  committed
  9ff009a
- Fix comment about default behaviour in cache manager
  mkleen
  committed
  07ce945
- Fix variable name in test
  mkleen
  committed
  c4531f7
- Fix variable name in test
  mkleen
  committed
  069c3e4
- Disable cache for sql logic test
  mkleen
  committed
  41535e7
- Include key into memory estimation
  mkleen
  committed
  d75a8db
- Fix fmt
  mkleen
  committed
  f7f9c6a
- Fix clippy
  mkleen
  committed
  5cfa8cd
- minor
  mkleen
  committed
  5087d4f
- Add more key memory accounting
  mkleen
  committed
  01d0b90
- Fix Formatting
  mkleen
  committed
  04dbf42
- Account path as string and remove dependency to object_store
  mkleen
  committed
  ee7518d
- Improve error handling
  mkleen
  committed
  44b3b52
- Fix fmt
  mkleen
  committed
  890d477
- Remove path.clone
  mkleen
  committed
  51df9a9
- Simplify accounting for statistics
  mkleen
  committed
  4d12496
- Adapt offset buffer
  mkleen
  committed
  b120b64
- Fix heap size for Arc
  mkleen
  committed
  66cbe38
- Adapt estimate in test
  mkleen
  committed
  c3cd8ae
- Fix sql logic test
  mkleen
  committed
  7395973
- Register cache from cachemanager at listing table
  mkleen
  committed
  caab254
- Revert slt
  mkleen
  committed
  6b08b90
- Add tablescoping for file stats cache
  mkleen
  committed
  296e718
- Adapt slt
  mkleen
  committed
  9b437c4
- Fix linter
  mkleen
  committed
  58a37e5
- Remove uneeded clone
  mkleen
  committed
  d0674c0
- Rename cache_unit to file_statistics_cache
  mkleen
  committed
  9fba2a9
- Simplify heap size accounting
  mkleen
  committed
  6745e07
- Adapt comments in test
  mkleen
  committed
  d610ea0
- Seperate drop table clean-ups
  mkleen
  committed
  b061eca
- fixup! Seperate drop table clean-ups
  mkleen
  committed
  8adb367
- Increase default limit to 10 mb
  mkleen
  committed
  5edee6e
- Increase default limit to 20 mb
  mkleen
  committed
  6d2266c
- Fix comment
  mkleen
  committed
  1bcfc75
- Fix deregister logic
  mkleen
  committed
  b8ecb11
- Fix slt
  mkleen
  committed
  a6439dc
- Add table reference to FileStatisticsCacheEntry
  mkleen
  committed
  2e30148
- fixup! Add table reference to FileStatisticsCacheEntry
  mkleen
  committed
  62e1c61
- Fix comment
  mkleen
  committed
  fb48bed
- Fix runtime_env entry
  mkleen
  committed
  b8acd8b
- Add cache for all benchmark runs
  mkleen
  committed
  1e02af5
- Add cache to listing table creation
  mkleen
  committed
  ffefbb8
- fixup! Add cache to listing table creation
  mkleen
  committed
  eb4a02f
- Adapt limit to 20M in configs.md
  mkleen
  committed
  ca56146
- fixup! Adapt limit to 20M in configs.md
  mkleen
  committed
  d3763da
- Fix linter
  mkleen
  committed
  a574ba2
- Add cache to listing table in _read_type()
  mkleen
  committed
  d1aa9c7
- Add ListView and LargeListView to heapsize
  mkleen
  committed
  3feb3bf
- fixup! Add ListView and LargeListView to heapsize
  mkleen
  committed
  b6126ca
- Remove array.slt
  mkleen
  committed
  2455918
- Add table ref to ListingTableUrl
  mkleen
  committed
  1a0f1b3
- Add heapsize for table-scoped-path
  mkleen
  committed
  3ebdbb0
- Make list_entries table-scoped
  mkleen
  committed
  69c3737
- fixup! Make list_entries table-scoped
  mkleen
  committed
  1cf57f9
- fixup! fixup! Make list_entries table-scoped
  mkleen
  committed
  2b1e926
- Improve heap size estimation for Arc
  mkleen
  committed
  82ea7f3
- fixup! Improve heap size estimation for Arc
  mkleen
  committed
  77c7e49
- Update migration guide
  mkleen
  committed
  bfd21a1
- fixup! Update migration guide
  mkleen
  committed
  a856387
- Improve heapsize estimation for TableReference
  mkleen
  committed
  eae9084
- Improve memory handling when inserting
  mkleen
  committed
  dba8fb9
- Fix comments in Cache Manager
  mkleen
  committed
  a0274fa
- Improve upgrade guide
  mkleen
  committed
  904a3e5
- Fix upgrade guide
  mkleen
  committed
  6a57aca
- Return stale entries from cache
  mkleen
  committed
  cd7db86
- Fix upgrade guide
  mkleen
  committed
  444004f
- Fix Arc<str> heapsize test
  mkleen
  committed
  5bcaa2e
- Remove const i32 cast from heapsize estimation
  mkleen
  committed
  112f7d6
- Fix heapsize estimation for Arc<T>
  mkleen
  committed
  39e150a
- Fix comment in cache_manager
  mkleen
  committed
  f7d85ab
- Fix linter + clippy
  mkleen
  committed
  5fa40c9
- Adapt test acording to heapsize estimation changes
  mkleen
  committed
  ae3d00b
- Always add tableref to partioned files
  mkleen
  committed
  6ba29d9
- fixup! Always add tableref to partioned files
  mkleen
  committed
  63e750d
- Add table to statistics_cache output
  mkleen
  committed
  abccc56
- Adopt test to new output
  mkleen
  committed
  9bd99a9
- Adopt configuration with '0' value
  mkleen
  committed
  c2450d4
- Update configs.md
  mkleen
  committed
  eb98fcb
- Add reset after show in slt
  mkleen
  committed
  4c8bd64
- Extract cache invalidation logic
  mkleen
  committed
  40b4550
Commits on May 15, 2026
- Merge branch 'main' into file-stats-cache
  mkleen
  authored
  f648bee

Commits

Commits on May 14, 2026

Commits on May 15, 2026