Repository navigation
Add 'Best Practices for Single Row Lookups' post - #209
nuno-faria wants to merge 1 commit into
Conversation
| !*/blog/ | ||
| infrastructure-actions/ | ||
|
|
||
| __pycache__ |
There was a problem hiding this comment.
The local build was adding these folders.
|
I see this and plan to review it |
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you @nuno-faria -- this is great. I think it will be very widely read and quite useful
I left a bunch of small comments, but I don't think any of them are required to publish this
Thanks again
|
|
||
| ## Methodology | ||
|
|
||
| The dataset we will be using here is formed by a single Parquet file with 100M |
There was a problem hiding this comment.
I don't know if it is possible, but providing a link here to the parquet file or the generator you used to make one, I think would make this post stronger / more concrete
| retrieve a random row from the table by key. Each test builds on the last. | ||
|
|
||
|
|
||
| ## Storage |
There was a problem hiding this comment.
Suggestion: add the word Parquet to the title somehow. You say this in the text, but i think it would also help to say in the titles again
Maybe something like
Storage (Optimizing Parquet Settings)
| [DuckDB]: https://duckdb.org/docs/lts/sql/statements/copy#parquet-options | ||
|
|
||
|
|
||
| ## Execution |
There was a problem hiding this comment.
similarly here I think it would help to highlight the DataFusion aspect of this
Execution (Optimizing DataFusion Settings)
| The dataset we will be using here is formed by a single Parquet file with 100M | ||
| rows (unless otherwise stated), using a key (`bigint`, unique) and value | ||
| (`varchar`, md5 of the key) columns. Unless otherwise stated, the queries tested | ||
| retrieve a random row from the table by key. Each test builds on the last. |
There was a problem hiding this comment.
It might also help to have a specific SQL example of this
select ... FROM ... That would also explain if this is a SELECT * or a SELECT col type query
| </div> | ||
|
|
||
| In summary, at least for sorted files with many unique values, it is better to | ||
| **disable both of these structures** when writing Parquet files. |
There was a problem hiding this comment.
It would be nice here to include links / examples of how to enable / set these configurations to make the recommendation concrete
For example a link to ArrowWriterProperties and the parquet writer settings `COPY ... FORMAT paruqet OPTIONS ..) and https://datafusion.apache.org/user-guide/sql/format_options.html#parquet-format-options
There was a problem hiding this comment.
or maybe you can add those links somewhere in the intro to parquet tuning
| # pushdown_filters disabled | ||
| FilterExec (output_rows=1) | ||
| DataSourceExec (output_rows=1k) | ||
|
|
||
| # pushdown_filters enabled | ||
| DataSourceExec (output_rows=1) |
There was a problem hiding this comment.
As written I found this confusing as the filter expression looks like it gets lost.
I recommend showing where the filter expression goes . perhaps something like this
| # pushdown_filters disabled | |
| FilterExec (output_rows=1) | |
| DataSourceExec (output_rows=1k) | |
| # pushdown_filters enabled | |
| DataSourceExec (output_rows=1) | |
| # pushdown_filters disabled | |
| FilterExec key=<VAL> (output_rows=1) | |
| DataSourceExec (output_rows=1k) | |
| # pushdown_filters enabled | |
| DataSourceExec key=<VAL> (output_rows=1) |
| For both smaller (<10k rows) and larger (>100k rows) reads, we conclude that | ||
| **enabling `pushdown_filters` results in a higher throughput**. Curiously, | ||
| medium sized reads (between 10k and 100k) are faster with `pushdown_filters` | ||
| disabled. It's not clear why this is the case, but I retested this multiple |
There was a problem hiding this comment.
It is almost certainly some combination of
- Different I/O patterns (pushdown filter fetches the predicate columns in a separate I/O)
- Filter representation efficiency (until recently, pushdown filter represents filter results as an RLE ype structure,
RowSelection).
@hhhizzz and others have been improving this over time
| As of the writing of the post, this feature is currently disabled by default due | ||
| to some performance regressions. More info can be found at [#3463] and [#20324]. |
There was a problem hiding this comment.
| As of the writing of the post, this feature is currently disabled by default due | |
| to some performance regressions. More info can be found at [#3463] and [#20324]. | |
| As of the writing of the post, this feature is currently disabled because it slows some queries | |
| down. More info can be found at [#3463] and [#20324]. |
| While reusing the logical plan offers an improvement over not reusing anything, | ||
| it's clear that **reusing the physical plan leads to a considerably higher | ||
| throughput**, doubling or even tripling performance. So when can we expect | ||
| prepared statements in native DataFusion that reuse the physical plan? First, we |
There was a problem hiding this comment.
It seems strange to list here the work items that would be needed to implement a DataFusion feature.
I recommend filing a ticket with the details and just refer to it here "See XYZ ticket for details"
| by releasing the current thread to another task. Andrew Lamb has given a talk at | ||
| Tokio Conf 2026 titled "[Using Tokio for CPU-Bound Tasks (Works Really Well)]" | ||
| which talks about this. |
There was a problem hiding this comment.
Nits to make this slightly more concise:
| by releasing the current thread to another task. Andrew Lamb has given a talk at | |
| Tokio Conf 2026 titled "[Using Tokio for CPU-Bound Tasks (Works Really Well)]" | |
| which talks about this. | |
| by releasing the current thread to another task. Andrew Lamb gave a talk at | |
| Tokio Conf 2026 titled "[Using Tokio for CPU-Bound Tasks (Works Really Well)]" | |
| about this. |
|
Thanks @alamb for the review. I will update the post soon. |
2 participants
Blog post based on my talk given at the Boston DataFusion meetup in September.