Repository navigation
[DISCUSSION] Is DataFusion's implementation growing too complicated? Are AI assisted PRs to blame? #26257
Description
TLDR; I worry that the ease of adding features with AI tools is leading DataFusion to become overly complicated and that the additional complexity will eventually slow down / ruin the project
I think @2010YOUY01's way of phrasing my concern in this comment: #24768 (comment)
My imaginary failure mode is, after exceeding some complexity threshold, AI always fix one bug by bringing in two new bugs
AI tools are amazing in terms of how much code you can write and how much it lowers the barriers to contribution. However, I worry about the cumulative effect on performance and understandability over the long term
Recent Examples:
- fix(substrait): Preserve pushed-down table scan offsets and limits #26035 from @viirya where there was quite a bit of test and code to handle LIMIT + OFFSET that overflows an i64 (which I think would never happen in practice);
- fix: do not rewrite
SUM(x + literal)for floating point arguments #26241 which looks very plausible, but I think the veneer of plausibility masks the fact it isn't really a bug.
I am not sure if anyone really has the experience yet to know if this is/will happen but I worry about it
Activity
haochunchang commentedon Oct 13, 2026
Just want to chime-in with my two-cents. :)
imo it is the drift between mental model and real codebase that will cause doubt/fear.
By addressing this, I still read through the AI generated code changes (and/or asking questions or use something like explain-diff skill to quiz myself) to understand as thoroughly as time allowed.
As one of the action items, my idea is to add safety rails that could help us understand the code changes, for example:
Maybe starting by refining the contributor guides section to be the context for AI review agents?
For performance, maybe encouraging PR authors to run benchmark to see if the changes would have performance degradation (since my understanding is that performance is really essential for datafusion)?
I also like the PR limits for each user, so that we can focus on what we have started.
alamb commentedon Oct 13, 2026
The intention was that this page is the context I think: https://datafusion.apache.org/contributor-guide/pr_review.html
Maybe we can add an explicit link 🤔
mbutrovich commentedon Oct 13, 2026
It's a fairly complicated project to begin with, and IME agent-generated PRs often lack the context for proper code reuse (either internally or from stuff like arrow-rs) and like to reinvent the wheel. I find myself looking at a number of PRs (here, Iceberg, Comet, etc.) thinking "don't we have code for this already?"
It's making reviewing even harder because new contributors show up (great!) with limited context and a ton of tokens to burn and dump massive PRs on us that pass the initial smell test, but if you dig into the details you find they were caused by the agent reward hacking or poorly scoped requirements to begin with.
I don't really have a solution here other than "actually we should move more slowly." Review cycles should be longer, particularly as average PR line count seems to be growing. I'm almost wondering, since reviews are so heavily AI-assisted, if we should consider bumping the threshold to two approving reviews.
haochunchang commentedon Oct 13, 2026
Do you mean maybe adding an explicit link to the page inside somewhere that AI agent can automatically pick up when contributors are using agents, for example, AGENTS.md (which already includes the link)?
haochunchang commentedon Oct 13, 2026
I think both longer review cycle, and assisting new-comers' agents to collect context all aim to build understandability among the maintainers. From a new comer's POV, maybe start by AI-assisted PR review can help understanding the codebase more quickly.
( I confess I also have opened massive PR, will address them later.)
pepijnve commentedon Oct 13, 2026
I got the feeling expressed in the description when @bert-beyondloops and I were dealing with incorrect/sub-optimal optimisation passes introduced by leaf expression pushdown. He expressed this concern in #25329 (comment)
I really don't want to throw any of the involved developers under the bus here. Just trying to express that both of us are having a hard time getting our heads around the concepts being introduced into the API (the notion of expression placement in this case). This makes it quite difficult to collaborate on fixing things. At the same time the LLM train keeps on thundering down the tracks while requests for clarification for the human trying to review things (me in this case) go unanswered. The end result is that we've basically given up and just turned the optimisation in question off for now.
phillipleblanc commentedon Oct 13, 2026
I found this post to be aligned with my current thinking on how AI is transforming software engineering: https://thorstenball.com/blog/2026/09/19/what-i-believe-about-the-future-of-software-development/
I don't think this will happen, and if it does then it can be solved. The frontier models are getting so good that most problems can be solved with more tokens.
However... I do think DataFusion falls into this edge case called out by the blog post above:
Telling Claude "make datafusion faster, no mistakes" won't work - that will still require human judgement and creativity. But all of the grunt work and stuff that would have distracted from focusing on that core can be handled by AI.
I do think DataFusion has the benefit of focusing on being a small extensible core rather than trying to solve the world. If anything we should focus more in that direction to enable the "building block economy": https://mitchellh.com/writing/building-block-economy And be ruthless about saying no and telling people to build on top of the building blocks that exist.
manuzhang commentedon Oct 13, 2026
Some personal experience to share. As a reviewer, I try to understand AI's review comments first, push hard on each point, and post comments by hand. As an author, I try to split big AI-assisted PRs into smaller ones to make it easier for review.
We learn from contributing and reviewing, and can steer the wheel when AI is going in the wrong direction. That makes me less worried.
comphead commentedon Oct 13, 2026
Yeah, this question is super fair.
IMO AI or not AI if the project follows to healthy project metrics below, it would mitigate lots of regressions.
The AI development is new normal, and we cannot get rid of it, but we need to make sure the AI generated content follows standards that we expect
Code quality
Testing
mainDependencies and API
Performance
Docs
Process
Please find LOC analytics for last 7 releases, the codebase is growing faster than ever
Some open issues to track code quality metrics:
Performance tracking and benchmarks
Binary size and build time
datafusion-clirequires more than 8GB to build #18471datafusion-clirequires more than 8GB to buildviirya commentedon Oct 13, 2026
To clarify, the PR was not intended to handle LIMIT + OFFSET overflowing an i64. 😄 The part @alamb mentioned was added to address a review comment. I was focused on addressing the comment and didn't think too much about it.
On this topic, I think using AI for coding, reviewing, and contributing to OSS projects like DataFusion is an unstoppable trend. Since we are still in the early wave of this trend, some chaos and concerns are unavoidable. AI is a tool, so ultimately the question is how we use it to achieve higher productivity and better results.
For example, I should have looked more carefully at the comment asking to handle i64 overflow. But even without AI, I might have misjudged the request and implemented the i64 handling anyway. AI just makes that easier, because I can simply ask it to address the comment. So I think everyone is still learning how to use AI at this early stage, and we can do better by reflecting like this each time.
I think encouraging PR authors to run benchmarks is helpful in many cases. With the help of AI, it is actually easier and less tedious to do.
The review process definitely helps, whether it comes from humans or from AI (with human direction).
asolimando commentedon Oct 13, 2026
Sharing a few ideas:
requiring an in-depth design discussion for all non-trivial new features
requiring a real SQL/SLT reproducer for bug fixes
perf- improvements: other than beating benchmarks, improvements should make sense on their own and they should make sense beyond pure numbers, like an intuition of why the perf improvment is good, why it would generalize beyond the current benchmarks, what are the blind spots, etc
Why I think they would help:
would probably reintroduce a bit of the friction AI tools lowered, but of a good kind, people must think things through and be able to condense complex concepts in quality examples and summarized explanations. It would also take a bit longer to get changes in, but for a good reason
is what Andrew has been frequently asking lately, and it helps removing all bugs that can't be reached by the end user, and it would make it easier to spot plausible but marginal bugs
limits the overfitting to existing benchmarks which are never really representative and might makes things worse over time if we overdo
AI amplifies good and bad habits, people have always been showing up in OSS projects trying to improve their downstream use cases without doing the effort to improve the abstractions and the general health of the project, it just became simpler to do that, but some healthy gatekeeping will help keeping quality standards and maintenability