Repository navigation
Feature: Support dataframes in lui() calls via Arrow upload #27
Description
Feature Request
Description
Enable passing dataframes directly to lui() function calls using Arrow format for efficient data transfer.
Proposed API
from typing import Union, Dict, Optional
import pandas as pd
import pyarrow as pa
DfLike = Union[pd.DataFrame, pa.Table]
# Single dataframe
lui('analyze this data', df=sales_df)
# Multiple named dataframes
lui('compare these datasets', dfs={'sales': sales_df, 'inventory': inventory_df})
# Optional parameters
def lui(prompt: str, df: Optional[DfLike] = None, dfs: Optional[Dict[str, DfLike]] = None):
...Use Case
Users working with dataframes currently need to convert them to text or manually serialize. Direct dataframe support would:
- Eliminate manual serialization steps
- Preserve data types and schemas
- Enable efficient transfer of large datasets
- Work seamlessly with existing pandas/pyarrow workflows
Implementation Details
- Accept
pd.DataFrameandpa.TableasDfLiketypes - Automatically convert pandas DataFrames to Arrow format internally
- Use Arrow IPC format for efficient binary serialization
- Support both single dataframe (
dfparam) and multiple named dataframes (dfsparam) - Handle upload behind the scenes transparently
Example Usage
import pandas as pd
from louie import lui
# Single dataframe
sales_df = pd.DataFrame({'product': ['A', 'B'], 'revenue': [1000, 1500]})
response = lui('What are the top selling products?', df=sales_df)
# Multiple dataframes
inventory_df = pd.DataFrame({'product': ['A', 'B'], 'stock': [50, 30]})
response = lui(
'Which products need restocking based on sales velocity?',
dfs={'sales': sales_df, 'inventory': inventory_df}
)
# With pyarrow.Table
import pyarrow as pa
arrow_table = pa.Table.from_pandas(sales_df)
response = lui('Analyze this Arrow table', df=arrow_table)Benefits
- Clean, intuitive API matching existing lui() signature
- Zero friction for data scientists already using pandas/pyarrow
- Type safety with Union types
- Efficient binary format reduces latency
- Preserves nulls, timestamps, and other complex types correctly
Technical Considerations
- Arrow IPC format for wire protocol
- Automatic chunking for large dataframes
- Memory-efficient streaming where possible
- Graceful fallback for unsupported column types
🤖 Generated with Claude Code
Activity
onestardao commentedon Aug 6, 2025
nice proposal — arrow definitely helps with bulk transfer and dtype clarity.
just wanted to flag a pattern we’ve seen before when piping dataframes directly into logic chains or semantic functions like lui(). once the data hits downstream models, things can get tricky if column names are too abstract or symbolic, especially when their semantic weight isn’t preserved across the pipeline. this can lead to broken embeddings or logic freezes even when the dataframe looks technically fine.
we’ve encountered this in systems where tabular inputs silently degrade reasoning quality due to mismatched field semantics. if your downstream expects any sort of semantic routing or inference, it might be worth considering lightweight field descriptors or ΔS-based heuristics to avoid symbolic collapse.
happy to share what we tried if that becomes relevant later.
lmeyerov commentedon Aug 6, 2025
Thanks @onestardao --Curious if you have recs?
The 80% case is probably doing with least user friction:
(Maybe also txt, pdf, or binary -- but I think we special case semistructured data for this issue)
Pandas, and esp Arrow/parquet, is pretty good at encoding data representation schema, so we get that.
Agreed that default not great at semantic / ontology. User might have good col names for that, or maybe use metadata field to add more. Maybe that's enough here for the typical case, passing that to the AI system? Or should something more be passed?
onestardao commentedon Aug 6, 2025
yeah... you just described one of the core failures we tracked in open-source RAG setups:
semantic collapse not because of missing data — but because the wrong chunk API was used, or internal chunking logic wasn't properly surfaced upstream.
we documented this as Problem No.1: "Mismatch in Chunk API Call", and it's way more common than folks realize.
breakdown here (with solution ideas):
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md
hope it saves you 2 weeks of tracing a bug that was never in the data itself.
(been there.)
lmeyerov commentedon Aug 6, 2025
Hi @onestardao thank you, maybe you can help clarify -
Afaict, txt os has powerful RAG reasoning methods, and as part of that, careful data prep methods that compute useful data representations. Reductively, it sounds like we should imagine extending the df type signature to support passing in RAG indexes that agents with custom retrievers can take advantage of, and maybe an opinionated 'common case' recommendation.
For example, in graph rag, the summary index structures are different representations for capturing vector & text search indexes, an edge table mapping them to one another, and a source citation index. As another, txtos has its own data representations, though I don't have a clear view of that internalized.
A key part of lui is keeping things serializable and composable. I can imagine two paths to supporting that here:
Ex: For GraphRAG, Imagine vector_df, edges_df, etc. that have different shapes and pluggable types vs original df. We would leave their handling uninterpreted and up to a specialized agent. This feels close to about something like langchain would do wrt tools & agents.
Note that by supporting a richer structure than a dataframe, every agent can benefit now as long as it sticks to retrieval methods like search and ask, In addition to usual DF methods.
Unfortunately this introduces mobile code for defining custom retrievers, so opens up questions I don't have clear answers to. But I can imagine somebody having a loader script to import these custom types depending on the environment...
Maybe you have thoughts here, here or can point to the specific type signatures that would be easy to understand?
onestardao commentedon Aug 6, 2025
great points — and yes, you’re absolutely right that aligning reasoning methods with structured data access (like dataframes or typed APIs) introduces sharp tradeoffs.
there are two main architectural frictions we’ve seen in systems like this:
this shows up in many real-world RAG setups when chunk metadata or index logic goes async — or when retrieval is optimized separately from agent alignment.
we wrote up some of this tension in our “rag architecture and recovery” notes, especially the parts on type flattening and post-hoc patching:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md
if useful, happy to share how we’ve worked around this with dual-mode agents or structure-preserving context bridges — let me know what setup you're working with!
lmeyerov commentedon Aug 6, 2025
I'm not sure how to make progress on your thinking. Maybe we should have a new issue about adding llmindex support, or a custom retriever type, as a way to support additional data types with rich & reusable retriever semantics? Would that solve your use case?
onestardao commentedon Aug 7, 2025
yeah, i think you're spot on
adding a new retriever class that encodes type signatures + bridges to structured semantics is definitely one clean way to go.
happy to jam on the interface design or show how we’ve been doing type‑preserving reasoning across agent hops if it helps — no pressure though, sounds like you're already on a great track
lmeyerov commentedon Aug 7, 2025
Awesome! Yeah I'll file a follow-up issue to explore this