An S3-compliant service for nemo-retriever

[veerkumar]

To contribute back to the community, we developed an S3-compliant client for nemo-retriever.

When we started using nemo-retriever for document parsing with an S3 bucket, we quickly realized the need for a service that could connect data storage to the embedding generation pipeline (which we call the AI Data Pipeline, or AIDP). To keep our vector database updated with the latest embeddings whenever a new document is added to the bucket, we needed to automate data ingestion.

In addition, we required a way to customize the processing tasks based on file type. For example, a .txt file does not require an image extraction task. To address these needs, we developed a scalable service that automates the entire workflow.

More details and source code are available at: https://github.com/nutanix/nutanix-aidp-connector.

[krishkhinchi]

This is actually a pretty useful addition. Automating ingestion + embedding updates from S3 events solves a very common real-world workflow problem for RAG systems.

The file-type-specific pipeline customization is also a nice touch since different document formats often need very different processing steps.