Repository navigation
feat: new agent template for tag and metadata filtering #33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
0a6df65
26bc34c
cb996e0
3d4d3bd
2054c45
e5eca60
6195e4c
63bf61c
569e5e0
78e0871
a1fb819
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Metadata Filtering Agent | ||
|
|
||
| The following examples demonstrate how SAS Retrieval Agent Manager (RAM) can use custom metadata to filter search results during retrieval. | ||
|
|
||
| The collection contains approximately 6,000 IT service management (ITSM) incidents with the following fields: | ||
|
|
||
| - incident_id | ||
| - description | ||
| - assigned_it_person | ||
| - category | ||
| - severity | ||
| - status | ||
| - date | ||
|
|
||
| ## Metadata | ||
|
|
||
| The following fields are configured as custom metadata columns in the RAM sources page: | ||
|
|
||
| - category (Access, Application, Database, Email, Hardware, Infrastructure, Network, Security, Software) | ||
| - severity (Low, Medium, High, Critical) | ||
| - status (Open, In Progress, Investigating, Resolved, Closed) | ||
| - date | ||
| - assigned_it_person | ||
|
|
||
| After assigning the desired metadata columns to the source file, you must create and vectorize a collection that the agent will use. | ||
|
|
||
| ## Approaches | ||
|
|
||
| After you create the collection, there are two ways to connect agents to it: | ||
| - [Tool-based experiment with agentic retrieval enabled](./tool_based/): | ||
|
Check failure on line 30 in examples/agents/metadata_filtering_agent/README.md
|
||
| - This is often the easiest option when users interact directly with the agent and are unlikely to know the underlying metadata schema. | ||
|
Check failure on line 31 in examples/agents/metadata_filtering_agent/README.md
|
||
| - In this approach, the LLM selects the appropriate filters to use based on the user's query. | ||
|
Check failure on line 32 in examples/agents/metadata_filtering_agent/README.md
|
||
| - [Template-based experiment](./template_based): | ||
|
Check failure on line 33 in examples/agents/metadata_filtering_agent/README.md
|
||
| - This approach gives the caller complete control over retrieval because filters are supplied explicitly to the agent. | ||
|
Check failure on line 34 in examples/agents/metadata_filtering_agent/README.md
|
||
| - A template-based approach may be a better fit when retrieval filters must be explicitly controlled, generated by another application, or applied consistently across repeated queries. | ||
|
Check failure on line 35 in examples/agents/metadata_filtering_agent/README.md
|
||
|
|
||
| ## How It Works | ||
|
|
||
| For each request: | ||
|
|
||
| 1. Apply metadata filters. | ||
| 2. Retrieve matching records. | ||
| 3. Rank results semantically. | ||
| 4. Return the most relevant matches. | ||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # Template-Based Experiment Metadata Filtering | ||
|
|
||
| This example uses a code template that applies explicit metadata filters supplied by the user. | ||
|
|
||
| ## Supported Metadata | ||
|
|
||
| - category | ||
| - severity | ||
| - status | ||
| - date | ||
| - assigned_it_person | ||
|
|
||
| ## Example Query | ||
|
|
||
| ```text | ||
| #filters: {"category":"Database","status":"Investigating"} | ||
| What database issues are currently under investigation? | ||
| ``` | ||
|
|
||
| RAM first applies the exact-match metadata filters and then performs semantic retrieval on the remaining records. | ||
|
|
||
| ## Why Use This Approach? | ||
|
|
||
| Use the template-based approach when: | ||
|
|
||
| - Exact filtering behavior is required | ||
| - Filters are generated programmatically | ||
| - Retrieval must be deterministic and reproducible | ||
| - Users already know which metadata filters should be applied |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| import json | ||
|
|
||
| from sasram.agent import Client | ||
|
|
||
|
|
||
| def _parse(text: str) -> tuple[str, dict[str, object]]: | ||
| content = text.strip() | ||
| # Without a directive, use the whole prompt as an unfiltered retrieval question. | ||
| if not content.lower().startswith("#filters:"): | ||
| return content, {} | ||
|
|
||
| # Optional first line: #filters: {"priority": "high"}; the question follows below. | ||
| directive, _, question = content.partition("\n") | ||
| try: | ||
| filters = json.loads(directive.split(":", 1)[1].strip()) | ||
| except json.JSONDecodeError as error: | ||
| raise ValueError("#filters must contain valid JSON.") from error | ||
|
|
||
| if not isinstance(filters, dict): | ||
| raise ValueError("#filters must contain a JSON object.") | ||
|
|
||
| return question.strip(), filters | ||
|
|
||
|
|
||
| async def exec(text: str, ram_client: Client) -> str: | ||
| try: | ||
| question, filters = _parse(text) | ||
| except ValueError as error: | ||
| return str(error) | ||
|
|
||
| if not question: | ||
| return "Include a question after the retrieval directives." | ||
|
|
||
| search_kwargs: dict[str, object] = {"k": 10} | ||
| if filters: | ||
| # Keys and values must match metadata already present in the collection. | ||
| search_kwargs["filter"] = filters | ||
|
|
||
| # RAM retrieves up to 10 matching results and generates the answer. | ||
| response = ram_client.post_query( | ||
| prompt=question, | ||
| search_kwargs=search_kwargs, | ||
| persistent=False, | ||
| ) | ||
| return response.response.answer | ||
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,45 @@ | ||||||||||||
| # Tool-Based Experiment Metadata Filtering | ||||||||||||
|
|
||||||||||||
| This example uses the default tool-based agent with Agentic Retrieval to automatically infer metadata filters from natural-language requests. | ||||||||||||
|
|
||||||||||||
| ## Configuration | ||||||||||||
|
|
||||||||||||
| When creating the agent experiment: | ||||||||||||
|
|
||||||||||||
| 1. Add the collection containing the IT incidents document. | ||||||||||||
| 2. Enable **Agentic Retrieval** toggle under Collections. | ||||||||||||
| 3. (Optional) Include instructions in the agent's system prompt on how/when to properly use the filters. | ||||||||||||
|
|
||||||||||||
| ## Example | ||||||||||||
|
|
||||||||||||
| User: | ||||||||||||
|
|
||||||||||||
| ```text | ||||||||||||
| Show me critical security incidents that are still open. | ||||||||||||
| ``` | ||||||||||||
|
|
||||||||||||
| The LLM can automatically infer filters such as: | ||||||||||||
|
|
||||||||||||
| ```json | ||||||||||||
| { | ||||||||||||
| "category": "Security", | ||||||||||||
| "severity": "Critical", | ||||||||||||
| "status": "Open" | ||||||||||||
| } | ||||||||||||
| ``` | ||||||||||||
|
|
||||||||||||
| before performing semantic search. | ||||||||||||
|
|
||||||||||||
| ## Why Use This Approach? | ||||||||||||
|
|
||||||||||||
| - No filter syntax required | ||||||||||||
| - Natural-language queries | ||||||||||||
| - Automatic metadata inference | ||||||||||||
| - Better experience for business users | ||||||||||||
| - Faster to configure than a custom filtering template | ||||||||||||
|
|
||||||||||||
| ## Agentic Retrieval vs Template-Based Filtering | ||||||||||||
|
|
||||||||||||
| Agentic Retrieval is often the easiest option when users interact directly with an agent and are unlikely to know the underlying metadata schema. | ||||||||||||
|
|
||||||||||||
| A template-based approach may be a better fit when retrieval filters must be explicitly controlled, generated by another application, or applied consistently across repeated queries. | ||||||||||||
|
Comment on lines
+41
to
+45
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,58 @@ | ||||||||||||||||||||||||||||||||||||||
| # File Tags Custom Source | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| This example generates three fictional IT support documents and applies different | ||||||||||||||||||||||||||||||||||||||
| file tags using the RAM Custom Source API. It demonstrates saving files and then | ||||||||||||||||||||||||||||||||||||||
| tagging them by their source filenames, including multiple tags on one file. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| ## How It Works | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| The entry point in [run.py](./run.py) writes three text files under `/tmp` | ||||||||||||||||||||||||||||||||||||||
| and saves each with `client.save_file(filepath)`. It then applies tags with | ||||||||||||||||||||||||||||||||||||||
| `client.add_tags_to_file(filename, tags)`, using the matching source filename | ||||||||||||||||||||||||||||||||||||||
| without the `/tmp/` prefix. | ||||||||||||||||||||||||||||||||||||||
| Each tag is a `SourceClient.FileTag` with a name and description. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| The password reset guide receives both its topic tag and an onboarding tag in | ||||||||||||||||||||||||||||||||||||||
| the same tagging call. The files are written directly under `/tmp`, not inside | ||||||||||||||||||||||||||||||||||||||
| a randomly named temporary directory, to keep source filenames stable. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| This example uses separate save and tag calls because `save_file(..., tags=...)` | ||||||||||||||||||||||||||||||||||||||
| was observed to produce an untagged file and a second, tagged file under a random | ||||||||||||||||||||||||||||||||||||||
| temporary path in the reported RAM runtime. This behavior may depend on the | ||||||||||||||||||||||||||||||||||||||
| runtime version; verify that synchronization produces only the three filenames | ||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+9
to
+22
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||||
| below with their expected tags. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| | Source File | Tags | | ||||||||||||||||||||||||||||||||||||||
| | ----------- | ---- | | ||||||||||||||||||||||||||||||||||||||
| | `password_reset.txt` | `account-access`, `employee-onboarding` | | ||||||||||||||||||||||||||||||||||||||
| | `vpn_connection.txt` | `networking` | | ||||||||||||||||||||||||||||||||||||||
| | `laptop_replacement.txt` | `hardware` | | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| The example does not clear the source or delete unrelated files. It uses fixed | ||||||||||||||||||||||||||||||||||||||
| filenames and tag names rather than generating new names for each synchronization. | ||||||||||||||||||||||||||||||||||||||
| If earlier versions created entries such as `tmp.../password_reset.txt`, remove | ||||||||||||||||||||||||||||||||||||||
| those obsolete entries manually; this example does not clean them up. | ||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+31
to
+34
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| ## Setup | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| Follow the [general custom source instructions](../README.md): | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| 1. Create a Custom Source template using [run.py](./run.py), then save and publish it. | ||||||||||||||||||||||||||||||||||||||
| 2. Create a source from the template and synchronize it. | ||||||||||||||||||||||||||||||||||||||
| 3. Inspect the source's files and confirm that their tags match the table above. | ||||||||||||||||||||||||||||||||||||||
| 4. Add the source to a collection and vectorize it for retrieval. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| No API keys, environment variables, LLM calls, or third-party packages are needed. | ||||||||||||||||||||||||||||||||||||||
| The example uses the Python standard library and `sasram` supplied by RAM. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| ## Adapting the Example | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| Replace `DOCUMENTS` with your own content, filenames, and tags. For files already | ||||||||||||||||||||||||||||||||||||||
| ingested into the source, use `client.add_tags_to_file` without uploading them again. | ||||||||||||||||||||||||||||||||||||||
| The API also provides `client.create_tag(tag)` to create a tag separately and | ||||||||||||||||||||||||||||||||||||||
| `client.untag_file(filename, tag_names)` to remove selected tags from a file. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| File tags label whole documents. They are separate from the record-level | ||||||||||||||||||||||||||||||||||||||
| metadata filters demonstrated by the | ||||||||||||||||||||||||||||||||||||||
| [metadata filtering agent](../../agents/metadata_filtering_agent/). | ||||||||||||||||||||||||||||||||||||||
| This source demonstrates applying tags, not configuring agent retrieval filters. | ||||||||||||||||||||||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| from pathlib import Path | ||
|
|
||
| from sasram.source import SourceClient | ||
|
|
||
|
|
||
| # Each entry contains a source filename, tag name, tag description, and document text. | ||
| DOCUMENTS = [ | ||
| ( | ||
| "password_reset.txt", | ||
| "account-access", | ||
| "Account access and password support.", | ||
| "Password reset guide\n\n" | ||
| "Use the organization's password reset portal to reset a forgotten password. " | ||
| "If you cannot complete identity verification, contact the IT help desk.\n", | ||
| ), | ||
| ( | ||
| "vpn_connection.txt", | ||
| "networking", | ||
| "Network connectivity and remote access support.", | ||
| "VPN connection guide\n\n" | ||
| "Connect to the internet, open the approved VPN client, and sign in. " | ||
| "If the connection fails, record the error message for the IT help desk.\n", | ||
| ), | ||
| ( | ||
| "laptop_replacement.txt", | ||
| "hardware", | ||
| "Device maintenance and replacement support.", | ||
| "Laptop replacement guide\n\n" | ||
| "Submit a device replacement request through the IT service portal. " | ||
| "Include the asset number and a description of the hardware problem.\n", | ||
| ), | ||
| ] | ||
|
|
||
|
|
||
| def exec(client: SourceClient): | ||
| for filename, tag_name, description, content in DOCUMENTS: | ||
| # Write directly under /tmp so source names do not include random subdirectories. | ||
| path = Path("/tmp") / filename | ||
| path.write_text(content, encoding="utf-8") | ||
| tags = [SourceClient.FileTag(name=tag_name, description=description)] | ||
| # A file can have multiple tags, such as a topic and an intended audience. | ||
| if filename == "password_reset.txt": | ||
| tags.append(SourceClient.FileTag( | ||
| name="employee-onboarding", | ||
| description="Guides relevant to new employees.", | ||
| )) | ||
| # Save and tag separately to avoid the tagged-save duplication seen in RAM. | ||
| client.save_file(str(path)) | ||
| # Tag the source filename, not the local /tmp path. | ||
| client.add_tags_to_file(filename, tags) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is persistent false here?