Truncation of message histories and individual messages to prevent context window overflows

[janspoerer]

This PR aims to add max_context_length_per_message and max_context_length_total as parameters to the FastAgent configuration.

[janspoerer]

Work in progress: File /Users/janspoerer/code/agents/fast-agent/src/m…cp_agent/llm/providers/augmented_llm_google_native.py, line 520, in _truncate_message_history if self.max_context_length_per_message is not None

[janspoerer]

Some help would be appreciated. I am stuck correctly passing the arguments down to the Google Provider.

Also, I would like to implement this for all providers, if possible. But we can also start with only one.

[evalstate]

Sorry, I missed this message yesterday. Can you take a look at release 0.2.34 as this is intended to make Usage, Context info. etc easy to work with auto-truncation etc. etc! hop on to discord if you want to chat.

[janspoerer]

Thanks for pointing this out. Missed that feature around context management.

I'd very much like to chat and will ping you when I come online on Discord (maybe later today or sometime over the week).

[evalstate]

It was only finished a couple of hours ago :) If you're using Anthropic models, I'd appreciate if you could try running with the feat/anthropic-cache branch if possible.

[janspoerer]

Oh great, we had similar thoughts about useful next features :-)

I responded here: #247

I may be available on Discord this week, but cannot yet say when exactly. I have some travel this week (to an AI conference, of course :-) ).

Hope to talk to you soon!

@evalstate Looks great, I can't wait to try the new tracking feature.
If I see this correctly, the usage is mainly there for informational purposes.
Is there also a truncation step anywhere that truncates either individual very long messages or that removes entire messages?
Or is this something that we should discuss on Discord as a next step?
(Refers to our short exchange in PR #246.)

[janspoerer]

If you're using Anthropic models, I'd appreciate if you could try running with the feat/anthropic-cache branch if possible.

Just to clarify: I should merge the feat/anthropic-cache branch into this one and see if this works?

@evalstate

[janspoerer]

Opened another PR that covers this feature: #311