KERIA Deployment Best Practices

[kentbull]

If you are deploying KERIA and could use some guidance then this discussion is for you. This will be updated as I gain new information on KERIA, KERIpy, and other KERI deployment best practices on any KERI implementation.

Overview

TOC

1. Deployment
   1. Component List
   2. Container-based deployments
      1. Kubernetes
2. Configuration
   1. KERIA configuration file
   2. Witness configuration file
   3. Signify Client Configuration
3. Common Problems
   1. 401 Unauthorized on OOBI resolutions; IPEX Grant Notifications never showing up

Contents

Deployment

Component List

A basic component list for a KERIA deployment includes the following:

1. A KERIA server (this repo).
2. One or more witness servers.
3. An ACDC schema hosting server (could be an NGINX server or the vLEI server)
4. Web apps running a Signify Client (SignifyTS or SignifyPy), like the Cardano Foundation's cf-identity-wallet

Container-based deployments

Kubernetes

Configuration

Common Problems

401 Unauthorized on OOBI resolutions

Problem:

This can often happen when you have a network load balancer or firewall that has filtering rules that do not properly forward HTTP headers or otherwise allow requests from the HIO HTTP client KERIA uses to make HTTP calls to witnesses, KERIA agents, and schema hosting servers.

Solution

Relax the filtering rules on your load balancer or firewall. This solution is lacking specificity and will be extended as the specific guidance for load balancers and firewalls are determined.

[2byrds]

We revisited this during the vLEI dev community call and perhaps if enough info is collected here we could make this part of the source repo as a markdown file.

[kentbull]

I like that idea.

[lenkan]

One of my main concerns regarding KERIA deployments is dynamic scaling. Most use cases that we are working with at the moment are very "spiky" in nature. Our KERIA deployments will have very little to do 99% of the time, but 1% of the time, we have a lot of usage.

I would like it to be possible to horizontally scale a KERIA deployment, and I want to be able to do so dynamically based on the load that the deployment is currently under.

Currently, each instance needs exclusive access to its LMDB database, which means we cannot run two instances against the same database.

Any thoughts?

[lenkan]

@iFergal I had thought of something similar regarding the already independent sub-directories for the agent.

I guess you mean that we could have multiple KERIA instances connected to the same file system. We just need to ensure exclusive access to the agency DB and each of the agent DBs.

[iFergal]

The agency DB probably needs some thought. The only write actions are creating new agents and deleting old ones so maybe a lock on that is not a bottleneck, but maybe that info could also be stored elsewhere so it's separate per agent. Not sure.

For the rest of the files at least, maybe the same file system, maybe there's some DevOps wizardry that can be done there to load and unload agent DBs into volumes without risking data loss. I could brainstorm with my devops lead.

[kentbull]

If the database structure was re-organised to partition the full DB of each agent, you could at least horizontally scale each agent

@iFergal what kind of load or use case are you trying to scale, a transaction processing use case or a redundancy and failover use case? And When you say scale one agent are you talking about replicating the same LMDB state across multiple instances of a running Agent? If so then trying to scale by replicating agent state will not get you what you want.

If you are talking about redundancy for failover and disaster recovery then I can see replication of KERIA Agent state as being useful.

The Agent is really just supposed to be two things: 1) a mailbox and 2) a delegation communication proxy. Since the Agent does not directly perform any signing at all of the edge events and is just a communication relay and proxy then scaling the Agent horizontally will not speed up issuance or anything else in a measurable way as the human will always be the bottleneck.

If, instead, you want to scale to a high transaction volume then a direct mode KERI Agent, like ReportingAgent here, would be better suited to your purposes. Since, in the Signify keys at the edge model, you have to send all events to the user for signing then the communications across the agent aren't really the bottleneck. The bottleneck is the human performing signing operations. If you do want to scale transaction load then some always on service in direct mode (runs its own mailbox) is likely your best solution.

If we do want to end up looking at the agency database here are some interesting facts about the key value pairs in KERIpy and KERIA:

• KERIpy: 155 key value pairs total (as of January 2025-ish)
  • Baser (KEL events): 79 key value pairs
  • Reger (TEL events) 65 key value pairs
  • Keeper (keys): 6 key value pairs
  • Noter (notifications): 3 key value pairs
  • Mailboxer: 2 key value pairs)
• KERIA: 22 key value pairs total (as of January 2025-ish)
  • AgencyBaser: 3 key value pairs
  • RemoteKeeper: 8 key value pairs
  • Operator: 1 key value pair
  • Seeker: 2 key value pairs
  • ExnSeeker: 8 key value pairs

The takeaway is that KERIA is small compared to KERIpy and that the Mailbox is very small. If we wanted to change things up then it's doable.

I think there's a delegation flag to indicate that the delegatee and delegator are the same entity, so it's for scaling.

You are correct. The TraitCodex.DelegateIsDelegator inception configuration trait indicates a delegate is to be treated the same as their delegator.

[lenkan]

I haven't yet posted any reproducible issues. But, in my experience, a keria instance will run at 100% CPU when you have ~100 agents on it. This is when it is idling. I intend to do proper measurements of how this affects the responsiveness and performance for doing actual work.

[iFergal]

@kentbull

what kind of load or use case are you trying to scale, a transaction processing use case or a redundancy and failover use case?

Scaling the processing (but not within a single agent), or how many agents it can concurrently handle. It's not something I need now but it's easier to make any changes now while the community is small. Redundancy and failover would also be cool.

And When you say scale one agent are you talking about replicating the same LMDB state across multiple instances of a running Agent? If so then trying to scale by replicating agent state will not get you what you want.

No, what I was suggesting was related to have the KERIA deployment be able to handle many more agents by horizontally scaling. I think having multiple instances for the same agent would cause a lot of problems. So taking advantage of DelegateIsDelegator where its running across multiple agents would be compatible with this.

The Agent is really just supposed to be two things: 1) a mailbox and 2) a delegation communication proxy. Since the Agent does not directly perform any signing at all of the edge events and is just a communication relay and proxy then scaling the Agent horizontally will not speed up issuance or anything else in a measurable way as the human will always be the bottleneck.

This assumes the Signify client is a human. It could be a backend or IoT device that needs to sign at scale. But again, this is about scaling the number of agents since the backend/device could delegate to itself across many agents.

[kentbull]

I have a blog post with a section on KERIA deployment best practices here: https://www.kentbull.com/posts/configuring-keripy-keria-controllers-witnesses/#part-4-keria-agency-configuration