+
+| Element | +Description | +
|---|---|
| failOnWarnings | +If true, the maven build will fail upon warnings for e.g. using Vespa types that are not annotated with @PublicApi. This should always be set to *true* to ensure that your project will compile successfully on future Vespa releases. Default is *false* | +
| allowEmbeddedArtifacts | +A comma-separated list of maven artifacts to allow embedding in the bundle, on the format *groupId:artifactId* | +
| attachBundleArtifact | +Whether to attach the bundle jar artifact to the build. Use this if you want to install and deploy the bundle jar along with the default jar. Default is *false* | +
| bundleClassifierName | +If *attachBundleArtifact* is true, this will be used as classifier for the bundle jar artifact. Default is *bundle* | +
| discApplicationClass | +The fully qualified class name of the Application to be started by JDisc | +
| discPreInstallBundle | +The name of the bundles that jDISC must pre-install | +
| bundleVersion | +The version of this bundle. Defaults to the Maven project version | +
| bundleSymbolicName | +The symbolic name of this bundle. Defaults to the Maven artifact ID | +
| bundleActivator | +The fully qualified class name of the bundle activator | +
| configGenVersion | +The version of *com.yahoo.vespa.configlib.config-class-plugin* that will be used to generate config classes | +
| configModels | +List of config models | +
| + | + |
|---|---|
| Compatible Changes | +These schema mismatches are handled automatically by the configserver: - C is missing a config value that S has: The server will omit that value from the response. - C has an additional config value with a default value: The server will include that value in the response. - C and S both have a config value, but the default values differ: The server will use C's default value. |
+
| Incompatible Changes | +These schema mismatches are not handled by the config server, and will typically lead to error in the subscription API because of missing values (though in principle some consumers of config may tolerate them): - C has an additional config value without a default value: The server will not include anything for that value. - C has the type of a config value changed, for example from string to int: The server will print an error message, and not include anything for that value. The user must use an entirely new name for the config if such a change must be made. |
+
| + | + |
|---|---|
| **Build** | +All the Vespa sample applications use the bundle plugin to build the components. | +
| **Configure** | +A key Vespa feature is code and configuration consistency, deployed using an application package. This ensures that code and configuration is in sync, and loaded atomically when deployed. This is done by generating config classes from config definition files. In Vespa and application code, configuration is therefore accessed through generated config classes. The Maven target {`generate-sources`} (invoked by {`mvn install`}) uses metal-names.def to generate {`target/generated-sources/vespa-configgen-plugin/com/mydomain/example/MetalNamesConfig.java`}. After generating config classes, they will resolve in tools like IntelliJ IDEA. |
+
| **Tests** | +Examples unit tests are found in MetalSearcherTest.java. {`testAddedOrTerm1`} and {`testAddedOrTerm2`} illustrates two ways of doing the same test: The first setting up the minimal search chain for YQL programmatically. The second uses {`com.yahoo.application.Application`}, which sets up the application package and simplifies testing. Read more in unit testing. |
+
| Return code | +Description | +
|---|---|
{`Progress.DONE`} |
+ Returned if a document processor has successfully processed a {`Processing`}. |
+
{`Progress.FAILED`} |
+ Processing failed and the input message should return a *fatal* failure back to the feeding application, meaning that this application will not try to re-feed this document operation. Return an error message/reason by calling {`withReason()`}. This result is represented as a {`500 Internal Server Error`} response in Document v1. Example: {`if (op instanceof DocumentPut) { return Progress.FAILED.withReason("PUT is not supported"); }`} |
+
{`Progress.INVALID_INPUT`} |
+ Available since 8.584. Processing failed due to invalid input, like a malformed document operation. This result is represented as a {`400 Bad Request`} response in Document v1. |
+
{`Progress.LATER`} |
+ See execution model. The document processor wants to release the calling thread and be called again later. This is useful if e.g. calling an external service with high latency. The document processor may then save its state in the {`Processing`} and resume when called again later. There are no guarantees as to *when* the processor is called again with this {`Processing`}; it is simply appended to the back of the input queue. By the use of {`Progress.LATER`}, this is an asynchronous model, where the processing of a document operation does not need to consume one thread for its entire lifespan. Note, however, that the document processors themselves are shared between all processing operations in a chain, and must thus be implemented thread-safe. |
+
| Exception | +Description | +
|---|---|
{`com.yahoo.docproc.TransientFailureException`} |
+ Processing failed and the input message should return a *transient* failure back to the feeding application, meaning that this application *may* try to re-feed this document operation. | +
{`RuntimeException`} |
+ Throwing any other {`RuntimeException`} means same behavior as for {`Progress.FAILED`}. |
+
| Handler | +Description | +
|---|---|
| DemoHandler | +A handler that modifies a request before dispatching it to the {`ProcessingHandler`}. This handler is also used in the HTTP API tutorial. Note that since this depends on ProcessingHandler you must add {`processing`} to your {`container`} tag to use it. If you want to issue Queries instead, have com.yahoo.search.searchchain.ExecutionFactory injected instead and use it to create executions and call search/fill on them. |
+
| Deployment environment | +Automated operations | +Suitable for | +
|---|---|---|
| Vespa self-managed/open source | +Application deployment (single application, single instance), application change (except rolling restarts), data redistribution, failover | +Development | +
| Vespa Kubernetes Operator | +Application deployment (single application, single instance), application change, data redistribution, failover, node provisioning, failed node replacement, node type change, autoscaling, endpoint routing, encryption | +Production in environments outside hyperscalers | +
| Vespa Cloud | +Application deployment (multiple applications, instances, regions, clouds), application change, data redistribution, failover, node provisioning, failed node replacement, node type change, autoscaling, endpoint routing, encryption, Vespa platform and OS upgrades, continuous deployment pipeline with verification, metrics and management console | +Development, production on hyperscalers (including in customer accounts and VPCs) | +
| Plan | +Quota | +
|---|---|
| Trial | +\$2/hour | +
| All other plans | +\$10/hour | +
| + | + |
|---|---|
| **fast-search** | +Also see the reference. Add an index structure to improve query performance: |
+
| **fast-access** | +For high-throughput updates, all nodes with a replica should have the attribute loaded in memory. Depending on replication factor and other configuration, this is not always the case. Use fast-access to increase feed rates by having replicas on all nodes in memory - see the reference and sizing feeding. |
+
| **distance-metric** | +Features like nearest neighbor search require a distance-metric, and can also have an {`hsnw index`} to speed up queries. Read more in approximate nearest neighbor. Pay attention to the field's {`index`} setting to enable the index: |
+
| Type | +Size | +Description | +
|---|---|---|
| Single-valued | +Fixed | +Like the "A" attribute, example {`int`}. The element size is the size of the type, like 4 bytes for an integer. A memory buffer (indexed by Local ID) holds all values directly. |
+
| Multi-valued | +Fixed | +Like the "B" attribute, example {`array. A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the *Multivalue Mapping* the arrays are stored. The *Multivalue Mapping* consists of multiple memory buffers, where arrays of the same size are co-located in the same buffer. |
+
| Multi-valued | +Variable | +Like the "B" attribute, example {`array. A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the *Multivalue Mapping* the arrays are stored. The unique strings are stored in the *Enum Store*, and the arrays in the *Multivalue Mapping* stores the references (32 bit) to the strings in the *Enum Store*. The *Enum Store* consists of multiple memory buffers. |
+
| Single-valued | +Variable | +Like the "C" attribute, example {`string`}. A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the *Enum Store* the strings are stored. |
+
| Tensor | +Fixed / Variable | +Like the "D" attribute, example {`tensor. A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the *Tensor Store* the tensor values are stored. The memory layout in the *Tensor Store* depends on the tensor type. |
+
| Abbreviation | +Concept | +Comment | +
|---|---|---|
| D | +Number of documents | +Number of documents on the node, or rather the maximum number of local document IDs allocated | +
| V | +Average number of values per document | +Only applicable for arrays and weighted sets | +
| U | +Number of unique values | +Only applies for strings or if fast-search is set | +
| FW | +Fixed data width | +sizeof(T) for numerics, 1 byte for strings, 1 bit for boolean | +
| WW | +Weight width | +Width of the weight in a weighted set, 4 bytes. 0 bytes for arrays. | +
| EIW | +Enum index width | +Width of the index into the enum store, 4 bytes. Used by all strings and other attributes if fast-search is set | +
| VW | +Variable data width | +strlen(s) for strings, 0 bytes for the rest | +
| PW | +Posting entry width | +Width of a posting list entry, 4 bytes for singlevalue, 8 bytes for array and weighted sets. Only applies if fast-search is set. | +
| PIW | +Posting index width | +Width of the index into the store of posting lists; 4 bytes | +
| MIW | +Multivalue index width | +Width of the index into the multivalue mapping; 4 bytes | +
| ROF | +Resize overhead factor | +Default is 6/5. This is the average overhead in any dynamic vector due to resizing strategy. Resize strategy is 50% indicating that structure is 5/6 full on average. | +
| Component | +Formula | +Approx Factor | +Applies to | +
|---|---|---|---|
| Document vector | +D * ((FW or EIW) or MIW) | +ROF | +FW for singlevalue numeric attributes and MIW for multivalue attributes. EIW for singlevalue string or if the attribute is singlevalue fast-search | +
| Multivalue mapping | +D * V * ((FW or EIW) + WW) | +ROF | +Applicable only for array or weighted sets. EIW if string or fast-search | +
| Enum store | +U * ((FW + VW) + 4 + ((EIW + PIW) or EIW)) | +ROF | +Applicable for strings or if fast-search is set. (EIW + PIW) if fast-search is set, EIW otherwise. | +
| Posting list | +D * V * PW | +ROF | +Applicable if fast-search is set | +
| Type | +Components | +Formula | +
|---|---|---|
| Numeric singlevalue plain | +Document vector | +D * FW * ROF | +
| Numeric multivalue value plain | +Document vector, Multivalue mapping | +D * MIW * ROF + D * V * (FW+WW) * ROF | +
| Numeric singlevalue fast-search | +Document vector, Enum store, Posting List | +D * EIW * ROF + U * (FW+4+EIW+PIW) * ROF + D * PW * ROF | +
| Numeric multivalue value fast-search | +Document vector, Multivalue mapping, Enum store, Posting List | +D * MIW * ROF + D * V * (EIW+WW) * ROF + U * (FW+4+EIW+PIW) * ROF + D * V * PW * ROF | +
| Singlevalue string plain | +Document vector, Enum store | +D * EIW * ROF + U * (FW+VW+4+EIW) * ROF | +
| Singlevalue string fast-search | +Document vector, Enum store, Posting List | +D * EIW * ROF + U * (FW+VW+4+EIW+PIW) * ROF + D * PW * ROF | +
| Multivalue string plain | +Document vector, Multivalue mapping, Enum store | +D * MIW * ROF + D * V * (EIW+WW) * ROF + U * (FW+VW+4+EIW) * ROF | +
| Multivalue string fast-search | +Document vector, Multivalue mapping, Enum store, Posting list | +D * MIW * ROF + D * V * (EIW+WW) * ROF + U * (FW+VW+4+EIW+PIW) * ROF + D * V * PW * ROF | +
| Boolean singlevalue | +Document vector | +D * FW * ROF | +
| + | + |
|---|---|
| **Split bucket** | +Split a bucket in two, by enforcing the documents within the new buckets to have more location bits in common. Buckets are split either because they have grown too big, or because the cluster wants to use more distribution bits. | +
| **Join bucket** | +Join two buckets into one. If a bucket has been previously split due to being large, but documents have now been deleted, the bucket can be joined again. | +
| **Merge bucket** | +If there are multiple replicas of a bucket, but they do not store the same set of versioned documents, _merge_ is used to synchronize the replicas. A special case of a merge is a one-way merge, which may be done if some of the replicas are to be deleted right after the merge. Merging is used not only to fix inconsistent bucket replicas, but also to move buckets between nodes. To move a bucket, an empty replica is created on the target node, a merge is executed, and the source bucket is deleted. | +
| **Create bucket** | +This operation exist merely for the distributor to notify a content node that it is now to store documents for this bucket too. This allows content nodes to refuse operations towards buckets it does not own. The ability to refuse traffic is a safeguard to avoid inconsistencies. If a client talks to a distributor that is no longer working correctly, we rather want its requests to fail than to alter the content cluster in strange ways. | +
| **Delete bucket** | +Drop stored state for a bucket and reject further requests for it | +
| **(De)activate bucket** | +Activate bucket for search results - refer to bucket management | +
| **Garbage collections** | +If configured, documents are periodically garbage collected through background maintenance operations. | +
| Task | +Description | +
|---|---|
| Node state polling | +The cluster controller polls nodes, sending the current cluster state. If the cluster state is no longer correct, the node returns correct information immediately. If the state is correct, the request lingers on the node, such that the node can reply to it immediately if its state changes. After a while, the cluster controller will send a new state request to the node, even with one pending. This triggers a reply to the lingering request and makes the new one linger instead. Hence, nodes have a pending state request. During a controlled node shutdown, it starts the shutdown process by responding to the pending state request that it is now stopping. |
+
| Cluster state generation | +The cluster controller translates unit and user states into the generated cluster state | +
| Cluster state broadcast | +When node unit states are received, a cluster controller internal cluster state is updated. New cluster states are distributed with a minimum interval between. A grace period per unit state too - e.g., distributors and content nodes that are on the same node often stop at the same time. The version number is incremented, and the new cluster state is broadcast. If cluster state version is reset, distributors and content node processes may have to be restarted in order for the system to converge to the new state. Nodes will reject lower cluster state versions to prevent race conditions caused by overlapping cluster controller leadership periods. |
+
| Metric | +Description | +
|---|---|
| .idealstate.idealstate_diff | +This metric tries to create a single value indicating distance to the ideal state. A value of zero indicates that the cluster is in the ideal state. Graphed values of this metric gives a good indication for how fast the cluster gets back to the ideal state after changes. Note that some issues may hide other issues, so sometimes the graph may appear to stand still or even go a bit up again, as resolving one issue may have detected one or several others. | +
| .idealstate.buckets_toofewcopies | +Specifically lists how many buckets have too few copies. Compare to the *buckets* metric to see how big a portion of the cluster this is. | +
| .idealstate.buckets_toomanycopies | +Specifically lists how many buckets have too many copies. Compare to the *buckets* metric to see how big a portion of the cluster this is. | +
| .idealstate.buckets | +The total number of buckets managed. Used by other metrics reporting bucket counts to know how big a part of the cluster they relate to. | +
| .idealstate.buckets_notrusted | +Lists how many buckets have no trusted copies. Without trusted buckets operations against the bucket may have poor performance, having to send requests to many copies to try and create consistent replies. | +
| .idealstate.delete_bucket.pending | +Lists how many buckets that needs to be deleted. | +
| .idealstate.merge_bucket.pending | +Lists how many buckets there are, where we suspect not all copies store identical document sets. | +
| .idealstate.split_bucket.pending | +Lists how many buckets are currently being split. | +
| .idealstate.join_bucket.pending | +Lists how many buckets are currently being joined. | +
| .idealstate.set_bucket_state.pending | +Lists how many buckets are currently altered for active state. These are high priority requests which should finish fast, so these requests should seldom be seen as pending. | +
| + | + |
|---|---|
| **Cluster upgrade** | +With multiple groups it becomes safe to take out a full group for upgrade instead of just one node at a time. Read more. | +
| **Query throughput** | +Applications with high query rates and/or high static query cost can use groups to scale to higher query rates since Vespa will automatically send a query to just a single group. Read more | +
| **Topology** | +By using groups you can control replica placement over network switches or racks to ensure there is redundancy at the switch and rack level. | +
| + | + |
|---|---|
| **Minimal reassignment on cluster state change** | +- If a node goes down, only buckets that resided on that node should be reassigned. - If a node comes up, only buckets that are moved to the new node should relocate. - Increasing the capacity of a single node should only move buckets to that node. - Reducing the capacity of a single node should only move buckets away from that node. |
+
| **No skew in distribution** | +- Nodes should get an amount of data relative to their capacity. | +
| **Lightweight** | +- A simple algorithm that is easy to understand is a plus. Being lightweight to calculate is also a plus, giving more options of how to use it, without needing to cache results. | +
| Bits \ Nodes | +1 | +2 | +3 | +4 | +5 | +6 | +7 | +8 | +9 | +10 | +11 | +12 | +13 | +14 | +15 | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | +0.0000 | +0.0000 | +0.3333 | +0.5000 | +0.6000 | +0.6667 | +0.7143 | +0.7500 | +0.7778 | +0.8000 | +0.8182 | +0.8333 | +0.8462 | +0.8571 | +0.8667 | +
| 2 | +0.0000 | +0.3333 | +0.3333 | +0.5000 | +0.2000 | +0.3333 | +0.4286 | +0.5000 | +0.5556 | +0.6000 | +0.6364 | +0.6667 | +0.6923 | +0.7143 | +0.7333 | +
| 3 | +0.0000 | +0.2000 | +0.1111 | +0.3333 | +0.2000 | +0.3333 | +0.6190 | +0.6667 | +0.8222 | +0.8400 | +0.8545 | +0.8333 | +0.6923 | +0.7143 | +0.7333 | +
| 4 | +0.0000 | +0.1111 | +0.1111 | +0.3333 | +0.3600 | +0.3333 | +0.4286 | +0.5000 | +0.7778 | +0.8000 | +0.8182 | +0.8095 | +0.6923 | +0.7143 | +0.6444 | +
| 5 | +- | +0.0588 | +0.1111 | +0.2727 | +0.2889 | +0.4074 | +0.2381 | +0.3333 | +0.8129 | +0.8316 | +0.8469 | +0.8519 | +0.8359 | +0.8367 | +0.8359 | +
| 6 | +- | +0.0000 | +0.0725 | +0.1579 | +0.1467 | +0.1111 | +0.1688 | +0.3846 | +0.7037 | +0.7217 | +0.7470 | +0.7460 | +0.7265 | +0.6952 | +0.6718 | +
| 7 | +- | +0.0725 | +0.0519 | +0.0857 | +0.0857 | +0.1111 | +0.2050 | +0.2000 | +0.4530 | +0.4667 | +0.5152 | +0.5152 | +0.4530 | +0.3905 | +0.3436 | +
| 8 | +- | +0.0000 | +0.0078 | +0.0725 | +0.0857 | +0.0922 | +0.1293 | +0.1351 | +0.1634 | +0.1742 | +0.1688 | +0.2381 | +0.2426 | +0.2967 | +0.3173 | +
| 9 | +- | +0.0039 | +0.0192 | +0.1467 | +0.1607 | +0.1203 | +0.1080 | +0.1111 | +0.1380 | +0.1322 | +0.1218 | +0.1795 | +0.1962 | +0.2381 | +0.2580 | +
| 10 | +- | +0.0019 | +0.0275 | +0.0922 | +0.0898 | +0.0623 | +0.0741 | +0.0922 | +0.1111 | +0.1018 | +0.1218 | +0.1203 | +0.1438 | +0.1688 | +0.1675 | +
| 11 | +- | +0.0019 | +0.0234 | +0.0430 | +0.0385 | +0.0248 | +0.0248 | +0.0483 | +0.0636 | +0.0648 | +0.0737 | +0.0725 | +0.0894 | +0.0800 | +0.0958 | +
| 12 | +- | +- | +0.0121 | +0.0285 | +0.0282 | +0.0121 | +0.0149 | +0.0571 | +0.0577 | +0.0562 | +0.0549 | +0.0412 | +0.0510 | +0.0439 | +0.0616 | +
| 13 | +- | +- | +0.0074 | +0.0019 | +0.0070 | +0.0177 | +0.0304 | +0.0303 | +0.0337 | +0.0189 | +0.0252 | +0.0358 | +0.0409 | +0.0501 | +0.0385 | +
| 14 | +- | +- | +0.0041 | +0.0024 | +0.0037 | +0.0027 | +0.0145 | +0.0073 | +0.0101 | +0.0130 | +0.0220 | +0.0234 | +0.0290 | +0.0248 | +0.0195 | +
| 15 | +- | +- | +0.0019 | +0.0021 | +0.0036 | +0.0083 | +0.0059 | +0.0056 | +0.0101 | +0.0097 | +0.0123 | +0.0163 | +0.0150 | +0.0186 | +0.0173 | +
| 16 | +- | +- | +0.0010 | +0.0007 | +0.0010 | +0.0030 | +0.0049 | +0.0039 | +0.0085 | +0.0072 | +0.0097 | +0.0108 | +0.0135 | +0.0141 | +0.0115 | +
| 17 | +- | +- | +- | +- | +- | +0.0030 | +0.0033 | +0.0024 | +0.0036 | +0.0030 | +0.0055 | +0.0091 | +0.0135 | +0.0156 | +0.0143 | +
| 18 | +- | +- | +- | +- | +- | +- | +0.0019 | +- | +0.0029 | +0.0027 | +0.0043 | +0.0040 | +0.0066 | +0.0061 | +0.0060 | +
| 19 | +- | +- | +- | +- | +- | +- | +- | +- | +0.0019 | +- | +0.0021 | +0.0030 | +0.0023 | +0.0031 | +0.0042 | +
| 20 | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +0.0029 | +0.0025 | +0.0037 | +0.0044 | +
| 21 | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +0.0026 | +0.0035 | +0.0040 | +
| Bits \ Nodes | +1 | +2 | +3 | +4 | +5 | +6 | +7 | +8 | +9 | +10 | +11 | +12 | +13 | +14 | +15 | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | +0.0000 | +0.0000 | +0.3333 | +0.5000 | +0.6000 | +0.6667 | +0.4286 | +0.5000 | +0.5556 | +0.6000 | +0.6364 | +0.6667 | +0.6923 | +0.7143 | +0.7333 | +
| 2 | +0.0000 | +0.0000 | +0.3333 | +0.3333 | +0.2000 | +0.3333 | +0.4286 | +0.5000 | +0.5556 | +0.6000 | +0.6364 | +0.6667 | +0.6923 | +0.4286 | +0.4667 | +
| 3 | +0.0000 | +0.0000 | +0.1111 | +0.2000 | +0.2000 | +0.3333 | +0.4286 | +0.5000 | +0.7037 | +0.7333 | +0.7576 | +0.7778 | +0.7949 | +0.7714 | +0.7333 | +
| 4 | +0.0000 | +0.0000 | +0.1111 | +0.2000 | +0.2000 | +0.3333 | +0.3469 | +0.2000 | +0.7460 | +0.7714 | +0.7762 | +0.7778 | +0.7949 | +0.7714 | +0.7630 | +
| 5 | +- | +- | +0.0725 | +0.1579 | +0.2471 | +0.2381 | +0.2967 | +0.2727 | +0.7265 | +0.7538 | +0.7673 | +0.7778 | +0.7949 | +0.7922 | +0.7968 | +
| 6 | +- | +- | +0.0519 | +0.1111 | +0.1742 | +0.1467 | +0.2050 | +0.2381 | +0.6908 | +0.7023 | +0.7016 | +0.7117 | +0.7265 | +0.7229 | +0.7247 | +
| 7 | +- | +- | +0.0303 | +0.0154 | +0.0340 | +0.0303 | +0.0857 | +0.1111 | +0.4921 | +0.4880 | +0.4828 | +0.4797 | +0.5077 | +0.4622 | +0.4667 | +
| 8 | +- | +- | +0.0078 | +0.0303 | +0.0248 | +0.0623 | +0.0857 | +0.0725 | +0.0970 | +0.1322 | +0.1049 | +0.1293 | +0.1620 | +0.1873 | +0.2242 | +
| 9 | +- | +- | +0.0019 | +0.0266 | +0.0519 | +0.0466 | +0.0682 | +0.0791 | +0.0824 | +0.0519 | +0.0691 | +0.0519 | +0.0623 | +0.0741 | +0.0898 | +
| 10 | +- | +- | +0.0063 | +0.0173 | +0.0154 | +0.0275 | +0.0116 | +0.0340 | +0.0558 | +0.0294 | +0.0452 | +0.0466 | +0.0567 | +0.0501 | +0.0584 | +
| 11 | +- | +- | +0.0078 | +0.0049 | +0.0154 | +0.0177 | +0.0149 | +0.0210 | +0.0275 | +0.0177 | +0.0252 | +0.0303 | +0.0305 | +0.0344 | +0.0317 | +
| 12 | +- | +- | +- | +0.0073 | +0.0112 | +0.0192 | +0.0231 | +0.0312 | +0.0296 | +0.0177 | +0.0278 | +0.0358 | +0.0245 | +0.0312 | +0.0385 | +
| 13 | +- | +- | +- | +0.0061 | +0.0049 | +0.0096 | +0.0112 | +0.0201 | +0.0218 | +0.0088 | +0.0077 | +0.0199 | +0.0138 | +0.0304 | +0.0317 | +
| 14 | +- | +- | +- | +0.0059 | +0.0058 | +0.0058 | +0.0057 | +0.0092 | +0.0128 | +0.0082 | +0.0139 | +0.0081 | +0.0096 | +0.0199 | +0.0213 | +
| 15 | +- | +- | +- | +- | +0.0014 | +0.0039 | +0.0052 | +0.0034 | +0.0051 | +0.0085 | +0.0044 | +0.0072 | +0.0107 | +0.0101 | +0.0082 | +
| 16 | +- | +- | +- | +- | +0.0016 | +0.0030 | +0.0026 | +0.0036 | +0.0065 | +0.0051 | +0.0061 | +0.0084 | +0.0065 | +0.0083 | +0.0100 | +
| 17 | +- | +- | +- | +- | +- | +- | +0.0010 | +0.0020 | +0.0028 | +- | +0.0040 | +0.0049 | +0.0067 | +0.0071 | +0.0062 | +
| 18 | +- | +- | +- | +- | +- | +- | +- | +- | +0.0032 | +- | +0.0024 | +- | +0.0034 | +0.0056 | +0.0041 | +
| 19 | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +0.0025 | +0.0018 | +- | +
| Bits \ Nodes | +16 | +20 | +32 | +48 | +64 | +100 | +128 | +160 | +200 | +256 | +350 | +500 | +800 | +1000 | +5000 | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 8 | +0.2000 | +0.3081 | +0.2727 | +0.5152 | +0.5294 | +0.5733 | +0.6364 | +0.7091 | +0.7673 | +0.8000 | +0.8537 | +0.8862 | +0.8933 | +0.8976 | +0.9659 | +
| 9 | +0.0725 | +0.2242 | +0.1795 | +0.1795 | +0.3043 | +0.3173 | +0.3846 | +0.5077 | +0.5345 | +0.6364 | +0.7340 | +0.7952 | +0.8400 | +0.8720 | +0.9317 | +
| 10 | +0.0725 | +0.1322 | +0.1233 | +0.2099 | +0.1579 | +0.2415 | +0.3333 | +0.5733 | +0.4611 | +0.5789 | +0.6558 | +0.7269 | +0.8293 | +0.8425 | +0.8976 | +
| 11 | +0.0340 | +0.0857 | +0.0922 | +0.1111 | +0.1233 | +0.1969 | +0.2558 | +0.5937 | +0.5643 | +0.5897 | +0.5965 | +0.6099 | +0.6587 | +0.7591 | +0.8830 | +
| 12 | +0.0448 | +0.0385 | +0.0623 | +0.1065 | +0.0986 | +0.1285 | +0.3725 | +0.3831 | +0.4064 | +0.4074 | +0.4799 | +0.4880 | +0.5124 | +0.8328 | +0.8976 | +
| 13 | +0.0340 | +0.0328 | +0.0554 | +0.0699 | +0.0623 | +0.0948 | +0.1049 | +0.2183 | +0.2344 | +0.3191 | +0.3498 | +0.4539 | +0.5733 | +0.6656 | +0.8870 | +
| 14 | +0.0140 | +0.0189 | +0.0376 | +0.0452 | +0.0466 | +0.0717 | +0.0986 | +0.1057 | +0.1047 | +0.2242 | +0.2853 | +0.2798 | +0.4064 | +0.4959 | +0.8830 | +
| 15 | +0.0094 | +0.0118 | +0.0385 | +0.0268 | +0.0331 | +0.0638 | +0.0708 | +0.0775 | +0.0898 | +0.1322 | +0.2133 | +0.2104 | +0.3550 | +0.4446 | +0.8752 | +
| 16 | +0.0097 | +0.0081 | +0.0380 | +0.0303 | +0.0362 | +0.0577 | +0.0501 | +0.0627 | +0.0717 | +0.1033 | +0.1733 | +0.1678 | +0.2586 | +0.3101 | +0.8511 | +
| 17 | +0.0075 | +0.0066 | +0.0346 | +0.0293 | +0.0154 | +0.0258 | +0.0466 | +0.0546 | +0.0704 | +0.1041 | +0.1469 | +0.1983 | +0.2702 | +0.2972 | +0.7740 | +
| 18 | +0.0053 | +0.0057 | +0.0098 | +0.0098 | +0.0122 | +0.0149 | +0.0238 | +0.0300 | +0.0394 | +0.0353 | +0.0434 | +0.0553 | +0.0611 | +0.1782 | +0.6334 | +
| 19 | +- | +0.0022 | +0.0050 | +0.0162 | +0.0098 | +0.0133 | +0.0149 | +0.0220 | +0.0242 | +0.0252 | +0.0333 | +0.0398 | +0.0495 | +0.0999 | +0.5145 | +
| 20 | +- | +- | +0.0030 | +0.0107 | +0.0088 | +0.0098 | +0.0144 | +0.0140 | +0.0148 | +0.0203 | +0.0195 | +0.0255 | +0.0348 | +0.1133 | +0.4481 | +
| 21 | +- | +- | +0.0043 | +0.0063 | +0.0051 | +0.0074 | +0.0079 | +0.0085 | +0.0086 | +0.0113 | +0.0147 | +0.0170 | +0.0237 | +0.1068 | +0.4422 | +
| 22 | +- | +- | +- | +0.0026 | +0.0035 | +0.0037 | +0.0082 | +0.0061 | +0.0077 | +0.0087 | +0.0101 | +0.0134 | +0.0193 | +0.1140 | +0.4635 | +
| 23 | +- | +- | +- | +0.0019 | +- | +0.0026 | +0.0080 | +0.0055 | +0.0056 | +0.0057 | +0.0063 | +0.0096 | +0.0155 | +0.1294 | +0.4982 | +
| 24 | +- | +- | +- | +0.0013 | +- | +- | +0.0074 | +0.0060 | +0.0058 | +0.0053 | +0.0049 | +0.0068 | +0.0112 | +0.0471 | +0.3219 | +
| 25 | +- | +- | +- | +- | +- | +- | +- | +- | +- | +0.0043 | +0.0043 | +0.0058 | +0.0067 | +0.0512 | +0.2543 | +
| 26 | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +0.0040 | +0.0042 | +0.0043 | +0.0051 | +0.0210 | +
| 27 | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +- | +0.0028 | +0.0157 | +0.0814 | +
| Node count | +1-4 | +5-199 | +200-> | +
|---|---|---|---|
| Distribution bit count | +8 | +16 | +24 | +
| Max calculated waste *) | +3.03 % | +7.17 % | +? | +
| Minimum buckets/node **) | +256 - 64 | +13108 - 329 | +83886 - | +
| Node count | +1-4 | +5-14 | +15-199 | +200-799 | +800-1499 | +1500-4999 | +5000-> | +
|---|---|---|---|---|---|---|---|
| Distribution bit count | +8 | +16 | +21 | +25 | +28 | +30 | +32 | +
| Max calculated waste *) | +3 % | +0.83 % | +0.86 % | +0.67 % | +? | +? | +? | +
| Minimum buckets/node **) | +256 - 64 | +13107 - 4681 | +139810 - 10538 | +167772 - 41995 | +335544 - 179076 | +715827 - 214791 | +858993 - | +
| + | + |
|---|---|
| **Not Ready** | +Holds the redundant documents that are not searchable, i.e. the not ready documents. Documents that are not ready are only stored, not indexed. It takes some processing to move from this state to the ready state. | +
| **Ready** | +Maintains attributes and indexes of all ready documents and keeps them searchable. One of the ready copies is active while the rest are not active: **Active** There should always be exactly one active copy of each document in the system, though intermittently there may be more. These documents produce results when queries are evaluated. >**Not Active** The ready copies that are not active are indexed but will not produce results. By being indexed, they are ready to take over immediately if the node holding the active copy becomes unavailable. Read more in searchable-copies. |
+
| **Removed** | +Keeps track of documents that have been removed. The id and timestamp for each document are kept. This information is used when buckets from two nodes are merged. If the removed document exists on another node but with a different timestamp, the most recent entry prevails. | +
| Job | +Description | +
|---|---|
| CPU | +Little - one thread flushes to disk | +
| Memory | +Little - some temporary use | +
| Disk | +A new file is written too, so 2x the size of an attribute on disk until the old flush file is deleted. | +
| Run metric | +content.proton.documentdb.job.attribute_flush | +
| content.proton.documentdb.[ready|notready].attribute.memory_usage. | +|
| Metrics | +allocated_bytes.average used_bytes.average dead_bytes.average onhold_bytes.average |
+
| CPU | +Little - one thread flushes to disk | +
| Memory | +Little | +
| Disk | +Creates a new disk index, size of the memory index. | +
| Run metric | +content.proton.documentdb.job.memory_index_flush | +
| Metric prefix | +content.proton.documentdb.index.memory_usage. | +
| Metrics | +allocated_bytes.average used_bytes.average dead_bytes.average onhold_bytes.average |
+
| CPU | +Multiple threads merge indices, configured as a function of feeding concurrency - refer to this for details |
+
| Memory | +Little | +
| Disk | +Creates a new index while serving from the current: 2x temporary disk usage for the given index. | +
| Run metric | +content.proton.documentdb.job.disk_index_fusion | +
| CPU | +Little | +
| Memory | +Little | +
| Disk | +Little | +
| Run metric | +content.proton.documentdb.job.document_store_flush | +
| CPU | +Little - one thread reads one file, sorts and writes a new file | +
| Memory | +Holds a document store file in memory plus memory for sorting the file. Note: This is important on hosts with little memory! Reduce maxfilesize to increase the number of files and use less temporary memory for compaction. |
+
| Disk | +A new file is written while the current is serving, max temporary usage is 2x. | +
| Run metric | +content.proton.documentdb.job.document_store_compact | +
| Metric prefix | +content.proton.documentdb.[ready|notready|removed].document_store. | +
| Metrics | +disk_usage.average disk_bloat.average max_bucket_spread.average memory_usage.allocated_bytes.average memory_usage.used_bytes.average memory_usage.dead_bytes.average memory_usage.onhold_bytes.average |
+
| CPU | +CPU similar to feeding. Consumes capacity from the write threads, so has feeding impact |
+
| Memory | +As feeding - e.g., the attribute memory usage and memory index in the ready sub-database will grow | +
| Disk | +As feeding | +
| Run metric | +content.proton.documentdb.job.bucket_move | +
| CPU | +Like feeding - add and remove documents | +
| Memory | +Little | +
| Disk | +0 | +
| Run metric | +content.proton.documentdb.job.lid_space_compact | +
| Metric prefix | +content.proton.documentdb.[ready|notready|removed].lid_space. | +
| Metrics | +lid_limit.last lid_bloat_factor.average lid_fragmentation_factor.average |
+
| CPU | +Little | +
| Memory | +Little | +
| Disk | +Little | +
| Run metric | +content.proton.documentdb.job.removed_documents_prune | +
| + | + |
|---|---|
| **Get** | +When the content node receives a get request, it scans through all the document databases, and for each one, it checks all three sub-databases. Once the document is found, the scan is stopped and the document returned. If the document is found in a Ready sub-database, the document retriever will apply any changes that are stored in the attributes before returning the document. | +
| **Visit** | +A visit request creates an iterator over each candidate bucket. This iterator will retrieve matching documents from all sub-databases of all document databases. As for get, attribute values are applied to document fields in the Ready sub-database. | +
| + | + |
|---|---|
| **Container** | +The Container knows all the document types and rewrites queries as a collection of queries, one for each type. Queries may have a restrict parameter, in which case the container will send the query only to the specified document types. It sends the query to content nodes and collects partial results. It pings all content nodes every second to know whether they are alive, and keeps open TCP connections to each one. If a node goes down, the elastic system will make the documents available on other nodes. | +
| **Content node matching** | +The *match engine* receives queries and routes them to the right document database based on the document type. The query is passed to the *Ready* sub-database, where the searchable documents are. Based on information stored in the document meta store, the query is augmented with a blocklist that ensures only *active* documents are matched. | +
| + | + |
|---|---|
| Only applicable to Vespa Cloud deployments. | +|
| Only applicable to self-managed deployments. | +|
| Not open source: Available commercially only (both self-managed and on cloud unless also marked by one of the other chips above). | +
| Metric | +Value | +
|---|---|
| Match Recall | +1.0000 | +
| Average Recall per Query | +1.0000 | +
| Total Relevant Documents | +51 | +
| Total Matched Relevant | +51 | +
| Average Matched per Query | +100.0000 | +
| Total Queries | +20 | +
| Search Time Average (s) | +0.0090 | +
| Search Time Q50 (s) | +0.0060 | +
| Search Time Q90 (s) | +0.0193 | +
| Search Time Q95 (s) | +0.0220 | +
| Metric | +Value | +
|---|---|
| Match Recall | +1.0000 | +
| Average Recall per Query | +1.0000 | +
| Total Relevant Documents | +51 | +
| Total Matched Relevant | +51 | +
| Average Matched per Query | +88.7000 | +
| Total Queries | +20 | +
| Search Time Average (s) | +0.0071 | +
| Search Time Q50 (s) | +0.0060 | +
| Search Time Q90 (s) | +0.0132 | +
| Search Time Q95 (s) | +0.0171 | +
| Metric | +Value | +
|---|---|
| Match Recall | +1.0000 | +
| Average Recall per Query | +1.0000 | +
| Total Relevant Documents | +51 | +
| Total Matched Relevant | +51 | +
| Average Matched per Query | +100.0000 | +
| Total Queries | +20 | +
| Search Time Average (s) | +0.0076 | +
| Search Time Q50 (s) | +0.0055 | +
| Search Time Q90 (s) | +0.0150 | +
| Search Time Q95 (s) | +0.0201 | +
| query_id | +doc_id | +relevance_label | +relevance_score | +match_avg_top_3_chunk_sim_scores | +match_avg_top_3_chunk_text_scores | +match_bm25(chunks) | +match_bm25(title) | +match_max_chunk_sim_scores | +match_max_chunk_text_scores | +
|---|---|---|---|---|---|---|---|---|---|
| alex_q_01 | +50 | +1 | +0.660597 | +0.248329 | +8.444725 | +7.717984 | +0. | +0.268457 | +8.444725 | +
| alex_q_01 | +82 | +1 | +0.649638 | +0.225300 | +12.327676 | +18.611592 | +2.453409 | +0.258905 | +15.644889 | +
| alex_q_01 | +1 | +1 | +0.245849 | +0.358027 | +15.100841 | +23.010389 | +4.333828 | +0.391143 | +20.582403 | +
| alex_q_01 | +28 | +0 | +0.988250 | +0.278074 | +0.179929 | +0.197420 | +0. | +0.278074 | +0.179929 | +
| alex_q_01 | +23 | +0 | +0.968268 | +0.203709 | +0.182603 | +0.196956 | +0. | +0.203709 | +0.182603 | +
| Feature | +Importance | +
|---|---|
| nativeProximity | +168.8498 | +
| firstPhase | +151.7382 | +
| max_chunk_sim_scores | +69.4377 | +
| avg_top_3_chunk_text_scores | +56.5079 | +
| avg_top_3_chunk_sim_scores | +31.8700 | +
| nativeRank | +20.0716 | +
| nativeFieldMatch | +15.9914 | +
| elementSimilarity(chunks) | +9.7003 | +
| bm25(body) | +bm25(title) | +nativeRank(body) | +nativeRank(title) | +docid | +qid | +relevant | +
|---|---|---|---|---|---|---|
| 25.792076 | +12.117309 | +0.322567 | +0.084239 | +D312959 | +3 | +1 | +
| 22.191228 | +0.043899 | +0.247145 | +0.017715 | +D3162299 | +3 | +0 | +
| 13.880625 | +0.098052 | +0.219413 | +0.036826 | +D2823827 | +3 | +0 | +
| Transformation | +Result | +
|---|---|
| Input | +hôtel | +
| Correctly URL quoted (Vespa always uses UTF-8 there) | +h%C3%B4tel | +
| Encoded as ISO-8859-1 (ISO Latin-1), then URL quoted | +h%F4tel | +
| Encoded as UTF-16 (as in Java strings), then URL quoted | +%00h%00%F4%00t%00e%00l | +
| For completeness, little endian UTF-16, including byte order marker | +%FF%FEh%00%F4%00t%00e%00l%00 | +
| Error | +Hex dump of code points | +Rendered | +
|---|---|---|
| UTF-8 input decoded as if it were ISO-8859-1 | +h\xc3\xb4tel | +hôtel | +
| UTF-8 input re-encoded as UTF-8, then decoded as UTF-8 again | +h\xc3\xb4tel | +hôtel | +
| Parameter | +Description | +Example | +
|---|---|---|
{`ecommerce.multicurrency.market`} |
+ Target market code | +{`NO`}, {`US`}, {`EU`}, {`NO-49`}, {`27`} |
+
{`ecommerce.multicurrency.currency`} |
+ Target currency code | +{`NOK`}, {`USD`}, {`EUR`} |
+
{`ecommerce.multicurrency.price-min`} |
+ Minimum price in target currency | +{`1000`} |
+
{`ecommerce.multicurrency.price-max`} |
+ Maximum price in target currency | +{`1500`} |
+
{`ecommerce.multicurrency.enrich`} |
+ Optional: expose forex rates as query tensor for ranking. Defaults to false | +{`true`} or {`false`} |
+
| State | +Description | +Query Behavior | +
|---|---|---|
{`READY`} |
+ Forex rates loaded and service is operational | +Queries with multi-currency filtering work normally | +
{`UNINITIALIZED`} |
+ No forex document has been loaded yet | +Queries return error: "forex rate service not initialized" | +
{`OUTAGE`} |
+ Refresh failed but stale data exists (cache stays ready for re-use once the retriever succeeds again) | +Queries return error: "forex rate service temporarily unavailable (last refresh failed)" | +
{`INVALID_FOREX_DOCUMENTS`} |
+ Multiple forex documents detected (expected exactly one) | +Queries return error: "ensure exactly one forex document exists" | +
| Type | +Prefix | +Format | +Purpose | +
|---|---|---|---|
| Filter parameters | +{`ecommerce.multicurrency.*`} |
+ Plain string or number | +Tells the searcher which market, currency, and price range to filter on. Consumed server-side ” never reach the rank profile. | +
| Ranking inputs | +{`ranking.features.query(...)`} |
+ One-hot tensor | +Passed directly to the rank profile to drive scoring expressions. The searcher does not read or modify these. | +
| Parameter | +Description | +Type | +Default | +
|---|---|---|---|
{`productFields.sellerCurrency`} |
+ Field name for the product's seller currency. | +{`string`} |
+ {`seller_currency`} |
+
{`productFields.perMarketPriceArrayStruct`} |
+ Array field name containing per-market prices. | +{`string`} |
+ {`per_market_price`} |
+
{`productFields.marketStructField`} |
+ Struct field name for market code. | +{`string`} |
+ {`market`} |
+
{`productFields.priceStructField`} |
+ Struct field name for price value. | +{`string`} |
+ {`price`} |
+
{`defaults.market`} |
+ Default market identifier used as fallback when no market-specific price exists. | +{`string`} |
+ {`DEFAULT`} |
+
{`rankProfileInputs.forexRates`} |
+ Query tensor name for forex rates in ranking. Used when {`enrich=true`} to inject the forex tensor into the query. |
+ {`string`} |
+ {`forexRates`} |
+
| Parameter | +Description | +Type | +Default value | +
|---|---|---|---|
{`notification.kind`} |
+ Method to use for sending notifications. | +{`enum {WEBHOOK, DUMMY, VESPA_SCHEMA}`} |
+ {`DUMMY`} |
+
{`notification.webhook.URL`} |
+ URL to send notification requests. | +{`string`} |
+ + |
{`notification.webhook.connectionPoolSize`} |
+ Number of HTTP client threads to use in the container cluster. | +{`int`} |
+ {`20`} |
+
{`notification.webhook.headers[].key`} |
+ Key of a header to add to all webhook requests. | +{`string`} |
+ + |
{`notification.webhook.headers[].value`} |
+ Value of a header to add to all webhook requests. | +{`string`} |
+ + |
{`notification.webhook.headers[].secret`} |
+ Use a secret from Vespa secret store instead of the value provided in {`.value`}. The value provided here should match the name of a secret specified with a {`secrets`} tag in {`services.xml`}. |
+ {`string`} |
+ + |
{`notification.vespaSchema.documentType`} |
+ Name of the Vespa document type to use for storing notifications. This document type has to be defined in the application. | +{`string`} |
+ saved_search_notification | +
{`notification.vespaSchema.namespace`} |
+ Namespace to use for creating document IDs for the notification documents. | +{`string`} |
+ saved_search | +
{`notification.vespaSchema.fieldPathProductId`} |
+ Fieldpath for storing the product id in the notification documents. | +{`string`} |
+ product_id | +
{`notification.vespaSchema.fieldPathSavedSearchId`} |
+ Fieldpath for storing saved search id in the notification documents. | +{`string`} |
+ saved_search_id | +
{`notification.vespaSchema.fieldPathTimestamp`} |
+ Fieldpath for storing timestamps in the notification documents. | +{`string`} |
+ timestamp | +
{`productDocumentType`} |
+ The name of the document type that can trigger notifications, e.g. {`product`}. |
+ {`string`} |
+ product | +
{`savedSearchDocumentType`} |
+ The name of the document type storing saved searches, e.g. {`saved_search`}. |
+ {`string`} |
+ saved_search | +
{`predicateFieldName`} |
+ The name of the field in {`savedSearchDocumentType`} storing the predicate query. |
+ {`string`} |
+ filters | +
{`savedSearchNumHits`} |
+ Maximum number of saved searches that can match per product update. Matches beyond this limit are silently dropped. Higher values increase work per update. | +{`int`} |
+ 100 | +
{`regularAttributes[].predicateName`} |
+ The name of a regular (string) attribute to be used in the saved search predicate field. | +{`string`} |
+ + |
{`regularAttributes[].fieldPath`} |
+ The field in the {`productDocumentType`} to be matched with this attribute. This field should be of type {`string`}. |
+ {`string`} |
+ + |
{`regularAttributes[].required`} |
+ Whether documents are required to specify this attribute. | +{`bool`} |
+ {`false`} |
+
{`rangeAttributes[].predicateName`} |
+ The name of a numerical range attribute to be used in the saved search predicate field. | +{`string`} |
+ + |
{`rangeAttributes[].fieldPath`} |
+ The field in the {`productDocumentType`} to be matched with this attribute. This field should be of a numeric type, e.g. {`int`}. |
+ {`string`} |
+ + |
{`rangeAttributes[].required`} |
+ Whether documents are required to specify this attribute. | +{`bool`} |
+ {`false`} |
+
| Parameter | +Description | +Type | +Default | +
|---|---|---|---|
{`multicurrency.enabled`} |
+ Enable generation of per-currency price features at feed time. | +{`bool`} |
+ {`false`} |
+
{`multicurrency.featurePrefix`} |
+ Prefix for the generated predicate range features. A feature is named {`{prefix}_{currency}_{market}`}. |
+ {`string`} |
+ {`price`} |
+
{`multicurrency.priceScaleFactor`} |
+ Integer multiplier applied to converted prices before feeding as predicate range features. Predicate ranges require integer values, so this preserves decimal precision. A factor of 100 preserves two decimal places. | +{`int`} |
+ {`100`} |
+
| Name | +Type | +Description | +Always present | +
|---|---|---|---|
| ip | +string | +The IP address request came from | +yes | +
| time | +number | +UNIX timestamp with millisecond decimal precision (e.g. 1477828938.123) when request is received | +yes | +
| duration | +number | +The duration of the request in seconds with millisecond decimal precision (e.g. 0.123) | +yes | +
| responsesize | +number | +The size of the response in bytes | +yes | +
| code | +number | +The HTTP status code returned | +yes | +
| method | +string | +The HTTP method used (e.g. 'GET') | +yes | +
| uri | +string | +The request URI from path and beyond (e.g. '/search?query=test') | +yes | +
| version | +string | +The HTTP version (e.g. 'HTTP/1.1') | +yes | +
| agent | +string | +The user agent specified in the request | +yes | +
| host | +string | +The host header provided in the request | +yes | +
| scheme | +string | +The scheme of the request | +yes | +
| port | +number | +The IP port number of the interface on which the request was received | +yes | +
| remoteaddr | +string | +The IP address of the remote client if specified in HTTP header | +no | +
| remoteport | +string | +The port used from the remote client if specified in HTTP header | +no | +
| peeraddr | +string | +Address of immediate client making request if different from *remoteaddr* | +no | +
| peerport | +string | +Port used by immediate client making request if different from *remoteport* | +no | +
| user-principal | +string | +The name of the authenticated user (java.security.Principal.getName()) if principal is set | +no | +
| ssl-principal | +string | +The name of the x500 principal if client is authenticated through SSL/TLS | +no | +
| search | +object | +Object holding search specific fields | +no | +
| search.totalhits | +number | +The total number of hits for the query | +no | +
| search.hits | +number | +The hits returned in this specific response | +no | +
| search.coverage | +object | +Object holding query coverage information similar to that returned in result set. | +no | +
| connection | +string | +Reference to the connection log entry. See Connection log | +no | +
| attributes | +object | +Object holding custom key/value pairs logged in searcher. | +no | +
| Field | +Format | +Meaning | +Example | +
|---|---|---|---|
{`%`}Y` |
+ YYYY | +Year | +2003 | +
{`%`}m` |
+ MM | +Month, numeric | +08 | +
{`%`}x` |
+ MMM | +Month, textual | +Aug | +
{`%`}d` |
+ dd | +Date | +25 | +
{`%`}H` |
+ HH | +Hour | +14 | +
{`%`}M` |
+ mm | +Minute | +30 | +
{`%`}S` |
+ ss | +Seconds | +35 | +
{`%`}s` |
+ SSS | +Milliseconds | +123 | +
{`%`}Z` |
+ Z | +Time zone | +-0400 | +
{`%`}T` |
+ Long | +System.currentTimeMillis | +1349333576093 | +
{`%`}%{` `} |
+ {`%`}` |
+ Escape percentage | +% | +
| Name | +Type | +Description | +Always present | +
|---|---|---|---|
| id | +string | +Unique ID of the connection, referenced from access log. | +yes | +
| timestamp | +number | +Timestamp (ISO8601 format) when the connection was opened | +yes | +
| duration | +number | +The duration of the request in seconds with millisecond decimal precision (e.g. 0.123) | +yes | +
| peerAddress | +string | +IP address used by immediate client making request | +yes | +
| peerPort | +number | +Port used by immediate client making request | +yes | +
| localAddress | +string | +The local IP address the request was received on | +yes | +
| localPort | +number | +The local port the request was received on | +yes | +
| remoteAddress | +string | +Original client ip, if proxy protocol enabled | +no | +
| remotePort | +number | +Original client port, if proxy protocol enabled | +no | +
| httpBytesReceived | +number | +Number of HTTP bytes sent over the connection | +no | +
| httpBytesSent | +number | +Number of HTTP bytes received over the connection | +no | +
| requests | +number | +Number of requests sent by the client | +no | +
| responses | +number | +Number of responses sent to the client | +no | +
| ssl | +object | +Detailed information on ssl connection | +no | +
| Name | +Type | +Description | +Always present | +
|---|---|---|---|
| clientSubject | +string | +Client certificate subject | +no | +
| clientNotBefore | +string | +Client certificate valid from | +no | +
| clientNotAfter | +string | +Client certificate valid to | +no | +
| sessionId | +string | +SSL session id | +no | +
| protocol | +string | +SSL protocol | +no | +
| cipherSuite | +string | +Name of session cipher suite | +no | +
| sniServerName | +string | +SNI server name | +no | +
| Shape | +Step type | +Description | +
|---|---|---|
| + + | +Instance | ++ The application instance. Hover to see target versions, + cancel/deploy/pin controls, and block windows. + | +
| + + | +Test | ++ System test, staging test, or production test. Hover to see run status, + versions, and abort/restart actions. + | +
| + + | +Production deployment | ++ A deployment to a production zone. Hover to see run status, versions, + and abort/restart/defer actions. + | +
| + + | +Delay | +A configured delay between steps. | +
| Indicator | +Meaning | +Description | +
|---|---|---|
| + + | +Completed | ++ The step has completed successfully on the current version. The color + corresponds to the deployed version. + | +
| + + | +Running | ++ A deployment or test is currently in progress. Shown as an animated + gradient between the source and target version colors. + | +
| + + | +Failed | +The last run of this step failed. | +
| + + | +Unknown / initial | +No version has been deployed to this step yet. | +
| + + | +Pending change | ++ A newer version is queued and waiting to be deployed to this step. + | +
| + + | +Paused / deferred | +Deployments to this step are temporarily postponed. | +
| + + | +Application blocked | ++ Application changes are blocked by a{" "} + + block window + + . Shown as vertical bars. + | +
| + + | +Platform blocked | ++ Platform upgrades are blocked by a{" "} + + block window + + . Shown as horizontal bars. + | +
| Name | +Description | +Expiry | +Cluster sizes | +
|---|---|---|---|
{`dev`} |
+ Used for manual development testing. | +14 days | +{`1`} |
+
{`test`} |
+ Used for automated system tests. | +- | +{`1`} |
+
{`staging`} |
+ Used for automated staging tests. | +- | +{`min(max(2, 0.05 * spec), spec)`} |
+
{`prod`} |
+ Hosts all production deployments. | +No expiry | +{`max(2, spec)`} |
+
| Field | +ConfigServer | +Container | +Content | +ClusterControllers | +
|---|---|---|---|---|
{`labels`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`annotations`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`volumes`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`nodeSelector`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`imagePullSecrets`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`tolerations`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`podAffinity`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`podAntiAffinity`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`nodeAffinity`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`containers[vespa].env`} |
+ ✅ | ✅ | ✅ | ✅ | +
{`containers[vespa].volumeMounts`} |
+ ✅ | ❌ | ❌ | ❌ | +
{`containers[vespa].resources`} |
+ ✅ | ❌ | ❌ | ❌ | +
{`containers[]`} |
+ ✅ | ✅ | ✅ | ✅ | +
| Field | +Recreate Pod | +Recreate Pod and volume | +
|---|---|---|
{`labels`} |
+ ✅ | + |
{`annotations`} |
+ ✅ | + |
{`nodeSelector`} |
+ ✅ | +|
{`nodeAffinity`} |
+ ✅ | +|
{`podAffinity`} |
+ ✅ | +|
{`podAntiAffinity`} |
+ ✅ | +|
{`tolerations`} |
+ ✅ | +
| Kubernetes Resource | +Required Permissions | +
|---|---|
| CustomResourceDefinitions | +create, get, list, watch | +
| VespaSet | +get, list, watch, create, update, patch, delete | +
| VespaSet Subresources | +{`vespasets/status`}: update, patch {`vespasets/finalizers`}: update |
+
| ConfigMaps | +get, list, watch, create, update, patch, delete | +
| Services | +get, list, watch, create, update, patch, delete | +
| Pods | +get, list, watch, create, update, patch, delete | +
| Pod Execution | +get, create | +
| Events | +create, patch | +
| PersistentVolumeClaims | +get, list, watch, create, update, patch, delete | +
| ServiceAccounts | +get, list, watch, create, update, patch, delete | +
| Roles | +get, list, watch, create, update, patch, delete | +
| RoleBindings | +get, list, watch, create, update, patch, delete | +
| Endpoint Type | +Kubernetes Service Type | +Use Case | +
|---|---|---|
{`LOAD_BALANCER`} |
+ {`LoadBalancer`} |
+ Provision the cloud-native (AWS, GCP, Azure) load-balancer. | +
{`NODE_PORT`} |
+ {`NodePort`} |
+ Expose a static port across every worker node, allowing external traffic to access the cluster from any node's IP. | +
{`CLUSTER_IP`} |
+ {`ClusterIP`} |
+ Each Container Pod will expose an internal IP address. Should not be used for production use-cases. | +
{`NONE`} |
+ N/A | +An external access layer will not be provisioned. Custom networking setups (Istio, Ingress Controllers) where no automatic service is desired. | +
| Type | +Scenario | +Behavior | +
|---|---|---|
| **Voluntary Disruption** | +Scaling down, rolling upgrades, or node maintenance. | +The preStop hook detects a voluntary disruption, stops the Vespa Container cluster from accepting new traffic, flushes in-memory data to disk for Content clusters, and ensures a clean exit before the Pod is deleted. | +
| **Involuntary Disruption** | +Node hardware failure, kernel panic, or eviction. | +Kubernetes initiates the termination. The preStop hook attempts to run to flush data and close connections. However, if the Pod is lost abruptly. the hook cannot run, and recovery relies on Vespa's data replication. | +
| Suffix Name | +Aggregation | +
|---|---|
{`max`} |
+ Use the highest value available {`MAX(max)`}. |
+
{`min`} |
+ Use the lowest value available {`MIN(min)`}. |
+
{`sum`} |
+ Use the sum of all values {`SUM(sum)`}. |
+
{`count`} |
+ Use the sum of all values {`SUM(count)`}. |
+
{`average`} |
+ To generate an average value you want to do {`SUM(sum) / SUM(count)`} where you generate the graph. Don’t use the {`average`} suffix itself if you have the {`sum`} and {`count`} suffixes available. Using this will easily lead to computing averages of averages, which will easily become very distorted and noisy. |
+
{`last`} |
+ Avoid this except for metrics you expect to be stable, such as amount of memory available on a node, etc. This value is the last from a metrics snapshot period, hence basically a single value picked from all values during the snapshot period. Typically, very noisy for volatile metrics. It does not make sense to aggregate on this value at all, but if you must then choose a value with the same combination of tags over time. | +
{`95percentile`} |
+ This value cannot be aggregated in a way that gives a mathematically correct value. But where you have to either compute the average value for the most realistic value, {`AVERAGE(95percentile)`}, or max if the goal is to better identify outliers, {`MAX(95percentile)`}. Regardless, this value is best used when considered at the most granular level, with all tag values specified. |
+
{`99percentile`} |
+ Same as for the {`95percentile`} suffix. |
+
| Tab | +What it shows | +When to use it | +
|---|---|---|
| **Overview** | +Health indicators, request rates, QoS, latency summary, HTTP status codes, resource utilization | +Daily health check, first stop during incidents | +
| **Query** | +Container- and content-node query latency, per-rank-profile breakdown, match/docsum executors | +Investigating read latency, query quality issues | +
| **Feed** | +Feed operation rates and latency at each layer, feed blocking | +Investigating write latency or throughput issues | +
| **Nearest Neighbor Search** | +NNS distance computations, visit efficiency | +Tuning HNSW parameters (hidden when not in use) | +
| **Content Node** | +Document counts, Proton resource usage, executor utilization, maintenance jobs | +Deep investigation of search engine internals | +
| **Resources** | +CPU, memory, disk, GPU, JVM, thread pools | +Sizing and scaling decisions | +
| **Health** | +Cluster state, data consistency, restarts, reindexing, resource limits | +Stability monitoring, post-incident review | +
| Annotation | +Triggered by | +Why it matters | +
|---|---|---|
| **Feed blocked in cluster** | +A content node crosses its disk/memory feed-block limit | +Writes are paused cluster-wide until remediated | +
| **Vespa upgrade** | +A new Vespa version is rolled out | +Brief rolling-restart latency spikes are expected around this marker | +
| **Data migration** | +Bucket merges pending exceed a threshold | +Explains elevated CPU/IO and latency during redistribution | +
| **Document re-indexing** | +A reindexing job is running | +Explains elevated CPU and search-side load | +
| **Auto-scaling** | +The autoscaler changed the cluster shape | +Brief capacity drop during reshuffle | +
| **Service restart** | +{`delta(sentinel_totalRestarts[10m]) > 0`} — a Vespa service process restarted on one or more nodes |
+ Unexpected restarts usually indicate a crash, OOM, or forced stop; outside of planned upgrades these are always worth investigating | +
| **Core dump** | +{`delta(coredumps_processed[1h]) > 0`} — a process core-dumped |
+ Signals a crash; cross-reference with Service restart. Should be extremely rare | +
| Indicator | +What it counts | +Healthy value | +
|---|---|---|
| **Core Dumps (1h)** | +Core dumps processed across all clusters in the last hour | +0 — any non-zero value is a crash to investigate | +
| **Restarts (1h)** | +Vespa service restarts across all clusters in the last hour | +0 during steady state; brief spikes are normal during upgrades | +
| **Feed Blocked** | +Nodes currently above a feed-block resource limit | +0 — non-zero means writes are being rejected cluster-wide | +
| **Content: Groups/Nodes Down** | +Content groups with at least one node down | +0 during steady state. 1 group down is normal during rolling restarts or maintenance; 2 or more should be investigated | +
| **Container: Services Down** | +Active container nodes where some service isn't running | +0 during steady state; brief spikes during deployments are expected | +
| Job | +Resource impact | +
|---|---|
| Attribute Flush | +Low | +
| Memory Index Flush | +Moderate | +
| Disk Index Fusion | +High — temporary 2× disk usage | +
| Document Store Compaction | +High — holds file in memory | +
| Bucket Move | +High — competes with feeding | +
| LID-Space Compaction | +Moderate | +
| Resource | +Healthy | +Concerning | +Action needed | +
|---|---|---|---|
| **CPU** | +{`< 70%`} |
+ {`70-85%`} |
+ {`> 85%`} sustained |
+
| **CPU IOWait** | +{`< 5%`} |
+ {`5-10%`} |
+ {`> 10%`} (I/O bottleneck) |
+
| **Memory** | +{`< 70%`} |
+ {`70-80%`} |
+ Approaching feed-block limit | +
| **Disk** | +{`< 70%`} |
+ {`70-80% `} |
+ Approaching feed-block limit | +
| **JVM GC Overhead** | +{`< 5%`} |
+ {`5-15%`} |
+ {`> 15%`} (severe latency impact) |
+
| **Threadpool utilization** | +{`< 70%`} |
+ {`70-90%`} |
+ Rejected tasks = requests dropped | +
| Thread pool | +Present when | +
|---|---|
{`default-handler-common`} |
+ Always (handler executor used by anything without its own pool) | +
{`search-handler`} |
+ {` element is present |
+
{`feedapi-handler`} |
+ {` element is present |
+
| Metric name | +Threshold | +Dimension aggregation | +
|---|---|---|
| content_proton_resource_usage_disk_average | +{`>`} 0.9 |
+ max by(applicationId, clusterId, zone) | +
| content_proton_resource_usage_memory_average | +{`>`} 0.8 |
+ max by(applicationId, zone, clusterId) | +
| cpu_util | +{`>`} 90 |
+ max by(applicationId, zone, clusterId) | +
| content_proton_resource_usage_feeding_blocked_last | +{`>=`} 1 |
+ N/A | +
| Item | +Description | +
|---|---|
| **Your IAM account number** | +The numeric identifier for your AWS account. | +
| **VPC ID** | +The identifier of your AWS VPC where you wish to connect to the service endpoints from. | +
| **AWS region name** | +The name of the AWS region to connect from. Note that you can only connect to a service in the same region, or, if public endpoints are disabled, in the same AWS availability zone. | +
| Item | +Description | +
|---|---|
| **Enabled GCP APIs** | +The *Compute Engine*, *Service Directory* and *Cloud DNS* APIs must all be enabled in your GCP account:{`$ gcloud services enable compute.googleapis.com`}{`$ gcloud services enable dns.googleapis.com`}{`$ gcloud services enable servicedirectory.googleapis.com`} |
+
| **Your GCP project name** | +The string identifier for your GCP account, like *resonant-diode-123456* | +
| **VPC network and subnetwork names** | +The name of the network and subnetwork to create your consumer endpoint in. | +
| Use case | +Description | +
|---|---|
| **clear field** | +To clear a field, do a partial update of all documents with the value, say an empty string. It is also possible to use reindexing, but there is a twist - intuitively, this would work: {`field artist type string {`}{`indexing: "" | summary | index`}{`}`}However, the reset only works for synthetic fields. A solution is to deploy a document processor that empties the field, to the default indexing chain - then trigger a reprocessing. |
+
| **change indexing settings** | +As reindexing takes time, a field's data can be in one state or another, while the queries to it have the most current state. This is OK for many changes and applications. If not, it is possible to reindex to a new field for a more atomic change. Add a *synthetic field* outside the *document definition* and pipe the content of the current field to it: {`search mydocs {`} {`field title_non_stemmed type string {`} {`indexing: input title | index | summary`} {`stemming: none`} {`}`} {`document mydocs {`} {`field title type string`} {`{`} {`indexing: index | summary`} {`}`} Once reindexing is completed, switch queries to use the new field. This solution naturally increases memory and disk requirements in the transition. Going back to using the original field with the new settings can be done by changing the index settings for the original field, wait for reindexing to be finished and start using the original field again in queries, then remove the temporary synthetic field. |
+
| + | + |
|---|---|
| **No endpoint** | +Most problems with the quick start guides are due to Docker out of memory. Make sure at least 6G memory is allocated to Docker:{`$ docker info | grep "Total Memory"`}{`or`}{`$ podman info | grep "memTotal"`}OOM symptoms includeINFO: {`Problem with Handshake localhost:8080 ssl=false: localhost:8080 failed to respond `}The container is named *vespa* in the guides, for a shell do: {`$ docker exec -it vespa bash`} |
+
| **Log viewing** | +Use vespa-logfmt to view the vespa log - example:{`$ /opt/vespa/bin/vespa-logfmt -l warning,error`} |
+
| **Json** | +For json pretty-print, append {`| python -m json.tool`}to commands that output json - or use jq. |
+
| **Routing** | +Vespa lets application set up custom document processing / indexing, with different feed endpoints. Refer to indexing for how to configure this in *services.xml*. #13193 has a summary of problems and solutions. |
+
| **Tracing** | +Use tracelevel to dump the routes and hops for a write operation - example:{`$ curl -H Content-Type:application/json --data-binary @docs.json \\`}{`$ENDPOINT/document/v1/mynamespace/doc/docid/1?tracelevel=4 \\ jq .`}{`{`}{`"pathId": "/document/v1/mynamespace/doc/docid/1",`}{`"id": "id:mynamespace:doc::1",`}{`"trace": [`}{`{ "message": "[1623413878.905] Sending message (version 7.418.23) from client to ..." },`}{`{ "message": "[1623413878.906] Message (type 100004) received at 'default/container.0' ..." },`}{`{ "message": "[1623413878.907] Sending message (version 7.418.23) from 'default/container.0' ..." },`}{`{ "message": "[1623413878.907] Message (type 100004) received at 'default/container.0' ..." },`}{`{ "message": "[1623413878.909] Selecting route" },`}{`{ "message": "[1623413878.909] No cluster state cached. Sending to random distributor." }`} |
+
| + | + |
|---|---|
| **Availability vs resources** | +Keeping index structures costs resources. Not all replicas of buckets are necessarily searchable, unless configured using searchable-copies. As Vespa indexes buckets on-demand, the most cost-efficient setting is 1, if one can tolerate temporary coverage loss during node failures. | +
| **Data retention vs size** | +When a document is removed, the document data is not immediately purged. Instead, *remove-entries* (tombstones of removed documents) are kept for a configurable amount of time. The default is two weeks, refer to removed-db prune age. This ensures that removed documents stay removed in a distributed system where nodes change state. Entries are removed periodically after expiry. Hence, if a node comes back up after being down for more than two weeks, removed documents are available again, unless the data on the node is wiped first. A larger *prune age* will grow the storage size as this keeps document and tombstones longer. The backend does not store remove-entries for nonexistent documents. This to prevent clients sending wrong document identifiers from filling a cluster with invalid remove-entries. A side effect is that if a problem has caused all replicas of a bucket to be unavailable, documents in this bucket cannot be marked removed until at least one replica is available again. Documents are written in new bucket replicas while the others are down - if these are removed, then older versions of these will not re-emerge, as the most recent change wins. |
+
| **Transition time** | +See transition-time for tradeoffs for how quickly nodes are set down vs. system stability. | +
| **Removing unstable nodes** | +One can configure how many times a node is allowed to crash before it will automatically be removed. The crash count is reset if the node has been up or down continuously for more than the stable state period. If the crash count exceeds max premature crashes, the node will be disabled. Refer to troubleshooting. | +
| **Minimal amount of nodes required to be available** | +A cluster is typically sized to handle a given load. A given percentage of the cluster resources are required for normal operations, and the remainder is the available resources that can be used if some of the nodes are no longer usable. If the cluster loses enough nodes, it will be overloaded: • Remaining nodes may create disk full situation. This will likely fail a lot of write operations, and if disk is shared with OS, it may also stop the node from functioning. • Partition queues will grow to maximum size. As queues are processed in FIFO order, operations are likely to get long latencies. • Many operations may time out while being processed, causing the operation to be resent, adding more load to the cluster. • When new nodes are added, they cannot serve requests before data is moved to the new nodes from the already overloaded nodes. Moving data puts even more load on the existing nodes, and as moving data is typically not high priority this may never actually happen. To configure what the minimal cluster size is, use min-distributor-up-ratio and min-storage-up-ratio. |
+
| Image | +Description | +
|---|---|
| docker.io/vespaengine/vespa ghcr.io/vespa-engine/vespa | +Container image for running Vespa. | +
| docker.io/vespaengine/vespa-build-almalinux-8 | +Container image for building Vespa on AlmaLinux 8. | +
| docker.io/vespaengine/vespa-dev-almalinux-8 | +Container image for development of Vespa on AlmaLinux 8. Used for incremental building and system testing. | +
| RPM | +Description | +
|---|---|
| **vespa-tools** | +Tools accessing Vespa endpoints for query or document operations: • vespa-destination • vespa-fbench • vespa-feeder • vespa-get • vespa-query-profile-dump-tool • vespa-stat • vespa-summary-benchmark • vespa-visit • vespa-visit-target |
+
| **vespa-malloc** | +Vespa has its own memory allocator, *vespa-malloc* - refer to */opt/vespa/etc/vespamalloc.conf* | +
| **vespa-clients** | +*vespa-feed-client.jar* - see vespa-feed-client | +
| Mode | +Description | +
|---|---|
| default | +Gets config from server and stores in memory cache. The config proxy will always be started in *default* mode. Serves from cache if possible. Always uses a config source. If restarted, it will lose all configs that were cached in memory. | +
| memorycache | +Serves config from memory cache only. Never uses a config source. A restart will lose all cached configs. Setting the mode to *memorycache* will make all applications on the node work as before (given that they have previously been running and requested config), since the config proxy will serve config from cache and work without connection to any config server. Applications on this node will not work if the config proxy stops, is restarted or crashes. | +
| Process | +Description | +
|---|---|
| config-proxy | +Proxies config requests between Vespa applications and the configserver node. All configuration is cached locally so that this node can maintain its current configuration, even if the configserver shuts down. | +
| **config-sentinel** | +Registers itself with the *config-proxy* and subscribes to and enforces node configuration, meaning the configuration of what services should be run locally, and with what parameters. | +
| vespa-logd | +Monitors *$VESPA_HOME/logs/vespa/vespa.log*, which is used by all other services, and relays everything to the log-server. | +
| metrics-proxy | +Provides APIs for metrics access to all nodes and services. | +
| Problem | +Description | +
|---|---|
| **Health checks** | +Verify that a config server is up and running using /state/v1/health, see start sequence. Status code is {`up`} if the server is up and has finished bootstrapping. Alternatively, use http://localhost:19071/status.html which will return response code 200 if server is up and has finished bootstrapping. Metrics are found at /state/v1/metrics. Use vespa-model-inspect to find host and port number, port is 19071 by default. |
+
| **Consistency** | +When having more than one config server, consistency between the servers is crucial. http://localhost:19071/status can be used to check that settings for config servers are the same for all servers. vespa-config-status can be used to check config on nodes. http://localhost:19071/application/v2/tenant/default/application/default displays active config generation and should be the same on all servers, and the same as in response from running vespa deploy |
+
| **Bad Node** | +If running with more than one config server and one of these goes down or has hardware failure, the cluster will still work and serve config as usual (clients will switch to use one of the good servers). It is not necessary to remove a bad server from the configuration. Deploying applications will take longer, as vespa deploy will not be able to complete a deployment on all servers when one of them is down. If this is troublesome, lower the barrier timeout - (default value is 120 seconds). Note also that if you have not configured cluster controllers explicitly, these will run on the config server nodes and the operation of these might be affected. This is another reason for not trying to manually remove a bad node from the config server setup. |
+
| **Stuck filedistribution** | +The config system distributes binary files (such as jar bundle files) using file-distribution - use vespa-status-filedistribution to see detailed status if it gets stuck. | +
| **Memory** | +Insufficient memory on the host / in the container running the config server will cause startup or deploy / configuration problems - see Docker containers. | +
| **ZooKeeper** | +The following can be caused by a full disk on the config server, or clocks out of sync:{`at com.yahoo.vespa.zookeeper.ZooKeeperRunner.startServer(ZooKeeperRunner.java:92)`}{`Caused by: java.io.IOException: The accepted epoch, 10 is less than the current epoch, 48`}Users have reported that "Copying the currentEpoch to acceptedEpoch fixed the problem". |
+
| Service | +Port 1 | +Port 2 | +
|---|---|---|
| QRS | +19015 | +19016 | +
| Docproc | +19123 | +19124 | +
| Directory | +Description | +
|---|---|
{`$VESPA_HOME/bin/`} |
+ Command line utilities and scripts | +
{`$VESPA_HOME/libexec/vespa/`} |
+ Command line utilities and scripts | +
{`$VESPA_HOME/sbin/`} |
+ Server programs, daemons, etc | +
{`$VESPA_HOME/lib64/`} |
+ Dynamically linked libraries, typically third-party libraries | +
{`$VESPA_HOME/lib/jars/`} |
+ Java archives | +
{`$VESPA_HOME/logs/vespa/`} |
+ Log files | +
{`$VESPA_HOME/var/db/vespa/config_server/serverdb/`} |
+ Config server database and user applications | +
{`$VESPA_HOME/share/vespa/`} |
+ A directory with config definitions and XML schemas for application package validation | +
{`$VESPA_HOME/conf/vespa`} |
+ Various config files used by Vespa or libraries Vespa depend on | +
| Process | +Host | +Port/range | +ps | +Function | +
|---|---|---|---|---|
| Config server | +Config server nodes | +19070-19071 | +{`java (...) -jar $VESPA_HOME/lib/jars/standalone-container-jar-with-dependencies.jar`} |
+ Vespa Configuration server | +
| + | + | 2181-2183 | ++ | Embedded Zookeeper cluster ports, see zookeeper-server.def | +
| Config sentinel | +All nodes | +19098 | +{`$VESPA_HOME/sbin/vespa-config-sentinel`} |
+ Sentinel that starts and stops vespa services and makes sure they are running unless they are manually stopped | +
| Config proxy | +All nodes | +19090 | +{`java (…) com.yahoo.vespa.config.proxy.ProxyServer`} |
+ Communication liaison between Vespa processes and config server. Caches config in memory | +
| Slobrok | +Admin nodes | +19099 for RPC port, HTTP port dynamically allocated in the 19100-19899 range | +{`$VESPA_HOME/sbin/vespa-slobrok`} |
+ Service location object broker | +
| logd | +All nodes | +19089 | +{`$VESPA_HOME/sbin/vespa-logd`} |
+ Reads local log files and sends them to log server | +
| Log server | +Log server node | +19080 | +{`java (...) -jar lib/jars/logserver-jar-with-dependencies.jar`} |
+ Vespa Log server | +
| Metrics proxy | +All nodes | +19092-19095 | +{`java (...) -jar $VESPA_HOME/lib/jars/container-disc-with-dependencies.jar`} |
+ Provides a single access point for metrics from all services on a Vespa node | +
| Distributor | +Content cluster | +dynamically allocated in the 19100-19899 range | +{`$VESPA_HOME/sbin/vespa-distributord-bin`} |
+ Content layer distributor processes | +
| Cluster controller | +Content cluster | +19050, plus ports dynamically allocated in the 19100-19899 range | +{`java (...) -jar $VESPA_HOME/lib/jars/container-disc-jar-with-dependencies.jar`} |
+ Cluster controller processes, manages state for content nodes | +
| proton | +Content cluster | +dynamically allocated in the 19100-19899 range | +{`$VESPA_HOME/sbin/vespa-proton-bin`} |
+ Searchnode process, receives queries from the container and returns results from the indexes. Also receives feed and indexes documents | +
| container | +Container cluster | +8080 | +{`java (...) -jar $VESPA_HOME/lib/jars/container-disc-with-dependencies.jar`} |
+ Container running servers, handlers and processing components | +
| Item | +Description | +
|---|---|
| action | +One of {`fallback`}, {`override`}, or {`unset`}. {`fallback`} sets the variable if it is unset (or empty). {`override`} set the value regardless. {`unset`} unsets the variable. |
+
| variablename | +The name of the variable, e.g. {`VESPA_CONFIGSERVERS`} |
+
| value | +The rest of the line is the variable's value. | +
| Environment variable | +Description | +
|---|---|
| VESPA_CONFIGSERVERS | +A comma-separated list of hosts to run configservers, use fully qualified hostnames. Should always be set to the same value on all hosts in a multi-host setup. If not set, {`localhost`} is assumed. Refer to configuration server operations. |
+
| VESPA_HOSTNAME | +Vespa uses {`hostname`} for node identity. But sometimes this doesn't work properly, either because that name can't be used to find an IP address which works for connecting to services running on the node, or it's just that the name doesn't agree with what the config server thinks the node's host name is. In this case, override by setting the {`VESPA_HOSTNAME`}, to be used instead of running the {`hostname`} command.Note that {`VESPA_HOSTNAME`} will be used *both* when a node identifies itself to the config server *and* when a service on that node registers a network connection point that other services can connect to.An error message with "hostname detection failed" is emitted if the {`VESPA_HOSTNAME`} isn't set and the hostname isn't usable. If {`VESPA_HOSTNAME`} is set to something that cannot work, an error with "hostname validation failed" is emitted instead. |
+
| VESPA_CONFIG_SOURCES | +Used by libraries like the Document API to set config server endpoints. Refer to configuration server operations for example use. | +
| VESPA_WEB_SERVICE_PORT | +The port number where REST apis will run, default {`8080`}. This isn't strictly needed, as the port number can be set for each HTTP server in {`services.xml`}, but with a big application it can be easier to set the default port number just once. Also note that this needs to be set when starting the *configserver*, since the REST api implementation gets its port number from there. |
+
| VESPA_TLS_CONFIG_FILE | +Absolute path to TLS configuration file. | +
| VESPA_CONFIGSERVER_JVMARGS | +JVM arguments for the config server - see tuning. | +
| VESPA_CONFIGPROXY_JVMARGS | +JVM arguments for the config proxy - see tuning. | +
| VESPA_LOG_LEVEL | +Tuning of log output from tools, see controlling log levels | +
| Service | +Description | +
|---|---|
| CloudWatch | +Metrics can be pulled into CloudWatch from both Vespa Cloud and self-hosted Vespa. The recommended solution is to use an AWS lambda function, as described in Pulling Vespa metrics to Cloudwatch. | +
| Datadog | +The Vespa team has created a Datadog Agent integration to allow real-time monitoring of Vespa in Datadog. The Datadog Vespa integration is not packaged with the agent, but is included in Datadog's integrations-extras repository. Clone it and follow the steps in the README. The Datadog Agent integration currently works for self-hosted Vespa only. |
+
| Prometheus | +Vespa exposes metrics in a text based format that can be scraped by Prometheus. For Vespa Cloud, append */prometheus/v1/values* to your endpoint URL. For self-hosted Vespa the URL is: *http://{`/prometheus/v1/values*, where the *port* is the same as for searching, e.g. 8080. Metrics for each individual host can also be retrieved at {`http://host:19092/prometheus/v1/values`}.See the below for a Prometheus / Grafana example. |
+
| type | +Private IP DNS name (IPv4 only) | +Public IPv4 DNS | +
|---|---|---|
| configserver | +ip-10-0-1-234.ec2.internal | +ec2-3-231-33-190.compute-1.amazonaws.com | +
| configserver | +ip-10-0-1-154.ec2.internal | +ec2-3-216-28-201.compute-1.amazonaws.com | +
| configserver | +ip-10-0-0-88.ec2.internal | +ec2-34-230-33-42.compute-1.amazonaws.com | +
| services | +ip-10-0-1-95.ec2.internal | +ec2-44-192-98-165.compute-1.amazonaws.com | +
| services | +ip-10-0-0-219.ec2.internal | +ec2-3-88-143-47.compute-1.amazonaws.com | +
| services | +ip-10-0-0-28.ec2.internal | +ec2-107-23-52-245.compute-1.amazonaws.com | +
| services | +ip-10-0-0-67.ec2.internal | +ec2-54-198-251-100.compute-1.amazonaws.com | +
| services | +ip-10-0-1-84.ec2.internal | +ec2-44-193-84-85.compute-1.amazonaws.com | +
| services | +ip-10-0-0-167.ec2.internal | +ec2-54-224-15-163.compute-1.amazonaws.com | +
| services | +ip-10-0-1-41.ec2.internal | +ec2-44-200-227-127.compute-1.amazonaws.com | +
| Cluster name | +vespa | +
|---|---|
| EC2 instance type | +t2.medium | +
| Number of instances | +10 | +
| Key pair | +*Select or create your keypair* | +
| Security group inbound rules - port range | +0 - 65535 | +
| Launch type | +EC2 | +
|---|---|
| Cluster | +vespa | +
| Number of tasks | +3 | +
| Placement templates | +One Task Per Host | +
| Launch type | +EC2 | +
|---|---|
| Cluster | +vespa | +
| Number of tasks | +7 | +
| Placement templates | +One Task Per Host | +
| Option | +Required | +Default | +Description | +
|---|---|---|---|
{`--dest-dir`} |
+ Yes | +— | +Directory where the diagnostic output files will be written. | +
{`--config-server-host`} |
+ No | +Auto-detected | +Host address of the Vespa config server. If not set, the tool resolves it automatically from the environment. | +
{`--config-server-port`} |
+ No | +19071 | +Port of the Vespa config server. | +
{`--timeout-secs`} |
+ No | +60 | +Timeout in seconds for operations that contact the config server. | +
| Environment | +Default | +Region | +Availability Zones | +
|---|---|---|---|
| dev | +Yes | +aws-us-east-1c | +use1-az6 | +
| dev | +No | +aws-euw1-az1 | +euw1-az1 | +
| dev | +No | +azure-eastus-az1 | ++ |
| dev | +No | +gcp-us-central1-f | ++ |
| Environment | +Region | +Availability Zones | +
|---|---|---|
| prod | +aws-us-east-1 | +use1-az1 to use1-az6, except use1-az3 | +
| prod | +aws-us-east-1c | +use1-az6 | +
| prod | +aws-use1-az4 | +use1-az4 | +
| prod | +aws-use2-az1 | +use2-az1 | +
| prod | +aws-use2-az3 | +use2-az3 | +
| prod | +aws-us-west-2a | +usw2-az1 | +
| prod | +aws-usw2-az2 | +usw2-az2 | +
| prod | +aws-usw2-az3 | +usw2-az3 | +
| prod | +aws-eu-west-1 | +euw1-az1 to euw1-az3 | +
| prod | +aws-eu-west-1a | +euw1-az2 | +
| prod | +aws-euw1-az1 | +euw1-az1 | +
| prod | +aws-euw1-az3 | +euw1-az3 | +
| prod | +aws-euc1-az1 | +euc1-az1 | +
| prod | +aws-euc1-az2 | +euc1-az2 | +
| prod | +aws-euc1-az3 | +euc1-az3 | +
| prod | +aws-eun1-az1 | +eun1-az1 | +
| prod | +aws-cac1-az1 | +cac1-az1 | +
| prod | +aws-cac1-az2 | +cac1-az2 | +
| prod | +aws-cac1-az4 | +cac1-az4 | +
| prod | +aws-aps1-az1 | +aps1-az1 | +
| prod | +aws-ap-northeast-1a | +apne1-az4 | +
| prod | +aws-apne1-az1 | +apne1-az1 | +
| prod | +aws-apne1-az2 | +apne1-az2 | +
| prod | +gcp-europe-west3-b | ++ |
| prod | +gcp-us-central1-a | ++ |
| prod | +gcp-us-central1-b | ++ |
| prod | +gcp-us-central1-c | ++ |
| prod | +gcp-us-central1-f | ++ |
| prod | +gcp-us-east4-c | ++ |
| prod | +gcp-us-west1-c | ++ |
| Environment | +Region | +Availability Zones | +
|---|---|---|
| test | +aws-us-east-1c | +use1-az6 | +
| test | +gcp-us-central1-f | ++ |
| staging | +aws-us-east-1c | +use1-az6 | +
| staging | +gcp-us-central1-f | ++ |
| Type | +Description | +
|---|---|
| double | +The default tensor cell type is the 64-bit floating-point {`double`} format. It gives the best precision at the cost of high memory usage and somewhat slower calculations. Using a smaller value type increases performance, trading off precision, so consider changing to one of the cell types below before scaling the application. |
+
| float | +The 32-bit floating-point format {`float`} should usually be used for all tensors when scaling for production. Note that some frameworks like TensorFlow prefer 32-bit floats. A vector with 1000 dimensions, {`tensor uses approximately 4K memory per tensor value. |
+
| bfloat16 | +This type has the range as a normal 32-bit float but only 8 bits of precision and can be thought of as a "float with lossy compression" - see Wikipedia. If memory (or memory bandwidth) is a concern, change the most space-consuming tensors to use the {`bfloat16`} cell type. Some careful analysis of the data is required before using this type. When doing calculations, {`bfloat16`} will act as if it was a 32-bit float, but the smaller size comes with a potential computational overhead. In most cases, the {`bfloat16`} needs conversion to a 32-bit float before the actual calculation can occur, adding an extra conversion step. In some cases, having tensors with {`bfloat16`} cells might bypass some built-in optimizations (like matrix multiplication) that will be hardware-accelerated only if the cells are of the same type. To avoid this, use the cell_cast tensor operation to make sure the cells are of the right type before doing the more expensive operations. |
+
| int8 | +If using machine learning to generate a model with data quantization, one can target the {`int8`} cell value type, which is a signed integer with a range from -128 to +127 only. This is also treated like a "float with limited range and lossy compression" by the Vespa tensor framework, and gives results as if it were a 32-bit float when any calculation is done. This type is also suitable when representing boolean values (0 or 1).If the input for an {`int8`} cell is not directly representable, the resulting cell value is undefined, so take care to only input numbers in the {`[-128,127]`} range. It's also possible to use {`int8`} representing binary data for hamming distance Nearest-Neighbor search. Refer to billion-scale-knn for example use.> |
+
| tensor a | +tensor b | +product | +sum | +comment | +
|---|---|---|---|---|
| tensor(x[3]):[1.0, 2.0, 3.0] | +tensor(x[3]):[4.0, 5.0, 6.0] | +tensor(x[3]):[4.0, 10.0, 18.0] | +32 | +Playground example. The dimension name and size are the same in both tensors - this is an inner product with a scalar result. | +
| tensor(x[3]):[1.0, 2.0, 3.0] | +tensor(y[3]):[4.0, 5.0, 6.0] | +tensor(x[3],y[3]):[ [4.0, 5.0, 6.0], [8.0, 10.0, 12.0], [12.0, 15.0, 18.0] ] | +90 | +Playground example. The dimension size is the same in both tensors, but dimensions have different names -> this is an outer product; the result is a two-dimensional tensor. | +
| tensor(x[3]):[1.0, 2.0, 3.0] | +tensor(x[2]):[4.0, 5.0] | +undefined | ++ | Playground example. Two tensors in the same dimension but with different lengths -> undefined. | +
| tensor(x[3]):[1.0, 2.0, 3.0] | +tensor(y[2]):[4.0, 5.0] | +tensor(x[3],y[2]):[ [4.0, 5.0], [8.0, 10.0], [12.0, 15.0] ] | +54 | +Playground example. Two tensors with different names and dimensions -> this is an outer product; the result is a two-dimensional tensor. | +
| tensor name | +tensor type | +
|---|---|
| model_id | +{`tensor(m_id{})`} |
+
| models | +{`tensor(m_id{}, x[3])`} |
+
| tensor name | +tensor type | +
|---|---|
| query(model_id) | +{`tensor(model{})`} |
+
| model_weights | +{`tensor(model{}, feature{})`} |
+
| model_features | +{`tensor(feature{})`} |
+
| tensor name | +tensor type | +
|---|---|
| query(model_id) | +{`tensor(model{})`} |
+
| model_weights | +{`tensor(model{}, feature[2])`} |
+
| model_features | +{`tensor(feature[2])`} |
+
| Architecture | +CPU cores | +Memory (GB) | +Local SSD (GB) | +GPU Memory (GB) | ++ |
|---|---|---|---|---|---|
| arm64 | +1.0 | +8 | +- | +- | ++ |
| arm64 | +1.0 | +8 | +59 | +- | ++ |
| arm64 | +1.0 | +16 | +- | +- | ++ |
| arm64 | +1.0 | +16 | +59 | +- | ++ |
| arm64 | +2.0 | +8 | +- | +- | ++ |
| arm64 | +2.0 | +8 | +118 | +- | ++ |
| arm64 | +2.0 | +16 | +- | +- | ++ |
| arm64 | +2.0 | +16 | +118 | +- | ++ |
| arm64 | +2.0 | +16 | +468 | +- | ++ |
| arm64 | +2.0 | +32 | +- | +- | ++ |
| arm64 | +2.0 | +32 | +118 | +- | ++ |
| arm64 | +4.0 | +8 | +- | +- | ++ |
| arm64 | +4.0 | +8 | +237 | +- | ++ |
| arm64 | +4.0 | +16 | +- | +- | ++ |
| arm64 | +4.0 | +16 | +237 | +- | ++ |
| arm64 | +4.0 | +32 | +- | +- | ++ |
| arm64 | +4.0 | +32 | +237 | +- | ++ |
| arm64 | +4.0 | +32 | +937 | +- | ++ |
| arm64 | +4.0 | +64 | +- | +- | ++ |
| arm64 | +4.0 | +64 | +237 | +- | ++ |
| arm64 | +8.0 | +16 | +- | +- | ++ |
| arm64 | +8.0 | +16 | +474 | +- | ++ |
| arm64 | +8.0 | +32 | +- | +- | ++ |
| arm64 | +8.0 | +32 | +474 | +- | ++ |
| arm64 | +8.0 | +64 | +- | +- | ++ |
| arm64 | +8.0 | +64 | +474 | +- | ++ |
| arm64 | +8.0 | +64 | +1875 | +- | ++ |
| arm64 | +8.0 | +128 | +- | +- | ++ |
| arm64 | +8.0 | +128 | +475 | +- | ++ |
| arm64 | +16.0 | +32 | +- | +- | ++ |
| arm64 | +16.0 | +32 | +950 | +- | ++ |
| arm64 | +16.0 | +64 | +- | +- | ++ |
| arm64 | +16.0 | +64 | +950 | +- | ++ |
| arm64 | +16.0 | +128 | +- | +- | ++ |
| arm64 | +16.0 | +128 | +950 | +- | ++ |
| arm64 | +16.0 | +128 | +3750 | +- | ++ |
| arm64 | +16.0 | +256 | +- | +- | ++ |
| arm64 | +16.0 | +256 | +950 | +- | ++ |
| arm64 | +32.0 | +64 | +- | +- | ++ |
| arm64 | +32.0 | +64 | +1900 | +- | ++ |
| arm64 | +32.0 | +128 | +- | +- | ++ |
| arm64 | +32.0 | +128 | +1900 | +- | ++ |
| arm64 | +32.0 | +256 | +- | +- | ++ |
| arm64 | +32.0 | +256 | +1900 | +- | ++ |
| arm64 | +32.0 | +256 | +7500 | +- | ++ |
| arm64 | +32.0 | +512 | +- | +- | ++ |
| arm64 | +32.0 | +512 | +1900 | +- | ++ |
| arm64 | +48.0 | +96 | +- | +- | ++ |
| arm64 | +48.0 | +96 | +2850 | +- | ++ |
| arm64 | +48.0 | +192 | +- | +- | ++ |
| arm64 | +48.0 | +192 | +2850 | +- | ++ |
| arm64 | +48.0 | +384 | +- | +- | ++ |
| arm64 | +48.0 | +384 | +2850 | +- | ++ |
| arm64 | +48.0 | +384 | +11250 | +- | ++ |
| arm64 | +48.0 | +768 | +- | +- | ++ |
| arm64 | +48.0 | +768 | +2850 | +- | ++ |
| arm64 | +64.0 | +128 | +- | +- | ++ |
| arm64 | +64.0 | +128 | +3800 | +- | ++ |
| arm64 | +64.0 | +256 | +- | +- | ++ |
| arm64 | +64.0 | +256 | +3800 | +- | ++ |
| arm64 | +64.0 | +512 | +- | +- | ++ |
| arm64 | +64.0 | +512 | +3800 | +- | ++ |
| arm64 | +64.0 | +512 | +15000 | +- | ++ |
| arm64 | +64.0 | +1024 | +- | +- | ++ |
| arm64 | +64.0 | +1024 | +3800 | +- | ++ |
| arm64 | +96.0 | +192 | +- | +- | ++ |
| arm64 | +96.0 | +192 | +5700 | +- | ++ |
| arm64 | +96.0 | +384 | +- | +- | ++ |
| arm64 | +96.0 | +384 | +5700 | +- | ++ |
| arm64 | +96.0 | +768 | +- | +- | ++ |
| arm64 | +96.0 | +768 | +5700 | +- | ++ |
| arm64 | +96.0 | +768 | +22500 | +- | ++ |
| arm64 | +96.0 | +1536 | +- | +- | ++ |
| arm64 | +192.0 | +384 | +- | +- | ++ |
| arm64 | +192.0 | +384 | +11400 | +- | ++ |
| arm64 | +192.0 | +768 | +- | +- | ++ |
| arm64 | +192.0 | +768 | +11400 | +- | ++ |
| arm64 | +192.0 | +1536 | +- | +- | ++ |
| arm64 | +192.0 | +1536 | +11400 | +- | ++ |
| arm64 | +192.0 | +1536 | +45000 | +- | ++ |
| arm64 | +192.0 | +3072 | +- | +- | ++ |
| x86_64 | +2.0 | +8 | +- | +- | ++ |
| x86_64 | +2.0 | +8 | +75 | +- | ++ |
| x86_64 | +2.0 | +8 | +118 | +- | ++ |
| x86_64 | +2.0 | +16 | +- | +- | ++ |
| x86_64 | +2.0 | +16 | +75 | +- | ++ |
| x86_64 | +2.0 | +16 | +468 | +- | ++ |
| x86_64 | +2.0 | +16 | +1250 | +- | ++ |
| x86_64 | +4.0 | +8 | +- | +- | ++ |
| x86_64 | +4.0 | +8 | +100 | +- | ++ |
| x86_64 | +4.0 | +8 | +237 | +- | ++ |
| x86_64 | +4.0 | +16 | +- | +- | ++ |
| x86_64 | +4.0 | +16 | +125 | +16.0 | ++ |
| x86_64 | +4.0 | +16 | +150 | +- | ++ |
| x86_64 | +4.0 | +16 | +237 | +- | ++ |
| x86_64 | +4.0 | +32 | +- | +- | ++ |
| x86_64 | +4.0 | +32 | +150 | +- | ++ |
| x86_64 | +4.0 | +32 | +237 | +- | ++ |
| x86_64 | +4.0 | +32 | +937 | +- | ++ |
| x86_64 | +4.0 | +32 | +2500 | +- | ++ |
| x86_64 | +8.0 | +16 | +- | +- | ++ |
| x86_64 | +8.0 | +16 | +200 | +- | ++ |
| x86_64 | +8.0 | +16 | +474 | +- | ++ |
| x86_64 | +8.0 | +32 | +- | +- | ++ |
| x86_64 | +8.0 | +32 | +225 | +16.0 | ++ |
| x86_64 | +8.0 | +32 | +300 | +- | ++ |
| x86_64 | +8.0 | +32 | +474 | +- | ++ |
| x86_64 | +8.0 | +64 | +- | +- | ++ |
| x86_64 | +8.0 | +64 | +300 | +- | ++ |
| x86_64 | +8.0 | +64 | +1875 | +- | ++ |
| x86_64 | +8.0 | +64 | +5000 | +- | ++ |
| x86_64 | +12.0 | +96 | +7500 | +- | ++ |
| x86_64 | +16.0 | +32 | +- | +- | ++ |
| x86_64 | +16.0 | +32 | +400 | +- | ++ |
| x86_64 | +16.0 | +32 | +950 | +- | ++ |
| x86_64 | +16.0 | +64 | +- | +- | ++ |
| x86_64 | +16.0 | +64 | +600 | +- | ++ |
| x86_64 | +16.0 | +64 | +950 | +- | ++ |
| x86_64 | +16.0 | +128 | +- | +- | ++ |
| x86_64 | +16.0 | +128 | +600 | +- | ++ |
| x86_64 | +16.0 | +128 | +3750 | +- | ++ |
| x86_64 | +24.0 | +192 | +900 | +- | ++ |
| x86_64 | +24.0 | +192 | +15000 | +- | ++ |
| x86_64 | +32.0 | +64 | +- | +- | ++ |
| x86_64 | +32.0 | +64 | +1200 | +- | ++ |
| x86_64 | +32.0 | +64 | +1900 | +- | ++ |
| x86_64 | +32.0 | +128 | +- | +- | ++ |
| x86_64 | +32.0 | +128 | +1200 | +- | ++ |
| x86_64 | +32.0 | +128 | +1900 | +- | ++ |
| x86_64 | +32.0 | +256 | +- | +- | ++ |
| x86_64 | +32.0 | +256 | +1200 | +- | ++ |
| x86_64 | +32.0 | +256 | +7500 | +- | ++ |
| x86_64 | +36.0 | +72 | +- | +- | ++ |
| x86_64 | +36.0 | +72 | +900 | +- | ++ |
| x86_64 | +48.0 | +96 | +- | +- | ++ |
| x86_64 | +48.0 | +96 | +1800 | +- | ++ |
| x86_64 | +48.0 | +96 | +2850 | +- | ++ |
| x86_64 | +48.0 | +192 | +- | +- | ++ |
| x86_64 | +48.0 | +192 | +1800 | +- | ++ |
| x86_64 | +48.0 | +192 | +2850 | +- | ++ |
| x86_64 | +48.0 | +384 | +- | +- | ++ |
| x86_64 | +48.0 | +384 | +1800 | +- | ++ |
| x86_64 | +48.0 | +384 | +30000 | +- | ++ |
| x86_64 | +64.0 | +128 | +- | +- | ++ |
| x86_64 | +64.0 | +128 | +3800 | +- | ++ |
| x86_64 | +64.0 | +256 | +- | +- | ++ |
| x86_64 | +64.0 | +256 | +2400 | +- | ++ |
| x86_64 | +64.0 | +256 | +3800 | +- | ++ |
| x86_64 | +64.0 | +512 | +- | +- | ++ |
| x86_64 | +64.0 | +512 | +2400 | +- | ++ |
| x86_64 | +64.0 | +512 | +15000 | +- | ++ |
| x86_64 | +72.0 | +144 | +- | +- | ++ |
| x86_64 | +72.0 | +144 | +1800 | +- | ++ |
| x86_64 | +72.0 | +576 | +45000 | +- | ++ |
| x86_64 | +96.0 | +192 | +- | +- | ++ |
| x86_64 | +96.0 | +192 | +3600 | +- | ++ |
| x86_64 | +96.0 | +192 | +5700 | +- | ++ |
| x86_64 | +96.0 | +384 | +- | +- | ++ |
| x86_64 | +96.0 | +384 | +3600 | +- | ++ |
| x86_64 | +96.0 | +768 | +- | +- | ++ |
| x86_64 | +96.0 | +768 | +3600 | +- | ++ |
| x86_64 | +96.0 | +768 | +60000 | +- | ++ |
| x86_64 | +128.0 | +1024 | +30000 | +- | ++ |
| x86_64 | +192.0 | +1536 | +120000 | +- | ++ |
| Architecture | +CPU cores | +Memory (GB) | +Local SSD (GB) | +GPU Memory (GB) | ++ |
|---|---|---|---|---|---|
| x86_64 | +2.0 | +8 | +- | +- | ++ |
| x86_64 | +2.0 | +8 | +75 | +- | ++ |
| x86_64 | +2.0 | +8 | +118 | +- | ++ |
| x86_64 | +2.0 | +16 | +- | +- | ++ |
| x86_64 | +2.0 | +16 | +75 | +- | ++ |
| x86_64 | +2.0 | +16 | +118 | +- | ++ |
| x86_64 | +4.0 | +8 | +- | +- | ++ |
| x86_64 | +4.0 | +8 | +150 | +- | ++ |
| x86_64 | +4.0 | +16 | +- | +- | ++ |
| x86_64 | +4.0 | +16 | +150 | +- | ++ |
| x86_64 | +4.0 | +16 | +236 | +- | ++ |
| x86_64 | +4.0 | +32 | +- | +- | ++ |
| x86_64 | +4.0 | +32 | +150 | +- | ++ |
| x86_64 | +4.0 | +32 | +236 | +- | ++ |
| x86_64 | +8.0 | +16 | +- | +- | ++ |
| x86_64 | +8.0 | +16 | +300 | +- | ++ |
| x86_64 | +8.0 | +32 | +- | +- | ++ |
| x86_64 | +8.0 | +32 | +300 | +- | ++ |
| x86_64 | +8.0 | +32 | +472 | +- | ++ |
| x86_64 | +8.0 | +64 | +- | +- | ++ |
| x86_64 | +8.0 | +64 | +300 | +- | ++ |
| x86_64 | +8.0 | +64 | +472 | +- | ++ |
| x86_64 | +16.0 | +32 | +- | +- | ++ |
| x86_64 | +16.0 | +32 | +600 | +- | ++ |
| x86_64 | +16.0 | +64 | +- | +- | ++ |
| x86_64 | +16.0 | +64 | +600 | +- | ++ |
| x86_64 | +16.0 | +64 | +944 | +- | ++ |
| x86_64 | +16.0 | +128 | +- | +- | ++ |
| x86_64 | +16.0 | +128 | +600 | +- | ++ |
| x86_64 | +16.0 | +128 | +944 | +- | ++ |
| x86_64 | +20.0 | +160 | +- | +- | ++ |
| x86_64 | +20.0 | +160 | +750 | +- | ++ |
| x86_64 | +20.0 | +160 | +1181 | +- | ++ |
| x86_64 | +32.0 | +64 | +- | +- | ++ |
| x86_64 | +32.0 | +64 | +1200 | +- | ++ |
| x86_64 | +32.0 | +128 | +- | +- | ++ |
| x86_64 | +32.0 | +128 | +1200 | +- | ++ |
| x86_64 | +32.0 | +128 | +1889 | +- | ++ |
| x86_64 | +32.0 | +256 | +- | +- | ++ |
| x86_64 | +32.0 | +256 | +1200 | +- | ++ |
| x86_64 | +32.0 | +256 | +1889 | +- | ++ |
| x86_64 | +48.0 | +96 | +- | +- | ++ |
| x86_64 | +48.0 | +96 | +1800 | +- | ++ |
| x86_64 | +48.0 | +192 | +- | +- | ++ |
| x86_64 | +48.0 | +192 | +1800 | +- | ++ |
| x86_64 | +48.0 | +192 | +2834 | +- | ++ |
| x86_64 | +48.0 | +384 | +- | +- | ++ |
| x86_64 | +48.0 | +384 | +1800 | +- | ++ |
| x86_64 | +48.0 | +384 | +2834 | +- | ++ |
| x86_64 | +64.0 | +128 | +- | +- | ++ |
| x86_64 | +64.0 | +128 | +2400 | +- | ++ |
| x86_64 | +64.0 | +256 | +- | +- | ++ |
| x86_64 | +64.0 | +256 | +2400 | +- | ++ |
| x86_64 | +64.0 | +256 | +3779 | +- | ++ |
| x86_64 | +64.0 | +512 | +- | +- | ++ |
| x86_64 | +64.0 | +512 | +2400 | +- | ++ |
| x86_64 | +64.0 | +512 | +3779 | +- | ++ |
| x86_64 | +96.0 | +192 | +- | +- | ++ |
| x86_64 | +96.0 | +192 | +3600 | +- | ++ |
| x86_64 | +96.0 | +384 | +- | +- | ++ |
| x86_64 | +96.0 | +384 | +3600 | +- | ++ |
| x86_64 | +96.0 | +384 | +5669 | +- | ++ |
| x86_64 | +96.0 | +672 | +- | +- | ++ |
| x86_64 | +96.0 | +672 | +3600 | +- | ++ |
| x86_64 | +96.0 | +672 | +5669 | +- | ++ |
| Architecture | +CPU cores | +Memory (GB) | +Local SSD (GB) | +GPU Memory (GB) | ++ |
|---|---|---|---|---|---|
| arm64 | +2.0 | +8 | +- | +- | ++ |
| arm64 | +2.0 | +16 | +- | +- | ++ |
| arm64 | +4.0 | +8 | +- | +- | ++ |
| arm64 | +4.0 | +16 | +- | +- | ++ |
| arm64 | +4.0 | +16 | +375 | +- | ++ |
| arm64 | +4.0 | +32 | +- | +- | ++ |
| arm64 | +4.0 | +32 | +375 | +- | ++ |
| arm64 | +8.0 | +16 | +- | +- | ++ |
| arm64 | +8.0 | +32 | +- | +- | ++ |
| arm64 | +8.0 | +32 | +750 | +- | ++ |
| arm64 | +8.0 | +64 | +- | +- | ++ |
| arm64 | +8.0 | +64 | +750 | +- | ++ |
| arm64 | +16.0 | +32 | +- | +- | ++ |
| arm64 | +16.0 | +64 | +- | +- | ++ |
| arm64 | +16.0 | +64 | +1500 | +- | ++ |
| arm64 | +16.0 | +128 | +- | +- | ++ |
| arm64 | +16.0 | +128 | +1500 | +- | ++ |
| arm64 | +32.0 | +64 | +- | +- | ++ |
| arm64 | +32.0 | +128 | +- | +- | ++ |
| arm64 | +32.0 | +128 | +2250 | +- | ++ |
| arm64 | +32.0 | +256 | +- | +- | ++ |
| arm64 | +32.0 | +256 | +2250 | +- | ++ |
| arm64 | +48.0 | +96 | +- | +- | ++ |
| arm64 | +48.0 | +192 | +- | +- | ++ |
| arm64 | +48.0 | +192 | +3750 | +- | ++ |
| arm64 | +48.0 | +384 | +- | +- | ++ |
| arm64 | +48.0 | +384 | +3750 | +- | ++ |
| arm64 | +64.0 | +128 | +- | +- | ++ |
| arm64 | +64.0 | +256 | +- | +- | ++ |
| arm64 | +64.0 | +256 | +5250 | +- | ++ |
| arm64 | +64.0 | +512 | +- | +- | ++ |
| arm64 | +64.0 | +512 | +5250 | +- | ++ |
| arm64 | +72.0 | +144 | +- | +- | ++ |
| arm64 | +72.0 | +288 | +- | +- | ++ |
| arm64 | +72.0 | +288 | +6000 | +- | ++ |
| arm64 | +72.0 | +576 | +- | +- | ++ |
| arm64 | +72.0 | +576 | +6000 | +- | ++ |
| x86_64 | +2.0 | +8 | +- | +- | ++ |
| x86_64 | +2.0 | +8 | +375 | +- | ++ |
| x86_64 | +2.0 | +8 | +750 | +- | ++ |
| x86_64 | +2.0 | +8 | +1500 | +- | ++ |
| x86_64 | +2.0 | +8 | +3000 | +- | ++ |
| x86_64 | +2.0 | +8 | +6000 | +- | ++ |
| x86_64 | +2.0 | +8 | +9000 | +- | ++ |
| x86_64 | +2.0 | +16 | +- | +- | ++ |
| x86_64 | +2.0 | +16 | +375 | +- | ++ |
| x86_64 | +2.0 | +16 | +750 | +- | ++ |
| x86_64 | +2.0 | +16 | +1500 | +- | ++ |
| x86_64 | +2.0 | +16 | +3000 | +- | ++ |
| x86_64 | +2.0 | +16 | +6000 | +- | ++ |
| x86_64 | +2.0 | +16 | +9000 | +- | ++ |
| x86_64 | +4.0 | +8 | +- | +- | ++ |
| x86_64 | +4.0 | +16 | +- | +- | ++ |
| x86_64 | +4.0 | +16 | +375 | +- | ++ |
| x86_64 | +4.0 | +16 | +750 | +- | ++ |
| x86_64 | +4.0 | +16 | +1500 | +- | ++ |
| x86_64 | +4.0 | +16 | +3000 | +- | ++ |
| x86_64 | +4.0 | +16 | +6000 | +- | ++ |
| x86_64 | +4.0 | +16 | +9000 | +- | ++ |
| x86_64 | +4.0 | +32 | +- | +- | ++ |
| x86_64 | +4.0 | +32 | +375 | +- | ++ |
| x86_64 | +4.0 | +32 | +750 | +- | ++ |
| x86_64 | +4.0 | +32 | +1500 | +- | ++ |
| x86_64 | +4.0 | +32 | +3000 | +- | ++ |
| x86_64 | +4.0 | +32 | +6000 | +- | ++ |
| x86_64 | +4.0 | +32 | +9000 | +- | ++ |
| x86_64 | +8.0 | +8 | +- | +- | ++ |
| x86_64 | +8.0 | +8 | +375 | +- | ++ |
| x86_64 | +8.0 | +8 | +750 | +- | ++ |
| x86_64 | +8.0 | +8 | +1500 | +- | ++ |
| x86_64 | +8.0 | +8 | +3000 | +- | ++ |
| x86_64 | +8.0 | +8 | +6000 | +- | ++ |
| x86_64 | +8.0 | +8 | +9000 | +- | ++ |
| x86_64 | +8.0 | +16 | +- | +- | ++ |
| x86_64 | +8.0 | +32 | +- | +- | ++ |
| x86_64 | +8.0 | +32 | +375 | +- | ++ |
| x86_64 | +8.0 | +32 | +750 | +- | ++ |
| x86_64 | +8.0 | +32 | +1500 | +- | ++ |
| x86_64 | +8.0 | +32 | +3000 | +- | ++ |
| x86_64 | +8.0 | +32 | +6000 | +- | ++ |
| x86_64 | +8.0 | +32 | +9000 | +- | ++ |
| x86_64 | +8.0 | +64 | +- | +- | ++ |
| x86_64 | +8.0 | +64 | +375 | +- | ++ |
| x86_64 | +8.0 | +64 | +750 | +- | ++ |
| x86_64 | +8.0 | +64 | +1500 | +- | ++ |
| x86_64 | +8.0 | +64 | +3000 | +- | ++ |
| x86_64 | +8.0 | +64 | +6000 | +- | ++ |
| x86_64 | +8.0 | +64 | +9000 | +- | ++ |
| x86_64 | +16.0 | +16 | +- | +- | ++ |
| x86_64 | +16.0 | +16 | +375 | +- | ++ |
| x86_64 | +16.0 | +16 | +750 | +- | ++ |
| x86_64 | +16.0 | +16 | +1500 | +- | ++ |
| x86_64 | +16.0 | +16 | +3000 | +- | ++ |
| x86_64 | +16.0 | +16 | +6000 | +- | ++ |
| x86_64 | +16.0 | +16 | +9000 | +- | ++ |
| x86_64 | +16.0 | +32 | +- | +- | ++ |
| x86_64 | +16.0 | +64 | +- | +- | ++ |
| x86_64 | +16.0 | +64 | +375 | +- | ++ |
| x86_64 | +16.0 | +64 | +750 | +- | ++ |
| x86_64 | +16.0 | +64 | +1500 | +- | ++ |
| x86_64 | +16.0 | +64 | +3000 | +- | ++ |
| x86_64 | +16.0 | +64 | +6000 | +- | ++ |
| x86_64 | +16.0 | +64 | +9000 | +- | ++ |
| x86_64 | +16.0 | +128 | +- | +- | ++ |
| x86_64 | +16.0 | +128 | +375 | +- | ++ |
| x86_64 | +16.0 | +128 | +750 | +- | ++ |
| x86_64 | +16.0 | +128 | +1500 | +- | ++ |
| x86_64 | +16.0 | +128 | +3000 | +- | ++ |
| x86_64 | +16.0 | +128 | +6000 | +- | ++ |
| x86_64 | +16.0 | +128 | +9000 | +- | ++ |
| x86_64 | +32.0 | +32 | +- | +- | ++ |
| x86_64 | +32.0 | +32 | +750 | +- | ++ |
| x86_64 | +32.0 | +32 | +1500 | +- | ++ |
| x86_64 | +32.0 | +32 | +3000 | +- | ++ |
| x86_64 | +32.0 | +32 | +6000 | +- | ++ |
| x86_64 | +32.0 | +32 | +9000 | +- | ++ |
| x86_64 | +32.0 | +64 | +- | +- | ++ |
| x86_64 | +32.0 | +128 | +- | +- | ++ |
| x86_64 | +32.0 | +128 | +750 | +- | ++ |
| x86_64 | +32.0 | +128 | +1500 | +- | ++ |
| x86_64 | +32.0 | +128 | +3000 | +- | ++ |
| x86_64 | +32.0 | +128 | +6000 | +- | ++ |
| x86_64 | +32.0 | +128 | +9000 | +- | ++ |
| x86_64 | +32.0 | +256 | +- | +- | ++ |
| x86_64 | +32.0 | +256 | +750 | +- | ++ |
| x86_64 | +32.0 | +256 | +1500 | +- | ++ |
| x86_64 | +32.0 | +256 | +3000 | +- | ++ |
| x86_64 | +32.0 | +256 | +6000 | +- | ++ |
| x86_64 | +32.0 | +256 | +9000 | +- | ++ |
| x86_64 | +48.0 | +48 | +- | +- | ++ |
| x86_64 | +48.0 | +48 | +750 | +- | ++ |
| x86_64 | +48.0 | +48 | +1500 | +- | ++ |
| x86_64 | +48.0 | +48 | +3000 | +- | ++ |
| x86_64 | +48.0 | +48 | +6000 | +- | ++ |
| x86_64 | +48.0 | +48 | +9000 | +- | ++ |
| x86_64 | +48.0 | +96 | +- | +- | ++ |
| x86_64 | +48.0 | +192 | +- | +- | ++ |
| x86_64 | +48.0 | +192 | +750 | +- | ++ |
| x86_64 | +48.0 | +192 | +1500 | +- | ++ |
| x86_64 | +48.0 | +192 | +3000 | +- | ++ |
| x86_64 | +48.0 | +192 | +6000 | +- | ++ |
| x86_64 | +48.0 | +192 | +9000 | +- | ++ |
| x86_64 | +48.0 | +384 | +- | +- | ++ |
| x86_64 | +48.0 | +384 | +750 | +- | ++ |
| x86_64 | +48.0 | +384 | +1500 | +- | ++ |
| x86_64 | +48.0 | +384 | +3000 | +- | ++ |
| x86_64 | +48.0 | +384 | +6000 | +- | ++ |
| x86_64 | +48.0 | +384 | +9000 | +- | ++ |
| x86_64 | +64.0 | +64 | +- | +- | ++ |
| x86_64 | +64.0 | +64 | +1500 | +- | ++ |
| x86_64 | +64.0 | +64 | +3000 | +- | ++ |
| x86_64 | +64.0 | +64 | +6000 | +- | ++ |
| x86_64 | +64.0 | +64 | +9000 | +- | ++ |
| x86_64 | +64.0 | +128 | +- | +- | ++ |
| x86_64 | +64.0 | +256 | +- | +- | ++ |
| x86_64 | +64.0 | +256 | +1500 | +- | ++ |
| x86_64 | +64.0 | +256 | +3000 | +- | ++ |
| x86_64 | +64.0 | +256 | +6000 | +- | ++ |
| x86_64 | +64.0 | +256 | +9000 | +- | ++ |
| x86_64 | +64.0 | +512 | +- | +- | ++ |
| x86_64 | +64.0 | +512 | +1500 | +- | ++ |
| x86_64 | +64.0 | +512 | +3000 | +- | ++ |
| x86_64 | +64.0 | +512 | +6000 | +- | ++ |
| x86_64 | +64.0 | +512 | +9000 | +- | ++ |
| x86_64 | +80.0 | +80 | +- | +- | ++ |
| x86_64 | +80.0 | +80 | +1500 | +- | ++ |
| x86_64 | +80.0 | +80 | +3000 | +- | ++ |
| x86_64 | +80.0 | +80 | +6000 | +- | ++ |
| x86_64 | +80.0 | +80 | +9000 | +- | ++ |
| x86_64 | +80.0 | +160 | +- | +- | ++ |
| x86_64 | +80.0 | +320 | +- | +- | ++ |
| x86_64 | +80.0 | +320 | +1500 | +- | ++ |
| x86_64 | +80.0 | +320 | +3000 | +- | ++ |
| x86_64 | +80.0 | +320 | +6000 | +- | ++ |
| x86_64 | +80.0 | +320 | +9000 | +- | ++ |
| x86_64 | +80.0 | +640 | +- | +- | ++ |
| x86_64 | +80.0 | +640 | +1500 | +- | ++ |
| x86_64 | +80.0 | +640 | +3000 | +- | ++ |
| x86_64 | +80.0 | +640 | +6000 | +- | ++ |
| x86_64 | +80.0 | +640 | +9000 | +- | ++ |
| x86_64 | +96.0 | +96 | +- | +- | ++ |
| x86_64 | +96.0 | +96 | +3000 | +- | ++ |
| x86_64 | +96.0 | +96 | +6000 | +- | ++ |
| x86_64 | +96.0 | +96 | +9000 | +- | ++ |
| x86_64 | +96.0 | +384 | +- | +- | ++ |
| x86_64 | +96.0 | +384 | +3000 | +- | ++ |
| x86_64 | +96.0 | +384 | +6000 | +- | ++ |
| x86_64 | +96.0 | +384 | +9000 | +- | ++ |
| x86_64 | +96.0 | +768 | +- | +- | ++ |
| x86_64 | +96.0 | +768 | +3000 | +- | ++ |
| x86_64 | +96.0 | +768 | +6000 | +- | ++ |
| x86_64 | +96.0 | +768 | +9000 | +- | ++ |
| x86_64 | +128.0 | +128 | +- | +- | ++ |
| x86_64 | +128.0 | +128 | +3000 | +- | ++ |
| x86_64 | +128.0 | +128 | +6000 | +- | ++ |
| x86_64 | +128.0 | +128 | +9000 | +- | ++ |
| x86_64 | +128.0 | +512 | +- | +- | ++ |
| x86_64 | +128.0 | +512 | +3000 | +- | ++ |
| x86_64 | +128.0 | +512 | +6000 | +- | ++ |
| x86_64 | +128.0 | +512 | +9000 | +- | ++ |
| x86_64 | +224.0 | +224 | +- | +- | ++ |
| x86_64 | +224.0 | +224 | +3000 | +- | ++ |
| x86_64 | +224.0 | +224 | +6000 | +- | ++ |
| x86_64 | +224.0 | +224 | +9000 | +- | ++ |
| x86_64 | +224.0 | +896 | +- | +- | ++ |
| x86_64 | +224.0 | +896 | +3000 | +- | ++ |
| x86_64 | +224.0 | +896 | +6000 | +- | ++ |
| x86_64 | +224.0 | +896 | +9000 | +- | ++ |
| + | + |
|---|---|
| **vmstat** | +*vmstat* can be used to figure out what kind of resources are used: • cpu usage split in user, system, idle, and io wait: system should be low( {`<10`})• swap in/out: should be zero. A maxed out system should have either maxed out disks or cpu ( {`idle == 0`}). If not, there might be lock contention or the system is bottlenecked by upstream services.Example: {`$ vmstat 1`}{`procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----`}{`r b swpd free buff cache si so bi bo in cs us sy id wa`}{`0 0 5628 3315460 304024 23008616 0 0 14 34 0 0 0 0 99 0`}{`1 0 5628 3298884 304024 23008640 0 0 0 396 33 4615 9 1 90 0`}{`0 0 5628 3316336 304028 23008644 0 0 0 0 15 4469 4 1 95 0`}{`0 0 5628 3316592 304028 23008644 0 0 0 0 24 4364 0 0 100 0`}{`0 0 5628 3316592 304028 23008644 0 0 0 2948 20 4305 0 0 100 0`}{`0 0 5628 3316468 304028 23008644 0 0 0 0 22 4259 0 0 100 0`}{`0 0 5628 3316468 304028 23008644 0 0 0 180 20 4279 0 0 100 0`}{`0 0 5628 3316468 304028 23008644 0 0 0 0 26 4349 0 0 100 0`}{`16 0 5628 3284236 304056 23008688 0 0 12 188 17 9196 38 2 60 0`}{`19 0 5628 3267020 304056 23008732 0 0 8 128 44 6408 99 1 0 0`}{`16 0 5628 3245472 304060 23008840 0 0 20 0 9 7191 99 1 0 0`}{`17 0 5628 3227784 304060 23008872 0 0 20 0 27 6420 99 1 0 0`} |
+
| **top** | +Use top to get a real-time view of which processes consume CPU and memory. | +
| **iostat** | +Use iostat to monitor disk IO. Note that the % busy is useless for SSD/NVMe storage disks, see Two traps in iostat: %util and svctm. | +
| Argument | +Type | +Description | +
|---|---|---|
| rate.id | +String | +The id of the client from rate limiting perspective. | +
| rate.cost | +Double | +The cost Double of this query. This is read after executing the query and so can be set by downstream searchers inspecting the result to allow differencing the cost of various queries. Default is 1. | +
| rate.quota | +Double | +The cost per second a particular id is allowed to consume in this system. | +
| rate.idDimension | +String | +The name of the rate-id dimension used when logging metrics. If this is not specified, the metric will be logged without dimensions. | +
| rate.dryRun | +Boolean | +Emit metrics on rejected requests but don't actually reject them. | +
| Thread pool | +Description | ++ |
|---|---|---|
| **master** | +Updates the document metastore, prepares tasks to the index and summary threads, and splits a write operation into a set of tasks to update individual attributes, executed by the threads in the field writer. | ++ |
| + | **Threads** | +1 | +
| + | **Instances** | +One instance per document database. | +
| + | **Metric prefix** | +*content.proton.documentdb.threading_service.master.* | +
| **index** | +Manages writing of index fields in the memory index. It splits a write operation into a set of tasks to update individual index fields, executed by the threads in the field writer. | ++ |
| + | **Threads** | +1 | +
| + | **Instances** | +One instance per document database. | +
| + | **Metric prefix** | +*content.proton.documentdb.threading_service.index.* | +
| **summary** | +Writes documents to the document store. | ++ |
| + | **Threads** | +1 | +
| + | **Instances** | +One instance per document database. | +
| + | **Metric prefix** | +*content.proton.documentdb.threading_service.summary.* | +
| **field writer** | +The threads in this thread pool are used to invert index fields, write changes to the memory index, and write changes to attributes. Index fields and attribute fields across all document databases are randomly assigned to one of the threads in this thread pool. A field that is costly to write or update might become the bottleneck during feeding. | ++ |
| + | **Threads** | +Many, controlled by feeding concurrency. | +
| + | **Instances** | +One instance shared between all document databases. | +
| + | **Metric prefix** | +*content.proton.executor.field_writer.* | +
| **shared** | +The threads in this thread pool are among other used to compress and de-compress documents in the document store, merge files as part of disk index fusion, and prepare for inserting a vector into a HNSW index. | ++ |
| + | **Threads** | +Many, controlled by feeding concurrency. | +
| + | **Instances** | +One instance shared between all document databases. | +
| + | **Metric prefix** | +*content.proton.executor.shared.* | +
| Attribute | +Description | +
|---|---|
| **weightedset** | +Memory-only operation when updating: read full set, update, write back. Make the update as inexpensive as possible using numeric types instead of strings, where possible Example: a weighted set of string with many (1000+) elements. Adding an element to the set means an enum store lookup/add and add/sort of the attribute multivalue map - details in attributes. Use a numeric type instead to speed this up - this has no string comparisons. | +
| **array/map of struct/map** | +Update to array of struct/map and map of struct/map requires a read from the document store and will reduce update rate - see #10892. | +
| **tensor** | +Updating tensor cell values is a memory-only operation: copy tensor, update, write back. For large tensors, this implicates reading and writing a large chunk of memory for single cell updates. | +
| + | + |
|---|---|
| **Metrics** | +Use metrics from content nodes and look at queues - queue wait time and queue size (all metrics in milliseconds):{`vds.filestor.averagequeuewait.sum`}{`vds.filestor.queuesize`}Check content node metrics across all nodes to see if there are any outliers. Also check latency metrics per operation type: {`vds.filestor.allthreads.put.latency`}{`vds.filestor.allthreads.update.latency`}{`vds.filestor.allthreads.remove.latency`} |
+
| **Bottlenecks** | +One of the threads used to handle write operations might become the bottleneck during feeding. Look at the *.utilization* metrics for all thread pools:{`content.proton.documentdb.threading_service.master.utilization`}{`content.proton.documentdb.threading_service.index.utilization`}{`content.proton.documentdb.threading_service.summary.utilization`}{`content.proton.executor.field_writer.utilization`}{`content.proton.executor.shared.utilization`} If utilization is high for field writer or shared, adjust feeding concurrency to allow more CPU cores to be used for feeding. For the field writer also look at the *.saturation* metric: {`content.proton.executor.field_writer.saturation`}If this is close to 1.0 and higher than *.utilization* it indicates that one of its worker threads is a bottleneck. The reason can be that this particular thread is handling a large index or attribute field that is naturally expensive to write and update. Use the custom component state API to find which index and attribute fields are assigned to which thread (identified by *executor_id*), and look at the detailed statistics of the field writer to find which thread is the actual bottleneck: {`state/v1/custom/component/documentdb/mydoctype/subdb/ready/index`}{`state/v1/custom/component/documentdb/mydoctype/subdb/ready/attributewriter`}`state/v1/custom/component/threadpools/field_writer |
+
| **Failure rates** | +Inspect these metrics for failures during load testing:{`vds.distributor.updates.latency`}{`vds.distributor.updates.ok`}{`vds.distributor.updates.failures.total`}{`vds.distributor.puts.latency`}{`vds.distributor.puts.ok`}{`vds.distributor.puts.failures.total`}{`vds.distributor.removes.latency`}{`vds.distributor.removes.ok`}{`vds.distributor.removes.failures.total`} |
+
| **Blocked feeding** | +This metric should be 0 - refer to feed block:{`content.proton.resource_usage.feeding_blocked`} |
+
| **Concurrent mutations** | +Multiple clients updating the same document concurrently will stall writes:{`vds.distributor.updates.failures.concurrent_mutations`}Mutating client operations towards a given document ID are sequenced on the distributors. If an operation is already active towards a document, a subsequently arriving one will be bounced back to the client with a transient failure code. Usually this happens when users send feed from multiple clients concurrently without synchronisation. Note that feed operations sent by a single client are sequenced client-side, so this should not be observed with a single client only. Bounced operations are never sent on to the backends and should not cause elevated latencies there, although the client will observe higher latencies due to automatic retries with back-off. |
+
| **Wrong distribution** | +{`vds.distributor.updates.failures.wrongdistributor`}Indicates that clients keep sending to the wrong distributor. Normally this happens infrequently (but is *does* happen on client startup or distributor state transitions), as clients update and cache all state required to route directly to the correct distributor (Vespa uses a deterministic CRUSH-based algorithmic distribution). Some potential reasons for this: 1. Clients are being constantly re-created with no cached state. 2. The system is in some kind of flux where the underlying state keeps changing constantly. 3. The client distribution policy has received so many errors that it throws away its cached state to start with a clean slate to e.g. avoid the case where it only has cached information for the bad side of a network partition. 4. The system has somehow failed to converge to a shared cluster state, causing parts of the cluster to have a different idea of the correct state than others. |
+
| **Cluster out of sync** | +*update_puts/gets* indicate "two-phase" updates:{`vds.distributor.update_puts.latency`}{`vds.distributor.update_puts.ok`}{`vds.distributor.update_gets.latency`}{`vds.distributor.update_gets.ok`} {`vds.distributor.update_gets.failures.total`}{`vds.distributor.update_gets.failures.notfound`}If replicas are out of sync, updates cannot be applied directly on the replica nodes as they risk ending up with diverging state. In this case, Vespa performs an explicit read-consolidate-write (write repair) operation on the distributors. This is usually a lot slower than the regular update path because it doesn't happen in parallel. It also happens in the write-path of other operations, so risks blocking these if the updates are expensive in terms of CPU. Replicas being out of sync is by definition not the expected steady state of the system. For example, replica divergence can happen if one or more replica nodes are unable to process or persist operations. Track (pending) merges: {`vds.idealstate.buckets`}{`vds.idealstate.merge_bucket.pending`}{`vds.idealstate.merge_bucket.done_ok`}{`vds.idealstate.merge_bucket.done_failed`} |
+
| + | + |
|---|---|
| redundancy | +Defines the total number of copies of each piece of data the cluster will store and maintain to avoid data loss. Example: with a redundancy of 2, the system tolerates 1 node failure before any further node failures may cause data to become unavailable. | +
| searchable-copies | +Configures how many of the copies (as configured with *redundancy*) to be indexed (*ready*) at any time. Configuring *searchable-copies* to be less than *redundancy* saves resources (memory, disk, cpu), as not all copies are indexed (*ready*). In case of node failure, the remaining nodes needs to index the *not ready* documents which belonged to the failed node. In this transition period, the search has reduced search coverage. | +
| + | + |
|---|---|
| **static query work** | +Portion of the query work on a content node that does not depend on the number of documents indexed on the node. This is an administrative overhead caused by system design and abstractions, e.g. number of memory allocations per query term. Typically, a large query tree means higher static work, and this work cannot be parallelized over multiple processors, threads or nodes. The static query work portion is described in step 1 to 4 and step 9 in the detailed life of a query explanation above. | +
| **dynamic query work** | +Portion of the query work on a content node that depends on the number of documents indexed and active on the node. This portion of the work scales mostly linearly with the number of matched documents. The dynamic query work can be parallelized over multiple processors and nodes. Referenced later as *DQW*. The *DQW* also depends on the phase two protocol summary fill where the actual contents of the global best documents is fetched from the content nodes which produced the hit in the first protocol phase. | +
| **Total query work** | +The total query work is given as the dynamic query work (*DQW*) + static query work (*SQW*). | +
| + | + |
|---|---|
| **Add nodes in a flat distribution** | +When DQW/TQW is large (close to 1.0), throughput QPS can be scaled by just adding more content nodes in a system using flat distribution. This will reduce the number of documents per node, and thus reduce the *DQW* per node. | +
| **Add groups using grouped distribution** | +When DQW/TQW is low, one can no longer just add more content nodes to scale throughput and must instead use a grouped distribution to scale throughput. | +
| Airport code | +City | +Location | +
|---|---|---|
| SFO | +San Francisco, USA | +{`{ "lat": 37.618806, "lng": -122.375416 }`} |
+
| LAX | +Los Angeles, USA | +{`{ "lat": 33.942496, "lng": -118.408048 }`} |
+
| JFK | +New York, USA | +{`{ "lat": 40.639928, "lng": -73.778692 }`} |
+
| LHR | +London, UK | +{`{ "lat": 51.477500, "lng": -0.461388 }`} |
+
| SYD | +Sydney, Australia | +{`{ "lat": -33.946110, "lng": 151.177222 }`} |
+
| TRD | +Trondheim, Norway | +{`{ "lat": 63.457556, "lng": 10.924250 }`} |
+
| OSL | +Oslo, Norway | +{`{ "lat": 60.193917, "lng": 11.100361 }`} |
+
| GRU | +São Paulo, Brazil | +{`{ "lat": -23.435555, "lng": -46.473055 }`} |
+
| GIG | +Rio de Janeiro, Brazil | +{`{ "lat": -22.809999, "lng": -43.250555 }`} |
+
| BLR | +Bangalore, India | +{`{ "lat": 13.198867, "lng": 77.705472 }`} |
+
| FCO | +Rome, Italy | +{`{ "lat": 41.804475, "lng": 12.250797 }`} |
+
| NRT | +Tokyo, Japan | +{`{ "lat": 35.765278, "lng": 140.385556 }`} |
+
| PEK | +Beijing, China | +{`{ "lat": 40.073, "lng": 116.598 }`} |
+
| CPT | +Cape Town, South Africa | +{`{ "lat": -33.971368, "lng": 18.604292 }`} |
+
| ACC | +Accra, Ghana | +{`{ "lat": 5.605186, "lng": -0.166785 }`} |
+
| TBU | +Nuku'alofa, Tonga | +{`{ "lat": -21.237999, "lng": -175.137166 }`} |
+
| Date | +Price | +Tax | +Item | +Customer | +Is paid | +
|---|---|---|---|---|---|
| 2006-09-06 09:00:00 | +$1 000 | +0.24 | +Intake valve | +Smith | +true | +
| 2006-09-07 10:00:00 | +$1 000 | +0.12 | +Rocker arm | +Smith | +false | +
| 2006-09-07 11:00:00 | +$2 000 | +0.24 | +Spring | +Smith | +true | +
| 2006-09-08 12:00:00 | +$3 000 | +0.12 | +Valve cover | +Jones | +false | +
| 2006-09-08 10:00:00 | +$5 000 | +0.24 | +Intake port | +Jones | +true | +
| 2006-09-08 11:00:00 | +$8 000 | +0.12 | +Head | +Brown | +false | +
| 2006-09-09 12:00:00 | +$1 300 | +0.24 | +Coolant | +Smith | +true | +
| 2006-09-09 10:00:00 | +$2 100 | +0.12 | +Engine block | +Jones | +false | +
| 2006-09-09 11:00:00 | +$3 400 | +0.24 | +Oil pan | +Brown | +true | +
| 2006-09-09 12:00:00 | +$5 500 | +0.12 | +Oil sump | +Smith | +false | +
| 2006-09-10 10:00:00 | +$8 900 | +0.24 | +Camshaft | +Jones | +true | +
| 2006-09-10 11:00:00 | +$1 440 | +0.12 | +Exhaust valve | +Brown | +false | +
| 2006-09-10 12:00:00 | +$2 330 | +0.24 | +Rocker arm | +Brown | +true | +
| 2006-09-10 10:00:00 | +$3 770 | +0.12 | +Spring | +Brown | +false | +
| 2006-09-10 11:00:00 | +$6 100 | +0.24 | +Spark plug | +Smith | +true | +
| 2006-09-11 12:00:00 | +$9 870 | +0.12 | +Exhaust port | +Jones | +false | +
| 2006-09-11 10:00:00 | +$1 597 | +0.24 | +Piston | +Brown | +true | +
| 2006-09-11 11:00:00 | +$2 584 | +0.12 | +Connection rod | +Smith | +false | +
| 2006-09-11 12:00:00 | +$4 181 | +0.24 | +Rod bearing | +Jones | +true | +
| 2006-09-11 13:00:00 | +$6 765 | +0.12 | +Crankshaft | +Jones | +false | +
| GroupId | +Sum(price) | +
|---|---|
| Brown | +$20 537 | +
| Jones | +$39 816 | +
| Smith | +$19 484 | +
| GroupId | +sum(price) | ++ | + | + | + | + |
|---|---|---|---|---|---|---|
| Brown | +$20 537 | ++ | + | + | + | + |
| + | Date | +Price | +Tax | +Item | +Customer | ++ |
| + | 2006-09-08 11:00 | +$8 000 | +0.12 | +Head | +Brown | ++ |
| + | GroupId | +Sum(price) | ++ | + | + | + |
| + | 2006-09-08 | +$8 000 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-08 11:00 | +$8 000 | +0.12 | +Head | +Brown | +
| + | 2006-09-09 | +$3 400 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-09 11:00 | +$3 400 | +0.12 | +Oil pan | +Brown | +
| + | 2006-09-10 | +$7 540 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-10 10:00 | +$3 770 | +0.12 | +Spring | +Brown | +
| + | + | 2006-09-10 12:00 | +$2 330 | +0.24 | +Rocker arm | +Brown | +
| + | + | 2006-09-10 11:00 | +$1 440 | +0.12 | +Exhaust valve | +Brown | +
| + | 2006-09-11 | +$1 597 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-11 10:00 | +$1 597 | +0.24 | +Piston | +Brown | +
| Jones | +$39 816 | ++ | + | + | + | + |
| + | Date | +Price | +Tax | +Item | +Customer | ++ |
| + | 2006-09-11 12:00 | +$9 870 | +0.12 | +Exhaust port | +Jones | ++ |
| + | GroupId | +Sum(price) | ++ | + | + | + |
| + | 2006-09-08 | +$8 000 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-08 10:00 | +$5 000 | +0.24 | +Intake port | +Jones | +
| + | + | 2006-09-08 12:00 | +$3 000 | +0.12 | +Valve cover | +Jones | +
| + | 2006-09-09 | +$2 100 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-09 10:00 | +$2 100 | +0,12 | +Engine block | +Jones | +
| + | 2006-09-10 | +$8 900 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-10 10:00 | +$8 900 | +0.24 | +Camshaft | +Jones | +
| + | 2006-09-11 | +$20 816 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-11 12:00 | +$9 870 | +0.12 | +Exhaust port | +Jones | +
| + | + | 2006-09-11 13:00 | +$6 765 | +0.12 | +Crankshaft | +Jones | +
| + | + | 2006-09-11 12:00 | +$4 181 | +0.24 | +Rod bearing | +Jones | +
| Smith | +$19 484 | ++ | + | + | + | + |
| + | Date | +Price | +Tax | +Item | +Customer | ++ |
| + | 2006-09-10 11:00 | +$6 100 | +0.24 | +Spark plug | +Smith | ++ |
| + | GroupId | +Sum(price) | ++ | + | + | + |
| + | 2006-09-06 | +$1 000 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-06 09:00 | +$1 000 | +0.24 | +Intake valve | +Smith | +
| + | 2006-09-07 | +$3 000 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-07 11:00 | +$2 000 | +0.24 | +Spring | +Smith | +
| + | + | 2006-09-07 10:00 | +$1 000 | +0.12 | +Rocker arm | +Smith | +
| + | 2006-09-09 | +$6 800 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-09 12:00 | +$5 500 | +0.12 | +Oil sump | +Smith | +
| + | + | 2006-09-09 12:00 | +$1 300 | +0.24 | +Coolant | +Smith | +
| + | 2006-09-10 | +$6 100 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-10 11:00 | +$6 100 | +0.24 | +Spark plug | +Smith | +
| + | 2006-09-11 | +$2 584 | ++ | + | + | + |
| + | + | Date | +Price | +Tax | +Item | +Customer | +
| + | + | 2006-09-11 11:00 | +$2 584 | +0.12 | +Connection rod | +Smith | +
| Transformation | +Type | +
|---|---|
| Hardwired...To → hardwire to | +Tokenization - split terms on non-characters, here "..." | +
| Head → head | +Lowercasing | +
| für → fur | +Normalizing | +
| dreams → dream | +Stemming | +
| <Columns cols={3}><Frame>!DandelionFlower.jpg</Frame></Columns> | +[0.560, 0.001, 0.223, ...] | +
|---|---|
| <Columns cols={2}><Frame>!Dandelion bu Anna of the North</Frame></Columns> | +[0.0, 0.011, 0.0, ...] | +
| "Taraxacum is a large genus of flowering plants in the family Asteraceae, which consists of species commonly known as dandelions." | +[0.002, 0.001, 0.411, ...] | +
| dandelions | +[0.002, 0.021, 0.355, ...] | +
| Int8 | +8 bits, 1 byte per dimension | +
|---|---|
| bfloat16 | +16 bits, 2 bytes per dimension | +
| float | +32 bits, 4 bytes per dimension | +
| double | +64 bits, 8 bytes per dimension | +
| input vector | +binarized floats | +pack_bits (to INT8) | +
|---|---|---|
| [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] | +[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] | +-1 | +
| [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | +[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | +0 | +
| [-1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0] | +[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | +0 | +
| [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | +[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | +-128 | +
| [2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0] | +[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0] | +-127 | +
| document embedding | +binarized floats | +pack_bits (to INT8) | +
|---|---|---|
| [-1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0] | +[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | +0 | +
| **query embedding** | +**binarized floats** | +**to INT8** | +
| [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0] | +[1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0] | +-119 | +
| Parameter (Vespa) | +Parameter (OpenAI) | +Description | +
|---|---|---|
{`maxTokens`} |
+ {`max_completion_tokens`} |
+ Maximum number of tokens that can be generated in the chat completion. | +
{`temperature`} |
+ {`temperature`} |
+ Number between 0 and 2. Higher values like 0.8 make output more random, while lower values like 0.2 make it more focused and deterministic. | +
{`topP`} |
+ {`top_p`} |
+ An alternative to temperature sampling. Model considers tokens with top_p probability mass (0-1). Value of 0.1 means only tokens comprising top 10% probability are considered. | +
{`seed`} |
+ {`seed`} |
+ If specified, the system will attempt to sample deterministically, so repeated requests with the same seed should return similar results. Determinism is not guaranteed. | +
{`npredict`} |
+ {`n`} |
+ How many chat completion choices to generate for each input message. Note that you will be charged based on the number of generated tokens across all choices. | +
{`frequencypenalty`} |
+ {`frequency_penalty`} |
+ Number between -2.0 and 2.0. Positive values penalize new tokens based on their frequency in the text so far, decreasing the likelihood of repetition. Negative values encourage repetition. | +
{`presencepenalty`} |
+ {`presence_penalty`} |
+ Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics. Negative values encourage repeating content from the prompt. | +
| + | + |
|---|---|
| GTE (General Text Embedding) model trained from ModernBERT-base. | ++ |
| Model id | +{`alibaba-gte-modernbert`} |
+
| Tensor definition | +{`tensor |
+
| Matryoshka dimensions | +{`x[768]`}, {`x[256]`} |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/Alibaba-NLP/gte-modernbert-base @ 3ab3f8c | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| INT8 quantized variant of alibaba-gte-modernbert. Offers faster inference with minimal accuracy loss. | ++ |
| Model id | +{`alibaba-gte-modernbert-int8`} |
+
| Tensor definition | +{`tensor |
+
| Matryoshka dimensions | +{`x[768]`}, {`x[256]`} |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/Alibaba-NLP/gte-modernbert-base @ e7f32e3 | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| The base model of the _E5_ family. | ++ |
| Model id | +{`e5-base-v2`} |
+
| Tensor definition | +{`tensor or {`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +MIT | +
| Source | +https://huggingface.co/intfloat/e5-base-v2 @ 121b23b | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| The largest model of the _E5_ family, at time of writing, this is the best performing embedding model on the MTEB benchmark. | ++ |
| Model id | +{`e5-large-v2`} |
+
| Tensor definition | +{`tensor or {`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +MIT | +
| Source | +https://huggingface.co/intfloat/e5-large-v2 | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| The smallest and most cost-efficient model from the _E5_ family. | ++ |
| Model id | +{`e5-small-v2`} |
+
| Tensor definition | +{`tensor or {`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +MIT | +
| Source | +https://huggingface.co/intfloat/e5-small-v2 | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| Trained from ModernBERT-large on the Nomic Embed datasets, bringing the new advances of ModernBERT to embeddings. | ++ |
| Model id | +{`lightonai-modernbert-large`} |
+
| Tensor definition | +{`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/lightonai/modernbert-embed-large @ b3a781f | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| INT8 quantized variant of lightonai-modernbert-large. Offers faster inference with minimal accuracy loss. | ++ |
| Model id | +{`lightonai-modernbert-large-int8`} |
+
| Tensor definition | +{`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/lightonai/modernbert-embed-large @ 95a19bf | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| The multilingual model of the _E5_ family. Use this model for multilingual queries and documents. | ++ |
| Model id | +{`multilingual-e5-base`} |
+
| Tensor definition | +{`tensor or {`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +MIT | +
| Source | +https://huggingface.co/intfloat/multilingual-e5-base | +
| Language | +Multilingual | +
| Component declaration | + |
+
| + | + |
|---|---|
| Trained from ModernBERT-base on the Nomic Embed datasets, bringing the new advances of ModernBERT to embeddings. | ++ |
| Model id | +{`nomic-ai-modernbert`} |
+
| Tensor definition | +{`tensor |
+
| Matryoshka dimensions | +{`x[768]`}, {`x[256]`} |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/nomic-ai/modernbert-embed-base @ 92168cb | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| INT8 quantized variant of nomic-ai-modernbert. Offers faster inference with minimal accuracy loss. | ++ |
| Model id | +{`nomic-ai-modernbert-int8`} |
+
| Tensor definition | +{`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/nomic-ai/modernbert-embed-base @ d556a88 | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| Embedding model based on snowflake-arctic-embed-m-v2.0. | ++ |
| Model id | +{`snowflake-arctic-embed-m-v2.0`} |
+
| Tensor definition | +{`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0 @ 95c2741 | +
| Language | +Multilingual | +
| Component declaration | + |
+
| + | + |
|---|---|
| INT8 quantized variant of snowflake-arctic-embed-m-v2.0. Offers faster inference with minimal accuracy loss. | ++ |
| Model id | +{`snowflake-arctic-embed-m-v2.0-int8`} |
+
| Tensor definition | +{`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0 @ 95c2741 | +
| Language | +Multilingual | +
| Component declaration | + |
+
| + | + |
|---|---|
| Embedding model based on voyage-4-nano-ONNX. | ++ |
| Model id | +{`voyage-4-nano`} |
+
| Tensor definition | +{`tensor |
+
| Matryoshka dimensions | +{`x[2048]`}, {`x[1024]`}, {`x[512]`}, {`x[256]`} |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/thomasht86/voyage-4-nano-ONNX @ fcf290d | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| INT8 quantized variant of voyage-4-nano. Offers faster inference with minimal accuracy loss. | ++ |
| Model id | +{`voyage-4-nano-int8`} |
+
| Tensor definition | +{`tensor |
+
| Matryoshka dimensions | +{`x[2048]`}, {`x[1024]`}, {`x[512]`}, {`x[256]`} |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/thomasht86/voyage-4-nano-ONNX @ fcf290d | +
| Language | +English | +
| Component declaration | + |
+
| + | + |
|---|---|
| Model-id | +minilm-l6-v2 | +
| Tensor definition | +{`tensor or {`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 | +
| Language | +English | +
| + | + |
|---|---|
| Model-id | +mpnet-base-v2 | +
| Tensor definition | +{`tensor or {`tensor |
+
| distance-metric | +{`angular`} |
+
| License | +apache-2.0 | +
| Source | +https://huggingface.co/sentence-transformers/all-mpnet-base-v2 | +
| Language | +English | +
| + | + |
|---|---|
| A vocabulary text (_vocab.txt_) file on the format expected by WordPiece: A text token per line. | +|
| Model-id | +bert-base-uncased | +
| License | +apache-2.0 | +
| Source | +https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 | +
| + | + |
|---|---|
A _tokenizer.json_ configuration file on the format expected by HF tokenizer. This tokenizer configuration can be used with {`e5-base-v2`}, {`e5-small-v2`} and {`e5-large-v2`}. |
+ |
| Model-id | +e5-base-v2-vocab | +
| License | +MIT | +
| Source | +https://huggingface.co/intfloat/e5-base-v2 | +
| Language | +English | +
| + | + |
|---|---|
A _tokenizer.json_ configuration file on the format expected by HF tokenizer. This tokenizer configuration can be used with {`multilingual-e5-base-vocab`}. |
+ |
| Model-id | +multilingual-e5-base-vocab | +
| License | +MIT | +
| Source | +https://huggingface.co/intfloat/multilingual-e5-base | +
| Language | +Multilingual | +
| + | + |
|---|---|
| This significance model was generated from English Wikipedia dump data from 2024-08-01. Available in Vespa as of version 8.426.8. | +|
| Model-id | +significance-en-wikipedia-v1 | +
| License | +Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) License. | +
| Source | +https://data.vespa-cloud.com/significance_models/significance-en-wikipedia-v1.json.zst | +
| Language | +English | +
| Feature input | +Description | +
|---|---|
| Weight | +Set query term weight. Example: {`... where (title contains ({weight:200}"heads") AND title contains "tails")`} specifies that {`heads`} is twice as important for the final rank score than {`tails`} (the default weight is 100). The term weight is used in several text scoring features, including fieldMatch(*name*).weight and nativeRank. Note that the term weight is not applicable for all text scoring features, for example bm25 does not use the term weight. Configure static field weights in the schema. |
+
| Significance | +Significance is an indication of how rare a term is in the corpus of the language, used by a number of text matching rank features. This can be set explicitly for each term in the query, or by calling item.setSignificance() in a Searcher. With *indexed search*, default significance values are calculated automatically during indexing. However, unless the indexed corpus is representative of the word frequencies in the user's language, relevance can be improved by passing significances derived from a representative corpus. Relative significance is accessible in ranking through the fieldMatch(*name*).significance feature. Weight and significance are also averaged into fieldMatch(*name*).importance for convenience. *Streaming search* does not compute term significance, queries should pass this with the query terms. Read more. |
+
| Connectedness | +Signify the degree of connection between adjacent terms in the query - set query term connectivity to another term. For example, the query {`new york newspaper`} should have a higher connectedness between the terms "new" and "york" than between "york" and "newspaper" to rank documents higher if they contain "new york" as a phrase. Term connectedness is taken into account by fieldMatch(*name*).proximity, which is also an important contribution to fieldMatch(*name*). Connectedness is a normalized value which is 0.1 by default. It must be set by a custom Searcher, looking up connectivity information from somewhere - there is no query syntax for it. |
+
| document | +query | +value | +
|---|---|---|
| /en/jdisc/container-components.html | ++ | 0 | +
| + | /en/overview.html | +0 | +
| /en/page-templates.html | +/en/page-templates.html | +1 | +
| /en/query-profiles.html | +/en/query-profiles.html | +1 | +
| Parameter | +Type | +Description | +
|---|---|---|
| **format.tensors** | +String | +Controls how tensors are rendered in the result. Value: {`short`} Description: **Default**. Render the tensor value in a JSON object having two keys, "type" containing the value, and "cells"/"blocks"/"values" (depending on the type) containing the tensor content. Render the tensor content in the type-appropriate short form. Value: {`long`}Description: Render the tensor value in a JSON object having two keys, "type" containing the value, and "cells" containing the tensor content. Render the tensor content in the general verbose form. Value: {`short-value`} Description: Render the tensor content directly as a JSON value. Render the tensor content in the type-appropriate short form. Value: {`long-value`} Description: Render the tensor content directly as a JSON value. Render the tensor content in the general verbose form. Value: {`string`} Description: Render the tensor content as a string on the appropriate literal short form. Value: {`string-long`}Description: Render the tensor content as a string on the general literal form. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| intraop-threads | +optional | +number | +max(1, CPU count / 4) | +The number of threads available for running operations with multithreaded implementations. | +
| interop-threads | +optional | +number | +{`max(1, CPU count / 4)`} if execution mode {`parallel`} |
+ The number of threads available for running multiple operations in parallel. This is only applicable for {`parallel`} execution mode. |
+
| execution-mode | +optional | +string | +sequential | +Controls how the operators of a graph are executed, either {`sequential`} or {`parallel`}. |
+
| gpu-device | +optional | +number | ++ | Set the GPU device number to use for computation, starting at 0, i.e. if your GPU is {`/dev/nvidia0`} set this to 0. This must be an Nvidia CUDA-enabled GPU. |
+
| HTTP request | +application/v2/tenant operation | +Description | +
|---|---|---|
| GET | +List tenant information. | ++ |
| + | List tenants | +{`/application/v2/tenant/`} Example response: {`[`} {`"default"`} {`]`} |
+
| + | Get tenant | +{`/application/v2/tenant/default`} Example response: {`{`} {`"message": "Tenant 'default' exists."`} {`}`} |
+
| PUT | +Create a new tenant. | ++ |
| + | Create tenant | +{`/application/v2/tenant/default`} Response: A message with the name of the tenant created - example: {`{`} {`"message" : "Tenant default created."`} {`}`} **Note:** This operation is asynchronous, it will eventually propagate to all config servers. |
+
| DELETE | +Delete a tenant. | ++ |
| + | Delete tenant | +{`/application/v2/tenant/default`} Response: A message with the deleted tenant: {`{`} {`"message" : "Tenant default deleted."`} {`}`} **Note:** This operation is asynchronous, it will eventually propagate to all config servers. |
+
| HTTP request | +cluster/v2 operation | +Description | +
|---|---|---|
| GET | +List cluster and nodes. Get cluster, node or disk states. | ++ |
| + | List content clusters | +{`/cluster/v2/`} |
+
| + | Get cluster state and list service types within cluster | +{`/cluster/v2/ |
+
| + | List nodes per service type for cluster | +{`/cluster/v2/ |
+
| + | Get node state | +{`/cluster/v2/ |
+
| PUT | +Set node state | ++ |
| + | Set node user state | +{`/cluster/v2/ |
+
| State | +Description | +
|---|---|
{`Up`} |
+ The node is up and available to keep buckets and serve requests. | +
{`Down`} |
+ The node is not available, and can not be used. | +
{`Stopping`} |
+ This node is stopping and is expected to be down soon. This state is typically only exposed to the cluster controller to tell why the node stopped. The cluster controller will expose the node as down or in maintenance mode for the rest of the cluster. This state is thus not seen by the distribution algorithm. | +
{`Maintenance`} |
+ This node is temporarily unavailable. The node is available for bucket placement, so redundancy is lower. Using this mode, new replicas of the documents stored on this node will not be created, allowing the node to be down with less of a performance impact on the rest of the cluster. This mode is typically used to mask a down state during controlled node restarts, or by an administrator that need to do some short maintenance work, like upgrading software or restart the node. | +
{`Retired`} |
+ A retired node is available and serves requests. This state is used to remove nodes while keeping redundancy. Buckets are moved to other nodes (with low priority), until empty. Special considerations apply when using grouped distribution as buckets are not necessarily removed. | +
| Type | +Spec | +Description | +
|---|---|---|
| cluster | +*{`* |
+ The name given to a content cluster in a Vespa application. | +
| description | +*.** | +Description can contain anything that is valid JSON. However, as the information is presented in various interfaces, some which may present reasons for all the states in a cluster or similar, keeping it short and to the point makes it easier to fit the information neatly into a table and get a better cluster overview. | +
| group-spec | +*{`* (. *{`* )* |
+ The hierarchical group assignment of a given content node. This is a dot separated list of identifiers given in the application services.xml configuration. | +
| node | +[0-9]+ | +The index or distribution key identifying a given node within the context of a content cluster and a service type. | +
| service-type | +(distributor|storage) | +The type of the service to look at state for, within the context of a given content cluster. | +
| state-disk | +(up|down) | +One of the valid disk states. | +
| state-unit | +up | stopping | down | +The cluster controller fetches states from all nodes, called *unit states*. States reported from the nodes are either {`up`} or {`stopping`}. If the node can not be reached, a {`down`} state is assumed. This means, the cluster controller detects failed nodes. The subsequent *generated states* will have nodes in {`down`}, and the ideal state algorithm will redistribute buckets of documents. |
+
| state-user | +up | down | maintenance | retired | +Use tools for user state management. Retire a node from a cluster - use {`retired`} to move buckets to other nodes Short-lived maintenance work - use {`maintenance`} to avoid merging buckets to other nodes Fail a bad node. The cluster controller or an operator can set a node {`down`} |
+
| state-generated | +up | down | maintenance | retired | +The cluster controller generates the cluster state from the {`unit`} and {`user`} states, over time. The generated state is called the *cluster state*. |
+
| Parameter | +Type | +Description | +
|---|---|---|
| recursive | +number | +Number of levels, or {`true`} for all levels. Examples: Use {`recursive=1`} for a node request to also see all datause {`recursive=2`} to see all the node data within each service type In recursive mode, you will see the same output as found in the spec below. However, where there is a {`{ "link" : " element, this element will be replaced by the content of that request, given a recursive value of one less than the request above. |
+
| Term | +Description | +
|---|---|
| Parameters | +**Parameter:** recursive; **Default:** false; **Description:** If true, include each config id in the model which produces the config, and list only the links to the config payload. If false, include the first level of the config ids in the listing of new list URLs, as explained above. | +
| Request body | +None | +
| Response | +A list response includes two arrays: List-links to descend one level down in the config id hierarchy, named {`children`}. Config payload links for the current (top) level, named {`configs`}. |
+
| Error Response | +N/A | +
| Term | +Description | +
|---|---|
| Parameters | +Same as above. | +
| Request body | +None | +
| Response | +List the configs in the model with the given namespace and name. List semantics as above. | +
| Error Response | +404 if the given namespace.name is not known to the config model. | +
| Term | +Description | +
|---|---|
| Parameters | +Same as above. | +
| Request body | +None | +
| Response | +List the configs in the model with the given namespace and name, and for which the given config id segment is a prefix. | +
| Error Response | +404 if the given namespace.name is not known to the config model. 404 if the given config id is not in the model. | +
| Term | +Description | +
|---|---|
| Parameters | +None | +
| Request body | +None | +
| Response | +Returns the config payload of the given {`namespace.name/config/id`}, formatted as JSON. |
+
| Error Response | +Same as above. | +
| Term | +Description | +
|---|---|
| session-id | +The session-id used in this API is generated by the server and is required for all operations after creating a session. The session-id is valid if it is an active session, or it was created before session lifetime has expired, the default value being 1 hour. | +
| path | +An application file path in a request URL or parameter refers to a relative path in the application package. A path ending with "/" refers to a directory. | +
| Term | +Description | +
|---|---|
| Parameters | ++ |
| Request body | +**Required:** Yes; **Content:** A compressed application package (with gzip or zip compression); **Note:** Set {`Content-Type`} HTTP header to {`application/x-gzip`} or {`application/zip`} . |
+
| Response | +See active. | +
| Term | +Description | +
|---|---|
| Parameters | +**Name:** from; **Default:** N/A; **Description:** Use when you want to create a new session based on an active application. The value supplied should be a URL to an active application. | +
| Request body | +**Required:** Yes, unless {`from`} parameter is used; **Content:** A compressed application package (with gzip or zip compression); **Note:** It is required to set the {`Content-Type`} HTTP header to {`application/x-gzip`} or {`application/zip`} , unless the {`from`} parameter is used. |
+
| Response | +The response contains: A session-id to the application that was created. A prepared URL for preparing the application. | +
| Term | +Description | +
|---|---|
| Parameters | +None | +
| Request body | +If path is a directory, none. If path is a file, the contents of the file. | +
| Response | +None Any errors or warnings from writing the file/creating the directory. | +
| Term | +Description | +
|---|---|
| Parameters | +**Name:** recursive; **Default:** false; **Description:** If *true* , directory content will be listed recursively. **Name:** return; **Default:** content; **Description:** If set to content and path refers to a file, the content will be returned. If set to content and path refers to a directory, the files and subdirectories in the directory will be listed. If set to status and path refers to a file, the file status and hash will be returned. If set to status and path refers to a directory, a list of file/subdirectory statuses and hashes will be returned. |
+
| Request body | +None. | +
| Response | +If path is a directory: a JSON array of URLs to the files and subdirectories of that directory. If path is a file: the contents of the file. If status parameter is set, the status and hash will be returned. | +
| Term | +Description | +
|---|---|
| Parameters | +None | +
| Request body | +None | +
| Response | +Any errors or warnings from deleting the resource. | +
| Term | +Description | +
|---|---|
| Parameters | +**Parameter:** applicationName; **Default:** N/A; **Description:** Name of the application to be deployed **Parameter:** environment; **Default:** default; **Description:** Environment where application should be deployed **Parameter:** region; **Default:** default; **Description:** Region where application should be deployed **Parameter:** instance; **Default:** default; **Description:** Name of application instance **Parameter:** debug; **Default:** false; **Description:** If true, include stack trace in response if prepare fails. **Parameter:** timeout; **Default:** 360 seconds; **Description:** Timeout in seconds to wait for session to be prepared. |
+
| Request body | +None | +
| Response | +Returns a session-id and a link to activate the session. Log with any errors or warnings from preparing the application. An activate URL for activating the application with this session-id, if there were no errors. A list of actions (possibly empty) that must be performed in order to apply some config changes between the current active application and this next prepared application. These actions are organized into three categories; *restart*, *reindex*, and *refeed*: *Restart* actions are done after the application has been activated and are handled by restarting all listed services. See schemas for details. *Reindex* actions are special refeed actions that Vespa handles automatically, if the reindex endpoint below is used. *Refeed* actions require several steps to handle. See schemas for details. | +
| Term | +Description | +
|---|---|
| Parameters | +**Parameter:** timeout; **Default:** 60 seconds; **Description:** Timeout in seconds to wait for session to be activated (when several config servers are used, they might need to sync before activate can be done). | +
| Request body | +None | +
| Response | +Returns a session-id, a message and a URL to the activated application. session-id Message | +
| Term | +Description | +
|---|---|
| Parameters | +None | +
| Request body | +None | +
| Response | +Returns a list of applications Array of active applications | +
| Term | +Description | +
|---|---|
| Parameters | +None | +
| Request body | +None | +
| Response | +Returns information about the application specified. config generation | +
| Term | +Description | +
|---|---|
| Parameters | +N/A | +
| Request body | +N/A | +
| Response | +JSON detailing current reindexing status for the application, with all its clusters and document types. Status for each content cluster in the application, by name: Status of each document type in the cluster, by name: Last time reindexing was triggered for this document type.Current status of reindexing.Optional start time of reindexing.Optional end time of reindexing.Optional progress of reindexing, from 0 to 1.Pseudo-speed of reindexing. | +
| Term | +Description | +
|---|---|
| Parameters | +**Name:** clusterId; **Description:** A comma-separated list of content clusters to limit reindexing to. All clusters are reindexed if this is not present. **Name:** documentType; **Description:** A comma-separated list of document types to limit reindexing to. All document types are reindexed if this is not present. **Name:** indexedOnly; **Description:** Boolean: whether to mark reindexing ready only for document types with indexing mode *index* and at least one field with the indexing statement {`index`} . Default is {`false`} .**Name:** speed; **Description:** Number (0–10], default 1: Indexing pseudo speed - balance speed vs. resource use. Example: speed=0.1 |
+
| Request body | +N/A | +
| Response | +A human-readable message indicating what reindexing was marked as ready. | +
| Term | +Description | +
|---|---|
| Parameters | +**Name:** clusterId; **Description:** A comma-separated list of content clusters to limit the changes to. Reindexing for all clusters are modified if this is not present. **Name:** documentType; **Description:** A comma-separated list of document types to limit the changes to. Reindexing for all document types are modified if this is not present. **Name:** indexedOnly; **Description:** Boolean: whether to modify reindexing only for document types with indexing mode *index* and at least one field with the indexing statement {`index`} . Default is {`false`} .**Name:** speed; **Description:** Number [0–10], required: Indexing pseudo speed - balance speed vs. resource use. Example: speed=0.1 |
+
| Request body | +N/A | +
| Response | +A human-readable message indicating what reindexing was modified. | +
| Term | +Description | +
|---|---|
| Parameters | +None | +
| Request body | +None | +
| Response | +Returns a message stating if the operation was successful or not | +
| Term | +Description | +
|---|---|
| Parameters | +None | +
| Request body | +None | +
| Response | +Returns a message with tenant and application details. | +
| HTTP request | +document/v1 operation | +Description | +
|---|---|---|
| GET | +*Get* a document by ID or *Visit* a set of documents by selection. | ++ |
| + | Get | +Get a document: {`/document/v1/ {`/document/v1/ {`/document/v1/ Optional parameters: cluster fieldSet timeout tracelevel |
+
| + | Visit | +Iterate over and get all documents, or a selection of documents, in chunks, using continuation tokens to track progress. Visits are a linear scan over the documents in the cluster. {`/document/v1/`} It is possible to specify namespace and document type with the visit path: {`/document/v1/ Documents can be grouped to limit accesses to a subset. A group is defined by a numeric ID or string — see id scheme . {`/document/v1/ {`/document/v1/ Mandatory parameters: cluster - Visits can only retrieve data from *one* content cluster, so {`cluster`} **must** be specified for requests at the root {`/document/v1/`} level, or when there is ambiguity. This is required even if the application has only one content cluster. Optional parameters: bucketSpace - Parent documents are global and in the {`global`} bucket space. By default, visit will visit non-global documents in the {`default`} bucket space, unless document type is indicated, and is a global document type. concurrency - Use to configure backend parallelism for each visit HTTP request. continuation fieldSet selection sliceId slices - Split visiting of the document corpus across more than one HTTP request—thus allowing the concurrent use of more HTTP containers—use the {`slices`} and {`sliceId`} parameters. stream - It's recommended enabling streamed HTTP responses, with the stream parameter, as this reduces memory consumption and reduces HTTP overhead. timeout tracelevel wantedDocumentCount fromTimestamp toTimestamp includeRemoves Optional request headers: Accept - specify the desired response format. |
+
| POST | +*Put* a given document, by ID, or *Copy* a set of documents by selection from one content cluster to another. | ++ |
| + | Copy | +Write documents visited in source cluster to the destinationCluster in the same application. A selection is mandatory — typically the document type. Supported paths (see visit above for semantics): {`/document/v1/`} {`/document/v1/ {`/document/v1/ {`/document/v1/ Mandatory parameters: cluster destinationCluster selection Optional parameters: bucketSpace continuation timeChunk timeout tracelevel |
+
| PUT | +*Update* a document with the given partial update, by ID, or *Update where* the given selection is true. | ++ |
| + | Update | +Update a document with the partial update contained in the request body in the document update JSON format . {`/document/v1/ Optional parameters: condition - use for conditional writes create - use to create empty documents when updating non-existent ones. route timeout tracelevel |
+
| + | Update where | +Update visited documents in cluster with the partial update contained in the request body in the document update JSON format. Supported paths (see visit above for semantics): {`/document/v1/ {`/document/v1/ {`/document/v1/ Mandatory parameters: cluster selection Optional parameters: bucketSpace - See visit, {`default`} or {`global`} bucket space continuation stream timeChunk timeout tracelevel |
+
| DELETE | +*Remove* a document, by ID, or *Remove where* the given selection is true. | ++ |
| + | Remove | +Remove a document. {`/document/v1/ Optional parameters: condition route timeout tracelevel |
+
| + | Delete where | +Delete visited documents from cluster. Supported paths (see visit above for semantics): {`/document/v1/`} {`/document/v1/ {`/document/v1/ {`/document/v1/ Mandatory parameters: cluster selection Optional parameters: bucketSpace - See visit, {`default`} or {`global`} bucket space continuation stream timeChunk timeout tracelevel |
+
| HTTP request | +metrics/v1 operation | +Description | +
|---|---|---|
| GET | ++ | + |
| + | Node metrics | +{`/metrics/v1/values`} See monitoring for examples. |
+
| Parameter | +Type | +Description | +
|---|---|---|
| consumer | +String | +Specify response consumer, i.e. set of metrics. An unknown / empty value will return the {`default`} metric set. Built-in: {`default`} - see DefaultMetrics. {`vespa`} - see VespaMetricSet. |
+
| Element | +Parent | +Type | +Description | +
|---|---|---|---|
| services | ++ | Object | +Root for /metrics/v1/values. Contains service objects. | +
| name | +services | +String | +Service name. | +
| timestamp | +services | +Number | +EPOCH in seconds - time of metrics fetch from service. | +
| status | +services | +Object | +Status from metrics fetch. | +
| code | +status | +String | +The status for each service is one of: {`up`} {`down`} {`unknown`} {`unknown`} is used if the service seems to be alive, but does not report metrics. |
+
| description | +status | +String | +Textual status. | +
| metrics | +services | +Array | +Array of metric objects. | +
| values | +metrics | +Object | +Set of metric-name/value pairs. | +
| dimensions | +metrics | +Object | +Set of metric dimension-name/value pairs. | +
| HTTP request | +metrics/v2 operation | +Description | +
|---|---|---|
| GET | ++ | + |
| + | Application metrics | +{`/metrics/v2/values`} See monitoring for examples. |
+
| Parameter | +Type | +Description | +
|---|---|---|
| consumer | +String | +Specify response consumer, i.e. set of metrics. See metrics/v1 for details. | +
| HTTP request | +prometheus/v1 operation | +Description | +
|---|---|---|
| GET | ++ | + |
| + | Node metrics | +{`/prometheus/v1/values`} See monitoring for examples. |
+
| Parameter | +Type | +Description | +
|---|---|---|
| consumer | +String | +Specify response consumer, i.e. set of metrics. An unknown / empty value will return the {`default`} metric set. Built-in (note: case-sensitive): {`default`} {`Vespa`} |
+
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| yql | ++ | String | ++ | See the YQL query guide for examples, and the reference for details. | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| hits | +count | +Number | +10 | +A positive integer, including 0. The maximum number of hits to return from the result set. {`hits`} is capped at {`maxHits`}, default 400. {`maxHits`} can be set in a query profile. Number of hits can also be set in YQL. |
+
| offset | +start | +Number | +0 | +To implement pagination: The number of hits to skip when returning the result. A positive integer, including 0. {`offset`} is capped at {`maxOffset`}, default 1000. {`maxOffset`} can be set in a query profile. Offset can also be set in YQL. |
+
| queryProfile | ++ | String | +{`default`} |
+ A query profile id with format {`name:version`}, where version can be omitted or partially specified, e.g. {`myprofile:2.1`}. A query profile has default properties for a query. The default query profile is named *default*. |
+
| groupingSessionCache | ++ | Boolean | +true | +Set to true to enable grouping session cache. See the grouping reference for details. | +
| searchChain | ++ | String | +{`default`} |
+ A search chain id with format {`name:version`}, where version can be omitted or partially specified, e.g. {`mychain:2.1.3`}. The search chain initially invoked when processing the query. This search chain may invoke other chains. |
+
| timeout | ++ | String | +0.5s | +Positive floating point number with an optional unit. Default unit is seconds (s), valid unit strings are e.g. *ms* and *s*. To set a timeout of one minute, the argument could be set to *60 s*. Space between the number and the unit is optional. It specifies the overall timeout of the query execution and can be defined in a query profile. Different classes of queries can then easily have a different latency budget/timeout using different profiles. At timeout, the hits generated thus far are returned, refer to ranking.softtimeout.enable for details on HTTP status codes and response elements. Refer to the Query API guide for more details on timeout handling. | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| model.defaultIndex | +default-index | +String | +{`default`} |
+ An index name. The field which is searched for query terms which doesn't explicitly specify an index. Also see the defaultIndex query annotation. | +
| model.encoding | +encoding | +String | +{`utf-8`} |
+ Encoding names or aliases defined in the IANA character sets. Sets the encoding to use when returning a result. The query is always encoded as UTF-8, independently of how the result will be encoded. The encodings {`big5`}, {`euc-jp`}, {`euc-kr`}, {`gb2312`}, {`iso-2022-jp`} and {`shift-jis`} also influences how tokenization is done in the absence of an explicit language setting. |
+
| model.filter | +filter | +String | ++ | A filter string in the Simple Query Language. Sets a filter to be combined with the model.queryString. Typical use of a filter is to add machine generated or preferences based filter terms to the user query. Terms which are passed in the filter are not bolded. The filter is parsed the same way as a query of type {`any`}, the full syntax is available. The positive terms (preceded by +) and phrases act as AND filters, the negative terms (preceded by -) act as NOT filters, while the unprefixed terms will be used to RANK the results. Unless the query has no positive terms, the filter will only restrict and influence ranking of the result set, never cause more matches than the query. The model.queryString must be present for this to have any effect. To add filters to the YQL string, use query profiles. See example. |
+
| model.locale | +locale | +String | ++ | A language tag from RFC 5646. Sets the locale and language to use when parsing queries from a language tag, such as {`en-US`}. This attribute should always be set when it is known. If this parameter is not set, it will be guessed from the query and encoding, and default to english if it cannot be guessed. |
+
| model.language | +lang, language | +String | ++ | A language tag from RFC 5646, but allowing underscore instead of dash as separator character. A legacy alternative to locale. When this value is accessed, underscores will be replaced by dashes in the returned value. Also see the language query term annotation. | +
| model.queryString | +query | +String | ++ | A query string in the Simple Query Language. It is combined with model.filter. See the userQuery operator for how to combine with YQL. Can also be used without YQL. | +
| model.restrict | +restrict | +String | ++ | A comma-delimited list of document type (schema) names, defaulting to all schemas if not set. See federation. Use model.sources to restrict to content cluster names or other source names. | +
| model.searchPath | +searchpath | +String | ++ | Specification of which content nodes a query should be sent to. This is useful for debugging/monitoring and when using Rank phase statistics. Note that in a content cluster with flat distribution (i.e. no <group> element in *services.xml*), there is 1 implicit group. If not set, defaults to all nodes in one group, selected by load balancing. {`searchpath::ELEMENT [';' ELEMENT]*`} {`ELEMENT::NODE ['/' GROUP]`} {`NODE::EXP [',' EXP]*`} {`EXP::NUM | RANGE`} {`GROUP::NUM | '*'`} {`RANGE::'['NUM ',' NUM ' >'`} Examples: {`7/3`} = node 7, group 3. {`7/`} = node 7, any group. {`*/0`} = all nodes in group 0 {`7,1,9/0`} = nodes 1,7 and 9, group 0. {`1,[3,9>/0`} = nodes 1,3,4,5,6,7,8, group 0. |
+
| model.sources | +search, sources | +String | ++ | A comma-separated list of content cluster names or other source names, defaulting to all sources/clusters if not set. The names of the sources to query, e.g., one or more content clusters and/or federated sources - see federation. Use model.restrict to only search a subset of the schemas in a cluster. | +
| model.type | +type | +String | +{`weakAnd`} |
+ Sets all the model.type parameters (composite, tokenization, and syntax) specifying how to parse a model.queryString parameter at once, according to the given table: {`all`} → composite {`and`}, tokenization {`internal`}, syntax {`simple`} {`any`} → composite {`or`}, tokenization {`internal`}, syntax {`simple`} {`linguistics`} → composite {`weakAnd`}, tokenization {`linguistics`}, syntax {`none`} {`phrase`} → composite {`phrase`}, tokenization {`internal`}, syntax {`none`} {`tokenize`} → composite {`weakAnd`}, tokenization {`internal`}, syntax {`none`} {`weakAnd`} → composite {`weakAnd`}, tokenization {`internal`}, syntax {`simple`} {`web`} → composite {`and`}, tokenization {`internal`}, syntax {`web`} {`yql`} → composite {`and`}, tokenization {`internal`}, syntax {`yql`} Also see YQL grammar. |
+
| model.type.composite | ++ | String | +{`Determined by model.type`} |
+ Sets the Vespa query composite type that will collect parsed terms of the query by default. and Create an AndItem which only matches if *all* terms are present. nearCreate a NearItem which matches if all the terms appear near each other (gap of 1 by default). oNear — Create an ONearItem which matches if all the terms appear near each other (gap of 1 by default), in the given order. or — Create an OrItem which matches if *any* of the terms are present. phrase — Create a PhraseItem which matches if all the terms are present in the given order with no gaps. weakAnd — Create a WeakAndItem which has the semantics of {`or`} with performance approaching {`and`} . |
+
| model.type.tokenization | ++ | String | +{`Determined by model.type`} |
+ Sets the tokenizer used to split the query string into tokens. internal Use the tokenizer built into the query parser. linguisticsPass the full query string as-is to the linguistics component for tokenization, exactly as on the indexing side, and collect any text and numeric token returned as-is, with no further stemming or normalization even when specified in the schema. This is only supported in conjunction with the {`none`} syntax option. |
+
| model.type.syntax | ++ | String | +{`Determined by model.type`} |
+ Sets the syntax used to interpret the query string. Options: {`none`}: No syntax: Disregard any non-searchable terms {`simple`}: Use the simple query language suitable for end users. {`web`}: Like the simple query language , but '+' in front of a term means "search for this term as-is", and 'a OR b' (capital OR) means match either a or b. {`yql`}: Parse as a YQL query . |
+
| model.type.profile | ++ | String | +{`*(null)*`} |
+ Overrides the linguistics profile assigned to the field(s) searched. The linguistics profile is used to choose the processing done in the linguistics component. | +
| model.type.isYqlDefault | ++ | Boolean | +{`false`} |
+ Whether the model.type settings should be used as the default settings for terms in YQL queries. With this parameter turned on, the model.type settings become the default "grammar" settings in userQuery, and with tokenization set to {`linguistics`} this will also cause "contains" terms to not undergo stemming, normalization and lowercasing as separate operations, as using this mode delegates all token processing to a single pass through the lingustics module. |
+
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| ranking.location | ++ | String | ++ | See Geo search. Point (two-dimensional location) to use as base for location ranking. **Important:** Deprecated in favor of adding a geoLocation item to the query tree. Use inside a rank operator if it should be used only for ranking). | +
| ranking.features .*featurename* |
+ input . *featurename* , rankfeature . *featurename* |
+ String | ++ | Set a query rank feature input to a value. The key must be a query feature - {`query(anyname)`}, and the value must be a double, string (to be hashed to a double), or a tensor matching the declared input type on tensor literal form - see the tensor user guide. Examples: {`input.query(userageDouble)=42.1`} {`input.query(stringToBeHashed)=abcd`} {`input.query(myIndexedTensor)=[1.0, 2.0, 3.0]`} {`input.query(myMappedTensor)={"Tablet Keyboard Cases": 0.8, "Keyboards":0.3}`} |
+
| ranking.listFeatures | +rankfeatures | +Boolean | +false | +Set to true to request *all* rank-features to be calculated and returned. The rank features will be returned in the summary field *rankfeatures*. This option is typically used for MLR training, should not to be used for production. | +
| ranking.profile | +ranking | +String | +{`default`} |
+ Sets rank profile to use for assigning rank scores for documents. The {`default`} rank profile will be used for backends which does not have the given rank profile. |
+
| ranking.properties .*propertyname* |
+ rankproperty . *propertyname* |
+ String | ++ | Set a rank property that is passed to, and used by a feature executor for this query. Example: {`query=foo&ranking.properties.dotProduct.X={a:1,b:2}`} |
+
| ranking.softtimeout .enable |
+ + | Boolean | +true | +By default, the hits available are returned on timeout. To return no hits at timeout instead, set {`ranking.softtimeout.enable=false`}. Softtimeout uses {`ranking.softtimeout.factor`} of the timeout, default 70%. The rest of the time budget is spent on later ranking phases. The factor is adaptive, per rank profile - the factor is adjusted based on remaining time after all ranking phases, unless overridden in the query using {`ranking.softtimeout.factor`}. A timeout element is returned in the query response at timeout. Example: query with 500ms timeout, use 300ms in first-phase ranking: {`&ranking.softtimeout.enable=true The {`ranking.softtimeout`} settings controls what the content nodes should do in the case where the latency budget has almost been used (timeout times a factor). Return the documents recalled and ranked with the first phase function within the time used, or simply don't produce a result: With soft timeout disabled, the Vespa container will return a 504 timeout without any results. When enabled, it will return the documents matched and ranked up until the timeout was reached, with a 200 OK response along with the reason the result set was degraded. The container might respond with a timeout error with HTTP response code 504 even with soft timeout enabled if the timeout is set so low that the query does not make it to the content nodes, or the container does not have any time left after input and query processing to dispatch the query to the content nodes. Read more about soft timeout in coverage degradation. |
+
| ranking.softtimeout .factor |
+ + | Number | +0.7 | +See ranking.softtimeout.enable. | +
| ranking.sorting | +sorting | +String | ++ | A valid sort specification. Fields you want to sort on must be stored as document attributes in the index structure by adding attribute to the indexing statement. | +
| ranking.significance.useModel | ++ | Boolean | +false | +Enables or disables the use of significance models specified in service.xml. Overrides use-model set in the rank profile. | +
| ranking.freshness | ++ | String | ++ | Sets the time which will be used as *now* during execution. {`[integer]`}, an absolute time in seconds since epoch, or {`now-[number]`}, to use a time [integer] seconds into the past, or {`now`} to use the current time. |
+
| ranking.queryCache | ++ | Boolean | +false | +Turns query cache on or off. Query is a two-phase process. If the query cache is on, the query is stored on the content nodes between the first and second phase, saving network bandwidth and also query setup time, at the expense of using more memory. It only affects the protocol phase two, see caches in Vespa. It does not cache the result, it just saves resources by not forwarding the query twice (one for the first protocol phase which is find the best k documents from all nodes, to the second phase which is to fill summary data and potentially ranking features listed in summary-features in the rank profile). The summary-features are re-calculated but this setting avoids sending the query down once more. There is little downside of using it, and it can save resources and latency in cases where the query tree and query ranking features (e.g. tensors used in ranking) are large. As this is a protocol optimization, it also works with changing filter, it's not cached cross independent queries, it's just saving having to send the same query twice. | +
| ranking.secondPhase.totalRerankCount | ++ | Number | ++ | Specifies the number of hits that should be ranked in the second ranking phase in total over the queried content nodes. Overrides the total-rerank-count set in the rank profile. Setting to 0 disables second phase reranking. | +
| ranking.secondPhase.rerankCount | ++ | Number | ++ | Specifies the number of hits that should be ranked in the second phase *per node*. Prefer using totalRerankCount over this. | +
| ranking.totalKeepRankCount | ++ | Number | ++ | Specifies the number of hits for which the rank score should be kept after first phase ranking in total over the nodes participating in the query. Overrides the total-keep-rank-count set in the rank profile. | +
| ranking.keepRankCount | ++ | Number | ++ | Specifies the number of hits for which the rank score should be kept after first phase ranking on each node. Overrides the keep-rank-count set in the rank profile. Prefer total-keep-rank-count over this. | +
| ranking.rankScoreDropLimit | ++ | Number | ++ | Minimum rankscore for a document to be considered a hit. Overrides the rank-score-drop-limit set in the rank profile. | +
| ranking.secondPhase.rankScoreDropLimit | ++ | Number | ++ | Minimum rank score for a document to be considered a hit after second phase reranking or rescoring. Overrides the second phase rank-score-drop-limit set in the rank profile. | +
| ranking.globalPhase.rerankCount | ++ | Number | ++ | Specifies the number of hits that should be re-ranked in the global ranking phase. Overrides the rerank-count set in the rank profile. Setting to 0 disables the global phase reranking. | +
| ranking.globalPhase.rankScoreDropLimit | ++ | Number | ++ | Minimum rank score for a document to be considered a hit after global phase reranking or rescoring. Overrides the global phase rank-score-drop-limit set in the rank profile. | +
| ranking.elementGap.*fieldName* | ++ | Integer | ++ | Set or overrides element-gap configured for a given *fieldName* in the rank profile. Note: Can be the integer "0" to consider elements to be adjacent, or the string "infinity" to signal that words in different elements never are considered "close". | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| ranking.matching .numThreadsPerSearch |
+ + | integer | ++ | Rank profile equivalent: num-threads-per-search Overrides the global persearch threads to a **lower** value. | +
| ranking.matching .minHitsPerThread |
+ + | integer | ++ | Rank profile equivalent: min-hits-per-thread After estimating the number of hits for a query, this number is used to decide how many search threads to use. | +
| ranking.matching .numSearchPartitions |
+ + | integer | ++ | Rank profile equivalent: num-search-partitions Number of logical partitions the corpus on a content node is divided in. A partition is the smallest unit a search thread will handle. | +
| ranking.matching .termwiseLimit |
+ + | double [0.0, 1.0] | ++ | Rank profile equivalent: termwise-limit If estimated number of hits > corpus * termwise-limit, document candidates are pruned with a TAAT evaluation for query terms not needed for ranking. | +
| ranking.matching .postFilterThreshold |
+ + | double [0.0, 1.0] | +1.0 | +Rank profile equivalent: post-filter-threshold Threshold value deciding if a query with an approximate nearestNeighbor operator combined with filters is evaluated using post-filtering. | +
| ranking.matching .approximateThreshold |
+ + | double [0.0, 1.0] | +0.02 | +Rank profile equivalent: approximate-threshold Threshold value deciding if a query with an approximate nearestNeighbor operator combined with filters is evaluated by searching for approximate or exact nearest neighbors. | +
| ranking.matching .filterFirstThreshold |
+ + | double [0.0, 1.0] | +0.2 | +Rank profile equivalent: filter-first-threshold Threshold value deciding if the filter is checked before computing a distance (*filter-first heuristic*) while searching the HNSW graph for approximate neighbors with filtering. | +
| ranking.matching .filterFirstExploration |
+ + | double [0.0, 1.0] | +0.01 | +Rank profile equivalent: filter-first-exploration Value specifying how aggressively the filter-first heuristic searches the HNSW graph for approximate neighbors with filtering. | +
| ranking.matching .explorationSlack |
+ + | double [0.0, 1.0] | +0.0 | +Rank profile equivalent: exploration-slack Value specifying slack to delay the termination of the search of the HNSW graph for nearest neighbors with or without filtering. | +
| ranking.matching .targetHitsMaxAdjustmentFactor |
+ + | double [1.0, inf] | ++ | Rank profile equivalent: target-hits-max-adjustment-factor Value used to control the auto-adjustment of totalTargetHits used when evaluating an approximate nearestNeighbor operator with post-filtering. | +
| ranking.matching .filterThreshold |
+ + | double [0.0, 1.0] | ++ | Rank profile equivalent: filter-threshold Threshold value (in the range [0, 1]) deciding when matching in *index* fields should be treated as filters. This happens for query terms with estimated hit ratios that are above the *filterThreshold*. | +
| ranking.matching.weakand .stopwordLimit |
+ + | double [0.0, 1.0] | ++ | Rank profile equivalent: weakand stopword-limit A number in the range [0, 1] representing the maximum normalized document frequency a query term can have in the corpus before it's considered a stopword and dropped entirely from being a part of the {`weakAnd`} evaluation. |
+
| ranking.matching.weakand .adjustTarget |
+ + | double [0.0, 1.0] | ++ | Rank profile equivalent: weakand adjust-target A number in the range [0, 1] representing normalized document frequency. Used to derive a per-query document score threshold, where documents scoring lower than the threshold will not be considered as potential hits from the {`weakAnd`} operator. |
+
| ranking.matching.weakand .allowDropAll |
+ + | boolean | +false | +Rank profile equivalent: weakand allow-drop-all A boolean value that, if set to {`true`}, will allow the {`weakAnd`} operator to drop *all* terms from the query if all terms are considered stopwords (i.e. by setting {`weakAnd.stopwordLimit`}). Typically used in conjunction with nearestNeighbor or other operators to ensure that the query will return hits even when all terms are considered stopwords. |
+
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| ranking.matchPhase .attribute |
+ + | string | ++ | Rank profile equivalent: match-phase: attribute The attribute used to limit matches by if more than maxHits hits will be produced. | +
| ranking.matchPhase .totalMaxHits |
+ + | long | ++ | The max number of hits that should be generated in total over the content nodes during the match phase. Setting the value to {`0`} disables match phase early termination. Rank profile equivalent: match-phase: total-max-hits |
+
| ranking.matchPhase .maxHits |
+ + | long | ++ | The max number of hits that should be generated on eache content nodes during the match phase. Prefer using totalMaxHits over this. Rank profile equivalent: match-phase: max-hits | +
| ranking.matchPhase .ascending |
+ + | boolean | ++ | Rank profile equivalent: match-phase: order Whether to keep the documents having the highest (false) or lowest (true) values of the match phase attribute. | +
| ranking.matchPhase .diversity.attribute |
+ + | string | ++ | Rank profile equivalent: diversity: attribute The attribute to use when deciding diversity. | +
| ranking.matchPhase .diversity.minGroups |
+ + | long | ++ | Rank profile equivalent: diversity: min-groups The minimum number of groups that should be returned from the match phase grouped by the diversity attribute. | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| dispatch.topKProbability | ++ | double | ++ | Probability to use when computing how many hits to fetch from each partition when merging and creating the final result set. See services for details. Default: none. | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| presentation.bolding | +bolding | +Boolean | +true | +Whether to bold query terms in schema fields defined with bolding: on or summary: dynamic. | +
| presentation.format | +format | +String | +{`default`} |
+ {`Value`}: Description {`*No value* or [default](/en/reference/querying/default-result-format)`}: The default, builtin JSON format {`[json](/en/reference/querying/default-result-format)`}: Builtin JSON format {``}cbor{``}: Builtin CBOR format. Binary encoding, responses are smaller and faster to render than JSON, especially for numeric data. Semantically equivalent to JSON. Cannot be used with {`jsoncallback`} (JSONP). Requires Vespa 8.623.5 or later. {``}xml{``}: Builtin XML format. **Important:** See deprecations. {`[page](/en/reference/querying/page-result-format)`}: XML format which is suitable for use with page templates . **Important:** See deprecations. {`*Any other value*`}: A custom result renderer supplied by the application The response format can also be selected via the HTTP {`Accept`} header. If the Accept header specifies {`application/cbor`} with higher priority than {`application/json`}, CBOR will be used. The {`format`} query parameter overrides the Accept header. |
+
| presentation.summary | +summary | +String | ++ | The name of the summary class used to select fields in results. Default: The default summary class of the schema. | +
| presentation.template | ++ | String | ++ | The id of a deployed page template to use for this result. This should be used with the page result format. | +
| presentation.timing | ++ | Boolean | +false | +Whether a result renderer should try to add optional timing information to the rendered page - see the result reference. | +
| presentation.format.tensors | ++ | String | +{`short`} |
+ Controls how tensors are rendered in the result. {`Value`}: Description {``}short{``}: Render the tensor value in an object having two keys, "type" containing the value, and "cells"/"blocks"/"values" ( depending on the type ) containing the tensor content. Render the tensor content in the type-appropriate short form . {``}long{``}: Render the tensor value in an object having two keys, "type" containing the value, and "cells" containing the tensor content. Render the tensor content in the general verbose form . {``}short-value{``}: Render the tensor content directly. Render the tensor content in the type-appropriate short form . {``}long-value{``}: Render the tensor content directly. Render the tensor content in the general verbose form . {``}hex{``}: Use {`short`} form, and render dense values hex encoded . {``}hex-value{``}: Use {`short-value`} form, and render dense values hex encoded . |
+
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| select | ++ | String | ++ | Requests specific multi-level result set statistics and/or hit groups to be returned in the result. Fields you want to retrieve statistics or hit groups for must be stored as document attributes in the index structure by adding attribute to the indexing statement. Default is no grouping. See the grouping guide for examples. | +
| collapsefield | ++ | String | ++ | Comma-separated list of field names, that should only appear uniquely in a result. Hits with values in these fields which are already present in a higher-ranked hit will be filtered out. Read more in result diversity to compare this with other options. Default is no field collapsing. | +
| collapsesize | ++ | Number | +1 | +The number of hits to keep in each collapsed bucket - used for all collapsefields. | +
| collapsesize.*fieldname* | ++ | Number | +1 | +The number of hits to keep in each collapsed bucket - used for the specified field. This value takes precedence over the value specified in {`collapsesize`}. |
+
| collapse.summary | ++ | String | ++ | A valid name of a document summary class. Use this summary class to fetch the fields used for collapsing. Default: Use default summary or attributes. | +
| grouping.defaultMaxGroups | ++ | Number | +10 | +Positive integer or {`-1`} to disable. The default number of groups to return when max is not specified. |
+
| grouping.defaultMaxHits | ++ | Number | +10 | +Positive integer or {`-1`} to disable. The default number of hits to return when max is not specified. |
+
| grouping.globalMaxGroups | ++ | Number | +10000 | +Positive integer or {`-1`} to disable. A cost limit for grouping queries. Any query that may exceed this threshold will be preemptively failed by the container. The limit is defined as the total number of groups and document summaries a query may produce. A query that does not have an implicit or explicit {`max`} defined for all levels will always fail if limit is enabled. This parameter can only be overridden in a query profile. See the grouping guide for practical examples. |
+
| grouping.defaultPrecisionFactor | ++ | Decimal number |
+ 2.0 | +The default precision scale factor when precision is not specified. The final precision value is calculated by multiplying the effective {`max`} value with the scale factor. |
+
| timezone | ++ | String | +{`utc`} |
+ Specifies a timezone that will be used to offset all {`time`} related expressions in grouping. See Java's definition for valid timezones. See the grouping guide for examples. |
+
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| streaming.groupname | ++ | A string | ++ | Sets the group (specified by g=<groupname>) of the documents to stream through. | +
| streaming.selection | ++ | A document selection | ++ | Restricts streaming search using a selection expression instead of a group id. If the selection is on the form {`id.group == "foo" or id.group == "bar" or id.group == ...`} this will only stream documents in those groups, which is efficient for a small number of groups. If any other selection is used, this will stream through *all* groups, which is very costly. |
+
| streaming.maxbucketspervisitor | ++ | An integer Positive infinity If set, limit backend bucket concurrency to the specified number of buckets. Can be used to explicitly control resource usage for extremely large streaming search locations. This is an expert option. | ++ | + |
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| trace.profile | +profile | +Boolean | +false | +True to produce a structured trace for performance analysis. Returns a structured trace for performance analysis. This is a shorthand to set various other parameters to the suitable values to generate a performance trace. | +
| trace.level | +tracelevel | +Number | +0 | +A positive number. Default is no tracing. Collect trace information for debugging when running a query. Higher numbers give progressively more detail on query transformations, searcher execution and content node(s) query execution. See query tracing for details and examples. Tracing is subject to change at any time, the below is a guide: {`Level`}: Description {`1`}: Basic tracing in container {`2`}: Basic tracing, more details {`3`}: Basic tracing, even more details {`4`}: Include timing info from content nodes {`5`}: Even more timing info from content nodes {`6`}: Include the query execution plan (blueprint) {`7`}: Include the query execution tree |
+
| trace.explainLevel | +explainlevel | +Number | +0 | +Set to a positive number to collect query execution information for debugging when running a query. Higher numbers give progressively more detail on content node query execution. Tuning this parameter is useful if we want to get more information from the content nodes without gathering lots of trace information from the container chain. Explanation is subject to change at any time, the below is a guide: {`Level`}: Description {`1`}: Timing and overall query plan (blueprint) from each content node {`2`}: Timing per search thread and execution tree (search iterator tree) Note that you might get the same at trace.level 5 and above. Default is no explanation. Tracing with {`trace.explainLevel`} also requires that trace.level is positive. |
+
| trace.profileDepth | ++ | Number | +0 | +Turns on performance profiling of the content node query execution for matching, first-phase ranking, and second-phase ranking. How profiling is performed is based on whether {`trace.profileDepth`} is positive or negative: {`Type`}: Description {`Tree`}: A positive number specifies the depth used by a tree profiler. A higher number means more profiler data. The output resembles the structure of the search iterator tree or rank expression tree being profiled, with total time and self time tracked per component (node in the tree). {`Flat`}: A negative number specifies the topn (cut-off) used by a flat profiler. The output returns the topn components that use the most self time. The performance profiling output is subject to change at any time. Default is no information. Tracing with {`trace.profileDepth`} also requires that trace.level is positive. |
+
| trace.profiling.matching.depth | ++ | Number | +0 | +Turns on profiling of matching of the content node query execution. This exposes information about how time spent on matching is distributed between individual search iterators. The profiling output is tagged *match_profiling* and is subject to change at any time. Default is no information. See trace.profileDepth for semantics of this parameter. Tracing with {`trace.profiling.matching.depth`} requires that trace.level is positive. |
+
| trace.profiling.firstPhaseRanking.depth | ++ | Number | +0 | +Turns on profiling of the first-phase ranking of the content node query execution. This exposes information about how time spent on first-phase ranking is distributed between individual rank features. The profiling output is tagged *first_phase_profiling* and is subject to change at any time. Default is no information. See trace.profileDepth for semantics of this parameter. Tracing with {`trace.profiling.firstPhaseRanking.depth`} also requires that trace.level is positive. |
+
| trace.profiling.secondPhaseRanking.depth | ++ | Number | +0 | +Turns on profiling of the second-phase ranking of the content node query execution. This exposes information about how time spent on second-phase ranking is distributed between individual rank features. The profiling output is tagged *second_phase_profiling* and is subject to change at any time. Default is no information. See trace.profileDepth for semantics of this parameter. Tracing with {`trace.profiling.secondPhaseRanking.depth`} also requires that trace.level is positive. |
+
| trace.timestamps | ++ | Boolean | +false | +Enable to get timing information already at trace.level=1. This is useful for debugging latency spent at different components in the container search chain without rendering a lot of string data which is associated with higher trace levels. | +
| trace.query | ++ | Boolean | +true | +Whether to include the query in any trace messages. This is useful for avoiding query serialization with very large queries to avoid impact from it on performance and excessively large traces. | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| rules.off | ++ | Boolean | +true | +Turn rule evaluation off for this query. | +
| rules.rulebase | ++ | String | ++ | A rule base name - the name of the rule base to use for these queries. | +
| tracelevel.rules | ++ | Number | ++ | The amount of rule evaluation trace output to show, higher number means more details. This is useful to see a trace from rule evaluation without having to see trace from all other searchers at the same time. | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| recall | ++ | String | ++ | Any allowed collection of recall terms. Sets a recall parameter to be combined with the query. This is identical to filter, except that recall terms are not exposed to the ranking framework and thus not ranked. As such, one can not use unprefixed terms; they must either be positive or negative. | +
| user | ++ | String | ++ | The id of the user making the query. The content of the argument is made available to the search chain, but it triggers no features in Vespa apart from being propagated to the access log. | +
| hitcountestimate | ++ | Boolean | +false | +Make this an estimation query. No hits will be returned, and total hit count will be set to an estimate of what executing the query as a normal query would give. | +
| metrics.ignore | ++ | Boolean | +false | +Ignore metric collection for this query request, useful for warm-up queries. | +
| weakAnd.replace | ++ | Boolean | +false | +Replace all instances of OR in the query tree with weakAnd. | +
| wand.hits | ++ | Number | +100 | +Used in combination with weakAnd.replace. Sets the targetHits of the new weakAnds to the specified value. | +
| sorting.degrading | ++ | Boolean | +true | +When sorting on a single-value numeric attribute with fast-search an optimization is activated to return early, with an inaccurate total-hits count. Set {`sorting.degrading`} to false to disable this optimization. This optimization sets the primary sorting attribute as the match phase attribute, and match phase maxHits equal to {`max(10000, maxHits+maxOffset)`}. maxHits and maxOffset can be set in a query profile. |
+
| noCache | +nocache | +Boolean | +false | +Sets whether this query should never be served from a cache. Vespa has few caches, and this parameter does not control any of them. Therefore, this parameter has no effect | +
| Code | +Description | +
|---|---|
| 200 | +OK | +
| 400 | +Bad Request | +
| 401 | +Unauthorized | +
| 403 | +Forbidden | +
| 404 | +Not Found | +
| 405 | +Method Not Allowed | +
| 408 | +Request Timeout | +
| 428 | +Precondition Required | +
| 431 | +Request Header Fields Too Large | +
| 500 | +Internal Server Error | +
| 502 | +Bad Gateway | +
| 503 | +Service Unavailable; no available search handler threads in the jdisc container to serve the request. See Container Tuning on sizing thread pools. | +
| 504 | +Gateway Timeout | +
| 507 | +Insufficient Storage | +
| Error Code | +HTTP Code | +
|---|---|
| com.yahoo.container.protect.Error.BAD_REQUEST | +400 | +
| com.yahoo.container.protect.Error.UNAUTHORIZED | +401 | +
| com.yahoo.container.protect.Error.FORBIDDEN | +403 | +
| com.yahoo.container.protect.Error.NOT_FOUND | +404 | +
| com.yahoo.container.protect.Error.INTERNAL_SERVER_ERROR | +500 | +
| com.yahoo.container.protect.Error.INSUFFICIENT_STORAGE | +507 | +
| Parameter | +Alias | +Type | +Default | +Description | +
|---|---|---|---|---|
| where | ++ | String | ++ | A string with JSON. Refer to the select reference for details. | +
| grouping | ++ | String | ++ | A string with JSON. Refer to the select reference for details. | +
| fields | ++ | String | ++ | A JSON array of summary field names to include in each hit. Equivalent to the field list in a YQL {`select`} clause. Refer to the select reference for details. |
+
| HTTP request | +state/v1 operation | +Description | +
|---|---|---|
| GET | ++ | + |
| + | Service config generation | +{`/state/v1/config`} In the response, config has a mandatory generation and one or more <service> elements: sentinel container distributor logd slobroks servicelayer proton Note: Other configuration elements can also be added as a service. A <service> has a mandatory generation. An optional message can be returned. Example: {``}{`{ "config": {"generation": 11, "slobroks": {"generation": 11, "message": "ok"}}}`}{``} |
+
| + | Service version | +{`/state/v1/version`} Returns a mandatory service version. Example: {``}{`{ "version": "8.43.64"}`}{``} |
+
| + | Service health | +{`/state/v1/health`} Returns the service status, with time, status and metrics. Metrics contains {`requestsPerSecond`} and {`latencySeconds`}, see StateHandler. Example: {`{`} {`"time": 1661863544346,`} {`"status": {`} {`"code": "up"`} {`},`} {`"metrics": {`} {`"snapshot": {`} {`"from": 1661863483.422,`} {`"to": 1661863543.38`} {`},`} {`"values": [`} {`{`} {`"name": "requestsPerSecond",`} {`"values": {`} {`"count": 30,`} {`"rate": 0.5`} {`}`} {`},`} {`{`} {`"name": "latencySeconds",`} {`"values": {`} {`"average": 0.001,`} {`"sum": 0,`} {`"count": 0,`} {`"last": 0.001,`} {`"max": 0.001,`} {`"min": 0.001,`} {`"rate": 0`} {`}`} {`}`} {`]`} {`}`} {`}`} |
+
| + | Service metrics | +{`/state/v1/metrics`} Same as {`/state/v1/health`}, but with a full metrics set. A metric has a name and values, and can have a description and a set of dimensions: {`{`} {`"name": "content.proton.documentdb.matching.rank_profile.query_setup_time",`} {`"description": "Average time (sec) spent setting up and tearing down queries",`} {`"values": {`} {`"average": 0,`} {`"sum": 0,`} {`"count": 0,`} {`"rate": 0,`} {`"min": 0,`} {`"max": 0,`} {`"last": 0`} {`},`} {`"dimensions": {`} {`"documenttype": "music",`} {`"rankProfile": "default"`} {`}`} {`}`} |
+
| + | Service metric histograms | +{`/state/v1/metrics/histograms`} See histograms for usage. The histograms are implemented using HdrHistogram, and the CSV result is what that library generates. |
+
| Element | +Parent | +Type | +Description | +
|---|---|---|---|
| config | ++ | Object | +Root element for /state/v1/config. | +
| generation | +config | +Number | +The generation number is the number for the config that is active in the application. | +
| message | +config | +String | +An info or error message. | +
| version | ++ | String | +Vespa version. | +
| time | ++ | Number | +Epoch in microseconds. | +
| status | ++ | Object | ++ |
| code | +status | +String | +Service status code - one of: up down initializing Containers with the query API enabled return {`initializing`} while waiting for content nodes to start, see example. {`up`} means that the service is fully up. Assume status {`down`} if no response. Refer to StateMonitor for implementation. |
+
| message | +status | +String | +Message is optional - it is normally empty if the service is up, while it is set to a textual reason for why it is unavailable, if so. | +
| metrics | ++ | Object | +Snapshot of metric values. | +
| snapshot | +metrics | +Object | +Time period for metrics snapshot. | +
| from | +snapshot | +Number | +Epoch in seconds, with microseconds fraction. | +
| to | +snapshot | +Number | +Epoch in seconds, with microseconds fraction. | +
| values | +metrics | +Array | +Array of metric objects. | +
| name | +values | +String | +Metric name. | +
| description | +values | +String | +Textual description of the metric. | +
| dimensions | +values | +Object | +Set of dimension name/value pairs. | +
| values | +values | +Object | +Set of metric values. | +
| average | +values | +Number | +Average metric value, typically *sum* divided by *count* . | +
| sum | +values | +Number | +Sum of metric values in snapshot. | +
| count | +values | +Number | +Number of times metric has been set. For instance in a metric counting number of operations done, it will give the number of operations added for that snapshot period. For a value metric, for instance latency of operations, the count will give how many times latencies have been added to the metric. | +
| last | +values | +Number | +Last metric value. | +
| max | +values | +Number | +Max metric value in snapshot. | +
| min | +values | +Number | +Min metric value in snapshot. | +
| rate | +values | +Number | +Metric rate: *count* divided by *snapshot interval* . | +
| Directory/file | +Required | +Description | +
|---|---|---|
| services.xml | +Yes | +Describes which services to run where, and their main configuration. | +
| hosts.xml | +No | +Vespa Cloud: Not used. See node counts in services.xml. Self-managed: The mapping from logical nodes to actual hosts. |
+
| deployment.xml | +Yes, for Vespa Cloud | +Specifies which environments and regions the application is deployed to during automated application deployment, as which application instances. This file also specifies other deployment-related configurations like cloud accounts and private endpoints. The file is required when deploying to the prod environment - it is ignored (with some exceptions) when deploying to the *dev* environment. |
+
| validation-overrides.xml | +No | +Override, allowing this package to deploy even if it fails validation. | +
| .vespaignore | +No | +Contains a list of path patterns that should be excluded from the {`application.zip`} deployed to Vespa. |
+
| models/ | +No | +Machine-learned models in the application package. Refer to stateless model evaluation, Tensorflow, Onnx, XGBoost, and LightGBM. | +
| schemas/ | +No | +Contains the *.sd files describing the document types of the application and how they should be queried and processed. | +
| schemas/[schema]/ | +No | +Contains *.profile files defining rank profiles. This is an alternative to defining rank profiles inside the schema. | +
| security/clients.pem | +Yes, for Vespa Cloud | +PEM encoded X.509 certificates for data plane access. See the security guide for how to generate and use. | +
| components/ | +No | +Contains *.jar files containing searcher(s) for the JDisc Container. | +
| rules/ | +No | +Contains *.sr files containing rule bases for semantic recognition and translation of the query | +
| search/query-profiles/ | +No | +Contains *.xml files containing a named set of search request parameters with values | +
| constants/ | +No | +Constant tensors | +
| tests/ | +No | +Test files for automated tests | +
| ext/ | +No | +Files that are guaranteed to be ignored by Vespa: They are excluded when processing the application package and cannot be referenced from any other element in it. | +
| Command | +Description | +
|---|---|
| **upload** | +Uploads an application package to the config server. Normally not used, as *prepare* includes *upload* | +
| **prepare** | +1. Verifies that a configuration server is up and running 2. Uploads the application to the configuration server, which stores it in * {`$VESPA_HOME/var/db/vespa/config_server/serverdb/tenants/default/sessions/[sessionid]`}*. *[sessionid]* increases for each *prepare*-call. The config server also stores the application in a ZooKeeper instance at */config/v2/tenants/default/sessions/[sessionid]* - this distributes the application to all config servers 3. Creates metadata about the deployed the applications package (which user deployed it, which directory was it deployed from and at what time was it deployed) and stores it in *...sessions/[sessionid]/.applicationMetaData* 4. Verifies that the application package contains the required files and performs a consistency check 5. Validates the xml config files using the schema, found in * {`$VESPA_HOME/share/vespa/schema`}* 6. Checks if there are config changes between the active application and this prepared application that require actions like restart or re-feed (like changes to schemas). These actions are returned as part of the prepare step in the deployment API. This prevents breaking changes to production - also read about validation overrides 7. Distributes constant tensors and bundles with components to nodes using file distribution. Files are downloaded to * {`$VESPA_HOME/var/db/vespa/filedistribution`}*, URL download starts downloading to *{`$VESPA_HOME/var/db/vespa/download`}* |
+
| **activate** | +1. Waits for prepare to complete 2. Activates new configuration version 3. Signals to containers to load new bundles - read more in container components |
+
| **fetch** | +Use *fetch* to download the active application package | +
| Component type | +Description | +
|---|---|
| **Request handler** | +Request handlers allow applications to implement arbitrary HTTP APIs. A request handler accepts a request and returns a response. Custom request handlers are subclasses of ThreadedHttpRequestHandler. | +
| **Processor** | +The processing framework can be used to create general composable synchronous request-response systems. Searchers and search chains are an instantiation (through subclasses) of this general framework for a specific domain. Processors are invoked synchronously and the response is a tree of arbitrary data elements. Custom output formats can be defined by adding renderers. | +
| **Renderer** | +Renderers convert a Response (or query Result) into a serialized form sent over the network. Renderers are subclasses of com.yahoo.processing.rendering.Renderer. | +
| **Searcher** | +Searchers processes Queries and their Results. Since they are synchronous, they can issue multiple queries serially or in parallel to e.g. implement federation or decorate queries with information fetched from a content cluster. Searchers are composed into *search chains* defined in services.xml. A query request selects a particular search chain which implements the logic of that query. Read more. | +
| **Document processor** | +Document processors processes incoming document operations. Similar to Searchers and Processors they can be composed in chains, but document processors are asynchronous. Read more. | +
| **Binding** | +A binding matches a request URI to the correct filter chain or request handler, and route outgoing requests to the correct client. For instance, the binding *http://*/** would match any HTTP request, while *http://*/processing* would only match that specific path. If several bindings match, the most specific one is chosen. **Server binding** A server binding is a rule for matching incoming requests to the correct request handler, basically the JDisc building block for implementing RESTful APIs. **Client binding** A client binding is a pattern which is used to match requests originating inside the container, e.g. when doing federation, to a client provider. That is, it is a rule which determines what code should handle a given outgoing request. | |
+
| **Filter** | +A filter is a lightweight request checker. It may set some specific request property, or it may do security checking and simply block requests missing some mandatory property or header. | +
| **Client** | +Clients, or client providers, are implementations of clients for different protocols, or special rules for given protocols. When a JDisc application acts as a client, e.g. fetches a web page from another host, it is a client provider that handles the transaction. Bindings are used, as with request handlers and filters, to choose the correct client, matching protocol, server, etc., and then hands off the request to the client provider. There is no problem in using arbitrary other types of clients for external services in processors and request handlers. | +
| Component | +Description | +
|---|---|
| **Always available** | ++ |
| AthenzIdentityProvider | +Provides the application's Athenz-identity and gives access to identity/role certificate and tokens. | +
| BertBaseEmbedder | +A BERT-Base compatible embedder, see BertBase embedder. | +
| ConfigInstance | +Configuration is injected into components as {`ConfigInstance`} components - see configuring components. |
+
| Executor | +Default threadpool for processing requests in threaded request handler | +
| Linguistics | +Inject a Linguistics component like SimpleLinguistics or provide a custom implementation - see linguistics. | +
| Metric | +Jdisc core interface for metrics. Required by all subclasses of ThreadedRequestHandler. | +
| MetricReceiver | +Use to emit metrics from a component. Find an example in the metrics guide. | +
| ModelsEvaluator | +Evaluates machine-learned models added to Vespa applications and available as config form. | +
| SentencePieceEmbedder | +A native Java implementation of SentencePiece, see SentencePiece embedder. | +
| VespaCurator | +A client for ZooKeeper. For use in container clusters that have ZooKeeper enabled. See using ZooKeeper. | +
| VipStatus | +Use this to gain control over the service status (up/down) to be emitted from this container. | +
| WordPieceEmbedder | +An implementation of the WordPiece embedder, usually used with BERT models. Refer to WordPiece embedder. | +
| SystemInfo | +Vespa Cloud: Provides information about the environment the component is running in. Read more. | +
**Available in containers having {`search`}** |
+ + |
| DocumentAccess | +To use the Document API. | +
| ExecutionFactory | +To execute new queries from code. Read more. | +
Map{` |
+ Use to inject a set of Models, see Stateless Model Evaluation. | +
**Available in containers having {`document-api`} or {`document-processing`}** |
+ + |
| DocumentAccess | +To use the Document API. | +
| int | +32 bit signed integer value | +
|---|---|
| long | +64 bit signed integer value | +
| double | +64 bit IEEE float value | +
| enum | +Enumerated types. A set of strings representing the valid values for the parameter, e.g: {`foo enum {BAR, BAZ, QUUX} default=BAR`} |
+
| bool | +A boolean (true/false) value | +
| string | +A String value. Default values must be enclosed in quotation marks (" "), and any internal quotation marks must be escaped by backslash. Likewise, newlines must be escaped to {`\\n`} |
+
| path | +A path to a physical file or directory in the application package. This makes it possible to access files from the application package in container components. The path is relative to the root of the application package. A path parameter cannot have a default value, but may be optional (using the *optional* keyword after the type). An optional path does not have to be set, in which case it will be an empty value. The content will be available as a {`java.nio.file.Path`} instance when the component accessing this config is constructed, or an {`Optional if the *optional* keyword is used. |
+
| url | +Similar to {`path`}, an arbitrary URL of a file that should be downloaded and made available to container components. The file content will be available as a java.io.File instance when the component accessing this config is constructed. Note that if the file takes a long time to download, it will also take a long time for the container to come up with the configuration referencing it. See also the note about changing contents for such a url. |
+
| model | +A pointer to a machine-learned model. This can be a model-id, url or path, and multiple of these can be specified as a single config value, where one is used depending on the deployment environment: • If a model-id is specified and the application is deployed on Vespa Cloud, the model-id is used. • Otherwise, if a URL is specified, it is used. • Otherwise, path is used. You may also use remote URLs protected by bearer-token authentication by supplying the optional {`secret-ref`} attribute. See using private Huggingface models. On the receiving side, this config value is simply represented as a file path regardless of how it is resolved. This makes it easy to refer to models in multiple ways such that the appropriate one is used depending on the context. The special syntax for setting these config values is documented in adding files to the configuration. |
+
| reference | +A config id to another configuration (only for internal vespa usage) | +
| individual service elements | +(e.g. *searcher*, *handler*, *searchnode*) - creates a service, but has no child elements that create services | +
|---|---|
| **service group elements** | +(e.g. *content*, *container*, *document-processing* - creates a group of services and can have all types of child elements | +
| **dedicated config elements** | +(e.g. *accesslog*) - configures a service or a group of services and can only have other dedicated config elements as children | +
| **generic config elements** | +always named *config* | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| version | +Yes | +1.0 | +
| major-version | +No | +The major version number this application is valid for. | +
| cloud-account | +No | +Account to deploy to with Vespa Cloud Enclave. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| id | +Yes | +The unique name of the instance. | +
| tags | +No | +Space-separated tags which can be referenced to make deployment variants. | +
| cloud-account | +No | +Account to deploy to with Vespa Cloud Enclave. Overrides parent's use of cloud-account. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| revision | +No, default {`true`} |
+ Set to {`false`} to allow application deployments |
+
| version | +No, default {`true`} |
+ Set to {`false`} to allow Vespa platform upgrades |
+
| maintenance | +No, default {`false`} |
+ Set to {`true`} to disallow Vespa maintenance operations. This is best effort, maintenance can still happen (e.g. for security reasons). The block window for maintenance should be open at least 10% of the time calculated over a week, that is, at least 17 hours per week. |
+
| days | +No, default {`mon-sun`} |
+ List of days this block is effective - a comma-separated list of single days or day intervals where the start and end day are separated by a dash and are inclusive. Each day is identified by its english name or three-letter abbreviation. | +
| hours | +No, default {`0-23`} |
+ List of hours this block is effective - a comma-separated list of single hours or hour intervals where the start and end hour are separated by a dash and are inclusive. Each hour is identified by a number in the range 0 to 23. | +
| time-zone | +No, default UTC | +The name of the time zone used to interpret the hours attribute. Time zones are full names or short forms, when the latter is unambiguous. See ZoneId.of for the full spec of acceptable values. | +
| from-date | +No | +The inclusive starting date of this block (ISO-8601, {`YYYY-MM-DD`}). |
+
| to-date | +No | +The inclusive ending date of this block (ISO-8601, {`YYYY-MM-DD`}). |
+
| Attribute | +Mandatory | +Values | +
|---|---|---|
| frequency | +Yes | +A positive integer with a suffix {`h`} (hours) or {`d`} (days), e.g. {`24h`} or {`7d`}. Minimum 24h. |
+
| granularity | +No, default {`cluster`} |
+ • {`cluster`}: all content nodes in the cluster• {`group`}: all content nodes in a single group |
+
| Attribute | +Mandatory | +Values | +
|---|---|---|
| key | +Yes | +The tag key. Must be non-empty. Allowed characters and maximum length depend on the target cloud; see per-cloud rules below. The {`vai_`} prefix is reserved for internal use. |
+
| value | +Yes | +The tag value. Must be non-empty. May contain template variables. Allowed characters and maximum length depend on the target cloud; see per-cloud rules below. | +
| Constraint | +AWS | +Azure | +GCP | +
|---|---|---|---|
| Key characters | +{`[a-zA-Z0-9 +-=._:/@]`} |
+ Unicode, except {`< > % & \\ ? /`} |
+ {`[a-z][a-z0-9_-]*`} (must start with lowercase letter) |
+
| Value characters | +{`[a-zA-Z0-9 +-=._:/@]`} |
+ No restrictions | +{`[a-z0-9_-]*`} |
+
| Key max length | +128 | +512 | +63 | +
| Value max length | +256 | +256 | +63 | +
| Max tags | +50 | +50 | +64 | +
| Variable | +Description | +
|---|---|
{`\${tenant}`} |
+ The tenant name, e.g. {`mytenant`}. |
+
{`\${application}`} |
+ The application name, e.g. {`myapp`}. |
+
{`\${instance}`} |
+ The instance name, e.g. {`default`}, {`beta`}. |
+
{`\${environment}`} |
+ The deployment environment, e.g. {`prod`}, {`dev`}. |
+
{`\${region}`} |
+ The deployment region, e.g. {`aws-us-east-1c`}. |
+
{`\${clustername}`} |
+ The cluster ID from services.xml, e.g. {`default`}, {`music`}. |
+
{`\${clustertype}`} |
+ The Vespa cluster type: {`container`}, {`content`}, or {`admin`}. |
+
| Attribute | +Mandatory | +Values | +
|---|---|---|
| rollout | +No, default {`simultaneous`} |
+ • {`separate`}: When a revision catches up to a platform upgrade, it stays behind, unless the upgrade alone fails.• {`leading`}: When a revision catches up to a platform upgrade, they fuse and roll out together.• {`simultaneous`} is the default, and favors revision roll-out. Revision changes deploy independently of platform upgrades. When a revision catches up to a platform upgrade, it joins, and then passes the upgrade. |
+
| revision-target | +No, default {`latest`} |
+ • {`latest`} is the default. When rolling out a new revision to an instance, the latest available revision is chosen.• {`next`} trades speed for smaller changes. When rolling out a new revision to an instance, the next available revision is chosen.The available revisions for an instance are revisions which are not yet deployed, or revisions which have rolled out in previous instances. |
+
| revision-change | +No, default {`when-failing`} |
+ • {`always`} is the most aggressive setting. A new, available revision may always replace the one which is currently rolling out.• {`when-failing`} is the default. A new, available revision may replace the one which is currently rolling out if this is failing.• {`when-clear`} is the most conservative setting. A new, available revision may never replace one which is currently rolling out.Revision targets will never automatically change inside revision block window, but may be set by manual intervention at any time. |
+
| max-risk | +No, default {`0`} |
+ May only be used with {`revision-change="when-clear"`} and {`revision-target="next"`}. The maximum amount of *risk* to roll out per new revision target. The default of {`0`} results in the next build always being chosen, while a higher value allows skipping intermediate builds, as long as the cumulative risk does not exceed what is configured here. |
+
| min-risk | +No, default {`0`} |
+ Must be less than or equal to the configured {`max-risk`}. The minimum amount of *risk* to start rolling out a new revision. The default of {`0`} results in a new revision rolling out as soon as anything is ready, while a higher value lets the system wait until enough cumulative risk is available. This can be used to avoid blocking a lengthy deployment process with trivial changes. |
+
| max-idle-hours | +No, default {`8`} |
+ May only be used when {`min-risk`} is specified, and greater than {`0`}. The maximum number of hours to wait for enough cumulative risk to be available, before rolling out a new revision. |
+
| Parent | +Description | +
|---|---|
{` {` |
+ If present, the application is deployed to the {`test`} environment, and system tested there, even if no prod zones are deployed to. Also, when specified, system tests *must* be present in the application test package. See guides for getting to production. If present in an {` element, system tests are run for that specific instance before any production deployments of the instance may proceed — otherwise, previous system tests for any instance are acceptable. |
+
{` {` {` |
+ If present, production tests are run against the production region with id contained in this element. A test must be *after* a corresponding region element. When specified, production tests *must* be preset in the application test package. See guides for getting to production. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| cloud-account | +No | +For system tests only: account to deploy to with Vespa Cloud Enclave. Overrides parent's use of cloud-account. Cloud account *must not* be specified for production tests, which always run in the account of the corresponding deployment. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| cloud-account | +No | +Account to deploy to with Vespa Cloud Enclave. Overrides parent's use of cloud-account. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| cloud-account | +No | +Account to deploy to with Vespa Cloud Enclave. Overrides parent's use of cloud-account. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| name | +Yes | +The region identifier, e.g. aws-us-east-1c. See zones for the list of available regions. Mandatory unless the region name is specified in the element body, but this is not compatible with specifying <availability-zone> children. |
+
| fraction | +No | +Only when this region is inside a group: The fractional membership in the group. | +
| cloud-account | +No | +Account to deploy to with Enclave. Overrides parent's use of cloud-account. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| tags | +No | +Space-separated tags which can be referenced to make deployment variants. | +
| cloud-account | +No | +Account to deploy to with Vespa Cloud Enclave. Overrides parent's use of cloud-account. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| id | +No | +The identifier for the endpoint. This will be part of the endpoint name that is generated. If not specified, the endpoint will be the default global endpoint for the application. | +
| container-id | +Yes | +The id of the container cluster to which requests to the global endpoint is forwarded. | +
| Attribute | +Mandatory | +Values | +
|---|---|---|
| type | +Yes | +Private endpoints are specified with {`type='zone'`}. |
+
| container-id | +Yes | +The id of the container cluster to disable public endpoints for. | +
| enabled | +No | +Whether a public endpoint for this container cluster should be enabled; default {`true`}. |
+
| Attribute | +Mandatory | +Values | +
|---|---|---|
| type | +Yes | +Private endpoints are specified with {`type='private'`}. |
+
| container-id | +Yes | +The id of the container cluster to which requests to the private endpoint service is forwarded. | +
| auth-method | +No | +The authentication method to use with this private endpoint. Must be either {`mtls`} or {`token`}. Defaults to mTLS if not included. |
+
| Attribute | +Mandatory | +Values | +
|---|---|---|
| with | +Yes | +The private endpoint access type; must be {`aws-private-link`} or {`gcp-service-connect`}. |
+
| arn | +Maybe | +Must be specified with {`aws-private-link`}. See AWS documentation for more details. |
+
| project | +Maybe | +Must be specified with {`gcp-service-connect`}. See GCP documentation for more details. |
+
| Attribute | +Mandatory | +Values | +
|---|---|---|
| deadline | +No | +The max time after a region becomes unreachable until the other regions in its BCP group must be able to handle the traffic of it, given as a number of minutes followed by 'm', 'h' or 'd' (for minutes, hours or days). The default deadline is 0: Regions must at all times have capacity to handle BCP traffic immediately. By providing a deadline, autoscaling can avoid the cost of provisioning additional resources for BCP capacity if it predicts that it can grow to handle the traffic faster than the deadline in a given cluster. This is the default deadline to be used for all groups that don't specify one themselves. |
+
| Attribute | +Mandatory | +Values | +
|---|---|---|
| deadline | +No | +The deadline of this BCP group. See deadline on the BCP element. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **version** | +required | +number | ++ | 2.0 | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **hostalias** | +required | +string | ++ | + |
| **baseport** | +optional | +number | ++ | + |
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **standalone-zookeeper** | +optional | +true/false | +false | +Will by default share the ZooKeeper instance with configserver. If configured to true a separate ZooKeeper instance will be configured and started on the set of nodes where you run cluster controller on. The set of cluster controllers nodes cannot overlap with the set of nodes where config server is running. If this setting is changed from false to true in a running system, all previous cluster state information will be lost as the underlying ZooKeeper changes. Cluster controllers will re-discover the state, but nodes that have been manually set as down will again be considered to be up. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **hostalias** | +required | +string | ++ | + |
| **baseport** | +optional | +number | ++ | + |
| **jvm-options** | +optional | +string | ++ | + |
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **hostalias** | +required | +string | ++ | + |
| **baseport** | +optional | +number | ++ | + |
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **hostalias** | +required | +string | ++ | + |
| **baseport** | +optional | +number | ++ | + |
| **jvm-options** | +optional | +string | ++ | + |
| **jvm-gc-options** | +optional | +string | ++ | + |
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **hostalias** | +required | +string | ++ | + |
| **baseport** | +optional | +number | ++ | + |
| + | + |
|---|---|
| systemname | +The name of the application in question in the monitoring system, default is "vespa" | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The name of the consumer to export metrics to. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The id of the metric set to include. Built-in metric sets are: • {`default`}• {`Vespa`} |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The name of the metric as defined in custom code or in process metrics api | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **region** | +required | +string | ++ | Your AWS region | +
| **namespace** | +required | +string | ++ | The metrics namespace in CloudWatch | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **file** | +required | +string | ++ | The path to the shared-credentials file | +
| **profile** | +optional | +string | +default | +The profile in the shared-credentials file | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **version** | +required | +number | ++ | 1.0 in this version of Vespa | +
| **id** | +required | +string | ++ | the id of this cluster | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **class** | +optional | +string | ++ | The class of the handler, defaults to id | +
| **bundle** | +optional | +string | ++ | The bundle to load the handler from: The name in {` in pom.xml. Defaults to class or id (if no class is given) |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **class** | +optional | +string | ++ | The class of the server, defaults to id | +
| **bundle** | +optional | +string | ++ | The bundle to load the server from: The name in {` in the pom.xml. Defaults to class or id (if no class is given). |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The client ID | +
| **permissions** | +required | +string | ++ | Permissions, see the security guide. One of: • {`read`}• {`write`}• {`read,write`} |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **file** | +required | +string | ++ | Path to the certificate file, see the security guide. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | Token ID, see the security guide. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **class** | +optional | +string | ++ | The class of the component, defaults to id | +
| **bundle** | +optional | +string | ++ | The bundle to load the component from: The name in {` in the pom.xml. Defaults to class or id (if no class is given). |
+
{``} {`http://*/document-api/ `}{`https://*/document-api/ `}{``}{`http://*/document-api/feed/`} and {`https://*/document-api/feed/`}. For other handlers, just replace 'feed/' with the appropriate suffix, e.g. 'get/', 'remove/' etc.{`true`} will abort the feed on such errors, while setting it to {`false`} will cause Vespa to simply skip to the next document in the feed. Note that malformed XML in the input will abort the feed regardless of this setting.{``} is 0 and this is set to 0, this defaults to 100 MB. If {``} is more than 0, and this is set to 0, the send-window is only limited by number of messages sent, not the memory footprint.{``}{`xml `} {``}{``}{`10MiB `}{``}{``}{`{``}| Element | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **memory** | +optional | +string | +auto-estimated | +Container memory reserved for model inference, covering both model weights and inference requests. This memory is subtracted from the memory available to the JVM heap on the same node. When not set, Vespa estimates the required inference memory automatically. The automatic estimate can be inaccurate for some models and workloads, which may lead to out-of-memory errors. Set this element explicitly to override the estimate. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **type** | +optional | +string | +json | +The accesslog type: *json*, *vespa* or *disabled* | +
| **fileNamePattern** | +required* | +string | +{`JsonAccessLog. |
+ File name pattern. * Note: Optional when *type* is *disabled* | +
| **symlinkName** | +optional | +string | +{`JsonAccessLog. |
+ Symlink name | +
| **rotationInterval** | +optional | +string | +0 60 ... | +Rotation interval | +
| **rotationScheme** | +optional | +string | +date | +Valid values are *date* or *sequence* | +
| Element | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **samples-per-second** | +required | +double | ++ | Probabilistic sample rate per second | +
| **path-prefix** | +required | +string | ++ | URI path prefix to match for logging | +
| **max-bytes** | +required | +integer | ++ | Maximum size in bytes to log, only prefix will be kept for larger requests | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **dir** | +required | +string | ++ | The directory to include files from. File inclusion order is undefined. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **allocated-memory** | +optional | +percentage | ++ | |
| **jvm-options** | +optional | +string | ++ | |
| **jvm-gc-options** | +optional | +string | ++ |
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **allocated-memory** | +optional | +percentage | ++ | Memory to allocate to each JVM instance as a percentage of available memory. Must be an integer percentage followed by *%* | +
| **options** | +optional | +string | ++ | Generic JVM options | +
| **gc-options** | +optional | +string | ++ | JVM GC options. Garbage Collector specific parameters | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **type** | +required | +string | ++ | Value: "oath-ckms" | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **name** | +required | +string | ++ | Key group name | +
| **environment** | +required | +string | ++ | Value one of: "alpha" "corp" "prod" "aws" "aws_stage" | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **max** | +optional | +number | +100 | +The maximum number of threads relative to vCPU cores. Value must be greater than or equal to {`. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **version** | +required | +number | ++ | 1.0 in this version of Vespa | +
| **id** | +required for multiple clusters | +string | ++ | Name of the content cluster. If none is supplied, the cluster name will be {`content`}. Cluster names must be unique within the application, if multiple clusters are configured, the name must be set for all but one at minimum. Renaming a cluster is the same as dropping the current cluster and adding a new one. This makes data unavailable or lost, depending on hosting model. Deploying with a changed cluster id will therefore fail with a validation override requirement: {`Content cluster 'music' is removed. This will cause loss of all data in this cluster. To allow this add . |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **selection** | +optional | +string | ++ | A document selection, restricting documents that are routed to this cluster. Defaults to a selection expression matching everything. This selection can be specified to match document identifier specifics that are *independent* of document types. For restrictions that apply only to a *specific* document type, this must be done within that particular document type's document element. Trying to use document type references in this selection makes an error during deployment. The selection given here will be merged with per-document type selections specified within document tags, if any, meaning that any document in the cluster must match *both* selections to be accepted and kept. This feature is primarily used to expire documents. |
+
| **garbage-collection** | +optional | +true / false | +false | +If true, regularly verify the documents stored in the cluster to see if they belong in the cluster, and delete them if not. If false, garbage collection is not run. | +
| **garbage-collection-interval** | +optional | +integer | +3600 | +Time (in seconds) between garbage collection cycles. Note that the deletion of documents is spread over this interval, so more resources will be used for deleting a set of documents with a small interval than with a larger interval. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **type** | +required | +string | ++ | Document type name | +
| **mode** | +required | +index / store-only / streaming | ++ | The mode of storing and indexing. Refer to streaming search for *store-only*, as documents are stored the same way for both cases. Changing mode requires an *indexing-mode-change* validation override, and documents must be re-fed. |
+
| **selection** | +optional | +string | ++ | A document selection, restricting documents that are routed to this cluster. Defaults to a selection expression matching everything. This selection must apply to fields in *this document type only*. Selection will be merged together with selection for other types and global selection from documents to form a full expression for what documents belong to this cluster. |
+
| **global** | +optional | +true / false | +false | +Set to *true* to distribute all documents of this type to all nodes in the content cluster it is defined. Fields in global documents can be imported into documents to implement joins - read more in parent/child. Vespa will detect when a new (or outdated) node is added to the cluster and prevent it from taking part in searches until it has received all global documents. Changing from *false* to *true* or vice versa requires a *global-document-change* validation override. First, stop services on all content nodes. Then, deploy with the validation override. Finally, start services on all content nodes. *global* is only supported for *mode="index"*. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **cluster** | +optional | +string | +Container cluster on content node | +Name of a document-processing container cluster that does index preprocessing. Use cluster to specify an alternative cluster, other than the default cluster on content nodes. | +
| **chain** | +optional | +string | +{`indexing`} chain |
+ A document processing chain in the container cluster specified by *cluster* to use for index preprocessing. The chain must inherit the {`indexing`} chain. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| distribution-key | +required | +integer | ++ | The unique data distribution id of this node. This **must** remain unchanged for the host's lifetime. Distribution keys of a fresh system should be contiguous and start from zero. Distribution keys are used to identify nodes and groups for the distribution algorithm. If a node changes distribution key, the distribution algorithm regards it as a new node, so buckets are redistributed. |
+
| capacity | +optional | +double | +1 | +Capacity of this node, relative to other nodes. A node with capacity 2 will get double the data and feed requests of a node with capacity 1. This feature is deprecated and expert mode only. Don't use in production, Vespa assumes homogenous cluster capacity. |
+
| baseport | +optional | +integer | ++ | baseport The first port in the port range allocated by this node. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **distribution-key** | +required | +integer | ++ | Sets the distribution key of a group. It is not allowed to change this for a given group. Group distribution keys only need to be unique among groups that share the same parent group. | +
| **name** | +required | +string | ++ | The name of the group, used for access from status pages and the like. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **partitions** | +required if there are subgroups in the group | +string | ++ | String conforming to the partition specification: Partition specification    Description *   Distribute all copies over 1 of N groups 1|*    Distribute all copies over 2 of N groups 1|1|*   Distribute all copies over 3 of N groups |
+
| Element | +Required | +Quantity | +
|---|---|---|
| searchnode | +No | +Zero or one | +
| Element | +Required | +Quantity | +
|---|---|---|
| lidspace | +No | +Zero or one | +
| + | ||
| requestthreads | +No | +Zero or one | +
| flushstrategy | +No | +Zero or one | +
| initialize | +No | +Zero or one | +
| feeding | +No | +Zero or one | +
| index | +No | +Zero or one | +
| summary | +No | +Zero or one | +
| Element | +Required | +Default | +Description | +
|---|---|---|---|
| **search** | +Optional | +**Vespa Cloud:** min(vcpu*4 + persearch - 1, vcpu*persearch) **Self-hosted:** 64. | +Total size of the match engine thread pool. Together with {`persearch`}, this determines the maximum number of queries that can execute concurrently: {`search / persearch`}. See the Vespa serving scaling guide for sizing guidance. |
+
| **persearch** | +Optional | +1 | +Maximum number of threads used per search. A higher value reduces the time queries spend in query evaluation, except time spent in ANN which is single-threaded. This number of threads is held for each query for the duration of the query, also when much of the time is spent on single-threaded operations. See the Vespa serving scaling guide for an introduction of using multiple threads per search per node to reduce query latency. Number of threads per search can be adjusted down per *rank-profile* using num-threads-per-search. | +
| **summary** | +Optional | +**Vespa Cloud:** vcpu **Self-hosted:** 16 | +Number of summary threads. | +
| Element | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **disk** | +optional | +float [0, 1] | +0.9 | +Fraction of total space on the disk partition used before put and update operations are rejected | +
| **memory** | +optional | +float [0, 1] | +0.9 | +Fraction of physical memory that can be resident memory in anonymous mapping by proton before put and update operations are rejected | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **max-documents** | +optional | +integer | +1024 | +Maximum number of documents per content bucket. Buckets are split in two if they have more documents than this. Keep this value below 16K. | +
| **max-size** | +optional | +integer | +32MiB | +Maximum size (in bytes) of a bucket. This is the sum of the serialized size of all documents kept in the bucket. Buckets are split in two if they have a larger size than this. Keep this value below 100 MiB. | +
| **minimum-bits** | +optional | +integer | ++ | Override the ideal distribution bit count configured for this cluster. Prefer to use the distribution type setting instead if the default distribution bit count does not fit the cluster. This variable is intended for testing and to work around possible distribution bit issues. Most users should not need this option. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **type** | +optional | +loose | strict | legacy | +loose | +When the number of a nodes configured in a system changes over certain limits, the system will automatically trigger major redistributions of documents. This is to ensure that the number of buckets is appropriate for the number of nodes in the cluster. This enum value specifies how aggressive the system should be in triggering such distribution changes. The default of {`loose`} strikes a balance between rarely altering the distribution of the cluster and keeping the skew in document distribution low. It is recommended that you use the default mode unless you have empirically observed that it causes too much skew in load or document distribution.Note that specifying {`minimum-bits`} under bucket-splitting overrides this setting and effectively "locks" the distribution in place. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **max-per-node** | +optional | +number | ++ | Maximum number of parallel active bucket merge operations. | +
| **max-queue-size** | +optional | +number | ++ | Maximum size of the merge bucket queue, before reporting BUSY back to the distributors. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **thread-count** | +optional | +number | ++ | The maximum number of threads in which to execute visitor operations. A higher number of threads may increase performance, but may use more memory. | +
| **max-queue-size** | +optional | +number | ++ | Maximum size of the pending visitor queue, before reporting BUSY back to the distributors. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **fixed** | +optional | +number | +16 | +The fixed component of the maximum active count | +
| **variable** | +optional | +number | +64 | +The variable component of the maximum active count | +
| Element | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **disk** | +optional | +float [0, 1] | +0.8 | +Fraction of total space on the disk partition used on a content node before feed is blocked | +
| **memory** | +optional | +float [0, 1] | +0.8/0.75 | +Fraction of physical memory that can be resident memory in anonymous mapping on a content node before feed is blocked. Total physical memory is sampled as the minimum of {`sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE)`} and the cgroup (v1 or v2) memory limit. Nodes with 8 Gib or less memory in Vespa Cloud has a limit of 0.75. |
+
| Element | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **max-hits-per-partition** | +optional | +Integer | +No capping: Return all | +Maximum number of hits to return from a content node. By default, a query returns the requested number of hits + offset from every content node to the container. The container orders the hits globally according to the query, then discards all hits beyond the number requested. In a system with a large fan-out, this consumes network bandwidth and the container nodes easily network saturated. Containers will also sort and discard more hits than optimal. When there are sufficiently many search nodes, assuming an even distribution of the hits, it suffices to only return a fraction of the request number of hits from each node. Note that changing this number will have global ordering impact. See *top-k-probability* below for improving performance with fewer hits. |
+
| **dispatch-policy | +optional** | +adaptive / best-of-random-2 / round-robin | +adaptive | +With grouped distribution: Configure policy for choosing which group shall receive the next query request. Coverage requirements is considered when choosing a group. Note that multiphase requests that requires or benefits from hitting the same group in all phases are always hashed. **adaptive** Measures latency, preferring lower latency groups, selecting group {`i`} has a probability proportional to 1 / (latency for group {`i`}). **best-of-random-2** Selects 2 random groups and selects the one with the lowest latency. **round-robin** Selects groups in a round-robin manner, giving fair distribution of queries to each group. |
+
| **prioritize-availability** | +optional | +Boolean | +true | +With grouped distribution: If true, or by default, all groups that are within min-active-docs-coverage of the **median** of the document count of other groups will be used to service queries. If set to false, only groups within min-active-docs-coverage of the **max** document count will be used, with the consequence that full coverage is prioritized over availability when multiple groups are lacking content, since the remaining groups may not be able to service the full query load. | +
| **min-active-docs-coverage** | +optional | +A float percentage | +97 | +With grouped distribution: The percentage of active documents a group must have, relative to the median across all groups in the content cluster, to be considered active for serving queries. Because of measurement timing differences, it is not advisable to tune this above 99 percent. | +
| **top-k-probability** | +optional | +Double | +0.9999 | +Probability that the top K hits will be the globally best. Based on this probability, the dispatcher will fetch enough hits from each node to achieve this. The only way to guarantee a probability of 1.0 is to fetch K hits from each partition. However, by reducing the probability from 1.0 to 0.99999, one can significantly reduce number of hits fetched and save both bandwidth and latency. The number of hits to fetch from each partition is computed as: $$ q = \frac{k}{n} + q T \left(\right. p , 30 \left.\right) \times \sqrt{k \times \frac{1}{n} \times \left(\right. 1 - \frac{1}{n} \left.\right)} $$ where qT is a Student's t-distribution. With n=10 partitions, k=200 hits and p=0.99999, only 45 hits per partition is needed, as opposed to 200 when p=1.0. Use this option to reduce network and container cpu/memory in clusters with many nodes per group - see Vespa Serving Scaling Guide. |
+
| Element | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **init-progress-time** | +optional | ++ | + | If the initialization progress count have not been altered for this amount of seconds, the node is assumed to have deadlocked and is set down. Note that initialization may actually be prioritized lower now, so setting a low value here might cause false positives. Though if it is set down for wrong reason, when it will finish initialization and then be set up again. | +
| **transition-time** | +optional | ++ | storage_transition_time distributor_transition_time | +The transition time states how long (in seconds) a node will be in maintenance mode during what looks like a controlled restart. Keeping a node in maintenance mode during a restart allows a restart without the cluster trying to create new copies of all the data immediately. If the node has not started or got back up within the transition time, the node is set down, in which case, new full bucket copies will be created. Note separate defaults for distributor and storage (i.e. search) nodes. | +
| **max-premature-crashes** | +optional | ++ | max_premature_crashes | +The maximum number of crashes allowed before a content node is permanently set down by the cluster controller. If the node has a stable up or down state for more than the *stable-state-period*, the crash count is reset. However, resetting the count will not re-enable the node again if it has been disabled - restart the cluster controller to reset. | +
| **stable-state-period** | +optional | ++ | stable_state_time_period | +If a content node's state doesn't change for this many seconds, it's state is considered *stable*, clearing the premature crash count. | +
| **min-distributor-up-ratio** | +optional | ++ | min_distributor_up_ratio | +The minimum ratio of distributors that are required to be *up* for the cluster state to be *up*. | +
| **min-storage-up-ratio** | +optional | ++ | min_storage_up_ratio | +The minimum ratio of content nodes that are required to be *up* for the cluster state to be *up*. | +
| **groups-allowed-down-ratio** | +optional | ++ | groups-allowed-down-ratio | +A ratio for the number of content groups that are allowed to be down simultaneously. A value of 0.5 means that 50% of the groups are allowed to be down. The default is to allow only one group to be down at a time. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **numnodesperclient** | +optional | ++ | + | Set to some number below the amount of nodes in the cluster to limit how many nodes a single client can connect to. If you have many clients, this can reduce the memory usage on both document-processing and client nodes. |
+
| **preferlocalnode** | +optional | ++ | false | +Set to always prefer sending to a document-processing node running on the same host as the client. You should use this if you are running a client on each document-processing node. |
+
| **maxmessagesinqueue** | ++ | + | + | + |
| **maxqueuebytesize** | ++ | + | + | |
| **maxqueuewait** | +optional | ++ | + | The maximum number of seconds a message should wait in queue before being processed. Docproc will adapt its queue size to adhere to this. If the queue is full, new messages will be replied to with SESSION_BUSY. | +
| **maxconcurrentfactor** | ++ | + | + | + |
| **documentexpansionfactor** | +optional | ++ | + | + |
| **containercorememory** | ++ | + | + | + |
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| class | ++ | + | + | + |
| bundle | ++ | + | + | + |
| id | +required | ++ | + | The component id of the documentprocessor instance. | +
| idref | ++ | + | + | + |
| provides | +optional | ++ | + | A space-separated list of names that represents what this documentprocessor produces. | +
| before | +optional | ++ | + | A space-separated list of phase or provided names. Phases or documentprocessors providing these names will be placed later in the docproc chain than this document processor. | +
| after | +optional | ++ | + | A space-separated list of phase or provided names. Phases or documentprocessors providing these names will be placed earlier in the docproc chain than this document processor. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| id | +required | ++ | + | The component id of the documentprocessor instance. | +
| class | +optional | ++ | + | A component specification containing the name of the class to instantiate to create the document processor instance. If missing, copied from id. | +
| bundle | +optional | ++ | + | The bundle containing the class: The name in {` in pom.xml. If a bundle is not specified, the bundle containing document processors bundled with Vespa is used. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **port** | +optional | +number | +The web services port of the environment variables | +Server port | +
| **default-request-chain** | +optional | +string | ++ | The default request chain to use for unmatched requests | +
| **default-response-chain** | +optional | +string | ++ | The default response chain to use for unmatched requests | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **class** | +required | +string | ++ | The class name | +
| **bundle** | +required | +string | ++ | The bundle name | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **strict-mode** | +optional | +boolean | +true | +When set to true, all requests must match a filter. For any requests not matching, an HTTP 403 response is returned. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **class** | +optional | +string | +id | +The class of the component, defaults to id | +
| **bundle** | +optional | +string | +id or class | +The bundle to load the component from, defaults to class or id (if no class is given) | +
| **before** | +optional | +string | ++ | Space separated list of phases and/or filters which should succeed this phase | +
| **class** | +optional | +string | +id | +Space separated list of phases and/or filters which should precede this phase | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **inherits** | ++ | string | ++ | A space separated list of chains this chain should include the contents of | +
| **excludes** | ++ | string | ++ | A space separated list of filters (contained in an inherited chain) this chain should not include | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The ID, or name, which other phases and filters may depend upon as a successor or predecessor | +
| **before** | +optional | +string | ++ | Space separated list of phases and/or filters which should succeed this phase | +
| **after** | +optional | +string | ++ | Space separated list of phases and/or filters which should precede this phase | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **class** | +optional | +string | ++ | The class of the component, defaults to id | +
| **bundle** | +optional | +string | ++ | The bundle to load the component from, defaults to class or id (if no class is given) | +
| **before** | +optional | +string | ++ | Space separated list of phases and/or processors which should succeed this processor | +
| **after** | +optional | +string | ++ | Space separated list of phases and/or processors which should precede this processor | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **class** | +optional | +string | ++ | The class of the component, defaults to id | +
| **bundle** | +optional | +string | ++ | The bundle to load the component from, defaults to class or id (if no class is given) | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **idref** | ++ | string | ++ | Reference to the definition of this processor. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **idref** | ++ | string | ++ | A reference to a defined chain. Mutually exclusive with *id*. If *idref* is used, no other attributes apply. | +
| **id** | +required | +string | ++ | The chain ID. Required unless *idref* is used | +
| **inherits** | +optional | +string | ++ | A space-separated list of chains this chain should include the contents of - see example below. | +
| **excludes** | +optional | +string | ++ | A space-separated list of processors (contained in an inherited chain) this chain should not include. The exclusion is done before any consolidation of component references when inheriting chains. Example:{` |
+
| **class** | +optional | +string | ++ | + |
| **name** | ++ | + | + | + |
| **documentprocessors** | ++ | + | + | + |
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The ID, or name, which other phases and processors may depend upon as a successor or predecessor. | +
| **before** | +optional | +string | ++ | Space-separated list of phases and/or processors which should succeed this phase | +
| **after** | +optional | +string | ++ | Space-separated list of phases and/or processors which should precede this phase | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component id of the searcher instance. For inner searchers, the id must be unique inside the search chain. For outer searchers, the id must be unique. An inner searcher is not permitted to have the same id as an outer searcher. | +
| **class** | +optional | ++ | + | A component specification containing the name of the class to instantiate to create the searcher instance. If missing, copied from id | +
| **bundle** | +optional | ++ | + | A component specification containing the bundle symbolic name and version used to select the bundle: The name in {` in pom.xml. The class is loaded from this bundle. If no bundle is specified, it defaults to the bundle containing the searchers bundled with Vespa. |
+
| **provides** | +optional | ++ | + | A space-separated list of names that represents what this searcher produces. For more information on provides, before and after, see chained components | +
| **before** | +optional | ++ | + | A space-separated list of phase or provided names. Phases or searchers providing these names will be placed later in the search chain than this searcher | +
| **after** | +optional | ++ | + | A space-separated list of phase or provided names. Phases or searchers providing these names will be placed earlier in the search chain than this searcher | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **idref** | +required | +string | ++ | Reference to a searcher definition | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **timeout** | +optional | +number | ++ | The minimum number of seconds or milliseconds (if ms is present) the federation searcher waits for the federated search chain executing the query | +
| **requestTimeout** | +optional | +number | ++ | The minimum number of seconds or milliseconds (if ms is present) the search chain executing the query should continue execution. In some cases it is useful to set this higher than the timeout, such that a chain can keep waiting for requested data longer than the query is waiting for the chain. This allows queries to populate caches within the search chain even though populating the caches requires waiting longer than the query timeout | +
| **optional** | +optional | +true/false | +false | +Determines if the federation searcher should wait for this search chain at all. Normally, it only waits for mandatory (i.e. not optional) search chains, and when they are done, cancels the remaining search chains that are not finished. If all the search chains federated to are optional, all of them will be treated as mandatory. All search chains are per default mandatory | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | The component ID | +
| **class** | +optional | +string | ++ | The class of the component, defaults to id | +
| **bundle** | +optional | +string | ++ | The bundle to load the component from: The name in {` in your pom.xml. If no bundle is given, the bundle containing renderers provided by Vespa is used. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **id** | +required | +string | ++ | ID | +
| **excludes** | +optional | ++ | + | + |
| **type** | +optional | +local | ++ | Determines which searchers are implicitly added to this search chain to talk to the external service. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **cluster** | +required | +string | ++ | The name of the local cluster. | +
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| **max** | +optional | +number | +equal to {` |
+ The maximum number of threads relative to vCPU cores. Value must be greater than or equal to {`. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| version | +required | +number | ++ | 1.0 in this version of Vespa | +
| Attribute | +type | +Default | +Description | +
|---|---|---|---|
| **count** | +integer or range | ++ | Vespa Cloud: The number of nodes of the cluster. | +
| **exclusive** | +boolean | +false | +Optional. Vespa Cloud: If true these nodes will never be placed on shared hosts even when this would otherwise be allowed (which is only for content nodes in some environments). When nodes are allocated exclusively, the resources must match the resources of the host exactly. | +
| **groups** | +integer or range | ++ | Vespa Cloud content nodes only, optional: Integer or range. Sets the number of groups into which content nodes should be divided. Each group will have an equal share of the nodes, and one or more complete copies of the corpus and index, and each query will be routed to just one group - see grouped distribution. This allows scaling to a higher query load than is possible with just a single group. | +
| **group-size** | +integer or range | ++ | Vespa Cloud content nodes only, optional: Integer or range where either value can be skipped (replaced by an empty string) to create a one-sided limit. This can be set as an alternative to explicitly setting {`groups`}: The group sizes used will always be within these limits (inclusive), for any {`count`}. |
+
| Attribute | +type | +Default | +Description | +
|---|---|---|---|
| **vcpu** | +float or range | +2 | +CPU (virtual threads) | +
| **memory** | +float or range, each followed by a byte unit, such as "Gb" | +8 Gb in container clusters, 16 Gb in content clusters | +Memory | +
| **disk** | +float or range, each followed by a byte unit, such as "Gb" | +50 in container clusters, 300 in content clusters | +Disk space. To fit core dumps/heap dumps, the disk space should be larger than 3 x memory size for content nodes, 2 x memory size for container nodes. If disk size is not explicitly specified, Vespa Cloud chooses a default disk size. The default may be automatically increased to satisfy the minimum disk-to-memory ratio. When both disk and memory are explicitly specified, Vespa Cloud enforces the same minimum ratios. | +
| **storage-type** | +string (enum) | +{`any`} |
+ The type of storage to use. This is useful to specify local storage when network storage provides insufficient io operations or too noisy io performance: • {`local`}: Node-local storage is required. • {`remote`}: Network storage must be used.• {`any`}: Both remote or local storage may be used. |
+
| **disk-speed** | +string (enum) | +{`fast`} |
+ The required disk speed category: • {`fast`}: SSD-like disk speed is required• {`slow`}: This is sized for spinning disk speed • {`any`}: Performance does not depend on disk speed (often suitable for container clusters). |
+
| **architecture** | +string (enum) | +{`any`} |
+ Node CPU architecture: • {`x86_64`} • {`arm64`}• {`any`}: Use any of the available architectures. |
+
| **max-cost-factor** | +float >= 1.0 | +{`1.0`} |
+ Used to allow provisioning of machine types larger than what's specified by these resources when no exactly matching types are currently available for provisioning. This number specifies the max oversize of a provisioned host measured in list price. For example, if this is set to 2, up to twice as costly machines as required may be provisioned. Exact matches are always preferred when available, and the system will migrate to exactly matching resources in the background as they become available. This has no impact on the resources available to the container node of the Vespa cluster, which will always match the resource spec. This can only be set on enclave deployed applications. |
+
| Attribute | +Required | +Value | +Default | +Description | +
|---|---|---|---|---|
| hostalias | +required | +string | ++ | a host name which must be mapped to a full hostname in hosts.xml | +
| Attribute | +type | +Description | +
|---|---|---|
| **count** | +integer | +Number of GPUs | +
| **memory** | +integer, followed by a byte unit, such as "Gb" | +Amount of memory per GPU. Total amount of GPU memory available is this number multiplied by {`count`}. |
+
| Category | +Annotation | +JUnit tag | +Description | +
|---|---|---|---|
| System test | +@SystemTest | +system | +Independent, functional tests | +
| Staging setup | +@StagingSetup | +staging-setup | +Set state before upgrade | +
| Staging test | +@StagingTest | +staging | +Verify state after upgrade | +
| Production test | +@ProductionTest | +production | +Verify domain specific metrics | +
| Category | +Directory | +Description | +
|---|---|---|
| System test | +tests/system-test/ | +Independent, functional tests | +
| Staging setup | +tests/staging-setup/ | +Set state before upgrade | +
| Staging test | +tests/staging-test/ | +Verify state after upgrade | +
| Production test | +tests/production-test/ | +Verify domain specific metrics | +
| Name | +Parent | +Type | +Default | +Description | +
|---|---|---|---|---|
| name | +root step | +string | +file name, step *n* | +Name used for display purposes in the test report. The file name is used by default for the test, while the 1-indexed "step n" is used for steps. | +
| defaults | +root | +object | ++ | Default settings for all steps in this test. May be overridden in each step. | +
| steps | +root | +array | ++ | The non-empty list of steps that constitute this test. | +
| request | +step | +object | ++ | A specification of a request to send, to Vespa, or to an external service. | +
| cluster | +defaults request | +string | ++ | The name of the Vespa cluster to send a request to, as specified in services.xml. If this is not specified, and the application has a single container cluster, this is used. | +
| method | +request | +string | +"GET" | +The HTTP method to use for a request. | +
| uri | +request | +string | +"/search/" | +When this is path + (encoded) query, the host is determined by the specified cluster; otherwise, it must be an absolute URI (with scheme), and its host is used. Query parameters specified here override those specified in the defaults. | +
| parameters | +defaults request | +string object | ++ | HTTP request query parameters. The values should not be encoded. These are merged with parameters from the specified URI, and override those specified in the defaults. If the value is a string, it must be a relative file reference to a parameters object. | +
| body | +request response | +string object | ++ | The body for a request, or the partial body (see matching) for a response. If the value is a string, it must be a relative file reference to a JSON object to be used in its place. | +
| response | +step | +object | ++ | A specification for assertions to make on the body of the HTTP response obtained by executing the HTTP request in the same step. | +
| code | +response | +number | +200 | +The status code the response should have. | +
| Attribute | +Mandatory | +Value | +
|---|---|---|
| until | +Yes | +The last day this change is allowed, as a ISO-8601-format date in UTC, e.g. 2016-01-30. Dates may at most be 30 days in the future, but should be as close to now as possible for safety, while allowing time for review and propagation to all deployed zones. {`allow`}-tags with dates in the past are ignored. |
+
| comment | +No | +Text explaining the reason for the change to humans. | +
| Log field | +Description | +
|---|---|
| *time* | +Time in seconds since 1970-01-01 UTC, with optional fractional seconds after. E.g. 1102675319.726342 | +
| *host* | +The hostname of the machine that produced this log entry | +
| *pid* | +The process id, and an optional thread-id of the process/thread that logged the message | +
| *service* | +The Vespa service name of the logger | +
| *component* | +The component name that logged. An application may have multiple subcomponents with their own component names, usually starts with the name of the binary | +
| *level* | +One of fatal, error, warning, info, config, event, debug, or spam | +
| *message* | +The log message itself. All dangerous characters are escaped (CR, NL, TAB, \, ASCII < 32 and ASCII 128..159) | +
| Log level | +Description | +
|---|---|
| *fatal* | +Fatal error messages. The application must exit immediately, and restarting it will not help | +
| *error* | +Error messages. These are serious, the application cannot function correctly | +
| *warning* | +Warnings - the application may be able to continue, but the situation should be looked into | +
| *info* | +Informational messages that are not reporting error conditions, but should still be useful to the operator | +
| *config* | +Configuration settings | +
| *event* | +Machine-readable events. May contain information about processes starting and stopping, and various metrics | +
| *debug* | +Debug messages - normally suppressed | +
| *spam* | +Low-level debug messages, normally suppressed. Generates massive amounts of logs when enabled | +
| Event | +Description | +
|---|---|
| starting | +Payload: *{`name=*This event is sent by processes when they are about to start another process. Typical for, but not limited to, shell scripts. This event is not required to track processes, but is useful in cases where a sub-process may fail during startup. Example: {`starting container for default/container.0`} |
+
| started | +Payload: *{`name=* The *started* event is sent by a service that just started up. Example: {`started/1 name="vespa-proton"`} |
+
| stopping | +Payload: *{`name=*The *stopping* event is sent by a process that is about to exit. Example: {`stopping/1 name="vespa-proton" why="clean shutdown"`} |
+
| stopped | +Payload: *{`name=*This event is sent by a process monitoring when a sub-process exits. Example: {`stopped/1 name="vespa-proton" pid=14523 exitcode=0`} |
+
| crash | +Payload: *{`name=* Submitted by a process monitoring a sub-process when the sub-process crashes (dumps core etc.). Example: {`crash/1 name="vespa-proton" pid=12345 signal=11`} |
+
| count | +Payload: *{`name=* General event for counts - for tracking any type of counter metric. The *name* is specific to each library/application. Counters are assumed to increase with time, counting the number of events since the program was started. Example: {`count/1 name="queries" value=10`} |
+
| value | +Payload: *{`name=* General event for values - for tracking any type of value metric. *Value is for values that cannot be counts*. Typical values are queue lengths, transaction frequencies and so on. Example: {`value/1 name="peak_qps" value=200`} |
+
| state | +Payload: *{`name=*General event for components in a process. *value* contains a string with more detailed information on what has happened. Note that the format and content of such strings can change between releases. Example: {`state/1 name="transactionlog.replay.start" value="{"domain":"test","serialnum":{"first":1,"last":1000}}"`} |
+
| Name | +Unit | +Description | +
|---|---|---|
| cluster-controller.down.count | +node | +Number of content nodes down | +
| cluster-controller.initializing.count | +node | +Number of content nodes initializing | +
| cluster-controller.maintenance.count | +node | +Number of content nodes in maintenance | +
| cluster-controller.retired.count | +node | +Number of content nodes that are retired | +
| cluster-controller.stopping.count | +node | +Number of content nodes currently stopping | +
| cluster-controller.up.count | +node | +Number of content nodes up | +
| cluster-controller.cluster-state-change.count | +node | +Number of nodes changing state | +
| cluster-controller.nodes-not-converged | +node | +Number of nodes not converging to the latest cluster state version | +
| cluster-controller.stored-document-count | +document | +Total number of unique documents stored in the cluster | +
| cluster-controller.stored-document-bytes | +byte | +Combined byte size of all unique documents stored in the cluster (not including replication) | +
| cluster-controller.cluster-buckets-out-of-sync-ratio | +fraction | +Ratio of buckets in the cluster currently in need of syncing | +
| cluster-controller.busy-tick-time-ms | +millisecond | +Time busy | +
| cluster-controller.idle-tick-time-ms | +millisecond | +Time idle | +
| cluster-controller.work-ms | +millisecond | +Time used for actual work | +
| cluster-controller.is-master | +binary | +1 if this cluster controller is currently the master, or 0 if not | +
| cluster-controller.remote-task-queue.size | +operation | +Number of remote tasks queued | +
| cluster-controller.node-event.count | +operation | +Number of node events | +
| cluster-controller.resource_usage.nodes_above_limit | +node | +The number of content nodes above resource limit, blocking feed | +
| cluster-controller.resource_usage.max_memory_utilization | +fraction | +Current memory utilisation, for content node with the highest value | +
| cluster-controller.resource_usage.max_disk_utilization | +fraction | +Current disk space utilisation, for content node with the highest value | +
| cluster-controller.resource_usage.memory_limit | +fraction | +Memory space limit as a fraction of available memory | +
| cluster-controller.resource_usage.disk_limit | +fraction | +Disk space limit as a fraction of available disk space | +
| reindexing.progress | +fraction | +Re-indexing progress | +
| Name | +Unit | +Description | +
|---|---|---|
| configserver.requests | +request | +Number of requests processed | +
| configserver.failedRequests | +request | +Number of requests that failed | +
| configserver.latency | +millisecond | +Time to complete requests | +
| configserver.cacheConfigElems | +item | +Time to complete requests | +
| configserver.cacheChecksumElems | +item | +Number of checksum elements in the cache | +
| configserver.hosts | +node | +The number of nodes being served configuration from the config server cluster | +
| configserver.tenants | +instance | +The number of tenants being served configuration from the config server cluster | +
| configserver.applications | +instance | +The number of applications being served configuration from the config server cluster | +
| configserver.delayedResponses | +response | +Number of delayed responses | +
| configserver.sessionChangeErrors | +session | +Number of session change errors | +
| configserver.unknownHostRequests | +request | +Config requests from unknown hosts | +
| configserver.newSessions | +session | +New config sessions | +
| configserver.preparedSessions | +session | +Prepared config sessions | +
| configserver.activeSessions | +session | +Active config sessions | +
| configserver.inactiveSessions | +session | +Inactive config sessions | +
| configserver.addedSessions | +session | +Added config sessions | +
| configserver.removedSessions | +session | +Removed config sessions | +
| configserver.rpcServerWorkQueueSize | +item | +Number of elements in the RPC server work queue | +
| maintenanceDeployment.transientFailure | +operation | +Number of maintenance deployments that failed with a transient failure | +
| maintenanceDeployment.failure | +operation | +Number of maintenance deployments that failed with a permanent failure | +
| maintenanceDeployment.reason | +operation | +Reason for maintenance deployment | +
| maintenance.successFactorDeviation | +fraction | +Configserver: Maintenance Success Factor Deviation | +
| maintenance.duration | +millisecond | +Configserver: Maintenance Duration | +
| maintenance.congestion | +failure | +Configserver: Maintenance Congestion | +
| configserver.zkConnectionLost | +connection | +Number of ZooKeeper connections lost | +
| configserver.zkReconnected | +connection | +Number of ZooKeeper reconnections | +
| configserver.zkConnected | +node | +Number of ZooKeeper nodes connected | +
| configserver.zkSuspended | +node | +Number of ZooKeeper nodes suspended | +
| configserver.zkZNodes | +node | +Number of ZooKeeper nodes present | +
| configserver.zkAvgLatency | +millisecond | +Average latency for ZooKeeper requests | +
| configserver.zkMaxLatency | +millisecond | +Max latency for ZooKeeper requests | +
| configserver.zkConnections | +connection | +Number of ZooKeeper connections | +
| configserver.zkOutstandingRequests | +request | +Number of ZooKeeper requests in flight | +
| orchestrator.lock.acquire-latency | +second | +Time to acquire zookeeper lock | +
| orchestrator.lock.acquire-success | +operation | +Number of times zookeeper lock has been acquired successfully | +
| orchestrator.lock.acquire-timedout | +operation | +Number of times zookeeper lock couldn't be acquired within timeout | +
| orchestrator.lock.acquire | +operation | +Number of attempts to acquire zookeeper lock | +
| orchestrator.lock.acquired | +operation | +Number of times zookeeper lock was acquired | +
| orchestrator.lock.hold-latency | +second | +Time zookeeper lock was held before it was released | +
| nodes.active | +node | +The number of active nodes in a cluster | +
| nodes.nonActive | +node | +The number of non-active nodes in a cluster | +
| nodes.nonActiveFraction | +node | +The fraction of non-active nodes vs total nodes in a cluster | +
| nodes.exclusiveSwitchFraction | +fraction | +The fraction of nodes in a cluster on exclusive network switches | +
| nodes.emptyExclusive | +node | +The number of exclusive hosts that do not have any nodes allocated to them | +
| nodes.expired.deprovisioned | +node | +The number of deprovisioned nodes that have expired | +
| nodes.expired.dirty | +node | +The number of dirty nodes that have expired | +
| nodes.expired.inactive | +node | +The number of inactive nodes that have expired | +
| nodes.expired.provisioned | +node | +The number of provisioned nodes that have expired | +
| nodes.expired.reserved | +node | +The number of reserved nodes that have expired | +
| cluster.cost | +dollar_per_hour | +The cost of the nodes allocated to a certain cluster, in $/hr | +
| cluster.load.ideal.cpu | +fraction | +The ideal cpu load of a certain cluster | +
| cluster.load.ideal.memory | +fraction | +The ideal memory load of a certain cluster | +
| cluster.load.ideal.disk | +fraction | +The ideal disk load of a certain cluster | +
| cluster.load.peak.cpu | +fraction | +The peak cpu load in the period considered of a certain cluster | +
| cluster.load.peak.memory | +fraction | +The peak memory load in the period considered of a certain cluster | +
| cluster.load.peak.disk | +fraction | +The peak disk load in the period considered of a certain cluster | +
| cluster.backup.age | +fraction | +Age of the most recent cluster backup as a fraction of the backup interval | +
| cluster.snapshot.busySeconds | +second | +The maximum time a snapshot has been busy (creating or restoring) for a cluster | +
| zone.working | +binary | +The value 1 if zone is considered healthy, 0 if not. This is decided by considering the number of non-active nodes vs the number of active nodes in a zone | +
| cache.nodeObject.hitRate | +fraction | +The fraction of cache hits vs cache lookups for the node object cache | +
| cache.nodeObject.evictionCount | +item | +The number of cache elements evicted from the node object cache | +
| cache.nodeObject.size | +item | +The number of cache elements in the node object cache | +
| cache.curator.hitRate | +fraction | +The fraction of cache hits vs cache lookups for the curator cache | +
| cache.curator.evictionCount | +item | +The number of cache elements evicted from the curator cache | +
| cache.curator.size | +item | +The number of cache elements in the curator cache | +
| wantedRestartGeneration | +generation | +Wanted restart generation for tenant node | +
| currentRestartGeneration | +generation | +Current restart generation for tenant node | +
| wantToRestart | +binary | +One if node wants to restart, zero if not | +
| wantedRebootGeneration | +generation | +Wanted reboot generation for tenant node | +
| currentRebootGeneration | +generation | +Current reboot generation for tenant node | +
| wantToReboot | +binary | +One if node wants to reboot, zero if not | +
| retired | +binary | +One if node is retired, zero if not | +
| wantedVespaVersion | +version | +Wanted vespa version for the node, in the form MINOR.PATCH. Major version is not included here | +
| currentVespaVersion | +version | +Current vespa version for the node, in the form MINOR.PATCH. Major version is not included here | +
| wantToChangeVespaVersion | +binary | +One if node want to change Vespa version, zero if not | +
| hasWireguardKey | +binary | +One if node has a WireGuard key, zero if not | +
| wantToRetire | +binary | +One if node wants to retire, zero if not | +
| wantToDeprovision | +binary | +One if node wants to be deprovisioned, zero if not | +
| failReport | +binary | +One if there is a fail report for the node, zero if not | +
| suspended | +binary | +One if the node is suspended, zero if not | +
| suspendedSeconds | +second | +The number of seconds the node has been suspended | +
| activeSeconds | +second | +The number of seconds the node has been active | +
| numberOfServicesUp | +instance | +The number of services confirmed to be running on a node | +
| numberOfServicesNotChecked | +instance | +The number of services supposed to run on a node, that has not checked | +
| numberOfServicesDown | +instance | +The number of services confirmed to not be running on a node | +
| someServicesDown | +binary | +One if one or more services has been confirmed to not run on a node, zero if not | +
| numberOfServicesUnknown | +instance | +The number of services the config server does not know is running on a node | +
| nodeFailerBadNode | +binary | +One if the node is failed due to being bad, zero if not | +
| downInNodeRepo | +binary | +One if the node is registered as being down in the node repository, zero if not | +
| numberOfServices | +instance | +Number of services supposed to run on a node | +
| lockAttempt.acquireMaxActiveLatency | +second | +Maximum duration for keeping a lock, ending during the metrics snapshot, or still being kept at the end or this snapshot period | +
| lockAttempt.acquireHz | +operation_per_second | +Average number of locks acquired per second the snapshot period | +
| lockAttempt.acquireLoad | +operation | +Average number of locks held concurrently during the snapshot period | +
| lockAttempt.lockedLatency | +second | +Longest lock duration in the snapshot period | +
| lockAttempt.lockedLoad | +operation | +Average number of locks held concurrently during the snapshot period | +
| lockAttempt.acquireTimedOut | +operation | +Number of locking attempts that timed out during the snapshot period | +
| lockAttempt.deadlock | +operation | +Number of lock grab deadlocks detected during the snapshot period | +
| lockAttempt.errors | +operation | +Number of other lock related errors detected during the snapshot period | +
| hostedVespa.docker.totalCapacityCpu | +vcpu | +Total number of VCPUs on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.totalCapacityMem | +gigabyte | +Total amount of memory on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.totalCapacityDisk | +gigabyte | +Total amount of disk space on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.freeCapacityCpu | +vcpu | +Total number of free VCPUs on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.freeCapacityMem | +gigabyte | +Total amount of free memory on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.freeCapacityDisk | +gigabyte | +Total amount of free disk space on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.allocatedCapacityCpu | +vcpu | +Total number of allocated VCPUs on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.allocatedCapacityMem | +gigabyte | +Total amount of allocated memory on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.docker.allocatedCapacityDisk | +gigabyte | +Total amount of allocated disk space on tenant hosts managed by hosted Vespa in a zone | +
| hostedVespa.pendingRedeployments | +task | +The number of hosted Vespa re-deployments pending | +
| hostedVespa.docker.skew | +fraction | +A number in the range 0..1 indicating how well allocated resources are balanced with availability on hosts | +
| hostedVespa.activeHosts | +host | +The number of managed hosts that are in state "active" | +
| hostedVespa.breakfixedHosts | +host | +The number of managed hosts that are in state "breakfixed" | +
| hostedVespa.deprovisionedHosts | +host | +The number of managed hosts that are in state "deprovisioned" | +
| hostedVespa.dirtyHosts | +host | +The number of managed hosts that are in state "dirty" | +
| hostedVespa.failedHosts | +host | +The number of managed hosts that are in state "failed" | +
| hostedVespa.inactiveHosts | +host | +The number of managed hosts that are in state "inactive" | +
| hostedVespa.parkedHosts | +host | +The number of managed hosts that are in state "parked" | +
| hostedVespa.provisionedHosts | +host | +The number of managed hosts that are in state "provisioned" | +
| hostedVespa.readyHosts | +host | +The number of managed hosts that are in state "ready" | +
| hostedVespa.reservedHosts | +host | +The number of managed hosts that are in state "reserved" | +
| hostedVespa.activeNodes | +host | +The number of managed nodes that are in state "active" | +
| hostedVespa.breakfixedNodes | +host | +The number of managed nodes that are in state "breakfixed" | +
| hostedVespa.deprovisionedNodes | +host | +The number of managed nodes that are in state "deprovisioned" | +
| hostedVespa.dirtyNodes | +host | +The number of managed nodes that are in state "dirty" | +
| hostedVespa.failedNodes | +host | +The number of managed nodes that are in state "failed" | +
| hostedVespa.inactiveNodes | +host | +The number of managed nodes that are in state "inactive" | +
| hostedVespa.parkedNodes | +host | +The number of managed nodes that are in state "parked" | +
| hostedVespa.provisionedNodes | +host | +The number of managed nodes that are in state "provisioned" | +
| hostedVespa.readyNodes | +host | +The number of managed nodes that are in state "ready" | +
| hostedVespa.reservedNodes | +host | +The number of managed nodes that are in state "reserved" | +
| overcommittedHosts | +host | +The number of hosts with over-committed resources | +
| spareHostCapacity | +host | +The number of spare hosts | +
| throttledHostFailures | +host | +Number of host failures stopped due to throttling | +
| throttledNodeFailures | +host | +Number of node failures stopped due to throttling | +
| nodeFailThrottling | +binary | +Metric indicating when node failure throttling is active. The value 1 means active, 0 means inactive | +
| clusterAutoscaled | +operation | +Number of times a cluster has been rescaled by the autoscaler | +
| clusterAutoscaleDuration | +second | +The currently predicted duration of a rescaling of this cluster | +
| deployment.prepareMillis | +millisecond | +Duration of deployment preparations | +
| deployment.activateMillis | +millisecond | +Duration of deployment activations | +
| throttledHostProvisioning | +binary | +Value 1 if host provisioning is throttled, 0 if not | +
| Name | +Unit | +Description | +
|---|---|---|
| http.status.1xx | +response | +Number of responses with a 1xx status | +
| http.status.2xx | +response | +Number of responses with a 2xx status | +
| http.status.3xx | +response | +Number of responses with a 3xx status | +
| http.status.4xx | +response | +Number of responses with a 4xx status | +
| http.status.5xx | +response | +Number of responses with a 5xx status | +
| application_generation | +version | +The currently live application config generation (aka session id) | +
| in_service | +binary | +This will have the value 1 if the node is in service, 0 if not. | +
| jdisc.gc.count | +operation | +Number of JVM garbage collections done | +
| jdisc.gc.ms | +millisecond | +Time spent in JVM garbage collection | +
| jdisc.jvm | +version | +JVM runtime version | +
| cpu | +thread | +Container service CPU pressure | +
| jdisc.memory_mappings | +operation | +JDISC Memory mappings | +
| jdisc.open_file_descriptors | +item | +JDISC Open file descriptors | +
| jdisc.thread_pool.unhandled_exceptions | +thread | +Number of exceptions thrown by tasks | +
| jdisc.thread_pool.work_queue.capacity | +thread | +Capacity of the task queue | +
| jdisc.thread_pool.work_queue.size | +thread | +Size of the task queue | +
| jdisc.thread_pool.rejected_tasks | +thread | +Number of tasks rejected by the thread pool | +
| jdisc.thread_pool.size | +thread | +Size of the thread pool | +
| jdisc.thread_pool.max_allowed_size | +thread | +The maximum allowed number of threads in the pool | +
| jdisc.thread_pool.active_threads | +thread | +Number of threads that are active | +
| jdisc.deactivated_containers.total | +item | +JDISC Deactivated container instances | +
| jdisc.deactivated_containers.with_retained_refs.last | +item | +JDISC Deactivated container nodes with retained refs | +
| jdisc.application.failed_component_graphs | +item | +JDISC Application failed component graphs | +
| jdisc.application.component_graph.creation_time_millis | +millisecond | +JDISC Application component graph creation time | +
| jdisc.application.component_graph.reconfigurations | +item | +JDISC Application component graph reconfigurations | +
| jdisc.singleton.is_active | +item | +JDISC Singleton is active | +
| jdisc.singleton.activation.count | +operation | +JDISC Singleton activations | +
| jdisc.singleton.activation.failure.count | +operation | +JDISC Singleton activation failures | +
| jdisc.singleton.activation.millis | +millisecond | +JDISC Singleton activation time | +
| jdisc.singleton.deactivation.count | +operation | +JDISC Singleton deactivations | +
| jdisc.singleton.deactivation.failure.count | +operation | +JDISC Singleton deactivation failures | +
| jdisc.singleton.deactivation.millis | +millisecond | +JDISC Singleton deactivation time | +
| jdisc.http.ssl.handshake.failure.missing_client_cert | +operation | +JDISC HTTP SSL Handshake failures due to missing client certificate | +
| jdisc.http.ssl.handshake.failure.expired_client_cert | +operation | +JDISC HTTP SSL Handshake failures due to expired client certificate | +
| jdisc.http.ssl.handshake.failure.invalid_client_cert | +operation | +JDISC HTTP SSL Handshake failures due to invalid client certificate | +
| jdisc.http.ssl.handshake.failure.incompatible_protocols | +operation | +JDISC HTTP SSL Handshake failures due to incompatible protocols | +
| jdisc.http.ssl.handshake.failure.incompatible_chifers | +operation | +JDISC HTTP SSL Handshake failures due to incompatible chifers | +
| jdisc.http.ssl.handshake.failure.connection_closed | +operation | +JDISC HTTP SSL Handshake failures due to connection closed | +
| jdisc.http.ssl.handshake.failure.unknown | +operation | +JDISC HTTP SSL Handshake failures for unknown reason | +
| jdisc.http.latency | +millisecond | +Request latency including the HTTP layer | +
| jdisc.http.time_to_first_byte | +millisecond | +Time from request has been received by the server until the first byte is returned to the client | +
| jdisc.http.request.prematurely_closed | +request | +HTTP requests prematurely closed | +
| jdisc.http.request.requests_per_connection | +request | +HTTP requests per connection | +
| jdisc.http.request.uri_length | +byte | +HTTP URI length | +
| jdisc.http.request.content_size | +byte | +HTTP request content size | +
| jdisc.http.requests | +request | +HTTP requests | +
| jdisc.http.requests.status | +request | +Number of requests to the built-in status handler | +
| jdisc.http.filter.rule.blocked_requests | +request | +Number of requests blocked by filter | +
| jdisc.http.filter.rule.allowed_requests | +request | +Number of requests allowed by filter | +
| jdisc.http.filtering.request.handled | +request | +Number of filtering requests handled | +
| jdisc.http.filtering.request.unhandled | +request | +Number of filtering requests unhandled | +
| jdisc.http.filtering.response.handled | +request | +Number of filtering responses handled | +
| jdisc.http.filtering.response.unhandled | +request | +Number of filtering responses unhandled | +
| jdisc.http.handler.unhandled_exceptions | +request | +Number of unhandled exceptions in handler | +
| jdisc.tls.capability_checks.succeeded | +operation | +Number of TLS capability checks succeeded | +
| jdisc.tls.capability_checks.failed | +operation | +Number of TLS capability checks failed | +
| jdisc.http.jetty.threadpool.thread.max | +thread | +Configured maximum number of threads | +
| jdisc.http.jetty.threadpool.thread.min | +thread | +Configured minimum number of threads | +
| jdisc.http.jetty.threadpool.thread.reserved | +thread | +Configured number of reserved threads or -1 for heuristic | +
| jdisc.http.jetty.threadpool.thread.busy | +thread | +Number of threads executing internal and transient jobs | +
| jdisc.http.jetty.threadpool.thread.idle | +thread | +Number of idle threads | +
| jdisc.http.jetty.threadpool.thread.total | +thread | +Current number of threads | +
| jdisc.http.jetty.threadpool.queue.size | +thread | +Current size of the job queue | +
| jdisc.http.jetty.http_compliance.violation | +failure | +Number of HTTP compliance violations | +
| serverNumOpenConnections | +connection | +The number of currently open connections | +
| serverNumConnections | +connection | +The total number of connections opened | +
| serverBytesReceived | +byte | +The number of bytes received by the server | +
| serverBytesSent | +byte | +The number of bytes sent from the server | +
| handled.requests | +operation | +The number of requests handled per metrics snapshot | +
| handled.latency | +millisecond | +The time used for handling requests, excluding HTTP layer and rendering | +
| httpapi_latency | +millisecond | +Duration for requests to the HTTP document APIs | +
| httpapi_pending | +operation | +Document operations pending execution | +
| httpapi_num_operations | +operation | +Total number of document operations performed | +
| httpapi_num_updates | +operation | +Document update operations performed | +
| httpapi_num_removes | +operation | +Document remove operations performed | +
| httpapi_num_puts | +operation | +Document put operations performed | +
| httpapi_ops_per_sec | +operation_per_second | +Document operations per second | +
| httpapi_succeeded | +operation | +Document operations that succeeded | +
| httpapi_failed | +operation | +Document operations that failed | +
| httpapi_parse_error | +operation | +Document operations that failed due to document parse errors | +
| httpapi_condition_not_met | +operation | +Document operations not applied due to condition not met | +
| httpapi_not_found | +operation | +Document operations not applied due to document not found | +
| httpapi_failed_unknown | +operation | +Document operations failed by unknown cause | +
| httpapi_failed_timeout | +operation | +Document operations failed by timeout | +
| httpapi_failed_insufficient_storage | +operation | +Document operations failed by insufficient storage | +
| httpapi_queued_operations | +operation | +Document operations queued for execution in /document/v1 API handler | +
| httpapi_queued_bytes | +byte | +Total operation bytes queued for execution in /document/v1 API handler | +
| httpapi_queued_age | +second | +Age in seconds of the oldest operation in the queue for /document/v1 API handler | +
| httpapi_mbus_window_size | +operation | +The window size of Messagebus's dynamic throttle policy for /document/v1 API handler | +
| mem.heap.total | +byte | +Total available heap memory | +
| mem.heap.free | +byte | +Free heap memory | +
| mem.heap.used | +byte | +Currently used heap memory | +
| mem.direct.total | +byte | +Total available direct memory | +
| mem.direct.free | +byte | +Currently free direct memory | +
| mem.direct.used | +byte | +Direct memory currently used | +
| mem.direct.count | +byte | +Number of direct memory allocations | +
| mem.native.total | +byte | +Total available native memory | +
| mem.native.free | +byte | +Currently free native memory | +
| mem.native.used | +byte | +Native memory currently used | +
| athenz-tenant-cert.expiry.seconds | +second | +Time remaining until Athenz tenant certificate expires | +
| container-iam-role.expiry.seconds | +second | +Time remaining until IAM role expires | +
| peak_qps | +query_per_second | +The highest number of qps for a second for this metrics snapshot | +
| search_connections | +connection | +Number of search connections | +
| feed.operations | +operation | +Number of document feed operations | +
| feed.latency | +millisecond | +Feed latency | +
| feed.http-requests | +operation | +Feed HTTP requests | +
| queries | +operation | +Query volume | +
| query_container_latency | +millisecond | +The query execution time consumed in the container | +
| query_latency | +millisecond | +The overall query latency as observed by the container cluster, excluding HTTP layer and rendering | +
| query_timeout | +millisecond | +The amount of time allowed for query execution, from the client | +
| failed_queries | +operation | +The number of failed queries | +
| degraded_queries | +operation | +The number of degraded queries, e.g. due to some content nodes not responding in time | +
| hits_per_query | +hit_per_query | +The number of hits returned | +
| query_hit_offset | +hit | +The offset for hits returned | +
| documents_covered | +document | +The combined number of documents considered during query evaluation | +
| documents_total | +document | +The number of documents to be evaluated if all requests had been fully executed | +
| documents_target_total | +document | +The target number of total documents to be evaluated when all data is in sync | +
| jdisc.render.latency | +nanosecond | +The time used by the container to render responses | +
| query_item_count | +item | +The number of query items (terms, phrases, etc.) | +
| docproc.proctime | +millisecond | +Time spent processing document | +
| docproc.documents | +document | +Number of processed documents | +
| totalhits_per_query | +hit_per_query | +The total number of documents found to match queries | +
| empty_results | +operation | +Number of queries matching no documents | +
| requestsOverQuota | +operation | +The number of requests rejected due to exceeding quota | +
| relevance.at_1 | +score | +The relevance of hit number 1 | +
| relevance.at_3 | +score | +The relevance of hit number 3 | +
| relevance.at_10 | +score | +The relevance of hit number 10 | +
| error.timeout | +operation | +Requests that timed out | +
| error.backends_oos | +operation | +Requests that failed due to no available backends nodes | +
| error.plugin_failure | +operation | +Requests that failed due to plugin failure | +
| error.backend_communication_error | +operation | +Requests that failed due to backend communication error | +
| error.empty_document_summaries | +operation | +Requests that failed due to missing document summaries | +
| error.illegal_query | +operation | +Requests that failed due to illegal queries | +
| error.invalid_query_parameter | +operation | +Requests that failed due to invalid query parameters | +
| error.internal_server_error | +operation | +Requests that failed due to internal server error | +
| error.misconfigured_server | +operation | +Requests that failed due to misconfigured server | +
| error.invalid_query_transformation | +operation | +Requests that failed due to invalid query transformation | +
| error.results_with_errors | +operation | +The number of queries with error payload | +
| error.unspecified | +operation | +Requests that failed for an unspecified reason | +
| error.unhandled_exception | +operation | +Requests that failed due to an unhandled exception | +
| serverRejectedRequests | +operation | +Deprecated. Use jdisc.thread_pool.rejected_tasks instead. | +
| serverThreadPoolSize | +thread | +Deprecated. Use jdisc.thread_pool.size instead. | +
| serverActiveThreads | +thread | +Deprecated. Use jdisc.thread_pool.active_threads instead. | +
| jrt.transport.tls-certificate-verification-failures | +failure | +TLS certificate verification failures | +
| jrt.transport.peer-authorization-failures | +failure | +TLS peer authorization failures | +
| jrt.transport.server.tls-connections-established | +connection | +TLS server connections established | +
| jrt.transport.client.tls-connections-established | +connection | +TLS client connections established | +
| jrt.transport.server.unencrypted-connections-established | +connection | +Unencrypted server connections established | +
| jrt.transport.client.unencrypted-connections-established | +connection | +Unencrypted client connections established | +
| max_query_latency | +millisecond | +Deprecated. Use query_latency.max instead | +
| mean_query_latency | +millisecond | +Deprecated. Use the expression (query_latency.sum / query_latency.count) instead | +
| jdisc.http.filter.athenz.accepted_requests | +request | +Number of requests accepted by the AthenzAuthorization filter | +
| jdisc.http.filter.athenz.rejected_requests | +request | +Number of requests rejected by the AthenzAuthorization filter | +
| jdisc.http.filter.athenz.grid_requests | +request | +Number of grid requests | +
| serverConnectionsOpenMax | +connection | +Maximum number of open connections | +
| serverConnectionDurationMax | +millisecond | +Longest duration a connection is kept open | +
| serverConnectionDurationMean | +millisecond | +Average duration a connection is kept open | +
| serverConnectionDurationStdDev | +millisecond | +Standard deviation of open connection duration | +
| serverNumRequests | +request | +Number of requests | +
| serverNumSuccessfulResponses | +request | +Number of successful responses | +
| serverNumFailedResponses | +request | +Number of failed responses | +
| serverNumSuccessfulResponseWrites | +request | +Number of successful response writes | +
| serverNumFailedResponseWrites | +request | +Number of failed response writes | +
| serverStartedMillis | +millisecond | +Time since the service was started | +
| embedder.latency | +millisecond | +Time spent creating an embedding | +
| embedder.sequence_length | +item | +Number of tokens in the input sequence | +
| embedder.request.count | +request | +Number of embedder API requests | +
| embedder.request.failure.count | +request | +Number of failed embedder API requests | +
| embedder.batch.size | +item | +Number of items in each dispatched batch | +
| embedder.batch.queue_time | +millisecond | +Time spent waiting in queue before batch dispatch | +
| embedder.batch.count | +operation | +Number of batch dispatches | +
| inference.pending | +item | +Number of pending inference requests in a queue | +
| inference.request.rate | +operation_per_second | +Successful inference requests per second | +
| inference.failure.rate | +operation_per_second | +Failed inference requests per second | +
| inference.request.latency | +millisecond | +Average inference request latency | +
| inference.queue.latency | +millisecond | +Average inference queue latency | +
| inference.compute.latency | +millisecond | +Average inference compute latency | +
| inference.queue_compute.ratio | +ratio | +Ratio of inference queue time to compute time | +
| jvm.buffer.count | +buffer | +An estimate of the number of buffers in the pool | +
| jvm.buffer.memory.used | +byte | +An estimate of the memory that the Java virtual machine is using for this buffer pool | +
| jvm.buffer.total.capacity | +byte | +An estimate of the total capacity of the buffers in this pool | +
| jvm.classes.loaded | +class | +The number of classes that are currently loaded in the Java virtual machine | +
| jvm.classes.unloaded | +class | +The total number of classes unloaded since the Java virtual machine has started execution | +
| jvm.gc.concurrent.phase.time | +second | +Time spent in concurrent phase | +
| jvm.gc.live.data.size | +byte | +Size of long-lived heap memory pool after reclamation | +
| jvm.gc.max.data.size | +byte | +Max size of long-lived heap memory pool | +
| jvm.gc.memory.allocated | +byte | +Incremented for an increase in the size of the (young) heap memory pool after one GC to before the next | +
| jvm.gc.memory.promoted | +byte | +Count of positive increases in the size of the old generation memory pool before GC to after GC | +
| jvm.gc.overhead | +percentage | +An approximation of the percent of CPU time used by GC activities | +
| jvm.gc.pause | +second | +Time spent in GC pause | +
| jvm.memory.committed | +byte | +The amount of memory in bytes that is committed for the Java virtual machine to use | +
| jvm.memory.max | +byte | +The maximum amount of memory in bytes that can be used for memory management | +
| jvm.memory.usage.after.gc | +percentage | +The percentage of long-lived heap pool used after the last GC event | +
| jvm.memory.used | +byte | +The amount of used memory | +
| jvm.threads.daemon | +thread | +The current number of live daemon threads | +
| jvm.threads.live | +thread | +The current number of live threads including both daemon and non-daemon threads | +
| jvm.threads.peak | +thread | +The peak live thread count since the Java virtual machine started or peak was reset | +
| jvm.threads.started | +thread | +The total number of application threads started in the JVM | +
| jvm.threads.states | +thread | +The current number of threads (in each state) | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| cluster-controller.down.count | +node | +last, max | +Number of content nodes down | +
| cluster-controller.maintenance.count | +node | +last, max | +Number of content nodes in maintenance | +
| cluster-controller.up.count | +node | +last, max | +Number of content nodes up | +
| cluster-controller.is-master | +binary | +last, max | +1 if this cluster controller is currently the master, or 0 if not | +
| cluster-controller.resource_usage.nodes_above_limit | +node | +last, max | +The number of content nodes above resource limit, blocking feed | +
| cluster-controller.resource_usage.max_memory_utilization | +fraction | +last, max | +Current memory utilisation, for content node with the highest value | +
| cluster-controller.resource_usage.max_disk_utilization | +fraction | +last, max | +Current disk space utilisation, for content node with the highest value | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| http.status.1xx | +response | +rate | +Number of responses with a 1xx status | +
| http.status.2xx | +response | +rate | +Number of responses with a 2xx status | +
| http.status.3xx | +response | +rate | +Number of responses with a 3xx status | +
| http.status.4xx | +response | +rate | +Number of responses with a 4xx status | +
| http.status.5xx | +response | +rate | +Number of responses with a 5xx status | +
| jdisc.gc.ms | +millisecond | +average, max | +Time spent in JVM garbage collection | +
| jdisc.thread_pool.work_queue.capacity | +thread | +max | +Capacity of the task queue | +
| jdisc.thread_pool.work_queue.size | +thread | +count, max, min, sum | +Size of the task queue | +
| jdisc.thread_pool.size | +thread | +max | +Size of the thread pool | +
| jdisc.thread_pool.active_threads | +thread | +count, max, min, sum | +Number of threads that are active | +
| jdisc.application.failed_component_graphs | +item | +rate | +JDISC Application failed component graphs | +
| jdisc.singleton.is_active | +item | +last, max | +JDISC Singleton is active | +
| jdisc.http.ssl.handshake.failure.missing_client_cert | +operation | +rate | +JDISC HTTP SSL Handshake failures due to missing client certificate | +
| jdisc.http.ssl.handshake.failure.incompatible_protocols | +operation | +rate | +JDISC HTTP SSL Handshake failures due to incompatible protocols | +
| jdisc.http.ssl.handshake.failure.incompatible_chifers | +operation | +rate | +JDISC HTTP SSL Handshake failures due to incompatible chifers | +
| jdisc.http.ssl.handshake.failure.unknown | +operation | +rate | +JDISC HTTP SSL Handshake failures for unknown reason | +
| jdisc.http.latency | +millisecond | +count, max, sum | +Request latency including the HTTP layer | +
| mem.heap.free | +byte | +average | +Free heap memory | +
| athenz-tenant-cert.expiry.seconds | +second | +last, max, min | +Time remaining until Athenz tenant certificate expires | +
| feed.operations | +operation | +rate | +Number of document feed operations | +
| feed.latency | +millisecond | +count, sum | +Feed latency | +
| queries | +operation | +rate | +Query volume | +
| query_latency | +millisecond | +average, count, max, sum | +The overall query latency as observed by the container cluster, excluding HTTP layer and rendering | +
| failed_queries | +operation | +rate | +The number of failed queries | +
| degraded_queries | +operation | +rate | +The number of degraded queries, e.g. due to some content nodes not responding in time | +
| hits_per_query | +hit_per_query | +average, count, max, sum | +The number of hits returned | +
| docproc.documents | +document | +sum | +Number of processed documents | +
| totalhits_per_query | +hit_per_query | +average, count, max, sum | +The total number of documents found to match queries | +
| serverActiveThreads | +thread | +average | +Deprecated. Use jdisc.thread_pool.active_threads instead. | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| vds.distributor.docsstored | +document | +average | +Number of documents stored in all buckets controlled by this distributor | +
| vds.bouncer.clock_skew_aborts | +operation | +count | +Number of client operations that were aborted due to clock skew between sender and receiver exceeding acceptable range | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| endpoint.certificate.expiry.seconds | +second | +N/A | +Time until node endpoint certificate expires | +
| node-certificate.expiry.seconds | +second | +N/A | +Time until node certificate expires | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| content.proton.documentdb.documents.total | +document | +last, max | +The total number of documents in this documents db (ready + not-ready) | +
| content.proton.documentdb.documents.ready | +document | +last, max | +The number of ready documents in this document db | +
| content.proton.documentdb.documents.active | +document | +last, max | +The number of active / searchable documents in this document db | +
| content.proton.documentdb.disk_usage | +byte | +last | +The total disk usage (in bytes) for this document db | +
| content.proton.documentdb.memory_usage.allocated_bytes | +byte | +last | +The number of allocated bytes | +
| content.proton.search_protocol.query.latency | +second | +average, count, max, sum | +Query request latency (seconds) | +
| content.proton.search_protocol.docsum.latency | +second | +average, count, max, sum | +Docsum request latency (seconds) | +
| content.proton.search_protocol.docsum.requested_documents | +document | +rate | +Total requested document summaries | +
| content.proton.resource_usage.disk | +fraction | +average | +The relative amount of disk used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller | +
| content.proton.resource_usage.memory | +fraction | +average | +The relative amount of memory used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller | +
| content.proton.resource_usage.feeding_blocked | +binary | +last, max | +Whether feeding is blocked due to resource limits being reached (value is either 0 or 1) | +
| content.proton.transactionlog.disk_usage | +byte | +last | +The disk usage (in bytes) of the transaction log | +
| content.proton.documentdb.matching.docs_matched | +document | +rate | +Number of documents matched | +
| content.proton.documentdb.matching.docs_reranked | +document | +rate | +Number of documents re-ranked (second phase) | +
| content.proton.documentdb.matching.rank_profile.query_latency | +second | +average, count, max, sum | +Total average latency (sec) when matching and ranking a query | +
| content.proton.documentdb.matching.rank_profile.query_setup_time | +second | +average, count, max, sum | +Average time (sec) spent setting up and tearing down queries | +
| content.proton.documentdb.matching.rank_profile.rerank_time | +second | +average, count, max, sum | +Average time (sec) spent on 2nd phase ranking | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| sentinel.totalRestarts | +restart | +last, max, sum | +Total number of service restarts done by the sentinel since the sentinel was started | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| vds.filestor.allthreads.put.count | +operation | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.remove.count | +operation | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.update.count | +request | +rate | +Number of requests processed. | +
| Name | +Unit | +Description | +
|---|---|---|
| vds.idealstate.buckets_rechecking | +bucket | +The number of buckets that we are rechecking for ideal state operations | +
| vds.idealstate.idealstate_diff | +bucket | +A number representing the current difference from the ideal state. This is a number that decreases steadily as the system is getting closer to the ideal state | +
| vds.idealstate.buckets_toofewcopies | +bucket | +The number of buckets the distributor controls that have less than the desired redundancy | +
| vds.idealstate.buckets_toomanycopies | +bucket | +The number of buckets the distributor controls that have more than the desired redundancy | +
| vds.idealstate.buckets | +bucket | +The number of buckets the distributor controls | +
| vds.idealstate.buckets_notrusted | +bucket | +The number of buckets that have no trusted copies. | +
| vds.idealstate.bucket_replicas_moving_out | +bucket | +Bucket replicas that should be moved out, e.g. retirement case or node added to cluster that has higher ideal state priority. | +
| vds.idealstate.bucket_replicas_copying_out | +bucket | +Bucket replicas that should be copied out, e.g. node is in ideal state but might have to provide data other nodes in a merge | +
| vds.idealstate.bucket_replicas_copying_in | +bucket | +Bucket replicas that should be copied in, e.g. node does not have a replica for a bucket that it is in ideal state for | +
| vds.idealstate.bucket_replicas_syncing | +bucket | +Bucket replicas that need syncing due to mismatching metadata | +
| vds.idealstate.max_observed_time_since_last_gc_sec | +second | +Maximum time (in seconds) since GC was last successfully run for a bucket. Aggregated max value across all buckets on the distributor. | +
| vds.idealstate.delete_bucket.done_ok | +operation | +The number of operations successfully performed | +
| vds.idealstate.delete_bucket.done_failed | +operation | +The number of operations that failed | +
| vds.idealstate.delete_bucket.pending | +operation | +The number of operations pending | +
| vds.idealstate.delete_bucket.blocked | +operation | +The number of operations blocked by blocking operation starter | +
| vds.idealstate.delete_bucket.throttled | +operation | +The number of operations throttled by throttling operation starter | +
| vds.idealstate.merge_bucket.done_ok | +operation | +The number of operations successfully performed | +
| vds.idealstate.merge_bucket.done_failed | +operation | +The number of operations that failed | +
| vds.idealstate.merge_bucket.pending | +operation | +The number of operations pending | +
| vds.idealstate.merge_bucket.blocked | +operation | +The number of operations blocked by blocking operation starter | +
| vds.idealstate.merge_bucket.throttled | +operation | +The number of operations throttled by throttling operation starter | +
| vds.idealstate.merge_bucket.source_only_copy_changed | +operation | +The number of merge operations where source-only copy changed | +
| vds.idealstate.merge_bucket.source_only_copy_delete_blocked | +operation | +The number of merge operations where delete of unchanged source-only copies was blocked | +
| vds.idealstate.merge_bucket.source_only_copy_delete_failed | +operation | +The number of merge operations where delete of unchanged source-only copies failed | +
| vds.idealstate.split_bucket.done_ok | +operation | +The number of operations successfully performed | +
| vds.idealstate.split_bucket.done_failed | +operation | +The number of operations that failed | +
| vds.idealstate.split_bucket.pending | +operation | +The number of operations pending | +
| vds.idealstate.split_bucket.blocked | +operation | +The number of operations blocked by blocking operation starter | +
| vds.idealstate.split_bucket.throttled | +operation | +The number of operations throttled by throttling operation starter | +
| vds.idealstate.join_bucket.done_ok | +operation | +The number of operations successfully performed | +
| vds.idealstate.join_bucket.done_failed | +operation | +The number of operations that failed | +
| vds.idealstate.join_bucket.pending | +operation | +The number of operations pending | +
| vds.idealstate.join_bucket.blocked | +operation | +The number of operations blocked by blocking operation starter | +
| vds.idealstate.join_bucket.throttled | +operation | +The number of operations throttled by throttling operation starter | +
| vds.idealstate.garbage_collection.done_ok | +operation | +The number of operations successfully performed | +
| vds.idealstate.garbage_collection.done_failed | +operation | +The number of operations that failed | +
| vds.idealstate.garbage_collection.pending | +operation | +The number of operations pending | +
| vds.idealstate.garbage_collection.documents_removed | +document | +Number of documents removed by GC operations | +
| vds.idealstate.garbage_collection.blocked | +operation | +The number of operations blocked by blocking operation starter | +
| vds.idealstate.garbage_collection.throttled | +operation | +The number of operations throttled by throttling operation starter | +
| vds.distributor.puts.latency | +millisecond | +The latency of put operations | +
| vds.distributor.puts.ok | +operation | +The number of successful put operations performed | +
| vds.distributor.puts.failures.total | +operation | +Sum of all failures | +
| vds.distributor.puts.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.puts.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.puts.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.puts.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.puts.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.puts.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.puts.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.puts.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.puts.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.puts.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.puts.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.removes.latency | +millisecond | +The latency of remove operations | +
| vds.distributor.removes.ok | +operation | +The number of successful removes operations performed | +
| vds.distributor.removes.failures.total | +operation | +Sum of all failures | +
| vds.distributor.removes.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.removes.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.removes.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.removes.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.removes.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.removes.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.removes.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.removes.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.removes.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.removes.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.removes.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.updates.latency | +millisecond | +The latency of update operations | +
| vds.distributor.updates.ok | +operation | +The number of successful updates operations performed | +
| vds.distributor.updates.failures.total | +operation | +Sum of all failures | +
| vds.distributor.updates.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.updates.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.updates.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.updates.diverging_timestamp_updates | +operation | +Number of updates that report they were performed against divergent version timestamps on different replicas | +
| vds.distributor.updates.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.updates.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.updates.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.updates.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.updates.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.updates.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.updates.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.updates.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.updates.fast_path_restarts | +operation | +Number of safe path (write repair) updates that were restarted as fast path updates because all replicas returned documents with the same timestamp in the initial read phase | +
| vds.distributor.removelocations.ok | +operation | +The number of successful removelocations operations performed | +
| vds.distributor.removelocations.failures.total | +operation | +Sum of all failures | +
| vds.distributor.removelocations.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.removelocations.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.removelocations.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.removelocations.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.removelocations.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.removelocations.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.removelocations.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.removelocations.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.removelocations.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.removelocations.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.removelocations.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.removelocations.latency | +millisecond | +The average latency of removelocations operations | +
| vds.distributor.gets.latency | +millisecond | +The average latency of gets operations | +
| vds.distributor.gets.ok | +operation | +The number of successful gets operations performed | +
| vds.distributor.gets.failures.total | +operation | +Sum of all failures | +
| vds.distributor.gets.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.gets.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.gets.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.gets.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.gets.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.gets.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.gets.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.gets.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.gets.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.gets.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.gets.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.visitor.latency | +millisecond | +The average latency of visitor operations | +
| vds.distributor.visitor.ok | +operation | +The number of successful visitor operations performed | +
| vds.distributor.visitor.failures.total | +operation | +Sum of all failures | +
| vds.distributor.visitor.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.visitor.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.visitor.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.visitor.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.visitor.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.visitor.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.visitor.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.visitor.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.visitor.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.visitor.bytes_per_visitor | +operation | +The number of bytes visited on content nodes as part of a single client visitor command | +
| vds.distributor.visitor.docs_per_visitor | +operation | +The number of documents visited on content nodes as part of a single client visitor command | +
| vds.distributor.visitor.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.visitor.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.docsstored | +document | +Number of documents stored in all buckets controlled by this distributor | +
| vds.distributor.bytesstored | +byte | +Number of bytes stored in all buckets controlled by this distributor | +
| metricmanager.periodichooklatency | +millisecond | +Time in ms used to update a single periodic hook | +
| metricmanager.resetlatency | +millisecond | +Time in ms used to reset all metrics. | +
| metricmanager.sleeptime | +millisecond | +Time in ms worker thread is sleeping | +
| metricmanager.snapshothooklatency | +millisecond | +Time in ms used to update a single snapshot hook | +
| metricmanager.snapshotlatency | +millisecond | +Time in ms used to take a snapshot | +
| vds.distributor.activate_cluster_state_processing_time | +millisecond | +Elapsed time where the distributor thread is blocked on merging pending bucket info into its bucket database upon activating a cluster state | +
| vds.distributor.bucket_db.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| vds.distributor.bucket_db.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| vds.distributor.bucket_db.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| vds.distributor.bucket_db.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| vds.distributor.getbucketlists.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.getbucketlists.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.getbucketlists.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.getbucketlists.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.getbucketlists.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.getbucketlists.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.getbucketlists.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.getbucketlists.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.getbucketlists.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.getbucketlists.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.getbucketlists.failures.total | +operation | +Total number of failures | +
| vds.distributor.getbucketlists.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.getbucketlists.latency | +millisecond | +The average latency of getbucketlists operations | +
| vds.distributor.getbucketlists.ok | +operation | +The number of successful getbucketlists operations performed | +
| vds.distributor.recoverymodeschedulingtime | +millisecond | +Time spent scheduling operations in recovery mode after receiving new cluster state | +
| vds.distributor.set_cluster_state_processing_time | +millisecond | +Elapsed time where the distributor thread is blocked on processing its bucket database upon receiving a new cluster state | +
| vds.distributor.state_transition_time | +millisecond | +Time it takes to complete a cluster state transition. If a state transition is preempted before completing, its elapsed time is counted as part of the total time spent for the final, completed state transition | +
| vds.distributor.stats.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.stats.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.stats.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.stats.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.stats.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.stats.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.stats.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.stats.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.stats.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.stats.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.stats.failures.total | +operation | +The total number of failures | +
| vds.distributor.stats.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.stats.latency | +millisecond | +The average latency of stats operations | +
| vds.distributor.stats.ok | +operation | +The number of successful stats operations performed | +
| vds.distributor.update_gets.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.update_gets.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.update_gets.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.update_gets.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.update_gets.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.update_gets.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.update_gets.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.update_gets.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.update_gets.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.update_gets.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.update_gets.failures.total | +operation | +The total number of failures | +
| vds.distributor.update_gets.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.update_gets.latency | +millisecond | +The average latency of update_gets operations | +
| vds.distributor.update_gets.ok | +operation | +The number of successful update_gets operations performed | +
| vds.distributor.update_metadata_gets.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.update_metadata_gets.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.update_metadata_gets.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.update_metadata_gets.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.update_metadata_gets.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.update_metadata_gets.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.update_metadata_gets.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.update_metadata_gets.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.update_metadata_gets.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.update_metadata_gets.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.update_metadata_gets.failures.total | +operation | +The total number of failures | +
| vds.distributor.update_metadata_gets.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.update_metadata_gets.latency | +millisecond | +The average latency of update_metadata_gets operations | +
| vds.distributor.update_metadata_gets.ok | +operation | +The number of successful update_metadata_gets operations performed | +
| vds.distributor.update_puts.failures.busy | +operation | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.update_puts.failures.concurrent_mutations | +operation | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.update_puts.failures.inconsistent_bucket | +operation | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.update_puts.failures.notconnected | +operation | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.update_puts.failures.notfound | +operation | +The number of operations that failed because the document did not exist | +
| vds.distributor.update_puts.failures.notready | +operation | +The number of operations discarded because distributor was not ready | +
| vds.distributor.update_puts.failures.safe_time_not_reached | +operation | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.update_puts.failures.storagefailure | +operation | +The number of operations that failed in storage | +
| vds.distributor.update_puts.failures.test_and_set_failed | +operation | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.update_puts.failures.timeout | +operation | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.update_puts.failures.total | +operation | +The total number of put failures | +
| vds.distributor.update_puts.failures.wrongdistributor | +operation | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.update_puts.latency | +millisecond | +The average latency of update_puts operations | +
| vds.distributor.update_puts.ok | +operation | +The number of successful update_puts operations performed | +
| vds.distributor.mutating_op_memory_usage | +byte | +Estimated amount of memory used by active mutating operations across all distributor stripes, in bytes | +
| vds.idealstate.nodes_per_merge | +node | +The number of nodes involved in a single merge operation. | +
| vds.idealstate.set_bucket_state.blocked | +operation | +The number of operations blocked by blocking operation starter | +
| vds.idealstate.set_bucket_state.done_failed | +operation | +The number of operations that failed | +
| vds.idealstate.set_bucket_state.done_ok | +operation | +The number of operations successfully performed | +
| vds.idealstate.set_bucket_state.pending | +operation | +The number of operations pending | +
| vds.idealstate.set_bucket_state.throttled | +operation | +The number of operations throttled by throttling operation starter | +
| vds.bouncer.clock_skew_aborts | +operation | +Number of client operations that were aborted due to clock skew between sender and receiver exceeding acceptable range | +
| Name | +Unit | +Description | +
|---|---|---|
| logd.processed.lines | +item | +Number of log lines processed | +
| Unit | +Description | +
|---|---|
| binary | +Zero or one. Zero typically indicate "false" while one indicate "true" | +
| bucket | +A chunk of documents managed by a distributor service | +
| buffer | +A buffer | +
| byte | +A collection of 8 bits | +
| byte/second | +A unit of storage capable of holding 8 bits | +
| class | +A instance of a Java class | +
| connection | +A link used for communication between a client and a server | +
| context switch | +A context switch | +
| deployment | +A deployment on hosted Vespa | +
| distance | +A number describing the distance of two tensors | +
| document | +Vespa document, a collection of fields defined in a schema file | +
| documentid | +A unique document identifier | +
| dollar | +US dollar | +
| dollar/hour | +Total current cost of the cluster in $/hr | +
| failure | +Failures, typically for requests, operations or nodes | +
| file | +Data file stored on the disk on a node | +
| fraction | +A value in the range [0..1]. Higher values can occur for some metrics, but would indicate the value is outside the allowed range. | +
| ratio | +A dimensionless ratio between two values. | +
| generation | +Typically, generation of configuration or application package | +
| gigabyte | +One billion bytes | +
| graph node | +A node in a graph | +
| hit | +Document that meets the filtering/restriction criteria specified by a given query | +
| hit/query | +Number of hits per query over a period of time | +
| host | +Bare metal computer that contain nodes | +
| instance | +Typically, tenant or application | +
| item | +Object or unit maintained in e.g. a queue | +
| millisecond | +Millisecond, 1/1000 of a second | +
| nanosecond | +Nanosecond, 1/1000.000.000 of a second | +
| node | +(Virtual) computer that is part of a Vespa cluster | +
| packet | +Collection of data transmitted over the network as a single unit | +
| operation | +A clearly defined task | +
| operation/second | +Number of operations per second | +
| percentage | +A number expressed as a fraction of 100, normally in the range [0..100]. | +
| query | +A request for matching, grouping and/or scoring documents stored in Vespa | +
| query/second | +Number of queries per second. | +
| record | +A collection of information, typically a set of key/value, e.g. stored in a transaction log | +
| request | +A request sent from a client to a server | +
| response | +A response from a server to a client, typically as a response to a request | +
| restart | +A service or node restarts | +
| routing rotation | +Routing rotation | +
| score | +Relevance score for a document | +
| second | +Time span of 1 second | +
| seconds since epoch | +Seconds since Unix Epoch | +
| session | +A set of operations taking place during one connection or as part of a higher level operation | +
| task | +Piece of work executed by a server, e.g. to perform back-ground data maintenance | +
| tenant | +Tenant that owns zero or more applications in a managed Vespa system | +
| thread | +Computer thread for executing e.g. tasks, operations or queries | +
| vcpu | +Virtual CPU | +
| version | +Software or config version | +
| wakeup | +Computer thread wake-ups for doing some work | +
| Aggregator name (metric suffix) | +Explanation | +
|---|---|
| 95percentile | +The 95 percentile of samples in the period | +
| 99percentile | +The 99 percentile of samples in the period | +
| average | +The average of samples in the period | +
| count | +The count of samples in the period | +
| last | +The last value sampled in the period | +
| max | +The max value sampled in the period | +
| min | +The min value sampled in the period | +
| rate | +The count of samples divided by the length of the period in seconds | +
| sum | +The sum of the sampled values in the period | +
| Name | +Unit | +Description | +
|---|---|---|
| endpoint.certificate.expiry.seconds | +second | +Time until node endpoint certificate expires | +
| node-certificate.expiry.seconds | +second | +Time until node certificate expires | +
| Name | +Unit | +Description | +
|---|---|---|
| content.proton.config.generation | +version | +The oldest config generation used by this search node | +
| content.proton.documentdb.documents.total | +document | +The total number of documents in this documents db (ready + not-ready) | +
| content.proton.documentdb.documents.ready | +document | +The number of ready documents in this document db | +
| content.proton.documentdb.documents.active | +document | +The number of active / searchable documents in this document db | +
| content.proton.documentdb.documents.removed | +document | +The number of removed documents in this document db | +
| content.proton.documentdb.index.docs_in_memory | +document | +Number of documents in memory index | +
| content.proton.documentdb.disk_usage | +byte | +The total disk usage (in bytes) for this document db | +
| content.proton.documentdb.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| content.proton.documentdb.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| content.proton.documentdb.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.heart_beat_age | +second | +How long ago (in seconds) heart beat maintenance job was run | +
| content.proton.docsum.count | +request | +Docsum requests handled | +
| content.proton.docsum.docs | +document | +Total docsums returned | +
| content.proton.docsum.latency | +millisecond | +Docsum request latency | +
| content.proton.search_protocol.query.latency | +second | +Query request latency (seconds) | +
| content.proton.search_protocol.query.request_size | +byte | +Query request size (network bytes) | +
| content.proton.search_protocol.query.reply_size | +byte | +Query reply size (network bytes) | +
| content.proton.search_protocol.docsum.latency | +second | +Docsum request latency (seconds) | +
| content.proton.search_protocol.docsum.request_size | +byte | +Docsum request size (network bytes) | +
| content.proton.search_protocol.docsum.reply_size | +byte | +Docsum reply size (network bytes) | +
| content.proton.search_protocol.docsum.requested_documents | +document | +Total requested document summaries | +
| content.proton.executor.proton.queuesize | +task | +Size of executor proton task queue | +
| content.proton.executor.proton.accepted | +task | +Number of executor proton accepted tasks | +
| content.proton.executor.proton.wakeups | +wakeup | +Number of times an executor proton worker thread has been woken up | +
| content.proton.executor.proton.utilization | +fraction | +Ratio of time the executor proton worker threads has been active | +
| content.proton.executor.proton.rejected | +task | +Number of rejected tasks | +
| content.proton.executor.flush.queuesize | +task | +Size of executor flush task queue | +
| content.proton.executor.flush.accepted | +task | +Number of accepted executor flush tasks | +
| content.proton.executor.flush.wakeups | +wakeup | +Number of times an executor flush worker thread has been woken up | +
| content.proton.executor.flush.utilization | +fraction | +Ratio of time the executor flush worker threads has been active | +
| content.proton.executor.flush.rejected | +task | +Number of rejected tasks | +
| content.proton.executor.match.queuesize | +task | +Size of executor match task queue | +
| content.proton.executor.match.accepted | +task | +Number of accepted executor match tasks | +
| content.proton.executor.match.wakeups | +wakeup | +Number of times an executor match worker thread has been woken up | +
| content.proton.executor.match.utilization | +fraction | +Ratio of time the executor match worker threads has been active | +
| content.proton.executor.match.rejected | +task | +Number of rejected tasks | +
| content.proton.executor.docsum.queuesize | +task | +Size of executor docsum task queue | +
| content.proton.executor.docsum.accepted | +task | +Number of executor accepted docsum tasks | +
| content.proton.executor.docsum.wakeups | +wakeup | +Number of times an executor docsum worker thread has been woken up | +
| content.proton.executor.docsum.utilization | +fraction | +Ratio of time the executor docsum worker threads has been active | +
| content.proton.executor.docsum.rejected | +task | +Number of rejected tasks | +
| content.proton.executor.shared.queuesize | +task | +Size of executor shared task queue | +
| content.proton.executor.shared.accepted | +task | +Number of executor shared accepted tasks | +
| content.proton.executor.shared.wakeups | +wakeup | +Number of times an executor shared worker thread has been woken up | +
| content.proton.executor.shared.utilization | +fraction | +Ratio of time the executor shared worker threads has been active | +
| content.proton.executor.shared.rejected | +task | +Number of rejected tasks | +
| content.proton.executor.warmup.queuesize | +task | +Size of executor warmup task queue | +
| content.proton.executor.warmup.accepted | +task | +Number of accepted executor warmup tasks | +
| content.proton.executor.warmup.wakeups | +wakeup | +Number of times a warmup executor worker thread has been woken up | +
| content.proton.executor.warmup.utilization | +fraction | +Ratio of time the executor warmup worker threads has been active | +
| content.proton.executor.warmup.rejected | +task | +Number of rejected tasks | +
| content.proton.executor.field_writer.queuesize | +task | +Size of executor field writer task queue | +
| content.proton.executor.field_writer.accepted | +task | +Number of accepted executor field writer tasks | +
| content.proton.executor.field_writer.wakeups | +wakeup | +Number of times an executor field writer worker thread has been woken up | +
| content.proton.executor.field_writer.utilization | +fraction | +Ratio of time the executor fieldwriter worker threads has been active | +
| content.proton.executor.field_writer.saturation | +fraction | +Ratio indicating the max saturation of underlying worker threads. A higher saturation than utilization indicates a bottleneck in one of the worker threads. | +
| content.proton.executor.field_writer.rejected | +task | +Number of rejected tasks | +
| content.proton.documentdb.job.total | +fraction | +The job load average total of all job metrics | +
| content.proton.documentdb.job.attribute_flush | +fraction | +Flushing of attribute vector(s) to disk | +
| content.proton.documentdb.job.memory_index_flush | +fraction | +Flushing of memory index to disk | +
| content.proton.documentdb.job.disk_index_fusion | +fraction | +Fusion of disk indexes | +
| content.proton.documentdb.job.document_store_flush | +fraction | +Flushing of document store to disk | +
| content.proton.documentdb.job.document_store_compact | +fraction | +Compaction of document store on disk | +
| content.proton.documentdb.job.bucket_move | +fraction | +Moving of buckets between 'ready' and 'notready' sub databases | +
| content.proton.documentdb.job.lid_space_compact | +fraction | +Compaction of lid space in document meta store and attribute vectors | +
| content.proton.documentdb.job.removed_documents_prune | +fraction | +Pruning of removed documents in 'removed' sub database | +
| content.proton.documentdb.threading_service.master.queuesize | +task | +Size of threading service master task queue | +
| content.proton.documentdb.threading_service.master.accepted | +task | +Number of accepted threading service master tasks | +
| content.proton.documentdb.threading_service.master.wakeups | +wakeup | +Number of times a threading service master worker thread has been woken up | +
| content.proton.documentdb.threading_service.master.utilization | +fraction | +Ratio of time the threading service master worker threads has been active | +
| content.proton.documentdb.threading_service.master.rejected | +task | +Number of rejected tasks | +
| content.proton.documentdb.threading_service.index.queuesize | +task | +Size of threading service index task queue | +
| content.proton.documentdb.threading_service.index.accepted | +task | +Number of accepted threading service index tasks | +
| content.proton.documentdb.threading_service.index.wakeups | +wakeup | +Number of times a threading service index worker thread has been woken up | +
| content.proton.documentdb.threading_service.index.utilization | +fraction | +Ratio of time the threading service index worker threads has been active | +
| content.proton.documentdb.threading_service.index.rejected | +task | +Number of rejected tasks | +
| content.proton.documentdb.threading_service.summary.queuesize | +task | +Size of threading service summary task queue | +
| content.proton.documentdb.threading_service.summary.accepted | +task | +Number of accepted threading service summary tasks | +
| content.proton.documentdb.threading_service.summary.wakeups | +wakeup | +Number of times a threading service summary worker thread has been woken up | +
| content.proton.documentdb.threading_service.summary.utilization | +fraction | +Ratio of time the threading service summary worker threads has been active | +
| content.proton.documentdb.threading_service.summary.rejected | +task | +Number of rejected tasks | +
| content.proton.documentdb.threading_service.attribute_field_writer.accepted | +task | +Number of accepted tasks | +
| content.proton.documentdb.threading_service.attribute_field_writer.queuesize | +task | +Size of task queue | +
| content.proton.documentdb.threading_service.attribute_field_writer.rejected | +task | +Number of rejected tasks | +
| content.proton.documentdb.threading_service.attribute_field_writer.utilization | +fraction | +Ratio of time the worker threads has been active | +
| content.proton.documentdb.threading_service.attribute_field_writer.wakeups | +wakeup | +Number of times a worker thread has been woken up | +
| content.proton.documentdb.threading_service.index_field_inverter.accepted | +task | +Number of accepted tasks | +
| content.proton.documentdb.threading_service.index_field_inverter.queuesize | +task | +Size of task queue | +
| content.proton.documentdb.threading_service.index_field_inverter.rejected | +task | +Number of rejected tasks | +
| content.proton.documentdb.threading_service.index_field_inverter.utilization | +fraction | +Ratio of time the worker threads has been active | +
| content.proton.documentdb.threading_service.index_field_inverter.wakeups | +wakeup | +Number of times a worker thread has been woken up | +
| content.proton.documentdb.threading_service.index_field_writer.accepted | +task | +Number of accepted tasks | +
| content.proton.documentdb.threading_service.index_field_writer.queuesize | +task | +Size of task queue | +
| content.proton.documentdb.threading_service.index_field_writer.rejected | +task | +Number of rejected tasks | +
| content.proton.documentdb.threading_service.index_field_writer.utilization | +fraction | +Ratio of time the worker threads has been active | +
| content.proton.documentdb.threading_service.index_field_writer.wakeups | +wakeup | +Number of times a worker thread has been woken up | +
| content.proton.documentdb.ready.lid_space.lid_bloat_factor | +fraction | +The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) | +
| content.proton.documentdb.ready.lid_space.lid_fragmentation_factor | +fraction | +The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) | +
| content.proton.documentdb.ready.lid_space.lid_limit | +documentid | +The size of the allocated lid space | +
| content.proton.documentdb.ready.lid_space.highest_used_lid | +documentid | +The highest used lid | +
| content.proton.documentdb.ready.lid_space.used_lids | +documentid | +The number of lids used | +
| content.proton.documentdb.ready.lid_space.lowest_free_lid | +documentid | +The lowest free local document id | +
| content.proton.documentdb.notready.lid_space.lid_bloat_factor | +fraction | +The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) | +
| content.proton.documentdb.notready.lid_space.lid_fragmentation_factor | +fraction | +The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) | +
| content.proton.documentdb.notready.lid_space.lid_limit | +documentid | +The size of the allocated lid space | +
| content.proton.documentdb.notready.lid_space.highest_used_lid | +documentid | +The highest used lid | +
| content.proton.documentdb.notready.lid_space.used_lids | +documentid | +The number of lids used | +
| content.proton.documentdb.notready.lid_space.lowest_free_lid | +documentid | +The lowest free local document id | +
| content.proton.documentdb.removed.lid_space.lid_bloat_factor | +fraction | +The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) | +
| content.proton.documentdb.removed.lid_space.lid_fragmentation_factor | +fraction | +The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) | +
| content.proton.documentdb.removed.lid_space.lid_limit | +documentid | +The size of the allocated lid space | +
| content.proton.documentdb.removed.lid_space.highest_used_lid | +documentid | +The highest used lid | +
| content.proton.documentdb.removed.lid_space.used_lids | +documentid | +The number of lids used | +
| content.proton.documentdb.removed.lid_space.lowest_free_lid | +documentid | +The lowest free local document id | +
| content.proton.documentdb.bucket_move.buckets_pending | +bucket | +The number of buckets left to move | +
| content.proton.resource_usage.disk | +fraction | +The relative amount of disk used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller | +
| content.proton.resource_usage.disk_usage.total | +fraction | +The total relative amount of disk used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.disk_usage.total_utilization | +fraction | +The relative amount of disk used compared to the content node disk resource limit | +
| content.proton.resource_usage.disk_usage.transient | +fraction | +The relative amount of transient disk used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.disk_usage.reserved | +fraction | +The relative amount of reserved disk space for this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.disk_usage.used_and_reserved | +fraction | +The relative amount of disk used and reserved disk space by this content node (transient usage not included, value in the range [0, 1]) | +
| content.proton.resource_usage.memory | +fraction | +The relative amount of memory used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller | +
| content.proton.resource_usage.memory_usage.total | +fraction | +The total relative amount of memory used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.memory_usage.total_utilization | +fraction | +The relative amount of memory used compared to the content node memory resource limit | +
| content.proton.resource_usage.memory_usage.transient | +fraction | +The relative amount of transient memory used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.memory_mappings | +file | +The number of memory mapped files | +
| content.proton.resource_usage.open_file_descriptors | +file | +The number of open files | +
| content.proton.resource_usage.feeding_blocked | +binary | +Whether feeding is blocked due to resource limits being reached (value is either 0 or 1) | +
| content.proton.resource_usage.malloc_arena | +byte | +Size of malloc arena | +
| content.proton.documentdb.attribute.resource_usage.address_space | +fraction | +The max relative address space used among components in all attribute vectors in this document db (value in the range [0, 1]) | +
| content.proton.documentdb.attribute.resource_usage.feeding_blocked | +binary | +Whether feeding is blocked due to attribute resource limits being reached (value is either 0 or 1) | +
| content.proton.documentdb.attribute.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| content.proton.documentdb.attribute.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.attribute.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| content.proton.documentdb.attribute.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.resource_usage.cpu_util.setup | +fraction | +cpu used by system init and (re-)configuration | +
| content.proton.resource_usage.cpu_util.read | +fraction | +cpu used by reading data from the system | +
| content.proton.resource_usage.cpu_util.write | +fraction | +cpu used by writing data to the system | +
| content.proton.resource_usage.cpu_util.compact | +fraction | +cpu used by internal data re-structuring | +
| content.proton.resource_usage.cpu_util.other | +fraction | +cpu used by work not classified as a specific category | +
| content.proton.transactionlog.entries | +record | +The current number of entries in the transaction log | +
| content.proton.transactionlog.disk_usage | +byte | +The disk usage (in bytes) of the transaction log | +
| content.proton.transactionlog.replay_time | +second | +The replay time (in seconds) of the transaction log during start-up | +
| content.proton.documentdb.ready.document_store.disk_usage | +byte | +Disk space usage in bytes | +
| content.proton.documentdb.ready.document_store.disk_bloat | +byte | +Disk space bloat in bytes | +
| content.proton.documentdb.ready.document_store.max_bucket_spread | +fraction | +Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) | +
| content.proton.documentdb.ready.document_store.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| content.proton.documentdb.ready.document_store.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.ready.document_store.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.ready.document_store.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| content.proton.documentdb.notready.document_store.disk_usage | +byte | +Disk space usage in bytes | +
| content.proton.documentdb.notready.document_store.disk_bloat | +byte | +Disk space bloat in bytes | +
| content.proton.documentdb.notready.document_store.max_bucket_spread | +fraction | +Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) | +
| content.proton.documentdb.notready.document_store.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| content.proton.documentdb.notready.document_store.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.notready.document_store.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.notready.document_store.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| content.proton.documentdb.removed.document_store.disk_usage | +byte | +Disk space usage in bytes | +
| content.proton.documentdb.removed.document_store.disk_bloat | +byte | +Disk space bloat in bytes | +
| content.proton.documentdb.removed.document_store.max_bucket_spread | +fraction | +Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) | +
| content.proton.documentdb.removed.document_store.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| content.proton.documentdb.removed.document_store.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.removed.document_store.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.removed.document_store.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| content.proton.documentdb.ready.document_store.cache.elements | +item | +Number of elements in the cache | +
| content.proton.documentdb.ready.document_store.cache.memory_usage | +byte | +Memory usage of the cache (in bytes) | +
| content.proton.documentdb.ready.document_store.cache.hit_rate | +fraction | +Rate of hits in the cache compared to number of lookups | +
| content.proton.documentdb.ready.document_store.cache.lookups | +operation | +Number of lookups in the cache (hits + misses) | +
| content.proton.documentdb.ready.document_store.cache.invalidations | +operation | +Number of invalidations (erased elements) in the cache. | +
| content.proton.documentdb.notready.document_store.cache.elements | +item | +Number of elements in the cache | +
| content.proton.documentdb.notready.document_store.cache.memory_usage | +byte | +Memory usage of the cache (in bytes) | +
| content.proton.documentdb.notready.document_store.cache.hit_rate | +fraction | +Rate of hits in the cache compared to number of lookups | +
| content.proton.documentdb.notready.document_store.cache.lookups | +operation | +Number of lookups in the cache (hits + misses) | +
| content.proton.documentdb.notready.document_store.cache.invalidations | +operation | +Number of invalidations (erased elements) in the cache. | +
| content.proton.documentdb.removed.document_store.cache.elements | +item | +Number of elements in the cache | +
| content.proton.documentdb.removed.document_store.cache.hit_rate | +fraction | +Rate of hits in the cache compared to number of lookups | +
| content.proton.documentdb.removed.document_store.cache.invalidations | +item | +Number of invalidations (erased elements) in the cache. | +
| content.proton.documentdb.removed.document_store.cache.lookups | +operation | +Number of lookups in the cache (hits + misses) | +
| content.proton.documentdb.removed.document_store.cache.memory_usage | +byte | +Memory usage of the cache (in bytes) | +
| content.proton.documentdb.ready.attribute.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| content.proton.documentdb.ready.attribute.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.ready.attribute.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.ready.attribute.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| content.proton.documentdb.ready.attribute.disk_usage | +byte | +Disk space usage (in bytes) of the flushed snapshot of this attribute for this document type | +
| content.proton.documentdb.notready.attribute.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| content.proton.documentdb.notready.attribute.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.notready.attribute.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.notready.attribute.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| content.proton.index.cache.postinglist.elements | +item | +Number of elements in the cache. Contains disk index posting list files across all document types | +
| content.proton.index.cache.postinglist.memory_usage | +byte | +Memory usage of the cache (in bytes). Contains disk index posting list files across all document types | +
| content.proton.index.cache.postinglist.hit_rate | +fraction | +Rate of hits in the cache compared to number of lookups. Contains disk index posting list files across all document types | +
| content.proton.index.cache.postinglist.lookups | +operation | +Number of lookups in the cache (hits + misses). Contains disk index posting list files across all document types | +
| content.proton.index.cache.postinglist.invalidations | +operation | +Number of invalidations (erased elements) in the cache. Contains disk index posting list files across all document types | +
| content.proton.index.cache.bitvector.elements | +item | +Number of elements in the cache. Contains disk index bitvector files across all document types | +
| content.proton.index.cache.bitvector.memory_usage | +byte | +Memory usage of the cache (in bytes). Contains disk index bitvector files across all document types | +
| content.proton.index.cache.bitvector.hit_rate | +fraction | +Rate of hits in the cache compared to number of lookups. Contains disk index bitvector files across all document types | +
| content.proton.index.cache.bitvector.lookups | +operation | +Number of lookups in the cache (hits + misses). Contains disk index bitvector files across all document types | +
| content.proton.index.cache.bitvector.invalidations | +operation | +Number of invalidations (erased elements) in the cache. Contains disk index bitvector files across all document types | +
| content.proton.documentdb.index.memory_usage.allocated_bytes | +byte | +The number of allocated bytes for the memory index for this document type | +
| content.proton.documentdb.index.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) for the memory index for this document type |
+
| content.proton.documentdb.index.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) for the memory index for this document type |
+
| content.proton.documentdb.index.memory_usage.onhold_bytes | +byte | +The number of bytes on hold for the memory index for this document type | +
| content.proton.documentdb.index.disk_usage | +byte | +Disk space usage (in bytes) of all disk indexes for this document type | +
| content.proton.documentdb.index.indexes | +item | +Number of disk or memory indexes | +
| content.proton.documentdb.index.io.search.read_bytes | +byte | +Bytes read from disk index posting list and bitvector files as part of search for this document type | +
| content.proton.documentdb.index.io.search.cached_read_bytes | +byte | +Bytes read from cached disk index posting list and bitvector files as part of search for this document type | +
| content.proton.documentdb.ready.index.memory_usage.allocated_bytes | +byte | +The number of allocated bytes for this index field in the memory index for this document type | +
| content.proton.documentdb.ready.index.disk_usage | +byte | +Disk space usage (in bytes) of this index field in all disk indexes for this document type | +
| content.proton.documentdb.matching.queries | +query | +Number of queries executed | +
| content.proton.documentdb.matching.soft_doomed_queries | +query | +Number of queries hitting the soft timeout | +
| content.proton.documentdb.matching.query_latency | +second | +Total average latency (sec) when matching and ranking a query | +
| content.proton.documentdb.matching.query_setup_time | +second | +Average time (sec) spent setting up and tearing down queries | +
| content.proton.documentdb.matching.docs_matched | +document | +Number of documents matched | +
| content.proton.documentdb.matching.docs_ranked | +document | +Number of documents ranked (first phase) | +
| content.proton.documentdb.matching.docs_reranked | +document | +Number of documents re-ranked (second phase) | +
| content.proton.documentdb.matching.exact_nns_distances_computed | +distance | +Number of distances computed in exact nearest-neighbor search | +
| content.proton.documentdb.matching.approximate_nns_distances_computed | +distance | +Number of distances computed in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.approximate_nns_nodes_visited | +graph_node | +Number of nodes visited in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.queries | +query | +Number of queries executed | +
| content.proton.documentdb.matching.rank_profile.soft_doomed_queries | +query | +Number of queries hitting the soft timeout | +
| content.proton.documentdb.matching.rank_profile.soft_doom_factor | +fraction | +Factor used to compute soft-timeout | +
| content.proton.documentdb.matching.rank_profile.query_latency | +second | +Total average latency (sec) when matching and ranking a query | +
| content.proton.documentdb.matching.rank_profile.query_setup_time | +second | +Average time (sec) spent setting up and tearing down queries | +
| content.proton.documentdb.matching.rank_profile.grouping_time | +second | +Average time (sec) spent on grouping | +
| content.proton.documentdb.matching.rank_profile.rerank_time | +second | +Average time (sec) spent on 2nd phase ranking | +
| content.proton.documentdb.matching.rank_profile.docs_matched | +document | +Number of documents matched | +
| content.proton.documentdb.matching.rank_profile.docs_ranked | +document | +Number of documents ranked (first phase) | +
| content.proton.documentdb.matching.rank_profile.docs_reranked | +document | +Number of documents re-ranked (second phase) | +
| content.proton.documentdb.matching.rank_profile.exact_nns_distances_computed | +distance | +Number of distances computed in exact nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.approximate_nns_distances_computed | +distance | +Number of distances computed in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.approximate_nns_nodes_visited | +graph_node | +Number of nodes visited in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.limited_queries | +query | +Number of queries limited in match phase | +
| content.proton.documentdb.matching.rank_profile.docid_partition.active_time | +second | +Time (sec) spent doing actual work | +
| content.proton.documentdb.matching.rank_profile.docid_partition.docs_matched | +document | +Number of documents matched | +
| content.proton.documentdb.matching.rank_profile.docid_partition.docs_ranked | +document | +Number of documents ranked (first phase) | +
| content.proton.documentdb.matching.rank_profile.docid_partition.docs_reranked | +document | +Number of documents re-ranked (second phase) | +
| content.proton.documentdb.matching.rank_profile.docid_partition.wait_time | +second | +Time (sec) spent waiting for other external threads and resources | +
| content.proton.documentdb.matching.rank_profile.match_time | +second | +Average time (sec) for matching a query (1st phase) | +
| content.proton.documentdb.feeding.commit.operations | +operation | +Number of operations included in a commit | +
| content.proton.documentdb.feeding.commit.latency | +second | +Latency for commit in seconds | +
| content.proton.session_cache.grouping.num_cached | +session | +Number of currently cached sessions | +
| content.proton.session_cache.grouping.num_dropped | +session | +Number of dropped cached sessions | +
| content.proton.session_cache.grouping.num_insert | +session | +Number of inserted sessions | +
| content.proton.session_cache.grouping.num_pick | +session | +Number if picked sessions | +
| content.proton.session_cache.grouping.num_timedout | +session | +Number of timed out sessions | +
| content.proton.session_cache.search.num_cached | +session | +Number of currently cached sessions | +
| content.proton.session_cache.search.num_dropped | +session | +Number of dropped cached sessions | +
| content.proton.session_cache.search.num_insert | +session | +Number of inserted sessions | +
| content.proton.session_cache.search.num_pick | +session | +Number if picked sessions | +
| content.proton.session_cache.search.num_timedout | +session | +Number of timed out sessions | +
| metricmanager.periodichooklatency | +millisecond | +Time in ms used to update a single periodic hook | +
| metricmanager.resetlatency | +millisecond | +Time in ms used to reset all metrics. | +
| metricmanager.sleeptime | +millisecond | +Time in ms worker thread is sleeping | +
| metricmanager.snapshothooklatency | +millisecond | +Time in ms used to update a single snapshot hook | +
| metricmanager.snapshotlatency | +millisecond | +Time in ms used to take a snapshot | +
| Name | +Unit | +Description | +
|---|---|---|
| sentinel.restarts | +restart | +Number of service restarts done by the sentinel | +
| sentinel.totalRestarts | +restart | +Total number of service restarts done by the sentinel since the sentinel was started | +
| sentinel.uptime | +second | +Time the sentinel has been running | +
| sentinel.running | +instance | +Number of services the sentinel has running currently | +
| Name | +Unit | +Description | +
|---|---|---|
| slobrok.heartbeats.failed | +request | +Number of heartbeat requests failed | +
| slobrok.requests.register | +request | +Number of register requests received | +
| slobrok.requests.mirror | +request | +Number of mirroring requests received | +
| slobrok.requests.admin | +request | +Number of administrative requests received | +
| slobrok.missing.consensus | +second | +Number of seconds without full consensus with all other brokers | +
| Name | +Unit | +Description | +
|---|---|---|
| vds.datastored.alldisks.buckets | +bucket | +Number of buckets managed | +
| vds.datastored.alldisks.docs | +document | +Number of documents stored | +
| vds.datastored.alldisks.bytes | +byte | +Number of bytes stored | +
| vds.datastored.alldisks.activebuckets | +bucket | +Number of active buckets on the node | +
| vds.datastored.alldisks.readybuckets | +bucket | +Number of ready buckets on the node | +
| vds.visitor.allthreads.averagevisitorlifetime | +millisecond | +Average lifetime of a visitor | +
| vds.visitor.allthreads.averagequeuewait | +millisecond | +Average time an operation spends in input queue. | +
| vds.visitor.allthreads.queuesize | +operation | +Size of input message queue. | +
| vds.visitor.allthreads.completed | +operation | +Number of visitors completed | +
| vds.visitor.allthreads.created | +operation | +Number of visitors created. | +
| vds.visitor.allthreads.failed | +operation | +Number of visitors failed | +
| vds.visitor.allthreads.averagemessagesendtime | +millisecond | +Average time it takes for messages to be sent to their target (and be replied to) | +
| vds.visitor.allthreads.averageprocessingtime | +millisecond | +Average time used to process visitor requests | +
| vds.visitor.allthreads.aborted | +instance | +Number of visitors aborted. | +
| vds.visitor.allthreads.averagevisitorcreationtime | +millisecond | +Average time spent creating a visitor instance | +
| vds.visitor.allthreads.destination_failure_replies | +instance | +Number of failure replies received from the visitor destination | +
| vds.filestor.queuesize | +operation | +Size of input message queue. | +
| vds.filestor.averagequeuewait | +millisecond | +Average time an operation spends in input queue. | +
| vds.filestor.active_operations.size | +operation | +Number of concurrent active operations | +
| vds.filestor.active_operations.latency | +millisecond | +Latency (in ms) for completed operations | +
| vds.filestor.throttle_window_size | +operation | +Current size of async operation throttler window size | +
| vds.filestor.throttle_waiting_threads | +thread | +Number of threads waiting to acquire a throttle token | +
| vds.filestor.throttle_active_tokens | +instance | +Current number of active throttle tokens | +
| vds.filestor.allthreads.mergemetadatareadlatency | +millisecond | +Time spent in a merge step to check metadata of current node to see what data it has. | +
| vds.filestor.allthreads.mergedatareadlatency | +millisecond | +Time spent in a merge step to read data other nodes need. | +
| vds.filestor.allthreads.mergedatawritelatency | +millisecond | +Time spent in a merge step to write data needed to current node. | +
| vds.filestor.allthreads.mergeavgdatareceivedneeded | +byte | +Amount of data transferred from previous node in chain that we needed to apply locally. | +
| vds.filestor.allthreads.mergebuckets.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.mergebuckets.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.mergebuckets.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.mergelatencytotal | +millisecond | +Latency of total merge operation, from master node receives it, until merge is complete and master node replies. | +
| vds.filestor.allthreads.merge_put_latency | +millisecond | +Latency of individual puts that are part of merge operations | +
| vds.filestor.allthreads.merge_remove_latency | +millisecond | +Latency of individual removes that are part of merge operations | +
| vds.filestor.allstripes.throttled_rpc_direct_dispatches | +instance | +Number of times an RPC thread could not directly dispatch an async operation directly to Proton because it was disallowed by the throttle policy | +
| vds.filestor.allstripes.throttled_persistence_thread_polls | +instance | +Number of times a persistence thread could not immediately dispatch a queued async operation because it was disallowed by the throttle policy | +
| vds.filestor.allstripes.timeouts_waiting_for_throttle_token | +instance | +Number of times a persistence thread timed out waiting for an available throttle policy token | +
| vds.filestor.allstripes.averagequeuewait | +millisecond | +Average time an operation spends in input queue. | +
| vds.filestor.allthreads.put.count | +operation | +Number of requests processed. | +
| vds.filestor.allthreads.put.failed | +operation | +Number of failed requests. | +
| vds.filestor.allthreads.put.test_and_set_failed | +operation | +Number of operations that were skipped due to a test-and-set condition not met | +
| vds.filestor.allthreads.put.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.put.request_size | +byte | +Size of requests, in bytes | +
| vds.filestor.allthreads.remove.count | +operation | +Number of requests processed. | +
| vds.filestor.allthreads.remove.failed | +operation | +Number of failed requests. | +
| vds.filestor.allthreads.remove.test_and_set_failed | +operation | +Number of operations that were skipped due to a test-and-set condition not met | +
| vds.filestor.allthreads.remove.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.remove.request_size | +byte | +Size of requests, in bytes | +
| vds.filestor.allthreads.remove.not_found | +request | +Number of requests that could not be completed due to source document not found. | +
| vds.filestor.allthreads.get.count | +operation | +Number of requests processed. | +
| vds.filestor.allthreads.get.failed | +operation | +Number of failed requests. | +
| vds.filestor.allthreads.get.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.get.request_size | +byte | +Size of requests, in bytes | +
| vds.filestor.allthreads.get.not_found | +request | +Number of requests that could not be completed due to source document not found. | +
| vds.filestor.allthreads.update.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.update.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.update.test_and_set_failed | +request | +Number of requests that were skipped due to a test-and-set condition not met | +
| vds.filestor.allthreads.update.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.update.request_size | +byte | +Size of requests, in bytes | +
| vds.filestor.allthreads.update.latency_read | +millisecond | +Latency of the source read in the request. | +
| vds.filestor.allthreads.update.not_found | +request | +Number of requests that could not be completed due to source document not found. | +
| vds.filestor.allthreads.createiterator.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.createiterator.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.createiterator.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.visit.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.visit.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.visit.docs | +document | +Number of entries read per iterate call | +
| vds.filestor.allthreads.visit.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.remove_location.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.remove_location.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.remove_location.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.splitbuckets.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.splitbuckets.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.splitbuckets.latency | +request | +Latency of successful requests. | +
| vds.filestor.allthreads.joinbuckets.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.joinbuckets.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.joinbuckets.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.deletebuckets.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.deletebuckets.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.deletebuckets.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.remove_by_gid.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.remove_by_gid.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.remove_by_gid.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.setbucketstates.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.setbucketstates.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.setbucketstates.latency | +millisecond | +Latency of successful requests. | +
| vds.mergethrottler.averagequeuewaitingtime | +millisecond | +Time merges spent in the throttler queue | +
| vds.mergethrottler.queuesize | +instance | +Length of merge queue | +
| vds.mergethrottler.active_window_size | +instance | +Number of merges active within the pending window size | +
| vds.mergethrottler.estimated_merge_memory_usage | +byte | +An estimated upper bound of the memory usage (in bytes) of the merges currently in the active window | +
| vds.mergethrottler.bounced_due_to_back_pressure | +instance | +Number of merges bounced due to resource exhaustion back-pressure | +
| vds.mergethrottler.locallyexecutedmerges.ok | +instance | +The number of successful merges for 'locallyexecutedmerges' | +
| vds.mergethrottler.locallyexecutedmerges.failures.aborted | +operation | +The number of merges that failed because the storage node was (most likely) shutting down | +
| vds.mergethrottler.locallyexecutedmerges.failures.bucketnotfound | +operation | +The number of operations that failed because the bucket did not exist | +
| vds.mergethrottler.locallyexecutedmerges.failures.busy | +operation | +The number of merges that failed because the storage node was busy | +
| vds.mergethrottler.locallyexecutedmerges.failures.exists | +operation | +The number of merges that were rejected due to a merge operation for their bucket already being processed | +
| vds.mergethrottler.locallyexecutedmerges.failures.notready | +operation | +The number of merges discarded because distributor was not ready | +
| vds.mergethrottler.locallyexecutedmerges.failures.other | +operation | +The number of other failures | +
| vds.mergethrottler.locallyexecutedmerges.failures.rejected | +operation | +The number of merges that were rejected | +
| vds.mergethrottler.locallyexecutedmerges.failures.timeout | +operation | +The number of merges that failed because they timed out towards storage | +
| vds.mergethrottler.locallyexecutedmerges.failures.total | +operation | +Sum of all failures | +
| vds.mergethrottler.locallyexecutedmerges.failures.wrongdistribution | +operation | +The number of merges that were discarded (flushed) because they were initiated at an older cluster state than the current | +
| vds.mergethrottler.mergechains.ok | +operation | +The number of successful merges for 'mergechains' | +
| vds.mergethrottler.mergechains.failures.busy | +operation | +The number of merges that failed because the storage node was busy | +
| vds.mergethrottler.mergechains.failures.total | +operation | +Sum of all failures | +
| vds.mergethrottler.mergechains.failures.exists | +operation | +The number of merges that were rejected due to a merge operation for their bucket already being processed | +
| vds.mergethrottler.mergechains.failures.notready | +operation | +The number of merges discarded because distributor was not ready | +
| vds.mergethrottler.mergechains.failures.other | +operation | +The number of other failures | +
| vds.mergethrottler.mergechains.failures.rejected | +operation | +The number of merges that were rejected | +
| vds.mergethrottler.mergechains.failures.timeout | +operation | +The number of merges that failed because they timed out towards storage | +
| vds.mergethrottler.mergechains.failures.wrongdistribution | +operation | +The number of merges that were discarded (flushed) because they were initiated at an older cluster state than the current | +
| vds.server.network.tls-handshakes-failed | +operation | +Number of client or server connection attempts that failed during TLS handshaking | +
| vds.server.network.peer-authorization-failures | +failure | +Number of TLS connection attempts failed due to bad or missing peer certificate credentials | +
| vds.server.network.client.tls-connections-established | +connection | +Number of secure mTLS connections established | +
| vds.server.network.server.tls-connections-established | +connection | +Number of secure mTLS connections established | +
| vds.server.network.client.insecure-connections-established | +connection | +Number of insecure (plaintext) connections established | +
| vds.server.network.server.insecure-connections-established | +connection | +Number of insecure (plaintext) connections established | +
| vds.server.network.tls-connections-broken | +connection | +Number of TLS connections broken due to failures during frame encoding or decoding | +
| vds.server.network.failed-tls-config-reloads | +failure | +Number of times background reloading of TLS config has failed | +
| vds.bouncer.unavailable_node_aborts | +operation | +Number of operations that were aborted due to the node (or target bucket space) being unavailable | +
| vds.changedbucketownershiphandler.avg_abort_processing_time | +millisecond | +Average time spent aborting operations for changed buckets | +
| vds.changedbucketownershiphandler.external_load_ops_aborted | +operation | +Number of outdated external load operations aborted | +
| vds.changedbucketownershiphandler.ideal_state_ops_aborted | +operation | +Number of outdated ideal state operations aborted | +
| vds.communication.bucket_space_mapping_failures | +operation | +Number of messages that could not be resolved to a known bucket space | +
| vds.communication.convertfailures | +operation | +Number of messages that failed to get converted to storage API messages | +
| vds.communication.exceptionmessageprocesstime | +millisecond | +Time transport thread uses to process a single message that fails with an exception thrown into communication manager | +
| vds.communication.messageprocesstime | +millisecond | +Time transport thread uses to process a single message | +
| vds.communication.messagequeue | +item | +Size of input message queue. | +
| vds.communication.sendcommandlatency | +millisecond | +Average ms used to send commands to MBUS | +
| vds.communication.sendreplylatency | +millisecond | +Average ms used to send replies to MBUS | +
| vds.communication.toolittlememory | +operation | +Number of messages failed due to too little memory available | +
| vds.datastored.bucket_space.active_buckets | +bucket | +Number of active buckets in the bucket space | +
| vds.datastored.bucket_space.bucket_db.memory_usage.allocated_bytes | +byte | +The number of allocated bytes | +
| vds.datastored.bucket_space.bucket_db.memory_usage.dead_bytes | +byte | +The number of dead bytes ({`<=`} used_bytes) |
+
| vds.datastored.bucket_space.bucket_db.memory_usage.onhold_bytes | +byte | +The number of bytes on hold | +
| vds.datastored.bucket_space.bucket_db.memory_usage.used_bytes | +byte | +The number of used bytes ({`<=`} allocated_bytes) |
+
| vds.datastored.bucket_space.buckets_total | +bucket | +Total number buckets present in the bucket space (ready + not ready) | +
| vds.datastored.bucket_space.entries | +document | +Number of entries (documents + tombstones) stored in the bucket space | +
| vds.datastored.bucket_space.bytes | +byte | +Bytes stored across all documents in the bucket space | +
| vds.datastored.bucket_space.docs | +document | +Documents stored in the bucket space | +
| vds.datastored.bucket_space.ready_buckets | +bucket | +Number of ready buckets in the bucket space | +
| vds.datastored.fullbucketinfolatency | +millisecond | +Amount of time spent to process a full bucket info request | +
| vds.datastored.fullbucketinforeqsize | +node | +Amount of distributors answered at once in full bucket info requests. | +
| vds.datastored.simplebucketinforeqsize | +bucket | +Amount of buckets returned in simple bucket info requests | +
| vds.filestor.allthreads.applybucketdiff.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.applybucketdiff.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.applybucketdiff.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.applybucketdiffreply | +request | +Number of applybucketdiff replies that have been processed. | +
| vds.filestor.allthreads.bucketfixed | +bucket | +Number of times bucket has been fixed because of corruption | +
| vds.filestor.allthreads.bucketverified.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.bucketverified.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.bucketverified.latency | +request | +Latency of successful requests. | +
| vds.filestor.allthreads.bytesmerged | +byte | +Total number of bytes merged into this node. | +
| vds.filestor.allthreads.createbuckets.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.createbuckets.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.createbuckets.latency | +request | +Latency of successful requests. | +
| vds.filestor.allthreads.failedoperations | +operation | +Number of operations throwing exceptions. | +
| vds.filestor.allthreads.getbucketdiff.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.getbucketdiff.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.getbucketdiff.latency | +request | +Latency of successful requests. | +
| vds.filestor.allthreads.getbucketdiffreply | +request | +Number of getbucketdiff replies that have been processed. | +
| vds.filestor.allthreads.internaljoin.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.internaljoin.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.internaljoin.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.movedbuckets.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.movedbuckets.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.movedbuckets.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.operations | +operation | +Number of operations processed. | +
| vds.filestor.allthreads.readbucketinfo.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.readbucketinfo.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.readbucketinfo.latency | +request | +Latency of successful requests. | +
| vds.filestor.allthreads.readbucketlist.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.readbucketlist.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.readbucketlist.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.recheckbucketinfo.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.recheckbucketinfo.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.recheckbucketinfo.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.revert.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.revert.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.revert.latency | +millisecond | +Latency of successful requests. | +
| vds.filestor.allthreads.revert.not_found | +request | +Number of requests that could not be completed due to source document not found. | +
| vds.filestor.allthreads.stat_bucket.count | +request | +Number of requests processed. | +
| vds.filestor.allthreads.stat_bucket.failed | +request | +Number of failed requests. | +
| vds.filestor.allthreads.stat_bucket.latency | +request | +Latency of successful requests. | +
| vds.filestor.bucket_db_init_latency | +millisecond | +Time taken (in ms) to initialize bucket databases with information from the persistence provider | +
| vds.filestor.directoryevents | +operation | +Number of directory events received. | +
| vds.filestor.diskevents | +operation | +Number of disk events received. | +
| vds.filestor.partitionevents | +operation | +Number of partition events received. | +
| vds.filestor.pendingmerge | +bucket | +Number of buckets currently being merged. | +
| vds.filestor.waitingforlockrate | +operation | +Amount of times a filestor thread has needed to wait for lock to take next message in queue. | +
| vds.mergethrottler.mergechains.failures.aborted | +operation | +The number of merges that failed because the storage node was (most likely) shutting down | +
| vds.mergethrottler.mergechains.failures.bucketnotfound | +operation | +The number of operations that failed because the bucket did not exist | +
| vds.server.memoryusage | +byte | +Amount of memory used by the storage subsystem | +
| vds.server.memoryusage_visiting | +byte | +Message use from visiting | +
| vds.server.message_memory_use.highpri | +byte | +Message use from high priority storage messages | +
| vds.server.message_memory_use.lowpri | +byte | +Message use from low priority storage messages | +
| vds.server.message_memory_use.normalpri | +byte | +Message use from normal priority storage messages | +
| vds.server.message_memory_use.total | +byte | +Message use from storage messages | +
| vds.server.message_memory_use.veryhighpri | +byte | +Message use from very high priority storage messages | +
| vds.state_manager.invoke_state_listeners_latency | +millisecond | +Time spent (in ms) propagating state changes to internal state listeners | +
| vds.visitor.cv_queueevictedwaittime | +millisecond | +Milliseconds waiting in create visitor queue, for visitors that was evicted from queue due to higher priority visitors coming | +
| vds.visitor.cv_queuefull | +operation | +Number of create visitor messages failed as queue is full | +
| vds.visitor.cv_queuesize | +item | +Size of create visitor queue | +
| vds.visitor.cv_queuetimeoutwaittime | +millisecond | +Milliseconds waiting in create visitor queue, for visitors that timed out while in the visitor queue | +
| vds.visitor.cv_queuewaittime | +millisecond | +Milliseconds waiting in create visitor queue, for visitors that was added to visitor queue but scheduled later | +
| vds.visitor.cv_skipqueue | +operation | +Number of times we could skip queue as we had free visitor spots | +
| vds.server.network.rpc-capability-checks-failed | +failure | +Number of RPC operations that failed due to one or more missing capabilities | +
| vds.server.network.status-capability-checks-failed | +failure | +Number of status page operations that failed due to one or more missing capabilities | +
| vds.server.fnet.num-connections | +connection | +Total number of connection objects | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| cluster-controller.down.count | +node | +last, max | +Number of content nodes down | +
| cluster-controller.initializing.count | +node | +last, max | +Number of content nodes initializing | +
| cluster-controller.maintenance.count | +node | +last, max | +Number of content nodes in maintenance | +
| cluster-controller.retired.count | +node | +last, max | +Number of content nodes that are retired | +
| cluster-controller.stopping.count | +node | +last | +Number of content nodes currently stopping | +
| cluster-controller.up.count | +node | +last, max | +Number of content nodes up | +
| cluster-controller.nodes-not-converged | +node | +max | +Number of nodes not converging to the latest cluster state version | +
| cluster-controller.stored-document-count | +document | +max | +Total number of unique documents stored in the cluster | +
| cluster-controller.stored-document-bytes | +byte | +max | +Combined byte size of all unique documents stored in the cluster (not including replication) | +
| cluster-controller.cluster-buckets-out-of-sync-ratio | +fraction | +max | +Ratio of buckets in the cluster currently in need of syncing | +
| cluster-controller.busy-tick-time-ms | +millisecond | +count, last, max, sum | +Time busy | +
| cluster-controller.idle-tick-time-ms | +millisecond | +count, last, max, sum | +Time idle | +
| cluster-controller.work-ms | +millisecond | +count, last, sum | +Time used for actual work | +
| cluster-controller.is-master | +binary | +last, max | +1 if this cluster controller is currently the master, or 0 if not | +
| cluster-controller.remote-task-queue.size | +operation | +last | +Number of remote tasks queued | +
| cluster-controller.resource_usage.nodes_above_limit | +node | +last, max | +The number of content nodes above resource limit, blocking feed | +
| cluster-controller.resource_usage.max_memory_utilization | +fraction | +last, max | +Current memory utilisation, for content node with the highest value | +
| cluster-controller.resource_usage.max_disk_utilization | +fraction | +last, max | +Current disk space utilisation, for content node with the highest value | +
| cluster-controller.resource_usage.memory_limit | +fraction | +last, max | +Memory space limit as a fraction of available memory | +
| cluster-controller.resource_usage.disk_limit | +fraction | +last, max | +Disk space limit as a fraction of available disk space | +
| reindexing.progress | +fraction | +last, max | +Re-indexing progress | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| http.status.1xx | +response | +rate | +Number of responses with a 1xx status | +
| http.status.2xx | +response | +rate | +Number of responses with a 2xx status | +
| http.status.3xx | +response | +rate | +Number of responses with a 3xx status | +
| http.status.4xx | +response | +rate | +Number of responses with a 4xx status | +
| http.status.5xx | +response | +rate | +Number of responses with a 5xx status | +
| application_generation | +version | +N/A | +The currently live application config generation (aka session id) | +
| jdisc.gc.count | +operation | +average, last, max | +Number of JVM garbage collections done | +
| jdisc.gc.ms | +millisecond | +average, last, max | +Time spent in JVM garbage collection | +
| jdisc.jvm | +version | +last | +JVM runtime version | +
| jdisc.memory_mappings | +operation | +max | +JDISC Memory mappings | +
| jdisc.open_file_descriptors | +item | +max | +JDISC Open file descriptors | +
| jdisc.thread_pool.unhandled_exceptions | +thread | +count, last, max, min, sum | +Number of exceptions thrown by tasks | +
| jdisc.thread_pool.work_queue.capacity | +thread | +count, last, max, min, sum | +Capacity of the task queue | +
| jdisc.thread_pool.work_queue.size | +thread | +count, last, max, min, sum | +Size of the task queue | +
| jdisc.thread_pool.rejected_tasks | +thread | +count, last, max, min, sum | +Number of tasks rejected by the thread pool | +
| jdisc.thread_pool.size | +thread | +count, last, max, min, sum | +Size of the thread pool | +
| jdisc.thread_pool.max_allowed_size | +thread | +count, last, max, min, sum | +The maximum allowed number of threads in the pool | +
| jdisc.thread_pool.active_threads | +thread | +count, last, max, min, sum | +Number of threads that are active | +
| jdisc.deactivated_containers.total | +item | +last, sum | +JDISC Deactivated container instances | +
| jdisc.deactivated_containers.with_retained_refs.last | +item | +last | +JDISC Deactivated container nodes with retained refs | +
| jdisc.application.failed_component_graphs | +item | +rate | +JDISC Application failed component graphs | +
| jdisc.application.component_graph.creation_time_millis | +millisecond | +last | +JDISC Application component graph creation time | +
| jdisc.application.component_graph.reconfigurations | +item | +rate | +JDISC Application component graph reconfigurations | +
| jdisc.singleton.is_active | +item | +last, max, min | +JDISC Singleton is active | +
| jdisc.singleton.activation.count | +operation | +last | +JDISC Singleton activations | +
| jdisc.singleton.activation.failure.count | +operation | +last | +JDISC Singleton activation failures | +
| jdisc.singleton.activation.millis | +millisecond | +last | +JDISC Singleton activation time | +
| jdisc.singleton.deactivation.count | +operation | +last | +JDISC Singleton deactivations | +
| jdisc.singleton.deactivation.failure.count | +operation | +last | +JDISC Singleton deactivation failures | +
| jdisc.singleton.deactivation.millis | +millisecond | +last | +JDISC Singleton deactivation time | +
| jdisc.http.ssl.handshake.failure.missing_client_cert | +operation | +rate | +JDISC HTTP SSL Handshake failures due to missing client certificate | +
| jdisc.http.ssl.handshake.failure.expired_client_cert | +operation | +rate | +JDISC HTTP SSL Handshake failures due to expired client certificate | +
| jdisc.http.ssl.handshake.failure.invalid_client_cert | +operation | +rate | +JDISC HTTP SSL Handshake failures due to invalid client certificate | +
| jdisc.http.ssl.handshake.failure.incompatible_protocols | +operation | +rate | +JDISC HTTP SSL Handshake failures due to incompatible protocols | +
| jdisc.http.ssl.handshake.failure.incompatible_chifers | +operation | +rate | +JDISC HTTP SSL Handshake failures due to incompatible chifers | +
| jdisc.http.ssl.handshake.failure.connection_closed | +operation | +rate | +JDISC HTTP SSL Handshake failures due to connection closed | +
| jdisc.http.ssl.handshake.failure.unknown | +operation | +rate | +JDISC HTTP SSL Handshake failures for unknown reason | +
| jdisc.http.latency | +millisecond | +count, max, sum | +Request latency including the HTTP layer | +
| jdisc.http.request.prematurely_closed | +request | +rate | +HTTP requests prematurely closed | +
| jdisc.http.request.requests_per_connection | +request | +average, count, max, min, sum | +HTTP requests per connection | +
| jdisc.http.request.uri_length | +byte | +count, max, sum | +HTTP URI length | +
| jdisc.http.request.content_size | +byte | +count, max, sum | +HTTP request content size | +
| jdisc.http.requests | +request | +count, rate | +HTTP requests | +
| jdisc.http.filter.rule.blocked_requests | +request | +rate | +Number of requests blocked by filter | +
| jdisc.http.filter.rule.allowed_requests | +request | +rate | +Number of requests allowed by filter | +
| jdisc.http.filtering.request.handled | +request | +rate | +Number of filtering requests handled | +
| jdisc.http.filtering.request.unhandled | +request | +rate | +Number of filtering requests unhandled | +
| jdisc.http.filtering.response.handled | +request | +rate | +Number of filtering responses handled | +
| jdisc.http.filtering.response.unhandled | +request | +rate | +Number of filtering responses unhandled | +
| jdisc.http.handler.unhandled_exceptions | +request | +rate | +Number of unhandled exceptions in handler | +
| jdisc.tls.capability_checks.succeeded | +operation | +rate | +Number of TLS capability checks succeeded | +
| jdisc.tls.capability_checks.failed | +operation | +rate | +Number of TLS capability checks failed | +
| jdisc.http.jetty.threadpool.thread.max | +thread | +count, last, max, min, sum | +Configured maximum number of threads | +
| jdisc.http.jetty.threadpool.thread.min | +thread | +count, last, max, min, sum | +Configured minimum number of threads | +
| jdisc.http.jetty.threadpool.thread.reserved | +thread | +count, last, max, min, sum | +Configured number of reserved threads or -1 for heuristic | +
| jdisc.http.jetty.threadpool.thread.busy | +thread | +count, last, max, min, sum | +Number of threads executing internal and transient jobs | +
| jdisc.http.jetty.threadpool.thread.total | +thread | +count, last, max, min, sum | +Current number of threads | +
| jdisc.http.jetty.threadpool.queue.size | +thread | +count, last, max, min, sum | +Current size of the job queue | +
| jdisc.http.jetty.http_compliance.violation | +failure | +rate | +Number of HTTP compliance violations | +
| serverNumOpenConnections | +connection | +average, last, max | +The number of currently open connections | +
| serverNumConnections | +connection | +average, last, max | +The total number of connections opened | +
| serverBytesReceived | +byte | +count, sum | +The number of bytes received by the server | +
| serverBytesSent | +byte | +count, sum | +The number of bytes sent from the server | +
| handled.requests | +operation | +count | +The number of requests handled per metrics snapshot | +
| handled.latency | +millisecond | +count, max, sum | +The time used for handling requests, excluding HTTP layer and rendering | +
| httpapi_latency | +millisecond | +count, max, sum | +Duration for requests to the HTTP document APIs | +
| httpapi_pending | +operation | +count, max, sum | +Document operations pending execution | +
| httpapi_num_operations | +operation | +rate | +Total number of document operations performed | +
| httpapi_num_updates | +operation | +rate | +Document update operations performed | +
| httpapi_num_removes | +operation | +rate | +Document remove operations performed | +
| httpapi_num_puts | +operation | +rate | +Document put operations performed | +
| httpapi_succeeded | +operation | +rate | +Document operations that succeeded | +
| httpapi_failed | +operation | +rate | +Document operations that failed | +
| httpapi_parse_error | +operation | +rate | +Document operations that failed due to document parse errors | +
| httpapi_condition_not_met | +operation | +rate | +Document operations not applied due to condition not met | +
| httpapi_not_found | +operation | +rate | +Document operations not applied due to document not found | +
| httpapi_failed_unknown | +operation | +rate | +Document operations failed by unknown cause | +
| httpapi_failed_timeout | +operation | +rate | +Document operations failed by timeout | +
| httpapi_failed_insufficient_storage | +operation | +rate | +Document operations failed by insufficient storage | +
| httpapi_queued_operations | +operation | +last | +Document operations queued for execution in /document/v1 API handler | +
| httpapi_queued_bytes | +byte | +last | +Total operation bytes queued for execution in /document/v1 API handler | +
| httpapi_queued_age | +second | +last | +Age in seconds of the oldest operation in the queue for /document/v1 API handler | +
| httpapi_mbus_window_size | +operation | +last | +The window size of Messagebus's dynamic throttle policy for /document/v1 API handler | +
| mem.heap.total | +byte | +average | +Total available heap memory | +
| mem.heap.free | +byte | +average | +Free heap memory | +
| mem.heap.used | +byte | +average, max | +Currently used heap memory | +
| mem.direct.total | +byte | +average | +Total available direct memory | +
| mem.direct.free | +byte | +average | +Currently free direct memory | +
| mem.direct.used | +byte | +average, max | +Direct memory currently used | +
| mem.direct.count | +byte | +max | +Number of direct memory allocations | +
| mem.native.total | +byte | +average | +Total available native memory | +
| mem.native.free | +byte | +average | +Currently free native memory | +
| mem.native.used | +byte | +average | +Native memory currently used | +
| athenz-tenant-cert.expiry.seconds | +second | +last, max, min | +Time remaining until Athenz tenant certificate expires | +
| container-iam-role.expiry.seconds | +second | +N/A | +Time remaining until IAM role expires | +
| peak_qps | +query_per_second | +max | +The highest number of qps for a second for this metrics snapshot | +
| search_connections | +connection | +count, max, sum | +Number of search connections | +
| feed.operations | +operation | +rate | +Number of document feed operations | +
| feed.latency | +millisecond | +count, max, sum | +Feed latency | +
| feed.http-requests | +operation | +count, rate | +Feed HTTP requests | +
| queries | +operation | +rate | +Query volume | +
| query_container_latency | +millisecond | +count, max, sum | +The query execution time consumed in the container | +
| query_latency | +millisecond | +count, max, sum | +The overall query latency as observed by the container cluster, excluding HTTP layer and rendering | +
| query_timeout | +millisecond | +count, max, min, sum | +The amount of time allowed for query execution, from the client | +
| failed_queries | +operation | +rate | +The number of failed queries | +
| degraded_queries | +operation | +rate | +The number of degraded queries, e.g. due to some content nodes not responding in time | +
| hits_per_query | +hit_per_query | +count, max, sum | +The number of hits returned | +
| query_hit_offset | +hit | +count, max, sum | +The offset for hits returned | +
| documents_covered | +document | +count | +The combined number of documents considered during query evaluation | +
| documents_total | +document | +count | +The number of documents to be evaluated if all requests had been fully executed | +
| documents_target_total | +document | +count | +The target number of total documents to be evaluated when all data is in sync | +
| jdisc.render.latency | +nanosecond | +average, count, last, max, min, sum | +The time used by the container to render responses | +
| query_item_count | +item | +count, max, sum | +The number of query items (terms, phrases, etc.) | +
| docproc.proctime | +millisecond | +count, max, sum | +Time spent processing document | +
| docproc.documents | +document | +count, max, min, sum | +Number of processed documents | +
| totalhits_per_query | +hit_per_query | +count, max, sum | +The total number of documents found to match queries | +
| empty_results | +operation | +rate | +Number of queries matching no documents | +
| requestsOverQuota | +operation | +count, rate | +The number of requests rejected due to exceeding quota | +
| relevance.at_1 | +score | +count, sum | +The relevance of hit number 1 | +
| relevance.at_3 | +score | +count, sum | +The relevance of hit number 3 | +
| relevance.at_10 | +score | +count, sum | +The relevance of hit number 10 | +
| error.timeout | +operation | +rate | +Requests that timed out | +
| error.backends_oos | +operation | +rate | +Requests that failed due to no available backends nodes | +
| error.plugin_failure | +operation | +rate | +Requests that failed due to plugin failure | +
| error.backend_communication_error | +operation | +rate | +Requests that failed due to backend communication error | +
| error.empty_document_summaries | +operation | +rate | +Requests that failed due to missing document summaries | +
| error.invalid_query_parameter | +operation | +rate | +Requests that failed due to invalid query parameters | +
| error.internal_server_error | +operation | +rate | +Requests that failed due to internal server error | +
| error.misconfigured_server | +operation | +rate | +Requests that failed due to misconfigured server | +
| error.invalid_query_transformation | +operation | +rate | +Requests that failed due to invalid query transformation | +
| error.results_with_errors | +operation | +rate | +The number of queries with error payload | +
| error.unspecified | +operation | +rate | +Requests that failed for an unspecified reason | +
| error.unhandled_exception | +operation | +rate | +Requests that failed due to an unhandled exception | +
| serverRejectedRequests | +operation | +count, rate | +Deprecated. Use jdisc.thread_pool.rejected_tasks instead. | +
| serverThreadPoolSize | +thread | +last, max | +Deprecated. Use jdisc.thread_pool.size instead. | +
| serverActiveThreads | +thread | +count, last, max, min, sum | +Deprecated. Use jdisc.thread_pool.active_threads instead. | +
| jrt.transport.tls-certificate-verification-failures | +failure | +N/A | +TLS certificate verification failures | +
| jrt.transport.peer-authorization-failures | +failure | +N/A | +TLS peer authorization failures | +
| jrt.transport.server.tls-connections-established | +connection | +N/A | +TLS server connections established | +
| jrt.transport.client.tls-connections-established | +connection | +N/A | +TLS client connections established | +
| jrt.transport.server.unencrypted-connections-established | +connection | +N/A | +Unencrypted server connections established | +
| jrt.transport.client.unencrypted-connections-established | +connection | +N/A | +Unencrypted client connections established | +
| embedder.latency | +millisecond | +count, max, sum | +Time spent creating an embedding | +
| embedder.sequence_length | +item | +count, max, sum | +Number of tokens in the input sequence | +
| embedder.request.count | +request | +count | +Number of embedder API requests | +
| embedder.request.failure.count | +request | +count | +Number of failed embedder API requests | +
| embedder.batch.size | +item | +count, max, sum | +Number of items in each dispatched batch | +
| embedder.batch.queue_time | +millisecond | +count, max, sum | +Time spent waiting in queue before batch dispatch | +
| embedder.batch.count | +operation | +count | +Number of batch dispatches | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| vds.idealstate.buckets_rechecking | +bucket | +average | +The number of buckets that we are rechecking for ideal state operations | +
| vds.idealstate.idealstate_diff | +bucket | +average | +A number representing the current difference from the ideal state. This is a number that decreases steadily as the system is getting closer to the ideal state | +
| vds.idealstate.buckets_toofewcopies | +bucket | +average | +The number of buckets the distributor controls that have less than the desired redundancy | +
| vds.idealstate.buckets_toomanycopies | +bucket | +average | +The number of buckets the distributor controls that have more than the desired redundancy | +
| vds.idealstate.buckets | +bucket | +average | +The number of buckets the distributor controls | +
| vds.idealstate.buckets_notrusted | +bucket | +average | +The number of buckets that have no trusted copies. | +
| vds.idealstate.bucket_replicas_moving_out | +bucket | +average | +Bucket replicas that should be moved out, e.g. retirement case or node added to cluster that has higher ideal state priority. | +
| vds.idealstate.bucket_replicas_copying_out | +bucket | +average | +Bucket replicas that should be copied out, e.g. node is in ideal state but might have to provide data other nodes in a merge | +
| vds.idealstate.bucket_replicas_copying_in | +bucket | +average | +Bucket replicas that should be copied in, e.g. node does not have a replica for a bucket that it is in ideal state for | +
| vds.idealstate.bucket_replicas_syncing | +bucket | +average | +Bucket replicas that need syncing due to mismatching metadata | +
| vds.idealstate.max_observed_time_since_last_gc_sec | +second | +average | +Maximum time (in seconds) since GC was last successfully run for a bucket. Aggregated max value across all buckets on the distributor. | +
| vds.idealstate.delete_bucket.done_ok | +operation | +rate | +The number of operations successfully performed | +
| vds.idealstate.delete_bucket.done_failed | +operation | +rate | +The number of operations that failed | +
| vds.idealstate.delete_bucket.pending | +operation | +average | +The number of operations pending | +
| vds.idealstate.merge_bucket.done_ok | +operation | +rate | +The number of operations successfully performed | +
| vds.idealstate.merge_bucket.done_failed | +operation | +rate | +The number of operations that failed | +
| vds.idealstate.merge_bucket.pending | +operation | +average | +The number of operations pending | +
| vds.idealstate.merge_bucket.blocked | +operation | +rate | +The number of operations blocked by blocking operation starter | +
| vds.idealstate.merge_bucket.throttled | +operation | +rate | +The number of operations throttled by throttling operation starter | +
| vds.idealstate.merge_bucket.source_only_copy_changed | +operation | +rate | +The number of merge operations where source-only copy changed | +
| vds.idealstate.merge_bucket.source_only_copy_delete_blocked | +operation | +rate | +The number of merge operations where delete of unchanged source-only copies was blocked | +
| vds.idealstate.merge_bucket.source_only_copy_delete_failed | +operation | +rate | +The number of merge operations where delete of unchanged source-only copies failed | +
| vds.idealstate.split_bucket.done_ok | +operation | +rate | +The number of operations successfully performed | +
| vds.idealstate.split_bucket.done_failed | +operation | +rate | +The number of operations that failed | +
| vds.idealstate.split_bucket.pending | +operation | +average | +The number of operations pending | +
| vds.idealstate.join_bucket.done_ok | +operation | +rate | +The number of operations successfully performed | +
| vds.idealstate.join_bucket.done_failed | +operation | +rate | +The number of operations that failed | +
| vds.idealstate.join_bucket.pending | +operation | +average | +The number of operations pending | +
| vds.idealstate.garbage_collection.done_ok | +operation | +rate | +The number of operations successfully performed | +
| vds.idealstate.garbage_collection.done_failed | +operation | +rate | +The number of operations that failed | +
| vds.idealstate.garbage_collection.pending | +operation | +average | +The number of operations pending | +
| vds.idealstate.garbage_collection.documents_removed | +document | +count, rate | +Number of documents removed by GC operations | +
| vds.distributor.puts.latency | +millisecond | +count, max, sum | +The latency of put operations | +
| vds.distributor.puts.ok | +operation | +rate | +The number of successful put operations performed | +
| vds.distributor.puts.failures.total | +operation | +rate | +Sum of all failures | +
| vds.distributor.puts.failures.notfound | +operation | +rate | +The number of operations that failed because the document did not exist | +
| vds.distributor.puts.failures.test_and_set_failed | +operation | +rate | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.puts.failures.concurrent_mutations | +operation | +rate | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.puts.failures.notconnected | +operation | +rate | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.puts.failures.notready | +operation | +rate | +The number of operations discarded because distributor was not ready | +
| vds.distributor.puts.failures.wrongdistributor | +operation | +rate | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.puts.failures.safe_time_not_reached | +operation | +rate | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.puts.failures.storagefailure | +operation | +rate | +The number of operations that failed in storage | +
| vds.distributor.puts.failures.timeout | +operation | +rate | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.puts.failures.busy | +operation | +rate | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.puts.failures.inconsistent_bucket | +operation | +rate | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.removes.latency | +millisecond | +count, max, sum | +The latency of remove operations | +
| vds.distributor.removes.ok | +operation | +rate | +The number of successful removes operations performed | +
| vds.distributor.removes.failures.total | +operation | +rate | +Sum of all failures | +
| vds.distributor.removes.failures.notfound | +operation | +rate | +The number of operations that failed because the document did not exist | +
| vds.distributor.removes.failures.test_and_set_failed | +operation | +rate | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.removes.failures.concurrent_mutations | +operation | +rate | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.updates.latency | +millisecond | +count, max, sum | +The latency of update operations | +
| vds.distributor.updates.ok | +operation | +rate | +The number of successful updates operations performed | +
| vds.distributor.updates.failures.total | +operation | +rate | +Sum of all failures | +
| vds.distributor.updates.failures.notfound | +operation | +rate | +The number of operations that failed because the document did not exist | +
| vds.distributor.updates.failures.test_and_set_failed | +operation | +rate | +The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | +
| vds.distributor.updates.failures.concurrent_mutations | +operation | +rate | +The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | +
| vds.distributor.updates.diverging_timestamp_updates | +operation | +rate | +Number of updates that report they were performed against divergent version timestamps on different replicas | +
| vds.distributor.removelocations.ok | +operation | +rate | +The number of successful removelocations operations performed | +
| vds.distributor.removelocations.failures.total | +operation | +rate | +Sum of all failures | +
| vds.distributor.gets.latency | +millisecond | +count, max, sum | +The average latency of gets operations | +
| vds.distributor.gets.ok | +operation | +rate | +The number of successful gets operations performed | +
| vds.distributor.gets.failures.total | +operation | +rate | +Sum of all failures | +
| vds.distributor.gets.failures.notfound | +operation | +rate | +The number of operations that failed because the document did not exist | +
| vds.distributor.visitor.latency | +millisecond | +count, max, sum | +The average latency of visitor operations | +
| vds.distributor.visitor.ok | +operation | +rate | +The number of successful visitor operations performed | +
| vds.distributor.visitor.failures.total | +operation | +rate | +Sum of all failures | +
| vds.distributor.visitor.failures.notready | +operation | +rate | +The number of operations discarded because distributor was not ready | +
| vds.distributor.visitor.failures.notconnected | +operation | +rate | +The number of operations discarded because there were no available storage nodes to send to | +
| vds.distributor.visitor.failures.wrongdistributor | +operation | +rate | +The number of operations discarded because they were sent to the wrong distributor | +
| vds.distributor.visitor.failures.safe_time_not_reached | +operation | +rate | +The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | +
| vds.distributor.visitor.failures.storagefailure | +operation | +rate | +The number of operations that failed in storage | +
| vds.distributor.visitor.failures.timeout | +operation | +rate | +The number of operations that failed because the operation timed out towards storage | +
| vds.distributor.visitor.failures.busy | +operation | +rate | +The number of messages from storage that failed because the storage node was busy | +
| vds.distributor.visitor.failures.inconsistent_bucket | +operation | +rate | +The number of operations failed due to buckets being in an inconsistent state or not found | +
| vds.distributor.visitor.failures.notfound | +operation | +rate | +The number of operations that failed because the document did not exist | +
| vds.distributor.docsstored | +document | +average | +Number of documents stored in all buckets controlled by this distributor | +
| vds.distributor.bytesstored | +byte | +average | +Number of bytes stored in all buckets controlled by this distributor | +
| vds.distributor.mutating_op_memory_usage | +byte | +max | +Estimated amount of memory used by active mutating operations across all distributor stripes, in bytes | +
| vds.bouncer.clock_skew_aborts | +operation | +count | +Number of client operations that were aborted due to clock skew between sender and receiver exceeding acceptable range | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| logd.processed.lines | +item | +count | +Number of log lines processed | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| endpoint.certificate.expiry.seconds | +second | +N/A | +Time until node endpoint certificate expires | +
| node-certificate.expiry.seconds | +second | +N/A | +Time until node certificate expires | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| content.proton.config.generation | +version | +last | +The oldest config generation used by this search node | +
| content.proton.documentdb.documents.total | +document | +last, max | +The total number of documents in this documents db (ready + not-ready) | +
| content.proton.documentdb.documents.ready | +document | +last, max | +The number of ready documents in this document db | +
| content.proton.documentdb.documents.active | +document | +last, max | +The number of active / searchable documents in this document db | +
| content.proton.documentdb.documents.removed | +document | +last, max | +The number of removed documents in this document db | +
| content.proton.documentdb.index.docs_in_memory | +document | +last, max | +Number of documents in memory index | +
| content.proton.documentdb.disk_usage | +byte | +last | +The total disk usage (in bytes) for this document db | +
| content.proton.documentdb.memory_usage.allocated_bytes | +byte | +max | +The number of allocated bytes | +
| content.proton.documentdb.heart_beat_age | +second | +last, min | +How long ago (in seconds) heart beat maintenance job was run | +
| content.proton.docsum.docs | +document | +rate | +Total docsums returned | +
| content.proton.docsum.latency | +millisecond | +count, max, sum | +Docsum request latency | +
| content.proton.search_protocol.query.latency | +second | +count, max, sum | +Query request latency (seconds) | +
| content.proton.search_protocol.query.request_size | +byte | +count, max, sum | +Query request size (network bytes) | +
| content.proton.search_protocol.query.reply_size | +byte | +count, max, sum | +Query reply size (network bytes) | +
| content.proton.search_protocol.docsum.latency | +second | +average, count, max, sum | +Docsum request latency (seconds) | +
| content.proton.search_protocol.docsum.request_size | +byte | +count, max, sum | +Docsum request size (network bytes) | +
| content.proton.search_protocol.docsum.reply_size | +byte | +count, max, sum | +Docsum reply size (network bytes) | +
| content.proton.search_protocol.docsum.requested_documents | +document | +count, max, sum | +Total requested document summaries | +
| content.proton.executor.proton.queuesize | +task | +count, max, sum | +Size of executor proton task queue | +
| content.proton.executor.proton.accepted | +task | +rate | +Number of executor proton accepted tasks | +
| content.proton.executor.proton.wakeups | +wakeup | +rate | +Number of times an executor proton worker thread has been woken up | +
| content.proton.executor.proton.utilization | +fraction | +count, max, sum | +Ratio of time the executor proton worker threads has been active | +
| content.proton.executor.flush.queuesize | +task | +count, max, sum | +Size of executor flush task queue | +
| content.proton.executor.flush.accepted | +task | +rate | +Number of accepted executor flush tasks | +
| content.proton.executor.flush.wakeups | +wakeup | +rate | +Number of times an executor flush worker thread has been woken up | +
| content.proton.executor.flush.utilization | +fraction | +count, max, sum | +Ratio of time the executor flush worker threads has been active | +
| content.proton.executor.match.queuesize | +task | +count, max, sum | +Size of executor match task queue | +
| content.proton.executor.match.accepted | +task | +rate | +Number of accepted executor match tasks | +
| content.proton.executor.match.wakeups | +wakeup | +rate | +Number of times an executor match worker thread has been woken up | +
| content.proton.executor.match.utilization | +fraction | +count, max, sum | +Ratio of time the executor match worker threads has been active | +
| content.proton.executor.docsum.queuesize | +task | +count, max, sum | +Size of executor docsum task queue | +
| content.proton.executor.docsum.accepted | +task | +rate | +Number of executor accepted docsum tasks | +
| content.proton.executor.docsum.wakeups | +wakeup | +rate | +Number of times an executor docsum worker thread has been woken up | +
| content.proton.executor.docsum.utilization | +fraction | +count, max, sum | +Ratio of time the executor docsum worker threads has been active | +
| content.proton.executor.shared.queuesize | +task | +count, max, sum | +Size of executor shared task queue | +
| content.proton.executor.shared.accepted | +task | +rate | +Number of executor shared accepted tasks | +
| content.proton.executor.shared.wakeups | +wakeup | +rate | +Number of times an executor shared worker thread has been woken up | +
| content.proton.executor.shared.utilization | +fraction | +count, max, sum | +Ratio of time the executor shared worker threads has been active | +
| content.proton.executor.warmup.queuesize | +task | +count, max, sum | +Size of executor warmup task queue | +
| content.proton.executor.warmup.accepted | +task | +rate | +Number of accepted executor warmup tasks | +
| content.proton.executor.warmup.wakeups | +wakeup | +rate | +Number of times a warmup executor worker thread has been woken up | +
| content.proton.executor.warmup.utilization | +fraction | +count, max, sum | +Ratio of time the executor warmup worker threads has been active | +
| content.proton.executor.field_writer.queuesize | +task | +count, max, sum | +Size of executor field writer task queue | +
| content.proton.executor.field_writer.accepted | +task | +rate | +Number of accepted executor field writer tasks | +
| content.proton.executor.field_writer.wakeups | +wakeup | +rate | +Number of times an executor field writer worker thread has been woken up | +
| content.proton.executor.field_writer.utilization | +fraction | +count, max, sum | +Ratio of time the executor fieldwriter worker threads has been active | +
| content.proton.executor.field_writer.saturation | +fraction | +count, max, sum | +Ratio indicating the max saturation of underlying worker threads. A higher saturation than utilization indicates a bottleneck in one of the worker threads. | +
| content.proton.documentdb.job.total | +fraction | +average | +The job load average total of all job metrics | +
| content.proton.documentdb.job.attribute_flush | +fraction | +average | +Flushing of attribute vector(s) to disk | +
| content.proton.documentdb.job.memory_index_flush | +fraction | +average | +Flushing of memory index to disk | +
| content.proton.documentdb.job.disk_index_fusion | +fraction | +average | +Fusion of disk indexes | +
| content.proton.documentdb.job.document_store_flush | +fraction | +average | +Flushing of document store to disk | +
| content.proton.documentdb.job.document_store_compact | +fraction | +average | +Compaction of document store on disk | +
| content.proton.documentdb.job.bucket_move | +fraction | +average | +Moving of buckets between 'ready' and 'notready' sub databases | +
| content.proton.documentdb.job.lid_space_compact | +fraction | +average | +Compaction of lid space in document meta store and attribute vectors | +
| content.proton.documentdb.job.removed_documents_prune | +fraction | +average | +Pruning of removed documents in 'removed' sub database | +
| content.proton.documentdb.threading_service.master.queuesize | +task | +count, max, sum | +Size of threading service master task queue | +
| content.proton.documentdb.threading_service.master.accepted | +task | +rate | +Number of accepted threading service master tasks | +
| content.proton.documentdb.threading_service.master.wakeups | +wakeup | +rate | +Number of times a threading service master worker thread has been woken up | +
| content.proton.documentdb.threading_service.master.utilization | +fraction | +count, max, sum | +Ratio of time the threading service master worker threads has been active | +
| content.proton.documentdb.threading_service.index.queuesize | +task | +count, max, sum | +Size of threading service index task queue | +
| content.proton.documentdb.threading_service.index.accepted | +task | +rate | +Number of accepted threading service index tasks | +
| content.proton.documentdb.threading_service.index.wakeups | +wakeup | +rate | +Number of times a threading service index worker thread has been woken up | +
| content.proton.documentdb.threading_service.index.utilization | +fraction | +count, max, sum | +Ratio of time the threading service index worker threads has been active | +
| content.proton.documentdb.threading_service.summary.queuesize | +task | +count, max, sum | +Size of threading service summary task queue | +
| content.proton.documentdb.threading_service.summary.accepted | +task | +rate | +Number of accepted threading service summary tasks | +
| content.proton.documentdb.threading_service.summary.wakeups | +wakeup | +rate | +Number of times a threading service summary worker thread has been woken up | +
| content.proton.documentdb.threading_service.summary.utilization | +fraction | +count, max, sum | +Ratio of time the threading service summary worker threads has been active | +
| content.proton.documentdb.ready.lid_space.lid_bloat_factor | +fraction | +average | +The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) | +
| content.proton.documentdb.ready.lid_space.lid_fragmentation_factor | +fraction | +average | +The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) | +
| content.proton.documentdb.ready.lid_space.lid_limit | +documentid | +last, max | +The size of the allocated lid space | +
| content.proton.documentdb.ready.lid_space.highest_used_lid | +documentid | +last, max | +The highest used lid | +
| content.proton.documentdb.ready.lid_space.used_lids | +documentid | +last, max | +The number of lids used | +
| content.proton.documentdb.notready.lid_space.lid_bloat_factor | +fraction | +average | +The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) | +
| content.proton.documentdb.notready.lid_space.lid_fragmentation_factor | +fraction | +average | +The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) | +
| content.proton.documentdb.notready.lid_space.lid_limit | +documentid | +last, max | +The size of the allocated lid space | +
| content.proton.documentdb.notready.lid_space.highest_used_lid | +documentid | +last, max | +The highest used lid | +
| content.proton.documentdb.notready.lid_space.used_lids | +documentid | +last, max | +The number of lids used | +
| content.proton.documentdb.removed.lid_space.lid_bloat_factor | +fraction | +average | +The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) | +
| content.proton.documentdb.removed.lid_space.lid_fragmentation_factor | +fraction | +average | +The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) | +
| content.proton.documentdb.removed.lid_space.lid_limit | +documentid | +last, max | +The size of the allocated lid space | +
| content.proton.documentdb.removed.lid_space.highest_used_lid | +documentid | +last, max | +The highest used lid | +
| content.proton.documentdb.removed.lid_space.used_lids | +documentid | +last, max | +The number of lids used | +
| content.proton.documentdb.bucket_move.buckets_pending | +bucket | +last, max, sum | +The number of buckets left to move | +
| content.proton.resource_usage.disk | +fraction | +average | +The relative amount of disk used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller | +
| content.proton.resource_usage.disk_usage.total | +fraction | +max | +The total relative amount of disk used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.disk_usage.total_utilization | +fraction | +max | +The relative amount of disk used compared to the content node disk resource limit | +
| content.proton.resource_usage.disk_usage.transient | +fraction | +max | +The relative amount of transient disk used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.memory | +fraction | +average | +The relative amount of memory used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller | +
| content.proton.resource_usage.memory_usage.total | +fraction | +max | +The total relative amount of memory used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.memory_usage.total_utilization | +fraction | +max | +The relative amount of memory used compared to the content node memory resource limit | +
| content.proton.resource_usage.memory_usage.transient | +fraction | +max | +The relative amount of transient memory used by this content node (value in the range [0, 1]) | +
| content.proton.resource_usage.memory_mappings | +file | +max | +The number of memory mapped files | +
| content.proton.resource_usage.open_file_descriptors | +file | +max | +The number of open files | +
| content.proton.resource_usage.feeding_blocked | +binary | +last, max | +Whether feeding is blocked due to resource limits being reached (value is either 0 or 1) | +
| content.proton.resource_usage.malloc_arena | +byte | +max | +Size of malloc arena | +
| content.proton.documentdb.attribute.resource_usage.address_space | +fraction | +max | +The max relative address space used among components in all attribute vectors in this document db (value in the range [0, 1]) | +
| content.proton.documentdb.attribute.resource_usage.feeding_blocked | +binary | +max | +Whether feeding is blocked due to attribute resource limits being reached (value is either 0 or 1) | +
| content.proton.resource_usage.cpu_util.setup | +fraction | +count, max, sum | +cpu used by system init and (re-)configuration | +
| content.proton.resource_usage.cpu_util.read | +fraction | +count, max, sum | +cpu used by reading data from the system | +
| content.proton.resource_usage.cpu_util.write | +fraction | +count, max, sum | +cpu used by writing data to the system | +
| content.proton.resource_usage.cpu_util.compact | +fraction | +count, max, sum | +cpu used by internal data re-structuring | +
| content.proton.resource_usage.cpu_util.other | +fraction | +count, max, sum | +cpu used by work not classified as a specific category | +
| content.proton.transactionlog.entries | +record | +average | +The current number of entries in the transaction log | +
| content.proton.transactionlog.disk_usage | +byte | +average | +The disk usage (in bytes) of the transaction log | +
| content.proton.transactionlog.replay_time | +second | +last, max | +The replay time (in seconds) of the transaction log during start-up | +
| content.proton.documentdb.ready.document_store.disk_usage | +byte | +average | +Disk space usage in bytes | +
| content.proton.documentdb.ready.document_store.disk_bloat | +byte | +average | +Disk space bloat in bytes | +
| content.proton.documentdb.ready.document_store.max_bucket_spread | +fraction | +average | +Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) | +
| content.proton.documentdb.ready.document_store.memory_usage.allocated_bytes | +byte | +average | +The number of allocated bytes | +
| content.proton.documentdb.ready.document_store.memory_usage.used_bytes | +byte | +average | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.ready.document_store.memory_usage.onhold_bytes | +byte | +average | +The number of bytes on hold | +
| content.proton.documentdb.notready.document_store.disk_usage | +byte | +average | +Disk space usage in bytes | +
| content.proton.documentdb.notready.document_store.disk_bloat | +byte | +average | +Disk space bloat in bytes | +
| content.proton.documentdb.notready.document_store.max_bucket_spread | +fraction | +average | +Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) | +
| content.proton.documentdb.notready.document_store.memory_usage.allocated_bytes | +byte | +average | +The number of allocated bytes | +
| content.proton.documentdb.notready.document_store.memory_usage.used_bytes | +byte | +average | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.notready.document_store.memory_usage.dead_bytes | +byte | +average | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.notready.document_store.memory_usage.onhold_bytes | +byte | +average | +The number of bytes on hold | +
| content.proton.documentdb.removed.document_store.disk_usage | +byte | +average | +Disk space usage in bytes | +
| content.proton.documentdb.removed.document_store.disk_bloat | +byte | +average | +Disk space bloat in bytes | +
| content.proton.documentdb.removed.document_store.max_bucket_spread | +fraction | +average | +Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) | +
| content.proton.documentdb.removed.document_store.memory_usage.allocated_bytes | +byte | +average | +The number of allocated bytes | +
| content.proton.documentdb.removed.document_store.memory_usage.used_bytes | +byte | +average | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.removed.document_store.memory_usage.dead_bytes | +byte | +average | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.removed.document_store.memory_usage.onhold_bytes | +byte | +average | +The number of bytes on hold | +
| content.proton.documentdb.ready.document_store.cache.memory_usage | +byte | +average | +Memory usage of the cache (in bytes) | +
| content.proton.documentdb.ready.document_store.cache.hit_rate | +fraction | +average | +Rate of hits in the cache compared to number of lookups | +
| content.proton.documentdb.ready.document_store.cache.lookups | +operation | +rate | +Number of lookups in the cache (hits + misses) | +
| content.proton.documentdb.ready.document_store.cache.invalidations | +operation | +rate | +Number of invalidations (erased elements) in the cache. | +
| content.proton.documentdb.notready.document_store.cache.memory_usage | +byte | +average | +Memory usage of the cache (in bytes) | +
| content.proton.documentdb.notready.document_store.cache.hit_rate | +fraction | +average | +Rate of hits in the cache compared to number of lookups | +
| content.proton.documentdb.notready.document_store.cache.lookups | +operation | +rate | +Number of lookups in the cache (hits + misses) | +
| content.proton.documentdb.notready.document_store.cache.invalidations | +operation | +rate | +Number of invalidations (erased elements) in the cache. | +
| content.proton.documentdb.ready.attribute.memory_usage.allocated_bytes | +byte | +average | +The number of allocated bytes | +
| content.proton.documentdb.ready.attribute.memory_usage.used_bytes | +byte | +average | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.ready.attribute.memory_usage.dead_bytes | +byte | +average | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.ready.attribute.memory_usage.onhold_bytes | +byte | +average | +The number of bytes on hold | +
| content.proton.documentdb.ready.attribute.disk_usage | +byte | +average | +Disk space usage (in bytes) of the flushed snapshot of this attribute for this document type | +
| content.proton.documentdb.notready.attribute.memory_usage.allocated_bytes | +byte | +average | +The number of allocated bytes | +
| content.proton.documentdb.notready.attribute.memory_usage.used_bytes | +byte | +average | +The number of used bytes ({`<=`} allocated_bytes) |
+
| content.proton.documentdb.notready.attribute.memory_usage.dead_bytes | +byte | +average | +The number of dead bytes ({`<=`} used_bytes) |
+
| content.proton.documentdb.notready.attribute.memory_usage.onhold_bytes | +byte | +average | +The number of bytes on hold | +
| content.proton.index.cache.postinglist.memory_usage | +byte | +average | +Memory usage of the cache (in bytes). Contains disk index posting list files across all document types | +
| content.proton.index.cache.postinglist.hit_rate | +fraction | +average | +Rate of hits in the cache compared to number of lookups. Contains disk index posting list files across all document types | +
| content.proton.index.cache.postinglist.lookups | +operation | +rate | +Number of lookups in the cache (hits + misses). Contains disk index posting list files across all document types | +
| content.proton.index.cache.postinglist.invalidations | +operation | +rate | +Number of invalidations (erased elements) in the cache. Contains disk index posting list files across all document types | +
| content.proton.index.cache.bitvector.memory_usage | +byte | +average | +Memory usage of the cache (in bytes). Contains disk index bitvector files across all document types | +
| content.proton.index.cache.bitvector.hit_rate | +fraction | +average | +Rate of hits in the cache compared to number of lookups. Contains disk index bitvector files across all document types | +
| content.proton.index.cache.bitvector.lookups | +operation | +rate | +Number of lookups in the cache (hits + misses). Contains disk index bitvector files across all document types | +
| content.proton.index.cache.bitvector.invalidations | +operation | +rate | +Number of invalidations (erased elements) in the cache. Contains disk index bitvector files across all document types | +
| content.proton.documentdb.index.memory_usage.allocated_bytes | +byte | +average | +The number of allocated bytes for the memory index for this document type | +
| content.proton.documentdb.index.memory_usage.used_bytes | +byte | +average | +The number of used bytes ({`<=`} allocated_bytes) for the memory index for this document type |
+
| content.proton.documentdb.index.memory_usage.dead_bytes | +byte | +average | +The number of dead bytes ({`<=`} used_bytes) for the memory index for this document type |
+
| content.proton.documentdb.index.memory_usage.onhold_bytes | +byte | +average | +The number of bytes on hold for the memory index for this document type | +
| content.proton.documentdb.index.io.search.read_bytes | +byte | +count, sum | +Bytes read from disk index posting list and bitvector files as part of search for this document type | +
| content.proton.documentdb.index.io.search.cached_read_bytes | +byte | +count, sum | +Bytes read from cached disk index posting list and bitvector files as part of search for this document type | +
| content.proton.documentdb.ready.index.disk_usage | +byte | +average | +Disk space usage (in bytes) of this index field in all disk indexes for this document type | +
| content.proton.documentdb.matching.queries | +query | +rate | +Number of queries executed | +
| content.proton.documentdb.matching.soft_doomed_queries | +query | +rate | +Number of queries hitting the soft timeout | +
| content.proton.documentdb.matching.query_latency | +second | +count, max, sum | +Total average latency (sec) when matching and ranking a query | +
| content.proton.documentdb.matching.query_setup_time | +second | +count, max, sum | +Average time (sec) spent setting up and tearing down queries | +
| content.proton.documentdb.matching.docs_matched | +document | +count, rate | +Number of documents matched | +
| content.proton.documentdb.matching.exact_nns_distances_computed | +distance | +rate | +Number of distances computed in exact nearest-neighbor search | +
| content.proton.documentdb.matching.approximate_nns_distances_computed | +distance | +rate | +Number of distances computed in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.approximate_nns_nodes_visited | +graph_node | +rate | +Number of nodes visited in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.queries | +query | +rate | +Number of queries executed | +
| content.proton.documentdb.matching.rank_profile.soft_doomed_queries | +query | +rate | +Number of queries hitting the soft timeout | +
| content.proton.documentdb.matching.rank_profile.soft_doom_factor | +fraction | +count, max, min, sum | +Factor used to compute soft-timeout | +
| content.proton.documentdb.matching.rank_profile.query_latency | +second | +count, max, sum | +Total average latency (sec) when matching and ranking a query | +
| content.proton.documentdb.matching.rank_profile.query_setup_time | +second | +count, max, sum | +Average time (sec) spent setting up and tearing down queries | +
| content.proton.documentdb.matching.rank_profile.grouping_time | +second | +count, max, sum | +Average time (sec) spent on grouping | +
| content.proton.documentdb.matching.rank_profile.rerank_time | +second | +count, max, sum | +Average time (sec) spent on 2nd phase ranking | +
| content.proton.documentdb.matching.rank_profile.docs_matched | +document | +count, rate | +Number of documents matched | +
| content.proton.documentdb.matching.rank_profile.exact_nns_distances_computed | +distance | +rate | +Number of distances computed in exact nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.approximate_nns_distances_computed | +distance | +rate | +Number of distances computed in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.approximate_nns_nodes_visited | +graph_node | +rate | +Number of nodes visited in approximate nearest-neighbor search | +
| content.proton.documentdb.matching.rank_profile.limited_queries | +query | +rate | +Number of queries limited in match phase | +
| content.proton.documentdb.feeding.commit.operations | +operation | +count, max, rate, sum | +Number of operations included in a commit | +
| content.proton.documentdb.feeding.commit.latency | +second | +count, max, sum | +Latency for commit in seconds | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| sentinel.restarts | +restart | +count | +Number of service restarts done by the sentinel | +
| sentinel.totalRestarts | +restart | +last, max, sum | +Total number of service restarts done by the sentinel since the sentinel was started | +
| sentinel.uptime | +second | +last | +Time the sentinel has been running | +
| sentinel.running | +instance | +count, last | +Number of services the sentinel has running currently | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| slobrok.heartbeats.failed | +request | +count | +Number of heartbeat requests failed | +
| slobrok.missing.consensus | +second | +count | +Number of seconds without full consensus with all other brokers | +
| Name | +Unit | +Suffixes | +Description | +
|---|---|---|---|
| vds.datastored.alldisks.buckets | +bucket | +average | +Number of buckets managed | +
| vds.datastored.alldisks.docs | +document | +average | +Number of documents stored | +
| vds.datastored.alldisks.bytes | +byte | +average | +Number of bytes stored | +
| vds.visitor.allthreads.averagevisitorlifetime | +millisecond | +count, max, sum | +Average lifetime of a visitor | +
| vds.visitor.allthreads.averagequeuewait | +millisecond | +count, max, sum | +Average time an operation spends in input queue. | +
| vds.visitor.allthreads.queuesize | +operation | +count, max, sum | +Size of input message queue. | +
| vds.visitor.allthreads.completed | +operation | +rate | +Number of visitors completed | +
| vds.visitor.allthreads.created | +operation | +rate | +Number of visitors created. | +
| vds.visitor.allthreads.failed | +operation | +rate | +Number of visitors failed | +
| vds.visitor.allthreads.averagemessagesendtime | +millisecond | +count, max, sum | +Average time it takes for messages to be sent to their target (and be replied to) | +
| vds.visitor.allthreads.averageprocessingtime | +millisecond | +count, max, sum | +Average time used to process visitor requests | +
| vds.filestor.queuesize | +operation | +count, max, sum | +Size of input message queue. | +
| vds.filestor.averagequeuewait | +millisecond | +count, max, sum | +Average time an operation spends in input queue. | +
| vds.filestor.active_operations.size | +operation | +count, max, sum | +Number of concurrent active operations | +
| vds.filestor.active_operations.latency | +millisecond | +count, max, sum | +Latency (in ms) for completed operations | +
| vds.filestor.throttle_window_size | +operation | +count, max, sum | +Current size of async operation throttler window size | +
| vds.filestor.throttle_waiting_threads | +thread | +count, max, sum | +Number of threads waiting to acquire a throttle token | +
| vds.filestor.throttle_active_tokens | +instance | +count, max, sum | +Current number of active throttle tokens | +
| vds.filestor.allthreads.mergemetadatareadlatency | +millisecond | +count, max, sum | +Time spent in a merge step to check metadata of current node to see what data it has. | +
| vds.filestor.allthreads.mergedatareadlatency | +millisecond | +count, max, sum | +Time spent in a merge step to read data other nodes need. | +
| vds.filestor.allthreads.mergedatawritelatency | +millisecond | +count, max, sum | +Time spent in a merge step to write data needed to current node. | +
| vds.filestor.allthreads.merge_put_latency | +millisecond | +count, max, sum | +Latency of individual puts that are part of merge operations | +
| vds.filestor.allthreads.merge_remove_latency | +millisecond | +count, max, sum | +Latency of individual removes that are part of merge operations | +
| vds.filestor.allstripes.throttled_rpc_direct_dispatches | +instance | +rate | +Number of times an RPC thread could not directly dispatch an async operation directly to Proton because it was disallowed by the throttle policy | +
| vds.filestor.allstripes.throttled_persistence_thread_polls | +instance | +rate | +Number of times a persistence thread could not immediately dispatch a queued async operation because it was disallowed by the throttle policy | +
| vds.filestor.allstripes.timeouts_waiting_for_throttle_token | +instance | +rate | +Number of times a persistence thread timed out waiting for an available throttle policy token | +
| vds.filestor.allthreads.put.count | +operation | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.put.failed | +operation | +rate | +Number of failed requests. | +
| vds.filestor.allthreads.put.test_and_set_failed | +operation | +rate | +Number of operations that were skipped due to a test-and-set condition not met | +
| vds.filestor.allthreads.put.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.put.request_size | +byte | +count, max, sum | +Size of requests, in bytes | +
| vds.filestor.allthreads.remove.count | +operation | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.remove.failed | +operation | +rate | +Number of failed requests. | +
| vds.filestor.allthreads.remove.test_and_set_failed | +operation | +rate | +Number of operations that were skipped due to a test-and-set condition not met | +
| vds.filestor.allthreads.remove.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.remove.request_size | +byte | +count, max, sum | +Size of requests, in bytes | +
| vds.filestor.allthreads.get.count | +operation | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.get.failed | +operation | +rate | +Number of failed requests. | +
| vds.filestor.allthreads.get.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.get.request_size | +byte | +count, max, sum | +Size of requests, in bytes | +
| vds.filestor.allthreads.update.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.update.failed | +request | +rate | +Number of failed requests. | +
| vds.filestor.allthreads.update.test_and_set_failed | +request | +rate | +Number of requests that were skipped due to a test-and-set condition not met | +
| vds.filestor.allthreads.update.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.update.request_size | +byte | +count, max, sum | +Size of requests, in bytes | +
| vds.filestor.allthreads.createiterator.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.createiterator.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.visit.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.visit.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.remove_location.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.remove_location.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.splitbuckets.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.joinbuckets.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.deletebuckets.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.deletebuckets.failed | +request | +rate | +Number of failed requests. | +
| vds.filestor.allthreads.deletebuckets.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.remove_by_gid.count | +request | +rate | +Number of requests processed. | +
| vds.filestor.allthreads.remove_by_gid.failed | +request | +rate | +Number of failed requests. | +
| vds.filestor.allthreads.remove_by_gid.latency | +millisecond | +count, max, sum | +Latency of successful requests. | +
| vds.filestor.allthreads.setbucketstates.count | +request | +rate | +Number of requests processed. | +
| vds.mergethrottler.averagequeuewaitingtime | +millisecond | +count, max, sum | +Time merges spent in the throttler queue | +
| vds.mergethrottler.queuesize | +instance | +count, max, sum | +Length of merge queue | +
| vds.mergethrottler.active_window_size | +instance | +count, max, sum | +Number of merges active within the pending window size | +
| vds.mergethrottler.estimated_merge_memory_usage | +byte | +count, max, sum | +An estimated upper bound of the memory usage (in bytes) of the merges currently in the active window | +
| vds.mergethrottler.bounced_due_to_back_pressure | +instance | +rate | +Number of merges bounced due to resource exhaustion back-pressure | +
| vds.mergethrottler.locallyexecutedmerges.ok | +instance | +rate | +The number of successful merges for 'locallyexecutedmerges' | +
| vds.mergethrottler.mergechains.ok | +operation | +rate | +The number of successful merges for 'mergechains' | +
| vds.mergethrottler.mergechains.failures.busy | +operation | +rate | +The number of merges that failed because the storage node was busy | +
| vds.mergethrottler.mergechains.failures.total | +operation | +rate | +Sum of all failures | +
| vds.server.network.tls-handshakes-failed | +operation | +count | +Number of client or server connection attempts that failed during TLS handshaking | +
| vds.server.network.peer-authorization-failures | +failure | +count | +Number of TLS connection attempts failed due to bad or missing peer certificate credentials | +
| vds.server.network.client.tls-connections-established | +connection | +count | +Number of secure mTLS connections established | +
| vds.server.network.server.tls-connections-established | +connection | +count | +Number of secure mTLS connections established | +
| vds.server.network.client.insecure-connections-established | +connection | +count | +Number of insecure (plaintext) connections established | +
| vds.server.network.server.insecure-connections-established | +connection | +count | +Number of insecure (plaintext) connections established | +
| vds.server.network.tls-connections-broken | +connection | +count | +Number of TLS connections broken due to failures during frame encoding or decoding | +
| vds.server.network.failed-tls-config-reloads | +failure | +count | +Number of times background reloading of TLS config has failed | +
| vds.server.network.rpc-capability-checks-failed | +failure | +count | +Number of RPC operations that failed due to one or more missing capabilities | +
| vds.server.network.status-capability-checks-failed | +failure | +count | +Number of status page operations that failed due to one or more missing capabilities | +
| vds.server.fnet.num-connections | +connection | +count | +Total number of connection objects | +
| Option | +Description | +
|---|---|
| **-m** | +method, available methods are: **cache** Output the config proxy cache content (overview) **dumpcache** **statistics** **heap** **getConfig** Get config (see vespa-get-config) **getmode** Outputs the current mode of the config proxy **setmode** Use *default* or *memorycache* - example **invalidatecache** Clears the current cache in the config proxy **cachefull** Output the config proxy cache content (including config payload) **sources** Output the config proxy's upstream config sources **updatesources** Updates the config proxy's upstream config sources to the supplied ones |
+
| **-p** | +port number, optional | +
| **-s** | +hostname, optional | +
| **-h** | +help text and usage | +
| Option | +Description | +
|---|---|
| **-force** | +Do not ask for confirmation before removal | +
| Option | +Description | +
|---|---|
| **-v** | +Verbose - show all services, even if they are up-to-date | +
| **-c arg** | +Get the Vespa cluster configuration from the config server specified by host and port. Use *host* or *host:port* for config server | +
| **-f** | +Filter to only query config status for the given comma-separated set of hosts | +
| Command | +Description | +
|---|---|
| **prepare** | +*vespa-deploy prepare* combines the upload and prepare steps. | +
| **activate** | +*vespa-deploy activate* invokes the activate step. | +
| **upload** | +*vespa-deploy upload* uploads an application package | +
| **fetch** | +*vespa-deploy fetch* fetches an application package. Useful to get the active configuration for an instance. | +
| **help** | +Same as **-h** | +
| Option | +Description | +
|---|---|
| **-h** | +Show help text | +
| **-v** | +Verbose | +
| **-n** | +Dry-run deployment | +
| **-f** | +Force - ignore validation errors | +
| **-t** | +Timeout | +
| **-c** | +Config server hostname | +
| **-p** | +Config server port | +
| Option | +Description | +
|---|---|
| **--instant** | +Reply in message thread | +
| **--name arg** | +Slobrok name to register | +
| **--maxqueuetime arg** | +Adjust the in-queue size to have a maximum queue wait period of this many ms (default -1 = unlimited) | +
| **--silent #nummsg** | +Do not dump anything, but progress every #nummsg | +
| **--sleeptime arg** | +The number of milliseconds to sleep per message, to simulate processing time | +
| **--threads arg** | +The number of threads to process the incoming data | +
| **--verbose** | +Dump the contents of certain messages to stdout | +
| Option | +Description | +
|---|---|
| **--abortondataerror arg** | +Abort if the input has errors (true|false) - default true. Set to *false* in case the input has errors (e.g., invalid characters). *vespa-feeder* notifies on parsing errors at the end of the feed, but it will not abort | +
| **--abortonsenderror arg** | +Abort if an error occurred while sending operations to Vespa (true|false) - default true | +
| **--file arg** | +Input files to read. These can also be passed as arguments without the option prefix. If none is given, this tool parses identifiers from stdin | +
| **--maxpending arg** | +Maximum number of pending operations. This disables dynamic throttling, use with care | +
| **--maxpendingsize arg** | +Maximum size (in bytes) of pending operations | +
| **--maxfeedrate arg** | +Limits the feed rate to the given number (operations/second) | +
| **--mode** | +The mode to run vespa-feeder in (standard|benchmark) - default standard | +
| **--noretry** | +Disables retries of recoverable failures | +
| **--retrydelay arg** | +The time (in seconds) to wait between retries of a failed operation. Default 1 | +
| **--route arg** | +The route to send the data to. Default the *default* route | +
| **--timeout arg** | +Time (in seconds) allowed for sending operations. Default 180 | +
| **--trace arg** | +Trace level of network traffic. Default 0 | +
| **--validate** | +Run validation tool on input files - do not feed | +
**--dumpDocuments {`** |
+ File where documents in the put are serialized | +
| **--numthreads arg** | +How many threads to use for sending. Default 1 | +
| **--create-if-non-existent** | +Enable setting of create-if-non-existent to true on all document updates in the given feed | +
| **-v, --verbose** | +Enable verbose output of progress | +
| Option | +Description | +
|---|---|
| **-a,--trace *tracelevel*** | +Trace level to use (default 0) | +
| **-c,--configid *configid*** | +Use the specified config id for messagebus configuration | +
| **-f,--fieldset *fieldset*** | +Retrieve the specified fields only (see Document field sets). Default: {`[document]`} |
+
| **-h,--help** | +Show this syntax page | +
| **-i,--printids** | +Show only identifiers of retrieved documents | +
| **-j,--jsonoutput** | +JSON output (default) | +
| **-l,--loadtype *loadtype*** | +Load type (default "") | +
| **-n,--noretry** | +Do not retry operation on transient errors, as is default | +
| **-r,--route *route*** | +Send request to the given messagebus route | +
| **-s,--showdocsize** | +Show binary size of document | +
| **--shorttensor**s | +Output using tensor short form | +
| **-t,--timeout *timeout*** | +Set timeout for the request in seconds (default 0) | +
| **-u,--cluster *cluster*** | +Send request to the given content cluster | +
| Option | +Description | +
|---|---|
| **-h, --help** | +Show help | +
| **-v** | +More verbose output | +
| **-s** | +Less verbose output | +
| **--show-hidden** | +Also show hidden undocumented debug options | +
| **-c, --cluster** | +The cluster name of the cluster to query. If unspecified, and vespa is installed on the current node, information will be attempted auto-extracted | +
| **-f, --force** | +Force execution | +
| **--config-server** | +Host name of the config server to query | +
| **--config-server-port** | +Port to connect to the config server on | +
| **--config-request-timeout** | +Timeout of config request | +
| Option | +Description | +
|---|---|
| **-n** | +config definition name, including namespace (on the form {`) |
+
| **-i** | +config id, optional | +
| **-a** | +config def schema file, optional (if you want to use another schema than the one known for the config server) | +
| **-m** | +defMd5, optional | +
| **-c** | +configMd5, optional | +
| **-t** | +server timeout, in seconds, default value 3, optional | +
| **-w** | +timeout, default value 10, optional | +
| **-s** | +server hostname, default localhost, optional | +
| **-p** | +port, default 19090, optional | +
| **-d** | +debug mode, optional | +
| **-h** | +help text and usage | +
| State | +Description | +
|---|---|
| **Unit state** | +The state of the node seen from the cluster controller. | +
| **User state** | +The state the administrator wants the node to be in, default "up". Can be set by using vespa-set-node-state or by the cluster controller | +
| **Generated state** | +The state of a given node in the current cluster state. This is the state all the other nodes know about. This state is a product of the other two states and cluster controller logic to keep the cluster stable. | +
| Option | +Description | +
|---|---|
| **-h, --help** | +Show help | +
| **-v** | +More verbose output | +
| **-s** | +Less verbose output | +
| **--show-hidden** | +Also show hidden undocumented debug options | +
| **-c, --cluster** | +The cluster name of the cluster to query. If unspecified, and vespa is installed on the current node, information will be attempted auto-extracted | +
| **-f, --force** | +Force execution | +
| **-t, --type** | +Node type - can either be 'storage' or 'distributor'. If not specified, the operation will use state for both types | +
| **-i, --index** | +Node index. If not specified, all nodes found running on this host will be used | +
| **--config-server** | +Host name of the config server to query | +
| **--config-server-port** | +Port to connect to the config server on | +
| **--config-request-timeout** | +Timeout of config request | +
| Option | +Description | +
|---|---|
| **--indexdir *path*** | +Index location | +
| **--field *fieldname*** | +Field to analyze | +
| **--transpose** | +Dump all tokens | +
| **--minnumdocs *count*** | +Minimum number of documents to analyze | +
| **--docidlimit *docid*** | +Dump up to this doc id | +
| **--mindocid *docid*** | +Start from this docid | +
| **--wordnum** | +Also dump token numbers | +
| **--verbose** | +Verbose output | +
| Option | +Description | +
|---|---|
| **-p** | +print content to {` |
+
| **-s** | +save attribute to {` |
+
| Option | +Description | +
|---|---|
| **-c** | +Create the control file if it does not exist (implies -n) | +
| **-a** | +Update all .logcontrol files | +
| **-r** | +Reset to default levels | +
| **-n** | +Create the component entry if it does not exist | +
| **-f *file*** | +Use {` as the log control file |
+
| **-d *dir*** | +Look in {` for log control files |
+
| Option | +Description | +
|---|---|
| **-l *levellist* (--level=*levellist*)** | +Filter messages by log level. By default, only messages of level *fatal, error, warning*, and *info* will be included, while messages of level *config, event, debug*, and *spam* will be ignored. This option allows you to replace or modify the list of log levels to be included. *levellist* is a comma-separated list of level names. • The name *all* may be used to add all known levels • You may use + or - in front of terms to add or remove from the current (or default) list of levels instead of replacing it • Adding term |
+
| **-s *fieldlist*** | +Select which fields of log messages to show. The output field order is fixed. When using this option, only the named fields will be printed. The default fields are as [**-s fmttime,msecs,level,service,component,message**]. The fieldlist is a comma-separated list of field names. The name *all* may be used to add all possible fields. Prepending a minus sign will turn off the display of the named field. Starting the list with a plus sign will add and remove fields from the current (or default) list of fields instead of replacing it. Using this option several times works as if the given *fieldlist* arguments had been concatenated into one comma-separated list. Fields: **time** Print the time in seconds since the epoch. Ignored if *fmttime* is shown **fmttime** Print the time in human-readable [YYYY-MM-DD HH:mm:ss] format. Note that the time is printed in the local timezone. To get GMT output, use {`env TZ=GMT vespa-logfmt`} **msecs** Add milliseconds after the seconds in *time* and *fmttime* output. Ignored if *usecs* is in effect **usecs** Add microseconds after the seconds in *time* and *fmttime* output **host** Print the hostname field **level** Print the level field (upper-cased) **pid** Print the pid field **service** Print the service field **component** Print the component field **message** Print the message text field. You probably always want to add this |
+
| **-p *pid*** | +Select messages where the pid field matches the *pid* string | +
| **-S *service*** | +Select messages where the service field matches the *service* string | +
| **-H *host*** | +Select messages where the hostname field matches the *host* string | +
| **-c *regex*** | +Select messages where the component field matches the *regex*, using *perlre* regular expression matching | +
| **-m *regex*** | +Select messages where the message text field matches the *regex*, using *perlre* regular expression matching | +
| **-f** | +Invoke tail -F to follow the input file | +
| **-N** | +De-quote quoted newlines in the message text field to an actual newline plus tab | +
| **-t** | +Format the component field (if shown) as a fixed-width string, truncating if necessary | +
| **-ts** | +Format the service field (if shown) as a fixed-width string, truncating if necessary | +
| **-i, --internal** | +Only include log entries emitted by the Vespa platform, i.e., exclude log entries from custom components | +
| Command | +Description | +
|---|---|
| **hosts** | +Show hostnames of all hosts in the Vespa system | +
| **services** | +Show a list of all service types in the Vespa system | +
| **clusters** | +Show a list of all named clusters in the Vespa system | +
| **configids** | +Show a list of all config ids in the Vespa system | +
| **filter:ports** | +List ports matching filter options | +
| **host *hostname*** | +Show host details: What services are running, and what ports have they allocated | +
| **service *servicetype*** | +Show service details: What instances of the service are running, on what hosts, and what ports have they allocated | +
| **cluster *clustername*** | +Show all services in the cluster, with details on hostname and allocated ports | +
| **configid *configid*** | +Show all services using this configid | +
| **get-index-of *servicetype* *host*** | +Show all indexes for instances of the service type on the given host | +
| Option | +Description | +
|---|---|
| **-c *host* | *host:port*** | +Specify host and port (or just host) to use for getting the config that this tool displays. Default is to use the configserver. You might want to use localhost:19090 if you are on a host with a running Vespa system without a config server | +
| **-h** | +Show usage | +
| **-t *tag*** | +to filter on a port tag | +
| **-u** | +Show URLs for services | +
| **-v** | +Verbose mode | +
| Command | +Description | +
|---|---|
| **getProtonStatus** | +Get the current proton state and its components. | +
| **getState** | +Get the current proton state. | +
| **triggerFlush** | +Trigger flush as soon as possible for all document types. | +
| **prepareRestart** | +Estimates the cost of transaction log replay, and flushes data structures if that will speed up a subsequent start. If this is not called before stopping proton, there is no estimation and no flush. | +
| Option | +Description | +
|---|---|
| **-force** | +Do not require verification from the user before really removing index data | +
| **-cluster *name*** | +Only remove data for the given cluster name | +
| Option | +Description | +
|---|---|
**--documentmanagerconfigid {`** |
+ Sets the config id that supplies document configuration | +
| **--dump** | +Prints the complete content of the routing table | +
| **--help** | +Prints this help | +
**--hop {`** |
+ Prints detailed information about hop {` |
+
| **--hops** | +Prints a list of all available hops | +
**--identity {`** |
+ Sets the identity of message bus | +
**--listenport {`** |
+ Sets the port message bus will listen to | +
**--oosserverpattern {`** |
+ Sets the out-of-service server pattern for message bus | +
**--protocol {`** |
+ Sets the name of the protocol whose routing to inspect | +
**--route {`** |
+ Prints detailed information about route {` |
+
| **--routes** | +Prints a list of all available routes | +
**--routingconfigid {`** |
+ Sets the config id that supplies the routing tables | +
| **--services** | +Prints a list of all available services | +
**--slobrokconfigid {`** |
+ Sets the config id that supplies the slobrok server list | +
**--trace {`** |
+ Sets the trace level to use when visualizing the route | +
| **--verify** | +All hops and routes are verified when routing | +
| Option | +Description | +
|---|---|
| **-h** | +Help text | +
| Command | +Description | +
|---|---|
| **list** | +Lists the services running on this node and their status: **service name** **state** • RUNNING: Service is running • FINISHED: Service has been stopped • FAILED: Service has crashed and failed to restart • TERMINATING: Service is stopping **mode** • MANUAL: Service has to be started and stopped manually • AUTO: Service will restart automatically if it stops **pid** Pid of the process (main thread) **exitstatus** Exit code the last time the service stopped. **id** Config ID of the service |
+
| **restart [name]** | +Restarts the service with the given name. The name is the first string in the service list given by {`list`} |
+
| **stop [name]** | +Stops the service with the given name | +
| **start [name]** | +Starts the service with the given name | +
| **connectivity** | +Use to troubleshoot startup issues/network configuration/ACLs/iptables:{`$ vespa-sentinel-cmd connectivity`}{`vespa-sentinel-cmd 'connectivity' OK.`}{`node0.vespanet -> ok`}{`node1.vespanet -> ok`}{`node2.vespanet -> ok`} |
+
| Option | +Description | +
|---|---|
| **-h, --help** | +Show help | +
| **-v** | +More verbose output | +
| **-s** | +Less verbose output | +
| **--show-hidden** | +Also show hidden undocumented debug options | +
| **-n, --no-wait** | +Do not wait for node state changes to be visible in the cluster before returning | +
| **-c, --cluster** | +The cluster name of the cluster to query. If unspecified, and vespa is installed on the current node, information will be attempted auto-extracted | +
| **-f, --force** | +Force execution | +
| **-t, --type** | +Node type - can either be 'storage' or 'distributor'. If not specified, the operation will use state for both types | +
| **-i, --index** | +Node index. If not specified, all nodes found running on this host will be used | +
| **--config-server** | +Host name of the config server to query | +
| **--config-server-port** | +Port to connect to the config server on | +
| **--config-request-timeout** | +Timeout of config request | +
| Option | +Description | +
|---|---|
| **-h, --help** | +Show help for the subcommand. | +
**-i, --in {`** |
+ Input file. Default format is JSON Lines where each line is a Vespa document in JSON. Use {`--format vstsv`} to read VSTSV files. |
+
**--format {`** |
+ Input format. Format can be {`jsonl`} or {`vstsv`}. Defaults to {`jsonl`}. |
+
**-o, --out {`** |
+ Output significance model. | +
**-f, --field {`** |
+ JSONL: the name of the text field to analyse. VSTSV: No effect. | +
**-l, --language {`** |
+ Comma-separated ISO language tags. The first tag controls tokenization; additional tags are stored in the model. Required for JSONL, optional for VSTSV (defaults to {`un`}). See supported tags in linguistics in Vespa. |
+
**-zst, --zst-compression {`** |
+ Enable Zstandard compression of the output. Enabled can be {`true`} or {`false`}. Default {`false`}. When {`true`}, the output file must end with {`.zst`}. |
+
| Option | +Description | +
|---|---|
| **-h, --help** | +Show help for the subcommand. | +
**--index-dir {`** |
+ Explicit path to the index directory. If omitted, the tool locates the directory using cluster, schema, and node information. | +
**--out {`** |
+ Output VSTSV file. Defaults to {`export_*FIELD*.vstsv`}. Compression adds {`.zst`} automatically. |
+
**--field {`** |
+ Text field to export. Must correspond to a field directory within the selected index. This is the same as a field in the schema. | +
**--cluster {`** |
+ Specifies the content cluster name for locating the index directory. If multiple clusters exist per node, use --cluster to choose one. | +
**--schema {`** |
+ Specifies the schema (document type) name used when locating the index directory. | +
**--node-index {`** |
+ Specifies the content node index for locating the index directory. If there are multiple indexes on a node, use --node-index to choose one. | +
| **-zst, --zst-compression** | +Write the VSTSV output compressed with Zstandard. | +
| Option | +Description | +
|---|---|
| **-h, --help** | +Show help for the subcommand. | +
**--out {`** |
+ Output VSTSV file. Defaults to {`merged.vstsv`}. Compression adds {`.zst`}. |
+
**--min-keep {`** |
+ Filter out terms with document frequency strictly lower than {`NUMBER`}. |
+
| **-zst, --zst-compression** | +Write the merged VSTSV output compressed with Zstandard. | +
| Option | +Description | +
|---|---|
**-b, --bucket {`** |
+ Dump list of buckets that are contained in the given bucket, or that contain it | +
| **-d, --dump** | +Dump list of documents for all buckets matching the selection command. | +
**-g, --group {`** |
+ Dump list of buckets that can contain the given group | +
| **-h, --help | +Help text | +
**-l, --gid {`** |
+ Dump information about one specific document, as given by the GID (implies --dump) | +
**-o, --document {`** |
+ Dump information about one specific document (implies --dump) | +
**-r, --route {`** |
+ Route to send the messages to, usually the name of the storage cluster | +
**-s, --bucketspace {`** |
+ Bucket space (*default* or *global*). If not specified, *default* is used | +
**-u, --user {`** |
+ Dump list of buckets that can contain the given user | +
| Option | +Description | +
|---|---|
**--application {`** |
+ Application name | +
| **--debug** | +Print debug log | +
**--environment {`** |
+ Environment name | +
| **-h, --help** | +Display help information | +
**--instance {`** |
+ Instance name | +
**--region {`** |
+ Region name | +
**--tenant {`** |
+ Tenant name | +
**--timeout {`** |
+ timeout (in seconds) | +
| Processor Type | +Description | +
|---|---|
| **Dump visitor** | +The most commonly used visitor processor type is the dump visitor. All it does is to send the read documents on to some external target specified by the visitor. Using the command line tool *vespa-visit*, the default is to just send the documents back to the client, and have them printed to stdout. The dump visitor is used to implement reprocessing. Typically, using a messagebus route, which will send the documents through the document processing cluster and then back to the content cluster. Migration of documents from one cluster to another is also implemented using a dump visitor. | +
| **Streaming search visitor** | +The streaming search visitor runs in the Vespa container, making it transparent whether search results were created from streaming or indexed search - see indexing mode. | +
| Target Type | +Description | +
|---|---|
| **Message bus routes** | +You can specify a message bus route name directly, and this route will be used to send the results. This is typically used when doing reprocessing or migration. Message bus routes are set up in the application package. In addition, some routes may have been auto-generated in simple setups, for instance, a route called *default* is generated if your setup is simple enough for the config model to likely guess where you want to send your data. | +
| **Slobrok address** | +You can also specify a slobrok address for data to be sent to. A slobrok address is a slash-separated path where you can use an asterisk to mean any element within this path. For instance, if you have a docproc cluster called *mydpcluster*, it will have registered its nodes with slobrok names like *docproc/cluster.mydpcluster/docproc/0/feed_processor*, where the 0 here indicates the first node in the cluster. You can thus specify to send visit data to this docproc cluster by stating a slobrok address of *docproc/cluster.mydpcluster/docproc/*/feed_processor*. Note that this will not send all the data to one or all the nodes. The data sent from the visitor will be distributed among the matching nodes, but each message will just be sent to one node. Slobrok names can be used when using vespa-visit-target to retrieve the data from some location. If you start vespa-visit-target on two nodes, listening to slobrok names *mynode/0/visit-destination* and *mynode/1/visit-destination*, you can send the results to these nodes by specifying *mynode/*/visit-destination* as the data handler. vespa-destination is similar to vespa-visit-target in that it can receive messages from messagebus and print the contents to stdout. It can be useful in situations where you want to debug a route or a docproc, by using the vespadestination as the endpoint of your route. |
+
| **TCP socket** | +TCP sockets can also be specified directly. This requires that the endpoint speaks FNET RPC. This is typically done, either by using the *vespa-visit-target* tool, or by using a visitor destination programmatically by using a utility class in the document API. A socket address looks like the following: tcp/*hostname*:*port*/*servicename*. For instance, an address generated by *vespa-visit-target* might look like: *tcp/myhost.mydomain.com:12345/visit-destination* | +