how to configure index rollover

[toby181]

Hi together,

I try to configure an index rollover since a while but so far I got no luck with any of the configurations I tried. I'm aware that the datastream can handle that but I see some disadvantages there and, from my point of view, it should work with regular indices as well.

My goal is to create indices based on the scheme "logstash-${kubernetes.cluster_name}-${$.kubernetes.namespace_name}" and to roll them over once they reach a specific size or document count.
I'm using one default index template with no settings regarding alias, this one is just responsible for setting the primaries and replicas and I got a specific template for the index that actually requires a rollover, see below.
At the moment my indices are created on a weekly bases with the date suffix *-yyyy-wWW

fluentd opensearch configuration:

  <match **>
    @type opensearch
    @id out_os
    @log_level "debug"
    include_tag_key true
    host "opensearch-cluster-master"
    port 9200
    path ""
    scheme https
    ssl_verify false
    ssl_version TLSv1_2
    user "fluentd-cluster1"
    password xxxxxx
    reload_connections false
    reconnect_on_error true
    reload_on_failure true
    log_os_400_reason true
    logstash_prefix "logstash-${kubernetes.cluster_name}-${$.kubernetes.namespace_name}"
    logstash_dateformat "%G-W%V"
    logstash_format true
    index_name "logstash"
    include_timestamp true
    template_name
    template_file
    template_overwrite false
    sniffer_class_name "Fluent::Plugin::OpenSearchSimpleSniffer"
    request_timeout 120s
    suppress_type_name false
    <buffer tag, kubernetes.cluster_name, $.kubernetes.namespace_name>
      flush_thread_count 8
      flush_interval 5s
      chunk_limit_size 20M
      total_limit_size 2GB
      retry_max_interval 30
      retry_forever true
    </buffer>
  </match>

Simple index template for a specific namespace to add the rollover alias:

PUT _index_template/logstash-namespace1
{
  "index_patterns": [
    "logstash-cluster1-namespace1-*"
  ],
  "priority": 2,
  "template": {
    "settings": {
      "plugins.index_state_management.rollover_alias": "logstash-cluster1-namespace1",
      "index": {
        "number_of_shards": "1",
        "number_of_replicas": "1",
        "mapping": {
          "total_fields": {
            "limit": "2000"
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "time": {
          "format": "epoch_second",
          "type": "date"
        },
        "timestamp": {
          "format": "epoch_second||epoch_millis||strict_date_optional_time||strict_date_optional_time_nanos",
          "type": "date"
        }
      }
    }
  },
  "composed_of": []
}

rollover policy:

{
    "id": "logstash-rollover",
    "seqNo": 12772859,
    "primaryTerm": 16,
    "policy": {
        "policy_id": "logstash-rollover",
        "description": "A sample description of the policy",
        "last_updated_time": 1656329138006,
        "schema_version": 13,
        "error_notification": null,
        "default_state": "rollover",
        "states": [
            {
                "name": "rollover",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_doc_count": 350000,
                            "min_index_age": "1d",
                            "min_primary_shard_size": "5gb"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "3d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "logstash-cluster1-namespace1*"
                ],
                "priority": 3,
                "last_updated_time": 1656327417841
            }
        ]
    }
}

Based on the documentation from OpenSearch: https://opensearch.org/docs/2.0/im-plugin/ism/policies#sample-policy-with-ism-template-for-auto-rollover step 3 and 4 need to be handled by the plugin.

When I run a "GET _plugins/_ism/explain/logstash-cluster1-namespace1-2022-w25" in the DEV API in OpenSearch Dashboard, ILM complains about ""Missing alias or not the write index when rollover [index=logstash-cluster1-namespace1-2022-w25]""

---

When I do step 3 manually I got this exception:

      "cause" : """index name [logstash-cluster1-namespace1-2022-w25] does not match pattern '^.*-\d+$'""",
      "message" : "Failed to rollover index [index=logstash-dcek1445stage02-kube-infra-2022-w25]"

OS is missing the suffix 000001 and I did not figure out what configuration I need to apply so the plugin adds this suffix. No matter whether there's a date in the index name or not.

This is how I created the alias manually. Although, this should be handled by the plugin.

PUT logstash-cluster1-namespace1-2022-w25
{
  "aliases": {
    "logstash-cluster1-namespace1": {
      "is_write_index": true
    }
  }
}

---

Have you got any idea how to achieve that?
Also, I'm not sure whether this is a bug or feature request, so I put it into the discussion area.

Thanks in advance!

[yogrr]

+1