Remove pod_id and container_id from Loki Labels when using configure_kubernetes_labels: true

[PlayMTL]

Hey,

my question / discussion is specific to loki output from logging-operator using fluentd.

When configure_kubernetes_labels: true is enabled, the Loki output adds the labels container_id ($.kubernetes.docker_id) and pod_id ($.kubernetes.pod_id) to every stream. Both values change on every pod restart or redeployment, so each restart creates a new stream.

Ref:

logging-operator/pkg/sdk/logging/model/output/loki.go

Lines 130 to 158 in 5ef2a72

	if l.ConfigureKubernetesLabels != nil && *l.ConfigureKubernetesLabels { //nolint:nestif
	if l.Labels == nil {
	l.Labels = Label{}
	}
	l.Labels.merge(Label{
	"namespace": `$.kubernetes.namespace_name`,
	"pod": `$.kubernetes.pod_name`,
	"container_id": `$.kubernetes.docker_id`,
	"container": `$.kubernetes.container_name`,
	"pod_id": `$.kubernetes.pod_id`,
	"host": `$.kubernetes.host`,
	})
	if l.RemoveKeys != nil {
	if !util.Contains(l.RemoveKeys, "kubernetes") {
	l.RemoveKeys = append(l.RemoveKeys, "kubernetes")
	}
	} else {
	l.RemoveKeys = []string{"kubernetes"}
	}
	if l.ExtractKubernetesLabels == nil {
	l.ExtractKubernetesLabels = new(true)
	}
	if l.IncludeThreadLabel == nil {
	l.IncludeThreadLabel = new(true)
	}
	// Prevent meta configuration from marshaling
	l.ConfigureKubernetesLabels = nil
	}

This conflicts with Loki's labeling guidance (see Loki label best practices), which recommends using labels with bounded, low cardinality.

My proposal would be to remove both of these labels in the standard configuration. What do you think abount?

My current workaround (removing the labels with remove_keys didn't work for me):

    configure_kubernetes_labels: false
    extract_kubernetes_labels: false
    include_thread_label: false
    remove_keys:
      - kubernetes
    labels:
      container: $.kubernetes.container_name
      host: $.kubernetes.host
      namespace: $.kubernetes.namespace_name
      pod: $.kubernetes.pod_name

Thanks
PlayMTL

[sueun-dev]

remove_keys on the Loki output can't drop them because the plugin builds the stream labels before it applies remove_keys. In fluent-plugin-grafana-loki 1.3.0 (the version in the operator's fluentd image), line_to_loki evaluates the <label> accessors first and runs the remove_keys deleters afterwards, so they only trim the log line. Overriding pod_id under labels won't help either, since configure_kubernetes_labels merges its defaults over your labels.

Removing the two fields in the Flow, before the record reaches the output, does work. The plugin only keeps label values that are strings, so pod_id and container_id disappear while configure_kubernetes_labels: true still sets the other labels:

spec:
  filters:
    - record_transformer:
        remove_keys: $.kubernetes.pod_id,$.kubernetes.docker_id

This setup also keeps extract_kubernetes_labels on, though, so pod labels such as pod_template_hash still change on a Deployment rollout. Your workaround, with extraction off, avoids those.

I tested this with the 6.9.0-full fluentd image, an output config based on the operator's loki_test.go render, and a local stand-in for Loki. Two events with different pod_id/docker_id gave 2 streams by default and with those paths in the output's remove_keys, and 1 stream without pod_id/container_id with the filter. Not tested through a live operator. The filter applies to every output of that Flow.