Vector stops reading logs from EKS with AL2023 based AMIs

[mscanlon72]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Vector stops reading logs from EKS worker nodes with AL2023 based AMIs. New worker nodes with AL2023 backed AMI were rolled out for the EKS clusters. Some hours later, Vector pods stop collecting logs. Using vector tap we cannot see any events in the output of the kubernetes_logs source.

Configuration

customConfig:
    data_dir: /var/lib/vector
    api:
      enabled: true
      address: 127.0.0.1:8686
      playground: true
    # sources -- Sources for Vector to pull data
    ##
    ##
    sources:
      # Internal logs for Vector
      internal_logs:
        type: internal_logs
      # Internal metrics for Vector
      internal_metrics:
        type: internal_metrics
        scrape_interval_secs: 10
      # Kubernetes logs
      k8s_logs:
        type: kubernetes_logs
        glob_minimum_cooldown_ms: 500
        max_read_bytes: 2097152

Version

0.48.0, 0.49.0, 0.50.0 -distroless

Debug Output

Example Data

No response

Additional Context

Rollback to AL2 backed AMIs resolves the issue. AL2 is end of life in Nov '25.

References

No response

[mscanlon72]

I am assuming Vector is supported on AL2023 EKS AMIs?

[mscanlon72]

Is there a debug version of vector available?

[mscanlon72]

To add some more context, the kubernetes_log source stops working. The theory is a stuck thread. All other components are still working and processing data. We have trace logs sent into splunk, so we provide anything requested. We do not see any failure or error messages on the source. It no longer logs anything, supporting the claim of a stuck thread. The destination we use for logs is splunk_hec, and it continues to receive the internal logs we send. This is the same endpoint that we send the souce kubernetes logs to. This indicates that the destination isn't an issue as it is still receiving the internal logs.

We are trying the alpine variant. Any help would be appreciated.

@pront @jszwedko

[mscanlon72]

Another data point, when a pod exhibits the issue and stops reading logs, the number of timeouts we see on the splunk_hec destination increase dramatically on those pods only. Other pods do not show this increase.

[thomasqueirozb]

Hi @mscanlon72, thanks for this report

Is there a debug version of vector available?

You can run vector with the environment variable VECTOR_LOG=debug or even VECTOR_LOG=trace (which will be very spammy but will give you important trace logs such as this one and others in this file). This will show you debug information from all sources including the file source (the kubernetes_logs source is built on top of the file source).

A major difference between the kubernetes_logs source and the file source is that the former uses a custom paths provides, the K8sPathsProvider. If the source is not properly reading files it is very likely the issue is with that paths provider (granted the files weren't deleted/truncated/moved etc - if that's the case you may be hitting edge cases in the file source itself and then you'll need to debug the checkpoints.json file nested under the data_dir but this is likely not the case). K8sPathsProvider gets its events from kube::Api::all which gets passed into kube::runtime::watcher (source), meaning that all paths are essentially resolved by the kube library and the file source just naively loops over those.

One (maybe) big issue is that we're using kube v0.93.0 but they have since released v2.0.1, so it may be a matter of upgrading to the latest version and this issue could be fixed. That'd be the first step I'd take to try and fix this (along with turning on the debug/trace logs). You can try to update the dependency and create a custom build. Since kube has gone through two major version bumps since v0.93.0 (v1.0.0 in May 2025, v2.0.0 in Sept 2025), I'd recommend a progressive upgrade rather than jumping directly to v2.0.1. Their changelog provides more information as to what was updated, most notably v1.0.0 introduced support for Kubernetes v1.33 and v2.0.0 introduced support for Kubernetes v1.34.

[mscanlon72]

@thomasqueirozb, thank you for your response, we really appreciate it as we are kinda stuck with AL2 EKS AMIs expiring this month.

We have been running with TRACE logs enabled for about a week now, so we have lots of data to provide.

We are using the kubernetes_logs source not the file source. Been using this for a while now and this is really the first major issue we have run into. Also, we do see multiple clusters that are working without error. Some clusters that have the issue, but it only presents in some nodes, not all.

We are curious about the kube-rs library. However, the version of Kubernetes we are using is the same regardless of the AL2 or AL2023 EKS AMI release. Maybe something in the logs can provide some clarity on this potential issue.

We do have a team member that is looking into building a custom Vector release with an updated kube-rs library. We are planning to base it off of 0.49.0 as it is the version we are running in our infra.

We are running Kubernetes Server Version: v1.31.13-eks-113cf36, kube_node_kubelet_version:v1.31.7-eks-473151a with containerd kube_node_runtime_version:containerd://2.0.5

[thomasqueirozb]

We are using the kubernetes_logs source not the file source.

I saw that from your config. I mentioned the file source the kubernetes_logs source uses the FileServer from the file source, therefore you might see some logs that are tagged from there and you should also keep an eye out for those 🙂

Let us know how the deployment with the updated kube library goes, hopefully that should give us some insight

[mscanlon72]

We will, is there anything we can look at in the logs in the meantime? Something we can target for information? We have tons of trace logs ready to query in Splunk.

[thomasqueirozb]

Watching for the logs present in this file: https://github.com/vectordotdev/vector/blob/a488105710032b593051496bad9dc8df5e8cce6c/src/internal_events/file.rs

and these ones too

vector/lib/file-source/src/file_server.rs

Lines 199 to 227 in 02671f4

	trace!(
	message = "Continue watching file.",
	path = ?path,
	);
	} else if !was_found_this_cycle {
	// matches a file with a different path
	info!(
	message = "Watched file has been renamed.",
	path = ?path,
	old_path = ?watcher.path
	);
	watcher.update_path(path).await.ok(); // ok if this fails: might fix next cycle
	} else {
	info!(
	message = "More than one file has the same fingerprint.",
	path = ?path,
	old_path = ?watcher.path
	);
	let (old_path, new_path) = (&watcher.path, &path);
	if let (Ok(old_modified_time), Ok(new_modified_time)) = (
	fs::metadata(old_path).await.and_then(|m| m.modified()),
	fs::metadata(new_path).await.and_then(|m| m.modified()),
	) && old_modified_time < new_modified_time
	{
	info!(
	message = "Switching to watch most recently modified file.",
	new_modified_time = ?new_modified_time,
	old_modified_time = ?old_modified_time,
	);

should give you at least some insight.

Unfortunately it's harder to know why something isn't happening rather than when it is happening... Watching the utilization metric is also a good idea but since you're not seeing anything out of vector tap then it wouldn't be so useful

[mscanlon72]

Those watched files messages disappear once the issue shows up.