[Failover] NoExecute taint is not added to the unhealthy cluster after a period of time

[ryanwuer]

What happened:
NoExecute taint is not added to the unhealthy cluster after a period of time

What you expected to happen:
NoExecute taint will be added to the unhealthy cluster after a period of time

How to reproduce it (as minimally and precisely as possible):
The doc we find relatable with Failover is here: https://karmada.io/docs/v1.14/userguide/failover/failover-analysis

We got two clusters named gy1 and gy2. And a workload with weight 1:1 is deployed into gy1 and gy2.

We made a network fault injection to check Karmada's failover logic. The network fault is injected to gy2, making it cannot be reached. After a short while, the NoSchedule taint is added to gy2, just like the doc describes. But after 5min(whicn can be set by --failover-eviction-timeout flag), no NoExecute taint is added to gy2.

From the doc we know the default tolerations of PropagationPolicy like below are added by karamda-webhook:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
  namespace: default
spec:
  placement:
    clusterTolerations:
    - effect: NoExecute
      key: cluster.karmada.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: cluster.karmada.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: nginx
    namespace: default

The tolerations are not matched with NoSchedule taint. As a result, when we make a new deploy, all replicas are scheduled to gy1, which makes resources consumption doubled. It's not as expected. What we expect is the new version of the workload will be deployed to gy1 with the same number of replica just like that before the network injection is made. The gy2 may be not reached, but it's the control plane, the workloads in it may be running healthily as usual. We cannot double replicas in gy1, that could make resources being exhausted.

Anything else we need to know?:

Environment:

• Karmada version: v1.14.5
• kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version): kubectl karmada version: version.Info{GitVersion:"v1.12.2", GitCommit:"0e82ce1823fdff2859053e48eebce189d78dc9a1", GitTreeState:"clean", BuildDate:"2025-01-02T12:19:54Z", GoVersion:"go1.22.9", Compiler:"gc", Platform:"darwin/arm64"}
• Others: K8s version: v1.19.3

[ryanwuer]

@zhzhuang-zju Please help confirm if this is a bug or possible misconfiguration

[XiShanYongYe-Chang]

Hi @ryanwuer, the system will no longer automatically add NoExecute taints to the cluster. Please refer to https://karmada.io/docs/v1.14/userguide/failover/cluster-taint-management

[ryanwuer]

Hi @ryanwuer, the system will no longer automatically add NoExecute taints to the cluster. Please refer to https://karmada.io/docs/v1.14/userguide/failover/cluster-taint-management

All right, but the karmada-webhook only adds NoExecute tolerations to the PropagationPolicy, which could lead to the "resources doubled" problem. What's the best way to solve this?

[XiShanYongYe-Chang]

but the karmada-webhook only adds NoExecute tolerations to the PropagationPolicy,

No default tolerations have been added again, ref #6444. Did you not upgrade the karmada-webhook?

which could lead to the "resources doubled" problem

What is this problem?

[ryanwuer]

but the karmada-webhook only adds NoExecute tolerations to the PropagationPolicy,

No default tolerations have been added again, ref #6444. Did you not upgrade the karmada-webhook?

which could lead to the "resources doubled" problem

What is this problem?

We're still using v1.14.5 for all karmada components. In this version, default-not-ready-toleration-seconds and default-unreachable-toleration-seconds still exist and NoExecute tolerations will be added to PropagationPolicy by default, but the taint added to unHealthy cluster is NoSchedule. So two clusters, with weight 1:1, one cluster being unHealthy, replicas in the healthy cluster will be doubled after a new deploy happens.

[XiShanYongYe-Chang]

So two clusters, with weight 1:1, one cluster being unHealthy, replicas in the healthy cluster will be doubled after a new deploy happens.

Thank you for the explanation.

I think that this is the expected behavior: faulty clusters should not be scheduled, and all instances should only be scheduled to healthy clusters. This behavior is also unrelated to the tolerations on PP/CPP.

Could you help describe what kind of behavior you expect?

[ryanwuer]

So two clusters, with weight 1:1, one cluster being unHealthy, replicas in the healthy cluster will be doubled after a new deploy happens.

Thank you for the explanation.

I think that this is the expected behavior: faulty clusters should not be scheduled, and all instances should only be scheduled to healthy clusters. This behavior is also unrelated to the tolerations on PP/CPP.

Could you help describe what kind of behavior you expect?

The unHealthy cluster may be not reached, but it's the control plane, the workloads in computes nodes may be running healthily as usual. We cannot and don't need to double replicas in healthy cluster by default, that could make resources being exhausted.

Do i have any options to avoid the default behavior?

[XiShanYongYe-Chang]

Do i have any options to avoid the default behavior?

I understand that you don't need Failover, so you can just turn it off.

In addition, for newly created applications, if you want to ignore the NoSchedule taint, you can simply add the corresponding toleration on the Pod (PP). For applications that have already been deployed, the scheduler should not modify the scheduling results when rescheduling(This should be the current expected processing logic).

[ryanwuer]

For applications that have already been deployed, the scheduler should not modify the scheduling results when rescheduling

The question is the scheduler will modify the scheduling results(from 1:1 to 1:0) when rescheduling if one of the two clusters got into unHealthy state. The PropagationPolicy for each application is stored in the gitlab (we're using ArgoCD & Gitops), it's not easy to change all of the PP to add NoSchedule tolerations (we got hundreds of applications deployed by Karmada).

There seems no convenient way to change the default Failover logic globally without changing PP one by one.

[XiShanYongYe-Chang]

The question is the scheduler will modify the scheduling results(from 1:1 to 1:0) when rescheduling if one of the two clusters got into unHealthy state.

This behavior seems to differ from the expected outcome; for rescheduling, it will ignore the taints.

There seems no convenient way to change the default Failover logic globally without changing PP one by one.

As mentioned above, this does not seem to be the behavior of Failover. I would like to understand the specific reason why the previous scheduling results were modified during rescheduling.

[ryanwuer]

The question is the scheduler will modify the scheduling results(from 1:1 to 1:0) when rescheduling if one of the two clusters got into unHealthy state.

This behavior seems to differ from the expected outcome; for rescheduling, it will ignore the taints.

There seems no convenient way to change the default Failover logic globally without changing PP one by one.

As mentioned above, this does not seem to be the behavior of Failover. I would like to understand the specific reason why the previous scheduling results were modified during rescheduling.

After one cluster became unHealthy, we make a new deploy like changing image URL or something, the scheduling results will change from 1:1 to 1:0, making replicas in the other healthy cluster doubled. It's easy to reproduce.

[XiShanYongYe-Chang]

After one cluster became unHealthy, we make a new deploy like changing image URL or something, the scheduling results will change from 1:1 to 1:0, making replicas in the other healthy cluster doubled. It's easy to reproduce.

Let me test it. Could you provide your test YAML?

[ryanwuer]

After one cluster became unHealthy, we make a new deploy like changing image URL or something, the scheduling results will change from 1:1 to 1:0, making replicas in the other healthy cluster doubled. It's easy to reproduce.

Let me test it. Could you provide your test YAML?

Sure, here is the key PP yaml:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  labels:
    app.kubernetes.io/instance: music-cicd-autotest-test
    propagationpolicy.karmada.io/permanent-id: 252b70a4-15ff-4755-ac56-28bd7648292c
  name: music-cicd-autotest-test
  namespace: test-101
spec:
  conflictResolution: Abort
  dependentOverrides:
  - music-cicd-autotest-test
  placement:
    clusterTolerations:
    - effect: NoExecute
      key: cluster.karmada.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: cluster.karmada.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
        - targetCluster:
            clusterNames:
            - guiyang-test-1
          weight: 1
        - targetCluster:
            clusterNames:
            - guiyang-test-2
          weight: 1
  preemption: Never
  priority: 0
  resourceSelectors:
  - apiVersion: apps/v1
    kind: ReplicaSet
    labelSelector:
      matchLabels:
        cloudnative.music.netease.com/application: music-cicd-autotest
        cloudnative.music.netease.com/cluster: music-cicd-autotest-test
        cloudnative.music.netease.com/environment: test
    namespace: test-101
  schedulerName: default-scheduler

Anything else do i need to provide?
The network fault is injected to guiyang-test-2. All replicas are scheduled to guiyang-test-1 after RS's spec has been changed.