Ingress Rule route to multiple backend service ports.

[iacovosb]

Discussed in #2481

^{Originally posted by iacovosb March 11, 2022}
Hello, and thank you for accepting my inquiry.

General information:

kubernetes version: v1.23.3
nginx-ingress vesion: (nginxinc/nginx-ingress OSS) v2.1.1 / nginx:1.21.5
Installation type: bare metal

I have the following service configuration:

apiVersion: v1
kind: Service
metadata:
  name: myservice
  namespace: testing
  labels:
    app: myservice
spec:
  ports:
  - name: http-5591
    port: 5591
    protocol: TCP
    targetPort: 5591 ## this is one of the ports the application is running
  - name: http-5592
    port: 5592
    protocol: TCP
    targetPort: 5592 ## this is another port the application is running

And the respective Ingress config:

ApiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myservice
  namespace: testing
  annotations:
    nginx.org/redirect-to-https: "True"
  tls:
  - hosts:
    - myservice.example.com
    secretName: wildcard-example-com
  rules:
  - host: myservice.example.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: myservice
            port:
             number: 5591
        backend:
          service:
            name: myservice
            port:
             number: 5592

What I am trying to do is Load Balance the requests to myservice.example.com to the same backend service, but by using multiple ports (5591 and 5592 in this example).
Please note that the requests I want to load balance are in the same path.

paths:
   - pathType: Prefix
     path: "/"

The generated config is:

        server 10.244.1.2:5592 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.244.11.10:5592 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.244.18.28:5592 max_fails=1 fail_timeout=10s max_conns=0;

Targeted config (what I am trying to achieve) is:

        server 10.244.1.2:5591 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.244.11.10:5591 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.244.18.28:5591 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.244.1.3:5592 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.244.11.13:5592 max_fails=1 fail_timeout=10s max_conns=0;
        server 10.244.18.38:5592 max_fails=1 fail_timeout=10s max_conns=0;`

I have tried several approaches, but I always seem to fall into a duplication of some sort.
Duplicate resource (for the host if I create multiple Ingress rules, duplicate location if I add both services under the same path, or ignoring the first part of the configuration if it is repeated in any way).

I also tried with multiple services and VirtualServer:
Services configuration:

apiVersion: v1
kind: Service
metadata:
  name: myservice-5591
  namespace: testing
  labels:
    app: myservice
spec:
  ports:
  - name: http-5591
    port: 5591
    protocol: TCP
    targetPort: 5591 ## this is one of the ports the application is running
---
apiVersion: v1
kind: Service
metadata:
  name: myservice-5592
  namespace: testing
  labels:
    app: myservice
spec:
  ports:
  - name: http-5592
    port: 5592
    protocol: TCP
    targetPort: 5592 ## this is one of the ports the application is running

Virtual Server Config:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: myservice-vs 
  namespace: testing
spec:
  host: myservice.example.com
  tls:
    secret: wildcard-example-com 
  upstreams:
  - name: myservice-stream
    service: myservice-5591
    port: 5591
    service: myservice-5592
    port: 5592
  routes:
  - path: /
    action:
      pass: myservice-stream

Again, one of the two services gets ignored:
Generated Config:

upstream vs_myservice {
    zone vs_myservice 256k;

    random two least_conn;
   
    server 10.244.1.3:5592 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.11.13:5592 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.18.38:5592 max_fails=1 fail_timeout=10s max_conns=0

Desired Config:

upstream vs_myservice {
    zone vs_myservice 256k;

    random two least_conn;
    server 10.244.1.2:5591 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.11.10:5591 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.18.28:5591 max_fails=1 fail_timeout=10s max_conns=0  
    server 10.244.1.3:5592 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.11.13:5592 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.18.38:5592 max_fails=1 fail_timeout=10s max_conns=0

Is there any way I can achieve the desired configuration?
Thank you in advance

[github-actions]

Hi @iacovosb thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this 🙂

Cheers!

[brianehlert]

Are these two different upstream groups? (in nginx speak)
Are you trying to balance between the two groups or have equal weight?

Upstreams are generally considered homogeneous - as in, all listening on the same port. The generic nginx way to handle this is generally two different upstream groups. One on one port and the second on its port. With a weight set for each group.
This is how you would also perform a blue/green or canary style roll-out.
In the case of a rollout you would generally have your automation update the weight between the two groups to reduce one and increase the other until all traffic is steered to the new.

In your case the weights would be the same.

Let me see if I can scare up an example.

[brianehlert]

Try this. But setting equal weights for the myservice-5591 and myservice-5592
https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/custom-resources/traffic-splitting

[iacovosb]

Hello @brianehlert , and thank you for the prompt response.
I tried the example you have provided and it works as you suggested.

 kubectl describe virtualserver.k8s.nginx.org/myservice-vs -n testing

Result:

 Routes:
    Path:  /
    Splits:
      Action:
        Pass:  myservice-5591
      Weight:  50
      Action:
        Pass:  myservice-5592
      Weight:  50
  Tls:
    Secret:  wildcard-example-com
  Upstreams:
    Name:     myservice-5591
    Port:     5591
    Service:  myservice-5591
    Name:     myservice-5592
    Port:     5592
    Service:  myservice-5592

Thank you very much, greatly appreciated.