getting 404 not found with cafe.example.com ingress using examples with kubernete-ingress 1.11.3

[llyons]

Describe the bug
with kubernete 1.21.1 cluster (with 3 linux and 3 windows nodes) and nginx-ingress (kubernetes-ingress v1.11.3 ) we are constantly getting 404 not found

To Reproduce
Steps to reproduce the behavior:

1. installed the nginx-ingress controller with manifests using the documentation on https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
2. deploying the cafe.example.com from the version v1.11.3 of kubernetes-ingress using the documented process in README.md with the slight modification of using nodeselector to force the running on the linux nodes. We have added the nginx-ingress external ip (using metallb) to the host file for cafe.example.com (10.244.0.30 cafe.example.com)

kubectl get svc -o wide -n nginx-ingress

 nginx-ingress   LoadBalancer   10.111.32.192   10.244.0.30   80:32132/TCP,443:32544/TCP   14d   app=nginx-ingress   
   shows the ingress controller running with a metallb provided external IP

kubectl get ingress

NAME CLASS HOSTS ADDRESS PORTS AGE
cafe-ingress cafe.example.com 80, 443 39m

kubectl get pods -A showing only cafe related

default coffee-6d68b8ff4f-8jdkf 1/1 Running 0 31m
default coffee-6d68b8ff4f-g5rpk 1/1 Running 0 31m
default tea-766cdd6757-gdj57 1/1 Running 0 31m
default tea-766cdd6757-kswcj 1/1 Running 0 31m
default tea-766cdd6757-smqjx 1/1 Running 0 31m

doing curl --resolve cafe.example.com:80:10.244.0.30 -v http://cafe.example.com returns 404

• Added cafe.example.com:80:10.244.0.30 to DNS cache
• About to connect() to cafe.example.com port 80 (#0)
• Trying 10.244.0.30...
• Connected to cafe.example.com (10.244.0.30) port 80 (#0)

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: cafe.example.com
Accept: /

< HTTP/1.1 404 Not Found
< Server: nginx/1.21.0
< Date: Thu, 17 Jun 2021 19:22:37 GMT
< Content-Type: text/html
< Content-Length: 153
< Connection: keep-alive
<

<title>404 Not Found</title>

404 Not Found

---

nginx/1.21.0 * Connection #0 to host cafe.example.com left intact 2. View logs on '....' 3. See error

Expected behavior
expected the nginx-ingress to route to the cafe or tea deployments

Your environment

• Kubernetes v 1.21.1
• On premise bare metal setup with 3 linux 3 windows nodes... linux control plane.
• using nginx

kubectl get ingress cafe-ingress

NAME CLASS HOSTS ADDRESS PORTS AGE
cafe-ingress cafe.example.com 80, 443 43m

kubectl describe ingress cafe-ingress

Name: cafe-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
cafe-secret terminates cafe.example.com
Rules:
Host Path Backends

---

cafe.example.com
/tea tea-svc:80 (192.168.1.26:8080,192.168.2.33:8080,192.168.2.34:8080)
/coffee coffee-svc:80 (192.168.2.31:8080,192.168.2.32:8080)
Annotations:
Events:

curl 192.168.1.26:8080

Server address: 192.168.1.26:8080
Server name: tea-766cdd6757-kswcj
Date: 17/Jun/2021:19:26:10 +0000
URI: /
Request ID: 60d611f651c50b9186804c420882fc8e

curl cafe.example.com

<title>404 Not Found</title>

404 Not Found

---

nginx/1.21.0

[github-actions]

Hi @llyons thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this 🙂

Cheers!

[llyons]

A couple of additional notes. I was trying to check to see if the config for the ingress was generated and I am not able to see anything in /etc/nginx. There is no folder named /etc/nginx

running kubectl exec -n nginx-ingress -- cat /etc/nginx/nginx.conf

returns

`

worker_processes auto;
daemon off;

error_log stderr notice;
pid /var/lib/nginx/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';


access_log  /dev/stdout  main;




sendfile        on;
#tcp_nopush     on;

keepalive_timeout 65s;
keepalive_requests 100;

#gzip  on;

server_names_hash_max_size 1024;
server_names_hash_bucket_size 256;

variables_hash_bucket_size 256;
variables_hash_max_size 1024;

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
map $http_upgrade $vs_connection_header {
    default upgrade;
    ''      $default_connection_header;
}








server {
    # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
    set $default_connection_header "";
    set $resource_type "";
    set $resource_name "";
    set $resource_namespace "";
    set $service "";

    listen 80 default_server;


    listen 443 ssl default_server;


    ssl_certificate /etc/nginx/secrets/default;
    ssl_certificate_key /etc/nginx/secrets/default;





    server_name _;
    server_tokens "on";






    location / {
        return 404;
    }
}
# stub_status
server {
    listen 8080;

    allow 127.0.0.1;
    deny all;

    location /stub_status {
        stub_status;
    }
}

include /etc/nginx/config-version.conf;
include /etc/nginx/conf.d/*.conf;

server {
    listen unix:/var/lib/nginx/nginx-502-server.sock;
    access_log off;



    return 502;
}

server {
    listen unix:/var/lib/nginx/nginx-418-server.sock;
    access_log off;



    return 418;
}

}

stream {
log_format stream-main '$remote_addr [$time_local] '
'$protocol $status $bytes_sent $bytes_received '
'$session_time "$ssl_preread_server_name"';

access_log  /dev/stdout  stream-main;





include /etc/nginx/stream-conf.d/*.conf;

}`

[jasonwilliams14]

Hi @llyons
Can you tell me about the coffee/tea applications that you deployed?
What namespace did you deploy them into?
For the NGINX Ingress controller, what namespace did you deploy NGINX Ingress into?

could you provide the output of:

kubectl get po,svc -A

We will take a look at your results.
Thank you.

[llyons]

Thanks Jason, Much appreciated.

NGINX is in the nginx-ingress namespace
kubectl get po,svc -A

I suspect there is something small. I am not getting anything generated in the /etc/nginx/conf.d folder. There is actually no /etc/nginx folder

NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/aks-helloworld-one-75c9fcd88-sx48t 1/1 Running 0 8h
default pod/aks-helloworld-two-6dcdb7f788-w7nx2 1/1 Running 0 8h
default pod/coffee-6d68b8ff4f-8jdkf 1/1 Running 0 7h42m
default pod/coffee-6d68b8ff4f-g5rpk 1/1 Running 0 7h42m
default pod/tea-766cdd6757-gdj57 1/1 Running 0 7h42m
default pod/tea-766cdd6757-kswcj 1/1 Running 0 7h42m
default pod/tea-766cdd6757-smqjx 1/1 Running 0 7h42m
kube-system pod/coredns-558bd4d5db-2zwrp 1/1 Running 1 20d
kube-system pod/coredns-558bd4d5db-tsbb8 1/1 Running 1 20d
kube-system pod/etcd-iabrl-k8inp01.olh.local 1/1 Running 1 20d
kube-system pod/kube-apiserver-iabrl-k8inp01.olh.local 1/1 Running 1 20d
kube-system pod/kube-controller-manager-iabrl-k8inp01.olh.local 1/1 Running 1 20d
kube-system pod/kube-flannel-ds-4npfv 1/1 Running 1 20d
kube-system pod/kube-flannel-ds-fxqlw 1/1 Running 1 20d
kube-system pod/kube-flannel-ds-qpvpn 1/1 Running 3 20d
kube-system pod/kube-flannel-ds-windows-amd64-fd6pk 1/1 Running 3 12h
kube-system pod/kube-flannel-ds-windows-amd64-kxkvz 1/1 Running 6 9d
kube-system pod/kube-flannel-ds-windows-amd64-ppxd7 1/1 Running 9 9d
kube-system pod/kube-proxy-9q8f7 1/1 Running 1 20d
kube-system pod/kube-proxy-dtv2g 1/1 Running 1 20d
kube-system pod/kube-proxy-windows-d7wmq 1/1 Running 1 12h
kube-system pod/kube-proxy-windows-f5w95 1/1 Running 6 7d4h
kube-system pod/kube-proxy-windows-xx59w 1/1 Running 6 7d4h
kube-system pod/kube-proxy-xfdk5 1/1 Running 1 20d
kube-system pod/kube-scheduler-iabrl-k8inp01.olh.local 1/1 Running 1 20d
metallb-system pod/controller-6b78bff7d9-pqjsq 1/1 Running 0 11h
metallb-system pod/speaker-c4k5v 1/1 Running 0 11h
metallb-system pod/speaker-gdzpv 1/1 Running 0 11h
metallb-system pod/speaker-s85mj 1/1 Running 0 11h
nginx-ingress pod/nginx-ingress-55f77659df-nrsqp 1/1 Running 0 2d9h
nginx-ingress pod/nginx-ingress-5bw4f 1/1 Running 0 9d
nginx-ingress pod/nginx-ingress-m6btc 1/1 Running 0 9d

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/aks-helloworld-one ClusterIP 10.96.236.14 80/TCP 8h
default service/aks-helloworld-two ClusterIP 10.105.111.115 80/TCP 8h
default service/coffee-svc ClusterIP 10.109.253.105 80/TCP 7h52m
default service/kubernetes ClusterIP 10.96.0.1 443/TCP 20d
default service/tea-svc ClusterIP 10.102.6.16 80/TCP 7h52m
kube-system service/kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 20d
nginx-ingress service/nginx-ingress LoadBalancer 10.111.32.192 10.244.0.30 80:32132/TCP,443:32544/TCP 15d

``

[llyons]

Some additional results from some of the troubleshooting commands.

kubectl describe pol webapp-policy
Error from server (NotFound): policies.k8s.nginx.org "webapp-policy" not found

kubectl describe vsr coffee
Error from server (NotFound): virtualserverroutes.k8s.nginx.org "coffee" not found

kubectl describe vs cafe
Error from server (NotFound): virtualservers.k8s.nginx.org "cafe" not found

kubectl describe ing cafe-ingress

Name: cafe-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
cafe-secret terminates cafe.example.com
Rules:
Host Path Backends

---

cafe.example.com
/tea tea-svc:80 (192.168.1.26:8080,192.168.2.33:8080,192.168.2.34:8080)
/coffee coffee-svc:80 (192.168.2.31:8080,192.168.2.32:8080)
Annotations:
Events:

[llyons]

I have tried to set the cafe.example.com host to the external IP of the ingress controller served up by metallb and I have tried to set cafe.example.com to the ip of one of the linux nodes where ingress controllers are running. In both cases I get this.

$ curl http://cafe.example.com/

<title>404 Not Found</title>

404 Not Found

---

nginx/1.21.0

$ curl http://cafe.example.com/tea

<title>404 Not Found</title>

404 Not Found

---

nginx/1.21.0

$ curl http://cafe.example.com/coffee

<title>404 Not Found</title>

404 Not Found

---

nginx/1.21.0

$ curl 10.109.253.105
Server address: 192.168.2.32:8080
Server name: coffee-6d68b8ff4f-8jdkf
Date: 18/Jun/2021:13:58:28 +0000
URI: /
Request ID: 7914329ab48caa9fd5ffc35f2234c894

$ curl 192.168.2.32:8080
Server address: 192.168.2.32:8080
Server name: coffee-6d68b8ff4f-8jdkf
Date: 18/Jun/2021:13:58:57 +0000
URI: /
Request ID: 589d81df45de33ddff1df34b2f572899

[jasonwilliams14]

Hi @llyons
Can you verify if your ingress has a class set?
Here is an example from my lab:

NAMESPACE   NAME           CLASS   HOSTS              ADDRESS      PORTS     AGE
default     cafe-ingress   nginx   cafe.example.com   172.18.0.2   80, 443   5m4s

it is hard to tell from your output if a Class was set for the Ingress resource.
If it is not set, you can set a class in the Ingress manifest like so:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
spec:
  ingressClassName: nginx # change this to your Ingressclass you have created in your cluster
  tls:
  - hosts:
    - cafe.example.com
    secretName: cafe-secret
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

Try that and let us know.

[llyons]

I did add the ingressClassname or uncommented it in the example and it seems to work. I was able set the cafe.example.com to the IP of the ingress (that was served up by metallb) and go to http://cafe.example.com/tea and http://cafe.example.com/coffee and get this.

Server address: 192.168.2.31:8080
Server name: coffee-6d68b8ff4f-g5rpk
Date: 19/Jun/2021:00:52:53 +0000
URI: /coffee
Request ID: 4767e173143b06a517639133b8f34a6b

does that mean its working from your perspective?

I still dont see files generated in /etc/nginx/conf.d

[jasonwilliams14]

Hello @llyons
That looks correct. That is our sample coffee/tea app that responds back with some info on the app/pod, also verifying that the Ingress controller is accepting and routing traffic correctly to your applications in the cluster.

I still dont see files generated in /etc/nginx/conf.d

What you can do to check the config, logon to the NGINX ingress pod and execute:

nginx -T
You see at the top that main file being nginx.conf and any additional includes will also be listed.

Let us know if you have any other questions.
Cheers!

[llyons]

when you say logon to the NGINX ingress pod, how is that done? I have never done that. Are you meaning create a shell into the ingress container and run that command?

Thanks

[llyons]

i seem to be missing something that might help understand how the routing should be setup to make these work.

I have another helloworld example that I pulled over from an AKS deploy. it is tied to host helloworld.onlifehealth.com which points to the IP of the ingress controller like the cafe.example.com prior.

`apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  rules:
    - host: helloworld.onlifehealth.com
      http:
        paths:
        - path: /hello-world-one
          pathType: Prefix
          backend:
            service: 
              name: aks-helloworld-one
              port: 
                number: 80
        - path: /hello-world-two
          pathType: Prefix
          backend:
            service: 
              name: aks-helloworld-two
              port: 
                number: 80
        - path: /
          pathType: Prefix
          backend:
            service: 
              name: aks-helloworld-one
              port: 
                number: 80`

and the two deployments /services

`apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-one  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-one
  template:
    metadata:
      labels:
        app: aks-helloworld-one
    spec:
      containers:
      - name: aks-helloworld-one
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "Welcome to Azure Kubernetes Service (AKS)"
      nodeSelector:
        kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-one  
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-one`

`apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-two  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-two
  template:
    metadata:
      labels:
        app: aks-helloworld-two
    spec:
      containers:
      - name: aks-helloworld-two
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
      nodeSelector:
        kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-two  
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-two`

with the above example I can see a response on http://helloworld.onlifehealth.com but i am seeing

Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

with these URLs http://helloworld.onlifehealth.com/hello-world-one and http://helloworld.onlifehealth.com/hello-world-two

[jasonwilliams14]

Hi @llyons
Looking at your settings:

    nginx.ingress.kubernetes.io/ssl-redirect: "false"         
    nginx.ingress.kubernetes.io/use-regex: "true"      
    nginx.ingress.kubernetes.io/rewrite-target: /$1

Those specific annotations are for a different Ingress controller project and will not work with this particular NGINX Ingress controller. Those above are for the Community Edition of NGINX Ingress Controller maintained by the community.
This particular Ingress Controller on this repo, is maintained by NGINX inc.

You will want to look at these annotations to use with your Ingress:

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

You will need to convert your annotations to the ones supported by this Ingress product.
Keep in mind that for regex, that is supported in our CRD using VirtualServer and VirtualServerRoute, which provides more advanced functionality and routing capabilities:

https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/

Here is a quick write I did of converting your Ingress resource to a VirtualServer resource:

apiVersion: k8s.nginx.org/v1
  kind: VirtualServer
  metadata:
    name: aks-helloworld
  spec:
    host: helloworld.onlifehealth.com
    tls:
      secret: helloworld-secret
    upstreams:
    - name: aks-helloworld-one
      service: aks-helloworld-one-svc
      port: 80
    - name: aks-helloworld-two
      service: aks-helloworld-two-svc
      port: 80
    routes:
    - path: /
      action:
        pass: aks-helloworld-one
    - path: /hello-world-two
      action:
        pass: aks-helloworld-two

NOTE: I added svc to the end of the service just for clarity.

In the above, you can remove the tls/secret spec. That does TLS termination. I put it there for an example.

Any request that comes into helloworld.onelifehealth.com routes to aks-helloworld-one app.
Requests coming into helloworld.onelifehealth.com/hello-world-two get routed to aks-helloworld-two app.

Pardon any typos.
Let us know if that is helpful.

[llyons]

thanks alot. i appreciate that. I was able to get a few other of our apps working behind the ingress.

I will spend time this afternoon trying to digest the info you just provided like the virtualserver kind and the different annotations

again thanks

[jasonwilliams14]

@llyons You are very welcome.
Happy to help out anytime.
Cheers

[llyons]

one more question..

I change the hello world to use the virtual server and change the service definitions to have -svc as you did.

I am able to curl the IP of the pod and the IP of the service

NAMESPACE NAME READY STATUS RESTARTS AGE IP
default pod/aks-helloworld-one-75c9fcd88-bv8mk 1/1 Running 0 5m20s 192.168.1.28
default pod/aks-helloworld-two-6dcdb7f788-qgxh2 1/1 Running 0 5m15s 192.168.1.29

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/aks-helloworld-one-svc ClusterIP 10.110.232.106 80/TCP 5m21s app=aks-helloworld-one
default service/aks-helloworld-two-svc ClusterIP 10.103.104.138 80/TCP 5m16s app=aks-helloworld-two

so curl 192.168.1.28 or 10.110.232.106 returns data from app hello-world-one
so curl 192.168.1.29 or 10.103.104.138 returns data from app hello-world-two

however i have my ingress controller set to IP 10.244.0.30

nginx-ingress service/nginx-ingress LoadBalancer 10.111.32.192 10.244.0.30 80:32132/TCP,443:32544/TCP 18d app=nginx-ingress

and have helloworld.onlifehealth.com set to this ip in my hosts file.

http://helloworld.onlifehealth.com works
http://helloword.onlifehealth.com/hello-world-one or two give me this error.

should the virtualserver have an IP? I am guessing I cant get to the new virtualserver through the ingress controller IP.

kubectl get vs

NAME STATE HOST IP PORTS AGE
aks-helloworld Valid helloworld.onlifehealth.com 113m

also

kubectl describe vs aks-helloworld

Name: aks-helloworld
Namespace: default
Labels:
Annotations:
API Version: k8s.nginx.org/v1
Kind: VirtualServer
Metadata:
Creation Timestamp: 2021-06-21T18:58:08Z
Generation: 1
Managed Fields:
API Version: k8s.nginx.org/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:host:
f:routes:
f:upstreams:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-06-21T18:58:08Z
API Version: k8s.nginx.org/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:externalEndpoints:
f:message:
f:reason:
f:state:
Manager: nginx-ingress
Operation: Update
Time: 2021-06-21T18:58:08Z
Resource Version: 3505779
UID: 00cc13d8-6869-4c29-a450-2dd2b865509f
Spec:
Host: helloworld.onlifehealth.com
Routes:
Action:
Pass: aks-helloworld-one
Path: /
Action:
Pass: aks-helloworld-one
Path: /hello-world-one
Action:
Pass: aks-helloworld-two
Path: /hello-world-two
Upstreams:
Name: aks-helloworld-one
Port: 80
Service: aks-helloworld-one-svc
Name: aks-helloworld-two
Port: 80
Service: aks-helloworld-two-svc
Status:
External Endpoints:
Ip:
Ports:
Message: Configuration for default/aks-helloworld was added or updated
Reason: AddedOrUpdated
State: Valid
Events: