Debugging Kubernetes Ingress: What 502, 503, and 504 Are Actually Telling You

The status code is the first bisection

When traffic to a Kubernetes service is failing at the edge, the fastest thing you can do is stop treating “the ingress is broken” as one problem. The request path has distinct layers, the error code tells you which layer gave up, and each layer has a different set of usual suspects.

client → LB → ingress controller → Service → Endpoints → pod → container
                     ↑                ↑          ↑         ↑
                    503              503        503       502/504

• 503 Service Unavailable — the controller had no healthy backend. The request never reached your code.
• 502 Bad Gateway — the controller reached a pod and got back something it could not use.
• 504 Gateway Timeout — the controller reached a pod, and the pod never answered in time.
• 404 from the controller’s default backend — no Ingress rule matched the host or path at all.

That single distinction eliminates most of the search space before you have run a command.

503: nothing to route to

Almost every 503 resolves to one command:

kubectl get endpoints my-service -o wide

If ENDPOINTS is <none>, the ingress is behaving correctly. There is genuinely nothing to send traffic to, and the problem is upstream of routing. Three things produce an empty endpoint list.

The Service selector does not match the pods. This is the most common one, and it survives review because both objects look right in isolation:

kubectl get svc my-service -o jsonpath='{.spec.selector}'   # {"app":"my-api"}
kubectl get pods -l app=my-api                              # No resources found
kubectl get pods --show-labels | grep my-api                # app=my-api-server

A label renamed in the Deployment and not in the Service, or a Helm value that changed the app name in one template but not the other. The pods are running perfectly and are invisible to the Service.

The pods are running but not ready. Endpoints only include pods that pass readiness. A pod in Running with 0/1 in the READY column is excluded from the endpoint list by design:

kubectl get pods -l app=my-api
kubectl describe pod my-api-xxxx | grep -A5 'Readiness\|Warning'

Readiness probe failures at this stage are usually pointed at the wrong port, pointed at a path that requires auth, or firing before the application has finished a slow startup. That last one wants a startupProbe rather than a generous initialDelaySeconds — the startup probe lets the application take as long as it needs to boot without weakening the liveness check that protects it afterwards. If the pods are not merely unready but actively restarting, that is a different investigation, and the CrashLoopBackOff playbook is the faster path.

The Service targets a port the container is not serving. targetPort must match the container’s actual listening port, or a named port the container declares:

kubectl get svc my-service -o jsonpath='{.spec.ports[*]}'
kubectl exec -it my-api-xxxx -- ss -lntp

Then there is the 503 that comes with a healthy endpoint list, which means the Ingress and the Service are not connected at all:

kubectl describe ingress my-ingress

Look at three things in that output. ADDRESS empty means no controller has claimed this Ingress — usually a missing or misspelled ingressClassName, so the object exists and nothing is acting on it. A backend showing <error: endpoints "x" not found> means the Ingress names a Service that does not exist in that namespace, frequently because the Ingress and the Service live in different namespaces. And the port in the backend must match a port name or number on the Service, not the container’s port.

502: it connected, and the answer was unusable

A 502 means service discovery worked. The controller opened a connection to a pod and could not turn what came back into an HTTP response. Bypass the ingress and go straight at the pod to confirm the application itself is fine:

kubectl port-forward pod/my-api-xxxx 8080:8080
curl -v localhost:8080/healthz

If that works and the ingress still 502s, the mismatch is between the controller and the pod. The recurring causes:

Protocol mismatch. The container is serving TLS on its port and the controller is speaking plaintext to it, or the reverse. With nginx-ingress this is the backend-protocol annotation; if the app terminates TLS itself you need HTTPS, and the default assumes it does not.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

The gRPC variant of this is the same bug wearing different clothes: a gRPC backend behind a controller defaulting to HTTP/1.1 needs backend-protocol: "GRPC", and without it every call fails at the edge while the service is provably healthy from inside the cluster.

The pod died mid-request. A 502 that appears in bursts alongside restarts is not a routing problem — it is the application crashing under specific inputs or getting OOMKilled. kubectl logs --previous on a restarted pod shows what it was doing when it went.

Response too large for the proxy buffer. Large headers, or a big response body with buffering on, hits nginx’s proxy_buffer_size and returns 502 for exactly the requests that produce large responses and nothing else. It looks like a mysterious per-endpoint failure until you notice the pattern.

The pod was terminating. During a rolling update, a pod that receives SIGTERM and stops accepting connections while the endpoint list still contains it will refuse in-flight connections. A preStop sleep of a few seconds lets the endpoint removal propagate to the controller before the process stops listening. This is one of the small things that separates a rollout with a blip from a genuinely zero-downtime migration.

504: the request is queueing, not failing

A 504 means the pod accepted the request and the controller stopped waiting. The default proxy-read-timeout on nginx-ingress is 60 seconds, so a 504 says your application took longer than a minute to respond.

The distinction that matters: does it happen on every request to that path, or only under load?

Consistent 504s on a specific endpoint mean that endpoint is genuinely slow — an unindexed query, a synchronous call to a slow third party, a report generated inline that should be a background job. Raise the timeout for that path if the work genuinely takes that long, and move it off the request path if it does not.

Load-dependent 504s are queueing, and that is a different fix. The application is not slow at the work; requests are waiting for a resource before the work starts. Saturated worker threads, an exhausted database connection pool, a downstream dependency that got slow and is now holding your workers hostage. Raising proxy-read-timeout here is actively harmful — it lets the queue grow deeper before anything sheds load, converting a fast failure into a slow cascade. Look at concurrency limits, pool sizes, and whether the slow dependency has a timeout of its own.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"   # a real fix only if
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"   # the work is truly slow

The two that look like ingress bugs and are not

NetworkPolicy. If a namespace has a default-deny policy and no rule admitting the ingress controller’s namespace, every request 503s while every pod is healthy and every Service has endpoints. It is invisible in ingress, Service, and pod state — all three look correct. This is the failure that most often produces “it works in staging,” because staging usually has no policies at all.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
spec:
  podSelector:
    matchLabels: { app: my-api }
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: ingress-nginx }

TLS Secret problems. A tls.secretName pointing at a Secret in the wrong namespace, or one cert-manager has not issued yet, gives you the controller’s self-signed default certificate. Browsers show a certificate warning rather than a routing error, which sends people to look at DNS instead of at the Secret.

kubectl get secret my-tls-cert -n my-namespace
kubectl get certificate,certificaterequest -n my-namespace
openssl s_client -connect my-host:443 -servername my-host </dev/null 2>/dev/null | openssl x509 -noout -subject -dates

Work the path, in order

When it is live and you need a decision quickly, the order that resolves it fastest is: read the status code to pick the layer, kubectl get endpoints to confirm whether there is anything to route to, kubectl describe ingress to confirm the Ingress is claimed and points at a real Service and port, port-forward straight to the pod to separate an application problem from a routing problem, and only then look at NetworkPolicy and TLS for the cases where every object looks correct and traffic still does not arrive.

Most of the time the ingress is reporting accurately and something below it is broken. The controller is rarely the bug; it is the component honest enough to tell you which layer is.

That is the work I get called in for under API gateways and networking, and when it is a live outage, infrastructure debugging and incident response.