TL;DR: Envoy’s default response timeout is 15 seconds. If your backend takes longer than that, Envoy kills the connection and returns a 504 — even if your ALB is configured for 300 seconds. The fix is an HTTPProxy timeout policy. Scroll to the bottom for the YAML.

The 504s Came Out of Nowhere

I was on-call when the alerts fired. A service behind our ingress gateway started throwing 504 Gateway Timeouts. Not all requests — just the ones that took a bit longer to process. The kind that hit a database, did some heavy lifting, and needed more than a few seconds to respond.

My first instinct was to check the ALB. AWS Application Load Balancers have a default idle timeout of 60 seconds, and we had ours set to 300. Plenty of room. The ALB metrics looked clean — no spikes in 5xx errors from the target group itself. The backend pods were healthy, not OOM-killed, not restarting. Logs showed the requests arriving and being processed normally.

So where were the 504s coming from?

The Timeout Chain You Didn’t Know About

When a request enters your Kubernetes cluster through a typical ingress setup, it passes through several layers. Each layer has its own timeout, and the shortest one wins:

Client --> ALB (300s) --> Envoy Ingress (???) --> Backend Pod

I was focused on the ALB and the backend. I completely forgot about the middle layer — Envoy.

Here’s the thing about Envoy that catches people off guard: the default response timeout is 15 seconds. That’s it. Fifteen seconds for your backend to start sending a response, or Envoy drops the connection and hands back a 504.

It doesn’t matter that your ALB is configured for 300 seconds. It doesn’t matter that your backend is happily crunching data and about to return a perfectly valid response at second 16. Envoy has already given up.

Finding the Smoking Gun

Once I started looking at Envoy’s access logs, it was obvious. Requests that completed in under 15 seconds were fine. Anything over? Dead. The logs showed Envoy returning UC (upstream connection termination) response flags right at the 15-second mark.

The frustrating part is that this is well-documented behavior — Envoy’s route.timeout defaults to 15 seconds in Contour’s HTTPProxy. But when you’re setting up ingress for the first time, you’re thinking about TLS, routing rules, path prefixes… not the timeout buried three layers deep in a proxy you didn’t configure directly.

The Fix

If you’re using Contour with HTTPProxy resources (which we are), the fix is straightforward. Add a timeoutPolicy to your route:

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: my-service
  namespace: my-namespace
spec:
  virtualhost:
    fqdn: my-service.example.com
    tls:
      secretName: my-tls-secret
  routes:
    - conditions:
        - prefix: /
      services:
        - name: my-service
          port: 8080
      timeoutPolicy:
        response: "180s"
        idle: "180s"

The two key fields:

  • response: How long Envoy waits for the backend to start sending a response. This is the one that was killing us at 15 seconds.
  • idle: How long Envoy tolerates silence on an already-established connection. Set this to match or you’ll get bitten by a different timeout on streaming or chunked responses.

We went with 180 seconds — enough to cover the roughly 2-minute processing target with a small buffer. You could go higher (the ALB allows up to 300s), but keeping connections open for 5 minutes is wasteful and masks performance problems you should actually be fixing.

Getting Your Timeout Chain Right

After this incident, I mapped out all the timeouts in the request path and made sure they made sense together. The rule of thumb: each layer should have a timeout equal to or greater than the layer behind it.

Client timeout  >=  ALB timeout  >=  Envoy timeout  >=  Backend processing
     ???              300s             180s                ~120s

If your ALB timeout is shorter than Envoy’s, the ALB kills the connection first and Envoy’s setting doesn’t matter. If Envoy’s is shorter than your backend’s processing time (our situation), you get 504s that look like they’re coming from nowhere.

The Real Fix: Don’t Wait That Long

Bumping the timeout to 180 seconds solved the immediate problem, but let’s be honest — if your API regularly takes two minutes to respond, you have a design problem. The better long-term approach for long-running operations is the 202 Accepted + polling pattern:

  1. Client sends request
  2. Backend immediately returns 202 Accepted with a job ID
  3. Client polls a status endpoint until the job completes
  4. Client fetches the result

This keeps all your timeouts short, makes your API more resilient, and gives the client visibility into progress instead of staring at a loading spinner wondering if things are still alive. If your backend ever crosses that 2-minute mark and starts pushing toward the ALB’s 300-second limit, you’ll need this pattern anyway — or at minimum, sending response headers early to keep the connection alive.

Lessons Learned

  1. Know your timeout chain. Every proxy, load balancer, and gateway in your request path has a timeout. Map them all out before you go to production.

  2. Envoy’s 15-second default is aggressive. It’s a sane default for most web traffic, but it will silently kill anything that takes longer. If you’re using Contour, set timeoutPolicy on every HTTPProxy route that handles non-trivial work.

  3. Check Envoy access logs first. The response flags (UC, UF, UT) tell you exactly what happened. Don’t waste time investigating ALBs and backends when the answer is sitting in the proxy logs.

  4. Design for short responses. If a request legitimately takes minutes, use 202 + polling. Your future on-call self will thank you.

The 15-second timeout is one of those defaults that works perfectly until it doesn’t. And when it doesn’t, it fails in the most confusing way possible — silently, between two layers that both look healthy. Now you know where to look.