FAQ

Answers to recurring questions from GitHub issues and community discussions. If your question isn’t covered here, check the rest of the Troubleshooting section or open a GitHub issue.

Why does kubectl exec/port-forward fail when exposing kube-apiserver through Envoy Gateway?

You’ve routed the Kubernetes API server through Envoy Gateway, and plain calls work fine (kubectl get, list, watch), but kubectl exec, attach, or port-forward fail with:

error: unable to upgrade connection: empty server response

The Envoy Gateway access log for the failing request shows a protocol error followed by a rejected upgrade:

{ "method": "GET", "protocol": "HTTP/1.1", "response_code": 502, "response_code_details": "upstream_reset_before_response_started{protocol_error}", "response_flags": "UPE" }
{ "method": "POST", "protocol": "HTTP/1.1", "response_code": 403, "response_code_details": "upgrade_failed" }

Root cause: exec, attach, and port-forward aren’t ordinary request/response calls — they need a bidirectional streaming channel, which kubectl gets by asking the server to switch protocols mid-connection (the SPDY/3.1 or websocket upgrade). Protocol upgrade is an HTTP/1.1-only handshake (a 101 Switching Protocols response). If the connection between Envoy Gateway and kube-apiserver negotiates HTTP/2 instead, that upgrade request has nowhere to go and the request resets. get/list/watch don’t ask for an upgrade, so HTTP/2 handles them without issue.

There are two ways to fix this:

1. Match the upstream protocol to the client’s, on the route (recommended)

   Attach a BackendTrafficPolicy with useClientProtocol: true and an explicit httpUpgrade list. Envoy then negotiates the upstream connection per-request to match what the client asked for — ordinary calls still get HTTP/2, and upgrade requests get the HTTP/1.1 handshake they need.

   apiVersion: gateway.envoyproxy.io/v1alpha1
   kind: BackendTrafficPolicy
   metadata:
     name: kube-api
     namespace: default
   spec:
     targetRefs:
       - group: gateway.networking.k8s.io
         kind: HTTPRoute
         name: kube-api
     httpUpgrade:
       - type: "spdy/3.1"
       - type: "websocket"
     useClientProtocol: true
   

   httpUpgrade must name every subprotocol you need passed through — kubectl has used both spdy/3.1 (older versions) and websocket (current versions) across its history, so list both unless you control every client’s kubectl version.

2. Use a TLSRoute instead of an HTTPRoute

   Passing the connection through at L4 avoids terminating TLS (and negotiating HTTP/2) at the gateway at all. This only works if kube-apiserver’s own certificate is valid for the hostname your clients connect to — on a managed control plane, that certificate usually belongs to the cloud provider and won’t match a custom domain.

Why does kubectl logs -f fail or get cut off after a while?

This is a separate issue with the same symptom family: a long-lived streaming response getting cut off. HTTPRoute rules apply a default request timeout, which eventually terminates a follow-mode log stream. Set an explicit, unbounded timeout on the route rule:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: kube-api
  namespace: default
spec:
  rules:
    - backendRefs: [...]
      timeouts:
        request: 0s

Why are requests with underscored header names rejected with HTTP 400 or a stream reset?

Clients that send headers with underscores in the name (for example X_Forwarded_For) get HTTP/1 400 responses or HTTP/2 stream resets, even though you never configured a ClientTrafficPolicy. This commonly surprises users migrating from nginx-ingress, which accepts those headers by default.

Root cause: ClientTrafficPolicy headers.withUnderscoresAction defaults to RejectRequest even when the field is unset. Envoy Gateway follows Envoy’s edge proxy best practices rather than Envoy’s own ALLOW default. HTTP/1 requests are rejected with 400; HTTP/2 requests end with a stream reset.

To allow those headers, set withUnderscoresAction: Allow on a ClientTrafficPolicy that targets the Gateway:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: allow-underscores
  namespace: default
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  headers:
    withUnderscoresAction: Allow

See HeaderSettings for the other actions (RejectRequest, DropHeader).