Fixing HTTP 500/504 during Git push or fetch

Overview

HTTP 500 (Internal Server Error) and 504 (Gateway Timeout) during Git push or fetch usually point to:

• Reverse proxy timeouts or buffering (Nginx/Apache, load balancers).
• Server-side issues (hooks failing, disk/full inode, permission errors).
• Large transfers (big repos, LFS) that exceed body size or timeouts.
• HTTP/2 or proxy quirks with chunked uploads.

This guide focuses on practical DevOps fixes for Git over HTTP(S).

Quickstart (get unblocked fast)

• Re-run with trace to see where it fails:
  • GIT_TRACE=1 GIT_CURL_VERBOSE=1 git push origin main
• Force HTTP/1.1 to avoid flaky HTTP/2 proxies:
  • git -c http.version=HTTP/1.1 push origin main
• If you control the proxy, raise timeouts and disable request buffering (see minimal example below).
• Try a smaller transfer temporarily:
  • Push fewer refs or split large binary changes; for fetch, use --depth 1 or --filter=blob:none.
• If you can, switch to SSH to bypass HTTP path issues while you fix the server.

Minimal working example (Nginx reverse proxy)

The config below prevents 504s for large Git pushes by allowing long-lived, streaming requests.

# /etc/nginx/conf.d/git.conf
upstream git_backend {
  server 127.0.0.1:8080;  # Your Git server (e.g., Git HTTP backend, Gitea/GitLab Workhorse)
  keepalive 32;
}

server {
  listen 443 ssl;
  server_name git.example.com;

  # Allow large bodies and streaming uploads
  client_max_body_size 0;
  proxy_request_buffering off;
  proxy_buffering off;

  # Generous timeouts for big pushes/fetches
  proxy_read_timeout 3600s;
  proxy_send_timeout 3600s;
  send_timeout 3600s;

  location / {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://git_backend;
  }
}

Client-side diagnostic and fallback commands:

# Trace HTTP session
GIT_TRACE=1 GIT_CURL_VERBOSE=1 git push origin main

# Force HTTP/1.1 (avoid buggy HTTP/2 proxies)
git -c http.version=HTTP/1.1 push origin main

# Shallow fetch to reduce size
git fetch --depth 1 origin main

# Partial clone (skip blobs initially)
git clone --filter=blob:none https://git.example.com/org/repo.git

Apache httpd reverse proxy equivalent:

# In VirtualHost
ProxyPass        / http://127.0.0.1:8080/ keepalive=On retry=0
ProxyPassReverse / http://127.0.0.1:8080/

# Disable body limit and raise timeouts
LimitRequestBody 0
ProxyTimeout 3600
RequestReadTimeout header=20-40,MinRate=1 body=0

# HTTP/1.1 passthrough for streaming
SetEnv force-proxy-request-1.0 false
SetEnv proxy-sendchunks 1

Step-by-step diagnosis

1. Confirm the transport and URL

• git remote -v to ensure HTTPS is used (these errors are HTTP-specific).
• curl -I "https://host/repo.git/info/refs?service=git-receive-pack" to confirm reachability.

1. Enable client tracing

• Use GIT_TRACE=1 GIT_CURL_VERBOSE=1 for push/fetch.
• Indicators of proxy involvement: 100-continue stalls, HTTP/2 h2 framing issues, or a 504 from a gateway.

1. Check reverse proxy/load balancer

• Look for upstream timed out or gateway timeout in error logs.
• Increase proxy_read_timeout/proxy_send_timeout (Nginx) or ProxyTimeout (Apache).
• Disable request buffering so large request bodies stream to the backend (proxy_request_buffering off in Nginx).
• Ensure client_max_body_size/LimitRequestBody are not capping request size.

1. Inspect the Git service/backend

• Check service logs (Git HTTP backend, Gitea, GitLab Workhorse, etc.).
• Verify disk space and inodes: df -h, df -i.
• Validate repo permissions and ownership for the HTTP service user.
• Review server-side hooks (pre-receive, update) for failures or long runtimes.

1. Large repositories and LFS

• For fetch: reduce transfer size using --depth or partial clone filters.
• For push: consider splitting very large changes (especially big binaries) or adopting Git LFS.
• If using LFS, inspect: git lfs logs last and lower concurrency: git config -g lfs.concurrenttransfers 3.

1. HTTP version/protocol quirks

• Force HTTP/1.1 if HTTP/2 is unstable: git config --global http.version HTTP/1.1.
• Some proxies mishandle Expect: 100-continue or chunked uploads; disabling request buffering helps.

1. Retrying and resilience

• Transient 504s may succeed on retry once buffering/timeout issues are fixed.
• Network flaps: try a nearby mirror or SSH temporarily while root cause is addressed.

Common root causes and fixes

• Reverse proxy timeout too low

  • Fix: Increase proxy_read_timeout/proxy_send_timeout/ProxyTimeout to >= 900s for large repos.

• Request body buffering causes long stalls

  • Fix: proxy_request_buffering off (Nginx) or streaming in Apache; ensure backend can handle streaming.

• Body size limits

  • Fix: client_max_body_size 0 (Nginx) or LimitRequestBody 0 (Apache).

• Server-side hooks heavy work

  • Fix: Optimize or async long-running checks; raise application request timeouts accordingly.

• Disk/quotas

  • Fix: Ensure free space and inodes on repo storage and temp directories.

• HTTP/2/proxy bugs

  • Fix: Force HTTP/1.1 on client or disable HTTP/2 on the proxy for Git paths.

Pitfalls to avoid

• Blindly raising timeouts without fixing root cause can mask performance issues.
• Relying on http.postBuffer tweaks: modern Git/libcurl no longer benefits from this; prefer fixing proxy buffering and timeouts.
• Disabling hooks or verification to “make it pass” can break policy and security.
• Mixing LFS and non-LFS for large files may cause inconsistent behavior across proxies.

Performance notes

• Keep repositories healthy: git gc on the server and occasional repack reduce push/fetch size.
• Prefer partial clone and sparse checkout for large monorepos.
• Enable HTTP keepalive on proxies/backends to reduce connection churn.
• If your Git service supports it, store repos and temp space on fast disks and ensure adequate CPU for delta compression.
• SSH can be more tolerant for long-lived transfers; consider offering both transports.

Tiny FAQ

Q: Why do only large pushes fail with 504? A: The proxy likely times out or buffers the request body. Increase timeouts and disable request buffering.

Q: Should I set http.postBuffer to a huge value? A: No. It’s largely obsolete. Fix proxy buffering/limits and server timeouts instead.

Q: Fetch fails but web UI works. Why? A: Web UI uses short requests; Git operations stream large data and hit different proxy paths/timeouts.

Q: Is forcing HTTP/1.1 safe? A: Yes. It avoids certain HTTP/2 proxy bugs and is commonly used for Git over HTTP.

Q: Could hooks cause HTTP 500? A: Yes. A non-zero exit or crash in pre-receive/update hooks returns 500. Check server logs and fix the hook.