Symptom¶
While cutting 0.18.0, git push origin v0.18.0 exited 141 — 128 + 13, death by
SIGPIPE. The pre-push hook ran to completion and reported success, then the push
transferred nothing and git printed no diagnostic, because the process died from a
signal rather than returning an error.
Root cause¶
Git opens the connection to the remote before running the pre-push hook, and the
connection then sits idle for the hook’s entire duration. Confirmed with GIT_TRACE=1:
21:23:24.225841 start_command: /usr/bin/ssh git@github.com 'git-receive-pack ...'
21:23:25.296955 run_command: .../pre-push origin git@github.com:MudwoodLabs/pyrxd.git
The ssh transport starts 1.07 s before the hook. GitHub closes an idle
git-receive-pack session after a few minutes. When the hook finally finishes, git
writes the pack to a socket the far end has already dropped, takes SIGPIPE, and dies.
task ci measured 395 s, which is past GitHub’s limit. The evidence is explicit
in the failing output, though it is easy to miss because ssh’s message lands mid-line
in pytest’s progress dots:
...........sConnection to github.com closed by remote host.
Measured bounds on the timeout¶
An idle
git-receive-packsession to GitHub was still alive at 300 s (and had returned a 2032-byte ref advertisement, so it was genuinely established).A
sleep 500hook failed with 141 and the “closed by remote host” message.task ciat 395 s failed 3 times out of 3.task testalone (~194 s) succeeded.
So the cutoff sits between 300 s and 395 s — call it ~5 minutes. It was not pinned more precisely because the actionable guidance (“keep pre-push hooks well under 5 minutes”) does not depend on the exact value.
Test matrix¶
Every row was run; results are observed, not inferred.
Hook |
Duration |
Transport |
Pack sent |
Exit |
|---|---|---|---|---|
none |
0 s |
GitHub ssh |
no (dry-run) |
0 |
|
200 s |
GitHub ssh |
no (dry-run) |
0 |
|
200 s |
GitHub ssh |
yes |
0 |
|
4.5 s |
GitHub ssh |
yes |
0 |
|
~194 s |
GitHub ssh |
yes |
0 |
|
500 s |
GitHub ssh |
yes |
141 |
|
395 s |
GitHub ssh |
yes |
141 (3/3) |
|
395 s |
local file |
yes |
0 |
|
395 s |
|
yes |
0 |
|
200 s |
|
yes |
0 |
loud, 60k lines |
3 s |
local file |
yes |
0 |
The local rows pass at any duration because a local git-receive-pack has no idle
timeout. That asymmetry — same hook, GitHub fails, localhost succeeds — is fully
explained by the timeout and was the sharpest clue.
Genuinely ruled out¶
The hook failing to drain git’s stdin. Git feeds the hook ref data on stdin and
scripts/git-hooks/pre-pushnever reads it, but a throwaway repo with a stdin-ignoring hook pushed fine, as did one whose child drained stdin and exited.Resource exhaustion. No OOM kills in the kernel log; 40 GiB free; fd limit 1M.
Anything local killing git’s
sshchild. Nopkill/killall/killpganywhere intests/,src/orscripts/; the onlyos.killcalls target self-spawned PIDs.
Fix¶
The pre-push hook runs
task ci-fast(lint && format-check && typecheck && check-private-links) — 4.5 s measured, far under the limit. See the update section in local-ci-parity-via-task-ci-and-pre-push-hook.md for what that trades away.Releases do not push tags.
gh release create vX.Y.Z --target <sha>has GitHub create the tag server-side, so no push and no hook is involved. See docs/runbooks/cutting-a-release.md.
If you hit it anyway: retry the push, or run task ci yourself and push with
--no-verify.
Lessons¶
A piped push hides this completely.
git push ... | tailreportstail’s exit status, so 141 reads as 0. Redirect to a file and check$?instead.grepfor the right words. The “closed by remote host” line was present from the second failure onward; a grep forerror|fatal|rejected|deniedwalked straight past it. When a process dies by signal, search for transport-level messages, not application-level ones.A control has to match the case it is controlling for. A
sleep 200hook was used to “rule out” duration while the real hook ran 395 s. It proved nothing, and the wrong conclusion sent the investigation down several dead ends.