generic-worker: log HTTP response body when getting a bad HTTP response code from the queue when uploading artifacts
Categories
(Taskcluster :: Workers, enhancement)
Tracking
(Not tracked)
People
(Reporter: pmoore, Unassigned)
Details
Attachments
(1 file)
I noticed today that if a worker sleeps and loses its claim on a task, when it tries to upload artifacts, the Queue will return a HTTP 401 status code, and the worker will assume it is a worker bug, and exit. However, the HTTP response would probably give an explanation for the bad exit code, and it would be useful to log that in the worker logs (and arguably the task logs, just in case the worker gets a bad exit code for one artifact, but is still able to upload the task log for some reason).
We've had around 14000 failures due to a HTTP 401 status code in this sentry report:
https://sentry.prod.mozaws.net/operations/generic-worker/issues/3248027/?query=is:unresolved
So I am wondering if these might be due to Windows workers sleeping during task execution. In any case, having the extra logging about the queue response would be useful to understand what the root cause of that sentry report is.
Reporter | ||
Comment 1•5 years ago
|
||
Reporter | ||
Updated•5 years ago
|
Updated•5 years ago
|
Reporter | ||
Updated•5 years ago
|
Reporter | ||
Comment 2•5 years ago
|
||
Released in generic-worker 15.1.1.
Comment 3•5 years ago
|
||
deployment in progress...
https://tools.taskcluster.net/groups/RGUMliLGTY2X_euzU3Ajng
Comment 4•5 years ago
|
||
deployment reverted due to tree closure (bug 1566365)
Reporter | ||
Comment 5•5 years ago
|
||
Redeployed without issue second time around (original failure cause currently unknown).
Description
•