Closed Bug 1562964 Opened 5 years ago Closed 5 years ago

generic-worker: log HTTP response body when getting a bad HTTP response code from the queue when uploading artifacts

Categories

(Taskcluster :: Workers, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: pmoore, Unassigned)

Details

Attachments

(1 file)

I noticed today that if a worker sleeps and loses its claim on a task, when it tries to upload artifacts, the Queue will return a HTTP 401 status code, and the worker will assume it is a worker bug, and exit. However, the HTTP response would probably give an explanation for the bad exit code, and it would be useful to log that in the worker logs (and arguably the task logs, just in case the worker gets a bad exit code for one artifact, but is still able to upload the task log for some reason).

We've had around 14000 failures due to a HTTP 401 status code in this sentry report:

https://sentry.prod.mozaws.net/operations/generic-worker/issues/3248027/?query=is:unresolved

So I am wondering if these might be due to Windows workers sleeping during task execution. In any case, having the extra logging about the queue response would be useful to understand what the root cause of that sentry report is.

Assignee: nobody → pmoore
Status: NEW → ASSIGNED
Attachment #9075441 - Flags: review?(miles)
Assignee: pmoore → nobody
Status: ASSIGNED → NEW
Attachment #9075441 - Flags: review+
Attachment #9075441 - Flags: review?(miles)

Released in generic-worker 15.1.1.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

deployment reverted due to tree closure (bug 1566365)

Redeployed without issue second time around (original failure cause currently unknown).

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: