Nick Thomas [:nthomas] (UTC+12) Assignee
	Description • 8 years ago

Revealed by the slow network in bug 1250374, if we take more than 20 minutes to upload to taskcluster we'll fail to reclaim the task, it'll expire, and everything will go pear shaped. See bug 1250374 comment #8.

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildbase.py#1541
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/taskcluster_helper.py#12

mshal set this up originally but I think he's on a work-week this week.

	Justin Wood (:Callek)
	Comment 1 • 8 years ago

Dropping severity since this is not actively blocking anything other than resiliency of our network

	Michael Shal [:mshal]
	Comment 2 • 8 years ago

IMO the easiest thing to do is to call reclaimTask in between each file, which would mean the per-file limit is 20 minutes instead of a per-job limit of 20 minutes. It would be better still if there's an easy way to periodically call reclaimTask in a separate thread or something, but off-hand I don't know how hard that would be to do.

	Chris AtLee [:catlee]
	Comment 3 • 8 years ago

+1. It would help the most common failure modes without over-complicating this logic.

	Nick Thomas [:nthomas] (UTC+12) Assignee
	Comment 4 • 8 years ago

There's other things we could do here, but lets grab the low hanging fruit.

	Nick Thomas [:nthomas] (UTC+12) Assignee
	Comment 5 • 8 years ago

https://treeherder.mozilla.org/#/jobs?repo=try&revision=073959365ab7 if you're interested.

	Michael Shal [:mshal]
	Comment 6 • 8 years ago

Huh, I thought we could've needed to add a new reclaim_task method in taskcluster_helper.py to call http://docs.taskcluster.net/queue/api-docs/#reclaimTask

:jonasfj, does calling claimTask again effectively do the same thing here as reclaimTask as far as resetting the timer?

	Jonas Finnemann Jensen (:jonasfj)
	Comment 7 • 8 years ago

@mshal,
You are right, claimTask != reclaimTask, hmm, I see can't refer to the docs as I didn't write any...

> :jonasfj, does calling claimTask again effectively do the same thing here as reclaimTask as far as
> resetting the timer?
Calling claimTask(taskId, runId) on a task and run that is already running will return 409, conflict.

To post-pone the takenUntil timestamp call reclaimTask(taskId, runId)

	Michael Shal [:mshal]
	Comment 8 • 8 years ago

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #7)
> Calling claimTask(taskId, runId) on a task and run that is already running
> will return 409, conflict.

Hmm, that doesn't seem to jive with nthomas' try push - it looks like it is successful (or something is silently ignoring the error).

> 
> To post-pone the takenUntil timestamp call reclaimTask(taskId, runId)

So, I think we'll want a reclaim_task in taskcluster_helper that does something like:

        self.taskcluster_queue.reclaimTask(
            task['status']['taskId'],
            task['status']['runs'][-1]['runId'])

(untested)

	Rail Aliiev [:rail]
	Comment 10 • 8 years ago

Review commit: https://reviewboard.mozilla.org/r/45353/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/45353/

	Rail Aliiev [:rail]
	Comment 11 • 8 years ago

Comment on attachment 8739729 [details]
MozReview Request: Bug 1250458 - Reclaim task before file uploads r=nthomas

The same approach works fine in create_reference_artifact() in the same helper.

	Nick Thomas [:nthomas] (UTC+12) Assignee
	Comment 12 • 8 years ago

Comment on attachment 8739729 [details]
MozReview Request: Bug 1250458 - Reclaim task before file uploads r=nthomas

https://reviewboard.mozilla.org/r/45353/#review41893

lgtm

	Pulsebot
	Comment 13 • 8 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/b85e9878c32f

	Carsten Book [:Tomcat]
	Comment 14 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/b85e9878c32f