Closed
Bug 1250458
Opened 8 years ago
Closed 8 years ago
taskcluster upload should be able to cope with slow network
Categories
(Release Engineering :: Applications: MozharnessCore, defect)
Release Engineering
Applications: MozharnessCore
Tracking
(firefox48 fixed)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox48 | --- | fixed |
People
(Reporter: nthomas, Assigned: nthomas)
References
Details
Attachments
(1 file, 1 obsolete file)
58 bytes,
text/x-review-board-request
|
nthomas
:
review+
rail
:
checked-in+
|
Details |
Revealed by the slow network in bug 1250374, if we take more than 20 minutes to upload to taskcluster we'll fail to reclaim the task, it'll expire, and everything will go pear shaped. See bug 1250374 comment #8. https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildbase.py#1541 https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/taskcluster_helper.py#12 mshal set this up originally but I think he's on a work-week this week.
Comment 1•8 years ago
|
||
Dropping severity since this is not actively blocking anything other than resiliency of our network
Severity: blocker → major
Comment 2•8 years ago
|
||
IMO the easiest thing to do is to call reclaimTask in between each file, which would mean the per-file limit is 20 minutes instead of a per-job limit of 20 minutes. It would be better still if there's an easy way to periodically call reclaimTask in a separate thread or something, but off-hand I don't know how hard that would be to do.
Comment 3•8 years ago
|
||
+1. It would help the most common failure modes without over-complicating this logic.
Assignee | ||
Comment 4•8 years ago
|
||
There's other things we could do here, but lets grab the low hanging fruit.
Assignee | ||
Comment 5•8 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=073959365ab7 if you're interested.
Comment 6•8 years ago
|
||
Huh, I thought we could've needed to add a new reclaim_task method in taskcluster_helper.py to call http://docs.taskcluster.net/queue/api-docs/#reclaimTask :jonasfj, does calling claimTask again effectively do the same thing here as reclaimTask as far as resetting the timer?
Flags: needinfo?(jopsen)
Comment 7•8 years ago
|
||
@mshal,
You are right, claimTask != reclaimTask, hmm, I see can't refer to the docs as I didn't write any...
> :jonasfj, does calling claimTask again effectively do the same thing here as reclaimTask as far as
> resetting the timer?
Calling claimTask(taskId, runId) on a task and run that is already running will return 409, conflict.
To post-pone the takenUntil timestamp call reclaimTask(taskId, runId)
Flags: needinfo?(jopsen)
Comment 8•8 years ago
|
||
(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #7) > Calling claimTask(taskId, runId) on a task and run that is already running > will return 409, conflict. Hmm, that doesn't seem to jive with nthomas' try push - it looks like it is successful (or something is silently ignoring the error). > > To post-pone the takenUntil timestamp call reclaimTask(taskId, runId) So, I think we'll want a reclaim_task in taskcluster_helper that does something like: self.taskcluster_queue.reclaimTask( task['status']['taskId'], task['status']['runs'][-1]['runId']) (untested)
Updated•8 years ago
|
Attachment #8722797 -
Flags: review?(mshal) → feedback+
Assignee | ||
Updated•8 years ago
|
Attachment #8722797 -
Attachment is obsolete: true
Comment 10•8 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/45353/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/45353/
Attachment #8739729 -
Flags: review?(nthomas)
Comment 11•8 years ago
|
||
Comment on attachment 8739729 [details] MozReview Request: Bug 1250458 - Reclaim task before file uploads r=nthomas The same approach works fine in create_reference_artifact() in the same helper.
Assignee | ||
Comment 12•8 years ago
|
||
Comment on attachment 8739729 [details] MozReview Request: Bug 1250458 - Reclaim task before file uploads r=nthomas https://reviewboard.mozilla.org/r/45353/#review41893 lgtm
Attachment #8739729 -
Flags: review?(nthomas) → review+
Updated•8 years ago
|
Attachment #8739729 -
Flags: checked-in+
Comment 14•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/b85e9878c32f
You need to log in
before you can comment on or make changes to this bug.
Description
•