Closed
Bug 1225899
Opened 9 years ago
Closed 8 years ago
add beet mover tasks to releasetasks task graph
Categories
(Release Engineering :: Release Automation: Other, defect)
Release Engineering
Release Automation: Other
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jlund, Assigned: jlund)
References
Details
Attachments
(9 files, 2 obsolete files)
566 bytes,
patch
|
rail
:
review+
jlund
:
checked-in+
|
Details | Diff | Splinter Review |
47 bytes,
text/x-github-pull-request
|
rail
:
review+
jlund
:
checked-in+
|
Details | Review |
2.07 KB,
patch
|
Callek
:
review+
jlund
:
checked-in+
|
Details | Diff | Splinter Review |
2.21 KB,
patch
|
jlund
:
review+
jlund
:
checked-in+
|
Details | Diff | Splinter Review |
47 bytes,
text/x-github-pull-request
|
rail
:
review+
|
Details | Review |
826 bytes,
patch
|
rail
:
review+
|
Details | Diff | Splinter Review |
47 bytes,
text/x-github-pull-request
|
Details | Review | |
47 bytes,
text/x-github-pull-request
|
rail
:
review+
|
Details | Review |
48 bytes,
text/x-github-pull-request
|
rail
:
review+
|
Details | Review |
This bug tracks the scheduling of all things beet mover. there will be a task for each 1) en_us build (the promoted CI builds) 2) l10n repack 3) partial they will call a task that runs mozharness with something like: python scripts/beet_mover.py --template configs/beetmover/locale.jinja.template --platform win32 --version 44.0 --locale en_us --taskid xMLQpY17T16xqdPpOm1uTg --generate-manifest --verify-bits --upload-bits
Assignee | ||
Comment 1•8 years ago
|
||
took a look at this today and yesterday. should have some patches up tomorrow am.
Assignee | ||
Comment 2•8 years ago
|
||
this is for part 1: en-US complete push to candidates looks like our date copy of release-runner is not managed by puppet so I'll need to update release-runner.ini to reflect the aws creds addition
Attachment #8708891 -
Flags: review?(rail)
Comment 3•8 years ago
|
||
Comment on attachment 8708891 [details] [diff] [review] 160117_bug_1225899_beetmover_en_US-releaserunner.patch Review of attachment 8708891 [details] [diff] [review]: ----------------------------------------------------------------- ::: buildfarm/release/release-runner.py @@ +171,5 @@ > platform=platform, > )) > platforms[platform] = { > + "task_id": task["taskId"], > + "upload_platform": upload_platform_mapping[platform], Please use buildbot2ftp instead, see https://dxr.mozilla.org/build-central/source/tools/lib/python/release/platforms.py#75 @@ +229,5 @@ > configs_workdir = 'buildbot-configs' > balrog_username = get_config(config, "balrog", "username", None) > balrog_password = get_config(config, "balrog", "password", None) > + beetmover_aws_access_key_id = get_config(config, "beetmover", "aws_access_key_id", None) > + beetmover_aws_secret_access_key = get_config(config, "beetmover", "aws_secret_access_key", None) These are coming from puppet, correct? @@ +302,5 @@ > "signing_class": "dep-signing", > "bouncer_enabled": branchConfig["bouncer_enabled"], > "release_channels": branchConfig["release_channels"], > "signing_pvt_key": signing_pvt_key, > + "push_to_candidates_enabled": True Hmm, sounds more like a branch config variable to me.
Attachment #8708891 -
Flags: review?(rail) → review-
Assignee | ||
Comment 4•8 years ago
|
||
thanks for review :) interdiff: http://people.mozilla.org/~jlund/160118_bug_1225899_beetmover_en-US-tools-interdiff.patch
Attachment #8709157 -
Flags: review?(rail)
Assignee | ||
Updated•8 years ago
|
Attachment #8708891 -
Attachment is obsolete: true
Assignee | ||
Comment 5•8 years ago
|
||
Attachment #8709159 -
Flags: review?(rail)
Assignee | ||
Comment 6•8 years ago
|
||
updated to reflect review comments
Attachment #8709161 -
Flags: review?(rail)
Comment 7•8 years ago
|
||
Comment on attachment 8709157 [details] [diff] [review] 160118_bug_1225899_beetmover_en-US-tools.patch Review of attachment 8709157 [details] [diff] [review]: ----------------------------------------------------------------- ::: buildfarm/release/release-runner.py @@ +164,5 @@ > platform=platform, > )) > platforms[platform] = { > + "task_id": task["taskId"], > + "upload_platform": buildbot2ftp[platform], A nit, buildbot2ftp is a function, need to call it: buildbot2ftp(platform) @@ +222,5 @@ > configs_workdir = 'buildbot-configs' > balrog_username = get_config(config, "balrog", "username", None) > balrog_password = get_config(config, "balrog", "password", None) > + beetmover_aws_access_key_id = get_config(config, "beetmover", "aws_access_key_id", None) > + beetmover_aws_secret_access_key = get_config(config, "beetmover", "aws_secret_access_key", None) Don't forget to submit a puppet patch to add these variables to the config @@ +295,5 @@ > "signing_class": "dep-signing", > "bouncer_enabled": branchConfig["bouncer_enabled"], > "release_channels": branchConfig["release_channels"], > "signing_pvt_key": signing_pvt_key, > + "push_to_candidates_enabled": branchConfig['push_to_candidates_enabled'] Your comment in IRC, that this variable can be used as a environment-specific variable (prod/dev) makes sense to me. Feel free to use either approach.
Attachment #8709157 -
Flags: review?(rail) → review+
Updated•8 years ago
|
Attachment #8709159 -
Flags: review?(rail) → review+
Updated•8 years ago
|
Attachment #8709161 -
Flags: review?(rail) → review+
Assignee | ||
Comment 8•8 years ago
|
||
(In reply to Rail Aliiev [:rail] from comment #7) > Comment on attachment 8709157 [details] [diff] [review] > 160118_bug_1225899_beetmover_en-US-tools.patch > > Review of attachment 8709157 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: buildfarm/release/release-runner.py > @@ +164,5 @@ > > platform=platform, > > )) > > platforms[platform] = { > > + "task_id": task["taskId"], > > + "upload_platform": buildbot2ftp[platform], > > A nit, buildbot2ftp is a function, need to call it: > > buildbot2ftp(platform) > LOL > @@ +222,5 @@ > > configs_workdir = 'buildbot-configs' > > balrog_username = get_config(config, "balrog", "username", None) > > balrog_password = get_config(config, "balrog", "password", None) > > + beetmover_aws_access_key_id = get_config(config, "beetmover", "aws_access_key_id", None) > > + beetmover_aws_secret_access_key = get_config(config, "beetmover", "aws_secret_access_key", None) > > Don't forget to submit a puppet patch to add these variables to the config oh right. I thought because puppet doesn't manage our date instance that it wasn't puppetized yet. But I guess I can add it to https://dxr.mozilla.org/build-central/source/puppet/modules/releaserunner/templates/release-runner.ini.erb which will be shared by production releaserunner too and be a no-op for that. > @@ +295,5 @@ > > "signing_class": "dep-signing", > > "bouncer_enabled": branchConfig["bouncer_enabled"], > > "release_channels": branchConfig["release_channels"], > > "signing_pvt_key": signing_pvt_key, > > + "push_to_candidates_enabled": branchConfig['push_to_candidates_enabled'] > > Your comment in IRC, that this variable can be used as a > environment-specific variable (prod/dev) makes sense to me. Feel free to use > either approach. I'll leave this as is. we will want 'push_to_releases_enabled' to be a branchconfig item so I can keep those side by side.
Assignee | ||
Comment 9•8 years ago
|
||
see motivation from: https://bugzilla.mozilla.org/show_bug.cgi?id=1225899#c8
Attachment #8709206 -
Flags: review?(bugspam.Callek)
Assignee | ||
Comment 10•8 years ago
|
||
Comment on attachment 8709206 [details] [diff] [review] 160118_bug_1225899_beetmover_en-US-puppet.patch ftr - secrets are in secrets.eyaml
Assignee | ||
Comment 11•8 years ago
|
||
Comment on attachment 8709159 [details] [diff] [review] 160118_bug_1225899_beetmover_en-US-bbot-cfgs.patch https://hg.mozilla.org/build/buildbot-configs/rev/fca43032761e
Attachment #8709159 -
Flags: checked-in+
Assignee | ||
Comment 12•8 years ago
|
||
carrying r+ interdiff: diff --git a/buildfarm/release/release-runner.py b/buildfarm/release/release-runner.py index c1cc4e1..1512a60 100755 --- a/buildfarm/release/release-runner.py +++ b/buildfarm/release/release-runner.py @@ -165,7 +165,7 @@ def get_en_US_config(release, branchConfig, branch, index): )) platforms[platform] = { "task_id": task["taskId"], - "upload_platform": buildbot2ftp[platform], + "upload_platform": buildbot2ftp(platform), } return {
Attachment #8709157 -
Attachment is obsolete: true
Attachment #8709210 -
Flags: review+
Updated•8 years ago
|
Attachment #8709206 -
Flags: review?(bugspam.Callek) → review+
Comment 13•8 years ago
|
||
In production: https://hg.mozilla.org/build/buildbot-configs/rev/fca43032761e
Assignee | ||
Comment 14•8 years ago
|
||
Comment on attachment 8709206 [details] [diff] [review] 160118_bug_1225899_beetmover_en-US-puppet.patch thanks, checked in and merged to prod remote: https://hg.mozilla.org/build/puppet/rev/3d9034df8272 remote: https://hg.mozilla.org/build/puppet/rev/0928dc3fed91
Attachment #8709206 -
Flags: checked-in+
Assignee | ||
Updated•8 years ago
|
Attachment #8709161 -
Flags: checked-in+
Assignee | ||
Updated•8 years ago
|
Attachment #8709210 -
Flags: checked-in+
Assignee | ||
Comment 15•8 years ago
|
||
Attachment #8710200 -
Flags: review?(rail)
Assignee | ||
Comment 16•8 years ago
|
||
see https://github.com/mozilla/releasetasks/pull/86 for motivation
Attachment #8710201 -
Flags: review?(rail)
Assignee | ||
Updated•8 years ago
|
Assignee: nobody → jlund
Assignee | ||
Comment 17•8 years ago
|
||
Comment on attachment 8710201 [details] [diff] [review] 160120_bug_1225899_beetmover_en-US-remove_unneeded_buildbot2ftp-tools.patch ftr - I've landed this for now so I can test it.
Assignee | ||
Comment 18•8 years ago
|
||
Comment on attachment 8710200 [details] [review] beetmover template fix ups ftr - I've landed this for now so I can test it. still needs a stamp
Assignee | ||
Comment 19•8 years ago
|
||
hmm, release-runner failed to generate a usable task graph. full log is long because of the output of the task graph in json. end of log: http://people.mozilla.org/~jlund/failed_beetmover_release-runner.log Odd that our tests didn't catch this. Will have to look more tomorrow.
Assignee | ||
Comment 20•8 years ago
|
||
I probably shouldn't be doing it this way. Maybe I can just move my own copy of releasetasks to /home/cltbld/releasetasks and hack on that for a bit till it works rather than pollute github:mozilla/releasetasks with small fixes at a time. at any rate, this patch tries to tackle the issues I outlined in previous comment and it seems to get us past the issue. We even have a task graph submitted with beetmover tasks: https://tools.taskcluster.net/task-graph-inspector/#y0h2HaBZTl6yniUziS6vqA/
Updated•8 years ago
|
Attachment #8710200 -
Flags: review?(rail) → review+
Updated•8 years ago
|
Attachment #8710201 -
Flags: review?(rail) → review+
Assignee | ||
Comment 21•8 years ago
|
||
looking good now: https://console.aws.amazon.com/s3/home?region=us-east-1&bucket=mozilla-releng-beet-mover-dev&prefix=pub/firefox/candidates/46.0b39-candidates/build1/ next up, repacks :) note, `encryptedEnv:` doesn't work unless you have at least one `env:` var defined as well :(
Assignee | ||
Comment 22•8 years ago
|
||
Attachment #8714140 -
Flags: review?(rail)
Comment 23•8 years ago
|
||
Comment on attachment 8714140 [details] [review] adds repack l10n beetmover candidates tasks Commented in the PR. Please, fix the deadline/valid_for issue before you land.
Attachment #8714140 -
Flags: review?(rail) → review+
Assignee | ||
Comment 24•8 years ago
|
||
Assignee | ||
Comment 25•8 years ago
|
||
Comment on attachment 8716679 [details] [review] beetmover releasetasks partials associated mozharness patch: https://bugzilla.mozilla.org/attachment.cgi?id=8716677
Assignee | ||
Updated•8 years ago
|
Attachment #8716679 -
Flags: review?(rail)
Updated•8 years ago
|
Attachment #8716679 -
Flags: review?(rail) → review+
Assignee | ||
Updated•8 years ago
|
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•