Closed
Bug 1215204
Opened 9 years ago
Closed 8 years ago
Add bouncer submitter to release promotion graph
Categories
(Release Engineering :: Release Automation: Other, defect)
Release Engineering
Release Automation: Other
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: rail, Assigned: kmoir)
References
Details
Attachments
(18 files, 20 obsolete files)
The builder definition may need some changes to not require config time variables, like version number, build number etc.
Assignee | ||
Updated•9 years ago
|
Assignee: nobody → kmoir
Assignee | ||
Comment 1•9 years ago
|
||
A few questions So we are just going to replace the existing builders here http://hg.mozilla.org/build/buildbotcustom/file/664d99eb01d6/process/release.py#l1709 with jobs that can go through the buildbot bridge The changes to not require require config time variables, like version number, build number etc. are because these are provided in releasetasks?
Flags: needinfo?(rail)
Reporter | ||
Comment 2•9 years ago
|
||
(In reply to Kim Moir [:kmoir] from comment #1) > A few questions > > So we are just going to replace the existing builders here > http://hg.mozilla.org/build/buildbotcustom/file/664d99eb01d6/process/release. > py#l1709 > with jobs that can go through the buildbot bridge At this stage we are not replacing the builder yet. You need to add a new builder with a different name (including the branch name in the builder name should solve the problem) to generateReleasePromotionBuilders(), see http://hg.mozilla.org/build/buildbotcustom/file/664d99eb01d6/process/release.py#l2079. > The changes to not require require config time variables, like version > number, build number etc. are because these are provided in releasetasks? Correct. These variables will be set in the TC task generated by releasetasks. The only way to pass some variables via BBB runtime is using properties, similar to https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/l10n.yml.tmpl#L28-L36 You may need to change the script to read those properties, similar to https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py#335 and later access them similar to https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py#336
Flags: needinfo?(rail)
Assignee | ||
Comment 3•9 years ago
|
||
Assignee | ||
Comment 4•9 years ago
|
||
Assignee | ||
Comment 5•9 years ago
|
||
Assignee | ||
Comment 6•9 years ago
|
||
This is a draft of the templates file, there is still some content to be added
Assignee | ||
Comment 7•9 years ago
|
||
Comment on attachment 8689047 [details] [diff] [review] bug1215204bbcustom.patch Rail, I forgot to mention that we discussed yesterday that the properties that are defined in release tasks could be removed but I tried that and then the checkconfig failed. So I left them in so they could be overwritten
Assignee | ||
Updated•9 years ago
|
Attachment #8689046 -
Flags: review?(rail)
Reporter | ||
Updated•9 years ago
|
Attachment #8689046 -
Flags: review?(rail) → review+
Assignee | ||
Comment 8•9 years ago
|
||
Comment on attachment 8689047 [details] [diff] [review] bug1215204bbcustom.patch Not sure that this has everything but I need to test it and it is difficult to do so without landing and testing
Attachment #8689047 -
Flags: feedback+
Assignee | ||
Updated•9 years ago
|
Attachment #8689047 -
Flags: feedback+ → feedback?(rail)
Assignee | ||
Updated•9 years ago
|
Attachment #8689050 -
Flags: feedback?(rail)
Assignee | ||
Comment 9•9 years ago
|
||
Attachment #8689058 -
Attachment is obsolete: true
Reporter | ||
Comment 10•9 years ago
|
||
Comment on attachment 8689047 [details] [diff] [review] bug1215204bbcustom.patch Review of attachment 8689047 [details] [diff] [review]: ----------------------------------------------------------------- In overall you are in the proper direction, just need to address 2 major things: 1) don't iterate over platforms, the script has to be run once. 2) clean up unused properties ::: process/release.py @@ +1853,5 @@ > }, > } > builders.append(l10n_builder) > > + for platform in config["release_platforms"]: I don't think you need to iterate over all platforms. The script should handle all of them in a single run. Actually, it won't add new platforms after the first run, because ti tries to be idempotent and checks if the product already exists and if it does, the script skips it. So there should be a single call of this script, not per platform. @@ +1857,5 @@ > + for platform in config["release_platforms"]: > + bouncer_mh_cfg = { > + "script_name": "scripts/bouncer_submitter.py", > + "extra_args": [ > + "-c", "releases/bouncer_%s_release.py" % pf['product_name'], this won't work for beta, for example. I'd move bouncer_submitter_config (https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/release-fennec-mozilla-release.py.template#l178) to branch config, like you did in https://hg.mozilla.org/build/buildbot-configs/rev/93d843ea9c08 @@ +1876,5 @@ > + ) > + > + bouncer_builder = { > + "name": bouncer_buildername, > + "slavenames": pf["slaves"], this would run windows submission on windows slave, but there is no reason to do that. Once you get rid of the "for platform ..." loop, you can use linux+linux64 slaves for this job. @@ +1877,5 @@ > + > + bouncer_builder = { > + "name": bouncer_buildername, > + "slavenames": pf["slaves"], > + 'category': bouncer_buildername, this should be builderPrefix(''), it's the same for all builder, so we can group them @@ +1883,5 @@ > + 'slavebuilddir': normalizeName(bouncer_buildername), > + 'factory': bouncer_submitter_factory, > + #'env': builder_env > + 'properties': { > + 'slavebuilddir': normalizeName(bouncer_buildername), I don't think you need "slavebuilddir" in properties. @@ +1884,5 @@ > + 'factory': bouncer_submitter_factory, > + #'env': builder_env > + 'properties': { > + 'slavebuilddir': normalizeName(bouncer_buildername), > + 'balrog_api_root': config['balrog_api_root'], I don't think this script interacts with balrog, does it? @@ +1886,5 @@ > + 'properties': { > + 'slavebuilddir': normalizeName(bouncer_buildername), > + 'balrog_api_root': config['balrog_api_root'], > + 'bouncerServer': config['bouncerServer'], > + 'tuxedoServerUrl': config['tuxedoServerUrl'], it's already passed in extra_args, no need to pass it again. @@ +1887,5 @@ > + 'slavebuilddir': normalizeName(bouncer_buildername), > + 'balrog_api_root': config['balrog_api_root'], > + 'bouncerServer': config['bouncerServer'], > + 'tuxedoServerUrl': config['tuxedoServerUrl'], > + 'bouncerServer': config['bouncerServer'], This is passed twice. I don't think you need this. @@ +1892,5 @@ > + #restructure it or use json.dumps (see l10n script) > + 'bouncer_aliases': { > + 'Firefox-%(version)s': 'firefox-latest', > + 'Firefox-%(version)s-stub': 'firefox-stub', > + }, This won't work, because properties have to be strings. I don't think it is used by the script, so you probably don't need this. @@ +1894,5 @@ > + 'Firefox-%(version)s': 'firefox-latest', > + 'Firefox-%(version)s-stub': 'firefox-stub', > + }, > + #this should be in releasetasks > + 'platform': platform, should be platform=None to make buildbot happy. @@ +1896,5 @@ > + }, > + #this should be in releasetasks > + 'platform': platform, > + 'product': pf['product_name'], > + 'branch': 'releases/date', shouldn't be hardcoded. @@ +1897,5 @@ > + #this should be in releasetasks > + 'platform': platform, > + 'product': pf['product_name'], > + 'branch': 'releases/date', > + 'bouncer_submitter_config': "releases/bouncer_%s_release.py" % pf['product_name'], this is already in extra_args.
Attachment #8689047 -
Flags: feedback?(rail) → feedback+
Reporter | ||
Comment 11•9 years ago
|
||
Comment on attachment 8689050 [details] [diff] [review] bug1215204mh.patch Review of attachment 8689050 [details] [diff] [review]: ----------------------------------------------------------------- You also need to override --previous-version with multiple values. The values should be passed from shipt-it via releasetasks. something similar to https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/funsize.yml.tmpl#L1 Something like ... task: ... payload: ... properties: ... - partial_versions: {% for partial_version, partial_info in partial_updates.iteritems() %}{{ partial_version }}{% endfor %} ... and then something like self.config["prev_versions"] = self.buildbot_config["properties"].get("partial_versions").split() ::: testing/mozharness/scripts/bouncer_submitter.py @@ +72,5 @@ > + #override properties from buildbot properties here as defined by taskcluster properties > + self.read_buildbot_config() > + > + #check if release promotion is true first before overwriting these properties > + #? do I need to define buildbot_json_path I don't think you need to define it. @@ +76,5 @@ > + #? do I need to define buildbot_json_path > + if self.buildbot_config["properties"].get("release_promotion"): > + for prop in ['product', 'version', 'build_number', 'revision', 'bouncer_submitter_config']: > + if self.buildbot_config["properties"].get(prop): > + self.info("Overriding en_us_binary_url with %s" % the log message is wrong.
Attachment #8689050 -
Flags: feedback?(rail) → feedback+
Assignee | ||
Comment 12•9 years ago
|
||
I couldn't get 'category': builderPrefix('') to be in scope and work
Attachment #8689047 -
Attachment is obsolete: true
Assignee | ||
Comment 13•9 years ago
|
||
Attachment #8689050 -
Attachment is obsolete: true
Assignee | ||
Comment 14•9 years ago
|
||
Assignee | ||
Comment 15•9 years ago
|
||
Attachment #8689210 -
Attachment is obsolete: true
Assignee | ||
Comment 16•9 years ago
|
||
Attachment #8689132 -
Attachment is obsolete: true
Assignee | ||
Updated•9 years ago
|
Attachment #8689549 -
Flags: review?(rail)
Assignee | ||
Comment 17•9 years ago
|
||
Comment on attachment 8689200 [details] [diff] [review] bug1215204bbcustom2.patch I couldn't get 'category': builderPrefix('') to be in scope and work, not sure how to do this part (re your last review comments)
Attachment #8689200 -
Flags: feedback?(rail)
Assignee | ||
Updated•9 years ago
|
Attachment #8689555 -
Flags: feedback?(rail)
Assignee | ||
Updated•9 years ago
|
Attachment #8689564 -
Flags: feedback?(rail)
Comment 18•9 years ago
|
||
In production: https://hg.mozilla.org/build/buildbot-configs/rev/93d843ea9c08
Reporter | ||
Comment 19•9 years ago
|
||
Comment on attachment 8689200 [details] [diff] [review] bug1215204bbcustom2.patch Review of attachment 8689200 [details] [diff] [review]: ----------------------------------------------------------------- Almost there! :) ::: process/release.py @@ +1875,5 @@ > + > + bouncer_builder = { > + "name": bouncer_buildername, > + "slavenames": config["platforms"]["linux"]["slaves"] + config["platforms"]["linux64"]["slaves"], > + 'category': name, category has to be builderPrefix('') @@ +1883,5 @@ > + #'env': builder_env, > + 'properties': { > + 'bouncerServer': config['bouncerServer'], > + 'tuxedoServerUrl': config['tuxedoServerUrl'], > + 'bouncerServer': config['bouncerServer'], I don't think you need these 3 (actually 2, one duplicate) properties. The only valuable one (tuxedoServerUrl) is already passed via extra_args as --bouncer-api-prefix @@ +1884,5 @@ > + 'properties': { > + 'bouncerServer': config['bouncerServer'], > + 'tuxedoServerUrl': config['tuxedoServerUrl'], > + 'bouncerServer': config['bouncerServer'], > + #these should be in releasetasks You can remove this comment, the following 3 properties are required by log_upload.py
Attachment #8689200 -
Flags: feedback?(rail) → feedback+
Reporter | ||
Comment 20•9 years ago
|
||
Comment on attachment 8689549 [details] [diff] [review] bug1215204bbconfigs2.patch Review of attachment 8689549 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla/project_branches.py @@ +197,3 @@ > 'bouncerServer': 'download.mozilla.org', > 'tuxedoServerUrl': 'https://bounceradmin.mozilla.com/api', > 'bouncerServer': 'download.mozilla.org', Can you remove bouncerServer entries when you land this? I don't think we use them yet.
Attachment #8689549 -
Flags: review?(rail) → review+
Reporter | ||
Comment 21•9 years ago
|
||
Comment on attachment 8689555 [details] [diff] [review] bug1215204mh3.patch Review of attachment 8689555 [details] [diff] [review]: ----------------------------------------------------------------- Feel free to land it on date to test. ::: testing/mozharness/scripts/bouncer_submitter.py @@ +78,5 @@ > + if self.buildbot_config["properties"].get(prop): > + self.info("Overriding %s with %s" % > + prop, self.buildbot_config["properties"][prop]) > + self.config[prop] = self.buildbot_config["properties"][prop] > + if self.buildbot_config["properties".get("partial_versions")]: should be if self.buildbot_config["properties"].get("partial_versions"):
Attachment #8689555 -
Flags: feedback?(rail) → feedback+
Reporter | ||
Comment 22•9 years ago
|
||
Comment on attachment 8689564 [details] [diff] [review] bug1215204templates2.patch Review of attachment 8689564 [details] [diff] [review]: ----------------------------------------------------------------- BTW, it would be great to add unit tests tests and you can use github PRs for these patches so you get tests running automatically on travis. ::: releasetasks/templates/bouncer.yml.tmpl @@ +1,1 @@ > +{% for platform, platform_info in bouncer_config["platforms"].iteritems() %} It doesn't match the buildername(s) in buildbot. Don't need to iterate over platforms. @@ +1,5 @@ > +{% for platform, platform_info in bouncer_config["platforms"].iteritems() %} > +# TODO: make a helper function to generate consistent builder names? > +{% set buildername = "release-{}_{}_{}__bncr_sub".format(branch, product, platform) %} > +- > + taskId: "{{ stableSlugId('{}'.format(buildername)) }}" taskId: "{{ stableSlugId(buildername) }}" looks simpler ;)
Attachment #8689564 -
Flags: feedback?(rail) → feedback+
Assignee | ||
Comment 23•9 years ago
|
||
fixed project branches patch
Assignee | ||
Updated•9 years ago
|
Attachment #8689717 -
Flags: checked-in+
Assignee | ||
Comment 24•9 years ago
|
||
Attachment #8689564 -
Attachment is obsolete: true
Assignee | ||
Comment 25•9 years ago
|
||
Attachment #8689555 -
Attachment is obsolete: true
Assignee | ||
Comment 26•9 years ago
|
||
In comment 19, rail suggested that category has to be builderPrefix(''). However, builderPrefix is not in scope for generateReleasePromotionBuilders since it is defined within generateReleaseBranchObjects. So I just used the bit that applied to generateReleasePromotionBuilders for the category
Attachment #8689200 -
Attachment is obsolete: true
Attachment #8691534 -
Flags: review?(jlund)
Assignee | ||
Updated•9 years ago
|
Attachment #8690093 -
Flags: review?(jlund)
Comment 27•9 years ago
|
||
Comment on attachment 8691534 [details] [diff] [review] bug1215204bbcustom3.patch Review of attachment 8691534 [details] [diff] [review]: ----------------------------------------------------------------- looks good to me. I didn't cross examine line by line against the 'old way' but I suspect we can iterate on this if we need to :) tiny nit: there are about 5 spots where bugzilla's splinter view picked up whitespaces. ::: process/release.py @@ +1854,5 @@ > builders.append(l10n_builder) > + > + bouncer_mh_cfg = { > + "script_name": "scripts/bouncer_submitter.py", > + "extra_args": [ so args like --build-num, --version, and --revision no longer required? @@ +1873,5 @@ > + ) > + > + bouncer_builder = { > + "name": bouncer_buildername, > + "slavenames": config["platforms"]["linux"]["slaves"] + config["platforms"]["linux64"]["slaves"], oh neat, I didn't realize we could use linux32 or 64 for this. @@ +1877,5 @@ > + "slavenames": config["platforms"]["linux"]["slaves"] + config["platforms"]["linux64"]["slaves"], > + "builddir": bouncer_buildername, > + "slavebuilddir": normalizeName(bouncer_buildername), > + "factory": bouncer_submitter_factory, > + "category": "release-%s-%s" % (config["bouncer_branch"], ''), category ends with '-'?
Attachment #8691534 -
Flags: review?(jlund) → review+
Comment 28•9 years ago
|
||
Comment on attachment 8690093 [details] [diff] [review] bug1215204templates3.patch Review of attachment 8690093 [details] [diff] [review]: ----------------------------------------------------------------- awesome. please bear with me as I get up to speed with you. setting to r- for now. as always, it can be persuaded to an r+ if I am wrong :) ::: releasetasks/templates/bouncer.yml.tmpl @@ +1,2 @@ > +# TODO: make a helper function to generate consistent builder names? > +{% set buildername = "release-{}_{}__bncr_sub".format(branch, product) %} there is two underscores btween product and bncr. iiuc, this has to match the buildername you defined in bbot-cfg's release.py @@ +23,5 @@ > + product: "{{ product }}" > + # Quotes cannot be used around this string because the loop causes it to have trailing whitespace > + # (which gets stripped by the yaml parser when unquoted). Kindof hacky. > + version: {{ version }} > + build_number: {{ buildNumber }} ah right, we moved build number, revision, partials, and version here because they are release specific. ignore my comment in other attachment @@ +26,5 @@ > + version: {{ version }} > + build_number: {{ buildNumber }} > + release_promotion: true > + revision: "{{ revision }}" > + partial_versions: {% for partial_version, partial_info in partial_updates.iteritems() %}{{ partial_version }}{% endfor %} I'm a jinja noob. It looks like the output of this will be a string of partials concatenated together without a delim. Am I reading that right? @@ +33,5 @@ > + description: "Release Promotion bouncer submission job" > + owner: "release@mozilla.com" > + source: https://github.com/mozilla/releasetasks > + > + {% if running_tests is defined %} do we have tests for this? @@ +39,5 @@ > + task_name: "{{ buildername }}" > + {% endif %} > + > +{% endfor %} > +{% endfor %} hrm, iiuc, these two endfors are not closing any loops
Attachment #8690093 -
Flags: review?(jlund) → review-
Assignee | ||
Comment 29•9 years ago
|
||
Re review comments from comment 28 I don't have tests yet, I have to add them I'm a jinja noob too. For this part partial_versions: {% for partial_version, partial_info in partial_updates.iteritems() %}{{ partial_version }}{ I just copied it from the funsize template
Attachment #8690093 -
Attachment is obsolete: true
Attachment #8692017 -
Flags: review?(jlund)
Comment 30•9 years ago
|
||
Comment on attachment 8692017 [details] [diff] [review] bug1215204templates4.patch Review of attachment 8692017 [details] [diff] [review]: ----------------------------------------------------------------- again, I'm learning as I go so please bear with me if I end up wrong here... ::: releasetasks/templates/bouncer.yml.tmpl @@ +1,3 @@ > +# TODO: make a helper function to generate consistent builder names? > +{% set buildername = "release-{}_{}_bncr_sub".format(branch, product) %} > +{% for partial_version, partial_info in partial_updates.iteritems() %} hm, so what are we trying to do here? this will create a task for each partial iiuc and each task would trigger the same buildbot builder with the same properties no? Don't we just want to create one bouncer submitter task? @@ +27,5 @@ > + version: {{ version }} > + build_number: {{ buildNumber }} > + release_promotion: true > + revision: "{{ revision }}" > + partial_versions: {% for partial_version, partial_info in partial_updates.iteritems() %}{{ partial_version }}{% endfor %} this is the same look as the one at the top. iiuc you are trying to account for: https://dxr.mozilla.org/build-central/source/buildbotcustom/process/release.py#1444 by passing the previous partial versions via buildbot properties. But what I still think we need to do is have separator between partial versions so we don't end up with something like: partial_versions: "44.0b144.0b341.b3' maybe we should try something like: code: https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/l10n.yml.tmpl#L33 output: https://queue.taskcluster.net/v1/task/Fyw6nQxMSNGQxSSfddXqAQ
Attachment #8692017 -
Flags: review?(jlund) → review-
Assignee | ||
Comment 31•9 years ago
|
||
Sorry for the many review requests. I find this hard to write because I can't test it first, which I usually do so I'm kind of flying blind. I added the formatting for the partial_versions
Attachment #8692017 -
Attachment is obsolete: true
Attachment #8692578 -
Flags: review?(jlund)
Comment 32•9 years ago
|
||
Comment on attachment 8692578 [details] [diff] [review] bug1215204templates5.patch Review of attachment 8692578 [details] [diff] [review]: ----------------------------------------------------------------- I hear yeah. This looks great. We can iterate on this once we see what it's like on date :)
Attachment #8692578 -
Flags: review?(jlund) → review+
Assignee | ||
Comment 33•8 years ago
|
||
Attachment #8691534 -
Attachment is obsolete: true
Assignee | ||
Updated•8 years ago
|
Attachment #8692997 -
Flags: checked-in+
Assignee | ||
Comment 34•8 years ago
|
||
Comment on attachment 8692997 [details] [diff] [review] bug1215204bbcustom3nowhitespace.patch in production
Assignee | ||
Comment 35•8 years ago
|
||
Have a pull request that passes https://github.com/mozilla/releasetasks/pull/56 Not sure if I should merge as it seems I can kmoir I issued a pull request for my patches for the bouncer submitter but it failed https://github.com/mozilla/releasetasks/pull/54/files kmoir https://tools.taskcluster.net/task-graph-inspector/#VYZ_aV80Suqlh6ERzQIQFg/HoaNq5VESVuxh1w64jfijA kmoir not sure how to debug what went wrong jlund kmoir: so I think everytime we call https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/release_graph.yml.tmpl jlund via https://github.com/mozilla/releasetasks/blob/77dc076c4c8868eb0ca159f3b5b0bba4d4d9cd81/releasetasks/__init__.py#L17 jlund in all our tests (and of course in releaserunner), we need to add a 'bouncer_enabled' arg jlund e.g. https://github.com/mozilla/releasetasks/blob/master/releasetasks/test/test_releasetasks.py#L62 kmoir jlund: thanks! jlund kmoir: can you see the log and the failures in the taskcluster task url you pasted above? jlund it's in the 'Run 0' tab
Assignee | ||
Comment 36•8 years ago
|
||
So I'm trying to run this in release runner and it is failing to create the task graph https://index.taskcluster.net/v1/task/buildbot.revisions.c5283bf0194859ba1f0a3e3b0c2d889456b625e4.date.linux Not sure why, going to ask rail about it next week at our all hands
Reporter | ||
Comment 37•8 years ago
|
||
Looks like you need to add "bouncer_enabled": branchConfig[something_that_enables_bouncer] to kwargs in https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l267 which is passed to make_task_graph() in https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l301
Reporter | ||
Comment 38•8 years ago
|
||
Kim, can you also add releasetasks unittests for this feature.
Assignee | ||
Comment 39•8 years ago
|
||
Attachment #8697357 -
Flags: review?(rail)
Reporter | ||
Comment 40•8 years ago
|
||
Comment on attachment 8697357 [details] [diff] [review] bug1215204tools.patch Hmm, I don't see this variable name anywhere in the configs. Maybe you forgot to add it to project_branches.py?
Attachment #8697357 -
Flags: review?(rail) → review-
Assignee | ||
Comment 41•8 years ago
|
||
Attachment #8697378 -
Flags: review?(rail)
Assignee | ||
Updated•8 years ago
|
Attachment #8697378 -
Flags: review?(rail)
Assignee | ||
Comment 42•8 years ago
|
||
Attachment #8697378 -
Attachment is obsolete: true
Assignee | ||
Updated•8 years ago
|
Attachment #8697382 -
Flags: review?(rail)
Reporter | ||
Comment 43•8 years ago
|
||
Comment on attachment 8697382 [details] [diff] [review] bug1215204date.patch conditional r+: please remove surrounding quotes, it should be True.
Attachment #8697382 -
Flags: review?(rail) → review+
Assignee | ||
Comment 44•8 years ago
|
||
Comment on attachment 8697382 [details] [diff] [review] bug1215204date.patch http://hg.mozilla.org/build/buildbot-configs/rev/9cbdc3b456f1 removed quotes
Attachment #8697382 -
Flags: checked-in+
Assignee | ||
Comment 45•8 years ago
|
||
Comment on attachment 8697357 [details] [diff] [review] bug1215204tools.patch asking for review again since I updated project_branches.py
Attachment #8697357 -
Flags: review- → review?
Reporter | ||
Updated•8 years ago
|
Attachment #8697357 -
Flags: review? → review+
Assignee | ||
Updated•8 years ago
|
Attachment #8697357 -
Flags: checked-in+
Assignee | ||
Comment 46•8 years ago
|
||
So releaserunner is failing like this Traceback (most recent call last): File "release-runner.py", line 302, in main graph = make_task_graph(**kwargs) File "/home/cltbld/releasetasks/releasetasks/__init__.py", line 50, in make_task_graph return yaml.safe_load(template.render(**template_vars)) File "/builds/releaserunner/lib/python2.7/site-packages/jinja2/environment.py", line 894, in render return self.environment.handle_exception(exc_info, True) File "/home/cltbld/releasetasks/releasetasks/templates/release_graph.yml.tmpl", line 69, in top-level template code {{ bouncer_tasks()|indent(4) }} File "/home/cltbld/releasetasks/releasetasks/templates/release_graph.yml.tmpl", line 68, in template {% macro bouncer_tasks() %}{% include "bouncer.yml.tmpl" %}{% endmacro %} File "/home/cltbld/releasetasks/releasetasks/templates/bouncer.yml.tmpl", line 30, in top-level template code partial_versions: {% for p in partial_version %}{{ "{}, ".format(p) }}{% endfor %} UndefinedError: 'partial_version' is undefined because partial_versions is not defined properly in the template I changed it to partial_versions: {% for p in partial_updates.iteritems() %}{{ "{}, ".format(p) }}{% endfor %} https://github.com/kmoir/releasetasks/commit/20e8472b63447a4c6051ecb2d273ae1875b44bde but the task graph is failing by travis, not sure if this is because there are previous commits from other people that are causing it to fail. If you have suggestions on how to fix this it would be appreciated
Flags: needinfo?(rail)
Assignee | ||
Comment 47•8 years ago
|
||
so problem is that product is undefined for task
Flags: needinfo?(rail)
Reporter | ||
Comment 48•8 years ago
|
||
I believe you need to pass more variables to the call, something like in https://github.com/rail/releasetasks/blob/master/releasetasks/test/test_releasetasks.py#L174
Assignee | ||
Comment 49•8 years ago
|
||
missing bouncer_enabled in process/release.py
Attachment #8700741 -
Flags: review?(rail)
Reporter | ||
Comment 50•8 years ago
|
||
Comment on attachment 8700741 [details] [diff] [review] bug1215204bouncerenabled.patch Review of attachment 8700741 [details] [diff] [review]: ----------------------------------------------------------------- ::: process/release.py @@ +1882,5 @@ > "properties": { > "platform": None, > "product": pf["product_name"], > + "branch": config["bouncer_branch"], > + "branch": config["bouncer_enabled"] Hmm, forgot to fix the first part? "branch" is set twice here. Probably should be "bouncer_enabled"?
Attachment #8700741 -
Flags: review?(rail) → review-
Assignee | ||
Comment 51•8 years ago
|
||
Attachment #8700741 -
Attachment is obsolete: true
Attachment #8701027 -
Flags: review?(rail)
Reporter | ||
Updated•8 years ago
|
Attachment #8701027 -
Flags: review?(rail) → review+
Assignee | ||
Updated•8 years ago
|
Attachment #8701027 -
Flags: checked-in+
Assignee | ||
Comment 52•8 years ago
|
||
current status job is being scheduled but has error kmoir> now the error is that it "Failed to infer task payload format" 4:53 PM <kmoir> https://tools.taskcluster.net/task-graph-inspector/#Q02OdIodR8CtxK9--UX4gQ/oY1Aif18RWmroandG-OFnQ/ 4:54 PM <kmoir> also noticing the scope is buildbot-bridge:builder-name:release-date_firefox_bncr_sub 4:54 PM <kmoir> not sure if I have to change something there
Reporter | ||
Comment 53•8 years ago
|
||
(In reply to Kim Moir [:kmoir] from comment #52) > current status > > job is being scheduled but has error > kmoir> now the error is that it "Failed to infer task payload format" > 4:53 PM <kmoir> > https://tools.taskcluster.net/task-graph-inspector/#Q02OdIodR8CtxK9--UX4gQ/ > oY1Aif18RWmroandG-OFnQ/ It looks like buildbot-bridge is broken now, this not the only task with exception. I see l10n repacks orange, even though they worked fine in the past. I tried to grep logs, but they are too brief: [root@buildbot-master82.bb.releng.scl3.mozilla.com bbb]# grep -C3 oY1Aif18RWmroandG-OFnQ *.log tclistener.log-2015-12-22 13:37:21,713 - bbb.servicebase - Fetching task -Qk6iqXcRPODENVayEKlqg tclistener.log-2015-12-22 13:37:21,720 - requests.packages.urllib3.connectionpool - Starting new HTTPS connection (1): queue.taskcluster.net tclistener.log-2015-12-22 13:37:22,253 - bbb.servicebase - Fetching task 4B71MPC4TBqjGyqWi5thNQ tclistener.log:2015-12-22 13:38:29,822 - bbb.services - Handling Taskcluster exception (canceled) for oY1Aif18RWmroandG-OFnQ tclistener.log:2015-12-22 13:38:29,822 - bbb.servicebase - Fetching task oY1Aif18RWmroandG-OFnQ tclistener.log-2015-12-22 13:38:29,836 - bbb.services - No task found in our database, nothing to do. > 4:54 PM <kmoir> also noticing the scope is > buildbot-bridge:builder-name:release-date_firefox_bncr_sub The scope LGTM, compared with other jobs.
Reporter | ||
Comment 54•8 years ago
|
||
It turns out that the job fails in buildbot, see http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-date_firefox_bncr_sub/builds/0 Traceback (most recent call last): File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/process/buildstep.py", line 728, in startStep d.addCallback(self._startStep_2) File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 260, in addCallback callbackKeywords=kw) File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 249, in addCallbacks self._runCallbacks() File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) --- <exception caught here> --- File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/process/buildstep.py", line 769, in _startStep_2 skip = self.start() File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/steps/shell.py", line 212, in start command = properties.render(self.command) File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/process/properties.py", line 108, in render return [ self.render(e) for e in value ] File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/process/properties.py", line 106, in render return value.render(self.pmap) File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/process/properties.py", line 194, in render s = self.fmtstring % pmap File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/process/properties.py", line 169, in __getitem__ rv = properties[key] File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/process/properties.py", line 50, in __getitem__ rv = self.properties[name][0] exceptions.KeyError: 'repo_path'
Assignee | ||
Comment 55•8 years ago
|
||
patch to add missing repo_path as shown in the error here http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-date_firefox_bncr_sub/builds/2/steps/download_and_extract_scripts_archive/logs/err.html also, add script_repo_revision and change the order of the properties to be the same as the l10n builder
Attachment #8703700 -
Flags: review?(rail)
Reporter | ||
Comment 56•8 years ago
|
||
Comment on attachment 8703700 [details] [diff] [review] bug1215204fixproperties.patch lgtm!
Attachment #8703700 -
Flags: review?(rail) → review+
Assignee | ||
Updated•8 years ago
|
Attachment #8703700 -
Flags: checked-in+
Assignee | ||
Comment 57•8 years ago
|
||
to fix http://buildbot-master91.bb.releng.usw2.mozilla.com:8001/builders/release-date_firefox_bncr_sub/builds/0/steps/run_script/logs/stdio
Attachment #8703787 -
Flags: review?(rail)
Assignee | ||
Comment 58•8 years ago
|
||
Attachment #8703787 -
Attachment is obsolete: true
Attachment #8703787 -
Flags: review?(rail)
Attachment #8703794 -
Flags: review?(rail)
Reporter | ||
Comment 59•8 years ago
|
||
Comment on attachment 8703794 [details] [diff] [review] bug1215204bouncer.patch I think we wanted to have a separate config for date to avoid collisions with production submission or (better) to use staging/dev bouncer like in https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/staging_release-fennec-mozilla-release.py.template#191
Attachment #8703794 -
Flags: review?(rail) → review-
Assignee | ||
Comment 60•8 years ago
|
||
Assignee | ||
Comment 61•8 years ago
|
||
so this is to add the releases/bouncer_firefox_date.py to date, (just copied the beta version) and the previous patch is submit to the bouncer staging server. Hope this is the correct way forward
Attachment #8704185 -
Flags: feedback?(rail)
Assignee | ||
Updated•8 years ago
|
Attachment #8704183 -
Flags: feedback?(rail)
Assignee | ||
Comment 62•8 years ago
|
||
Attachment #8703794 -
Attachment is obsolete: true
Attachment #8704183 -
Attachment is obsolete: true
Attachment #8704185 -
Attachment is obsolete: true
Attachment #8704183 -
Flags: feedback?(rail)
Attachment #8704185 -
Flags: feedback?(rail)
Attachment #8704198 -
Flags: review?(rail)
Reporter | ||
Updated•8 years ago
|
Attachment #8704198 -
Flags: review?(rail) → review+
Assignee | ||
Updated•8 years ago
|
Attachment #8704198 -
Flags: checked-in+
Assignee | ||
Comment 63•8 years ago
|
||
I landed this on date yesterday
Attachment #8704704 -
Flags: checked-in+
Assignee | ||
Comment 64•8 years ago
|
||
set buildbot_json_path because it's missing
Attachment #8704706 -
Flags: review?(rail)
Reporter | ||
Comment 66•8 years ago
|
||
Comment on attachment 8704707 [details] [diff] [review] bug1215204mhargs.patch a nit, I'd add a trailing coma after the last line, so the next patch looks cleaner
Attachment #8704707 -
Flags: review?(rail) → review+
Reporter | ||
Comment 67•8 years ago
|
||
Comment on attachment 8704706 [details] [diff] [review] bug1215204properties.patch 302 jlund I see buildbot_json_path is mostly used in configs, not in code.
Attachment #8704706 -
Flags: review?(rail) → review?(jlund)
Assignee | ||
Comment 68•8 years ago
|
||
Comment on attachment 8704707 [details] [diff] [review] bug1215204mhargs.patch landed and added trailing comma
Attachment #8704707 -
Flags: checked-in+
Comment 69•8 years ago
|
||
Comment on attachment 8704706 [details] [diff] [review] bug1215204properties.patch Review of attachment 8704706 [details] [diff] [review]: ----------------------------------------------------------------- this would absolutely work fine but I would ask to tweak it a bit: ::: testing/mozharness/scripts/bouncer_submitter.py @@ +69,5 @@ > def _pre_config_lock(self, rw_config): > super(BouncerSubmitter, self)._pre_config_lock(rw_config) > > #override properties from buildbot properties here as defined by taskcluster properties > + self.config['buildbot_json_path'] = "buildprops.json" _pre_config_lock is generally reserved for dynamic config changes that are not known pre run time. since this value buildbot_json_path is pretty standard and static, we should put it into one of the config files for bouncer e.g. https://dxr.mozilla.org/build-central/source/mozharness/configs/releases/bouncer_firefox_beta.py?case=true&from=bouncer_firefox_beta.py saying that, since we will likely end up with more than one config file and you may not wish to put this item in each, you could define this in this class's BaseScript.__init__ under `config`: e.g. like https://dxr.mozilla.org/build-central/source/mozharness/scripts/desktop_l10n.py#168 mh looks for config items in 3 places and merges them by order: from highest precedence.. 1) CLI options 2) 'config' attr within BaseScript init. 3) config files
Attachment #8704706 -
Flags: review?(jlund) → review-
Comment 70•8 years ago
|
||
In production: https://hg.mozilla.org/build/buildbotcustom/rev/93196910e3df
Comment 71•8 years ago
|
||
In production: https://hg.mozilla.org/build/buildbotcustom/rev/4cda855c1404
Assignee | ||
Comment 72•8 years ago
|
||
Attachment #8704707 -
Attachment is obsolete: true
Assignee | ||
Comment 73•8 years ago
|
||
this worked on when I reran the the job on a spot instance and modified the scripts locally cltbld@bld-linux64-spot-535.build.releng.usw2.mozilla.com rel-date_fx_bncr_sub-000000000]$ /tools/buildbot/bin/python scripts/scripts/bouncer_submitter.py -c releases/bouncer_firefox_beta.py --credentials-file oauth.txt --bouncer-api-prefix https://bounceradmin.allizom.org/api --repo projects/date 18:17:23 INFO - MultiFileLogger online at 20160107 18:17:23 in /builds/slave/rel-date_fx_bncr_sub-000000000 18:17:23 INFO - Using buildbot properties: 18:17:23 INFO - { 18:17:23 INFO - "project": "", 18:17:23 INFO - "product": "firefox", 18:17:23 INFO - "build_number": 2, 18:17:23 INFO - "taskId": "BczAgDHfQdueE5rocPgjnQ", 18:17:23 INFO - "repository": "", 18:17:23 INFO - "buildername": "release-date_firefox_bncr_sub", 18:17:23 INFO - "basedir": "/builds/slave/rel-date_fx_bncr_sub-000000000", 18:17:23 INFO - "buildnumber": 7, 18:17:23 INFO - "slavename": "bld-linux64-spot-535", 18:17:23 INFO - "version": "44.0b31", 18:17:23 INFO - "release_promotion": true, 18:17:23 INFO - "platform": null, 18:17:23 INFO - "branch": "releases/date", 18:17:23 INFO - "bouncer_enabled": true, 18:17:23 INFO - "script_repo_revision": "production", 18:17:23 INFO - "master": "http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/", 18:17:23 INFO - "revision": "53f6e3a60916b82de924c8878249930453af9312", 18:17:23 INFO - "partial_versions": "44.0b2,", 18:17:23 INFO - "repo_path": "projects/date" 18:17:23 INFO - } 18:17:23 INFO - Overriding product with firefox 18:17:23 INFO - Overriding version with 44.0b31 18:17:23 INFO - Overriding build_number with 2 18:17:23 INFO - Overriding revision with 53f6e3a60916b82de924c8878249930453af9312 18:17:23 INFO - Run as scripts/scripts/bouncer_submitter.py -c releases/bouncer_firefox_beta.py --credentials-file oauth.txt --bouncer-api-prefix https://bounceradmin.allizom.org/api --repo projects/date
Attachment #8704706 -
Attachment is obsolete: true
Assignee | ||
Updated•8 years ago
|
Attachment #8705451 -
Flags: review?(rail)
Assignee | ||
Updated•8 years ago
|
Attachment #8705453 -
Flags: review?(rail)
Assignee | ||
Comment 74•8 years ago
|
||
Attachment #8705451 -
Attachment is obsolete: true
Attachment #8705451 -
Flags: review?(rail)
Attachment #8705630 -
Flags: review?(rail)
Reporter | ||
Updated•8 years ago
|
Attachment #8705453 -
Flags: review?(rail) → review+
Reporter | ||
Updated•8 years ago
|
Attachment #8705630 -
Flags: review?(rail) → review+
Assignee | ||
Updated•8 years ago
|
Attachment #8705453 -
Flags: checked-in+
Assignee | ||
Updated•8 years ago
|
Attachment #8705630 -
Flags: checked-in+
Assignee | ||
Comment 75•8 years ago
|
||
So the job is being scheduled and running but failing on the buildbot side because it can't access https://bounceradmin.allizom.org <kmoir> my job from last night failed because it couldn't access the bounceradmin dev instance 10:02 AM <kmoir> http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-date_firefox_bncr_sub/builds/3/steps/run_script/logs/stdio 10:02 AM <kmoir> how do you access this instance to ensure it is up? 10:02 AM <kmoir> https://bounceradmin.allizom.org/api/product_show?product=Firefox-44.0b33-Complete 10:02 AM <kmoir> I can't access it via a browser either 10:03 AM <kmoir> I looked in the wiki and mana but couldn't find anything on it
Reporter | ||
Comment 76•8 years ago
|
||
Wooohoo, great progress! I think it "Should Just Work (TM)" :), especially if the command line is the same with existing one.
Assignee | ||
Comment 77•8 years ago
|
||
new staging url as per discussion this morning
Attachment #8706459 -
Flags: review?(bugspam.Callek)
Assignee | ||
Comment 78•8 years ago
|
||
I ran a build with the last patch manually and I got this error message so will fix this 09:53:58 INFO - <?xml version="1.0" encoding="utf-8"?><locations><product id="5160" name="Firefox-44.0b33-stub"><location id="26241" os="win64">/firefox/releases/44.0b33/win64/:lang/Firefox%20Setup%20Stub%2044.0b33.exe</location></product></locations> 09:53:58 FATAL - Uncaught exception: Traceback (most recent call last): 09:53:58 FATAL - File "/builds/slave/rel-date_fx_bncr_sub-000000000/scripts/mozharness/base/script.py", line 1721, in run 09:53:58 FATAL - self.run_action(action) 09:53:58 FATAL - File "/builds/slave/rel-date_fx_bncr_sub-000000000/scripts/mozharness/base/script.py", line 1663, in run_action 09:53:58 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 09:53:58 FATAL - File "/builds/slave/rel-date_fx_bncr_sub-000000000/scripts/mozharness/base/script.py", line 1604, in _possibly_run_method 09:53:58 FATAL - return getattr(self, method_name)() 09:53:58 FATAL - File "scripts/scripts/bouncer_submitter.py", line 158, in submit 09:53:58 FATAL - self.submit_partials() 09:53:58 FATAL - File "scripts/scripts/bouncer_submitter.py", line 170, in submit_partials 09:53:58 FATAL - prev_version, prev_build_number = prev_version.split("build") 09:53:58 FATAL - ValueError: need more than 1 value to unpack 09:53:58 FATAL - Running post_fatal callback... 09:53:58 FATAL - Exiting -1
Comment 79•8 years ago
|
||
Comment on attachment 8706459 [details] [diff] [review] bug1215204tuxedourl.patch Review of attachment 8706459 [details] [diff] [review]: ----------------------------------------------------------------- Would be nice to also touch the rest of the files as well... (but not required by this review): https://dxr.mozilla.org/build-central/search?q=bounceradmin.allizom&redirect=false&case=false
Attachment #8706459 -
Flags: review?(bugspam.Callek) → review+
Assignee | ||
Comment 80•8 years ago
|
||
updates to other pointers to staging bouncer server
Attachment #8706518 -
Flags: review?(bugspam.Callek)
Assignee | ||
Updated•8 years ago
|
Attachment #8706459 -
Flags: checked-in+
Updated•8 years ago
|
Attachment #8706518 -
Flags: review?(bugspam.Callek) → review+
Assignee | ||
Updated•8 years ago
|
Attachment #8706518 -
Flags: checked-in+
Assignee | ||
Comment 81•8 years ago
|
||
My bouncer job is failing now because http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/builders/release-date_firefox_bncr_sub/builds/10/steps/run_script/logs/stdio the versions specified in taskcluster for previous versions look like this "partial_versions": "44.0b2, 44.0b4, 44.0b6,", which gets split and added into prev_versions while the ones with the old jobs look like this 'prev_versions': ('43.0.1build1', '43.0.3build1', '43.0.2build1'), Yet the code wants to split them both like this File "scripts/scripts/bouncer_submitter.py", line 170, in submit_partials 06:25:21 FATAL - prev_version, prev_build_number = prev_version.split("build") Should I just add some code to bouncer_submitter.py so that they are split differently for release promotion builds or should the original value be different?
Flags: needinfo?(rail)
Reporter | ||
Comment 82•8 years ago
|
||
I'd probably change https://github.com/rail/releasetasks/blob/master/releasetasks/templates/bouncer.yml.tmpl#L30 to something like partial_versions: "{{ ",".join(sorted(partial_updates.keys()) }}" and adjust the test accordingly.
Flags: needinfo?(rail)
Assignee | ||
Comment 83•8 years ago
|
||
I have a pull request to address the previous comment https://github.com/mozilla/releasetasks/pulls However, I'm stuck on another issue in the templates for releasetasks we don't input the build number for partials for bouncer (However the partial builder is included in the l10n and funsize tasks as a point of reference) the existing code splits the partials based on "build" in http://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l72 and https://hg.mozilla.org/projects/date/file/tip/testing/mozharness/scripts/bouncer_submitter.py#l170 So I think the https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/bouncer.yml.tmpl should include previousBuildNumber: {{ partial_info["buildNumber"] }} appended to this values in here partial_versions: "{{ ", ".join(sorted(partial_updates.keys())) }}" so that when the job runs it has values like 'prev_versions': ('44.0b2build1', '44.0b4build1', '44.0b6build1'), instead of the current values in the release promotion jobs on date that look like this "44.0b2, 44.0b4, 44.0b6" is this on the right track?
Reporter | ||
Comment 84•8 years ago
|
||
Yeah, looks like we need to pass the build numbers as well. probably something like https://gist.github.com/rail/e182c4cdd38fab03c9fa may resolve the issue. Not sure how to kill the trailing coma though. Maybe the coma is not that important.
Comment 85•8 years ago
|
||
(In reply to Rail Aliiev [:rail] from comment #84) > Yeah, looks like we need to pass the build numbers as well. probably > something like https://gist.github.com/rail/e182c4cdd38fab03c9fa may resolve > the issue. Not sure how to kill the trailing coma though. Maybe the coma is > not that important. Just found this: http://jinja.pocoo.org/docs/dev/templates/#joiner via: http://stackoverflow.com/a/33940873
Assignee | ||
Comment 86•8 years ago
|
||
Pull request to fix issue with missing build info in bouncer data + remove last comma in list
Assignee | ||
Updated•8 years ago
|
Attachment #8707946 -
Flags: checked-in+
Assignee | ||
Comment 87•8 years ago
|
||
bouncer submitter worked when I updated the dev instance to the latest version of releasetasks and ran another release http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-date_firefox_bncr_sub/builds/6
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 88•8 years ago
|
||
mh patches from date to land on m-i
Reporter | ||
Comment 89•8 years ago
|
||
Comment on attachment 8708020 [details] [diff] [review] bug1215204mhmi.patch lgtm
Attachment #8708020 -
Flags: review+
Assignee | ||
Comment 91•8 years ago
|
||
Comment on attachment 8708020 [details] [diff] [review] bug1215204mhmi.patch https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=85224fb4d75b
Attachment #8708020 -
Flags: checked-in+
Comment 92•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/85224fb4d75b
Reporter | ||
Comment 93•8 years ago
|
||
synced to esr45: https://hg.mozilla.org/releases/mozilla-esr45/pushloghtml?changeset=c6c98bb9b707
Reporter | ||
Comment 94•8 years ago
|
||
one more: https://hg.mozilla.org/releases/mozilla-esr45/rev/d3b53881c3f4
You need to log in
before you can comment on or make changes to this bug.
Description
•