Brad Lassey [:blassey] (use needinfo?) Reporter
	Description • 10 years ago

As I understand it, we already have this for nightly and aurora. This has been motivated by Android's lack of support for 3 char locale codes and our desire to ship support for Maithili.

	bhearsum@mozilla.com (:bhearsum)
	Comment 1 • 10 years ago

This is a big project, I doubt this is do-able before we ship. We don't even have another Beta test it with at this point...

	Chris Cooper [:coop] (he/him)
	Comment 2 • 10 years ago

Ben: agreed, it's not without peril on our side.  

In the mobile mtg, I suggested testing on beta first and then going to release, given that we're already in the release process for Fx30 next week. Ben also suggested that beta 1 may not be the best time to turn this on, because there is sometimes random bustage during the first beta cycle after uplift.

I'm going to assume we'd use balrog for this rather than snippets, but call me out if I'm wrong.

	Lukas Blakk [:lsblakk] use ?needinfo
	Comment 3 • 10 years ago

As this requires more effort from releng than we have time for between now and next Tuesday's ship date - this isn't a viable FF30 target.  Not even clear that this work needs to be tracked to a particular release, and with Nightly builds now having locale picker built in - do we still need to do this for Beta/Release channels? What's the win there?

	Chris Cooper [:coop] (he/him)
	Comment 4 • 10 years ago

(In reply to Lukas Blakk [:lsblakk] from comment #3)
> As this requires more effort from releng than we have time for between now
> and next Tuesday's ship date - this isn't a viable FF30 target.  Not even
> clear that this work needs to be tracked to a particular release, and with
> Nightly builds now having locale picker built in - do we still need to do
> this for Beta/Release channels? What's the win there?

blassey: after some more digging, it turns out we're *not* already serving single locale updates on nightly and Aurora, although we thought we were. Sorry for the misinformation, but that makes this a bigger ask.

We discussed this in our weekly post-mortem today and echoed Lukas' questions: is there a use case for single locale updates that will extend beyond when the locale picker hits the Release channel in 13 weeks? Are there benefits to having this in place in the intervening time, enabling a new locale+community notwithstanding?

	Mark Finkle (:mfinkle) (use needinfo?)
	Comment 5 • 10 years ago

We have an indic locale (Maithili in bug 1019726) that won't work with the current Android multi-locale build (and the locale picker). So we were hoping to give the localizers a show of support by creating a single locale Maithili build that could be installed directly from a Web or FTP listing. That build would not be updateable using the Play Store, so we wanted to get Mozilla-based updates working.

	Brad Lassey [:blassey] (use needinfo?) Reporter
	Comment 6 • 10 years ago

(In reply to Lukas Blakk [:lsblakk] from comment #3)
> As this requires more effort from releng than we have time for between now
> and next Tuesday's ship date - this isn't a viable FF30 target.  Not even
> clear that this work needs to be tracked to a particular release, and with
> Nightly builds now having locale picker built in - do we still need to do
> this for Beta/Release channels? What's the win there?

More specifically, the Maithili community is planning on pointing users to single locale builds of Fx30 which will not have any update mechanism, hence the request to track for 30.

Beyond the Maithili community, non-play-store updates will allow us to address non-play-store devices like the Kindle Fires and phones in China.

	Lukas Blakk [:lsblakk] use ?needinfo
	Comment 7 • 10 years ago

Let's bring Jeff into the conversation here too so we can ensure that the locale(s) we would do this work for are progressing along the channels as per l10n community standards and process.  

A few more questions:

* We have bug 713777 for looking at trying the Amazon app store again (I am currently trying to chase down John O'Duinn for the Mozilla account credentials there in order to progress the investigation)
* Are there other 3 char locales we would need to offer this for in the near future?
* Is there no way to have this locale turned into a 2 char locale name and fit it into the multi-locale build?

Releng (aka Coop) - is there any way to provide an update path for the build (ie: URL) even if that URL is not live *yet* so that the builds on FTP can, at some point a little further down the road, get updates?

	Nick Thomas [:nthomas] (UTC+12)
	Comment 8 • 10 years ago

On behalf of coop - I took the 29.0.1 es-ES apk and it has the updater enabled, and queries for updates like this
 https://aus4.mozilla.org/update/4/Fennec/29.0.1/20140505221201/Android_arm-eabi-gcc3/es-ES/release/4.3/default/default/29.0.1/update.xml
so it's using a sensible locale. The multi and en-US apks query the same except s/es-ES/en-US/.

So if we had a Maithili build (not currently enabled in beta or release) then we should be able to serve an update to it in the future.

	Chris Cooper [:coop] (he/him)
	Comment 9 • 10 years ago

Thanks, Nick.

	Jeff Beatty [:gueroJeff]
	Comment 10 • 10 years ago

(In reply to Lukas Blakk [:lsblakk] from comment #7)
> Let's bring Jeff into the conversation here too so we can ensure that the
> locale(s) we would do this work for are progressing along the channels as
> per l10n community standards and process.  
> 
> A few more questions:
> 
> * We have bug 713777 for looking at trying the Amazon app store again (I am
> currently trying to chase down John O'Duinn for the Mozilla account
> credentials there in order to progress the investigation)
> * Are there other 3 char locales we would need to offer this for in the near
> future?

There are only two (hsb and dsb), but since bug fix to support ISO 639-2 codes will arrive in Android developer tools in the near future (see bug 1013226 for reference). I'm convinced the best approach is to accomodate these three locales (Maithili) and keep the gate down for other three char locales until the fix is landed.

> * Is there no way to have this locale turned into a 2 char locale name and
> fit it into the multi-locale build?
It's a short term solution that requires significant testing (especially with the inclusion of the locale picker feature) and won't scale once Google's fix lands and we have to fix the locale code remap. The timeframe is too short and the workload too high. I can forward the email discussion between rnewman, nalexander, and I on the topic, if you'd like.
> 
> Releng (aka Coop) - is there any way to provide an update path for the build
> (ie: URL) even if that URL is not live *yet* so that the builds on FTP can,
> at some point a little further down the road, get updates?

	Lukas Blakk [:lsblakk] use ?needinfo
	Comment 11 • 10 years ago

Sounds like, barring a dot release, the users of the Maithili locale could feasibly download their builds from FTP and we have up to 6 weeks to provide updates for them.

Coop/Nick - can you provide a time estimation for getting single locale updates (esp. if only for up to 3 locales?) on Beta/Release?

Jeff - is Maithili ready to be enabled in Beta/Release for FF30?  If it's not already enabled, perhaps we are better to wait one more cycle and have the update mechanics lined up first?

	Jeff Beatty [:gueroJeff]
	Comment 12 • 10 years ago

(In reply to Lukas Blakk [:lsblakk] from comment #11)
> Sounds like, barring a dot release, the users of the Maithili locale could
> feasibly download their builds from FTP and we have up to 6 weeks to provide
> updates for them.
> 
> Coop/Nick - can you provide a time estimation for getting single locale
> updates (esp. if only for up to 3 locales?) on Beta/Release?
> 
> Jeff - is Maithili ready to be enabled in Beta/Release for FF30?  If it's
> not already enabled, perhaps we are better to wait one more cycle and have
> the update mechanics lined up first?
No, Maithili is targeted for FF31 and is ready for FF31.

	Chris Cooper [:coop] (he/him)
	Comment 13 • 10 years ago

Kim is going to pick this up, but likely won't be able to start on it for 2 weeks (July). 

She will sync up with bhearsum before he goes on PTO to get the lay of the land.

	bhearsum@mozilla.com (:bhearsum)
	Comment 14 • 10 years ago

bug 1019919 is going to be one of the main parts of this, meant to link it up earlier.

	Sylvestre Ledru [:Sylvestre]
	Comment 15 • 10 years ago

Kim, Ben, Are you still targeting 31? FYI, we release 31 in 3 weeks.

	Kim Moir [:kmoir] ET Assignee
	Comment 16 • 10 years ago

I haven't had a chance to look at this yet but it is a Q3 goal for me. I plan to look at this bug next week, am dealing with capacity issues right now on our CI farm (bug 1034055)

	bhearsum@mozilla.com (:bhearsum)
	Comment 17 • 10 years ago

I'm just back from PTO. It sounds like Kim is still planning to work on this, and I have other priorities. I'll be supporting her from the sidelines and probably doing reviews, but I'm not the one actively driving this.

	Brad Lassey [:blassey] (use needinfo?) Reporter
	Comment 18 • 10 years ago

Kim, any update?

	Kim Moir [:kmoir] ET Assignee
	Comment 19 • 10 years ago

The past week and a half I have been working on bug 1034055 (move all 2.3 tests to AWS).  This morning I wrote a patch and tested bug 1030753 (enable more tests on Android debug). Now I am looking at this bug :-)

	Kim Moir [:kmoir] ET Assignee
	Comment 20 • 10 years ago

I have patches to enable single locale updates to balrog for m-c and m-a in bug 1019919.  However, I don't see the mai locale enabled anymore here

http://mxr.mozilla.org/mozilla-aurora/source/mobile/android/locales/maemo-locales
http://mxr.mozilla.org/mozilla-beta/source/mobile/android/locales/maemo-locales 

which I understand is how they are enabled.  My single locale builds thus don't include the mai locale to enable Maithili.  Is this plan to be enabled soon or should I open a bug?

	Richard Newman [:rnewman]
	Comment 21 • 10 years ago

`mai` cannot appear in maemo-locales: it breaks the build.

See Bug 1013226 Comment 19.


(If it no longer breaks the build -- I haven't tested since May -- then we have much less urgency for single-locale Maithili builds. But I suspect that breakage is still the case.)

	Mark Finkle (:mfinkle) (use needinfo?)
	Comment 22 • 10 years ago

When we get single locale builds working, I believe we'll be pulling the locale list from "all-locales" and not "maemo-locales", which is really used to drive the multi-locale build.

This is kinda orthogonal to your question about the "mai" locale though, which might still break builds.

	Kim Moir [:kmoir] ET Assignee
	Comment 23 • 10 years ago

Have made good progress on this last night, they only thing that is missing is that the way the balrog credentials are consumed on release channels is different so I have to fix that up. 

Also :bhearsum do I have to add an explicit rule to balrog for each android beta and release, I notice that they are there for firefox

	bhearsum@mozilla.com (:bhearsum)
	Comment 24 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #23)
> Have made good progress on this last night, they only thing that is missing
> is that the way the balrog credentials are consumed on release channels is
> different so I have to fix that up. 
> 
> Also :bhearsum do I have to add an explicit rule to balrog for each android
> beta and release, I notice that they are there for firefox

Yeah, you'll need rules for Fennec there. There's a few other wrinkles here, too:
1) We need the fileUrls to point at either Bouncer or the "releases" directory on FTP. Eg: http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/32.0b10/. There's no guarantees about how long candidates directories will live, so we can't point updates there. If we want to be consistent with Firefox, Bouncer entries are what we should use. There's also extra work there because we'll need to add bouncer entry automation for Fennec. I think using direct links to FTP is OK, at least for now.

We actually handle Balrog somewhat differently for Releases and Nightlies. For releases, we submit the individual file information as part of each build, but then we submit a bunch of generic information later on. For example, here's a release blob:
https://aus4-admin.mozilla.org/releases/Firefox-32.0-build1/data

You'll note that the fileUrls have moved to the top level of the blob, and have platform/locale substitutions in them. Those are submitted as part of the "updates" builder, which we don't have for Fennec.

The scripts that do these submissions are https://github.com/mozilla/build-tools/blob/master/scripts/updates/balrog-release-pusher.py and https://github.com/mozilla/build-tools/blob/master/scripts/updates/balrog-submitter.py. They would work for Fennec with a few tweaks, I think.

2) We should probably talk to RelMan and/or QA about whether or not we want test channel rules, too. For Firefox, we use test channels to make sure updates work before we push them live.

3) Because we'll be pointing at the "releases" directory, we need to push Fennec files there before we ship. Currently, we push after QA signs off. RelMan and QA should be made aware of this.

	bhearsum@mozilla.com (:bhearsum)
	Comment 25 • 10 years ago

(In reply to Ben Hearsum [:bhearsum] from comment #24)
> (In reply to Kim Moir [:kmoir] from comment #23)
> 2) We should probably talk to RelMan and/or QA about whether or not we want
> test channel rules, too. For Firefox, we use test channels to make sure
> updates work before we push them live.

We can forget about this point. Nick told me yesterday that there's no way change the channel of a Fennec build, so there's no way to run tests on a test channel before shipping.

	Kim Moir [:kmoir] ET Assignee
	Comment 27 • 10 years ago

WIP patches.  Not sure of the correct values for testChannelRuleIds and testChannels

	Kim Moir [:kmoir] ET Assignee
	Comment 28 • 10 years ago

this isn't 100% complete but I thought I'd get feedback so far
One thing that is that I couldn't figure out a way to use the existing credentials file in configs/balrog so that's why they are added to the configs explicitly.  The way the releases are constructed are very different from the regular builds

	Kim Moir [:kmoir] ET Assignee
	Comment 29 • 10 years ago

Comment on attachment 8487896 [details] [diff] [review]
bug1019724bbcustom.patch

I think this could go in at any time

	Kim Moir [:kmoir] ET Assignee
	Comment 30 • 10 years ago

Forgot link, this is the data in balrog for a staging beta release https://aus4-admin-dev.allizom.org/releases/Fennec-32.0b8-build1/data

	bhearsum@mozilla.com (:bhearsum)
	Comment 31 • 10 years ago

Comment on attachment 8487896 [details] [diff] [review]
bug1019724bbcustom.patch

Review of attachment 8487896 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, this should be able to land anytime.

	bhearsum@mozilla.com (:bhearsum)
	Comment 32 • 10 years ago

Comment on attachment 8487913 [details] [diff] [review]
bug1019724buildbotc.patch

Review of attachment 8487913 [details] [diff] [review]:
-----------------------------------------------------------------

It looks like these patches cover the data submission part of it. There's a few tweaks, but they look good overall.

I think you still need to take care of the bouncer part. Lucky for you, Rail is already taking care of the pushing to the releases directory automatically as part of https://bugzilla.mozilla.org/show_bug.cgi?id=543109!

::: mozilla/release-fennec-mozilla-beta.py
@@ +142,5 @@
>  releaseConfig['enablePartialMarsAtBuildTime'] = False
>  releaseConfig['autoGenerateChecksums'] = False
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']

This is about to change, so it would be best to use the new style here. We'll just be using a single test channel for beta going forward, "beta-cdntest".

@@ +143,5 @@
>  releaseConfig['autoGenerateChecksums'] = False
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']
> +releaseConfig['partialUpdates']      = {}

Does this need to be set? I don't think it does...

@@ +145,5 @@
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']
> +releaseConfig['partialUpdates']      = {}
> +releaseConfig['bouncerServer']       = 'download.mozilla.org'
> +releaseConfig['testChannelRuleIds']    = []
\ No newline at end of file

Ping me and I'll help you figure out the rule ids.

::: mozilla/release-fennec-mozilla-release.py
@@ +151,5 @@
>  releaseConfig['enablePartialMarsAtBuildTime'] = False
>  releaseConfig['autoGenerateChecksums'] = False
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']

You should change these channel names too. For releases, they are "release-localtest" and "release-cdntest". (https://bugzilla.mozilla.org/show_bug.cgi?id=986990 talks about this in more detail if you're curious.)

	bhearsum@mozilla.com (:bhearsum)
	Comment 33 • 10 years ago

Comment on attachment 8487940 [details] [diff] [review]
bug1019724mh.patch

Review of attachment 8487940 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/single_locale/release_mozilla-beta_android.py
@@ +24,5 @@
> +    "balrog_credentials_file": "oauth.txt",
> +    "balrog_api_root": "https://aus4-admin.mozilla.org",
> +    'balrog_usernames': {
> +            'Fennec': 'ffxbld',
> +     },

To use the existing configs you'll need to pass them along to mobile_l10n.py in https://github.com/mozilla/build-buildbotcustom/blob/master/process/release.py#L814. It should be very similar to what you did in https://github.com/mozilla/build-buildbotcustom/commit/adfc9ff7134b74551e88e37354b26b28d98a394e, except "branchConfig" is what will be holding mozharness_configs.

@@ +25,5 @@
> +    "balrog_api_root": "https://aus4-admin.mozilla.org",
> +    'balrog_usernames': {
> +            'Fennec': 'ffxbld',
> +     },
> +    "tools_repo": "https://hg.mozilla.org/build/tools",

tools_repo still belongs here though, I think.

::: scripts/mobile_l10n.py
@@ +569,5 @@
> +     
> +    def query_is_release(self):
> +        
> +        if str(self.buildbot_config['properties']['branch']).startswith("release"):
> +            return True

Hmm, does this work? I guess it may if branch is relative to the root of hg.mozilla.org....

This may also be true if we do nightly l10n jobs on this branch (I have no idea if we do). It might be better to have an explicit flag for this in the mozharness configs rather than make this assumption.

@@ +612,5 @@
> +            else:
> +                self.submit_balrog_updates(release_type="release")
> +                product = self.buildbot_config["properties"]["product"]
> +                cmd = [os.path.join(dirs['abs_tools_dir'], "scripts/updates/balrog-release-pusher.py")]
> +                self.chmod(cmd[0], 0764)

Why do you need to chmod here?

@@ +621,5 @@
> +                cmd.extend(["--credentials-file", self.config.get("balrog_credentials_file")])
> +                cmd.extend(["--username", self.config.get("balrog_usernames")[product]])
> +                env = {'PATH': os.environ.get('PATH')}
> +                status = self.run_command(cmd, dirs['base_work_dir'], env=env, success_codes=(0,255), 
> +                                 return_type="num_errors")

All of this balrog-release-pusher.py stuff should go in BalrogMixin instead of here. You can either explicitly call it in a separate function, or do it as part of submit_balrog_updates if query_is_release() is True - I don't have a preference either way.

@@ +622,5 @@
> +                cmd.extend(["--username", self.config.get("balrog_usernames")[product]])
> +                env = {'PATH': os.environ.get('PATH')}
> +                status = self.run_command(cmd, dirs['base_work_dir'], env=env, success_codes=(0,255), 
> +                                 return_type="num_errors")
> +                if status not in [0, 255]:

Is 255 really a valid exit code?

	Kim Moir [:kmoir] ET Assignee
	Comment 35 • 10 years ago

reworked to address the previous feedback 

I had to chmod the script balrog-release-pusher.py or it wouldn't run because it didn't have x permissions.

For invoking scripts/bouncer_submitter.py, I tried a bunch of different approaches that aren't working right now so that part is yet to come.  Should 
scripts/bouncer_submitter.py just be called in buildbotcustom like scripts/mobile_l10n.py is invoked.  Because calling bouncer_submitter.py from within mobile_l10n.py seems problematic.

	bhearsum@mozilla.com (:bhearsum)
	Comment 36 • 10 years ago

Comment on attachment 8494026 [details] [diff] [review]
bug1019724mh-2.patch

Review of attachment 8494026 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/balrog/production.py
@@ +4,5 @@
>          'b2g': 'b2gbld',
>          'firefox': 'ffxbld',
>          'thunderbird': 'tbirdbld',
>          'mobile': 'ffxbld',
> +        'Fennec': 'ffxbld',

I'm a bit surprised this is necessary, but it's fine.

::: configs/releases/bouncer_fennec_beta.py
@@ +2,5 @@
> +config = {
> +    "shipped-locales-url": "https://hg.mozilla.org/%(repo)s/raw-file/%(revision)s/mobile/android/locales/all-locales",    
> +    "products": {
> +        "installer": {
> +            "product-name": "Firefox-%(version)s",

Rail tells me that the bouncer submitter code doesn't have any checks to make sure a product already exists before trying to create it, so this will fail right now because Firefox uses the same product. You can either update the Bouncer submitter code to deal with it (https://github.com/mozilla/build-mozharness/blob/master/scripts/bouncer_submitter.py) or use Fennec in the product name. Either is fine IMO.

::: scripts/mobile_l10n.py
@@ +605,5 @@
> +            if self.query_is_nightly():  
> +                self.submit_balrog_updates()                
> +            else:
> +                self.submit_balrog_updates(release_type="release")
> +                self.submit_balrog_release_pusher(dirs)               

I'm not going to r- for it, but it would be better to only have the release pusher run once, after all repacks have completed. Even if you continue to run this here, you'll need a builder like the "updates" one we have for Desktop to announce the updates to the release-drivers list. That is likely going to have to happen in this block of release.py: https://github.com/mozilla/build-buildbotcustom/blob/master/process/release.py#L1150

The simplest thing to do might be to use makeDummyBuilder if the product is Fennec, but that will require some testing. I think the "update shipping" builder will work fine as is, but that probably needs testing too.

(Apologies for not catching this earlier...)

	Kim Moir [:kmoir] ET Assignee
	Comment 37 • 10 years ago

bhearsum:

I fixed the issues issues you commented on re my earlier patches. thanks :-)  Regarding, 
http://hg.mozilla.org/build/buildbotcustom/file/a12657766d1d/process/release.py#l1150
I note that 

This seems to apply to Linux only
pf = branchConfig['platforms']['linux']

and fennec doesn't currently have verifyConfigs enabled.  

For firefox they are like this
releaseConfig['verifyConfigs']       = {
    'linux':  'mozBeta-firefox-linux.cfg',
    'linux64':  'mozBeta-firefox-linux64.cfg',
    'macosx64': 'mozBeta-firefox-mac64.cfg',
    'win32':  'mozBeta-firefox-win32.cfg'
}
which refers to, for instance tools/release/updates/mozBeta-firefox-linux.cfg

Looking at the hg history for the above file, it looks like it gets updated automatically. So not sure if I should create the blank files for Fennec.

Is the sole purpose of this builder to announce changes to the release drivers?

I looked for an example of update_shipping on our existing masters but couldn't find a job that had run recently

	bhearsum@mozilla.com (:bhearsum)
	Comment 38 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #37)
> bhearsum:
> 
> I fixed the issues issues you commented on re my earlier patches. thanks :-)
> Regarding, 
> http://hg.mozilla.org/build/buildbotcustom/file/a12657766d1d/process/release.
> py#l1150
> I note that 
> 
> This seems to apply to Linux only
> pf = branchConfig['platforms']['linux']
> 
> and fennec doesn't currently have verifyConfigs enabled.  
> 
> For firefox they are like this
> releaseConfig['verifyConfigs']       = {
>     'linux':  'mozBeta-firefox-linux.cfg',
>     'linux64':  'mozBeta-firefox-linux64.cfg',
>     'macosx64': 'mozBeta-firefox-mac64.cfg',
>     'win32':  'mozBeta-firefox-win32.cfg'
> }
> which refers to, for instance tools/release/updates/mozBeta-firefox-linux.cfg
> 
> Looking at the hg history for the above file, it looks like it gets updated
> automatically. So not sure if I should create the blank files for Fennec.

update verify isn't something we can run for Fennec, so you definitely don't need update verify configs.

> Is the sole purpose of this builder to announce changes to the release
> drivers?

For Desktop, the updates builder is responsible for running balrog-release-pusher.py as well as creating old style aus2/3 snippets (which is why it's not appropriate to use as-is for this). When it completes, Buildbot sends mail to announce the availability of the updates. It's that part that you're still missing, and you have two options for adding:
1) Stop running balrog release pusher as part of the repacks and create a new "updates" builder for Fennec that runs it.
2) Create a dummy "updates" builder that runs for Fennec.

In either case, it should be triggered after all of the repacks complete. You may need to add the productName to the updates builder because Fennec and Firefox will probably try to use the same one. A lot of the scheduler and builder name changes will probably be similar to what Rail did for push to mirrors in bug 543109 (eg: https://bug543109.bugzilla.mozilla.org/attachment.cgi?id=8488341).

The e-mail announcement should "just work" provided the templates in https://github.com/mozilla/build-buildbot-configs/tree/master/mozilla/release_templates are updated to match any changes in builder names and/or new builders.

I hope this helps clear things up a bit, but I know the builder+scheduler organization is quite complicated, so feel free to ping me for any further clarification.

> I looked for an example of update_shipping on our existing masters but
> couldn't find a job that had run recently

It's a bit easier to find these via the candidates directory, eg: http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/33.0b8-candidates/build1/logs/release-mozilla-beta-update_shipping-bm85-build1-build7.txt.gz.

	Kim Moir [:kmoir] ET Assignee
	Comment 39 • 10 years ago

Had a meeting with Ben and Rail regarding this issue this morning, thanks for the info it was very helpful

These are the steps forward I'll take
Create a dummy "updates" builder that runs for Fennec. This is just send out the email that the updates are available.
Add this dummy builder to the scheduler
add product name to it 
builder needs to be downtream of post_deliverables scheduler
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l1799
updates_success should be renamed to updates_firefox_sucess tb fennec etc
or add symlinks in buildbot-configs_kmoir/mozilla/release_templates/updates_success
strip release branch stuff out of updates_success

Bouncer 
disable bouncer is currently in fennec release configs
delete that from configs
will submit to bouncer automagically
product 
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l182
bouncer configs
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l1592
config from mozharness
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l1593
update configs here to include ones for fennec beta and release
http://hg.mozilla.org/build/mozharness/file/670222dfeb47/configs/releases

rail's patches https://bug543109.bugzilla.mozilla.org/attachment.cgi?id=8488341 are an example of how to specify a product when it wasn't there before 
https://bug543109.bugzilla.mozilla.org/attachment.cgi?id=8488341

	Kim Moir [:kmoir] ET Assignee
	Comment 40 • 10 years ago

Created attachment 8500432 [details] [diff] [review] [diff] [review]
bug1019724dummy.patch

I added this patch my staging env and my dummy build was triggered after all the l10n repacks completed.  I thought the email would magically get sent but I can't see that it happened here

http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/release-mozilla-beta-android_repack_complete

So I'm not sure what I'm missing, help appreciated

	bhearsum@mozilla.com (:bhearsum)
	Comment 41 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #40)
> Created attachment 8500436 [details] [diff] [review]
> bug1019724dummy.patch
> 
> Created attachment 8500432 [details] [diff] [review]
> bug1019724dummy.patch
> 
> I added this patch my staging env and my dummy build was triggered after all
> the l10n repacks completed.  I thought the email would magically get sent
> but I can't see that it happened here
> 
> http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/release-mozilla-
> beta-android_repack_complete
> 
> So I'm not sure what I'm missing, help appreciated

I think we resolved this on IRC - Kim didn't realize she needed to add herself to EMAIL_RECIPIENTS to get mail.

	Kim Moir [:kmoir] ET Assignee
	Comment 44 • 10 years ago

buildbot-configs.  Not sure if the bouncer stuff is quite right, not sure how it works to update the db

	bhearsum@mozilla.com (:bhearsum)
	Comment 45 • 10 years ago

Comment on attachment 8501168 [details] [diff] [review]
bug1019724bbcustom-3.patch

Review of attachment 8501168 [details] [diff] [review]:
-----------------------------------------------------------------

This looks right to me...

	bhearsum@mozilla.com (:bhearsum)
	Comment 46 • 10 years ago

Comment on attachment 8501258 [details] [diff] [review]
bug1019724mh-3.patch

Review of attachment 8501258 [details] [diff] [review]:
-----------------------------------------------------------------

This looks sensible to me. Rail, can you give the Bouncer parts a second set of eyes?

	bhearsum@mozilla.com (:bhearsum)
	Comment 47 • 10 years ago

Comment on attachment 8501278 [details] [diff] [review]
bug1019724bbconfigs-3.patch

Review of attachment 8501278 [details] [diff] [review]:
-----------------------------------------------------------------

This is fine, but I think there's two things missing:
1) "releaseChannel", which also means you need to enable the "update_shipping" builder in release.py (this is the one that lets us actually make the release live when we're ready to release).
2) I think you need to adjust e-mail templates still, otherwise we'll get generic success mail rather than "Updates available on xxxxx".

	Rail Aliiev [:rail]
	Comment 48 • 10 years ago

Comment on attachment 8501258 [details] [diff] [review]
bug1019724mh-3.patch

Review of attachment 8501258 [details] [diff] [review]:
-----------------------------------------------------------------

In overall the direction is correct. The patch just needs some fixes.

::: configs/releases/bouncer_fennec_beta.py
@@ +1,2 @@
> +# lint_ignore=E501
> +config = {

You can use a single file for both beta and release, they are identical. It's the same for Thunderbird. Firefox has different platforms for beta and release, this is why we use different files for them.

@@ +1,3 @@
> +# lint_ignore=E501
> +config = {
> +    "shipped-locales-url": "https://hg.mozilla.org/%(repo)s/raw-file/%(revision)s/mobile/android/locales/all-locales",    

Hmm, do we use this file to generate the list of locales we ship? If one of the locales is missing on FTP bouncer may mark this product as not ready.

::: mozharness/mozilla/updates/balrog.py
@@ +47,5 @@
> +    def submit_balrog_release_pusher(self, dirs): 
> +        dirs = dirs
> +        product = self.buildbot_config["properties"]["product"]
> +        cmd = [os.path.join(dirs['abs_tools_dir'], "scripts/updates/balrog-release-pusher.py")]
> +        self.chmod(cmd[0], 0764)

I would not change the state of the file.

cmd = [self.query_exe("python"), os.path.join(..., "scripts/updates/balrog-release-pusher.py")]

is how it's used in mozharness usually.

@@ +50,5 @@
> +        cmd = [os.path.join(dirs['abs_tools_dir'], "scripts/updates/balrog-release-pusher.py")]
> +        self.chmod(cmd[0], 0764)
> +        cmd.extend(["--build-properties", os.path.join(dirs["base_work_dir"], "balrog_props.json")])
> +        cmd.extend(["--api-root", self.config.get("balrog_api_root")])
> +        cmd.extend(["--buildbot-configs", "https://hg.mozilla.org/users/kmoir_mozilla.com/buildbot-configs"])

I bet this is going to be read from configs in the future. :)

@@ +54,5 @@
> +        cmd.extend(["--buildbot-configs", "https://hg.mozilla.org/users/kmoir_mozilla.com/buildbot-configs"])
> +        cmd.extend(["--release-config", os.path.join(dirs['build_dir'], self.config.get("release_config_file"))])
> +        cmd.extend(["--credentials-file", os.path.join(dirs['base_work_dir'], self.config.get("balrog_credentials_file"))])
> +        cmd.extend(["--username", self.config.get("balrog_usernames")[product]])
> +        env = {'PATH': os.environ.get('PATH')}

Why you need to set PATH here? I think self.query_exe("python") should make the line above redundant.

@@ +60,5 @@
> +        self.info("Calling Balrog release pusher script")
> +        return_code = self.retry(self.run_command,
> +                                 args=(cmd,),
> +                                 kwargs={'cwd': dirs['abs_work_dir'],
> +                                         'env': env})

If you don't need setting PATH, probably you can remove env from kwargs.

@@ +65,5 @@
> +       
> +        if return_code not in [0]:
> +            self.return_code = 1
> +        else:
> +            self.return_code = 0
\ No newline at end of file

Can you add comments why you needed this magic?

::: scripts/mobile_l10n.py
@@ +568,5 @@
> +    def query_is_release(self):
> +        
> +        c = self.config
> +        if str(self.config['is_release']):
> +            return True

I'd just implement the method body as single liner:

    return bool(self.config.get("is_release"))

using self.config["is_release"] may make the script barf if you don't have this variable set in the config. Also "c" is assigned, but never used.

@@ +574,3 @@
>      def submit_to_balrog(self):
> +        
> +        dirs = self.query_abs_dirs()

You can move the line above somewhere below (closer to the place where it's used) so you don't waste time to call this method if the following "if" doesn't pass.

@@ +574,5 @@
>      def submit_to_balrog(self):
> +        
> +        dirs = self.query_abs_dirs()
> +        if not self.query_is_nightly() and not self.query_is_release():
> +            self.info("Not a nightly or release build, skipping balrog submission.")

Hmmm, "return" is missing here I believe.

@@ +608,5 @@
>              self.set_buildbot_property("isOSUpdate", False)
>              self.set_buildbot_property("buildid", self.query_buildid())
>  
> +            if self.query_is_nightly():  
> +                self.submit_balrog_updates()

Can you explicitly pass release_type="nighly" here, just to make it more readable?

@@ +613,5 @@
> +            else:
> +                self.submit_balrog_updates(release_type="release")
> +                self.submit_balrog_release_pusher(dirs)
> +        if not self.query_is_nightly():
> +            self.submit_balrog_release_pusher(dirs)

It looks like you call self.submit_balrog_release_pusher(dirs) N+1 times for release jobs, where N = number of locales. N times in the "else" block and once here. And it looks like you call it twice for the last locale. It'd be great either to fix the logic or add comments here.

	Kim Moir [:kmoir] ET Assignee
	Comment 49 • 10 years ago

I have new patches which I've tested and are green.  However, I want to clean up the balrog db for my staging env to make sure it works and there's the bug where you can't delete them from the ui.

How do we remove a release from the balrog dev instance. I tried to do it as mentioned here but it didn't work for me https://bugzilla.mozilla.org/show_bug.cgi?id=1049550#c6
Do we have to open a bug with some IT or db or do we have access to the databases ourselves?

	bhearsum@mozilla.com (:bhearsum)
	Comment 50 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #49)
> I have new patches which I've tested and are green.  However, I want to
> clean up the balrog db for my staging env to make sure it works and there's
> the bug where you can't delete them from the ui.
> 
> How do we remove a release from the balrog dev instance. I tried to do it as
> mentioned here but it didn't work for me
> https://bugzilla.mozilla.org/show_bug.cgi?id=1049550#c6
> Do we have to open a bug with some IT or db or do we have access to the
> databases ourselves?

The best right now is to have me do it, unfortunately :(. Kim and coordinated on IRC and I deleted Firefox-32.0b8-build1 from the dev db.

	Kim Moir [:kmoir] ET Assignee
	Comment 51 • 10 years ago

Thanks Ben, rerunning builds now.

	Kim Moir [:kmoir] ET Assignee
	Comment 52 • 10 years ago

Latest data is here

https://aus4-admin-dev.allizom.org/releases/Fennec-32.0b8-build1/data

Looks okay but not sure if 
"completes": {
      "*": "fennec-32.0b8.complete.mar"
    }
should be there?

	bhearsum@mozilla.com (:bhearsum)
	Comment 53 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #52)
> Latest data is here
> 
> https://aus4-admin-dev.allizom.org/releases/Fennec-32.0b8-build1/data
> 
> Looks okay but not sure if 
> "completes": {
>       "*": "fennec-32.0b8.complete.mar"
>     }
> should be there?

That looks fine. The difference is that we created a new blob schema time mid way through your work on this bug, I think, and this is one of the changes. URLs like https://aus4-dev.allizom.org/update/4/Fennec/27.0/20140123190227/Android_arm-eabi-gcc3/eu/beta-cdntest/4.2.2/default/default/27.0/update.xml are returning updates, so you certainly don't have malformed data or anything.

	Kim Moir [:kmoir] ET Assignee
	Comment 54 • 10 years ago

new patch with feedback incorporated. Ran green in staging.

	Rail Aliiev [:rail]
	Comment 55 • 10 years ago

Comment on attachment 8507874 [details] [diff] [review]
bug1019724mh-4.patch

Review of attachment 8507874 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

::: mozharness/mozilla/updates/balrog.py
@@ +44,5 @@
> +        
> +        return return_code
> +        
> +    def submit_balrog_release_pusher(self, dirs): 
> +        dirs = dirs

the line above is redundant

	Rail Aliiev [:rail]
	Comment 57 • 10 years ago

Comment on attachment 8507898 [details] [diff] [review]
bug1019724mh-5.patch

Ship it!

	Rail Aliiev [:rail]
	Comment 58 • 10 years ago

Comment on attachment 8501168 [details] [diff] [review]
bug1019724bbcustom-3.patch

Review of attachment 8501168 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +576,5 @@
>              ))
>              xr_deliverables_builders.append(builderPrefix('xulrunner_source'))
> +    if releaseConfig['productName'] == "fennec":
> +        builders.append(makeDummyBuilder(
> +            name=builderPrefix('%s_updates' % releaseConfig['productName']),

feel free to use "fennec_updates" here (without interpolation) -- easier to grep ;)

@@ +1820,5 @@
> +                '%s_updates_ready' % releaseConfig['productName']),
> +            branch=sourceRepoInfo['path'],
> +            upstreamBuilders=deliverables_builders,
> +            builderNames=post_deliverables_builders,
> +        ))

You are adding a second aggregating scheduler with the same upstream and downstream builders what means that the downstream builders may be triggered twice. The only difference with the previous scheduler is the name. What's the purpose of this scheduler?

	Rail Aliiev [:rail]
	Comment 59 • 10 years ago

Comment on attachment 8501278 [details] [diff] [review]
bug1019724bbconfigs-3.patch

Review of attachment 8501278 [details] [diff] [review]:
-----------------------------------------------------------------

A nit

::: mozilla/release-fennec-mozilla-beta.py
@@ +145,5 @@
>  releaseConfig['ftpSymlinkName'] = 'latest-beta'
> +releaseConfig['enableAutomaticPushToMirrors'] = True
> +# Tuxedo/Bouncer configuration
> +releaseConfig['tuxedoServerUrl']     = 'https://bounceradmin.mozilla.com/api'
> +releaseConfig['bouncer_submitter_config'] = 'releases/bouncer_fennec_beta.py'

I believe this should be releases/bouncer_fennec.py per attachment 8507898 [details] [diff] [review]. The same applies to other hunks as well.

Conditional r+ if you run something like
sed 's,releases/bouncer_fennec_\(beta\|release\).py,releases/bouncer_fennec.py,g' on those files

	Kim Moir [:kmoir] ET Assignee
	Comment 60 • 10 years ago

patch to address feedback in comment #59

	Kim Moir [:kmoir] ET Assignee
	Comment 61 • 10 years ago

regarding comment #58, the purpose of this builder is to send an email that updates are available for fennec

release-mozilla-beta-fennec_updates and release-mozilla-release-fennec_updates builders are added for this purpose

Is there better way to do this?

	Rail Aliiev [:rail]
	Comment 62 • 10 years ago

Using a dummy builder is the easiest way to email. I still wonder why you need 2 schedulers (not builders) with the same up/down stream builders.

	Kim Moir [:kmoir] ET Assignee
	Comment 63 • 10 years ago

Right that's what I did, it adds a dummy builder like this

http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/release-mozilla-beta-fennec_updates/builds/0

	Kim Moir [:kmoir] ET Assignee
	Comment 64 • 10 years ago

Rail:

So this is the diff of the builders added, is this what you would expect?

(build1)[kmoir@dev-master1.srv.releng.scl3.mozilla.com build1]$ diff old new
22a23
> release-mozilla-release-fennec_updates DummyFactory
36a38
> release-mozilla-beta-fennec_updates DummyFactory

	Rail Aliiev [:rail]
	Comment 65 • 10 years ago

Yup, the diff looks OK to me. My issue is with the schedulers. :)

	Kim Moir [:kmoir] ET Assignee
	Comment 66 • 10 years ago

I'm testing this patch now.  To ensure that the correct jobs are being triggered, I had to fix my staging environment last night/this morning to run android builds since it was in a weird state but this seems to be working now. (Before I was just retriggering l10n repacks) Once this is complete I'll ask for review on this final patch.

	Kim Moir [:kmoir] ET Assignee
	Comment 67 • 10 years ago

This patch worked in staging and triggered the correct builds and sent emails.  

One thing I didn't do is change all the updates builders to include the product name.  Not sure if this should be included.

	Rail Aliiev [:rail]
	Comment 68 • 10 years ago

Comment on attachment 8509973 [details] [diff] [review]
bug1019724bbcustom-5.patch

Review of attachment 8509973 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry, have to r- this :(

::: process/release.py
@@ +1266,2 @@
>          builders.append(makeDummyBuilder(
> +            name=builderPrefix('updates_%s' % releaseConfig['productName']),

This won't work:
1) it will break the scheduler graph for non android builds when you set skip_updates to True
2) This dummy builder won't be triggered by the corresponding aggregating scheduler because it's not added to post_signing_builders list. This builder is referenced by name everywhere in the file :/

I'd changed "updates" everywhere in the release.py to include productName.

	Kim Moir [:kmoir] ET Assignee
	Comment 69 • 10 years ago

running tests with this patch now

	Kim Moir [:kmoir] ET Assignee
	Comment 70 • 10 years ago

Comment on attachment 8510303 [details] [diff] [review]
bug1019724bbcustom-6.patch

This worked in staging

	Rail Aliiev [:rail]
	Comment 71 • 10 years ago

Comment on attachment 8510303 [details] [diff] [review]
bug1019724bbcustom-6.patch

Woot!

When you land the patch, can you also incorporate this change to keep the directory names in sync with the builder names?

https://gist.github.com/rail/41a7261a3376d168ed4f

	Kim Moir [:kmoir] ET Assignee
	Comment 72 • 10 years ago

new patch with rail's other suggestions

	Rail Aliiev [:rail]
	Comment 73 • 10 years ago

Comment on attachment 8510756 [details] [diff] [review]
bug1019724-6.patch

just in case :)

	Kim Moir [:kmoir] ET Assignee
	Comment 74 • 10 years ago

I'll land these changes on Monday, FF33.0.1 build2 is running right now.

	Kim Moir [:kmoir] ET Assignee
	Comment 75 • 10 years ago

And we still have releases in flight today, this is why I haven't landed the changes.

	Kim Moir [:kmoir] ET Assignee
	Comment 76 • 10 years ago

patch to fix failing tests due to duplicate builder names etc

ran test-masters.sh in staging and it was all green

	Rail Aliiev [:rail]
	Comment 78 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #76)
> Created attachment 8512774 [details] [diff] [review]
> bug1019724bbcustom-8.patch
> 
> patch to fix failing tests due to duplicate builder names etc
> 
> ran test-masters.sh in staging and it was all green

For this to work as expected you also need to add new symlinks in buildbot-configs/mozilla/release_templaets

	Kim Moir [:kmoir] ET Assignee
	Comment 79 • 10 years ago

more template files

	Rail Aliiev [:rail]
	Comment 80 • 10 years ago

Comment on attachment 8512850 [details] [diff] [review]
bug1019724-morebbtemplates.patch

Can you also add thunderbird?

	Kim Moir [:kmoir] ET Assignee
	Comment 81 • 10 years ago

added tb

	Chris Cooper [:coop] (he/him)
	Comment 82 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/da3ac7ace7e3

	Chris Cooper [:coop] (he/him)
	Comment 83 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/eab4127639e5

	Chris Cooper [:coop] (he/him)
	Comment 84 • 10 years ago

In production: https://hg.mozilla.org/build/buildbotcustom/rev/0151fbedc670

	Chris Cooper [:coop] (he/him)
	Comment 85 • 10 years ago

In production: https://hg.mozilla.org/build/buildbotcustom/rev/a2841db14eaf

	Chris Cooper [:coop] (he/him)
	Comment 86 • 10 years ago

In production: https://hg.mozilla.org/build/mozharness/rev/282d655f1ab2

	bhearsum@mozilla.com (:bhearsum)
	Comment 87 • 10 years ago

Kim, I noticed we got a plain "[release] Firefox 33.1 build1: completed updates_firefox" instead of "Updates available on the betatest channel" mail for 33.1. This patch should fix that...

	Kim Moir [:kmoir] ET Assignee
	Comment 88 • 10 years ago

Comment on attachment 8514220 [details] [diff] [review]
add templates for fennec/firefox updates builders

thanks Ben!

	Chris Cooper [:coop] (he/him)
	Comment 89 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/d175e19ca8b7

	bhearsum@mozilla.com (:bhearsum)
	Comment 90 • 10 years ago

There's a few more templates that need fixing, I think. We got mail like:
31275     Oct 31 release@mozilla (0.1K) ├─>[release] Firefox 34.0b5 build2: completed ready_for_releasetest_testing_firefox
31277     Oct 31 release@mozilla (0.1K) └─>[release] Firefox 34.0b5 build2: completed ready_for_release_firefox

Which should have special subjects:
https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/ready_for_release_success
https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/ready_for_releasetest_testing_success

	Nick Thomas [:nthomas] (UTC+12)
	Comment 91 • 10 years ago

In Fennec 34.0b6 we got an error trying to submit to bouncer:

scripts/scripts/bouncer_submitter.py -c releases/bouncer_fennec_beta.py --revision FENNEC_34_0b6_RELEASE --repo releases/mozilla-beta --version 34.0b6 --credentials-file oauth.txt --bouncer-api-prefix https://bounceradmin.mozilla.com/api
 in dir /builds/slave/rel-m-beta-bncr_sub-0000000000/. (timeout 1200 secs)
...
IOError: Can't find releases/bouncer_fennec_beta.py in ['.', '/builds/slave/rel-m-beta-bncr_sub-0000000000/scripts/scripts/../configs', '/builds/slave/rel-m-beta-bncr_sub-0000000000/scripts/scripts/../../configs']!

Full log: http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/34.0b6-candidates/build1/logs/release-mozilla-beta-bouncer_submitter_fennec-bm73-build1-build0.txt.gz

Looks like we just have a mozharness/configs/release/bouncer_fennec.py, with no branch suffix.

	Kim Moir [:kmoir] ET Assignee
	Comment 92 • 10 years ago

patch to address comment #91

	Nick Thomas [:nthomas] (UTC+12)
	Comment 93 • 10 years ago

Comment on attachment 8516375 [details] [diff] [review]
bug1019724bbconfigs-5.patch

Looks good. Would you me to rerun the bouncer submitter for 34.06 with this ?

	Nick Thomas [:nthomas] (UTC+12)
	Comment 94 • 10 years ago

Oops, I meant 34.0b6. The next fennec release will be this time next week, 34.0b8.

	Kim Moir [:kmoir] ET Assignee
	Comment 95 • 10 years ago

nthomas: you can rerun the bouncer submitter if it's not too much trouble.  Sorry for the breakage :-)

	Nick Thomas [:nthomas] (UTC+12)
	Comment 96 • 10 years ago

Comment on attachment 8516375 [details] [diff] [review]
bug1019724bbconfigs-5.patch

Transplanted to production and moved release tags:
http://hg.mozilla.org/build/buildbot-configs/rev/3437ada679c9

	Nick Thomas [:nthomas] (UTC+12)
	Comment 97 • 10 years ago

I was watching the log as bouncer_submitter_fennec ran, and needed to add an 'android-x86' OS to bouncer. Then it was happy.

There's a problem with the path's though:
 Product         OS              Path
 Fennec-34.0b6   android-x86     /mobile/releases/34.0b6/android-x86/:lang/fennec-34.0b6.android-x86.apk
 Fennec-34.0b6   android         /mobile/releases/34.0b6/android/:lang/fennec-34.0b6.android-arm.apk

There should be another :lang in the filenames, eg fennec-34.0b6.en-US.android-arm.apk. I've tried fixing that up by hand this time, but sentry has issues with it:

[2014-11-03 19:32:56 -0800] HEAD http://72.21.81.253/pub/mobile/releases/34.0b6/android/en-US/fennec-34.0b6.:lang.android-arm.apk ... FAILED. rc=404 CACHE=:miss: TOOK=0.053443
[2014-11-03 19:32:56 -0800] HEAD http://72.21.81.253/pub/mobile/releases/34.0b6/android-x86/en-US/fennec-34.0b6.:lang.android-x86.apk ... FAILED. rc=404 CACHE=:miss: TOOK=0.054189
Finished. Elapsed time: 54.

Only one :lang replacement is being done at https://github.com/mozilla/tuxedo/blob/master/sentry/sentry.pl#L271 because it's not ...@g. We could fix that, and I think serving files would be ok (it uses php's str_replace which replaces all instances, https://github.com/mozilla/tuxedo/blob/master/bouncer/php/index.php#L160).  You might catch the push to prod from bug 1089729 if you're fast.

Alternatively we could switch to pretty naming (MOZ_PKG_PRETTYNAMES) if that exists for Fennec.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 98 • 10 years ago

Found another glitch - the uptake checker is looking for these:
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6&os=android
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6&os=android-x86
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6-Complete&os=android
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6-Complete&os=android-x86

The last two must be accidental, but the server returns a '400 BAD REQUEST', which shows up in the exception logs we get emailed by the master.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 99 • 10 years ago

Sorry, found something else. Wondering why we have new builder names like 
  release-mozilla-beta-bouncer_submitter_firefox
  release-mozilla-beta-updates_firefox
instead of
  release-mozilla-beta-firefox_bouncer_submitter
  release-mozilla-beta-firefox_updates
to fit with the existing style
  release-mozilla-beta-fennec_tag

Might be related to earlier issues with release emails.

	Kim Moir [:kmoir] ET Assignee
	Comment 100 • 10 years ago

patch for comment #97

	Kim Moir [:kmoir] ET Assignee
	Comment 101 • 10 years ago

another patch for comment #97

	Kim Moir [:kmoir] ET Assignee
	Comment 102 • 10 years ago

rail: What's the standard for including productName in the builder name?  If you look at release.py, some of the builders include it at the start of their name, some in the middle, and I added ones that include the product name in the end.  As Nick (comment#99) and Ben (comment #90) pointed out, this causing problems with the templates.

	Kim Moir [:kmoir] ET Assignee
	Comment 103 • 10 years ago

:nthomas regarding comment #98, so the complete mar entry should be removed from here https://aus4-admin.mozilla.org/releases/Fennec-34.0b6-build1/data Should it be replaced by something else?

	Rail Aliiev [:rail]
	Comment 104 • 10 years ago

I don't think we have some formal standards here, just common sense and consistency. We should fix the latter. :)

	Kim Moir [:kmoir] ET Assignee
	Comment 105 • 10 years ago

makes builder names more consistent, fixes templates

	Kim Moir [:kmoir] ET Assignee
	Comment 106 • 10 years ago

makes builder names more consistent, fixes templates

	Kim Moir [:kmoir] ET Assignee
	Comment 107 • 10 years ago

builderdiff from release.py changes

	Nick Thomas [:nthomas] (UTC+12)
	Comment 108 • 10 years ago

Comment on attachment 8516722 [details] [diff] [review]
bug1019724sentry.patch

Sorry, this will have to go to a bug in Webtools :: Bouncer, and rhelmer can review. Looks fine to me though.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 109 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #103)
> :nthomas regarding comment #98, so the complete mar entry should be removed
> from here https://aus4-admin.mozilla.org/releases/Fennec-34.0b6-build1/data
> Should it be replaced by something else?

It's something else. The bouncer check is this code
 http://mxr.mozilla.org/build/source/buildbotcustom/scheduler.py#218
which ends up at
 http://mxr.mozilla.org/build/source/tools/lib/python/util/tuxedo.py#66
checkMars defaults to True, hence the check for Fennec-34.0b6-Complete. It can be forced to False if we set skip_updates=True in the release config, which wouldn't have any other effects for mobile.

There's a few things that need to be consistent though:
* the fileUrls in balrog. Currently Fennec-34.0b6-Complete, which makes a kind of sense over Fennec-34.0b6 (we're serving updates, not installers). [1]
* the products in bouncer submitter, as interpolated from the mozharness config. Currently Fennec-version
* the products checked in TriggerBouncerCheck. I wonder if we should teach this to read the mozharness config instead (we also have bug 1082823 to move it from master to slave)

So there's a wee tangle to sort out there.

[1] Incidentally, the beta-localtest url of 
 http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/34.0b6-candidates/build1/update/%OS_FTP%/%LOCALE%/fennec-34.0b6.complete.mar
needs to lose the update/ part. And we only have en-US for android-x86, perhaps we should be turning on all locales, or not doing anything for it on this bug.

	Release Engineering SlaveAPI Service
	Comment 110 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/4fe5cfc3981b

	Release Engineering SlaveAPI Service
	Comment 111 • 10 years ago

In production: https://hg.mozilla.org/build/buildbotcustom/rev/cbc087de6234

	Release Engineering SlaveAPI Service
	Comment 112 • 10 years ago

In production: https://hg.mozilla.org/build/mozharness/rev/4e0acd0c72be

	Kim Moir [:kmoir] ET Assignee
	Comment 113 • 10 years ago

Okay after reading through a lot of balrog and friends code this morning I have a few questions regarding comment #109

So skip_updates=True should be set in the release configs for beta and release for Fennec, correct?

Regarding this comment
>>>There's a few things that need to be consistent though:
* the fileUrls in balrog. Currently Fennec-34.0b6-Complete, which makes a kind of sense over Fennec-34.0b6 (we're serving updates, not installers). [1]
* the products in bouncer submitter, as interpolated from the mozharness config. Currently Fennec-version
* the products checked in TriggerBouncerCheck. I wonder if we should teach this to read the mozharness config instead (we also have bug 1082823 to move it from master to slave)

I don't understand what needs to change here in the actual data to fix this other than removing updates in the first item
https://aus4-admin.mozilla.org/releases/Fennec-34.0b6-build1/data

The bata-localtest url needs to lose update. So here I would remove the update/ part by 
where ProductName is Fennec and channel is betatest. Because for Firefox it needs to remain
http://hg.mozilla.org/build/tools/file/046e9d5ec108/lib/python/balrog/submitter/cli.py#l183

Should I remove android-x86 from mozharness/configs/releases/bouncer_fennec.py?

If having a short vidyo to discuss would be easier, that would be great with me.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 114 • 10 years ago

We got an email '[release] Fennec 33.1 build1: Updates available on None', so templates might need a further tweak.

	Kim Moir [:kmoir] ET Assignee
	Comment 115 • 10 years ago

Hmm, I looked at the templates issue and couldn't seem to see where the problem is.  Will look at it a again a bit later

	Kim Moir [:kmoir] ET Assignee
	Comment 116 • 10 years ago

So I was talking to Nick about this bug today and he raised two important points

Will there every be anything else than en-US for Android-x86?  If not, it should not be submitted to balrog and removed from the configs that do so.

What sorts of QA will be done on these single locales for Android at the time of release?  If there isn't going to be any formal testing we should remove the localtest channels from these configs.

	Kim Moir [:kmoir] ET Assignee
	Comment 117 • 10 years ago

Other discussion with Nick re how to fix the remaining problems

Option 1
Rewrite get_release_update in https://mxr.mozilla.org/build/source/tools/lib/python/util/tuxedo.py#66 to read the to read the mozharness bouncer config files for balrog
http://hg.mozilla.org/build/mozharness/file/2212615c4fa4/configs/releases/bouncer_firefox_release.py#l94
http://hg.mozilla.org/build/mozharness/file/2d78545b5039/configs/releases/bouncer_fennec.py

loop over the keys in the keys and ensure they are correct

This is a change in both Fennec and desktop and will require testing in the staging bouncer + staging master + sentry.

Option 2:
skip_updates=True in the release config
leave bouncer config
change balrog submitter so so it accepts Fennec-Version-complete
wrangle tuxedo.py to parse this

	Nick Thomas [:nthomas] (UTC+12)
	Comment 118 • 10 years ago

Nice summary, one tweak ...

(In reply to Kim Moir [:kmoir] from comment #117)
> Option 2:
> skip_updates=True in the release config
> leave bouncer config
> change balrog submitter so so it accepts Fennec-Version-complete
> wrangle tuxedo.py to parse this

change balrog submitter so it posts Fennec-<version>. Should be OK in tuxedo.py then, as skip_updates makes checkMars False.

	Brad Lassey [:blassey] (use needinfo?) Reporter
	Comment 119 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #116)
> So I was talking to Nick about this bug today and he raised two important
> points
> 
> Will there every be anything else than en-US for Android-x86?  If not, it
> should not be submitted to balrog and removed from the configs that do so.
Yes, Android-x86 should have the same locales as Android-ARM
 
> What sorts of QA will be done on these single locales for Android at the
> time of release?  If there isn't going to be any formal testing we should
> remove the localtest channels from these configs.
That's a question for QA

	Kim Moir [:kmoir] ET Assignee
	Comment 120 • 10 years ago

> What sorts of QA will be done on these single locales for Android at the
> time of release?  If there isn't going to be any formal testing we should
> remove the localtest channels from these configs.
That's a question for QA

Aaron I'm working on enabling single locale updates for Android on beta and release.  Do you know the extent of testing these repacks will receive by QA?

	Nick Thomas [:nthomas] (UTC+12)
	Comment 121 • 10 years ago

Specifically, do you want to use the desktop model for update testing, which is to check those updates on a test channel prior shipping to users.

	Kim Moir [:kmoir] ET Assignee
	Comment 122 • 10 years ago

In the mobile meeting today kbrosan said that he would use the desktop model for testing.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 123 • 10 years ago

(In reply to Nick Thomas [:nthomas] from comment #114)
> We got an email '[release] Fennec 33.1 build1: Updates available on None',
> so templates might need a further tweak.

Any idea what's going on here ? Hit it on 34.0 build1 and seems to just be release builds, beta's OK.

	Kim Moir [:kmoir] ET Assignee
	Comment 124 • 10 years ago

With respect to comment 123, 

the templates for beta have 
releaseConfig['localTestChannel']      = 'betatest'
releaseConfig['cdnTestChannel']        = 'releasetest'
and the message templates such as 
fennec_updates_success reference localTestChannel

So should the templates for fennec releases also reference localtest or should both beta and releases be updated to new channel names for fennec

I read this wiki page but it was not clear to me what the correct channel should be
https://wiki.mozilla.org/Releases/Update_Channels

	Kim Moir [:kmoir] ET Assignee
	Comment 125 • 10 years ago

Latest data for the patches to address comment #118 is here

https://aus4-admin-dev.allizom.org/releases#Fennec%2034.0b8

Patches will be attached next.

	Kim Moir [:kmoir] ET Assignee
	Comment 127 • 10 years ago

green builds in staging

	Nick Thomas [:nthomas] (UTC+12)
	Comment 128 • 10 years ago

(In reply to Kim Moir [:kmoir] from comment #124)

I think you're right that it's that some of the configs don't have localTestChannel and cdnTestChannel definitions. For now lets copy what firefox does for mozilla-release builds, and use betatest and releasetest. Later, when we swap to Balrog for release we can change the test channel names for both Firefox and Fennec.

	Kim Moir [:kmoir] ET Assignee
	Comment 129 • 10 years ago

patch to address comment 128

	Kim Moir [:kmoir] ET Assignee
	Comment 130 • 10 years ago

noticed looking at the current release jobs that is was also missing releaseChannel and releaseChannelRuleIds.  Not sure what the releaseChannelRuleIds should be

	Chris Cooper [:coop] (he/him)
	Comment 131 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/2c26cb775d70

	Nick Thomas [:nthomas] (UTC+12)
	Comment 132 • 10 years ago

Comment on attachment 8529136 [details] [diff] [review]
bug1019724fennecrtemplates.patch

r+ with a some changes described below.

>diff --git a/mozilla/release-fennec-mozilla-release.py.template b/mozilla/release-fennec-mozilla-release.py.template
>+releaseConfig['releaseChannelRuleIds'] = []

That's fine, will change when we switch to Balrog. Should set up staging configs the same though, and add to all four files for beta.

>diff --git a/mozilla/staging_release-fennec-mozilla-release.py.template b/mozilla/staging_release-fennec-mozilla-release.py.template
>+releaseConfig['testChannels']          = ['release-localtest', 'release-cdntest']

testChannels can be removed complete, deprecated by bug 986990.

	Kim Moir [:kmoir] ET Assignee
	Comment 133 • 10 years ago

updated incorporating feedback from comment 132

	Jordan Lund (:jlund)
	Comment 134 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/367a1d157f91

	Rail Aliiev [:rail]
	Comment 135 • 10 years ago

BTW, there are 3 dangling symlinks introduced in http://hg.mozilla.org/build/buildbot-configs/rev/eab4127639e5:

$ find -L . -type l        
./mozilla/release_templates/fennec_ready_for_release
./mozilla/release_templates/firefox_ready_for_release
./mozilla/release_templates/thunderbird_ready_for_release

All of them point to ready_for_release, which doesn't exist. Can you fix them too.

	Kim Moir [:kmoir] ET Assignee
	Comment 136 • 10 years ago

remove extraneous files mentioned in comment #135

	Kim Moir [:kmoir] ET Assignee
	Comment 137 • 10 years ago

I think these should be ready to test for the next beta - Kbrosnan could you make sure they are verified by QE?

	Kim Moir [:kmoir] ET Assignee
	Comment 138 • 10 years ago

test channels for beta

	Chris Cooper [:coop] (he/him)
	Comment 139 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/d75ad61ca11c

	Chris Cooper [:coop] (he/him)
	Comment 140 • 10 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/1bda6639bf51

	Kim Moir [:kmoir] ET Assignee
	Comment 141 • 10 years ago

It would be good if someone from QE could verify the single locale updates from 35.0b4 on 
http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/35.0b4-candidates/build1/

They are allocated to channels beta-localtest and  beta-cdntest

	Nick Thomas [:nthomas] (UTC+12)
	Comment 142 • 10 years ago

Unfortunately, bug 792992 means we can't use test channels for Android (not yet anyway, I requested tracking on it). If there aren't any users out there already we can probably test on beta.

	Kim Moir [:kmoir] ET Assignee
	Comment 143 • 10 years ago

okay thanks I didn't realize that nthomas.  I've added a rule for the release channel id in balrog and linked it to 35.0b4. Here's a patch so it can be enabled for future betas automatically.

	Kevin Brosnan [Ex-Mozilla]
	Comment 144 • 9 years ago

See 1117831 comment 5 for current status. Blocked on development work or we need to use the beta channel.

	Mark Finkle (:mfinkle) (use needinfo?)
	Comment 145 • 9 years ago

(In reply to Nick Thomas [:nthomas] from comment #142)
> Unfortunately, bug 792992 means we can't use test channels for Android (not
> yet anyway, I requested tracking on it). If there aren't any users out there
> already we can probably test on beta.

Bug 792992 was fixed. What do we need to do to start some testing?

	Kim Moir [:kmoir] ET Assignee
	Comment 146 • 9 years ago

Were the patches from  bug 792992 uplifted to beta so it could be tested?  This is what I asked here
https://bugzilla.mozilla.org/show_bug.cgi?id=792992#c34 

We need the patch to be available in beta so QA can test

	Chris Cooper [:coop] (he/him)
	Comment 147 • 9 years ago

Any update here?

	Kim Moir [:kmoir] ET Assignee
	Comment 148 • 9 years ago

waiting for a needinfo from mkfinkle in bug 1139567 before we can proceed.  If you read through the comments in bug 1139567, the work required seems to have increased substantially with the introduction of split apks

	Chris Cooper [:coop] (he/him)
	Comment 149 • 9 years ago

(In reply to Kim Moir [:kmoir] from comment #148)
> waiting for a needinfo from mkfinkle in bug 1139567 before we can proceed. 
> If you read through the comments in bug 1139567, the work required seems to
> have increased substantially with the introduction of split apks

Karen answered the NI in bug 1139567. In short, this is still wanted.

	Brian King [:kinger]
	Comment 151 • 9 years ago

I'd like to advocate for a standalone, updateable APK for Indic Marathi (mr). See:

https://github.com/mozilla/participation-org/issues/99

	Justin Wood (:Callek)
	Comment 152 • 8 years ago

Kim, this had a BUNCH of patches on it, most checked in, and most over a year old, can we wrap this up and re-file with the current ask(s) as known, so the current state is easier to detangle, please.

	Kim Moir [:kmoir] ET Assignee
	Comment 153 • 8 years ago

The patches were implemented to lay the groundwork for supporting single locale builds but it was never fully implemented because other priorities were deemed more important to work on.  

Callek: Looking at the email thread that you started entitled "Fennec Single Locale Repacks (Nightly/Release"
with Barbara, Lawrence, Catlee etc on it, there still doesn't seem to be strong consensus that these are a priority to implement.

From catlee
I spoke with Axel and Flod about the single-locale repacks in London. They mentioned they were useful for localizers working on locales that weren't yet ready to be included in the multi-locale build.

From Lawrence:
Useful but necessary? Is there another option for these localizers?

Until we have strong consensus that these are needed, I'm not sure that this bug should stay open.  Most of the patches that were landed aren't won't be relevant for long anyways, given the move to release promotion and taskcluster.

	Chris Cooper [:coop] (he/him)
	Comment 154 • 8 years ago

(In reply to Kim Moir [:kmoir] from comment #153)
> From catlee
> I spoke with Axel and Flod about the single-locale repacks in London. They
> mentioned they were useful for localizers working on locales that weren't
> yet ready to be included in the multi-locale build.
> 
> From Lawrence:
> Useful but necessary? Is there another option for these localizers?

Is this something that we could offer as opt-in on Try only? 

I know that's not a model we generally support (i.e. builds on Try and not elsewhere), but if these really are only useful for vetting changes, do we really need regular builds elsewhere?