Nick Thomas [:nthomas] (UTC+12) Reporter
	Description • 9 years ago

Busted from the merges. Work around for bug 1195546.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 1 • 9 years ago

I've tried to limit the scope of this right down, so that it doesn't break anything else. It should fix Firefox & T'bird ci builds, as well as releases.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 2 • 9 years ago

Comment on attachment 8681016 [details] [diff] [review]
[buildbotcustom] Set MOZ_CURRENT_PROJECT for make package in opt mac builds

https://hg.mozilla.org/build/buildbotcustom/rev/09ad19193503
https://hg.mozilla.org/build/buildbotcustom/rev/ef70aa76c86d

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 3 • 9 years ago

Explanation of what's going on - the patch is setting MOZ_CURRENT_PROJECT=i386 in the env for the 'make package' steps of mac universal builds. We're calling that in %(basedir)s/build/obj-firefox/i386.

This is based on glandium's recommendation that we either do that or unset MOZ_OBJDIR.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 4 • 9 years ago

We got past make package on a mozilla-beta build. Jobs are running on other branches to make sure we haven't busted them.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 5 • 9 years ago

We need to apply the same treatment the make package calls with pretty names, to resolve the orange status of the builds. There's one for firefox & thunderbird on m-b, but three for Thunderbird builds on comm-central and comm-aurora. The extra TB errors, in 'make package-tests MOZ_PKG_PRETTYNAMES=1' and 'make -C objdir-tb/i386/tools/update-packaging MOZ_PKG_PRETTYNAMES=1', may be from something else given how long Thunderbird has been broken.

Firefox on m-r and m-esr38 seems to be fine. I won't have time to work on this, could someone else pick it up ?

	aleth [:aleth]
	Comment 6 • 9 years ago

(In reply to Nick Thomas [:nthomas] from comment #5)
> We need to apply the same treatment the make package calls with pretty
> names, to resolve the orange status of the builds. There's one for firefox &
> thunderbird on m-b, but three for Thunderbird builds on comm-central and
> comm-aurora. The extra TB errors, in 'make package-tests
> MOZ_PKG_PRETTYNAMES=1' and 'make -C objdir-tb/i386/tools/update-packaging
> MOZ_PKG_PRETTYNAMES=1', may be from something else given how long
> Thunderbird has been broken.
> 
> Firefox on m-r and m-esr38 seems to be fine. I won't have time to work on
> this, could someone else pick it up ?

Does the patch in bug 1214403 address some of these issues?

	Chris AtLee [:catlee]
	Comment 7 • 9 years ago

r+ from jlund on irc

	Jordan Lund (:jlund) Assignee
	Comment 8 • 9 years ago

comment 7 is the attempted fix for comment 5. which got us from an orange build to actually green[1]. however, we are, as philor pointed out, still failing the xpcshell and mochi-other tests. The suspected culprit is still the build as signs point to wrong arch (i386)

[1] https://treeherder.mozilla.org/logviewer.html#?job_id=617621&repo=mozilla-beta

	Jordan Lund (:jlund) Assignee
	Comment 9 • 9 years ago

xpcshell failure - https://treeherder.mozilla.org/logviewer.html#?job_id=617938&repo=mozilla-beta

mochi-other failure - https://treeherder.mozilla.org/logviewer.html#?job_id=617926&repo=mozilla-beta

glandium, ted - any ideas re: comment 8? we are likely going to find the culprit in a 'mach/mozharness' only change that is being ignored on beta as beta uses 'client.mk build' directly through buildbot (skipping mach/mozharness wrappers). e.g. https://hg.mozilla.org/releases/mozilla-beta/rev/9126c5fcc788#l1.23

	Phil Ringnalda (:philor)
	Comment 10 • 9 years ago

The third thing wrong is https://treeherder.mozilla.org/logviewer.html#?job_id=617949&repo=mozilla-beta - mochitest-2 failing EME and H264 tests, which could easily either be other things from the wrong arch, or using ctypes and being the same wrong arch thing.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 11 • 9 years ago

philor and I both discovered bug 1201224 at much the same time. Previously we were unifying i386 and x86_64 architectures into universal test binaries. Now we're packaging tests in <objdir>/i386 but running the tests on hosts that default to 64bit, and that doesn't go so well.

I've landed this r=bustage 
  https://hg.mozilla.org/build/buildbotcustom/rev/9797e9b039e9
  https://hg.mozilla.org/build/buildbotcustom/rev/a615dd77dfe3
but would appreciate retrospective review.

It also removes 'make package-tests MOZ_PKG_PRETTYNAMES=1' steps, since we don't run tests on release builds and those are the only things to use pretty names.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 12 • 9 years ago

Comment on attachment 8681721 [details] [diff] [review]
[buildbotcustom] Use 64bit part of objdir for test packaging

This turned out to be a bad idea because we don't end up with any test archives uploaded.

Here's the major steps and working directory they get called in:
build - %(basedir)s/build
buildsymbols - %(basedir)s/build/obj-firefox/i386
package-tests - should %(basedir)s/build/obj-firefox/x86_64  (with this patch, was i386)
package - %(basedir)s/build/obj-firefox/i386
python build/config/printconfigsetting.py build/obj-firefox/i386/dist/bin/application.ini x3 - .
upload - %(basedir)s/build/obj-firefox/i386
package MOZ_PKG_PRETTYNAMES=1 - %(basedir)s/build/obj-firefox/i386
check - %(basedir)s/build/obj-firefox/i386

Looking at running all the i386 tasks on x86_64 instead for m-b, pretty sure that's what m-c/m-a are doing.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 13 • 9 years ago

Comment on attachment 8681721 [details] [diff] [review]
[buildbotcustom] Use 64bit part of objdir for test packaging

Backed this out:
 https://hg.mozilla.org/build/buildbotcustom/rev/6f01af020d4e
 https://hg.mozilla.org/build/buildbotcustom/rev/8f4da13e9ffe

... and landed a config change instead:
 https://hg.mozilla.org/build/buildbot-configs/rev/615851873863
 https://hg.mozilla.org/build/buildbot-configs/rev/9442d0cd2501

NB - this is a very specific patch for mozilla-beta. I looked at changing the platform default, then setting it back for items_before(..., 42) (ie m-r, m-esr38) but the dump_master.py diff was alarmingly large. Don't have time to digest that or clobber a lot of things right now.

If this sticks, then we'll need revisit. m-r will need it at the next merge at the very least. And it's kinda sucky to have old branches still using i386 too, but l10n.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 14 • 9 years ago

Retrigger running on m-b 38f98cdc8d49, with clobber via cloberrer. Also retriggering m-esr38 13e1af34b11e because the buildbotcustom patch introduced the media errors we're seeing on beta.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 15 • 9 years ago

mozilla-beta went green and philor has reopened that. On esr38 he's clobbered and pushed a whitespace change to get new builds, fingers crossed that comes up green on the media tests too.

To do: 
* check state of esr38
* check the Thunderbird branches
* make sure we apply the buildbot-configs change on release at the next merge.


The net effect of the changes here is
* on mozilla-beta, the working directory is build/obj-firefox/x84_64 instead of build/obj-firefox/i386 when calling make with targets buildsymbols, package-tests, package, 3 printconfigsetting.py calls, upload, package with pretty names, and check
(https://hg.mozilla.org/build/buildbot-configs/rev/615851873863)

* on all branches still using buildbot factories for compilation (m-b, m-r, m-esr38, c-c, c-a, c-b, c-esr38, maybe a project branch or too) set MOZ_CURRENT_PROJECT to x86_64 or i386 for make sdk, package-tests, package, and any pretty name equivalents
(hg diff -r fa3d45a6ec1e:8f4da13e9ffe process/factory.py)

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 16 • 9 years ago

(In reply to Nick Thomas [:nthomas] from comment #15)
> To do: 
> * check state of esr38

Looks like philor has also retriggered a mac opt build on mozilla-release (no clobber there), so that needs checking too.

> * check the Thunderbird branches

I've retriggered on mozilla-aurora, since that only has this bug in play. The other branches are more complicated.

> * make sure we apply the buildbot-configs change on release at the next
> merge.

	(not currently active) Ted Mielczarek
	Comment 17 • 9 years ago

I asked this of catlee on IRC: what would we have to do to make the beta/etc build environments match aurora/nightly? This sort of thing is going to be come more commonplace as we make larger build system changes, and it's pretty hard for us to know that we'll break it in advance.

	Chris AtLee [:catlee]
	Comment 18 • 9 years ago

The biggest change and risk is to release automation, which is still using buildbot. That would have to be moved also. It's arguable that having beta/release use completely different build system invocations is a larger risk than changing release automation to use mozharness/mach though.

	Jordan Lund (:jlund) Assignee
	Comment 19 • 9 years ago

dump_master_diff: http://people.mozilla.org/~jlund/doUpload_fix_for_RBF-dump_master_diff.patch

this will also affect other branches like esr but I didn't have that in my master_config.json['release_branches']

	Jordan Lund (:jlund) Assignee
	Comment 20 • 9 years ago

same as before but with comment fix:

diff --git a/process/factory.py b/process/factory.py
index a8377a2..136f593 100644
--- a/process/factory.py
+++ b/process/factory.py
@@ -2636,7 +2636,7 @@ class ReleaseBuildFactory(MercurialBuildFactory):
         if self.enableUpdatePackaging:
             update_env = self.env.copy()
             if self.complete_platform == 'macosx64':
-                # make package is broken in universal (opt) builds) if
+                # automation make targets are broken in universal (opt) builds) if
                 # MOZ_OBJDIR is set but you're calling in a specific arch, bug 1195546
                 update_env.update({'MOZ_CURRENT_PROJECT': os.path.basename(self.objdir)})
             self.addStep(MockCommand(
@@ -2672,7 +2672,7 @@ class ReleaseBuildFactory(MercurialBuildFactory):
                           'UPLOAD_TO_TEMP': '1',
                           'UPLOAD_EXTRA_FILES': ' '.join(self.UPLOAD_EXTRA_FILES)})
         if self.complete_platform == 'macosx64':
-            # make package is broken in universal (opt) builds) if
+            # automation make targets are broken in universal (opt) builds) if
             # MOZ_OBJDIR is set but you're calling in a specific arch, bug 1195546
             uploadEnv.update({'MOZ_CURRENT_PROJECT': os.path.basename(self.objdir)})
         if self.stageSshKey:

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 21 • 9 years ago

Comment on attachment 8682134 [details] [diff] [review]
151102_bug_1220018_doUpload_fix_for_RBF_moz_current_project-2.patch

I think we shouldn't take this actually. The CI builds aren't setting this so lets keep release the same.

	Jordan Lund (:jlund) Assignee
	Comment 22 • 9 years ago

> To do: 
> * check state of esr38

m-r and esr38 look good to me. sheriffs feel free to correct me.

> * check the Thunderbird branches

fallen: given changes outlined in comment 15, how are continuous integration comm- trees holding up?

> * make sure we apply the buildbot-configs change on release at the next
> merge.
> 

the buildbot-config patch in question is: https://hg.mozilla.org/build/buildbot-configs/rev/615851873863

I've filed and blocked this against next uplift: Bug 1220979



if tb looks good and there is no objection, I'm going to close this bug since we are not changing ReleaseBuildFactory make targets (comment 21). 1220979 will track work required for next uplift

	Phil Ringnalda (:philor)
	Comment 23 • 9 years ago

m-r is... probably okay. A clobber and multiple retriggers to get past the way that our cache invalidation for retriggers is nonexistent got me green m2 and m-oth, and no green xpcshell. The odds are fairly good that https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=a420b49d3b37 will wind up green.

	Phil Ringnalda (:philor)
	Comment 24 • 9 years ago

m-r is okay.

	Release Engineering SlaveAPI Service
	Comment 25 • 9 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/615851873863

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 26 • 9 years ago

Thanks philor. I think we're all done here.

	Chris AtLee [:catlee]
	Comment 27 • 7 years ago

clearing needinfo