Chris Cooper [:coop] (he/him) Reporter
	Description • 9 years ago

Android x86 builds use the fx_desktop_build.py mozharness script invoked with the custom build variant of x86.

Example command line:

/tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_android_64_builds.py --custom-build-variant-cfg x86 --config balrog/production.py --branch mozilla-inbound --build-pool production


Example build logs:

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-x86/1434122155/mozilla-inbound-android-x86-bm74-build1-build1142.txt.gz

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 1 • 9 years ago

How is this different from bug 1118394?

	Chris Cooper [:coop] (he/him) Reporter
	Comment 2 • 9 years ago

I only see the Android 4.0 API11+ TC build in try treeherder right now, so I want to make sure API9, x86, and debug variants don't get missed.

If you're using that bug to track moving *all* the variants, that's fine. Feel free to dupe.

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 3 • 9 years ago

I'm assuming (!) those won't be much more work than the single build.  I set up just the one to save load on TC until people are actually looking.

	Kim Moir [:kmoir] ET Assignee
	Comment 4 • 8 years ago

wip patches.  They don't work yet due to missing java env on worker.  

https://treeherder.mozilla.org/#/jobs?repo=try&revision=028ec64f5e34c9be6df7214c3a00fe518155663f

	Kim Moir [:kmoir] ET Assignee
	Comment 5 • 8 years ago

I added openjdk to the tooltool manifest and it ran green.  Will do some further verification to ensure it is correct.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b10ccb969ddc&selectedJob=22887702

	Kim Moir [:kmoir] ET Assignee
	Comment 6 • 8 years ago

diff of unzip -l of the old and new apks.  Not sure if the different manifest names are just because it was run on try

	Kim Moir [:kmoir] ET Assignee
	Comment 7 • 8 years ago

Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

Not sure if it's worth having a creating a second tooltool manifest for android x86 since it's going away and having openjdk in the one for buildbot doesn't seem to harm anything.

	Geoff Brown [:gbrown]
	Comment 8 • 8 years ago

Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

Review of attachment 8765083 [details] [diff] [review]:
-----------------------------------------------------------------

I think adding on to the existing tooltool manifest is fine.

In your try run, I notice that the tc build had a substantially longer run-time than the buildbot build -- do you know why?

	Pulsebot
	Comment 9 • 8 years ago

Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3e53a0aae914
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

	Kim Moir [:kmoir] ET Assignee
	Comment 10 • 8 years ago

Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

http://hg.mozilla.org/integration/mozilla-inbound/rev/3e53a0aae914

Thanks, I'm investigating the issue wrt longer build times.  I know with some of the other conversions, the problem was that tc builds were running on less powerful instance types (with less RAM). I'm running another try build now to try and will look at the instance types each run with.

	Pulsebot
	Comment 11 • 8 years ago

Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6b992274233a
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

	Kim Moir [:kmoir] ET Assignee
	Comment 12 • 8 years ago

Yes, this is the problem.  Buildbot builds currently run on r3.xlarge and the tc worker-type I assigned it is android-api-15 (me.2xlarge) which has less memory.  Going to try to use a different worker type definition or create a new one.  None of the tc worker type definitions currently use r3.xlarge according to https://github.com/taskcluster/aws-provisioner

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 13 • 8 years ago

So this caused a problem in production.  The issue was that the inheritence chain of config files does the following:

taskcluster/ci/legacy/tasks/builds/android_api_15.yml inherits from
taskcluster/ci/legacy/tasks/builds/android_x86.yml also inherits from
taskcluster/ci/legacy/tasks/builds/mobile_base.yml which inherits from
taskcluster/ci/legacy/tasks/docker_build.yml

The variables passed from the inherit section in android_api_15.yml are the same as android_x86.yml, 

  variables:
    build_name: 'android'
    build_type: 'opt'

which resolve in docker_build.yml to the routes:

  routes:
    - 'index.gecko.v1.{{project}}.revision.linux.{{head_rev}}.{{build_name}}.{{build_type}}'
    - 'index.gecko.v1.{{project}}.latest.linux.{{build_name}}.{{build_type}}'

which will collide.

Joel helped debug this, and has steps for reproducing this issue here https://pastebin.mozilla.org/8880122 .  This should be a simple matter of changing build_name or build_type, but I'm not sure of the impact of making those changes.  Once I find someone who can do the backout, that's what I'll do

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 14 • 8 years ago

Joel's completed the backout: http://hg.mozilla.org/integration/mozilla-inbound/rev/73732455d235

	Kim Moir [:kmoir] ET Assignee
	Comment 15 • 8 years ago

new patch, running try run.  Ironically, I had a different build_name in my patch queue locally but not in the patch I submitted to bugzilla

	Kim Moir [:kmoir] ET Assignee
	Comment 16 • 8 years ago

I'm not sure what the output should be given the fix

Kims-MacBook-Pro:mozilla-central kmoir$  ./mach taskgraph target -p ~/Downloads/parameters.yml 
 0:00.19 Generating full task set
 0:00.56 Starting new HTTPS connection (1): hg.mozilla.org
 0:22.41 Generating full task graph
 0:22.42 Generating target task set
TaskLabel==Htn2LYRFRGGPONQ6Y7aNRw
TaskLabel==dtGSMvrkQ8aPgYfDEwoCrQ


jmaher: how do you tell if mach is generating the right taskgraph? Do the task graphs correspond to something that you can verify in the taskcluster tools?

	Kim Moir [:kmoir] ET Assignee
	Comment 17 • 8 years ago

Comment on attachment 8765924 [details] [diff] [review]
bug1174206-2.patch

I also changed the worker type to m3.2xlarge but it seems that linux64-opt was allocated to it even though there are r3.2xlarge in the pool so the build wasn't faster. https://tools.taskcluster.net/aws-provisioner/#opt-linux64/

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 18 • 8 years ago

:kmoir, this is all cryptic, but actually you solved it because when the patch that landed was there it was an error printed out while running |mach taskgraph target|, you don't see the error.

I have confidence all is well!

	Geoff Brown [:gbrown]
	Comment 19 • 8 years ago

Comment on attachment 8765924 [details] [diff] [review]
bug1174206-2.patch

Review of attachment 8765924 [details] [diff] [review]:
-----------------------------------------------------------------

You may have it right already, but in light of bug 1283091, maybe double-check the x86 route.

	Pulsebot
	Comment 20 • 8 years ago

Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1cee0d296a34
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

	Pulsebot
	Comment 21 • 8 years ago

Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/89d45ec9d5aa
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown DONTBUILD

	Kim Moir [:kmoir] ET Assignee
	Comment 22 • 8 years ago

I changed the workertype back to android15 as it had a better pool of high memory instance types

	Carsten Book [:Tomcat]
	Comment 23 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/1cee0d296a34
https://hg.mozilla.org/mozilla-central/rev/89d45ec9d5aa

	Kim Moir [:kmoir] ET Assignee
	Comment 24 • 8 years ago

I have to add another patch to enable on other branches, this was just on try.

	Kim Moir [:kmoir] ET Assignee
	Comment 25 • 8 years ago

patch to enable on other branches

	Pulsebot
	Comment 26 • 8 years ago

Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c4fc457de3a7
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

	Kim Moir [:kmoir] ET Assignee
	Comment 27 • 8 years ago

This is landed on inbound and I've verified on treeherder However, I'm going to ask in the next mobile meeting if Android x86 is still needed. I recall asking in a meeting about six months ago that Android x86 has very small market numbers compared to armv7. Going to confirm if this is still the case and what the future holds for x86 builds from a product standpoint.

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 28 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/c4fc457de3a7