Andrew Halberstadt [:ahal] Reporter
	Description • 10 years ago

There are some mochitests that need to run on emulator-jb. Eventually we'll want to run all mochitests there, but I don't know that it's worth running all of them on both platforms. For now, let's just get the ones that require it running.

	Jason Smith [:jsmith]
	Comment 1 • 10 years ago

Copying over the comment about the context of the need:

Importance - This is important to get done in the 1.4 timeframe. In the perfect world, we would need this done before FC (3/17/2014). But I know this is late breaking & you guys have other priorities, so feel free to adjust the date as necessary here against your other priorities.

Impact - Right now we know the cost is that we'll have no mochitests protecting against regressions for the OMX Encoder, which is a 1.4 committed feature. That's potentially scary, as OMX decoding I know historically was famous for a high regression rate. I'd imagine there's other costs here that have to do with not having automation coverage on JB in general - we should ask eng. leads to find out what's impacted here. This is important for 1.4, given that JB is the mainline Android OS we'll be doing testing on, not ICS.

	Andrew Halberstadt [:ahal] Reporter
	Comment 2 • 10 years ago

Hey Armen, would you mind looking over this patch and checking extra carefully if I made any mistakes in Aki's absence? Or let me know if I should f? someone else.. I have some question, will ask inline.

	Andrew Halberstadt [:ahal] Reporter
	Comment 3 • 10 years ago

Comment on attachment 8376537 [details] [diff] [review]
schedule_emulator_jb_mochitests

Review of attachment 8376537 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +58,5 @@
>      'linux32_gecko': {},
>      'linux64_gecko': {},
>      'macosx64_gecko': {},
>      'emulator': {},
> +    'emulator_jb': {},

How does this get tied to the emulator-jb build jobs? Is this all that's needed?

@@ +109,5 @@
>      'hg_bin': 'hg',
>      'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
>  }
>  
> +PLATFORMS['emulator_jb']['slave_platforms'] = ['ubuntu64_vm-b2g-emulator']

I left out the 'fedora-b2g-emulator' and 'ubuntu64_hw-b2g-emulator' as I figured we wouldn't be running these there. Should I add them in anyway just in case?

@@ +111,5 @@
>  }
>  
> +PLATFORMS['emulator_jb']['slave_platforms'] = ['ubuntu64_vm-b2g-emulator']
> +PLATFORMS['emulator_jb']['env_name'] = 'linux-perf'
> +PLATFORMS['emulator_jb']['ubuntu64_vm-b2g-emulator'] = {'name': "b2g_emulator_vm"}

Should this be b2g_emulator_jb_vm?

	Andrew Halberstadt [:ahal] Reporter
	Comment 4 • 10 years ago

Jason, could you point me at the minimum set of mochitest.ini manifest(s) needed to run all the tests you care about?

	Jason Smith [:jsmith]
	Comment 5 • 10 years ago

(In reply to Andrew Halberstadt [:ahal] from comment #4)
> Jason, could you point me at the minimum set of mochitest.ini manifest(s)
> needed to run all the tests you care about?

http://hg.mozilla.org/mozilla-central/file/33b3248b4aa0/content/media/test/mochitest.ini

Although in reality, I only need test_mediarecorder* tests. If we need to reduce load here, then we could write a patch to refactor the media recorder tests into a separate directory with it's own mochitest.ini file. Then, we would only have 16 tests running here.

	Randy Lin [:rlin] - Be Mozillian on 2015/01/26
	Comment 6 • 10 years ago

Thanks, If need us to have a patch to move recorder mochitest into another folder, please file a bug and assign to me, thanks.

	Andrew Halberstadt [:ahal] Reporter
	Comment 7 • 10 years ago

I don't think that's necessary. Running an extra hundred or so mochitests will be pretty small impact in terms of infrastructure load. Alternatively I could just create a new root manifest for emulator-jb in the mochitest directory that only contains those tests.

	Armen [:armenzg] Assignee
	Comment 8 • 10 years ago

Hi ahal,
Thanks for giving this a shot.

Dealing with buildbot platforms is not clear but this should help a bit.

If we load this:
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g&rev=23fb2aae7aad

We can see the b2g emulator ICS builds, if we open one of the logs we will see that a sendchange has a parameter --branch [1]:
--branch cedar-emulator-opt-unittest

This literally means "trigger all opt unittest jobs that are listening for the 'emulator' build on the Cedar branch'.

You can tell who listens to this type of build by looking at this line:
PLATFORMS['emulator']['slave_platforms'] = ['fedora-b2g-emulator', 'ubuntu64_vm-b2g-emulator', 'ubuntu64_hw-b2g-emulator']

From your patch I read that you want to add a new set of tests that listen to a new type of build.

I'm about to go to the doctor and be back in the afternoon. I will look closer to your patch at that time.

From a first pass, you seem in the right track, however, I believe that the issue lays on that the "emulator-jb" *build* is not enabled on Cedar.
No build builders -> No test builders.

This is modified on mozilla/b2g_config.py. I will look at it when I come back.

If it helps, I will paste in here what I do if my patches are adding or remove the builders that I want:
(buildbot)armenzg-thinkpad buildbot-configs hg:[default!] $ patch -p1 < attachment.cgi\?id\=8376537
(buildbot)armenzg-thinkpad buildbot-configs hg:[default!] $ ./setup-master.py -l | grep "linux"
bm51-tests1-linux
bm52-tests1-linux
bm53-tests1-linux
bm54-tests1-linux
bm67-tests1-linux
bm68-tests1-linux
(buildbot)armenzg-thinkpad buildbot-configs hg:[default!] $ ./setup-master.py master bm67-tests1-linux
(buildbot)armenzg-thinkpad buildbot-configs hg:[default!] $ ~/repos/releng/braindump/buildbot-related/builder_list.py master/master.cfg > list_of_builders.txt 
(buildbot)armenzg-thinkpad buildbot-configs hg:[default!] $ hg up -C
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(buildbot)armenzg-thinkpad buildbot-configs hg:[default!] $ ~/repos/releng/braindump/buildbot-related/builder_list.py master/master.cfg > old_list_of_builders.txt
(buildbot)armenzg-thinkpad buildbot-configs hg:[default!] $ diff -pU8 old_list_of_builders.txt list_of_builders.txt > differences

[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=34709556&tree=Cedar&full=1

	Armen [:armenzg] Assignee
	Comment 9 • 10 years ago

FTR what I mention on my last comment is briefly described in here:
https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques

	Andrew Halberstadt [:ahal] Reporter
	Comment 10 • 10 years ago

Thanks for the summary! I didn't notice we weren't running emulator-jb builds on Cedar

	Armen [:armenzg] Assignee
	Comment 11 • 10 years ago

Comment on attachment 8376537 [details] [diff] [review]
schedule_emulator_jb_mochitests

Review of attachment 8376537 [details] [diff] [review]:
-----------------------------------------------------------------

This was on the right direction. I will post a patch that will address this.

Unfortunately, adding tests for a "new" test platform involves 2 more patches.
I don't like the way that it is and I hope one day we will fix the status quo.

	Armen [:armenzg] Assignee
	Comment 14 • 10 years ago

ahal, this is the patch that replaces yours.
If you notice, you almost had it; there's barely any differences.

	Armen [:armenzg] Assignee
	Comment 15 • 10 years ago

ahal, mind if I grab the bug?

	Andrew Halberstadt [:ahal] Reporter
	Comment 16 • 10 years ago

Not at all, thanks Armen!

	Andrew Halberstadt [:ahal] Reporter
	Comment 17 • 10 years ago

Comment on attachment 8377799 [details] [diff] [review]
schedule jb mochitests on Cedar

Review of attachment 8377799 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thanks! Would you mind making a minor change to the manifest location? This will also burn until bug 974069 is finished.. not that it matters on Cedar anyway. I don't really care if you hold off landing or not :)

::: mozilla-tests/b2g_config.py
@@ +1638,5 @@
> +                    'extra_args': [
> +                        '--cfg', 'b2g/emulator_automation_config.py',
> +                        '--test-suite', 'mochitest',
> +                        '--this-chunk', '1', '--total-chunks', '1',
> +                        '--test-manifest', 'tests/mochitest/emulator-jb.ini',

Could you change this to point to 'tests/mochitest/manifests/emulator-jb.ini'? I realize we're probably going to need several, one per emulator platform at least and I'd rather not clutter the main mochitest directory.

	Justin Wood (:Callek)
	Comment 18 • 10 years ago

Comment on attachment 8377796 [details] [diff] [review]
emulator_jb.tools.diff

Review of attachment 8377796 [details] [diff] [review]:
-----------------------------------------------------------------

Please adjust this patch to add it to the new masters I just committed to product-masters.json

	Justin Wood (:Callek)
	Comment 19 • 10 years ago

Comment on attachment 8377797 [details] [diff] [review]
emulator_jb.puppet.diff

Review of attachment 8377797 [details] [diff] [review]:
-----------------------------------------------------------------

please e-mail release@, sfink, and :Fallen about this change since it will break staging masters (no puppet managing BuildSlaves.py)

	Justin Wood (:Callek)
	Comment 20 • 10 years ago

Please additionally make sure trychooser and regex's for slave_health, buildapi (reports), and aws_* scripts are updated accordingly

	bhearsum@mozilla.com (:bhearsum)
	Comment 21 • 10 years ago

in production

	Armen [:armenzg] Assignee
	Comment 22 • 10 years ago

I'm trying triggering a emulator-jb job.
Andrew's merge happened before the reconfig.

	Armen [:armenzg] Assignee
	Comment 23 • 10 years ago

I will investigate why the build did not trigger the tests.

	Armen [:armenzg] Assignee
	Comment 24 • 10 years ago

We're having a tbpl display issue.
You can see the tests running under ICS rather than JB.
You can use this direct URL:
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_emulator_jb_vm

	Andrew Halberstadt [:ahal] Reporter
	Comment 25 • 10 years ago

Thanks! Don't bother retriggering the tests yet. I'd expect them to fail until bug 974069 is finished.

	John Hopkins (:jhopkins)
	Comment 27 • 10 years ago

in production.

	Andrew Halberstadt [:ahal] Reporter
	Comment 28 • 10 years ago

In bug 974069 I decided to store the manifest in a 'manifests' subdirectory, so this requires a small tweak.

	Andrew Halberstadt [:ahal] Reporter
	Comment 29 • 10 years ago

https://hg.mozilla.org/build/buildbot-configs/rev/68cbd0408484

	John Hopkins (:jhopkins)
	Comment 30 • 10 years ago

In production.

	Armen [:armenzg] Assignee
	Comment 31 • 10 years ago

ahal: I believe we're done in here. Once things are green we can look at enabling across the place.
At that time, please file another bug in the same component.