Chris Cooper [:coop] (he/him) Reporter
	Description • 8 years ago

We currently have 3 Mac test masters:

bm106-tests1-macosx	
bm107-tests1-macosx
bm108-tests1-macosx

I tried connecting to bm108 today, and it was painfully slow. We have 336 Mac test machines trying to connect to those 3 masters, so that slowness doesn't really surprise me. We've had high pending counts for Mac tests recently (>3000 pending yesterday), so we should do what we can to improve the performance of those masters. 

On top of the general slowness, there are plans to add some 200 more r7 testers (bug 1184181).

We should create five new buildbot-masters to address the current slowdown (giving us 8 Mac testing masters total), and also provide us with master capacity for the new machines when they arrive.

536 Mac test machines / 8 masters = 67 machines per master

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 1 • 8 years ago

More information how to create a buildbot in AWS https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_buildbot_masters_in_AWS

How to add a master in slavealloc https://wiki.mozilla.org/ReleaseEngineering:Buildduty:Slave_Management#Adding_a_master

	Amy Rich [:arr] [:arich]
	Comment 2 • 8 years ago

We'll should be ready to start imaging machines in a couple days. Can someone please look at getting the new buildbot masters created and configs modified soon?

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 3 • 8 years ago

Attached you can find the patch that add new 5 buildbot master for macosx in moco-nodes

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 4 • 8 years ago

Attached you can find the patch that add 5 new macosx in production_master.json by default disabled

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 5 • 8 years ago

Attached you can find the csv file for the new masters in order to be added in slavealloc DB

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 6 • 8 years ago

Attached you can find the path to enable the new masters in production. This patch needs to be land the last one

	Kim Moir [:kmoir] ET
	Comment 7 • 8 years ago

Comment on attachment 8748720 [details] [diff] [review]
bug1264417_enable-masters.patch

Looks good. Remember to lock some machines to the masters
https://wiki.mozilla.org/ReleaseEngineering/AWS_Master_Setup#Lock_a_slave_and_let_it_take_jobs

before enabling it completely to ensure it runs green jobs etc

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 8 • 8 years ago

The new masters have been added in slavealloc DB

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 9 • 8 years ago

We have pointed 5 slaves to connect to every buildbot-master created and everything seems to work as intended.
More details can be found bellow :
http://buildbot-master132.bb.releng.scl3.mozilla.com:8201/one_line_per_build
http://buildbot-master133.bb.releng.scl3.mozilla.com:8201/one_line_per_build
http://buildbot-master134.bb.releng.scl3.mozilla.com:8201/one_line_per_build
http://buildbot-master135.bb.releng.scl3.mozilla.com:8201/one_line_per_build
http://buildbot-master136.bb.releng.scl3.mozilla.com:8201/one_line_per_build

From our point of view the new masters are working ok,can we enable them in the production.

	Kim Moir [:kmoir] ET
	Comment 10 • 8 years ago

great work! yes please land the final patch

	Vlad Ciobancai [:vladC] [:vciobancai] Assignee
	Comment 11 • 8 years ago

The patch landed https://hg.mozilla.org/build/tools/rev/183ee5c45256
The new masters have been enabled in to the production