Closed Bug 1204756 Opened 9 years ago Closed 9 years ago

increase size of tst-emulator64-spot instance pool

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kmoir, Assigned: vciobancai)

References

Details

Attachments

(4 files, 1 obsolete file)

I've noticed recently that the wait times for the tst-emulator64-spot pool are quite long.  In watch_pending.cfg the global pool is set to 1300 with us-east-1 and us-west-2 set at 1000 each. Perhaps we could try increasing the size of the pool by 50%

Need to write patches to update range mozilla-tests/production_config.py and add new instance machines to slavealloc

This pool will need more capacity when we enable the remaining Android 4.3 debug tests in bug
Created the patch to increase the tst-emulator64-spot
Attachment #8661246 - Flags: review?(kmoir)
Rail, why didn't we have a contiguous range of machines the last time we expanded the size of this pool. I recall there was a reason, but don't remember now.  Also, is there anything we need to change wrt the ip address pool assigned to this type of instance when we expand the pool size?
Flags: needinfo?(rail)
Assignee: nobody → vlad.ciobancai
No need to extend the address pool. Just 3 steps similar to bug 1090568.
Flags: needinfo?(rail)
Thanks Rail.  Vlad, you also need a patch to add them to slavealloc as you can see for example in bug 1090568.  Please don't add them to slavealloc until we are ready to go with the other patches. Also, we usually add them to slavealloc as disabled at first.
Attachment #8661246 - Flags: review?(kmoir) → review+
Attached file emulator.txt
vlad had problems with his computer today and was unable to write the patch to add the slaves to slavealloc so I went ahead and did it
I looked at configs/watch_pending.cfg but and I don't think we need to change the limits
It is 1300 globally, 1000 for each us-west-2 and us-east1 which is still under the total pool size of 600 instances
Attachment #8661246 - Flags: checked-in+
FYI, we saw a bunch of alerts for usw2 and use1 from the buildbot master twistd logs (many more than this):

[sns alert] Wed 09:09:02 PDT buildbot-master120.bb.releng.use1.mozilla.com watch_twistd_log.py: Count: 69 | First instance: 2015-09-16 08:55:53-0700 | Most recent instance: 2015-09-16 09:00:00-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.134.44.20
[sns alert] Wed 09:09:02 PDT buildbot-master118.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 155 | First instance: 2015-09-16 08:50:43-0700 | Most recent instance: 2015-09-16 08:59:58-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.157.184
[sns alert] Wed 09:09:02 PDT buildbot-master118.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 153 | First instance: 2015-09-16 08:50:57-0700 | Most recent instance: 2015-09-16 08:59:57-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.157.195
I can see new jobs running on the expanded pool now
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
kmoir think we may need to increase the tst-emulator64 pool again
kmoir	http://builddata.pub.build.mozilla.org/reports/pending/pending.html
kmoir	we have the same pool size as we used to with the pandas
kmoir	now that most tests run on android 4.3 on emulators
kmoir	but they take longer and have more chunks
kmoir	plus these instances are used for other tests other than android 4.3
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 1205409
Created a new patch in order to increase the number of tst-emulator64-spot
Attachment #8662265 - Flags: review?(kmoir)
Attached file emulator.csv (obsolete) —
Attachment #8662365 - Flags: review?(kmoir)
Created the patch emulator.csv to add the slaves to slavealloc
Attachment #8662265 - Flags: review?(kmoir) → review+
Comment on attachment 8662265 [details] [diff] [review]
production_config.py_v2.patch

Looks good except 900 needs to be excluded since they aren't in the range you specify in the python patch (I had a mistake in my original patch)
Attachment #8662265 - Flags: review+ → review-
Attachment #8662265 - Flags: review- → review+
Comment on attachment 8662365 [details]
emulator.csv

Looks good except 900 needs to be excluded since they aren't in the range you specify in the python patch (I had a mistake in my original patch)
Attachment #8662365 - Flags: review?(kmoir) → review-
Attached file emulator.csv
Attachment #8662365 - Attachment is obsolete: true
Attachment #8662916 - Flags: review?(kmoir)
Attachment #8662916 - Flags: review?(kmoir) → review+
Vlad, you could add these machines to mysql using this tool
https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Slave_Management#Adding_a_slave

The machines will be disabled by default until we do the other work to enable them

I have to add more masters in bug 1205409 first
(In reply to Kim Moir [:kmoir] from comment #16)
> Vlad, you could add these machines to mysql using this tool
> https://wiki.mozilla.org/ReleaseEngineering/Buildduty/
> Slave_Management#Adding_a_slave
> 
> The machines will be disabled by default until we do the other work to
> enable them
> 
> I have to add more masters in bug 1205409 first

I have imported the slaves with success
Attachment #8662265 - Flags: checked-in+
I enabled the new instances in slavealloc after the reconfig had completed
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: