Jordan Lund (:jlund) Reporter
	Description • 10 years ago

this requires a the following process:

1) stop watch_pending from starting up new usw2 instances
** pretty much this: https://bugzilla.mozilla.org/show_bug.cgi?id=1060407#c2

2) disable the existing running usw2 slaves
** find the usw2 slaves: by listing them against the associated usw2 ami: http://hg.mozilla.org/build/cloud-tools/file/default/scripts/get_spot_amis.py
** disabling them in slavealloc

one issue here is we might have a problem with jacazzis. They are supposed to have slaves in their own respective regions but that might not be the case. On friday when we did step (1) above, we had a couple m-a and m-c linux32 build pending backlog (2+ hours pending) while nothing else was pending badly.

I think jacuzzis like these are all in usw2: http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/Linux%20mozilla-central%20build

or so the console says with bld-linux64-spot-3* slaves

	Jordan Lund (:jlund) Reporter
	Comment 1 • 10 years ago

callek pointed out that hwine has a script to mass disable slaves in slavealloc db and then also re-enable ones that were disabled via the last script call

	Jordan Lund (:jlund) Reporter
	Comment 2 • 10 years ago

we currently have a limit set to usw2 slaves so we are seeing an increase in pending. This might have a side effect of fishing out the usw2 only jacuzzi builders:

b2g_mozilla-aurora_emulator-debug_dep   2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_emulator-jb-debug_dep    2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_emulator-jb_dep  2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_emulator_dep     2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_flame_eng_dep    2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_hamachi_eng_dep  2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_linux32_gecko build  2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_linux32_gecko-debug build    2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_linux64_gecko build  2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_linux64_gecko-debug build    2014-09-02 10:18:54     1:11:59
b2g_mozilla-aurora_emulator-debug_dep   2014-09-02 09:59:58     1:30:55
b2g_mozilla-aurora_linux32_gecko build  2014-09-02 09:59:58     1:30:55


I just checked and http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/b2g_mozilla-aurora_emulator-debug_dep has only 1 slave and it's usw2 only.

advice from catlee on how to deal with this:

10:44:50 <catlee> https://wiki.mozilla.org/ReleaseEngineering:Jacuzzis
11:24:01 <•catlee> jlund: so you could disable the slaves, and then remove them from the configs
11:24:09 <•catlee> and the allocator will re-populate with enabled slaves

one thing I'm not sure about is when we modify |v1/builders/* v1/machines/* v1/allocated/all| and when we just let the allocator handle things by modifying config.json

	Jordan Lund (:jlund) Reporter
	Comment 3 • 10 years ago

we may have flipped the usw2 tunnel already. It may or may not have been on purpose. This bug might not be necessary now. will confirm in irc/here. stand by.

	Chris Cooper [:coop] (he/him)
	Comment 4 • 10 years ago

AFAIK, we never re-enabled these usw2 slaves last week after the trouble had passed. Given pending job numbers on tst-linux64-spot, we should make sure these are available for jobs again.

	Justin Wood (:Callek)
	Comment 5 • 10 years ago

(In reply to Chris Cooper [:coop] from comment #4)
> AFAIK, we never re-enabled these usw2 slaves last week after the trouble had
> passed. Given pending job numbers on tst-linux64-spot, we should make sure
> these are available for jobs again.

Fully re-enabled per https://hg.mozilla.org/build/cloud-tools/rev/e47bf4c569dd and RESO/WONTFIX since the discussion here seems to have made this specific use-case of disable usw2 moot.