Jordan Lund (:jlund) Assignee
	Description • 9 years ago

there are a few intermittents windows jobs that could have many root causes however there are a few common slaves between them all that are appearing more than others.

namely:
b-2008-ix-0003
b-2008-ix-0007
b-2008-ix-0012
b-2008-ix-0013
b-2008-ix-0014

releng + relops should investigate these slaves and rule out 1) if there is something individually different about them and then 2) if our windows pool as a whole isn't meeting requirements.

	Phil Ringnalda (:philor)
	Comment 1 • 9 years ago

I rage-disabled 7 and 12/13/14 this morning, though conveniently for me I hadn't yet gotten around to filing an "investigate" bug for them :)

	Jordan Lund (:jlund) Assignee
	Comment 2 • 9 years ago

see:

https://bugzilla.mozilla.org/show_bug.cgi?id=1110236#c17

https://bugzilla.mozilla.org/show_bug.cgi?id=916765#c193

https://bugzilla.mozilla.org/show_bug.cgi?id=1115490#c40

	Ryan VanderMeulen [:RyanVM]
	Comment 3 • 9 years ago

Oh hey, same slaves in bug 1093664 too it appears.

	Phil Ringnalda (:philor)
	Comment 4 • 9 years ago

b-2008-ix-0003 offended me too, so I plucked it out as well.

	Jordan Lund (:jlund) Assignee
	Comment 5 • 9 years ago

bugs 1115490, 1110236, and 1093664 all hinted at RAM issues in their own unique way.

after poking the handful of bad offenders from this bug and comparing to a sample size of good slaves, I think I have a good idea of what's going on:

'known good slaves':
  cltbld@B-2008-IX-0166 ~
  cltbld@B-2008-IX-0106 ~
  cltbld@B-2008-IX-0115 ~
     Total Physical Memory:     8,183 MB

'all slaves from this bug':
  cltbld@B-2008-IX-0014 ~
  cltbld@B-2008-IX-0013 ~
  cltbld@B-2008-IX-0001 ~
  cltbld@B-2008-IX-0003 ~
  cltbld@B-2008-IX-0007 ~
  cltbld@B-2008-IX-0012 ~
    Total Physical Memory:     4,087 MB

seems like not all our slaves have the same amount of RAM and 4gb is just not good enough. next steps are figuring out which slaves have 4gb and which have 8gb. I think bumping RAM of these B-2008-IX machines will be hard to get budget/time approval for, particularly with our Q1 efforts of moving win builders to the cloud. If there are only a few 4gb slaves, I would like to propose the idea of disabling the bad ones and waiting for results of aws efforts.

I'll follow up with getting numbers for which slaves only have 4gb

	Jordan Lund (:jlund) Assignee
	Comment 6 • 9 years ago

just went through the b-2008 machines. automating with windows + ssh wasn't behaving so it ended up taking me a while to hack a tmux script together.

good news is there isn't many machines (12) with only 4gb of RAM and sheriffs pretty much discovered them all for this bug already.

I've made sure that all the 4gb ones are disabled for now. I'll follow up with a bug to request a RAM bump for the 12 4gb ones.

ftr here is the full list (surprise surprise, all the 4gb were the first numbered slaves which makes me think they all came from a different pool together):

> cat windows_ram_4gb
B-2008-IX-0001 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0002 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0003 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0004 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0005 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0007 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0007 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0009 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0010 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0011 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0012 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0013 ~ Total Physical Memory:     4,087 MB
B-2008-IX-0014 ~ Total Physical Memory:     4,087 MB



> cat windows_ram_8gb
B-2008-IX-0008 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0015 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0016 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0017 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0065 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0066 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0067 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0068 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0069 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0070 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0071 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0072 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0073 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0074 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0075 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0075 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0076 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0077 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0078 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0079 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0080 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0082 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0083 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0084 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0085 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0086 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0087 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0088 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0090 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0091 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0092 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0093 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0094 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0095 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0096 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0097 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0098 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0099 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0100 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0101 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0102 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0103 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0104 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0105 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0106 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0107 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0108 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0109 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0110 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0111 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0112 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0113 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0114 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0115 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0116 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0117 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0118 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0119 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0120 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0121 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0122 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0123 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0124 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0125 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0126 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0127 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0128 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0129 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0130 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0131 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0132 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0133 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0134 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0135 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0136 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0137 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0138 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0139 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0140 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0141 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0142 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0143 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0145 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0146 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0147 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0148 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0149 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0150 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0152 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0153 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0154 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0155 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0156 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0157 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0158 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0161 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0162 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0163 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0164 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0165 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0166 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0167 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0168 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0169 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0170 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0171 ~ Total Physical Memory:     8,183 MB
B-2008-IX-0172 ~ Total Physical Memory:     8,183 MB

	Jordan Lund (:jlund) Assignee
	Comment 7 • 9 years ago

unlucky 13

	Jordan Lund (:jlund) Assignee
	Comment 8 • 9 years ago

All dep slaves have 8gb now. It is getting late in my day. I'll enable them in the morning so I can keep an eye on them.

	Jordan Lund (:jlund) Assignee
	Comment 9 • 9 years ago

there were some slaves that didn't end up getting RAM upgrades. They should all have 8gb now though. Going to resolve this tracker as I think the thing that linked all the blocking bugs was around memory constraints.