Justin Wood (:Callek) Reporter
	Description • 10 years ago

This is having a bunch of problems translatable to the attached pandas, lets run diags first.

	Salvador Espinoza [:sal]
	Comment 1 • 10 years ago

running diagnostics

	Van Le [:van]
	Comment 2 • 10 years ago

memtest found 0 errors after 13 passes, running Western Digital's full media scan on hdd.

	Salvador Espinoza [:sal]
	Comment 3 • 10 years ago

passed hd diags, reimaging

	Salvador Espinoza [:sal]
	Comment 4 • 10 years ago

host back up.

sals-MacBook-Pro-3:~ sal$ sudo fping  10.26.19.126
10.26.19.126 is alive
sals-MacBook-Pro-3:~ sal$ sudo fping   10.26.131.21
10.26.131.21 is alive
sals-MacBook-Pro-3:~ sal$ ssh !$
ssh 10.26.131.21
The authenticity of host '10.26.131.21 (10.26.131.21)' can't be established.
RSA key fingerprint is 0f:fb:74:e6:23:32:2b:30:ca:e6:4c:b2:f7:97:f5:26.
Are you sure you want to continue connecting (yes/no)?

	Amy Rich [:arr] [:arich]
	Comment 5 • 10 years ago

It's still showing load issues. Do we have more invasive disk diags we can run (or other diags)?

	Amy Rich [:arr] [:arich]
	Comment 6 • 10 years ago

Oh, heck yeah, we definitely have drive issues. There are multiple:

Sep 30 18:27:53 foopy64 kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 30 18:27:53 foopy64 kernel: ata1.00: irq_stat 0x40000001
Sep 30 18:27:53 foopy64 kernel: ata1.00: failed command: FLUSH CACHE EXT
Sep 30 18:27:53 foopy64 kernel: ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Sep 30 18:27:53 foopy64 kernel:         res 51/04:00:38:df:f7/00:00:00:00:00/a7 Emask 0x1 (device error)
Sep 30 18:27:53 foopy64 kernel: ata1.00: status: { DRDY ERR }
Sep 30 18:27:53 foopy64 kernel: ata1.00: error: { ABRT }
Sep 30 18:27:53 foopy64 kernel: ata1.00: configured for UDMA/33
Sep 30 18:27:53 foopy64 kernel: ata1: EH complete

Please contact iX for a drive replacement.

	Amy Rich [:arr] [:arich]
	Comment 7 • 10 years ago

It may also be the controller, too, come to think of it. They'll probably ask for some diagnostic data we can pass onto them to help them diagnose the issue.

	Vinh Hua [:vinh]
	Comment 8 • 10 years ago

iX Systems RMA Case ID #AMA-717-88012

	Vinh Hua [:vinh]
	Comment 9 • 10 years ago

I've ran HDD diags but no errors were detected.  While we wait for iX to respond, I'll reimage the host.

	Amy Rich [:arr] [:arich]
	Comment 10 • 10 years ago

reimaging won't help, we've already done that. There's is most definitely issues with the disk or controller if you look at the system logs.

	Vinh Hua [:vinh]
	Comment 11 • 10 years ago

The host has been dropped off to iX for 48 hrs burn-in diags.

	Vinh Hua [:vinh]
	Comment 12 • 10 years ago

Feedback from iX support:

The node itself has passed our burn-in test. Have you run any additional tests on the drives associated with it?

	Vinh Hua [:vinh]
	Comment 14 • 10 years ago

Node came back from iX with no errors detected.  iX has loaned me a temp HDD to replace.  Upon reimaging, I get the following error; see attachment.

	Justin Wood (:Callek) Reporter
	Comment 16 • 10 years ago

fwiw, we're again getting:

[19:48:59]	nagios-releng	Tue 16:49:02 PDT [4219] foopy64.p4.releng.scl3.mozilla.com:load is CRITICAL: (Return code of 255 is out of bounds) (http://m.mozilla.org/load)

	Jake Watkins [:dividehex]
	Comment 17 • 10 years ago

(In reply to Vinh Hua [:vinh] from comment #15)
> Created attachment 8501434 [details]
> foopy64.JPG

Vinh:  according to dustin and lsscsi, foopies shouldn't have raid.  I think the disk needs to be zero'd to wipe the metadata that is hanging up anaconda

	Vinh Hua [:vinh]
	Comment 18 • 10 years ago

Host is online now with a loaner HDD. Let me know if issues persist.

	Vinh Hua [:vinh]
	Comment 19 • 10 years ago

:callek - How's foopy64 holding up after the HDD replacement?

	Justin Wood (:Callek) Reporter
	Comment 20 • 10 years ago

I'm not sure if its been put back into rotation, passing the question off to coop.

	Chris Cooper [:coop] (he/him)
	Comment 21 • 10 years ago

It's back in rotation, and seems to be working correctly thus far.

I'll reopen if anything goes wrong.

	Vinh Hua [:vinh]
	Comment 22 • 10 years ago

:coop - The replacement hard disk came in.  Can you take foopy64 down for the replacement?

	Chris Cooper [:coop] (he/him)
	Comment 23 • 10 years ago

(In reply to Vinh Hua [:vinh] from comment #22)
> :coop - The replacement hard disk came in.  Can you take foopy64 down for
> the replacement?

I've disabled all the pandas on foopy64 and shutdown the foopy. It's all yours.

	Vinh Hua [:vinh]
	Comment 24 • 10 years ago

:coop - Foopy64 has been reimaged with the replacement drive.