Phil Ringnalda (:philor) Reporter
	Description • 10 years ago

"<nagios-releng> Thu 09:27:16 PST [4498] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - Last modified 0:13:04 ago - 8191 bytes in 0.008 second response time (http://m.mozilla.org/http+file+age+-+/buildjson/builds-4hr.js.gz)"

and

"<nagios-releng> Thu 09:27:46 PST [4499] buildapi.pvt.build.mozilla.org:http - /buildapi/self-serve/jobs is CRITICAL: CRITICAL - Socket timeout after 10 seconds (http://m.mozilla.org/http+-+/buildapi/self-serve/jobs)"

All trees are closed.

	Phil Ringnalda (:philor) Reporter
	Comment 1 • 10 years ago

<gozer> [buildapi.lib.mq] [Dummy-1] Kombu connection revived
<gozer> philor: looks like its recovering, I kicked httpd on both webheads

So now we have working self-serve, and builds-pending.js/builds-running.js are okay, but builds-4hr.js.gz still isn't updating, and related-or-not, things like https://secure.pub.build.mozilla.org/slaveapi/slaves/t-snow-r4-0134 are returning 500 ISE.

	Phil Ringnalda (:philor) Reporter
	Comment 2 • 10 years ago

Progress is progressing - the problem is a try push which infinitely retried all tests very quickly, so that the last time I was able to look, it had done 81K jobs.

Nobody expected to have to handle 81K jobs on a single rev.

	Armen [:armenzg]
	Comment 3 • 10 years ago

What's left in here?
My apologies about this. I will fix the script to prevent this.

	Chris Cooper [:coop] (he/him)
	Comment 4 • 10 years ago

catlee has picked up bug 733663 for the releng-side fix.