Carsten Book [:Tomcat] Reporter
	Description • 9 years ago

nagios alert:

***** Nagios  *****

Notification Type: PROBLEM

Service: http file age - /buildjson/builds-4hr.js.gz
Host: builddata.pub.build.mozilla.org
Address: 63.245.215.57
State: CRITICAL

Date/Time: 01-22-2015 07:45:06

Additional Info:
HTTP CRITICAL: HTTP/1.1 200 OK - Last modified 0:10:29 ago - 8191 bytes in 0.013 second response time

	Chris Cooper [:coop] (he/him)
	Comment 1 • 9 years ago

We did an intentional buildbot db failover this morning, which cause in-progress scripts to timeout. I've cleaned up after them and we should be running again within 15min.

	Chris Cooper [:coop] (he/him)
	Comment 2 • 9 years ago

db failover is bug 1121516.

	Chris Cooper [:coop] (he/him)
	Comment 3 • 9 years ago

(In reply to Chris Cooper [:coop] from comment #2)
> db failover is bug 1121516.

Hasn't happened yet apparently, just slowdown in db replication. Sheeri flushed some tables, and builds-4hr is up-to-date again.

	Ryan VanderMeulen [:RyanVM]
	Comment 4 • 9 years ago

Trees are mostly reopened. Inbound has some new failures that need addressing before it can be reopened, though.