Ryan VanderMeulen [:RyanVM] Reporter
	Description • 11 years ago

Trunk trees are currently seeing a 1+ hour backlog on AWS-based builds. All trees except Try closed at 07:11 PT.

	John Hopkins (:jhopkins)
	Comment 1 • 11 years ago

When trying to manually start bld-linux64-ec2-013 and bld-linux64-ec2-056 via the EC2 console I got:

"Error starting instances
We currently do not have sufficient m3.xlarge capacity in the Availability Zone you requested (us-east-1c). Our system will be working on provisioning additional capacity. You can currently get m3.xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-1d, us-east-1a."

	Ryan VanderMeulen [:RyanVM] Reporter
	Comment 2 • 11 years ago

So basically my understanding is that this is "AWS is under heavy load during the holiday shopping season" and there's little we can do about this on our end besides closing if things get too backlogged until they catch back up.

Trees reopened at 08:40 PT.

	Rail Aliiev [:rail]
	Comment 3 • 11 years ago

FTR, when we hit AWS limits we still try to start other instances (which can be on other availability zones), but since it takes time to try to start all available instances in the failing zone we have some lag here.

I just checked our last watch pending and it tell me that we started enough bld-linux64 instances to cover all pending jobs.