Michael Shal [:mshal] Assignee
	Description • 9 years ago

Once build artifacts are in s3 (bug 1100624), we'll want to have testers pull data from s3 instead of ftp. The initial target is to support trunk & try builds.

	Michael Shal [:mshal] Assignee
	Comment 1 • 9 years ago

This still needs the dependent bug to land and stick in mozharness before it can land, but it seems to work fine on try.

I've just duplicated the annoying property_conditions for now. This set will be able to replace the ones in m-c/build/gen_mach_buildprops.py. The other sets in mozharness (MakeUploadOutputParser) and buildbotcustom (parse_make_upload) can hopefully be removed as things are converted mach+mozharness and take advantage of the S3 uploading.

I think I would prefer avoiding this silly pattern matching to determine the property type though, and somehow expose that from the build system into mach_build_properties.json - if you have any thoughts on that let me know.

In the try push I've just launched a single test and talos job to verify that the files are pulled from S3 instead of ftp.m.o:
try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8c0735a40931

	Jordan Lund (:jlund)
	Comment 2 • 9 years ago

Comment on attachment 8560689 [details] [diff] [review]
s3-tests

Review of attachment 8560689 [details] [diff] [review]:
-----------------------------------------------------------------

this is awesome. I see you took my painful logic in MakeUploadOutputParser and made it better! Though I agree it is still a pain. maybe you would be interested in talking with myself and nick next week about improving that: https://bugzilla.mozilla.org/show_bug.cgi?id=1122509#c3

	Michael Shal [:mshal] Assignee
	Comment 3 • 9 years ago

(In reply to Jordan Lund (:jlund) from comment #2)
> this is awesome. I see you took my painful logic in MakeUploadOutputParser
> and made it better! Though I agree it is still a pain. maybe you would be
> interested in talking with myself and nick next week about improving that:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1122509#c3

Definitely - let me know when you guys are meeting and I'll join in.

	Michael Shal [:mshal] Assignee
	Comment 4 • 9 years ago

:jonasfj recommended waiting until the TC queue rewrite is complete (bug 1121293) before starting to use the files from s3 for testers & other things.

	Michael Shal [:mshal] Assignee
	Comment 5 • 9 years ago

We'll also want to spruce up the python client error handling before switching testers over. I hit an issue where a tests.zip file wasn't uploaded correctly. We suspect that the upload died part way through, and then retries failed since the file-handle wasn't reset (bug 1133876). Even if a retry did succeed, it looks like it would have retried several times unnecessarily (bug 1133856).

	Michael Shal [:mshal] Assignee
	Comment 6 • 9 years ago

Comment on attachment 8560689 [details] [diff] [review]
s3-tests

https://hg.mozilla.org/build/mozharness/rev/32f80f4d499e

	Michael Shal [:mshal] Assignee
	Comment 8 • 9 years ago

bump mozharness rev, this time with a production revision.

Note I haven't tried this one on try, but the previous default rev looks good: https://treeherder.mozilla.org/#/jobs?repo=try&revision=53caf322caad

	Jordan Lund (:jlund)
	Comment 9 • 9 years ago

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

	Jordan Lund (:jlund)
	Comment 10 • 9 years ago

Comment on attachment 8581914 [details] [diff] [review]
0001-Bug-1117960-bump-mozharness.json-to-rev-93f17b170f43.patch

Review of attachment 8581914 [details] [diff] [review]:
-----------------------------------------------------------------

thanks for changing this to a prod rev :)

	Michael Shal [:mshal] Assignee
	Comment 11 • 9 years ago

bump mozharness rev: https://hg.mozilla.org/integration/mozilla-inbound/rev/50d681158e9f

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 12 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/50d681158e9f