James Graham [:jgraham] Reporter
	Description • 8 years ago

We want to run the web-platform-tests WebDriver tests against the
Firefox/GeckoDriver combination in CI (i.e. on
treeherder). GeckoDriver is a seperate Mozilla-owned binary written in
Rust. It currently lives on GitHub and makes extensive use of Cargo
for installing dependencies (at present there are 49 packages listed
in the Cargo.lock file).

Three options spring to mind:

1) Put GeckoDriver in m-c (mirrored or by moving the main repository),
and build it as part of the normal gecko build process in the same way
we do for e.g. the xpcshell or jsshell. This is the best solution from
the point of view of being able to test changes to both GeckoDriver
and gecko itself, but means finding some solution to the Cargo-in-CI
problem.

2) Put precompiled GeckoDriver binaries somewhere that CI
infrastructure can reach them, and download these as part of the test
process. This means that we can't directly test changes to GeckoDriver
using these tests, and have an ongoing maintainace hassle updating the
binaries. I'm also not sure what the precedent is for this kind of
approach.

3) Give up on using Mozilla-provided infrastructure and test
GeckoDriver on travis, or similar, with prebuilt Firefox downloaded as
part of the build. This could even be taught to report to treeherder,
although obviously not as part of the standard firefox projects. The
disadvantage of this approach is that it could not be used to test
changes to the Firefox backend, only to the GeckoDriver component.

Obviously 1) seems like the ideal solution from the point of view of
being able to test our WebDriver implementation, but it does
potentially involve solving some of the big outstanding problems for
building Rust as part of the Fx build process.

	David Burns :automatedtester
	Comment 1 • 8 years ago

1) or 2) would be my preference. Long term, building something with Rust is going to be painful if we are constantly downloading stuff from Crates (or internal Crates if we set one up).

	Andreas Tolfsen ❲:ato❳
	Comment 2 • 8 years ago

Each geckodriver binary compiled with debug symbols is ~18M on Linux
vs. ~3.5M for optimised.  For geckodriver specifically we could probably
get away with using only optimised binaries, even when testing against
debug Firefox builds.  If you multiply this up by the relevant platforms
(Windows, Mac, Linux) you will still arrive at something in excess of
10.5M in total.  Because VCSes tend not to compress/diff binary objects
very well, I’m guessing this rules out option 2.

It is an alternative to use a custom TC container with access to
_either_ releng infrastructure for downloading the binaries, or to fetch
them directly from Github in which case we’d have to make the job
Tier-2 or lower.  This has the downside that we would not be able to
test changes to geckodriver in sync with changes to Marionette and would
incur a bit of synchronisation overhead, but might quite possibly be a
realistic stopgap solution until we have support for Cargo in central.

I think geckodriver might be an ideal candidate to use for experimenting
with integrating Cargo in central: It does not ship in Firefox, i.e. it
has no linking, distribution, or platform problems; relies solely on
Rust dependencies available via Cargo; and it makes sense for it to live
inside central next Gecko/Marionette.

We do already build some Rust code in central today (with --enable-rust
flag) with limited dependencies (libstagefright has none as far as I
remember).  But I do not know the status of “offline” support in
Cargo; is it possible to drop a .cargo state/deps directory into central
and expect builds to be reproducible across platforms?  And it’s also
unclear to me what the status of this “oxidation” process is right
now.

	James Graham [:jgraham] Reporter
	Comment 3 • 8 years ago

Option 2 wasn't supposed to mean "put them in tree", just on a server where they could be fetched as part of the test job.

	Ralph Giles (:rillian)
	Comment 4 • 8 years ago

Fun to seem more code happening in rust!

I would do (2) today, and get ready to experiment with (1) when the new `cargo vendor` scripts for syncing a copy of dependencies in m-c become available. As :ato said, that's what we want, but I don't think it's quite possible yet. Hopefully very soon, but I haven't heard anything in a few weeks. I'm sure larsberg can elaborate.

If you want to pull binaries for gecko ci tests, I'd suggest uploading them to tooltool (https://github.com/mozilla/build-tooltool, self-serve credentials through login.mozilla.com). You can automate uploading builds in response to github pushes through taskcluster or travis. Ask me or dustin if you need help.

	Lars Bergstrom (:larsberg)
	Comment 5 • 8 years ago

I agree with rillian's suggestion, assuming that this is work that you primarily intend for Mozilla staff to work on. #2 is the easiest thing to do today, and #1 will preserve the current workflow, once the infrastructure bits are ready. The current status is that there's a proposal for all of the cargo vendor bits, and we've contracted with the cargo maintainer and are trying to get this designed and implemented ASAP. I unfortunately don't have a schedule in place, but will share it as soon as we have one!

All that said, I'd recommend at least considering an experiment around option #3 if you're looking to increase the volunteer contributor base for GeckoDriver. We've found in Servo and Rust that it's really easy to get new contributors when you use a GitHub+Travis PR workflow. That said, it's also *way* more work for the mozstaff - GH issues are pretty limiting, Travis/AppVeyor is never quite as nice as the hardware you can buy yourself, etc.

	Andreas Tolfsen ❲:ato❳
	Comment 6 • 7 years ago

We chose to move the geckodriver repository in to mozilla-central.
It now lives under testing/geckodriver.  WPT tests are run on
selected platforms (due to various Rust integration problems that
we are tracking separately) against the geckodriver built in
automation on try.  geckodriver is by default _not_ built locally
when building Firefox, but it can be enabled using the --enable-geckodriver
directive in mozconfig.