Andreas Tolfsen ❲:ato❳ Assignee
	Description • 7 years ago

Adding the geckodriver Rust unit tests to the try build in
https://bugzil.la/1395933 caused a spike in the intermittency rate
for https://bugzil.la/1420472.  Until this has been investigated I
will disable the Rust tests on Windows.

This will not fix the intermittent but will reduce it.  The
intermittent will again become a problem, I suspect, when someone
adds new Rust unit tests.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 2 • 7 years ago

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=074156f0ab9daf4856a22eda175b0b384f3ebddf

	Pulsebot
	Comment 3 • 7 years ago

Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/2abe464e8452
Disable geckodriver Rust tests on Windows. r=me a=Aryx

	Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)
	Comment 4 • 7 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/2abe464e8452

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 5 • 6 years ago

I don’t understand why this bug was re-opened.  Is there any further
work I need to do here?

	Henrik Skupin [:whimboo][⌚️UTC+1]
	Comment 6 • 6 years ago

We temporarily disabled the geckodriver unit tests on Windows due to the issue on bug 1420472. As such we still have to track the state, and re-enable the tests whenever this dependency gets fixed. This is how we handle such situations usually,

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 7 • 6 years ago

OK let me unassign myself so it doesn’t turn up in my work list then.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 8 • 6 years ago

try run to see if the situation has improved:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=30fa2d4facfa1df959a7f4b257193f008da8df06

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 9 • 6 years ago

Apparently Rust tests are in a job called BR now, which is tier-2
on Linux.  I can’t find a BR job at all for Windows anymore.
Do we not want to run Rust tests on Windows?

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 10 • 6 years ago

Aryx messaged my directly and said it runs under Windows _2012_
tier-2, not Windows 10 where I was looking.  I guess it makes sense
that they run on the builders.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 11 • 6 years ago

I will do another try run.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 12 • 6 years ago

Enabling geckodriver et al. tests on Windows makes the BR job fail,
but in a different way than before:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dcbfa82f80b6f4412afc8f429571cb972687042b

This seems to be an sccache timeout, but I’m not sure for what
reason.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 13 • 6 years ago

It looks like the webdriver and geckodriver unit tests are passing
(174 passed), but that sccache times out trying to compile serde:

> 8:07:20     INFO - test result: ok. 174 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
> 18:07:20     INFO - error: failed to execute compile
> 18:07:20     INFO - caused by: error reading compile response from server
> 18:07:20     INFO - caused by: Failed to read response header
> 18:07:20     INFO - caused by: An existing connection was forcibly closed by the remote host. (os error 10054)
> 18:07:20     INFO - mozmake.EXE[2]: Leaving directory 'z:/build/build/src/obj-firefox/testing/geckodriver'
> 18:07:20     INFO - error: Could not compile `serde`.
> 18:07:20     INFO - Caused by:
> 18:07:20     INFO -   process didn't exit successfully: `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name serde 'third_party\rust\serde\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="rc"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=3b988fac4a9d3e60 -C extra-filename=-3b988fac4a9d3e60 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' --target i686-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'serde_derive=z:/build/build/src/obj-firefox/toolkit/library\release\deps\serde_derive-2271e9788f4dcf51.dll' --cap-lints warn -C opt-level=2 -C debuginfo=2 --cfg de_boxed_c_str --cfg de_rc_dst --cfg core_duration --cfg integer128 --cfg num_nonzero` (exit code: 2)

It’s not clear to me why serde is not already built by this point.
I would’ve thought serde is being _used_ by tests such as
response::tests::test_json_window_rect_response?

	Henrik Skupin [:whimboo][⌚️UTC+1]
	Comment 14 • 6 years ago

Here some output from a job:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dcbfa82f80b6f4412afc8f429571cb972687042b&selectedJob=201764740

Serde gets compiled for toolkit first:

> 20:19:18     INFO -     Compiling serde v1.0.66
> 20:19:18     INFO -       Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name build_script_build 'third_party\rust\serde\build.rs' --crate-type bin --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="rc"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=a833dfff5d9941a9 -C extra-filename=-a833dfff5d9941a9 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\release\build\serde-a833dfff5d9941a9' -C 'linker=z:/build/build/src/obj-firefox/build/win64/cargo-linker.bat' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --cap-lints warn`

Then we compile it again for geckodriver:

> 20:20:14     INFO -     Compiling serde v1.0.66
> 20:20:14     INFO -       Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name build_script_build 'third_party\rust\serde\build.rs' --crate-type bin --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=1c947763419dfe71 -C extra-filename=-1c947763419dfe71 --out-dir 'z:\build\build\src\obj-firefox\testing\geckodriver\.\release\build\serde-1c947763419dfe71' -C 'linker=z:/build/build/src/obj-firefox

And once more but with msvc for geckodriver:

> 20:31:57     INFO -    Compiling serde v1.0.66
> 20:31:57     INFO -      Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name serde 'third_party\rust\serde\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=7a39c7d00625e267 -C extra-filename=-7a39c7d00625e267 --out-dir 'z:\build\build\src\obj-firefox\testing\geckodriver\.\i686-pc-windows-msvc\release\deps' --target i686-pc-windows-msvc -L 'dependency=z:\build\build\src\obj-firefox\testing\geckodriver\.\i686-pc-windows-msvc\release\deps' -L 'dependency=z:\build\build\src\obj-firefox\testing\geckodriver\.\release\deps' --cap-

... and for toolkit:

> 20:32:34     INFO -    Compiling serde v1.0.66
> 20:32:34     INFO -      Running `z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name serde 'third_party\rust\serde\src\lib.rs' --crate-type lib --emit=dep-info,link -C opt-level=2 -C codegen-units=1 --cfg 'feature="default"' --cfg 'feature="rc"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=3b988fac4a9d3e60 -C extra-filename=-3b988fac4a9d3e60 --out-dir 'z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' --target i686-pc-windows-msvc -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox/toolkit/library\release\deps' --extern 'serde_derive=z:/build/build/src/obj-firefox/toolkit/library\release\deps\serde_derive-2271e9788f4dcf51.dll' --cap-lints warn -C opt-level=2 -C debuginfo=2 --cfg de_boxed_c_str --cfg de_rc_dst --cfg core_duration --cfg integer128 --cfg num_nonzero`

I assume multiple processes are used for this job and as such we have an intersection. While serde is correctly compiled for geckodriver and our tests run successfully, it fails for the toolkit component. Not that the order we compile is different. But I doubt that this has any impact here.

	(not currently active) Ted Mielczarek
	Comment 15 • 6 years ago

I have a patch in bug 1495798 that I'm testing on try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c664672e729a7d9b7ea7c67491e7fd74eb372cb3

https://hg.mozilla.org/try/rev/a2c8beece69a5c508325c4cdd63df98b2595b0b2

Could you push your patch to try atop my patch and see what happens? If it doesn't fix the issue then it might at least give us better diagnostics. Also it sounds like bug 1420472 is no longer involved?

	Henrik Skupin [:whimboo][⌚️UTC+1]
	Comment 16 • 6 years ago

Andreas, the patch from Ted landed. Can you please try again to enable the geckodriver rust tests on Windows?

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 17 • 6 years ago

That’s a really good idea:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=caa862c1c4e3429074082acb8775b07503d35cb3

	(not currently active) Ted Mielczarek
	Comment 18 • 6 years ago

I can't guarantee that my patches will have fixed your issue, but hopefully if it still fails we can get a clearer picture of what's wrong.

	(not currently active) Ted Mielczarek
	Comment 19 • 6 years ago

Also: if this is fixed it's very likely that bug 1396845 was in fact what fixed it.

	Andreas Tolfsen ❲:ato❳ Assignee
	Comment 21 • 6 years ago

Thanks ted!  Let’s get these tests running.

	Pulsebot
	Comment 22 • 6 years ago

Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/04459ddd517a
geckodriver: re-enable rust tests on windows; r=whimboo

	Bogdan Tara[:bogdan_tara | bogdant]
	Comment 23 • 6 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/04459ddd517a