Chrome equivalent bug https://bugs.chromium.org/p/chromium/issues/detail?id=934984

Backed out 2 changesets (bug 1335740) for devtools failures. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=230728752&repo=autoland&lineNumber=16406

Push where it started:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=1bddabb7bafb168ba9c64b755ee4f2844f0ac9cc

Jan-Ivar, landing this bug caused bug 1500977 to permafail tier2:

https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&searchStr=android%2C8.0%2Cpixel2%2Cdebug%2Cmochitests%2Ctest-android-hw-p2-8-0-arm7-api-16%2Fdebug-mochitest-media-2%2Cm%28mda2%29&fromchange=8353a7e41bca13c5734623dbbc0f1c6da98ffdc2&selectedJob=234896321

I wanted to backout but it's tier2 so leaving the ni here. Adding Noemi also since she'll be on the day shift in case the backout is needed.

From those logs I suspect webaudio tests acting up.

This patch only modifies test_mediaStreamAudioSourceNodeNoGC, to run with science=https, but oddly that test does not appear to run anymore?? The test above it and the test below it are shown as run, but no sign of this one.

That was the only one I found using getUserMedia. Paul, do you know if any of the other tests indirectly rely on getUserMedia, or do know what might be happening here? Android only.

I have no idea what is happening here. Maybe it's set to run after, grouping the tests by protocol? Reading what the test harness is doing is probably the best course of action here.

I don't know of other Web Audio API tests that are using getUserMedia, I usually put getUserMedia tests in dom/media/tests/mochitest.

https://hg.mozilla.org/mozilla-central/rev/35abcd7c962a
https://hg.mozilla.org/mozilla-central/rev/7beefe9e4d81

This was not yet fixed

Backed out for permafailing mda tasks on Android 8.0:

https://hg.mozilla.org/integration/autoland/rev/a6c71c35b055432f4b24a673d41c11e2f917f0df

See bug 1537745 for more information.

https://hg.mozilla.org/mozilla-central/rev/a6c71c35b055

https://hg.mozilla.org/mozilla-central/rev/092aceb2b8dc
https://hg.mozilla.org/mozilla-central/rev/a91128c447b5

:bc, this was backed out for a perma fail on android as it was trying to test https instead of http for certain tests. The browser gets invoked like so:
am start -W -n org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp -a android.intent.action.VIEW --es env9 MOZ_CRASHREPORTER_NO_REPORT=1 --es env8 MOZ_UPLOAD_DIR=/sdcard/tests/mozlog --es args "-no-remote -profile /sdcard/tests/profile//" --es env3 DISABLE_UNSAFE_CPOW_WARNINGS=1 --es env2 R_LOG_VERBOSE=1 --es env1 XPCOM_DEBUG_BREAK=stack --es env0 MOZ_CRASHREPORTER=1 --es env7 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 MOZ_IN_AUTOMATION=1 --es env4 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env11 MOZ_HIDE_RESULTS_TABLE=1 --es env10 R_LOG_LEVEL=6 -d "https://example.com:443/tests?autorun=1&closeWhenDone=1&logFile=%2Fsdcard%2Ftests%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fsdcard%2Ftests"

and hangs with a 370 second timeout:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=mochitest-media%2Candroid&tochange=a6c71c35b055432f4b24a673d41c11e2f917f0df&fromchange=04fb1566c7c1201b649231e80b7c293d1ed0b543

do we have prior art of tests running on android with https? looking at a screenshot from a failure:
https://taskcluster-artifacts.net/HnpZZtNjS4-0g0f6Maqgew/0/public/test_info/mozilla-test-fail-screenshot_XbpIUx.png

it seems as though the proxy server is refusing the connection- is this an issue because we are running at bitbar and the networking could be different as we actually go over the network instead of 100% localhost?

We only use adb over tcp/ip when running the battery tests which isn't the case here. I don't recall any experience with https though gbrown has had some recent experience with ssl not being built into the python installation being used but was with python 3.7 on macos iirc. gbrown?

Perhaps our installation of python on the Bitbar containers is lacking ssl support.

I just realized that we use tooltool to download things and those are from https sites. If we can do that, wouldn't it mean we have sufficient python ssl support?

I've verified that the container's Python has SSL abilities and it does seem to be working.

root@dfb2c9022bff:~# python -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.0.2g 1 Mar 2016

$ python

import requests
r = requests.get("https://google.com")
r
<Response [200]>

maybe :bc or :gbrown could try to apply a patch locally and see what is going on.

given the proxy issues, I believe proxy is ssltunnel, do we know if this is setup to run non localhost and will work over adb?

(In reply to Bob Clary [:bc:] from comment #31)

We only use adb over tcp/ip when running the battery tests which isn't the case here. I don't recall any experience with https though gbrown has had some recent experience with ssl not being built into the python installation being used but was with python 3.7 on macos iirc. gbrown?

Right, that's bug 1534647 -- probably not much in common with this issue.

android-em runs many https tests, such as:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237644992&repo=mozilla-central&lineNumber=2037

[task 2019-04-02T18:04:37.563Z] 18:04:37     INFO -  runtests.py | Running with scheme: https
[task 2019-04-02T18:04:37.564Z] 18:04:37     INFO -  runtests.py | Running with e10s: False
[task 2019-04-02T18:04:37.564Z] 18:04:37     INFO -  runtests.py | Running with serviceworker_e10s: False
[task 2019-04-02T18:04:37.565Z] 18:04:37     INFO -  runtests.py | Running with socketprocess_e10s: False
[task 2019-04-02T18:04:37.565Z] 18:04:37     INFO -  runtests.py | Running tests: start.
[task 2019-04-02T18:04:37.877Z] 18:04:37     INFO -  remoteautomation.py | runApp deleted /sdcard/tests/logs/mochitest.log
[task 2019-04-02T18:04:38.187Z] 18:04:38     INFO -  adb launch_application: am start -W -n org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp -a android.intent.action.VIEW --es env9 MOZ_CRASHREPORTER_NO_REPORT=1 --es env8 MOZ_UPLOAD_DIR=/sdcard/tests/mozlog --es args "-no-remote -profile /sdcard/tests/profile//" --es env3 DISABLE_UNSAFE_CPOW_WARNINGS=1 --es env2 R_LOG_VERBOSE=1 --es env1 XPCOM_DEBUG_BREAK=stack --es env0 MOZ_CRASHREPORTER=1 --es env7 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 MOZ_IN_AUTOMATION=1 --es env4 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env11 MOZ_HIDE_RESULTS_TABLE=1 --es env10 R_LOG_LEVEL=6 -d "https://example.com:443/tests?autorun=1&closeWhenDone=1&logFile=%2Fsdcard%2Ftests%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fsdcard%2Ftests"
[task 2019-04-02T18:04:47.118Z] 18:04:47     INFO -  remoteautomation.py | Application pid: 2280
[task 2019-04-02T18:05:42.929Z] 18:05:42     INFO -  302 INFO SimpleTest START
[task 2019-04-02T18:05:42.929Z] 18:05:42     INFO -  303 INFO TEST-START | dom/cache/test/mochitest/test_cache_orphaned_body.html
[task 2019-04-02T18:05:53.145Z] 18:05:53     INFO -  304 INFO TEST-OK | dom/cache/test/mochitest/test_cache_orphaned_body.html | took 11281ms

so android works in general- does emulator use localhost for the proxy or a "remote" ip address on the host? I believe the actual phones use a remote ip address which is much different.

On android-em, runtestsremote.py is called with --remote-webserver=10.0.2.2 (the emulator's view of the host localhost) and --http-port=8854 --ssl-port=4454. For https tests, fennec is called with '-d "https://example.com:443/tests?autorun=1...'

I don't see a significant difference for android-hw.

ok, I am convinced this should be working; maybe there is an issue on the hardware.

While attempting to stress test my local p2 that was flashed to Android 9 and rooted with the latest magisk, I found some mochitests which failed due to proxy errors. I got distracted and haven't gotten back to figuring that out. I'll look again with the perspective of this failure and see if there is anything I can learn.

Cosmin, the push in comment 28 is just the web-platform-test modifications merged upstream coming back to us. The rest of the patch hasn't landed. Re-opening.

I'm still blocked from landing this over what looks like Android infra issues. Looking for solutions.

When the proxy is not at 127.0.0.1, the PAC based proxy works for Android for scheme http but not for https. I don't know why. This is not a new problem and means that we don't have ssl based testing for Android.

The PAC script for Android only differs from the Desktop by using the ip address of the host instead of 127.0.0.1. I tried forcing the forward in the ssltunnel config to be the host's ip address instead of 127.0.0.1 but that didn't help.

It may be that the PAC script doesn't work anywhere if the address of the proxy isn't 127.0.0.1.

I successfully worked around the issue by setting the remote web server to 127.0.0.1 which sets the proxy to use 127.0.0.1 then used adb reverse to tell the device to forward requests to the proxy on the device's 127.0.0.1 to the host. adb reverse is only available for Android 5 and later so the Android 4.3 emulators are out of luck. The current work around is only for mochitests. We would need to do something similar for reftests if we wished to use the https scheme there.

I did a try run with patches from this bug

https://hg.mozilla.org/try/rev/d5a2469392d0e26cb8690e0b878fe45e37a1d723
https://hg.mozilla.org/try/rev/07c565fb7f6d0aadfe6d2e6565dffccfd5da5d8e

along with this kludge

https://hg.mozilla.org/try/rev/025af1b16cf1f81f65b03b2607b3fcbf167101c8

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=b94e3cfb14aa29335ebcfe22cbd7e055456dfcbb

Overall it worked though there is one failure due to the kludge not being complete:

TEST-UNEXPECTED-FAIL | netwerk/test/mochitests/test_loadinfo_redirectchain.html | remote address should match - got "127.0.0.1", expected "10.0.2.2"

There are additional failures not related to the kludge or the ssl proxy which must be fixed before this lands again.

For comparison I did a try run with the patches from this bug but without the proxy hack:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=8b05e3957987990f83246327ca6091d3ec02c2e1

Of course the real fix would be to get the PAC settings to work for Android and SSL. snorp, mayhemer: Any thoughts?

I don't know why PAC wouldn't be working. I'm sure mayhemer will be able to enlighten us :)

One sentence is not clear to me: "the PAC based proxy works for Android for scheme http but not for https."

Did you mean that when a PAC is served (via http: url) and you visit/request an https: url we 1) don't use the proxy, 2) we fail to connect the proxy, 3) we fail to reach the end server?

Or, did you mean that when the PAC script is served via an https: url the proxy is not used? Or what?

Please clarify.

Note that PAC must be served via http: URL - yeah, not good, I know. We couldn't do the potential ocsp request for the certificate then - it would be an indefinite loop.

comment 44

In automation, the pac is set as a data uri.

For Firefox desktop the pac script contains

var proxyForScheme = { 'http': 'PROXY 127.0.0.1:8888', 'https': 'PROXY 127.0.0.1:4443', 'ws': 'PROXY 127.0.0.1:4443', 'wss': 'PROXY 127.0.0.1:4443'};

The web server and the ssltunnel run on the same host as Firefox. Thhs works fine for both requesting http and https pages.

For Android the pac script contains

var proxyForScheme = { 'http': 'PROXY 192.168.1.7:8888', 'https': 'PROXY 192.168.1.7:4443', 'ws': 'PROXY 192.168.1.7:4443', 'wss': 'PROXY 192.168.1.7:4443'};

where in this case the host where the web server and ssltunnel are running is 192.168.1.7.

This works for the device requesting http sites but not https. When a request for an https page is made, the browser responds that the proxy is refusing connections.

If I change the android pac script to use 127.0.0.1 instead of the host's ip address and then tell the device to forward requests to the proxied ports to the host, it will succeed in making requests to both http and https pages.

I'll attach the default prefs.js file used for Android which uses the ip address of my machine.

First thing I would check is if the context of the application is able to make requests to port 4443 (could be in some kind of a system protected range). Try to change it to 8889, perhaps? If that doesn't change anything then I would check the ssltunnel actually listens on 4443. It may happen it fails to. Have you tried to use direct connections (change the proxy config) and go to https://192.168.1.7:4443 directly? that should do something (not sure what exactly the outcome is going to be) but reject the connection. Then, check that the ssltunnel is correctly configured to reach the end server on 8888. It might be that we fail there - it tries to reach port 8888, but fails to and just closes the connection with a reset and we see it as proxy connection rejection. ssltunnel can log too: https://searchfox.org/mozilla-central/rev/8d78f219702286c873860f39f9ed78bad1a6d062/testing/mochitest/ssltunnel/ssltunnel.cpp#44

How complicated is to get a log from the device? https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging. If easy, add proxy:5, just in case. It may reveal something more complicated or just confirm we try to reach the port but get an immediate reset - which would mean there is no one listening.

with:
SSLTUNNEL_LOG_LEVEL=0
MOZ_LOG=proxy:5
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=5090712e68322e9ca7201786e68a97680c6ced8f

It looks like the android-em emulators actually contact the ssltunnel since SSLTUNNEL messages appear in the log. It is not clear to me that usermedia is actually enabled though since there are : TypeError: navigator.mediaDevices is undefined errors.

The android-hw devices do not contain SSLTUNNEL messages.

The android-hw devices also contain

EDITED TO ADD CORRECT MESSAGE

I/Gecko ( 6749): [(null) 6749: Main Thread]: D/proxy DisableProxy http 10.7.205.214:4454 1800

Bitbar says that all ports are open and not blocked by a firewall. I disabled my local firewall and still reproduced the issue.

Locally, Fennec can reach <myip>:8888 but not <myip>:4443.

the ssltunnel config file used locally is

$ cat /tmp/ssltunnelUXY_Cu.cfg
httpproxy:1
certdbdir:/home/bclary/mozilla/builds/inbound/mozilla/build/pgo/certs
forward:127.0.0.1:8888
websocketserver:192.168.1.7:9988
listen:*:4443:pgoserver
listen:self-signed.example.com:443:4443:selfsigned
listen:untrusted.example.com:443:4443:untrusted
listen:expired.example.com:443:4443:expired
clientauth:requestclientcert.example.com:443:4443:request
clientauth:requireclientcert.example.com:443:4443:require
listen:mismatch.expired.example.com:443:4443:expired
listen:mismatch.untrusted.example.com:443:4443:untrusted
listen:untrusted-expired.example.com:443:4443:untrustedandexpired
listen:mismatch.untrusted-expired.example.com:443:4443:untrustedandexpired
listen:no-subject-alt-name.example.com:443:4443:noSubjectAltName
listen:bug413909.xn--hxajbheg2az3al.xn--jxalpdlp:443:4443:bug413909cert
listen:www.bank1.com:443:4443:escapeattack1
redirhost:redirproxy.example.com:443:4443:test1.example.com
listen:include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningGood
listen:bad.include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningBad
listen:badchain.include-subdomains.pinning.example.com:443:4443:staticPinningBad
failHandshake:fail-handshake.example.com:443:4443
listen:sha1ee.example.com:443:4443:sha1_end_entity
listen:sha256ee.example.com:443:4443:sha256_end_entity
listen:imminently-distrusted.example.com:443:4443:imminently_distrusted
ssl3:ssl3.example.com:443:4443
rc4:rc4.example.com:443:4443
ssl3:ssl3rc4.example.com:443:4443
rc4:ssl3rc4.example.com:443:4443
tls1:tls1.example.com:443:4443

Locally, ssltunnel is running on my host as a linux x86_64 executable from host utils. My local fennec build does have a ssltunnel arm binary. Should we be running it on the device instead of the host?

General request: please be more specific in details.

(In reply to Bob Clary [:bc:] from comment #49)

with:
SSLTUNNEL_LOG_LEVEL=0
MOZ_LOG=proxy:5
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=5090712e68322e9ca7201786e68a97680c6ced8f

It looks like the android-em emulators actually contact the ssltunnel since SSLTUNNEL messages appear in the log.

Where is ssltunnel actually running in this situation? What "contact" actually means?

It is not clear to me that usermedia is actually enabled though since there are : TypeError: navigator.mediaDevices is undefined errors.

No idea how this is related.

The android-hw devices do not contain SSLTUNNEL messages.

What exactly does this mean?

The android-hw devices also contain

EDITED TO ADD CORRECT MESSAGE

I/Gecko ( 6749): [(null) 6749: Main Thread]: D/proxy DisableProxy http 10.7.205.214:4454 1800

It means we try to contact a proxy at 10.7.205.214:4454 and fail to. We then blacklist this proxy and try another one, if available. If all configured proxies are failing, we reset the disabled list and start over on next request.

Bitbar says that all ports are open and not blocked by a firewall. I disabled my local firewall and still reproduced the issue.

Bitbar is what? and running where?

Locally, Fennec can reach <myip>:8888 but not <myip>:4443.

Means what exactly? Navigating to it? What is the exact error? A timeout? An immediate connection reset?

the ssltunnel config file used locally is

$ cat /tmp/ssltunnelUXY_Cu.cfg
httpproxy:1
certdbdir:/home/bclary/mozilla/builds/inbound/mozilla/build/pgo/certs
forward:127.0.0.1:8888
websocketserver:192.168.1.7:9988
listen:*:4443:pgoserver
listen:self-signed.example.com:443:4443:selfsigned
listen:untrusted.example.com:443:4443:untrusted
listen:expired.example.com:443:4443:expired
clientauth:requestclientcert.example.com:443:4443:request
clientauth:requireclientcert.example.com:443:4443:require
listen:mismatch.expired.example.com:443:4443:expired
listen:mismatch.untrusted.example.com:443:4443:untrusted
listen:untrusted-expired.example.com:443:4443:untrustedandexpired
listen:mismatch.untrusted-expired.example.com:443:4443:untrustedandexpired
listen:no-subject-alt-name.example.com:443:4443:noSubjectAltName
listen:bug413909.xn--hxajbheg2az3al.xn--jxalpdlp:443:4443:bug413909cert
listen:www.bank1.com:443:4443:escapeattack1
redirhost:redirproxy.example.com:443:4443:test1.example.com
listen:include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningGood
listen:bad.include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningBad
listen:badchain.include-subdomains.pinning.example.com:443:4443:staticPinningBad
failHandshake:fail-handshake.example.com:443:4443
listen:sha1ee.example.com:443:4443:sha1_end_entity
listen:sha256ee.example.com:443:4443:sha256_end_entity
listen:imminently-distrusted.example.com:443:4443:imminently_distrusted
ssl3:ssl3.example.com:443:4443
rc4:rc4.example.com:443:4443
ssl3:ssl3rc4.example.com:443:4443
rc4:ssl3rc4.example.com:443:4443
tls1:tls1.example.com:443:4443

looks fine

Locally, ssltunnel is running on my host as a linux x86_64 executable from host utils. My local fennec build does have a ssltunnel arm binary. Should we be running it on the device instead of the host?

we have not identified the problem yet.

(In reply to Honza Bambas (:mayhemer) from comment #50)

General request: please be more specific in details.

(In reply to Bob Clary [:bc:] from comment #49)

with:
SSLTUNNEL_LOG_LEVEL=0
MOZ_LOG=proxy:5
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=5090712e68322e9ca7201786e68a97680c6ced8f

It looks like the android-em emulators actually contact the ssltunnel since SSLTUNNEL messages appear in the log.

Where is ssltunnel actually running in this situation? What "contact" actually means?

I believe the ssltunnel program is running in a Docker container on the host as a linux binary.

I meant "contact" to mean that the logs for the android emulators contained SSLTUNNEL debug messages. The logs for the android hardware devices did not which I took to mean that they were not "contacted".

It is not clear to me that usermedia is actually enabled though since there are : TypeError: navigator.mediaDevices is undefined errors.

No idea how this is related.

From what I understand, the patch was to enable getUserMedia only on secure origins. It seemed to me that the error messages mean that even though there were SSLTUNNEL debug messages in the android emulator logs, that the browser didn't consider the request to be over secure origins. If this wasn't relevant, I'm sorry for bringing it up. :jib may be able to say whether this is significant or not.

The android-hw devices do not contain SSLTUNNEL messages.

What exactly does this mean?

The logs for the android emulators contained messages of the form

[task 2019-04-09T17:48:28.619Z] 17:48:28     INFO -  remoteautomation.py | Application pid: 1132
[task 2019-04-09T17:49:02.691Z] 17:49:02     INFO -  Server listening on port 4454 with cert pgoserver
[task 2019-04-09T17:49:02.691Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): incoming connection csock(0)=0x7efc0bd2c8b0, ssock(1)=0x7efc0bd2c910
[task 2019-04-09T17:49:02.691Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): polling flags csock(0)=R-, ssock(1)=R-
[task 2019-04-09T17:49:02.692Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): csock(0)=0x7efc0bd2c8b0 out_flags=1 :reading, read 195 bytes parsing initial connect request, dump:
[task 2019-04-09T17:49:02.693Z] 17:49:02     INFO -  CONNECT example.com:443 HTTP/1.1
[task 2019-04-09T17:49:02.694Z] 17:49:02     INFO -  User-Agent: Mozilla/5.0 (Android 4.3.1; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0
[task 2019-04-09T17:49:02.694Z] 17:49:02     INFO -  Proxy-Connection: keep-alive
[task 2019-04-09T17:49:02.695Z] 17:49:02     INFO -  Connection: keep-alive
[task 2019-04-09T17:49:02.696Z] 17:49:02     INFO -  Host: example.com:443
[task 2019-04-09T17:49:02.701Z] 17:49:02     INFO -   accepted CONNECT request, connected to the server, sending OK to the client
[task 2019-04-09T17:49:02.702Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): polling flags csock(0)=RW, ssock(1)=R-
[task 2019-04-09T17:49:02.703Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): csock(0)=0x7efc0bd2c8b0 out_flags=2 :writing, written 50 bytes dump:
[task 2019-04-09T17:49:02.704Z] 17:49:02     INFO -  HTTP/1.1 200 Connected
[task 2019-04-09T17:49:02.705Z] 17:49:02     INFO -  Connection: keep-alive
[task 2019-04-09T17:49:02.706Z] 17:49:02     INFO -   proxy response sent to the client client socket updated to SSL dropping our write flag and setting other socket read flag
[task 2019-04-09T17:49:02.707Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): polling flags csock(0)=R-, ssock(1)=R-
[task 2019-04-09T17:49:02.708Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): csock(0)=0x7efc0bd2c8b0 out_flags=1 :reading would block
[task 2019-04-09T17:49:02.709Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): polling flags csock(0)=R-, ssock(1)=R-
[task 2019-04-09T17:49:02.711Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): csock(0)=0x7efc0bd2c8b0 out_flags=1 :reading would block
[task 2019-04-09T17:49:02.712Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): polling flags csock(0)=R-, ssock(1)=R-
[task 2019-04-09T17:49:02.713Z] 17:49:02     INFO -  SSLTUNNEL(0x7efc0bd703a0)): csock(0)=0x7efc0bd2c8b0 out_flags=1 :reading, read 509 bytes incoming request to adjust:

while the logs for the android hardware did not.

The android-hw devices also contain

EDITED TO ADD CORRECT MESSAGE

I/Gecko ( 6749): [(null) 6749: Main Thread]: D/proxy DisableProxy http 10.7.205.214:4454 1800

It means we try to contact a proxy at 10.7.205.214:4454 and fail to. We then blacklist this proxy and try another one, if available. If all configured proxies are failing, we reset the disabled list and start over on next request.

Bitbar says that all ports are open and not blocked by a firewall. I disabled my local firewall and still reproduced the issue.

Bitbar is what? and running where?

Sorry. The android hardware devices are hosted at a company named Bitbar. The devices are connected to Linux hosts. The test frameworks are executed in Docker containers on the hosts. All of the ports are open and are not being blocked.

Locally, Fennec can reach <myip>:8888 but not <myip>:4443.

Means what exactly? Navigating to it? What is the exact error? A timeout? An immediate connection reset?

During the test run, the Fennec browser attempted to load a url from https://example.com/ and displayed:

The proxy server is refusing connections

Firefox is configured to use a proxy server that is refusing connections.

1. Check the proxy settings to make sure that they are correct.

2. Contact your network administrator to make sure the proxy server is working.

I then opened a new tab in Fennec and navigated to http://192.168.1.7:8888 and successfully displayed the /MochiKit/ directory. I then attempted to load http://192.168.1.7:4443 and https://192.168.1.7:4443 and received

Unable to connect

Firefox can't establish a connection to the server at 192.168.1.7:4443

1. The site could be temporarily unavailable or too busy. Try again in a few moments.

2. If you are unable to load any pages, check your mobile device's data or Wi-Fi connection.

$ ps aux | grep ssltunnel
bclary   19581  0.0  0.0  22232 12716 pts/2    Sl   09:21   0:00 /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64/ssltunnel /tmp/ssltunnelHlSCYA.cfg

sudo netstat -n -p did not show an open port for 4443 or a program ssltunnel.

the ssltunnel config file used locally is

$ cat /tmp/ssltunnelUXY_Cu.cfg
httpproxy:1
certdbdir:/home/bclary/mozilla/builds/inbound/mozilla/build/pgo/certs
forward:127.0.0.1:8888
websocketserver:192.168.1.7:9988
listen:*:4443:pgoserver
listen:self-signed.example.com:443:4443:selfsigned
listen:untrusted.example.com:443:4443:untrusted
listen:expired.example.com:443:4443:expired
clientauth:requestclientcert.example.com:443:4443:request
clientauth:requireclientcert.example.com:443:4443:require
listen:mismatch.expired.example.com:443:4443:expired
listen:mismatch.untrusted.example.com:443:4443:untrusted
listen:untrusted-expired.example.com:443:4443:untrustedandexpired
listen:mismatch.untrusted-expired.example.com:443:4443:untrustedandexpired
listen:no-subject-alt-name.example.com:443:4443:noSubjectAltName
listen:bug413909.xn--hxajbheg2az3al.xn--jxalpdlp:443:4443:bug413909cert
listen:www.bank1.com:443:4443:escapeattack1
redirhost:redirproxy.example.com:443:4443:test1.example.com
listen:include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningGood
listen:bad.include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningBad
listen:badchain.include-subdomains.pinning.example.com:443:4443:staticPinningBad
failHandshake:fail-handshake.example.com:443:4443
listen:sha1ee.example.com:443:4443:sha1_end_entity
listen:sha256ee.example.com:443:4443:sha256_end_entity
listen:imminently-distrusted.example.com:443:4443:imminently_distrusted
ssl3:ssl3.example.com:443:4443
rc4:rc4.example.com:443:4443
ssl3:ssl3rc4.example.com:443:4443
rc4:ssl3rc4.example.com:443:4443
tls1:tls1.example.com:443:4443

looks fine

Locally, ssltunnel is running on my host as a linux x86_64 executable from host utils. My local fennec build does have a ssltunnel arm binary. Should we be running it on the device instead of the host?

we have not identified the problem yet.

Thanks Bob! I can see better how things are now.

There is one bit that I can see instantly: Server listening on port 4454 with cert pgoserver from ssl tunnel [1] but it's expected to be reachable on 4443. Is it really using the correct config file?

Did connecting https://192.168.1.7:4443 took some time (20+ secs) to get an error or was it instant?

[1] https://searchfox.org/mozilla-central/rev/69ace9da347adcc4a33c6fa3d8e074759b91068c/testing/mochitest/ssltunnel/ssltunnel.cpp#909

Let me check I understand how things are layered:

+-----------------------------------------------------
+ Linux host
+
+ +----------------- [all ports open] ----------------
+ + Docker
+ +
+ + +-------------------------------------------------
+ + + Docker container
+ + + ssltunnel, expected to listen on :8888 plain
+ + +            and on :4443 with TLS
+ + +-------------------------------------------------
+ +---------------------------------------------------
+
+ +---------------------------------------------------
+ + Android emulator
+ + Fennec, configured to proxy SSL to http proxy on 192.168.1.7:4443 
+ +---------------------------------------------------
+-----------------------------------------------------

Questions:
0. is this actually correct? :)

1. where in this picture is the actual httpd.js server listening plain on 8888?
2. how is Docker (and the container) networked to the host? (nat, bridge, something else?) I assume 192.168.1.7 is IP of the Docker container as seen inside the Linux host, right?
3. how exactly does the ssltunnel binary reach the httpd.js server?
4. what is the local IP of the linux host and how the Docker container can reach anything running on the host?

(In reply to Honza Bambas (:mayhemer) from comment #52)

Thanks Bob! I can see better how things are now.

There is one bit that I can see instantly: Server listening on port 4454 with cert pgoserver from ssl tunnel [1] but it's expected to be reachable on 4443. Is it really using the correct config file?

I think we are getting the addresses and ports used when running locally via mach and the addresses and ports used when running in production CI via mozharness mixed up.

The ports are set for android when running in production CI via mozharness in:

https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/android_emulator_unittest.py#215
https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/android_hardware_unittest.py#195

The ports are different when run locally via mach.

Whenever I mention locally or an address like 192.168.1.7, I am referring to results from running the test locally via mach with an attached pixel2 device or with an emulator running on my laptop.

Did connecting https://192.168.1.7:4443 took some time (20+ secs) to get an error or was it instant?

Immediately.

The 192.168.1.7 address is the address of my local laptop and is not related to the ip addresses used in production. It is just an example of of a host ip address found when running the tests locally via mach.

getUserMedia build from the patches in the bug

You can get a build with the relevent getUserMedia changes from the try run at https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=5090712e68322e9ca7201786e68a97680c6ced8f

Running locally using attached pixel2

This uses a custom build but you can download a build and change the appname to org.mozilla.fennec_aurora

./mach mochitest  --appname org.mozilla.fennec_bclary dom/media
$ ps aux | egrep '(inbound|mozbuild)'
bclary    7738  7.2  0.3 1506500 56828 pts/2   Sl   05:24   0:05 /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64/xpcshell -g /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64 -f /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64/components/httpd.js -e const _PROFILE_PATH = '/tmp/tmp0ofMeA.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '192.168.1.7'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false; -f /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest/server.js
bclary    7741  0.1  0.1 232356 17076 pts/2    S    05:24   0:00 /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_virtualenvs/init/bin/python /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest/pywebsocket_wrapper.py -H 127.0.0.1 -p 9988 -w /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest -l /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest/websock.log --log-level=debug --allow-handlers-outside-root-dir
bclary    7744  0.1  0.1 247164 19492 pts/2    S    05:24   0:00 /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_virtualenvs/init/bin/python websocketprocessbridge/websocketprocessbridge.py --port 8191
bclary    7769  0.0  0.0  22232 12764 pts/2    Sl   05:24   0:00 /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64/ssltunnel /tmp/ssltunnellCKaYX.cfg
bclary    7925  0.0  0.0 215748   900 pts/1    S+   05:26   0:00 grep -E --color=auto (inbound|mozbuild)

bclary@ruby ~
$ lsof -i 4 -a -P -p 7769
COMMAND    PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
ssltunnel 7769 bclary    7u  IPv4 7245225      0t0  TCP localhost:4443 (LISTEN)

So when I run this locally via mach, ssltunnel is listening on 4443 but only on localhost. Maybe that is the issue.

Running locally with an x86-7.0 emulator

Installing an x86 build on the emulator then running the test locally via mach. You should be able to reproduce using this approach without needing a physical device.

This reproduces the issue for me where fennec on the emulator received The proxy server is refusing connections when accessing https://example.com/

./mach  android-emulator --version 'x86-7.0'

./mach mochitest  --appname org.mozilla.fennec_aurora dom/media

$ ps aux | egrep '(inbound|mozbuild)'
bclary   12870 18.5  9.1 5185328 1476508 pts/2 Sl   05:56   2:46 /home/bclary/.mozbuild/android-sdk-linux/emulator/qemu/linux-x86_64/qemu-system-x86_64 -avd mozemulator-x86-7.0 -gpu swiftshader_indirect -skip-adb-auth -verbose -show-kernel -ranchu -engine qemu2 -selinux permissive -memory 3072 -cores 4 -qemu -enable-kvm
bclary   12877  0.0  0.0  48192 12028 pts/2    Sl   05:56   0:00 /home/bclary/.mozbuild/android-sdk-linux/emulator/emulator64-crash-service -pipe 6 -ppid 12870 -data-dir /tmp/android-bclary/88b692f3-f81f-4e3d-8405-ba3d73305fb2
bclary   15013  2.8  0.3 1490116 54172 pts/2   Sl   06:10   0:01 /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64/xpcshell -g /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64 -f /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64/components/httpd.js -e const _PROFILE_PATH = '/tmp/tmpDlmYr0.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '192.168.1.7'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false; -f /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest/server.js
bclary   15016  0.3  0.1 232356 17044 pts/2    S    06:10   0:00 /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_virtualenvs/init/bin/python /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest/pywebsocket_wrapper.py -H 127.0.0.1 -p 9988 -w /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest -l /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_tests/testing/mochitest/websock.log --log-level=debug --allow-handlers-outside-root-dir
bclary   15019  0.2  0.1 247168 19476 pts/2    S    06:10   0:00 /home/bclary/mozilla/builds/inbound/mozilla/fennec-opt-arm/_virtualenvs/init/bin/python websocketprocessbridge/websocketprocessbridge.py --port 8191
bclary   15084  0.0  0.0  22232 12700 pts/2    Sl   06:10   0:00 /home/bclary/.mozbuild/android-device/host-utils-67.0a1.en-US.linux-x86_64/ssltunnel /tmp/ssltunnelCHCbI1.cfg
bclary   15298  0.0  0.0 215880  2312 pts/1    S+   06:11   0:00 grep -E --color=auto (inbound|mozbuild)

bclary@ruby /tmp
$ lsof -i 4 -a -P -p 15084
COMMAND     PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
ssltunnel 15084 bclary    7u  IPv4 7404518      0t0  TCP localhost:4443 (LISTEN)

$ cat  /tmp/ssltunnelCHCbI1.cfg
httpproxy:1
certdbdir:/home/bclary/mozilla/builds/inbound/mozilla/build/pgo/certs
forward:127.0.0.1:8888
websocketserver:192.168.1.7:9988
listen:*:4443:pgoserver
listen:self-signed.example.com:443:4443:selfsigned
listen:untrusted.example.com:443:4443:untrusted
listen:expired.example.com:443:4443:expired
clientauth:requestclientcert.example.com:443:4443:request
clientauth:requireclientcert.example.com:443:4443:require
listen:mismatch.expired.example.com:443:4443:expired
listen:mismatch.untrusted.example.com:443:4443:untrusted
listen:untrusted-expired.example.com:443:4443:untrustedandexpired
listen:mismatch.untrusted-expired.example.com:443:4443:untrustedandexpired
listen:no-subject-alt-name.example.com:443:4443:noSubjectAltName
listen:bug413909.xn--hxajbheg2az3al.xn--jxalpdlp:443:4443:bug413909cert
listen:www.bank1.com:443:4443:escapeattack1
redirhost:redirproxy.example.com:443:4443:test1.example.com
listen:include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningGood
listen:bad.include-subdomains.pinning-dynamic.example.com:443:4443:dynamicPinningBad
listen:badchain.include-subdomains.pinning.example.com:443:4443:staticPinningBad
failHandshake:fail-handshake.example.com:443:4443
listen:sha1ee.example.com:443:4443:sha1_end_entity
listen:sha256ee.example.com:443:4443:sha256_end_entity
listen:imminently-distrusted.example.com:443:4443:imminently_distrusted
ssl3:ssl3.example.com:443:4443
rc4:rc4.example.com:443:4443
ssl3:ssl3rc4.example.com:443:4443
rc4:ssl3rc4.example.com:443:4443
tls1:tls1.example.com:443:4443

Running at Bitbar with a pixel2

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239125961&repo=try&lineNumber=1926

/builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.fennec_aurora --remote-webserver=10.7.205.214 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-67.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-67.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/M49m9XsuQXu2ad4oeFZwZA/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-media_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-media_errorsummary.log --log-tbpl-level=info --screenshot-on-fail --chunk-by-runtime --subsuite=media --deviceSerial=FA83V1A02375 --this-chunk 3 --total-chunks 3 --disable-e10s

Here 10.7.205.214 is the ip address of the Docker container running on this host.

Running at AWS with an emulator

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239125957&repo=try&lineNumber=1431

/builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.fennec_aurora --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-67.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-67.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/M49m9XsuQXu2ad4oeFZwZA/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-media_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-media_errorsummary.log --log-tbpl-level=info --screenshot-on-fail --chunk-by-runtime --subsuite=media --deviceSerial=emulator-5554 --disable-e10s --this-chunk 3 --total-chunks 3

[1] https://searchfox.org/mozilla-central/rev/69ace9da347adcc4a33c6fa3d8e074759b91068c/testing/mochitest/ssltunnel/ssltunnel.cpp#909

Let me check I understand how things are layered:

[snip]

I think that is mostly correct though the emulator runs in it own network.

https://developer.android.com/studio/run/emulator-networking

Questions:
0. is this actually correct? :)

1. where in this picture is the actual httpd.js server listening plain on 8888?

This is started by the framework and is run via xpcshell. You can see the command line used locally above.

2. how is Docker (and the container) networked to the host? (nat, bridge, something else?) I assume 192.168.1.7 is IP of the Docker container as seen inside the Linux host, right?

No. 192.168.1.7 was referring to my laptop when running tests locally via mach.

3. how exactly does the ssltunnel binary reach the httpd.js server?

I don't understand the question.

4. what is the local IP of the linux host and how the Docker container can reach anything running on the host?

I'm not sure. I'll defer to Sakari @ Bitbar.

maybe https://searchfox.org/mozilla-central/source/testing/mochitest/ssltunnel/ssltunnel.cpp#892 is the reason.

// Explicitly listen on loopback to avoid users getting errors from their
// firewalls about ssltunnel needing permission.

accidentally submitted the comment.

(In reply to Bob Clary [:bc:] from comment #54)

maybe https://searchfox.org/mozilla-central/source/testing/mochitest/ssltunnel/ssltunnel.cpp#892 is the reason.

// Explicitly listen on loopback to avoid users getting errors from their
// firewalls about ssltunnel needing permission.

Good catch! Was not aware of that change at all - this is how it goes when network changes don't go through necko peers ;) If the ports are otherwise correctly setup on ssltunnel and firefox but it still doesn't work, the loopback only limitation could be the cause, yes.

Probably an option will be needed in ssltunnel (probably should have been made in bug 1474895 already)

I was going to try out an

#ifdef XP_MACOSX
PR_InitializeNetAddr(PR_IpAddrLoopback, si->listen_port, &server_addr);
#else
PR_InitializeNetAddr(PR_IpAddrAny, si->listen_port, &server_addr);
#endif

change to ssltunnel.cpp but I'm not sure how to test it since by default ssltunnel comes from hostutils.

gbrown says to just copy my local version of he one installed in hostutils.

(In reply to Bob Clary [:bc:] from comment #53)

3. how exactly does the ssltunnel binary reach the httpd.js server?

I don't understand the question.

ssltunnel just forwards to the plain httpd server. so it has to know where it is (ip:port) and that ip and port must be reachable from the location/context ssltunnel is running at. if they are in the same docker container or same host, then it's easy. if they are not, you must ensure ssltunnel (standing as a client) can reach httpd (ip must be correct, ports must be open, and if httpd is NAT'ed than NAT rules has to be setup).

(In reply to Bob Clary [:bc:] from comment #56)

I was going to try out an

#ifdef XP_MACOSX
PR_InitializeNetAddr(PR_IpAddrLoopback, si->listen_port, &server_addr);
#else
PR_InitializeNetAddr(PR_IpAddrAny, si->listen_port, &server_addr);
#endif

change to ssltunnel.cpp but I'm not sure how to test it since by default ssltunnel comes from hostutils.

the firewall window can appear on windows too... this has to be configurable. as a quick test, just back out bug 1474895 altogether.

Confirmed backing out the patch fixes the problem for me. Filed bug 1544089.

jib: You should be able to test your patches now and confirm that everything is ok before landing this again.

Thanks Bob!

https://hg.mozilla.org/mozilla-central/rev/cec93c8315e9
https://hg.mozilla.org/mozilla-central/rev/1e2987994551

Posted site compatibility note: https://www.fxsitecompat.com/en-CA/docs/2019/getusermedia-and-enumeratedevices-can-no-longer-be-used-on-insecure-sites/

Delightfully, this work was already mostly done. All I did was add a mention to the Firefox 68 for devs page at https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/68. A few minor copy edits here and there in related content.

I seem to have read in the comments on the patch that file:/// URLs and content loaded from localhost are considered secure for the purposes of this. True or not?

True.

Documentation updates:

• Confirmed that the privacy and security section in the getUserMedia() article is accurate.
• Submitted BCD PR to cover support for this requirement in browsers

Noted on Firefox 68 for developers: APIs

Feel free to make corrections if any are required, or to let me know of work that needs to be done.

Hi Kohei, sorry for not catching it earlier, but the site compat note has one inaccuracy in it.

It correctly says "getUserMedia() ... can no longer be used on insecure sites", which is true in both 68 and 69. Good.

But then it incorrectly says "enumerateDevices() can no longer be used on insecure sites", which is true in 69, but not 68.

The correct information is that in 68, getUserMedia will throw NotAllowedError, while enumerateDevices will continue to work until 69. This matches how Chrome has worked for a good while (pre Chrome 74). This is the intermediate stepping stone we chose on the way to 69's [SecureContext] behavior.

I don't know if we want to fix this at this point or just leave it?

I've just added this blog post to clarify. Feel free to link to it. https://blog.mozilla.org/webrtc/camera-microphone-require-https-in-firefox-68/

Thank you :jib! I’ve just corrected my site compatibility note and added link to your blog post.