Dylan Hardison [:dylan] (he/him) Assignee
	Description • 7 years ago

Currently, bmo is configured to use a 'http://' proxy scheme which means, among other things, we can't speak to anything using SNI. 
I'm told the proper solution is to use the CONNECT method, to which I have updated BMO's code to support.

However currently BMO's web nodes aren't allowed to do this:

cd /data/www/bugzilla.mozilla.org
perl -Ilocal/lib/perl5 -MLWP::UserAgent -E 'my $ua = LWP::UserAgent->new; $ua->proxy("https", "connect://dc-proxy"); my $r = $ua->get("https://github.com/mozilla-bteam/bmo/pull/85.diff"); say $r->content'

500 error while CONNECT thru proxy: 405 Method Not Allowed

Among other things, this breaks auth delegation between sites that use SNI,
and more pressingly breaks pull-requests-as-patch viewing.

	Brian Hourigan [:digi]
	Comment 1 • 7 years ago

I don't think the 405 is being generated by the proxy, instead this error is coming from LWP. I confirmed this by running your one liner under strace and looking for calls to connect() and stat().

It is in fact looking for LWP::Protocol::connect:
> stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)

I found an article on stackoverflow.com[0] which references this exact issue, and someone commenting they wrote a LWP protocol module to address it[1]. The module doesn't appear to be maintained, not having any releases since 2013.

I turned to LWP source and found a PR[2] related to using CONNECT, but I wasn't able to get it to work. LWP still wants to send a GET to the proxies for a https target. I think LWP might only support CONNECT when it's using SSL to connect to the proxy, which we currently don't support. (But we should)

I don't have the appropriate flows from my dev host to test this with a SSL port on squid, or a dev squid host I can install a bunch of dependencies on to continue testing.

I'll ping you on IRC to follow up

[0] https://stackoverflow.com/questions/12116244/https-proxy-and-lwpuseragent
[1] http://search.cpan.org/~benning/LWP-Protocol-connect/
[2] https://github.com/libwww-perl/libwww-perl/compare/master...dod38fr:fix-https-proxy

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 2 • 7 years ago

Thanks digi, and sorry for the noise: The root cause is that LWP is too old. We're using the system-provided one currently, so we need to add it to our dependencies.

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 3 • 7 years ago

LWP::UserAgent upgrade:
   https://github.com/mozilla-bteam/carton-bundles/commit/7d03027254b18e07e85140ae51833cdb61abf401

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 4 • 7 years ago

(In reply to Brian Hourigan [:digi] from comment #1)
> I don't think the 405 is being generated by the proxy, instead this error is
> coming from LWP. I confirmed this by running your one liner under strace and
> looking for calls to connect() and stat().
> 
> It is in fact looking for LWP::Protocol::connect:
> > stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)
I just wanted to mention that this test should be true if run from the bmo root.

	Brian Hourigan [:digi]
	Comment 5 • 7 years ago

(In reply to Dylan Hardison [:dylan] (he/him) from comment #4)
> (In reply to Brian Hourigan [:digi] from comment #1)
> > I don't think the 405 is being generated by the proxy, instead this error is
> > coming from LWP. I confirmed this by running your one liner under strace and
> > looking for calls to connect() and stat().
> > 
> > It is in fact looking for LWP::Protocol::connect:
> > > stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)
> I just wanted to mention that this test should be true if run from the bmo
> root.

I ran that on stage from /data/www/bugzilla-dev.allizom.org

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 6 • 7 years ago

Aha! and dev hadn't been updated. 

Still, the root problem is LWP itself was too old. :-)

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 7 • 7 years ago

To git@github.com:mozilla-bteam/bmo.git
   37120e708..8bb543cad  master -> master