Closed Bug 1144872 Opened 9 years ago Closed 8 years ago

Deploy bundleclone extension for hg

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86_64
Linux
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

References

Details

Attachments

(4 files, 1 obsolete file)

http://mercurial.selenic.com/wiki/BundleCloneExtension

Having this will allow the hg server to point clients at static bundles to use instead of cloning the repo from the hg server itself.

This basically replaces the functionality of hgtool's --bundle option.
This extension is now deployed to hg.mozilla.org and is ready for use by release automation.

Instructions are at https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmo/bundleclone.html.

Please notify
Depends on: 1041173
Please notify people in #vcs when this is being deployed so we may monitor things accordingly.
catlee, jlal: I would like all release automation to rip out its custom "clone a repo from a bundle/archive" code and replace it with hg clone + bundleclone. This includes taskcluster's code to create repos from tarball archives. The overhead of Mercurial + stream bundles over tar I believe to be marginal. And the benefit is a lot of complex code can be flat out deleted.

If you don't have plans to work on this yourselves, this is something I can do. Since TC was responsible for the last few floods against hg.mozilla.org, I'm inclined to fix TC first. Just point me at the code if you want me to do this.
Flags: needinfo?(jlal)
Flags: needinfo?(catlee)
Bug 1144872 - Add bundleclone extension; r?catlee

The bundleclone extension has been copied from
https://hg.mozilla.org/hgcustom/version-control-tools unmodified.

This review should be a rubber stamp since bundleclone has already been
reviewed.
Attachment #8614428 - Flags: review?(catlee)
Bug 1144872 - Use bundleclone when cloning; r?catlee

The bundleclone extension is our preferred mechanism for cloning
repositories. It is integrated with Mercurial and requires no effort
from us to support bundles: only that the server advertise bundles.

Unconditionally activate the bundleclone extension during when cloning.

A subsequent commit will remove support for bundles from hgtool.
Attachment #8614429 - Flags: review?(catlee)
Bug 1144872 - Remove support for handling bundles; r?catlee

Now that Mercurial itself transparently handles bundles courtesy of the
bundleclone extension, we don't need explicit support for bundles in
hgtool. This patch removes them.

The command line argument for specifying a bundle is preserved, as
callers in the wild may still use it. However, the option now does
nothing.
Attachment #8614430 - Flags: review?(catlee)
Comment on attachment 8614430 [details]
MozReview Request: Bug 1144872 - Remove support for handling bundles; r?catlee

Bug 1144872 - Remove support for handling bundles; r?catlee

Now that Mercurial itself transparently handles bundles courtesy of the
bundleclone extension, we don't need explicit support for bundles in
hgtool. This patch removes them.

The command line argument for specifying a bundle is preserved, as
callers in the wild may still use it. However, the option now does
nothing.
Comment on attachment 8614430 [details]
MozReview Request: Bug 1144872 - Remove support for handling bundles; r?catlee

Bug 1144872 - Remove support for handling bundles; r?catlee

Now that Mercurial itself transparently handles bundles courtesy of the
bundleclone extension, we don't need explicit support for bundles in
hgtool. This patch removes them.

The command line argument for specifying a bundle is preserved, as
callers in the wild may still use it. However, the option now does
nothing.
https://reviewboard.mozilla.org/r/9953/#review8753

I may have been too greedy with removing uses of bundle. I suspect there are callers of some of these scripts outside of this repo. If only we were using a monorepo, I would be able to search all the code and remove all references with an atomic commit.
Comment on attachment 8614428 [details]
MozReview Request: Bug 1144872 - Add bundleclone extension; r?catlee

https://reviewboard.mozilla.org/r/9949/#review8787

Ship It!
Attachment #8614428 - Flags: review?(catlee) → review+
Comment on attachment 8614429 [details]
MozReview Request: Bug 1144872 - Use bundleclone when cloning; r?catlee

https://reviewboard.mozilla.org/r/9951/#review8789

Ship It!
Attachment #8614429 - Flags: review?(catlee) → review+
(In reply to Gregory Szorc [:gps] from comment #9)
> https://reviewboard.mozilla.org/r/9953/#review8753
> 
> I may have been too greedy with removing uses of bundle. I suspect there are
> callers of some of these scripts outside of this repo. If only we were using
> a monorepo, I would be able to search all the code and remove all references
> with an atomic commit.

a monorepo wouldn't completely solve this, since there's still deployment order to be concerned about.

buildbot-configs, buildbotcustom, mozharness and puppet all need patching.

The puppet piece in particular is a bit more challenging, since it relies on a "compiled" version of hgtool to clone other repositories.
Flags: needinfo?(catlee)
I wonder if a better approach would be to deploy the extension on the system, and update the user's hgrc. Then all invocations of hg would benefit.
Yeah, let's deploy support for bundleclone first then we can follow up with removal of bundles support from hgtool and its consumers once we know bundleclone is working.

From IRC conversations, I'm assuming you are working on getting the /etc/mercurial/hgrc in order before this can land. I'm assuming you'll let me know when this is clear to land.
Yes, I'm working on a puppet patch to deploy the extension itself, and to manage hgrc.
Assignee: nobody → catlee
I think I addressed all your comments from IRC. I would appreciate another look at this!

I added a mercurial::system_hgrc class which manages /etc/mercurial/hgrc, and removed /root/.hgrc in favour of that.

Should we remove any old user .hgrc files? mercurial does seem to merge /etc/mercurial/hgrc and /root/.hgrc correctly, at least in terms of bundleclone behaviour.
Attachment #8620563 - Flags: feedback?(dustin)
Regarding merging /etc/mercurial/hgrc and /root/.hgrc, that should "just work." Only thing I can think of is that Mercurial looks at the user/group owner of config files and only loads files owned by "trusted" users. Run `hg help config` and search for "trusted" to learn more. Mercurial should print a warning at run-time if it sees a config file in a well-defined location that is ignored for trust reasons.
Attachment #8620563 - Flags: feedback?(dustin) → feedback+
Comment on attachment 8620563 [details]
github comparison of bundleclone branch

good enough for an r+?
Attachment #8620563 - Flags: review?(dustin)
Comment on attachment 8620563 [details]
github comparison of bundleclone branch

I think so, yes.  It looks like there's a stray edit to modules/runner/files/influxdb_hook.py in there, too, but that's easily removed.
Attachment #8620563 - Flags: review?(dustin) → review+
Comment on attachment 8620563 [details]
github comparison of bundleclone branch

https://hg.mozilla.org/build/puppet/rev/86da2a3fc8cb

removed the changes to the runner tasks - will attach as a separate patch
Attachment #8620563 - Flags: checked-in+
Comment on attachment 8620563 [details]
github comparison of bundleclone branch

failing on OSX because there's no 'root' group
Attachment #8620563 - Flags: checked-in+ → checked-in-
this is what changed from before:
https://github.com/catlee/puppet/compare/895685a5792a3c7685dd23760a14f183ab8ba4d1...bundleclone
Attachment #8620563 - Attachment is obsolete: true
Attachment #8629039 - Flags: review?(rail)
Comment on attachment 8629039 [details] [diff] [review]
bundleclone-puppet-1.diff

:shipit:
Attachment #8629039 - Flags: review?(rail) → review+
Attachment #8629039 - Flags: checked-in+
One problem we hit on some systems is when an older system hg is invoked, e.g. 

[root@releng-puppet1.srv.releng.scl3.mozilla.com builder]# /usr/bin/hg --version
*** failed to import extension bundleclone from /usr/local/lib/hgext/bundleclone.py: 'module' object has no attribute 'command'
Mercurial Distributed SCM (version 1.4)

Copyright (C) 2005-2009 Matt Mackall <mpm@selenic.com> and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This doesn't seem to cause runtime failures, but makes these operations generate output on stderr whereas they were silent before.
Mercurial 1.4?! There's your bug right there :) FWIW, I tested bundleclone down to 2.5. That's as far back as I'm willing to go. I'd actually prefer to drop everything below 3.2.4 because those versions have a known vulnerability with path handling. See https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/installing.html#recommended-versions.

The good news is that Mercurial ignores import failures when loading extensions. So you'll get this warning message and nothing else should come of it.
On UTC day July 5, the following S3 transfers occurred in us-west-2:

4,751 GET requests
3,693 stream bundles for build/mozharness
  935 stream bundles for build/tools
   16 stream bundles for mozilla-central
   12 stream bundles for releases/mozilla-aurora
    4 stream bundles for comm-central

In total, 106,737,182,463 bytes were transferred.

The majority of requests (2915) came from IP 63.245.214.82. The rest is pretty much a long tail of 52.*.*.* addresses, which are almost certainly all AWS IPs. The total bytes transferred to 63.245.214.82 was 38,464,825,521.

Our month to date bill says we have transferred 187.577 GB from us-west-2 to the internet at a cost of $16.58.

I'm anxious to look at the data from today, as we haven't had a non-holiday of load since automation started using bundle clones. The UTC day ends in a few hours, so maybe I'll run this again at EOD. I also need to import metrics into Graphite or something. For now, I'm just grepping the server logs that S3 produces for us.
I should have said that everything is looking very good so far. No runaway costs or anything like that. But we should continue to keep a close eye on things. I'm anxious to run hg.mozilla.org's server stats in a post bundleclone world so we can see who is accounting for the remaining bandwidth.
On UTC day July 6, the following S3 transfers occurred in us-west-2:

33,040 GET requests
32,001 stream bundles for build/mozharness
 1,020 stream bundles for build/tools
     8 stream bundles for mozilla-central
     5 stream bundles for releases/mozilla-aurora
     3 stream bundles for releases/mozilla-release

In total, 228,842,738,345 bytes were transferred.

Total transfer to internet was 216.184 GB, an increase of 28.607 GB over yesterday. Data transfer is ~$0.09/GB. So, assuming UTC day July 6 was typical, we're looking at per-day data transfer costs of $2.57. In other words, pocket change.

Keep in mind this is only for us-west-2. us-east-1 does have a bunch of traffic. But it is mostly internal-to-AWS and we only pay literally pennies for the count of HTTP requests, regardless of how much data was transferred.

I'll need to look at the HTTP logs from hg.mozilla.org to see what repos are eating the remaining bandwidth.
Depends on: 1181783
From UTC day July across all regions:

68,797 GET requests
65,448 stream bundles for build/mozharness
 2,600 stream bundles for build/tools
   346 stream bundles for integration/mozilla-inbound
   205 stream bundles for integration/fx-team
   ... a long tail of other bundles

In total, 1,563,014,396,236 bytes were transferred. Remember, previous comments were only including us-west-2, which is a little less than 50% of total S3 traffic. The big jump in bytes transferred was attributed to the several hundred clones of the 1.5GB stream bundle for the Firefox repos.
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0b4-candidates/build1/logs/release-mozilla-beta-linux_repack_1-bm94-build1-build6.txt.gz complains about missing file (inside mock):

command: START
command: hg parent --template {node|short}
command: cwd: mozilla-beta
command: env: {'HGPLAIN': '1'}
*** failed to import extension bundleclone from /usr/local/lib/hgext/bundleclone.py: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
command: END (0.50s elapsed)
The "failed to import extension bundleclone" error is harmless and was known to at least catlee and me. It would be nice to have it resolved.

One way would be to only activate the bundleclone extension as part of hgtool's cloning code. e.g. `hg --config extensions.bundleclone=/path/to/bundleclone.py`. Although that runs the risk of not having every `hg clone` on the machine use bundleclone. I believe this was originally decided against for whatever reason.
Attachment #8614430 - Flags: review?(catlee)
Blocks: 1191859
Blocks: 1196067
Blocks: 1200792
Flags: needinfo?(jlal)
Newer versions of hg support this natively - we won't be deploying bundleclone as a separate extension any more.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: