Closed
Bug 1144872
Opened 9 years ago
Closed 8 years ago
Deploy bundleclone extension for hg
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: catlee, Assigned: catlee)
References
Details
Attachments
(4 files, 1 obsolete file)
39 bytes,
text/x-review-board-request
|
catlee
:
review+
|
Details |
39 bytes,
text/x-review-board-request
|
catlee
:
review+
|
Details |
39 bytes,
text/x-review-board-request
|
Details | |
35.67 KB,
patch
|
rail
:
review+
catlee
:
checked-in+
|
Details | Diff | Splinter Review |
http://mercurial.selenic.com/wiki/BundleCloneExtension Having this will allow the hg server to point clients at static bundles to use instead of cloning the repo from the hg server itself. This basically replaces the functionality of hgtool's --bundle option.
Comment 1•9 years ago
|
||
This extension is now deployed to hg.mozilla.org and is ready for use by release automation. Instructions are at https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmo/bundleclone.html. Please notify
Depends on: 1041173
Comment 2•9 years ago
|
||
Please notify people in #vcs when this is being deployed so we may monitor things accordingly.
Comment 3•9 years ago
|
||
catlee, jlal: I would like all release automation to rip out its custom "clone a repo from a bundle/archive" code and replace it with hg clone + bundleclone. This includes taskcluster's code to create repos from tarball archives. The overhead of Mercurial + stream bundles over tar I believe to be marginal. And the benefit is a lot of complex code can be flat out deleted. If you don't have plans to work on this yourselves, this is something I can do. Since TC was responsible for the last few floods against hg.mozilla.org, I'm inclined to fix TC first. Just point me at the code if you want me to do this.
Flags: needinfo?(jlal)
Flags: needinfo?(catlee)
Comment 4•9 years ago
|
||
Bug 1144872 - Add bundleclone extension; r?catlee The bundleclone extension has been copied from https://hg.mozilla.org/hgcustom/version-control-tools unmodified. This review should be a rubber stamp since bundleclone has already been reviewed.
Attachment #8614428 -
Flags: review?(catlee)
Comment 5•9 years ago
|
||
Bug 1144872 - Use bundleclone when cloning; r?catlee The bundleclone extension is our preferred mechanism for cloning repositories. It is integrated with Mercurial and requires no effort from us to support bundles: only that the server advertise bundles. Unconditionally activate the bundleclone extension during when cloning. A subsequent commit will remove support for bundles from hgtool.
Attachment #8614429 -
Flags: review?(catlee)
Comment 6•9 years ago
|
||
Bug 1144872 - Remove support for handling bundles; r?catlee Now that Mercurial itself transparently handles bundles courtesy of the bundleclone extension, we don't need explicit support for bundles in hgtool. This patch removes them. The command line argument for specifying a bundle is preserved, as callers in the wild may still use it. However, the option now does nothing.
Attachment #8614430 -
Flags: review?(catlee)
Comment 7•9 years ago
|
||
Comment on attachment 8614430 [details] MozReview Request: Bug 1144872 - Remove support for handling bundles; r?catlee Bug 1144872 - Remove support for handling bundles; r?catlee Now that Mercurial itself transparently handles bundles courtesy of the bundleclone extension, we don't need explicit support for bundles in hgtool. This patch removes them. The command line argument for specifying a bundle is preserved, as callers in the wild may still use it. However, the option now does nothing.
Comment 8•9 years ago
|
||
Comment on attachment 8614430 [details] MozReview Request: Bug 1144872 - Remove support for handling bundles; r?catlee Bug 1144872 - Remove support for handling bundles; r?catlee Now that Mercurial itself transparently handles bundles courtesy of the bundleclone extension, we don't need explicit support for bundles in hgtool. This patch removes them. The command line argument for specifying a bundle is preserved, as callers in the wild may still use it. However, the option now does nothing.
Comment 9•9 years ago
|
||
https://reviewboard.mozilla.org/r/9953/#review8753 I may have been too greedy with removing uses of bundle. I suspect there are callers of some of these scripts outside of this repo. If only we were using a monorepo, I would be able to search all the code and remove all references with an atomic commit.
Assignee | ||
Comment 10•9 years ago
|
||
Comment on attachment 8614428 [details] MozReview Request: Bug 1144872 - Add bundleclone extension; r?catlee https://reviewboard.mozilla.org/r/9949/#review8787 Ship It!
Attachment #8614428 -
Flags: review?(catlee) → review+
Assignee | ||
Comment 11•9 years ago
|
||
Comment on attachment 8614429 [details] MozReview Request: Bug 1144872 - Use bundleclone when cloning; r?catlee https://reviewboard.mozilla.org/r/9951/#review8789 Ship It!
Attachment #8614429 -
Flags: review?(catlee) → review+
Assignee | ||
Comment 12•9 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #9) > https://reviewboard.mozilla.org/r/9953/#review8753 > > I may have been too greedy with removing uses of bundle. I suspect there are > callers of some of these scripts outside of this repo. If only we were using > a monorepo, I would be able to search all the code and remove all references > with an atomic commit. a monorepo wouldn't completely solve this, since there's still deployment order to be concerned about. buildbot-configs, buildbotcustom, mozharness and puppet all need patching. The puppet piece in particular is a bit more challenging, since it relies on a "compiled" version of hgtool to clone other repositories.
Flags: needinfo?(catlee)
Assignee | ||
Comment 13•9 years ago
|
||
I wonder if a better approach would be to deploy the extension on the system, and update the user's hgrc. Then all invocations of hg would benefit.
Comment 14•9 years ago
|
||
Yeah, let's deploy support for bundleclone first then we can follow up with removal of bundles support from hgtool and its consumers once we know bundleclone is working. From IRC conversations, I'm assuming you are working on getting the /etc/mercurial/hgrc in order before this can land. I'm assuming you'll let me know when this is clear to land.
Assignee | ||
Comment 15•9 years ago
|
||
Yes, I'm working on a puppet patch to deploy the extension itself, and to manage hgrc.
Assignee | ||
Updated•9 years ago
|
Assignee: nobody → catlee
Assignee | ||
Comment 16•9 years ago
|
||
I think I addressed all your comments from IRC. I would appreciate another look at this! I added a mercurial::system_hgrc class which manages /etc/mercurial/hgrc, and removed /root/.hgrc in favour of that. Should we remove any old user .hgrc files? mercurial does seem to merge /etc/mercurial/hgrc and /root/.hgrc correctly, at least in terms of bundleclone behaviour.
Attachment #8620563 -
Flags: feedback?(dustin)
Comment 17•9 years ago
|
||
Regarding merging /etc/mercurial/hgrc and /root/.hgrc, that should "just work." Only thing I can think of is that Mercurial looks at the user/group owner of config files and only loads files owned by "trusted" users. Run `hg help config` and search for "trusted" to learn more. Mercurial should print a warning at run-time if it sees a config file in a well-defined location that is ignored for trust reasons.
Updated•9 years ago
|
Attachment #8620563 -
Flags: feedback?(dustin) → feedback+
Assignee | ||
Comment 18•9 years ago
|
||
Comment on attachment 8620563 [details]
github comparison of bundleclone branch
good enough for an r+?
Attachment #8620563 -
Flags: review?(dustin)
Comment 19•9 years ago
|
||
Comment on attachment 8620563 [details]
github comparison of bundleclone branch
I think so, yes. It looks like there's a stray edit to modules/runner/files/influxdb_hook.py in there, too, but that's easily removed.
Attachment #8620563 -
Flags: review?(dustin) → review+
Assignee | ||
Comment 20•9 years ago
|
||
Comment on attachment 8620563 [details] github comparison of bundleclone branch https://hg.mozilla.org/build/puppet/rev/86da2a3fc8cb removed the changes to the runner tasks - will attach as a separate patch
Attachment #8620563 -
Flags: checked-in+
Assignee | ||
Comment 21•9 years ago
|
||
Comment on attachment 8620563 [details]
github comparison of bundleclone branch
failing on OSX because there's no 'root' group
Attachment #8620563 -
Flags: checked-in+ → checked-in-
Assignee | ||
Comment 22•9 years ago
|
||
this is what changed from before: https://github.com/catlee/puppet/compare/895685a5792a3c7685dd23760a14f183ab8ba4d1...bundleclone
Attachment #8620563 -
Attachment is obsolete: true
Attachment #8629039 -
Flags: review?(rail)
Comment 23•9 years ago
|
||
Comment on attachment 8629039 [details] [diff] [review] bundleclone-puppet-1.diff :shipit:
Attachment #8629039 -
Flags: review?(rail) → review+
Assignee | ||
Updated•9 years ago
|
Attachment #8629039 -
Flags: checked-in+
Assignee | ||
Comment 24•9 years ago
|
||
One problem we hit on some systems is when an older system hg is invoked, e.g. [root@releng-puppet1.srv.releng.scl3.mozilla.com builder]# /usr/bin/hg --version *** failed to import extension bundleclone from /usr/local/lib/hgext/bundleclone.py: 'module' object has no attribute 'command' Mercurial Distributed SCM (version 1.4) Copyright (C) 2005-2009 Matt Mackall <mpm@selenic.com> and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This doesn't seem to cause runtime failures, but makes these operations generate output on stderr whereas they were silent before.
Comment 25•9 years ago
|
||
Mercurial 1.4?! There's your bug right there :) FWIW, I tested bundleclone down to 2.5. That's as far back as I'm willing to go. I'd actually prefer to drop everything below 3.2.4 because those versions have a known vulnerability with path handling. See https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/installing.html#recommended-versions. The good news is that Mercurial ignores import failures when loading extensions. So you'll get this warning message and nothing else should come of it.
Comment 26•9 years ago
|
||
On UTC day July 5, the following S3 transfers occurred in us-west-2: 4,751 GET requests 3,693 stream bundles for build/mozharness 935 stream bundles for build/tools 16 stream bundles for mozilla-central 12 stream bundles for releases/mozilla-aurora 4 stream bundles for comm-central In total, 106,737,182,463 bytes were transferred. The majority of requests (2915) came from IP 63.245.214.82. The rest is pretty much a long tail of 52.*.*.* addresses, which are almost certainly all AWS IPs. The total bytes transferred to 63.245.214.82 was 38,464,825,521. Our month to date bill says we have transferred 187.577 GB from us-west-2 to the internet at a cost of $16.58. I'm anxious to look at the data from today, as we haven't had a non-holiday of load since automation started using bundle clones. The UTC day ends in a few hours, so maybe I'll run this again at EOD. I also need to import metrics into Graphite or something. For now, I'm just grepping the server logs that S3 produces for us.
Comment 27•9 years ago
|
||
I should have said that everything is looking very good so far. No runaway costs or anything like that. But we should continue to keep a close eye on things. I'm anxious to run hg.mozilla.org's server stats in a post bundleclone world so we can see who is accounting for the remaining bandwidth.
Comment 28•9 years ago
|
||
On UTC day July 6, the following S3 transfers occurred in us-west-2: 33,040 GET requests 32,001 stream bundles for build/mozharness 1,020 stream bundles for build/tools 8 stream bundles for mozilla-central 5 stream bundles for releases/mozilla-aurora 3 stream bundles for releases/mozilla-release In total, 228,842,738,345 bytes were transferred. Total transfer to internet was 216.184 GB, an increase of 28.607 GB over yesterday. Data transfer is ~$0.09/GB. So, assuming UTC day July 6 was typical, we're looking at per-day data transfer costs of $2.57. In other words, pocket change. Keep in mind this is only for us-west-2. us-east-1 does have a bunch of traffic. But it is mostly internal-to-AWS and we only pay literally pennies for the count of HTTP requests, regardless of how much data was transferred. I'll need to look at the HTTP logs from hg.mozilla.org to see what repos are eating the remaining bandwidth.
Comment 29•9 years ago
|
||
From UTC day July across all regions: 68,797 GET requests 65,448 stream bundles for build/mozharness 2,600 stream bundles for build/tools 346 stream bundles for integration/mozilla-inbound 205 stream bundles for integration/fx-team ... a long tail of other bundles In total, 1,563,014,396,236 bytes were transferred. Remember, previous comments were only including us-west-2, which is a little less than 50% of total S3 traffic. The big jump in bytes transferred was attributed to the several hundred clones of the 1.5GB stream bundle for the Firefox repos.
Comment 30•9 years ago
|
||
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0b4-candidates/build1/logs/release-mozilla-beta-linux_repack_1-bm94-build1-build6.txt.gz complains about missing file (inside mock): command: START command: hg parent --template {node|short} command: cwd: mozilla-beta command: env: {'HGPLAIN': '1'} *** failed to import extension bundleclone from /usr/local/lib/hgext/bundleclone.py: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py' command: END (0.50s elapsed)
Comment 31•9 years ago
|
||
The "failed to import extension bundleclone" error is harmless and was known to at least catlee and me. It would be nice to have it resolved. One way would be to only activate the bundleclone extension as part of hgtool's cloning code. e.g. `hg --config extensions.bundleclone=/path/to/bundleclone.py`. Although that runs the risk of not having every `hg clone` on the machine use bundleclone. I believe this was originally decided against for whatever reason.
Assignee | ||
Updated•9 years ago
|
Attachment #8614430 -
Flags: review?(catlee)
Updated•9 years ago
|
Flags: needinfo?(jlal)
Assignee | ||
Comment 32•8 years ago
|
||
Newer versions of hg support this natively - we won't be deploying bundleclone as a separate extension any more.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Updated•6 years ago
|
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•