Chris AtLee [:catlee] Reporter
	Description • 9 years ago

We have bundleclone installed at the host level, and we refer to it from our hgrc. However, it's not available inside mock which leads to confusing log messages e.g.

command: START
command: hg parent --template {node|short}
command: cwd: mozilla-beta
command: env: {'HGPLAIN': '1'}
*** failed to import extension bundleclone from /usr/local/lib/hgext/bundleclone.py: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
command: END (0.50s elapsed)

Also some jobs clone large repositories from inside mock, and would benefit from the speed increase here.

I can think of two approaches:
- Update the various lists of files we specify to copy into the mock env to include bundleclone
- Update the base mock environments to include bundleclone and any required configuration

	Chris AtLee [:catlee] Reporter
	Comment 1 • 9 years ago

This looks interesting...

# config_opts['files']['path/name/no/leading/slash'] = "put file contents here."

	Gregory Szorc [:gps]
	Comment 2 • 9 years ago

Per IRC conversation with catlee, this is almost certainly the cause of the high load to hg.mozilla.org this morning. As part of the release process coupled with an increase in the amount of concurrent l10n builders, full clones of mozilla-release and mozilla-esr38 contributed significant load to hg.mozilla.org. We definitely flirted with a few capacity limits. Had these clones been serviced from S3-backed bundles, we wouldn't have had a problem.

	Chris AtLee [:catlee] Reporter
	Comment 3 • 9 years ago

simply adding to mock_files fails because the leading directories don't exist:

11:33:01     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 --copyin --unpriv /usr/local/lib/hgext/bundleclone.py /usr/local/lib/hgext/bundleclone.py
11:33:01     INFO -  INFO: mock_mozilla.py version 1.0.3 starting...
11:33:01     INFO -  State Changed: init plugins
11:33:01     INFO -  INFO: selinux disabled
11:33:01     INFO -  State Changed: start
11:33:01     INFO -  State Changed: lock buildroot
11:33:01     INFO -  Mock Version: 1.0.3
11:33:01     INFO -  INFO: Mock Version: 1.0.3
11:33:01     INFO -  INFO: copying /usr/local/lib/hgext/bundleclone.py to /builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py
11:33:01     INFO -  ERROR: [Errno 2] No such file or directory: '/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py'
11:33:01     INFO -  Traceback (most recent call last):
11:33:01     INFO -    File "/usr/sbin/mock_mozilla", line 862, in <module>
11:33:01     INFO -      main(retParams)
11:33:01     INFO -    File "/usr/sbin/mock_mozilla", line 823, in main
11:33:01     INFO -      shutil.copy(src, dest)
11:33:01     INFO -    File "/usr/lib64/python2.6/shutil.py", line 84, in copy
11:33:01     INFO -      copyfile(src, dst)
11:33:01     INFO -    File "/usr/lib64/python2.6/shutil.py", line 51, in copyfile
11:33:01     INFO -      with open(dst, 'wb') as fdst:
11:33:01     INFO -  IOError: [Errno 2] No such file or directory: '/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py'
11:33:01    ERROR - Return code: 1

	Chris AtLee [:catlee] Reporter
	Comment 6 • 9 years ago

if anybody wants to pick this up while I'm away, you can use this spec file / dockerfile to generate an rpm that will install bundleclone.py into /usr/local/lib/hgext.

I've put up a generated copy here: http://people.mozilla.org/~catlee/mercurial-bundleclone-1.0-1.el6.noarch.rpm

Still TODO:
- write puppet patches to install this rpm as part of the base mock package (modify mock configs)
- spec file cleanup
- publishing rpm to rpm repos

	Gregory Szorc [:gps]
	Comment 7 • 9 years ago

Since bundleclone canonically lives in version-control-tools, we can put the RPM generation in it. Although, we'll need to pin the install location, and /usr/local may not be right for all scenarios. So maybe this is releng-specific enough to not live in v-c-t.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 8 • 9 years ago

You can work around the leading directories issue (comment #3) by specifying /usr/local/lib/hgext (and recursive copy takes care of the rest). I'm testing that out for releases with https://hg.mozilla.org/users/stage-ffxbld/buildbot-configs/rev/ac01212b5048.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 9 • 9 years ago

As we've discussed, this should resolve the Traceback's we're getting with Fennec 41.0 build1 multi-locale. It works in staging with the slave's .hgrc set to traceback on errors.

	Jordan Lund (:jlund)
	Comment 10 • 9 years ago

Comment on attachment 8661536 [details] [diff] [review]
[buildbot-configs] Copy in bundleclone when setting up mock environment

Review of attachment 8661536 [details] [diff] [review]:
-----------------------------------------------------------------

r+

	Nick Thomas [:nthomas] (UTC+12)
	Comment 11 • 9 years ago

Comment on attachment 8661536 [details] [diff] [review]
[buildbot-configs] Copy in bundleclone when setting up mock environment

https://hg.mozilla.org/build/buildbot-configs/rev/0b08becdf2b5
https://hg.mozilla.org/build/buildbot-configs/rev/65ef8a7712e3

Build masters reconfiged.

	Jordan Lund (:jlund)
	Comment 12 • 9 years ago

the mozharness equivalent.

some of these changes will almost certainly be required to fix android nightlies

	Pulsebot
	Comment 13 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/3e8dde8f8c17

	Nick Thomas [:nthomas] (UTC+12)
	Comment 14 • 9 years ago

Comment on attachment 8661557 [details] [diff] [review]
150915_1191859_make_bundleclone_available_within_mock-mozharness.patch

https://hg.mozilla.org/mozilla-central/rev/3e8dde8f8c17
https://hg.mozilla.org/releases/mozilla-aurora/rev/7484c6088282

That should sort preempt bustage of Android nightlies. We'll have to wait and see if the release-style single-locale repacks blow up, and possibly uplift further.