Closed Bug 1094364 Opened 10 years ago Closed 9 years ago

switch win64 builds to use mozharness mach

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(firefox38 fixed, firefox39 fixed)

RESOLVED FIXED
Tracking Status
firefox38 --- fixed
firefox39 --- fixed

People

(Reporter: jlund, Assigned: jlund)

References

Details

Attachments

(10 files)

5.08 KB, patch
catlee
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
5.46 KB, patch
catlee
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
669 bytes, patch
mshal
: review+
Details | Diff | Splinter Review
3.09 KB, patch
catlee
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
309 bytes, patch
catlee
: review+
Details | Diff | Splinter Review
309 bytes, patch
catlee
: review+
Details | Diff | Splinter Review
541 bytes, patch
mshal
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
308 bytes, patch
jlund
: review+
Details | Diff | Splinter Review
309 bytes, patch
RyanVM
: review+
Details | Diff | Splinter Review
616 bytes, patch
mshal
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
      No description provided.
the buildbot parts needed to start playing with win64

note: I needed to add reboot and python interpreter to win64-debug only since it was already there (for l10n stuff) in win64 opt

[dev_master]jlund@Hastings163:~/devel/mozilla/dev_master/build-master
> peop ~/tmp/win64_mh_cedar-dump_master.diff
win64_mh_cedar-dump_master.diff
http://people.mozilla.org/~jlund/win64_mh_cedar-dump_master.diff

[dev_master]jlund@Hastings163:~/devel/mozilla/dev_master/build-master
> peop ~/tmp/win64_mh_cedar-builder_list.diff
win64_mh_cedar-builder_list.diff
http://people.mozilla.org/~jlund/win64_mh_cedar-builder_list.diff
Attachment #8559360 - Flags: review?(catlee)
between this and the previous bbot-config attachment, we should have win64 builds on cedar. who knows, it could 'just work'.

Note: I removed the nsis refs from both the win64 and win64-debug PATHS even though nsis still exists in win64-debug.

here is how this differs from the win32 configs:
http://people.mozilla.org/~jlund/win64_config_differences.diff
Attachment #8559381 - Flags: review?(catlee)
Comment on attachment 8559360 [details] [diff] [review]
150204_1094364_win64_mh_cedar-bbot-configs.diff

Review of attachment 8559360 [details] [diff] [review]:
-----------------------------------------------------------------

<stamp />
Attachment #8559360 - Flags: review?(catlee) → review+
Comment on attachment 8559381 [details] [diff] [review]
150204_1094364_win64_mh_cedar-mh.diff

Review of attachment 8559381 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/builds/releng_base_windows_64_builds.py
@@ +59,5 @@
> +
> +
> +     #########################################################################
> +     ###### 64 bit specific ######
> +    'base_name': 'WINNT_6.1_%(branch)s',

is this right? should it be  WINNT_6.1_x86-64_branch?
Attachment #8559381 - Flags: review?(catlee) → review+
Comment on attachment 8559381 [details] [diff] [review]
150204_1094364_win64_mh_cedar-mh.diff

Review of attachment 8559381 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/mozharness/rev/d5c3f2d98f46

::: configs/builds/releng_base_windows_64_builds.py
@@ +59,5 @@
> +
> +
> +     #########################################################################
> +     ###### 64 bit specific ######
> +    'base_name': 'WINNT_6.1_%(branch)s',

yup. thanks for catch.
Attachment #8559381 - Flags: checked-in+
Comment on attachment 8559360 [details] [diff] [review]
150204_1094364_win64_mh_cedar-bbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/88116f993c6d
Attachment #8559360 - Flags: checked-in+
poked this today. looks like we are failing to create a virtualenv: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-win64/1424449078/cedar-win64-bm85-build1-build2.txt.gz

snippet:
09:27:46     INFO - buildids match.
09:27:46     INFO - Creating virtualenv c:\builds\moz2_slave\ced-w64-0000000000000000000000\build\venv
09:27:46    FATAL - The executable 'virtualenv' is not found; not creating virtualenv!
09:27:46    FATAL - Running post_fatal callback...
09:27:46    ERROR - setting return code to 2 because fatal was called
09:27:46    FATAL - Exiting -1
09:27:46     INFO - Running post-run listener: _summarize
09:27:46    ERROR - # TBPL FAILURE #

this must be because it landed beside mshal's s3 taskcluster bits and the virtualenv executable wasn't included.

I think this patch is all that's needed.

tested and now allows win64 to complete: https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=d2d9810cf8d7
Attachment #8570041 - Flags: review?(mshal)
Comment on attachment 8570041 [details] [diff] [review]
150225_mh_builds_win64_tc_fix-mh.patch

Looks good!
Attachment #8570041 - Flags: review?(mshal) → review+
why not. We could do this more gradually by turning on a few branches at a time like I did initially for mh builds. However, seeing how mh builds have been working successfully for a while, I think the risk is less switching this over at once and backing out if needed.

note: this can't land till we update mozharness.json on trunk and m-a

builderlist diff: http://people.mozilla.org/~jlund/win64_mh_builds_m-c_and_m-a-builderlist.diff
Attachment #8570302 - Flags: review?(catlee)
Attachment #8570302 - Flags: review?(catlee) → review+
will be switching this on today
side note: we should probably stop asking for reviews of these. we still need a place to document the mh version bump on a branch as we are not only deploying revs that we need but also every rev in between current pin and this one. Reconfigs are merging mh to prod and updating bugs but most things are not going 'live' anymore.

this system of pinning by REV feels a little broken/awkward at the moment.
Attachment #8573086 - Flags: review?(catlee)
(In reply to Jordan Lund (:jlund) from comment #12)
> side note: we should probably stop asking for reviews of these. we still
> need a place to document the mh version bump on a branch as we are not only
> deploying revs that we need but also every rev in between current pin and
> this one. Reconfigs are merging mh to prod and updating bugs but most things
> are not going 'live' anymore.

I was wondering about this as well, especially since I'll need to bump it again for bug 1132123 :). Is the mh merge from default->production during a reconfig still useful when the version is pinned anyway? Maybe we should have a fool-proof mh bumper bot that updates the revision in m-c during a reconfig, and then nuke the production branch.
Attachment #8573083 - Flags: review?(catlee) → review+
Attachment #8573086 - Flags: review?(catlee) → review+
Comment on attachment 8570302 [details] [diff] [review]
150226_win64_mh_builds_trunk_and_aurora-bbotcfgs.patch

Review of attachment 8570302 [details] [diff] [review]:
-----------------------------------------------------------------

the mh rev required is on m-c and m-a now.

some proj branches that are lazy at merging m-c may break for a short while till they actually merge.

on default: remote:   https://hg.mozilla.org/build/buildbot-configs/rev/b895db57d5a1
Attachment #8570302 - Flags: checked-in+
reviewer: can I get your stamp for the subsequent mozharness.json bump too? It will need to land across trunk.

after landing, we had a regression for pgo builds: https://treeherder.mozilla.org/logviewer.html#?job_id=7373106&repo=mozilla-inbound

the reason is because I kept the max vsize linker check (and graph server post) enabled for win64 like it is for win32. However we don't do that for win64 buildbot factory case. You can see we only define enable_post_linker_size for the win32 case : http://mxr.mozilla.org/build/search?string=enable_post_linker_size&find=&findi=&filter=%5E%5B%5E%5C0%5D*%24&hitlimit=&tree=build


this patch disables the graph server post and linker check: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#1518
Comment on attachment 8574872 [details] [diff] [review]
150309_win64_mh_builds-disables_vsize_max_linker_check-mh.patch

can I get your approval here. we can figure out after we fix bustage whether we want to continue disabling win64 vsize linker checks or if we should update graph server to support win64 too
Attachment #8574872 - Flags: review?(mshal)
Attachment #8574872 - Flags: review?(mshal) → review+
Comment on attachment 8574872 [details] [diff] [review]
150309_win64_mh_builds-disables_vsize_max_linker_check-mh.patch

thank you, this has landed and m-i mozharness.json has been updated. once merged, pgo win64 will be fixed

remote:   https://hg.mozilla.org/build/mozharness/rev/0230a275037f
Attachment #8574872 - Flags: checked-in+
sigh. we are hitting failures with pgo win again.

looks like we lost my previous mozharness.json rev because we switched the pin rev from a default one to a prod one and the prod rev didn't have the latest default changes merged.

this points us to current tip
Attachment #8576194 - Flags: review?(coop)
Attachment #8576194 - Flags: review?(coop) → review+
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Attachment #8576989 - Flags: review?(mshal) → review+
Comment on attachment 8576989 [details] [diff] [review]
150309_win64_mh_builds-add_to_percheckin_pgo_strategy-mh.patch

remote:   https://hg.mozilla.org/build/mozharness/rev/c89055ada676
Attachment #8576989 - Flags: checked-in+
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: