Closed Bug 1287439 Opened 8 years ago Closed 8 years ago

Android L10n Nighly N1,N4, N5 broken

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox49 fixed, firefox50 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox49 --- fixed
firefox50 --- fixed

People

(Reporter: cbook, Assigned: gps)

References

Details

Attachments

(2 files)

Bug 1287439 - Update robustcheckout extension with upstream;
58 bytes, text/x-review-board-request
Callek
: review+
Sylvestre
: approval-mozilla-aurora+
Details
Bug 1287439 - Update robustcheckout extension with upstream;
58 bytes, text/x-review-board-request
Callek
: review+
Details 
for issues like 

 Can't copy /builds/slave/m-cen-and-api-15-l10n_4-000000/build/mozilla-central/mobile/android/config/mozconfigs/android-api-15/l10n-nightly to /builds/slave/m-cen-and-api-15-l10n_4-000000/build/mozilla-central/.mozconfig: [Errno 2] No such file or directory: '/builds/slave/m-cen-and-api-15-l10n_4-000000/build/mozilla-central/mobile/android/config/mozconfigs/android-api-15/l10n-nightly'! 

https://treeherder.mozilla.org/logviewer.html#?job_id=4388574&repo=mozilla-central

or hg errors like

https://treeherder.mozilla.org/logviewer.html#?job_id=4388571&repo=mozilla-central
...ok this is odd,

Its failing to clone the 'an' locale, but as :flod says in IRC its a locale that was explicitly not used on central.

However, the log:

 04:32:58     INFO - Reading from file /builds/slave/m-cen-and-api-15-l10n_1-000000/build/mozilla-central/mobile/android/locales/all-locales
 04:32:58     INFO - Contents:
 04:32:58     INFO -  ar
 04:32:58     INFO -  an
 04:32:58     INFO -  as
 04:32:58     INFO -  az
 04:32:58     INFO -  be
 04:32:58     INFO -  bn-IN
 04:32:58     INFO -  br
 04:32:58     INFO -  ca
 04:32:58     INFO -  cak
 04:32:58     INFO -  cs
 04:32:58     INFO -  cy
 04:32:58     INFO -  da
 04:32:58     INFO -  de
 04:32:58     INFO -  dsb
 04:32:58     INFO -  en-GB
 04:32:58     INFO -  en-ZA
 04:32:58     INFO -  eo
 04:32:58     INFO -  es-AR
 04:32:58     INFO -  es-ES
 04:32:58     INFO -  es-MX
 04:32:58     INFO -  et
 04:32:58     INFO -  eu
 04:32:58     INFO -  fa
 04:32:58     INFO -  ff
 04:32:58     INFO -  fi
 04:32:58     INFO -  fr
 04:32:58     INFO -  fy-NL
 04:32:58     INFO -  ga-IE
 04:32:58     INFO -  gd
 04:32:58     INFO -  gl
 04:32:58     INFO -  gu-IN
 04:32:58     INFO -  he
 04:32:58     INFO -  hi-IN
 04:32:58     INFO -  hr
 04:32:58     INFO -  hsb
 04:32:58     INFO -  hu
 04:32:58     INFO -  hy-AM
 04:32:58     INFO -  id
 04:32:58     INFO -  is
 04:32:58     INFO -  it
 04:32:58     INFO -  ja
 04:32:58     INFO -  kk
 04:32:58     INFO -  kn
 04:32:58     INFO -  ko
 04:32:58     INFO -  lo
 04:32:58     INFO -  lt
 04:32:58     INFO -  lv
 04:32:58     INFO -  mai
 04:32:58     INFO -  ml
 04:32:58     INFO -  mr
 04:32:58     INFO -  ms
 04:32:58     INFO -  my
 04:32:58     INFO -  nb-NO
 04:32:58     INFO -  nl
 04:32:58     INFO -  nn-NO
 04:32:58     INFO -  or
 04:32:58     INFO -  pa-IN
 04:32:58     INFO -  pl
 04:32:58     INFO -  pt-BR
 04:32:58     INFO -  pt-PT
 04:32:58     INFO -  rm
 04:32:58     INFO -  ro
 04:32:58     INFO -  ru
 04:32:58     INFO -  sk
 04:32:58     INFO -  sl
 04:32:58     INFO -  son
 04:32:58     INFO -  sq
 04:32:58     INFO -  sr
 04:32:58     INFO -  sv-SE
 04:32:58     INFO -  ta
 04:32:58     INFO -  te
 04:32:58     INFO -  th
 04:32:58     INFO -  tr
 04:32:58     INFO -  tsz
 04:32:58     INFO -  uk
 04:32:58     INFO -  ur
 04:32:58     INFO -  uz
 04:32:58     INFO -  wo
 04:32:58     INFO -  xh
 04:32:58     INFO -  zh-CN
04:32:58 INFO - zh-TW

Has it. And yet, https://dxr.mozilla.org/mozilla-central/source/mobile/android/locales/all-locales doesn't... investigating more...
For what it's worth, that file has a very short history, since we mostly work on aurora to enable/disable locales.
http://hg.mozilla.org/mozilla-central/log/default/mobile/android/locales/all-locales

As far as I can tell there shouldn't be any file even similar to this in mozilla-central. That list looks more similar (but not identical) to the one we had on aurora a few cycles ago than central.
It seems that the checked-out version of mozilla-central doesn't match central. Which could be a problem in the bundle creation, for example.

I haven't found out which version it actually thinks to be on. I see the following in the logs, but can't find where revision ae8724925add1dd45c3d98954c59bf92ec058930 would be, if it was a revision:

04:31:43     INFO - Copy/paste: hg --config ui.merge=internal:merge --config extensions.robustcheckout=/builds/slave/m-cen-and-api-15-l10n_4-000000/scripts/external_tools/robustcheckout.py robustcheckout https://hg.mozilla.org/mozilla-central mozilla-central --sharebase /builds/hg-shared --branch default
04:33:07     INFO -  ensuring https://hg.mozilla.org/mozilla-central@default is available at mozilla-central
04:33:07     INFO -  (sharing from existing pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
04:33:07     INFO -  searching for changes
04:33:07     INFO -  no changes found
04:33:07     INFO -  (pulling to obtain default)
04:33:07     INFO -  no changes found
04:33:07     INFO -  131299 files updated, 0 files merged, 0 files removed, 0 files unresolved
04:33:07     INFO -  updated to ae8724925add1dd45c3d98954c59bf92ec058930
04:33:07     INFO - Return code: 0
> or hg errors like
> 
> https://treeherder.mozilla.org/logviewer.html#?job_id=4388571&repo=mozilla-
> central

For these kind of errors the problems seems to be related to the  missing of some components in https://hg.mozilla.org/l10n-central/ ,for example :tsz,an,gn or hsb which are required in the failed nightly Android jobs.
The jobs that have been completed successfully passed this step because components like eu,fa,fi,fr which were needed were also present on https://hg.mozilla.org/l10n-central/.

04:31:15     INFO -  ensuring https://hg.mozilla.org/l10n-central/eu@default is available at eu
(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #4)
> > or hg errors like
> > 
> > https://treeherder.mozilla.org/logviewer.html#?job_id=4388571&repo=mozilla-
> > central
> 
> For these kind of errors the problems seems...

Yea in this case its a symptom not a cause
Assignee: nobody → bugspam.Callek
Component: Buildduty → General Automation
QA Contact: bugspam.Callek → catlee
So per IRC chat with gps,

* Mercurial's code on resolving named references changed in 3.8 (not clear if thats relevant here, just noting)
* We essentially `pull` from m-c's `default` -- then `update` to the local copies resolved @default. Which is not necessarily m-c's default. And thus our issue.
* GPS is working on a patch that essentially resolves the branch against the upstream repo and then updates to the changeset directly, rather than continuing to use the branch name locally.
** Bug 1287506

This has likely been a problem for a while, though admittedly we update (later on) to whatever rev en-US's nightly we pulled is using anyway, so its only a big problem when the locale in all-locales is not available.

"hg update -r 0fbdcd21fad76a00328e67875c6f40dc219235f4" from http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-l10n/mozilla-central-android-api-15-l10n_2-unknown-bm74-build1-build28.txt.gz

And mozharness itself is pulled based on the rev of the cset we're reporting to....

This round-robin has a LOT of potential to bite us though, and since configure also runs on the initially checked out code, its not as simple as fetching all-locales from somewhere else.
Depends on: 1287506
I'm not satisfied with the explanation here.

Which revision did we check-out that had the inconsistent all-locales content?
In which non-unified not-shared repo is that revision?
(In reply to Axel Hecht [:Pike] from comment #3)
> It seems that the checked-out version of mozilla-central doesn't match
> central. Which could be a problem in the bundle creation, for example.
> 
> I haven't found out which version it actually thinks to be on. I see the
> following in the logs, but can't find where revision
> ae8724925add1dd45c3d98954c59bf92ec058930 would be, if it was a revision:
> 
> 04:31:43     INFO - Copy/paste: hg --config ui.merge=internal:merge --config
> extensions.robustcheckout=/builds/slave/m-cen-and-api-15-l10n_4-000000/
> scripts/external_tools/robustcheckout.py robustcheckout
> https://hg.mozilla.org/mozilla-central mozilla-central --sharebase
> /builds/hg-shared --branch default
> 04:33:07     INFO -  ensuring https://hg.mozilla.org/mozilla-central@default
> is available at mozilla-central
> 04:33:07     INFO -  (sharing from existing pooled repository
> 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
> 04:33:07     INFO -  searching for changes
> 04:33:07     INFO -  no changes found
> 04:33:07     INFO -  (pulling to obtain default)
> 04:33:07     INFO -  no changes found
> 04:33:07     INFO -  131299 files updated, 0 files merged, 0 files removed,
> 0 files unresolved
> 04:33:07     INFO -  updated to ae8724925add1dd45c3d98954c59bf92ec058930
> 04:33:07     INFO - Return code: 0

(In reply to Axel Hecht [:Pike] from comment #7)
> I'm not satisfied with the explanation here.
> 
> Which revision did we check-out that had the inconsistent all-locales
> content?
> In which non-unified not-shared repo is that revision?

This was esr45: http://hg.mozilla.org/releases/mozilla-esr45/rev/ae8724925add1dd45c3d98954c59bf92ec058930
This should be fixed by upgrading the robustcheckout extension. I'll get some patches together.
Assignee: bugspam.Callek → gps
Status: NEW → ASSIGNED
Synchronized with hgext/robustcheckout/__init.py
from https://hg.mozilla.org/hgcustom/version-control-tools at revision
d2140637eaf3f91fefa7c2f44cbaabf4c19faeb3.

Review commit: https://reviewboard.mozilla.org/r/64980/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/64980/
Attachment #8772059 - Flags: review?(bugspam.Callek)
Comment on attachment 8772059 [details]
Bug 1287439 - Update robustcheckout extension with upstream;

https://reviewboard.mozilla.org/r/64980/#review61968

stamp this reviewed-in-dep-bug code ;-)
Attachment #8772059 - Flags: review?(bugspam.Callek) → review+
Pushed by Callek@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/f89771c94b4a
Update robustcheckout extension with upstream; r=Callek
Synchronized with hgext/robustcheckout/__init.py
from https://hg.mozilla.org/hgcustom/version-control-tools at revision
d2140637eaf3f91fefa7c2f44cbaabf4c19faeb3.

Review commit: https://reviewboard.mozilla.org/r/64988/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/64988/
Attachment #8772065 - Flags: review?(bugspam.Callek)
Comment on attachment 8772065 [details]
Bug 1287439 - Update robustcheckout extension with upstream;

https://reviewboard.mozilla.org/r/64988/#review61970
Attachment #8772065 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8772059 [details]
Bug 1287439 - Update robustcheckout extension with upstream;

Approval Request Comment
[Feature/regressing bug #]: bug 1287506
[User impact if declined]: automation may not check out appropriate revision
[Describe test coverage new/current, TreeHerder]: tested in version-control-tools repo
[Risks and why]: should be low
[String/UUID change made/needed]: not part of Firefox

We should uplift this to Aurora (the only other repo with a copy of this file) to prevent automation from doing wonky things.
Attachment #8772059 - Flags: approval-mozilla-aurora?
https://hg.mozilla.org/mozilla-central/rev/f89771c94b4a
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox50: --- → fixed
Resolution: --- → FIXED
For bug history, this absolutely affects aurora, and has been failing the same way for many nights (Since July 12!)


02:08:06     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', '--config', 'extensions.robustcheckout=/builds/slave/m-aurora-and-api-15-l10n_3-000/scripts/external_tools/robustcheckout.py', 'robustcheckout', 'https://hg.mozilla.org/releases/mozilla-aurora', 'mozilla-aurora', '--sharebase', '/builds/hg-shared', '--branch', 'default']
02:08:06     INFO - Copy/paste: hg --config ui.merge=internal:merge --config extensions.robustcheckout=/builds/slave/m-aurora-and-api-15-l10n_3-000/scripts/external_tools/robustcheckout.py robustcheckout https://hg.mozilla.org/releases/mozilla-aurora mozilla-aurora --sharebase /builds/hg-shared --branch default
02:08:59     INFO -  ensuring https://hg.mozilla.org/releases/mozilla-aurora@default is available at mozilla-aurora
02:08:59     INFO -  (sharing from existing pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
02:08:59     INFO -  searching for changes
02:08:59     INFO -  adding changesets
02:08:59     INFO -  adding manifests
02:08:59     INFO -  adding file changes
02:08:59     INFO -  added 6532 changesets with 9572 changes to 9860 files
02:09:30     INFO -  (pulling to obtain default)
02:09:30     INFO -  no changes found
02:09:30     INFO -  145023 files updated, 0 files merged, 0 files removed, 0 files unresolved
02:09:30     INFO -  updated to a5a2837e3fed3cfdf6be39fa756e3be7e1257190
02:09:31     INFO - Return code: 0


http://hg.mozilla.org/releases/mozilla-aurora/rev/a5a2837e3fed3cfdf6be39fa756e3be7e1257190 -- 404

http://hg.mozilla.org/mozilla-central/rev/a5a2837e3fed3cfdf6be39fa756e3be7e1257190  -- exists
Comment on attachment 8772059 [details]
Bug 1287439 - Update robustcheckout extension with upstream;

Needed by releng, taking it.
Attachment #8772059 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: