Closed Bug 1188992 Opened 9 years ago Closed 9 years ago

[OTA] Latest Flame build is a day behind

Categories

(Firefox OS Graveyard :: Gaia::System, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::System
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(b2g-master verified)

Status:
VERIFIED FIXED
Tracking Flags:
Tracking Status
b2g-master --- verified

People

(Reporter: Marty, Unassigned)

Details

(Keywords: qablocker, Whiteboard: [2.5-Daily-Testing], [b2g-build-support])

Attachments

(3 files)

logcat-flame-ota.txt
80.15 KB, text/plain
Details
verified_logcat_2039.txt
183.50 KB, text/plain
Details
verified_Flame KK_v2.5.3gp
3.94 MB, video/3gpp
Details
Attached file logcat-flame-ota.txt — 
Description:
Using yesterday's '20150728030208' build, there is no OTA update available on the 'nightlytest' channel.

According to nhirata on irc, this is due to an aliasing issue.

<nhirata_> B2G-mozilla-central-nightly-latest
<nhirata_> 20150729030209
<nhirata_> that shuld be the build
<nhirata_> oh it's not aliased
<nhirata_> 20150727162816
<nhirata_> file a bug where flame isn't being aliased for flame-kk
<nhirata_> and NI me please
<nhirata_> I just changed it so you have the right build.
<nhirata_> 20150729030209

Repro Steps:
1) Update a Flame to 20150728030208
2) Connect to a WiFi or Data network
3) Change the update channel to 'nightlytest'
4) Check for updates

Actual:
No system updates are found

Expected:
The latest Flame build system update  is found successfully.

Environmental Variables:
Device: Flame 2.5
Build ID: 20150728030208
Gaia: 14e32276025b0310d3e89027320cf4b2a24cedfb
Gecko: 33dc8a83cfc0
Gonk: 41d3e221039d1c4486fc13ff26793a7a39226423
Version: 42.0a1 (2.5)
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:42.0) Gecko/42.0 Firefox/42.0

Repro frequency: 5/5
See attached: Logcat
Flags: needinfo?(nhirata.bugzilla)
{
  "platforms": {
    "aries-userdebug": {
      "locales": {
        "en-US": {
          "buildID": "20150803170922",
          "platformVersion": "42.0a1",
          "displayVersion": "42.0a1",
          "appVersion": "42.0a1",
          "completes": [
            {
              "fileUrl": "https://queue.taskcluster.net/v1/task/mimzLu8_SqqPEFA26Sw1Bg/runs/0/artifacts/public/build/b2g-aries-gecko-update.mar",
              "from": "*",
              "hashValue": "4baeeff61875b96f742953eb494e2f42d59270408a1386b145867cd5ecae0d56fff663133593fa39f23e76f78be6ed1d61789279076f19eb4954c3a877f9abb7",
              "filesize": 125875211
            }
          ]
        }
      }
    },
    "aries": {
      "locales": {
        "en-US": {
          "buildID": "20150803171137",
          "platformVersion": "42.0a1",
          "displayVersion": "42.0a1",
          "appVersion": "42.0a1",
          "completes": [
            {
              "fileUrl": "https://queue.taskcluster.net/v1/task/t8LDwvzhR5ihqWZAw6O2uA/runs/0/artifacts/public/build/b2g-aries-gecko-update.mar",
              "from": "*",
              "hashValue": "a40ed60cc861771aa22b5f84da95ebee7802b58f0538bf069d05d0b80cd98cedfbc3f8842d762e350b4566121a72d22dc252a1c1e0a8a997e0c046887743ee2f",
              "filesize": 125679252
            }
          ]
        }
      }
    },
    "flame": {
      "locales": {
        "en-US": {
          "buildID": "20150803132043",
          "platformVersion": "42.0a1",
          "displayVersion": "42.0a1",
          "completes": [
            {
              "fileUrl": "https://queue.taskcluster.net/v1/task/-t1J0bTXTpyXvpcH2xfnvg/runs/0/artifacts/public/build/b2g-flame-gecko-update.mar",
              "from": "*",
              "hashValue": "d60aee0a563a6b92e2f5e8f152bff9dce881dcd1a2f95371d15facf172f3072cbd46cd7853ce903464dfe2584faea19367a6088ac8ec883a860501e26fd565d5",
              "filesize": 94557431
            }
          ],
          "appVersion": "42.0a1"
        }
      }
    },
    "flame-userdebug": {
      "locales": {
        "en-US": {
          "buildID": "20150803170724",
          "platformVersion": "42.0a1",
          "displayVersion": "42.0a1",
          "completes": [
            {
              "fileUrl": "https://queue.taskcluster.net/v1/task/ePvjJbhTRAuHGV-zy9bYUA/runs/0/artifacts/public/build/b2g-flame-gecko-update.mar",
              "from": "*",
              "hashValue": "41c74c3af3e9db006de2415a6a8b7e74c3aa4c5346ff03d3d5d9c4825bad778658855522bc172ef0504cebab8b3e8f6974075c60418782c267ac62c804f2a26c",
              "filesize": 94798429
            }
          ],
          "appVersion": "42.0a1"
        }
      }
    }
  },
  "hashFunction": "sha512",
  "name": "B2G-gecko-nightly-latest",
  "schema_version": 4
}


Changed by b2gbld 12 days ago
Keywords: qablocker
FYI, not sure, but I think there's something up with buildbot.
Flags: needinfo?(wcosta)
Sorry for the late response,

iirc, after mozharness went in-tree, blob name is now B2G-gecko-nightly-latest.
Flags: needinfo?(wcosta)
Thanks Wander.  I wasn't aware of this... Would there be a way that we can get a heads up on these changes before they take place?

Also flame is still a day behind.

 "flame": {
      "locales": {
        "en-US": {
          "buildID": "20150820202733",
          "platformVersion": "43.0a1",
          "displayVersion": "43.0a1",
          "completes": [
            {
              "fileUrl": "https://queue.taskcluster.net/v1/task/NBHLNIaWRgSZq5yN2zO87g/runs/0/artifacts/public/build/b2g-flame-gecko-update.mar",
              "from": "*",
              "hashValue": "f12e40a3c801d85870b826cfcf59988b64788e83a4f41a1784e88f8fb3c0d5188ed9a8a3b0db5a59d78fcb6a957c426430f53ebab6bedc186fee4b1bc589ea5c",
              "filesize": 95680379
            }
          ],
          "appVersion": "43.0a1"
        }
      }
    },
Flags: needinfo?(wcosta)
Flags: needinfo?(nhirata.bugzilla)
Flags: needinfo?(mshal)
Summary: [OTA] Latest Flame build is not being aliased for flame-kk on nightlytest channel → [OTA] Latest Flame build is a day behind
After chatting with wcosta and bhearsum, it sounds like the problem (or at least part of the problem) may be from how the balrog rules are configured. In particular, the update query is using "nightlytest", while balrog is configured to use "nightly-test". bhearsum also noted a few other issues, which he can explain better than I :). In any case, this doesn't appear to be a problem with buildbot at the moment.
Flags: needinfo?(mshal) → needinfo?(bhearsum)
To address the immediate issue, it looks to me that there's a mismatch between channel name in the rule ("nightly-test"), and the one in the update URL ("nightlytest"). The latter is what we typically use, so I'd advise changing the one in the rule.

I also noticed some other weird things that may or may not be causing issues with this or other things:
* Many other rules have "-test" in their name: dogfood-test, nightly-b2g34-test, nightly-b2g37-test
* There's a rule with "nightly-latest" for its channel, is that even used?
* It looks like all of the releases submitted by taskcluster start with "B2G-gecko". There's probably going to be some naming collisions if those are actually for different branches. Eg: if some are based on "master" and some are based on "b2g37", B2G-gecko-$buildid isn't a unique enough name.
* The "aurora" channel rule for flame-kk is pointing at mozilla-b2g34. This one looks like it might be dead now, since most things point at B2G-gecko releases rather than B2G-mozilla-whatever
Flags: needinfo?(bhearsum)
Flags: needinfo?(wcosta)
I ended up changing some of the rules at comment 4:
* The nightly-latest and dogfood-latest have been updated to use B2G-gecko-nightly-latest
* The dogfood-test and nightly-test are used for QA to change to an OTA test build.  nightlytest moved to nightly-latest
* dogfood and nightly are for public consumption.

I did that so both channels conform rather than having nightlytest and dogfood-latest as it was confusing people.

Aurora has been dead for a while, 2.1, 2.2 has discontinued on pvtbuild, there's a build to have task cluster pick those up.

The comment 4 was in regards to the wrong build for the release and I believe it may have to do with the explanation of : 
"It looks like all of the releases submitted by taskcluster start with "B2G-gecko". There's probably going to be some naming collisions if those are actually for different branches. Eg: if some are based on "master" and some are based on "b2g37", B2G-gecko-$buildid isn't a unique enough name."

I think there's some collision happening; it would be great if we could get some sort of logging that we can look at to troubleshoot the issue...
Flags: needinfo?(mshal)
Flags: needinfo?(bhearsum)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #7)
> I ended up changing some of the rules at comment 4:
> * The nightly-latest and dogfood-latest have been updated to use
> B2G-gecko-nightly-latest
> * The dogfood-test and nightly-test are used for QA to change to an OTA test
> build.  nightlytest moved to nightly-latest
> * dogfood and nightly are for public consumption.
> 
> I did that so both channels conform rather than having nightlytest and
> dogfood-latest as it was confusing people.
> 
> Aurora has been dead for a while, 2.1, 2.2 has discontinued on pvtbuild,
> there's a build to have task cluster pick those up.
> 
> The comment 4 was in regards to the wrong build for the release and I
> believe it may have to do with the explanation of : 
> "It looks like all of the releases submitted by taskcluster start with
> "B2G-gecko". There's probably going to be some naming collisions if those
> are actually for different branches. Eg: if some are based on "master" and
> some are based on "b2g37", B2G-gecko-$buildid isn't a unique enough name."
> 
> I think there's some collision happening; it would be great if we could get
> some sort of logging that we can look at to troubleshoot the issue...

I'm not really sure where to point you here, I have very little knowledge on these b2g builds. https://github.com/mozilla/gecko-dev/blob/master/testing/taskcluster/scripts/phone-builder/build-phone-ota.sh looks like it might be involved? But I'm not really sure.

There's sections in logs like https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/vvxdielpQnG2Uv9U5-k-tA/0/public/logs/live_backing.log that says " Balrog request to http://balrog/api/releases/B2G-gecko-nightly-2015082702214", so if all branches use the same base string (B2G-gecko-nightly), it's pretty clear to me that there's collisions happening.
Flags: needinfo?(bhearsum)
:wcosta, do you know or can you point us to someone who knows how these builds are deciding what to submit to balrog? I'm out of my league here :/
Flags: needinfo?(mshal) → needinfo?(wcosta)
mshal, I have a hard time tracking what is actually updating the rule.  I think if we had some patches for logging what actually updates balrog, it might shed some clarity why there's a disparity.
Any progress on this? Dogfooding builds are now 3 weeks old (20150820200530).
This doesn't block dogfood.  Please see https://wiki.mozilla.org/B2G/QA/Spark for Aries or https://wiki.mozilla.org/B2G/QA/Flame_OTA#Nightly for Flame for the information in regards to what's blocking them.
Are the channel names "nightly test" and "nightly latest" (mind the spaces) correct under https://wiki.mozilla.org/B2G/QA/Flame_OTA#Nightly ?
:nhirata - it looks like both the flame-kk-ota-* builds (running in Taskcluster) and the flame nightly build (running in buildbot) are submitting updates to balrog. Is that intended? Or should we be turning off the flame nightlies in buildbot?

Separately, what kind of logging are you looking for? We have the build logs for both when submitting things to balrog, but I don't know if that has the info that you're looking for.
Flags: needinfo?(nhirata.bugzilla)
I think we want to remove flame-kk-ota-* from updating the latest, right?  Because I think the goal is to have Nightly builds... I could be mistaken.  We don't have a test server, do we?

Basically a log file stating which script updated the information for the releases.
Flags: needinfo?(nhirata.bugzilla)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #16)
> I think we want to remove flame-kk-ota-* from updating the latest, right? 

We can disable the flame-kk-ota-* builds in taskcluster for now. Is that acceptable? 

The only difference between the -ota- version of these builds is that it submits to balrog. :)  I realise that we would like to get these jobs moved over, but we have a couple dependencies that need to be resolved in Q4 before that can happen.
Flags: needinfo?(nhirata.bugzilla)
Sounds good!  I think disabling it would help.  We can turn it back on once the dependencies are done.  Thanks!
Flags: needinfo?(nhirata.bugzilla) → needinfo?(sdeckelmann)
Flags: needinfo?(sdeckelmann)
Whiteboard: [2.5-Daily-Testing] → [2.5-Daily-Testing] [tc-emulator]
Whiteboard: [2.5-Daily-Testing] [tc-emulator] → [2.5-Daily-Testing] [tc-build-support]
Whiteboard: [2.5-Daily-Testing] [tc-build-support] → [2.5-Daily-Testing], [b2g-build-support]
Agreed - let's disable the flame-kk-ota builds
Following up on this bug, can we have https://bugzilla.mozilla.org/show_bug.cgi?id=1188992#c17 done please?
If you want us to file a new bug in regards to that with a dependency on this one, we can do this.
Flags: needinfo?(sdeckelmann)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #20)
> Following up on this bug, can we have
> https://bugzilla.mozilla.org/show_bug.cgi?id=1188992#c17 done please?
> If you want us to file a new bug in regards to that with a dependency on
> this one, we can do this.

This was done in bug 1212812
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
This issue can repro on the Flame KK v2.5 user build by the following STR.
STR:
1. Update Flame to  20151014150203.                                                   
2. Connect to a WiFi or Data network.
3. Tap on the Setting app on the homescreen.
4. Tap Developer button on the Settings.
5. In the Update Channel, Change the update channel to 'nightlytest'.
6. Tap Check Now button on the Device Information.

Actual results: No system updates are found.
See attachment: verified_logcat_2039.txt. verified_Flame KK_v2.5.3gp
Reproduce rate: 10/10

Device: Flame KK 2.5 user(Affected)
Build ID               20151014150203
Gaia Revision          c64d6e345aa477f2a24b676e84a47020a00d015d
Gaia Date              2015-10-14 15:54:16
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/4f4615ffec6a6a7ec40ff61ffda90a46c53f8d31
Gecko Version          44.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20151014.184235
Firmware Date          Wed Oct 14 18:42:47 EDT 2015
Firmware Version       v18D v4
Bootloader             L1TC000118D0
QA Whiteboard: [MGSEI-Triage+]
(In reply to Tony from comment #22)
> This issue can repro on the Flame KK v2.5 user build by the following STR.
> STR

So are you saying that this is broken still despite bug 1212812?
Flags: needinfo?(sdeckelmann)
Flags: needinfo?(mshuman)
(In reply to Tony from comment #22)
> This issue can repro on the Flame KK v2.5 user build by the following STR.
> STR:
> 1. Update Flame to  20151014150203.                                         
> 
> 2. Connect to a WiFi or Data network.
> 3. Tap on the Setting app on the homescreen.
> 4. Tap Developer button on the Settings.
> 5. In the Update Channel, Change the update channel to 'nightlytest'.
> 6. Tap Check Now button on the Device Information.
> 
> Actual results: No system updates are found.
> See attachment: verified_logcat_2039.txt. verified_Flame KK_v2.5.3gp
> Reproduce rate: 10/10
> 
> Device: Flame KK 2.5 user(Affected)
> Build ID               20151014150203
> Gaia Revision          c64d6e345aa477f2a24b676e84a47020a00d015d
> Gaia Date              2015-10-14 15:54:16
> Gecko Revision        
> https://hg.mozilla.org/mozilla-central/rev/
> 4f4615ffec6a6a7ec40ff61ffda90a46c53f8d31
> Gecko Version          44.0a1
> Device Name            flame
> Firmware(Release)      4.4.2
> Firmware(Incremental)  eng.cltbld.20151014.184235
> Firmware Date          Wed Oct 14 18:42:47 EDT 2015
> Firmware Version       v18D v4
> Bootloader             L1TC000118D0

Hi, Tony,

To my knowledge, we were using PVT build to do Flame OTA test because of some test purposes.
So, I am not sure if this problem relates to PVT build.

Can we try to use task cluster build to do Flame OTA test again?
Thank you.
Flags: needinfo?(tianxu)
Hi, William,
  Thanks for your reminding. With William's help, this bug has been verified as "pass" on the old build of Flame KK 2.5 user(Build ID 20151020150236). Please check the link below for verification information. thanks.

URI: https://aus5.mozilla.org/update/5/B2G/44.0a1/20151020003809/flame/en-US/nightly-test/Boot2Gecko 2.5.0.0-prerelease (SDK 19)/default/default/default/update.xml?force=1.
Actual results: system updates are found.
Reproduce rate: 0/10.


Device: Flame KK 2.5 user(Pass)
Build ID               20151020150236
Gaia Revision          11eb5d4cb2675d359d277ae17772bc75f7ccedbc
Gaia Date              2015-10-20 16:22:28
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/f397034950304b845175f8dab8fccbdd0e8bf995
Gecko Version          44.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20151020.184701
Firmware Date          Tue Oct 20 18:47:13 EDT 2015
Firmware Version       v18D v4
Bootloader             L1TC000118D0
Flags: needinfo?(tianxu)
Status: RESOLVED → VERIFIED
status-b2g-master: affected → verified
Flags: needinfo?(whsu)
Thank you, everyone.
Looks good to me.
Flags: needinfo?(whsu)
Flags: needinfo?(mshuman)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: