Closed Bug 1111890 Opened 9 years ago Closed 9 years ago

[Nexus 5] Create Lollipop based pvtbuilds for 2.2

Categories

(Release Engineering :: General, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(blocking-b2g:2.2+, b2g-v2.2 fixed, b2g-master fixed)

RESOLVED FIXED
blocking-b2g 2.2+
Tracking Status
b2g-v2.2 --- fixed
b2g-master --- fixed

People

(Reporter: tchung, Assigned: catlee)

References

Details

(Whiteboard: [leave open])

Attachments

(11 files)

1.02 KB, patch
hwine
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
6.62 KB, patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
12.75 KB, patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
816 bytes, patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.02 KB, patch
pmoore
: review+
pmoore
: checked-in+
Details | Diff | Splinter Review
650 bytes, patch
catlee
: review+
Details | Diff | Splinter Review
1.10 KB, patch
nhirata
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.35 KB, patch
catlee
: review+
Details | Diff | Splinter Review
52 bytes, text/x-github-pull-request
seinlin
: review+
Details | Review
1.89 KB, patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
766 bytes, patch
nhirata
: review-
Details | Diff | Splinter Review
+++ This bug was initially created as a clone of Bug #1055305 +++

[Blocking Requested - why for this release]:

We need to create Nexus 5 builds against pvtbuilds server, since this is a defined device for testing and development in 2.2 timeframe.   They should have builds for both Master/mozilla-central and 2.2 branch (when it becomes a branch).   This will block 2.2 work.

Who can own this?   (ie. https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/)
Flags: needinfo?(sku)
Flags: needinfo?(kkuo)
Flags: needinfo?(khu)
Flags: needinfo?(bbajaj)
Blocks: 1098983
marking 2.2+, since this will block testing on a affected 2.2 device.
blocking-b2g: 2.2? → 2.2+
I think this is best done in Task Cluster.
Component: General Automation → TaskCluster
Product: Release Engineering → Testing
QA Contact: catlee
(In reply to Tony Chung [:tchung] from comment #0)
> +++ This bug was initially created as a clone of Bug #1055305 +++
> 
> [Blocking Requested - why for this release]:
> 
> We need to create Nexus 5 builds against pvtbuilds server, since this is a
> defined device for testing and development in 2.2 timeframe.   They should
> have builds for both Master/mozilla-central and 2.2 branch (when it becomes
> a branch).   This will block 2.2 work.
> 
> Who can own this?   (ie.
> https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/)

Hi Tony:
 Since there is no WIP allowed on Moz git repo. We are still working on make build pass w/o WIP. Once we get it done, manifest for nexus5-l can be ready.
(The ETA is 2 weeks to make manifest ready per my team's perspective.)

We can have a discussion about PVT build then.

Thanks!!
Shawn
Flags: needinfo?(bbajaj)
(In reply to shawn ku [:sku] from comment #3)
> (In reply to Tony Chung [:tchung] from comment #0)
> > +++ This bug was initially created as a clone of Bug #1055305 +++
> > 
> > [Blocking Requested - why for this release]:
> > 
> > We need to create Nexus 5 builds against pvtbuilds server, since this is a
> > defined device for testing and development in 2.2 timeframe.   They should
> > have builds for both Master/mozilla-central and 2.2 branch (when it becomes
> > a branch).   This will block 2.2 work.
> > 
> > Who can own this?   (ie.
> > https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/)
> 
> Hi Tony:
>  Since there is no WIP allowed on Moz git repo. We are still working on make
> build pass w/o WIP. Once we get it done, manifest for nexus5-l can be ready.
> (The ETA is 2 weeks to make manifest ready per my team's perspective.)
> 
> We can have a discussion about PVT build then.
> 
> Thanks!!
> Shawn

Tony once we are able to complete the porting work, we can talk to releng to have the pvt builds..but the first step isn't done yet :(
(In reply to bhavana bajaj [:bajaj] from comment #4)
> Tony once we are able to complete the porting work, we can talk to releng to
> have the pvt builds..but the first step isn't done yet :(

So, when we're ready to talk with releng, who could own the communication? Bhavana, are you the best one to own it? Thanks.
Flags: needinfo?(khu) → needinfo?(bbajaj)
Bug 1098983 is resolved fixed. PVT build can be proceeded IMO.
(In reply to shawn ku [:sku] from comment #6)
> Bug 1098983 is resolved fixed. PVT build can be proceeded IMO.

Yes, it's time that PVT build can be processed.

--
Keven
Flags: needinfo?(kkuo)
:catlee or james, not sure if taskcluster is ready or we still need help from catlee's team to support this, so NI'ing both to get this bug moving.
Flags: needinfo?(jlal)
Flags: needinfo?(catlee)
Flags: needinfo?(bbajaj)
Please let me know if any help from gonk team is needed.
(clear ni? for myself first.)
Flags: needinfo?(sku)
Depends on: 1121839
Depends on: 1119980
In addition to the manifest added in bug 1098983, the build must succeed using mozharness for either automation system to be able to schedule it.

From reading above, it is not clear if that work has been done.

ni :kkuo - can you give status of mozharness integration, please?
Flags: needinfo?(kkuo)
we'll most likely need binary blobs extracted from the device before we can have builds in automation. QA has generally done that for us, and then we can put them into tooltool and in-tree manifests like this:
http://hg.mozilla.org/mozilla-central/file/38e4719e71af/b2g/config/nexus-4/releng-mako.tt

James, are you in a position to take these on yet?
Flags: needinfo?(catlee)
To ease infrastructure load, it would be helpful if at least one of the older builds can be disabled. See bug 1121495 for list.
Depends on: 1121495
(In reply to Hal Wine [:hwine] (use needinfo) from comment #10)
> In addition to the manifest added in bug 1098983, the build must succeed
> using mozharness for either automation system to be able to schedule it.
> 
> From reading above, it is not clear if that work has been done.
> 
> ni :kkuo - can you give status of mozharness integration, please?

Hal,
 mozharness parts were done by catlee's team.
If need device team to finish this task, please kindly let us know.

Seinlin:
 Please follow up the things we need to be done.
Flags: needinfo?(kkuo) → needinfo?(kli)
(In reply to Chris AtLee [:catlee] from comment #11)
> we'll most likely need binary blobs extracted from the device before we can
> have builds in automation. QA has generally done that for us, and then we
> can put them into tooltool and in-tree manifests like this:
> http://hg.mozilla.org/mozilla-central/file/38e4719e71af/b2g/config/nexus-4/
> releng-mako.tt

From email thread after b2g-drivers meeting:

On 2015-01-27 07:44 , Naoki wrote:

Ok thanks!  I noticed that there's a config called : nexus-5-l
  ie. : ./config.sh nexus-5-l 
for a new local repo after cloning the B2G repo ( ./repo init nexus-5-l for
reconfiguring a repo) and that should automatically pull the correct lollipop
parts from what I see.

I gave it a try and it indeed does pull all the vendor drivers down.

Hal, if you need any help please let me know.  It seems that everything that needs to be done is already scripted for the blobs for the N5 to go to Lollipop.
Assignee: nobody → catlee
Component: TaskCluster → General Automation
Product: Testing → Release Engineering
QA Contact: catlee
Depends on: 1126491
Chris, I try to upload nexus-5 binary to tooltool but I don't have the permission. I think you can download it from nexus binary site. Now our porting is based on 'LRX21O'. Please ni me if necessary.
Flags: needinfo?(kli)
Depends on: 1127166
I think these bugs need to land to make a proper build:
bug 1127166 and bug 1126499
Now temporary fix of bug 1119980 is landed, it doesn't block build.
No longer depends on: 1119980
Depends on: 1127998
Depends on: 1128002
Attachment #8557260 - Flags: review?(hwine)
Comment on attachment 8557260 [details] [diff] [review]
bump nexus-5-l on master

5-(lowercase-L) lgtm
Attachment #8557260 - Flags: review?(hwine) → review+
Attachment #8557260 - Flags: checked-in+
I think bug 1128113 needs to land as well?

Once I had that patch I was able to successfully build a N5L build.
Attachment #8559132 - Flags: review?(rail)
Comment on attachment 8559132 [details] [diff] [review]
in-tree configs for nexus-5

stamp
Attachment #8559132 - Flags: review?(rail) → review+
Attachment #8559132 - Flags: checked-in+
Attachment #8559273 - Flags: review?(rail)
need to shorten these otherwise we die on this builder:
b2g_mozilla-b2g34_v2_1s_nexus-5-l_eng_dep

which is weird because I'm not enabling it on that branch...I think the builder is created, but not added anywhere in misc.py
Attachment #8559276 - Flags: review?(rail)
Attachment #8559273 - Flags: review?(rail) → review+
Attachment #8559276 - Flags: review?(rail) → review+
Whiteboard: [leave open]
Turns out that we can't recover the phone, due to the apps being on the data partiion.

Could you include B2G_SYSTEM_APPS=1 variable in the build script please? see bug 1128836 comment 22~24 for more details.
Should I file a different bug for comment 29?  ( please needinfo me if that is the case )
Please uplift to 2.2, thanks.
Attachment #8559778 - Flags: review?(pmoore)
Attachment #8559273 - Flags: checked-in+
Attachment #8559276 - Flags: checked-in+
++ I Just want to thank catlee/rail here I dropped the ball
Flags: needinfo?(jlal)
Patch for building with VARIANT=userdebug + B2G_SYSTEM_APPS=1 based on comment 29.
The nexus-5-l.zip is only 27KB in size, it seems a little bit too small.

nexus-5-l.zip	06-Feb-2015 01:40 	27K	 

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-nexus-5-l-eng/2015/02/2015-02-05-16-04-02/
Comment on attachment 8560181 [details] [diff] [review]
nexus-5-l_build.patch

Review of attachment 8560181 [details] [diff] [review]:
-----------------------------------------------------------------

r+, but can you update the entries in the "zip_files" section that reference mako and replace them with hammerhead?
Attachment #8560181 - Flags: review?(catlee) → review+
Attachment #8560666 - Flags: review?(nhirata.bugzilla)
Comment on attachment 8560666 [details] [diff] [review]
update files to include in zip

Review of attachment 8560666 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.  Sorry I didn't see that section the first time around.
Attachment #8560666 - Flags: review?(nhirata.bugzilla) → review+
Attachment #8560666 - Flags: checked-in+
Chris, Now we can see nexus-5-l builds are on m/c. Can we also have it on v2.2? Thanks!
Flags: needinfo?(catlee)
Comment on attachment 8559778 [details] [diff] [review]
bump manifests on b2g37/v2.2

Review of attachment 8559778 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for delay in reviewing this! I'll check it in too, to compensate. =)
Attachment #8559778 - Flags: review?(pmoore) → review+
Comment on attachment 8559778 [details] [diff] [review]
bump manifests on b2g37/v2.2

I've landed on default, but not merged to production since there is quite a backlog (http://hg.mozilla.org/build/mozharness/graph), and this is no doubt better handled by build duty.

On default:
https://hg.mozilla.org/build/mozharness/rev/cc61099cd96a
Attachment #8559778 - Flags: checked-in+
I saw comment 39.  Do we need to do anything else to have 2.2 builds for Nexus L on pvtbuilds?  
ie is there anything else blocking us?  Please let us know.  :)
Hi Naoki,

We required a merge of default branch -> production branch in mozharness, and this has now been done:
http://hg.mozilla.org/build/mozharness/rev/937da8fabeaf

So I believe this change should be active now.

Thanks,
Pete
Flags: needinfo?(pmoore)
Are the builds on master functional now?
Flags: needinfo?(catlee)
For what I know,
1. We cannot boot from user build, but engineering build is okay.
2. We have issues when trying to run gaia-ui-test automation, issues with test cases with keyboard, we just cannot enter text with it by automation, and more than 50% of our test cases are failed.

Hermes should have more details on this.
Flags: needinfo?(hcheng)
(In reply to Eric Chang [:ericcc] [:echang] from comment #46)
> For what I know,
> 1. We cannot boot from user build, but engineering build is okay.
This problem is not confirmed. I will flash another phone with user build again.
> 2. We have issues when trying to run gaia-ui-test automation, issues with
> test cases with keyboard, we just cannot enter text with it by automation,
> and more than 50% of our test cases are failed.
> 
> Hermes should have more details on this.

For the pvt build on master, I will perform a test run and send out test report.
Flags: needinfo?(hcheng)
Ok, thanks Pmoore.  From the sounds of things, we just need to wait for the next build cycle.

Catlee, we'll (QA: TW/US) take a look into it.
Parul, if you have the time please take a look as well.
Flags: needinfo?(pmathur)
(In reply to Hermes Cheng[:hermescheng] from comment #47)
> (In reply to Eric Chang [:ericcc] [:echang] from comment #46)
> > For what I know,
> > 1. We cannot boot from user build, but engineering build is okay.
> This problem is not confirmed. I will flash another phone with user build
> again.
> > 2. We have issues when trying to run gaia-ui-test automation, issues with
> > test cases with keyboard, we just cannot enter text with it by automation,
> > and more than 50% of our test cases are failed.
> > 
> > Hermes should have more details on this.
> 
> For the pvt build on master, I will perform a test run and send out test
> report.


After flash my Nexus 5 with user build, the phone cannot reboot. Please see bug 1132837.
(In reply to Pete Moore [:pete][:pmoore] from comment #44)
> Hi Naoki,
> 
> We required a merge of default branch -> production branch in mozharness,
> and this has now been done:
> http://hg.mozilla.org/build/mozharness/rev/937da8fabeaf
> 
> So I believe this change should be active now.
> 
> Thanks,
> Pete

Hi Pete,

It seems not build on v2.2 now. Could you please check it? Thanks.


@Chris, in my last test run on pvt master build, the builds on master is ok.
* Test env info
Build ID 20150212160523
Gaia Revision 2a2b008f9ae957fe19ad540d233d86b5c0b6829e
Gaia Date 2015-02-12 07:44:13
Gecko Revision https://hg.mozilla.org/mozilla-central/rev/2f5c5ec1a24b
Gecko Version 38.0a1
Device Name hammerhead
Firmware(Release) 5.0
Firmware(Incremental) eng.cltbld.20150212.192802
Firmware Date Thu Feb 12 19:28:15 EST 2015
Bootloader HHZ12d
Flags: needinfo?(pmoore)
Catlee,

Should we also add something like:

BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l']['enable_nightly'] = True
BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l_eng']['enable_nightly'] = True
BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l_eng']['consider_for_nightly'] = False

to:

https://hg.mozilla.org/build/buildbot-configs/rev/5e87f62bd6a4#l1.117 ?

Not sure what "consider_for_nightly" is.

Pete
Flags: needinfo?(pmoore) → needinfo?(catlee)
Catlee,

We need to use nexus binary files in our build. But I didn't see them in b2g/config/nexus-5-l. I already uploaded these files to /tooltool/uploads/kli/pvt.

Could you help to review and land this if it is really missing?

Thanks!
Attachment #8565113 - Flags: review?(catlee)
Attachment #8565113 - Flags: review?(catlee) → review+
(In reply to Pete Moore [:pete][:pmoore] from comment #52)
> Catlee,
> 
> Should we also add something like:
> 
> BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l']['enable_nightly'] =
> True
> BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-
> l_eng']['enable_nightly'] = True
> BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-
> l_eng']['consider_for_nightly'] = False
> 

Yes, but we'll also need config.json and releng-nexus5.tt changes ported over to mozilla-b2g37_v2_2 before these will be successful.

> to:
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/5e87f62bd6a4#l1.117 ?
> 
> Not sure what "consider_for_nightly" is.

It determines which builds are considered for choosing a "good" nightly revision to use. If any build with "consider_for_nightly" has failed for a revision, then that revision won't be used for subsequent nightly builds.
Flags: needinfo?(catlee)
Pull request to correct a nonexistant revision for v2.2 nexus.
Attachment #8566495 - Flags: review?(kli) → review+
https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/9140d8c61635
https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/df38c1e22633

Note that the status flags here are being set for the in-tree patches. We'll leave the bug open until they're enabled in production.
Thanks, Ryan!  

Ok, it sounds like we have everything in place.  We just have to wait for the builds?
Attached patch enable on 2.2Splinter Review
Attachment #8568073 - Flags: review?(rail)
Attachment #8568073 - Flags: review?(rail) → review+
Attachment #8568073 - Flags: checked-in+
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #34)
> Created attachment 8560181 [details] [diff] [review]
> nexus-5-l_build.patch
> 
> Patch for building with VARIANT=userdebug + B2G_SYSTEM_APPS=1 based on
> comment 29.

Could you check in this patch for Bug 1128836?
Flags: needinfo?(nhirata.bugzilla)
NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: pvtbuilds for 2.2 nexus-5-l won't show apps/boot properly
Testing completed: 
Risk to taking this patch (and alternatives if risky): 
String or UUID changes made by this patch:
Flags: needinfo?(nhirata.bugzilla)
Attachment #8568844 - Flags: review?(kli)
Attachment #8568844 - Flags: approval-mozilla-b2g37?
When B2G_SYSTEM_APPS = 1 in userconfig, apps will always be installed to '/system/b2g/webapps'. This patch will affect to 'eng' build only, because in 'user' and 'userdebug' B2G_SYSTEM_APPS is already set to 1 by default. Is it the same with what you expected?

BTW, I think process this in a different bug could be better for tracking.
Flags: needinfo?(nhirata.bugzilla)
I think we do want the engineering builds to be default to the system partition, don't we?  Somewhere along the lines I think we did have this discussion before on another thread when we ran out of space on the system partition...

Hermes had asked for this patch to be placed on 2.2 as well.  Could you talk to Hermes in person also to see if this is a blocker for him or not?

If we don't need this additional patch to make sure we have working user(debug) and eng builds, then I'm fine not pushing it.
Flags: needinfo?(nhirata.bugzilla)
Flags: needinfo?(kli)
Flags: needinfo?(hcheng)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #65)
> I think we do want the engineering builds to be default to the system
> partition, don't we?  Somewhere along the lines I think we did have this
> discussion before on another thread when we ran out of space on the system
> partition...
> 
> Hermes had asked for this patch to be placed on 2.2 as well.  Could you talk
> to Hermes in person also to see if this is a blocker for him or not?
> 
> If we don't need this additional patch to make sure we have working
> user(debug) and eng builds, then I'm fine not pushing it.

Sorry, it does not block me now after I have pvt build instead of local build for v2.2.
Flags: needinfo?(kli)
Flags: needinfo?(hcheng)
Comment on attachment 8568844 [details] [diff] [review]
b2gsystemapp_v2.2.patch

Review of attachment 8568844 [details] [diff] [review]:
-----------------------------------------------------------------

I have no particular opinion on the patch and I also think the patch is also good to land. 

But I notice all other devices build didn't set B2G_SYSTEM_APPS and all apps are under /system/b2g. In nexus-5-l apps are also under /system/b2g. 

Finally, I found the answer in [1], all device eng build use releng-otoro-eng.py, and B2G_SYSTEM_APPS is set to 1 there. Now I can sure we don't need to land this patch.

[1] http://hg.mozilla.org/build/mozharness/file/f2510cc466c0/configs/b2g/releng-otoro-eng.py#l73
Attachment #8568844 - Flags: review?(kli)
Comment on attachment 8568844 [details] [diff] [review]
b2gsystemapp_v2.2.patch

minusing approval and review then because we don't need the patch.
Attachment #8568844 - Flags: review-
Attachment #8568844 - Flags: approval-mozilla-b2g37?
Attachment #8568844 - Flags: approval-mozilla-b2g37-
I tested the builds, they are flashable.

I think there's something odd about the flash script (at least running on the mac).  It didn't quite seem to flash or was taking a really long time so I ran the fastboot flash manually.

I think we might have to update the flash scripts, which we can handle in a different bug.  I think this bug can be marked resolved/verified fixed.
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(pmathur)
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: