Robert Chira [:RobertC] Reporter
	Description • 9 years ago

We are seeing an intermittent issue in the python automation tests where dialer fails to make an outgoing call.
After typing the phone number and tapping call a different callscreen is displayed (screenshot attached).

Issue reproduced locally with automation: 4 out of 10.
I was not able to reproduce the issue manually after 20 attempts.

Build info m-c v188-1 319MB Flame KK:
Gaia-Rev        b02ec9713e6de8d96c6954d2c0dfd0442b0656ac
Gecko-Rev       https://hg.mozilla.org/mozilla-central/rev/38e4719e71af
Build-ID        20150126160233
Version         38.0a1
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20150126.192457
FW-Date         Mon Jan 26 19:25:08 EST 2015
Bootloader      L1TC10011880

STR:
1. open dialer app
2. type phone number
3. tap call button

Expected result:
Callscreen is displayed and outgoing call works as expected

Actual result:
The callscreen has a different layout (phone number missing as well as call type) and the call fails, the displayed app after this is the dialer.

The issue is not reproduced on base build v18D

	Robert Chira [:RobertC] Reporter
	Comment 1 • 9 years ago

Logcat of the automated test failing locally.

	Johan Lorenzo [:jlorenzo]
	Comment 2 • 9 years ago

Would you mind providing a job where we see the call screen failing?

	Robert Chira [:RobertC] Reporter
	Comment 3 • 9 years ago

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-inbound.ui.functional.sanity/3274/HTML_Report/

The test fails at a later step, after the call screen closes so the image is different.
The screenshot that is attached was taken manually on my device while the test was running.

	Robert Chira [:RobertC] Reporter
	Comment 4 • 9 years ago

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.ui.functional.non-smoke.2/148/HTML_Report/

In that test run this failure was also encountered at incoming call.

	Johan Lorenzo [:jlorenzo]
	Comment 5 • 9 years ago

I reproduce this issue manually by chance today on master.

[Blocking Requested - why for this release]: Regression that an end user would hit.

	Johan Lorenzo [:jlorenzo]
	Comment 6 • 9 years ago

QA wanted for a branch check.

	KTucker [:KTucker][Inactive 3/4/2016]
	Comment 7 • 9 years ago

I was unable to reproduce this issue on Flame 2.2 after 50 plus attempts

The correct call screen was shown and they call went through as expected. 

Environmental Variables:
Device: Flame 2.2 (Full Flash)(KK)(319mb)
BuildID: 20150128002506
Gaia: cd42b034fd2825c3675ace3a67f5775eb61c2d60
Gecko: d824c65a6a2b
Gonk: e7c90613521145db090dd24147afd5ceb5703190
Version: 37.0a2 (2.2) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Going to get a regression window on this.

	Wesley Huang [:wesley_huang] (EPM) (NI me)
	Comment 8 • 9 years ago

triage: blocker

	Jayme Mercado [:JMercado] (Inactive 3/4/2016)
	Comment 9 • 9 years ago

I could not get this issue to reproduce on the latest build as it seems to be fixed.  Using the automation tests I was able to reproduce this bug on builds from the 26th easily, but on the latest working central build I repeated the script 40 times with 0 reproductions.  Removing the regression window tag for now as this issue seems to be fixed.  

Robert Chira if you see this issue again please add the tag again and I will find the regression window.

Issue does NOT occur:

Environmental Variables:
Device: Flame 3.0
BuildID: 20150128181137
Gaia: 9d2378a9ef092ab1fc15c3a9f7fc4171aab59d57
Gecko: ea54e6623f59
Version: 38.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0

Issue DOES occur

Environmental Variables:
Device: Flame 3.0
BuildID: 20150126134900
Gaia: b02ec9713e6de8d96c6954d2c0dfd0442b0656ac
Gecko: 38e4719e71af
Version: 38.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0

	KTucker [:KTucker][Inactive 3/4/2016]
	Comment 10 • 9 years ago

Closing this as worksforme. Please reopen if the issue is encountered again.

	Johan Lorenzo [:jlorenzo]
	Comment 11 • 9 years ago

Encountered this issue again on today's master, with automation:
Gaia-Rev        8238eeacc7030b2cdbf7ab4eba2f36779b702599
Gecko-Rev       https://hg.mozilla.org/mozilla-central/rev/29b05d283b00
Build-ID        20150130010210
Version         38.0a1
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20150130.044333
FW-Date         Fri Jan 30 04:43:44 EST 2015
Bootloader      L1TC000118D0

	KTucker [:KTucker][Inactive 3/4/2016]
	Comment 12 • 9 years ago

Adding qawanted so we can get better logs.

	KTucker [:KTucker][Inactive 3/4/2016]
	Comment 14 • 9 years ago

Let's try to get a regression window again.

	Stoyan Dimitrov [:stoyan]
	Comment 15 • 9 years ago

My experience with this bug is mainly in v2.2 and with incoming calls. The device is ZTE OPEN. It is 8/10 times reproducible (if not more) when given contact calls me (by chance this is my wife). The contact has image. The image is taken with the device camera and is not resized. My feeling is that it is performance issue.

	Jayme Mercado [:JMercado] (Inactive 3/4/2016)
	Comment 16 • 9 years ago

Bug 1116588 seems to be the cause for this issue.

Mozilla-inbound Regression Window

Last Working 
Environmental Variables:
Device: Flame 3.0
BuildID: 20150126062631
Gaia: 793773bb2944b42a85dd160049e605cbd880c4da
Gecko: 0bec74187553
Version: 38.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0

First Broken 
Environmental Variables:
Device: Flame 3.0
BuildID: 20150126065334
Gaia: 793773bb2944b42a85dd160049e605cbd880c4da
Gecko: babd56077826
Version: 38.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0

Last Working gaia / First Broken gecko - Issue DOES occur
Gaia: 793773bb2944b42a85dd160049e605cbd880c4da
Gecko: babd56077826

First Broken gaia / Last Working gecko - Issue does NOT occur
Gaia: 793773bb2944b42a85dd160049e605cbd880c4da
Gecko: 0bec74187553

Gecko Pushlog: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=0bec74187553&tochange=babd56077826

	KTucker [:KTucker][Inactive 3/4/2016]
	Comment 17 • 9 years ago

Kartikaya, can you take a look at this please? Looks like the work done on bug 1116588 might have caused this issue to occur.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 18 • 9 years ago

1) What STR were used to find the regression window in comment 16?
2) The cset 7c4059ebfcf7 identified in comment 16 was backed out two days later (see bug 1116588 comment 29), so please re-check if this is still happening on master. If it is, it may have come back with the second patch on bug 1116588 which was a modified version of the original patch.

	Jayme Mercado [:JMercado] (Inactive 3/4/2016)
	Comment 19 • 9 years ago

The test_dialer python automation script was used to reproduce this bug for the regression window.  If the issue did not reproduce within 40 retries, the build was considered working for the regression window.

In comment 9 this issue had been fixed on master on the 28th of January, but the issue began occurring again as of comment 11 on the 30th of January.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 20 • 9 years ago

How can I run the test_dialer python automation script? Are there instructions somewhere?

	Jayme Mercado [:JMercado] (Inactive 3/4/2016)
	Comment 21 • 9 years ago

The steps we use to do this were modified from http://blargon7.com/2013/07/running-firefox-os-ui-tests-without-a-device-revised/ but to use with a device.  Maybe Naoki can give you more correct guidance.

	Naoki Hirata :nhirata (please use needinfo instead of cc)
	Comment 22 • 9 years ago

kats, you'll have to do some prep work : 
https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Setting_up_Marionette

you can then run on device : 
https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests#Running_the_tests_2

Dialer tests are located here : 
https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/dialer

If you run into issues, please let me know.

	Florin Strugariu [:Bebe]
	Comment 24 • 9 years ago

Comment on attachment 8560438 [details] [review]
[PullReq] bebef1987:dial1 to mozilla-b2g:master

I updated some waits in our pull

We wait for the main window and the call screen to be displayed and then try to interact with it.

I think this will fix our test fails.

	Johan Lorenzo [:jlorenzo]
	Comment 25 • 9 years ago

Comment on attachment 8560438 [details] [review]
[PullReq] bebef1987:dial1 to mozilla-b2g:master

This still looks like an actual bug in the production code, to me. I ran another adhoc job and test_dialer_clear_miss_call_notification.TestDialerClearMissCallNotification failed 4 out of 11 times with the same screenshot: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/681/HTML_Report/

By the way, most of the test failed there probably due to connectivity issues. Could it be a race condition depending on the network coverage?

	Geo Mealer [:geo] -- This account is inactive after 2015-07-07
	Comment 26 • 9 years ago

Comment on attachment 8560438 [details] [review]
[PullReq] bebef1987:dial1 to mozilla-b2g:master

Going to r- based on Johan's estimation that we might be covering up a real product bug.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 27 • 9 years ago

Sorry for the delay. I finally got some time to go through these instructions but for some reason when I run the gaiatest app it can't connect to marionette on the device. The connection to port 2828 fails. Do I need to do something in particular to make something on the device listen to that port? I built gecko with ENABLE_MARIONETTE=1 and as far as I can tell that's all I should need.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 28 • 9 years ago

Oh, ignore comment 27. I was port-forwarding through SSH and I screwed up the tunneling. It seems to be running now.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 29 • 9 years ago

So I ran all of the dialer tests and got 19 failures and 4 passes. Pretty sure that's not normal, so I don't know if my results are representative of what's happening in automation. However test_dialer.py was one of the failures and I can run it in isolation with a 100% failure rate (3/3 so far). It fails with the attached output. Looking up in this bug I don't see any information about how this test is failing in automation so I can't say if debugging the failure I'm seeing locally will be of any use. If anybody has more info please let me know.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 30 • 9 years ago

A little more investigation indicates that more likely my testvars file doesn't match what is used in automation, and also I don't have a SIM for the phone so I don't know if I can even reproduce the right failure locally.

	Johan Lorenzo [:jlorenzo]
	Comment 31 • 9 years ago

Not related to the call log db refactor.

	Naoki Hirata :nhirata (please use needinfo instead of cc)
	Comment 32 • 9 years ago

Florin or RobertC, can you send kats your testvars file ( you can remove anything that doesn't relate to this particular test )?

	Johan Lorenzo [:jlorenzo]
	Comment 33 • 9 years ago

I just sent a private email to Kats with the information contained in the testvars.json needed here.

	Naoki Hirata :nhirata (please use needinfo instead of cc)
	Comment 34 • 9 years ago

Thanks, Johan!

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 35 • 9 years ago

I used a Nexus 4 device with a SIM card and the testvars.json that Johan sent me to run through the dialer tests, and stuff was passing. At least, I didn't run into any of the errors in the report that Johan sent me. That report indicated an "Expected Failure" in test_call_log_groups, which I saw. It also indicated "Error" results in test_dialer_find_contact, test_dialer_clear_miss_call_notification, and test_redial_last_number. I didn't see any of these. I did see a failure in test_dialer_clear_miss_call_notification because the python code couldn't find the "plivo" module but that seems like a setup problem. I'm attaching the output I got from running these three tests locally. I also ran test_dialer.py locally a few times and that passed fine.

Unless somebody can point out if I'm doing something wrong, I think the only way forward here is for somebody to debug the issue on a machine that can reproduce it.

	Doug Sherk (:drs) (inactive)
	Comment 36 • 9 years ago

Johan, could you drive checking if this is still an issue, before we commit to investigating and fixing it?

	Johan Lorenzo [:jlorenzo]
	Comment 37 • 9 years ago

I ran the tests again, and the issue can't be seen anymore[1]. There are some failures, but none of them shows the "blank" call screen. I also don't remember seeing this issue recently in the reports. Let's close this bug and reopen it if it comes back.

[1] http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/797/HTML_Report/

	Derek Harris [:DerekH] - Dont need info. Last day 6/5/15
	Comment 38 • 9 years ago

I was able to reproduce this bug today on Flame 3.0. I have been seeing this bug about twice a week for the last couple weeks, but about a month ago I was seeing it almost every day. It seems as though the repro rate has dropped, but I still dont know any manual STR except that it seems to only happen one time during the first couple calls of a fresh flash.

When making an outgoing call the callscreen appeared blank, with broken UI elements

Environmental Variables:
Device: Flame 3.0 (319mb)(Kitkat)(Full Flash)
Build ID: 20150421010201
Gaia: a8e4f95dce9db727dda5d408b038f97fb4296557
Gecko: 7b823253d9f2
Gonk: b83fc73de7b64594cd74b33e498bf08332b5d87b
Version: 40.0a1 (3.0)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:40.0) Gecko/40.0 Firefox/40.0

Reopening this bug

	Doug Sherk (:drs) (inactive)
	Comment 39 • 9 years ago

Bhavana, this should block Spark.

Gabriele, could you take a look into this?

	Gabriele Svelto [:gsvelto] Assignee
	Comment 40 • 9 years ago

Sure, I'll have a look though I've never encountered this issue in practice.

	Peter Bylenga [:PBylenga]
	Comment 41 • 9 years ago

NI on Johan to take a look.

	Johan Lorenzo [:jlorenzo]
	Comment 42 • 9 years ago

I found an accurate (5/5 times) way to reproduce the problem. Here are the steps with today's build[1]:

1. Set the memory to 260MB (That's correct, this value is not supposed to be supported. I'll explain the reason of this value below).
2. Run test_dialer.py[2]

How did we come to 260MB?
* I tried to repro the problem like usual, by repeating multiple times the test with a restart between each test, on a 319MB device => 0/10 repro.
* Same thing without a restart in between => 0/10 repro
* Still no restart in between, on a 300MB device => 0/10 repro
* Same thing on 280MB => Repro'd at the 3rd attempt
* Same thing on 260MB => 5/5 times reproduced.

I got a logcat with RIL and Gaia debug traces (attached).
At first, the RIL doesn't seem the cause of the issue. Even with the blank call screen, I am able to hear the phone call and I can hang up by pressing the end call button. 

I haven't managed to see something in the gaia traces. Do you, Gabriele?

[1] Build ID               20150503160200
Gaia Revision          e18cce173840d6ff07fb6f1f0e0ffb58b99aab3e
Gaia Date              2015-05-02 04:27:01
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/dc5f85980a82
Gecko Version          40.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150503.193941
Firmware Date          Sun May  3 19:39:52 EDT 2015
Bootloader             L1TC000118D0

[2] https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/dialer/test_dialer.py
In gaia:
> cd tests/python/gaia-ui-tests
> virtualenv .env && source .env/bin/activate/ && python setup.py develop && pip install plivo
Edit testvars.json to fill in "remote_phone_number" and the plivo token. PM me if you need these values.
> adb forward tcp:2828 tcp:2828 && gaiatest --testvars=testvars.json --address=localhost:2828 --timeout=30000 gaiatest/tests/functional/dialer/test_dialer.py

	Gabriele Svelto [:gsvelto] Assignee
	Comment 44 • 9 years ago

I've started working on this, hopefully I'll be able to repro soon. BTW I've never seen this while dogfooding but I'm using the full 1024MiB configuration on my Flame and it seems like this is related to a low memory scenario.

I have once met a similar issue which led to the same screen to be displayed but the STR was completely different (two calls were involved with one being held). The log didn't show anything interesting at the time.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 45 • 9 years ago

OK, I've identified the issue. When receiving a 'memorypressure' event we're unconditionally freeing the callscreen window, even if there are ongoing calls:

https://github.com/mozilla-b2g/gaia/blob/bc8be5081676ffed4cf9c2c9c53fc48caa0f1a49/apps/system/js/dialer_agent.js#L60

It's a miracle this hasn't bitten us in the past, though it's perfectly possible that other intermittent dialer bugs were caused by this. For example if the 'memorypressure' condition arises after the callscreen has been shown and populated correctly it turns white even though the call keeps going. I'll cook up a patch shortly.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 46 • 9 years ago

Moving to the right component, I'm writing unit-tests for the patch and will upload it soon. BTW I think we should backport this to v2.2 too since the change is very small but the impact is significant on memory-constrained devices.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 48 • 9 years ago

Comment on attachment 8612304 [details] [review]
[gaia] gabrielesvelto:bug-1126260-callscreen-memory-pressure > mozilla-b2g:master

Here's a patch that fixes the problem and introduces unit-tests to cover this scenario. In a nutshell my change prevents the callscreen from being freed when under memory pressure if there's any ongoing call or if it's visible. The latter condition is important because when we close the callscreen it remains visible for a couple of seconds *after* the last call has been disconnected and we don't want to turn it into a white window while the user is still seeing it.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 49 • 9 years ago

This affects v2.2 and the repro rate is going to be high on memory constrained devices so nom'ing for 2.2+. When it happens it effectively makes the callscreen unusable, possibly preventing a user from answering a call, preventing the user from making a call or even making the user unable to hang up a call if it happens while one is already connected. I think this should be enough to consider this a cert blocker. The upside is that the patch is only 7 lines, carries very little risk and all possible scenarios are covered with unit-tests.

	(inactive after 6/18) Alive Kuo [:alive]
	Comment 50 • 9 years ago

Comment on attachment 8612304 [details] [review]
[gaia] gabrielesvelto:bug-1126260-callscreen-memory-pressure > mozilla-b2g:master

LGTM, thanks.

	howie [:howie]
	Comment 51 • 9 years ago

since it's very close to CC June 8th, ni RM Josh, I suggest to make it 2.2+.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 52 • 9 years ago

Thanks for the quick review Alive! Ready to land.

	GitHub Autolander Bot
	Comment 53 • 9 years ago

Pull request has landed in master: https://github.com/mozilla-b2g/gaia/commit/60fea9c155ef017f0c3f4ef0f2f338d479a0518c

	Josh Cheng [:josh]
	Comment 54 • 9 years ago

2.2+ as this could break major function.
Hi Gabriele,
Could you please raise 2.2 approval?
Thanks

	Gabriele Svelto [:gsvelto] Assignee
	Comment 55 • 9 years ago

Comment on attachment 8612304 [details] [review]
[gaia] gabrielesvelto:bug-1126260-callscreen-memory-pressure > mozilla-b2g:master

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): bug 999478
[User impact] if declined: In low-memory situations the callscreen might stop working, preventing the user from receiving a call, making one or hanging up one already connected
[Testing completed]: Tested on a Flame by setting the memory pressure trigger so low as to force memory pressure events to be fired almost permanently, the fix is also covered with unit-tests
[Risk to taking this patch] (and alternatives if risky): Disabling memory pressure detection, but this will hamper applications consuming a lot of memory and memory-constrained devices in general
[String changes made]: None

	Jayme Mercado [:JMercado] (Inactive 3/4/2016)
	Comment 56 • 9 years ago

I ran the automation script that often reproduced this issue 40 times on a full flashed engineering build from today and saw 0 reproductions:

Actual Results: All calls were made correctly.

	Jayme Mercado [:JMercado] (Inactive 3/4/2016)
	Comment 57 • 9 years ago

Forgot to paste the build variables for the above comment.

Environmental Variables:
Device: Flame 3.0
BuildID: 20150529063340
Gaia: a0d4c222c1f1a8d2954539dac1537a1166aab320
Gecko: 9738f055d98c
Gonk: 040bb1e9ac8a5b6dd756fdd696aa37a8868b5c67
Version: 41.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0

	Josh Cheng [:josh]
	Comment 58 • 9 years ago

Comment on attachment 8612304 [details] [review]
[gaia] gabrielesvelto:bug-1126260-callscreen-memory-pressure > mozilla-b2g:master

Approving as this causing major function failure.

	Ryan VanderMeulen [:RyanVM]
	Comment 59 • 9 years ago

v2.2: https://github.com/mozilla-b2g/gaia/commit/055a5b8e0400c46ebcdca6d8b3fe4c8c85eaa56d