Closed Bug 1000863 Opened 10 years ago Closed 9 years ago

Investigate test_fmradio_find_stations.py failure

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P2)

ARM
Gonk (Firefox OS)
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: RobertC, Assigned: martijn.martijn)

Details

(Whiteboard: [xfail])

Attachments

(4 files)

test_fmradio_find_stations.py is failing because when tapping on next station it cannot find any.

We could not reproduce this manually or with automated tests, but someone from MV should check the phones manually. A possible issue would be that the phones are in an area with radio interference.

Environment:
Gaia      b6a41ee9e2934fe9692da6fb1cb310cb759e9870
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/2e9abf89c087
BuildID   20140424000203
Version   30.0a2
ro.build.version.incremental=eng.tclxa.20131223.163538
ro.build.date=Mon Dec 23 16:36:04 CST 2013

Stacktrace:
Traceback (most recent call last):
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.ui.smoketest/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/marionette_test.py", line 158, in run
testMethod()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.ui.smoketest/tests/python/gaia-ui-tests/gaiatest/tests/functional/cards_view/test_cards_view_with_two_apps.py", line 39, in test_that_app_can_be_launched_from_cards_view
cards_view.wait_for_cards_view_not_displayed()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.ui.smoketest/tests/python/gaia-ui-tests/gaiatest/apps/system/regions/cards_view.py", line 50, in wait_for_cards_view_not_displayed
self.wait_for_element_not_displayed(*self._cards_view_locator)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.ui.smoketest/tests/python/gaia-ui-tests/gaiatest/apps/base.py", line 48, in wait_for_element_not_displayed
lambda m: not m.find_element(by, locator).is_displayed())
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.ui.smoketest/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/wait.py", line 143, in until
cause=last_exc)
TimeoutException: TimeoutException: Timed out after 30.1214268208 seconds

STR (from automated test):
1. launch FmRadio
2. wait for radio start-up
3. search for next station

Expected result:
The radio changes to a new station

Actual result:
The frequency does not change. No new station is found.
Leaving qawanted to see if we can reproduce on 1.4.
Keywords: qawanted
This is still reproducible in the latest v1.4 build. 
Gaia      d23e479e8a4ce0bc620acb2d7e2f82801aa4d0ea
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/36f67ce46855
BuildID   20140428000206
Version   30.0a2
ro.build.version.incremental=eng.tclxa.20131223.163538
ro.build.date=Mon Dec 23 16:36:04 CST 2013

Raymond, can you, please, test the phones manually to see if you can reproduce the failure, according to comment 1?
Flags: needinfo?(mozbugs.retornam)
Sorry, according to STR in comment 0.
We can see this on 2.0 in Jenkins.

This might be because of the recent infrastructure move.


Gaia      f50d8a3504e0a57d371457c50a6ced333e20724d
Gecko     https://hg.mozilla.org/mozilla-central/rev/4d926af89907
BuildID   20140428040200
Version   31.0a1
ro.build.version.incremental=324
ro.build.date=Thu Dec 19 14:04:55 CST 2013


Traceback (most recent call last):
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.non-smoketest/.env/local/lib/python2.7/site-packages/marionette_client-0.7.6-py2.7.egg/marionette/marionette_test.py", line 163, in run
testMethod()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.non-smoketest/tests/python/gaia-ui-tests/gaiatest/tests/functional/fmradio/test_fmradio_find_stations.py", line 34, in test_find_next_previous_station
self.fm_radio.tap_next()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.non-smoketest/tests/python/gaia-ui-tests/gaiatest/apps/fmradio/app.py", line 39, in tap_next
self.wait_for_condition(lambda m: self.frequency != current_frequency)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.non-smoketest/tests/python/gaia-ui-tests/gaiatest/apps/base.py", line 54, in wait_for_condition
Wait(self.marionette, timeout).until(method, message=message)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.ui.non-smoketest/.env/local/lib/python2.7/site-packages/marionette_client-0.7.6-py2.7.egg/marionette/wait.py", line 143, in until
cause=last_exc)
TimeoutException: TimeoutException: Timed out after 30.0 seconds
Raymond can you check and see if we have FM signal where the phones are ?
Summary: [v1.4] Investigate test_fmradio_find_stations.py failure → Investigate test_fmradio_find_stations.py failure
(In reply to Florin Strugariu [:Bebe] from comment #5)
> Raymond can you check and see if we have FM signal where the phones are ?

Each phone has a headphone connected to it and is able to receive an FM signal.
Flags: needinfo?(mozbugs.retornam)
Retornam, can you, from inside the QA lab count how many stations the device can find. Then take the device outside the office into the carpark and count how many stations it can find?

We strongly suspect the signal inside the office is much poorer. With an intermittent test we need a bit more hard data than "it has a signal".
Flags: needinfo?(mozbugs.retornam)
It's also possible that this is tapping before the fm radio has started up properly.
Under normal use in the Kirkland, Wa office, QA did not experience this issue. User was able to switch the radio station as expected.  

Following the steps in Comment #8 User was able to temporarily not switch stations. I spammed the 'next' button at radio launch, but it eventually (5-8 seconds after app launched) caught up and start switching.

Tester used a Faraday bag, and could reproduce the issue but this is as expected since user was blocking the FM signal. When the 'next' button was pushed, it highlighted, and stayed in that state. The radio station never switched. This maybe a bug in itself because the user is not notified that there is no available signal.

1.4 Environmental Variables:
Device: Buri 1.4 MOZ
BuildID: 20140430000201
Gaia: 81e97c3ca58be0487292011bc59efa4cebab30be
Gecko: 123485e733d5
Version: 30.0
Firmware Version: V1.2-devices.cfg
Keywords: qawanted
All the devices have their ear-buds connected in the lab. I tried manually tuning in to a FM station with one of the devices and was able to receive a radio signal.
Flags: needinfo?(mozbugs.retornam)
Whiteboard: [xfail]
I cannot replicate this locally on the latest buid.
Let's just xfail this until so we can investigate this further.
Attachment #8417895 - Flags: review?(zcampbell)
Attachment #8417895 - Flags: review?(florin.strugariu)
Comment on attachment 8417895 [details] [review]
XFAIL PR https://github.com/mozilla-b2g/gaia/pull/18975

r+!
Attachment #8417895 - Flags: review?(zcampbell)
Attachment #8417895 - Flags: review?(florin.strugariu)
Attachment #8417895 - Flags: review+
Comment on attachment 8417895 [details] [review]
XFAIL PR https://github.com/mozilla-b2g/gaia/pull/18975

sorry wrong bug!
Attachment #8417895 - Flags: review?(zcampbell)
Attachment #8417895 - Flags: review?(florin.strugariu)
Attachment #8417895 - Flags: review+
I'll have a look into the timing of this now.
Assignee: nobody → zcampbell
Spent quite a bit of time on this today and the test is fine.

I can replicate the bug manually by scrunching up the headphones and moving into a corner of the office. Tapping the find station just times out, nothing is found. The timeout takes around 10 seconds (not very long).

Then by extending the headphones (ie let them dangle down) I can immediately find stations.

Thus I'm 100% sure it's a problem with the phone or antenna. Now that we have re-enabled test_fm_flick I think we'll see problems with that, too.
Priority: -- → P2
Flags: needinfo?(stephen.donner)
Has this been isolated to a specific phone?  That'd help us triage it quite a bit too.  As Raymond says, it's able to tune in to stations.

Inside the lab, B2G-8 was able to tune in to 17 stations; at my desk, and outside, the number increases to ~24.  I had a full passing run of 10x, on the same device, when I physically put the B2G-8 phone atop the cluster of the rest of the phones.  I'll keep digging on Monday (will come in early).
Flags: needinfo?(stephen.donner)
Let's xfail this on v1.4 also, until we figure out what's causing this.
Attachment #8420945 - Flags: review?(zcampbell)
Attachment #8420945 - Flags: review?(florin.strugariu)
Comment on attachment 8420945 [details] [review]
XFAIL PR v1.4 https://github.com/mozilla-b2g/gaia/pull/19161

This will have no effect in build stability 

If we Xfail this and the test will pass we will still have a red build because of the unexpected pass.

Let's wait until we fix the test or find out the root cause of the issue
Attachment #8420945 - Flags: review?(florin.strugariu) → review-
Comment on attachment 8420945 [details] [review]
XFAIL PR v1.4 https://github.com/mozilla-b2g/gaia/pull/19161

r-, Did not fail on today's build
Attachment #8420945 - Flags: review?(zcampbell) → review-
Reverted the master xfail because of unexpected passes:
https://github.com/mozilla-b2g/gaia/commit/4a9c9d6146d0efdeeaaff5e0fe48de886872bd32
I've been checking the devices on which this test was ran in jenkins, in order to try and correlate them with one specific device. 
But the test failed on several devices, more frequently on b2g-9, b2g-11 and b2g-8, once in b2g-7 and twice in b2g-2.
Assignee: zcampbell → nobody
What's the resolution path here to getting this test on? Do you need someone from Mt View to investigate?
Flags: needinfo?(zcampbell)
I think there were some environmental issues here. The test was not xfailed or disabled. It was just intermittently failing.

But this is not reproducible in Mt View for some time now.
The last failure was 20-May-2014 at 08:18:06 and it is passing since then.
Flags: needinfo?(zcampbell)
As this started right after we moved the phones in the lab and locally everything it's OK I think it is a issue with where the phones; where actually located and if the headphones where extended or not.

From the build it seems like the test is passing for almost a week now on both 2.0 and 1.4

Let's double check that all the phones have the headphones(FM Antennas) extended and that they are not in a corner where the FM signal might be bad.
Flags: needinfo?(stephen.donner)
(In reply to Florin Strugariu [:Bebe] from comment #25)
> As this started right after we moved the phones in the lab and locally
> everything it's OK I think it is a issue with where the phones; where
> actually located and if the headphones where extended or not.
> 
> From the build it seems like the test is passing for almost a week now on
> both 2.0 and 1.4
> 
> Let's double check that all the phones have the headphones(FM Antennas)
> extended and that they are not in a corner where the FM signal might be bad.

There's no way to really tell this, I'm afraid.  None of us are RF experts, and although we can move them around, there's no clear way (yet, that I can tell) of moving them outside of the rack they're all in.  If this ends up not being reliable, we might just have to give up this test -- let us know.  (I've looked at extending the earbud cables through the wire doors of the rack, and that's not possible, either, due to the earbuds themselves, size-wise.)
Flags: needinfo?(stephen.donner)
This might be a bad idea but:

In Ro we use some FM modulators to play mp3 on the car radio.

They look something like this:
http://www.amazon.com/GSI-Quality-Wireless-Hands-Free-Modulator-Transmitter/dp/B003L89FMW/ref=sr_1_20?s=electronics&ie=UTF8&qid=1401281940&sr=1-20&keywords=fm+modulator+for+car

This should work to simulate 2 or 3 radio networks in the lab

If we would add 2 or 3 of these in our lab setup we can be sure we have FM stations in there and also we can set them up on specified FM frequency that we check in the test :D
I tried to move to some earplugs higher in the rack. Earplugs of B2G-{2,7,8,11} are now taped to one side of the rack. Let's see if the issue is getting better or not.
(In reply to Johan Lorenzo [:jlorenzo] from comment #28)
> I tried to move to some earplugs higher in the rack. Earplugs of
> B2G-{2,7,8,11} are now taped to one side of the rack. Let's see if the issue
> is getting better or not.

Thanks!
This has not failed for a while.

we can reopen it if we see this fails
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
I had to reopen this as test_fmradio_find_stations.py failed again on both Flame and Hamachi:
http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Flame/job/b2g.flame.mozilla-central.ui.non-smoketest/49/HTML_Report/
http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-central.ui.non-smoketest/398/console
Raymond, can you, please, check again if the devices are able to find radio stations?
Flags: needinfo?(mozbugs.retornam)
Johan, would you please take a look in the lab, when you get a chance?  (Perhaps we need a more robust solution?)  Thanks!
Flags: needinfo?(jlorenzo)
All ear plugs have fallen down. Does anybody removed them from the sides of the rack? If not, we need to get better duck tape than the regular I used.
Flags: needinfo?(jlorenzo)
Stephen what do you think about: comment #27 ?
Flags: needinfo?(stephen.donner)
(In reply to Johan Lorenzo [:jlorenzo] from comment #33)
> All ear plugs have fallen down. Does anybody removed them from the sides of
> the rack? If not, we need to get better duck tape than the regular I used.

Can you try with electrical tape, or somehow actually hook their cable to the inside of the cage?
Flags: needinfo?(stephen.donner) → needinfo?(jlorenzo)
(In reply to Florin Strugariu [:Bebe] from comment #27)
> This might be a bad idea but:
> 
> In Ro we use some FM modulators to play mp3 on the car radio.
> 
> They look something like this:
> http://www.amazon.com/GSI-Quality-Wireless-Hands-Free-Modulator-Transmitter/
> dp/B003L89FMW/ref=sr_1_20?s=electronics&ie=UTF8&qid=1401281940&sr=1-
> 20&keywords=fm+modulator+for+car
> 
> This should work to simulate 2 or 3 radio networks in the lab
> 
> If we would add 2 or 3 of these in our lab setup we can be sure we have FM
> stations in there and also we can set them up on specified FM frequency that
> we check in the test :D

This sounds like a lot of complexity; I'd rather see if a more-robust tape or somehow hooking the cables into the frame, would still work.
Ordered gaffer tape today on service now. I'll update this bug when I have some news.
Flags: needinfo?(jlorenzo)
Flags: needinfo?(mozbugs.retornam)
Gaffer is here. I tapped the earplugs to the left side of the rack. It's kind of messy. If this test doesn't fail for the next days, we should think about cleaning up that pile of cables ;)
This failed again in:
http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Aurora/job/b2g.flame.mozilla-aurora.ui.non-smoketest/44/HTML_Report/

Build:
Gaia      8df02268fcd7e80c5fab8c1ec099772e37f8659d                         
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/731a5e8831e6  
BuildID   20140627000201                                                   
Version   32.0a2                                                           
ro.build.version.incremental=94                                            
ro.build.date=Tue May 20 09:29:20 CST 2014
Test also failed on v2.1 while running on b2g-7
http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Master/job/b2g.flame.mozilla-central.ui.non-smoketest/98/HTML_Report/

Build info flame:
Gaia      b8f36518696f3191a56e4f33447ee9d6ec820da1
Gecko     https://hg.mozilla.org/mozilla-central/rev/9290d7995f98
BuildID   20140627040205
Version   33.0a1
ro.build.version.incremental=94
ro.build.date=Tue May 20 09:29:20 CST 2014
Crap! They are still taped on the side of the rack. The trick didn't worked. We need to figure out a new way to avoid interferences.
Flags: needinfo?(jlorenzo)
Can this be investigated even further? 
Ran the test in parallel at the same time:

On b2g-4 test works perfectly: http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Flame/job/b2g.flame.mozilla-central.ui.adhoc/74/HTML_Report/

On b2g-7 and b2g-9 test fails every time:
http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Flame/job/b2g.flame.mozilla-central.ui.adhoc/73/HTML_Report/

Can you guys take a look and see what is the difference between this phones, maybe we can find the reason why this is still failing?
Flags: needinfo?(mozbugs.retornam)
Flags: needinfo?(jlorenzo)
Flags: needinfo?(dylan)
And another thing, for b2g-15 the test is intermittent failing: 
http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Flame/job/b2g.flame.mozilla-central.ui.adhoc/76/console

Maybe we can find out what is causing this.
Sorry, I'm not involved in this product -- did you mean to needinfo? another Dylan?
Flags: needinfo?(dylan)
Flags: needinfo?(dwong)
Currently working remotely as there as an intern event occurring, but yesterday when I checked the phones I can confirm manually that there is trouble finding next stations, either it takes a very long time or not at all. When trying b2g-9 I believe I got it to change stations once, but after that no luck. I'm not sure what can be done if it's a reception issue, currently the earbuds of all flames are taped along the wall of the server rack to get better reception, so they are all under similar conditions. If this is truly a hardware problem we can replace it and see if results are less cloudy.
Flags: needinfo?(dwong)
do you think it's possible some types of headphones give a better reception than others?
(In reply to Zac C (:zac) from comment #49)
> do you think it's possible some types of headphones give a better reception
> than others?

The headphones are not the problem. It is the amount of interference from other devices in the lab that is causing this. We need to come up with a way to reduce the interference.
Flags: needinfo?(mozbugs.retornam)
(In reply to raymond [:retornam] (needinfo? me) from comment #50)
> (In reply to Zac C (:zac) from comment #49)
> > do you think it's possible some types of headphones give a better reception
> > than others?
> 
> The headphones are not the problem. It is the amount of interference from
> other devices in the lab that is causing this. We need to come up with a way
> to reduce the interference.

I agree. Also, all the devices use the same type of headphone (the one included with the Flame).
Flags: needinfo?(jlorenzo)
Might be a longshot, but let's see how this fares, post-landing from bug 1037646.  And, yes, the earbuds aren't likely to be the problem, and for the performance tests' integrity, we need to keep the original Flame ones paired to the phones.
Comment on attachment 8453651 [details] [review]
Disable on master https://github.com/mozilla-b2g/gaia/pull/21577

We worked out that these seem to be more device related. If we get consistent failures, disable the device and alert MV.
Attachment #8453651 - Flags: review?(zcampbell)
This test is still failing intermittently on v2.2 and v2.1. Lately the reproduction rate on Jenkins has increased a lot.
http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/164/console

From what I observed in the last few weeks this is not isolated to a certain device.
Can someone from MV please check the device antennas and make sure they are in the correct position and check the devices to make sure we are finding at least 2 radio stations?

On local device I was never able to reproduce the issue.
Tony, as mentioned in comment 54 we are encountering a lot of failures with this test. Could you look over comment 27 and see if the possible solution discussed there is viable?
Flags: needinfo?(tchung)
(In reply to Robert Chira [:RobertC] from comment #55)
> Tony, as mentioned in comment 54 we are encountering a lot of failures with
> this test. Could you look over comment 27 and see if the possible solution
> discussed there is viable?

which phones are failing lately?  i'll take a look at the lab today.
Flags: needinfo?(tchung)
I checked the job history and found that he test failed on the following devices: b2g-28.2, b2g-24.2, b2g-23.1, b2g-20.2, b2g-21.1, b2g-07.1. Since Jenkins tend to run on the same devices if they are available, I cannot guarantee that other devices are not affected.
QA Whiteboard: [fxosqa-auto-from-s2] [fxosqa-auto-s3]
Assignee: nobody → gmealer
FM transmitter on route to at least isolate issue
QA Whiteboard: [fxosqa-auto-from-s2] [fxosqa-auto-s3] → [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-s4]
FM transmitters are present in the lab, but are shutting down automatically due to no audio source. Am sourcing transmitters that either have a hard power switch or sourcing a more permanent audio source.
QA Whiteboard: [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-s4] → [fxosqa-auto-from-s1] [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-s4]
QA Whiteboard: [fxosqa-auto-from-s1] [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-s4] → [fxosqa-auto-from-s1] [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-from-s4] [fxosqa-auto-s5]
Hard-switched transmitters came in during workweek, one good and one DOA. Returned/reordered for DOA, ETA Thursday.
QA Whiteboard: [fxosqa-auto-from-s1] [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-from-s4] [fxosqa-auto-s5] → [fxosqa-auto-from-s1] [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-from-s4] [fxosqa-auto-from-s5][fxosqa-auto-s6]
OK, I've staged the transmitters. Original transmitters were both on 102.7. I'm going to put these two on two different settings since the test seems to want that.

That said, I have concerns about this test. It seems to assume that the initial station that the FM Radio tunes to when you activate it is a valid station to come back to when you seek previous. 

We have another test that tests that the FM Radio initializes to the lower bound (I assume 87.5 MHz) every time, so this seems unlikely to be true. It's possible we may need to add testvars to start the next/prev at a known station.

In the meantime, I'm going to put these two at 87.5 and 89.5 Mhz respectively. In reality, we have stations between those points (and on almost every .x interval, Bay Area sucks for radio that way) but it's about as good a punt as I know.

Reopen if more issues occur, please.
Status: REOPENED → RESOLVED
Closed: 10 years ago9 years ago
Resolution: --- → FIXED
Missed that we disabled the test. I ran this 20 times in the lab successfully to verify and will have a pull request to re-enable later today.
Status: RESOLVED → REOPENED
QA Whiteboard: [fxosqa-auto-from-s1] [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-from-s4] [fxosqa-auto-from-s5][fxosqa-auto-s6] → [fxosqa-auto-from-s1] [fxosqa-auto-from-s2] [fxosqa-auto-from-s3] [fxosqa-auto-from-s4] [fxosqa-auto-from-s5][fxosqa-auto-from-s6][fxosqa-auto-s7]
Resolution: FIXED → ---
Assignee: gmealer → nobody
Assignee: nobody → martijn.martijn
Comment on attachment 8682145 [details] [review]
[gaia] mwargers:1000863 > mozilla-b2g:master

This test works fine locally, even after a repeat of 10.
And from comment 62, I get the impression that this was only a matter of turning back on.
Attachment #8682145 - Flags: review?(npark)
Attachment #8682145 - Flags: review?(jlorenzo)
Attachment #8682145 - Flags: review?(npark) → review+
Comment on attachment 8682145 [details] [review]
[gaia] mwargers:1000863 > mozilla-b2g:master

Yeah, the test was already passing in comment 62. I'm glad it's still the case.
Attachment #8682145 - Flags: review?(jlorenzo) → review+
Merged: https://github.com/mozilla-b2g/gaia/commit/5309b1918d95ce52c35663693d8b138d96491c70
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: