Closed Bug 1003795 Opened 10 years ago Closed 10 years ago

[meta] Fix the failing marionette tests after using v0.3.7 marionette-apps

Categories

(Firefox OS Graveyard :: Gaia, defect)

x86
macOS
defect
Not set
normal

Tracking

(b2g-v2.0 fixed)

RESOLVED FIXED
Tracking Status
b2g-v2.0 --- fixed

People

(Reporter: evanxd, Unassigned)

References

Details

Once we use the patch of Bug 950673, some tests will be failed.

Always failed:
apps/system/test/marionette/notification_get_test.js:247
apps/system/test/marionette/edges_gesture_test.js:30
apps/music/test/marionette/Player_test.js:103
apps/calendar/test/marionette/month_view_test.js:16
apps/calendar/test/marionette/toggle_calendar_test.js:13
apps/calendar/test/marionette/day_view_test.js:25
apps/calendar/test/marionette/create_event_test.js:22

Intermittent failed:
apps/system/test/marionette/fullscreen_statusbar_test.js:30
apps/search/test/marionette/places_search_test.js:
apps/search/test/marionette/app_search_test.js:15
apps/homescreen/test/marionette/install_bookmark_test.js:43

We could refer to https://travis-ci.org/evanxd/gaia/builds/23940818.
Depends on: 1003788
Currently, we need to disable the below tests with the https://github.com/evanxd/marionette-apps/commit/2665d203c85eb9b02a000ac9716edcc15b274ee3 marionette-apps change.
1. apps/calendar/test/marionette/create_event_test.js
2. apps/calendar/test/marionette/day_view_test.js
3. apps/calendar/test/marionette/month_view_test.js
4. apps/calendar/test/marionette/toggle_calendar_test.js
5. apps/homescreen/test/marionette/install_bookmark_test.js
6. apps/music/test/marionette/Player_test.js
7. apps/search/test/marionette/app_search_test.js
8. apps/search/test/marionette/places_search_test.js
9. apps/system/test/marionette/desktop_notification_test.js
10. apps/system/test/marionette/edges_gesture_test.js
11. apps/system/test/marionette/fullscreen_statusbar_test.js
12. apps/system/test/marionette/notification_test.js
13. apps/system/test/marionette/notification_get_test.js
And all tests will be passed after we do the marionette-apps patch at Comment 1.
See the travis job at https://travis-ci.org/evanxd/gaia/builds/24507793.
After we fix this bug, we should find a long term solution to deal with changes of System app for the marionette-apps module.
We will disable the below tests in Bug 950673, because we will use updated marionette-apps module and the logic of switchToApp method is changed.

home2:
"dev_apps/home2/test/marionette/bookmark_test.js"

calendar:
"apps/calendar/test/marionette/create_event_test.js"
"apps/calendar/test/marionette/day_view_test.js"
"apps/calendar/test/marionette/month_view_test.js"
"apps/calendar/test/marionette/toggle_calendar_test.js"
"apps/homescreen/test/marionette/install_bookmark_test.js"

music:
"apps/music/test/marionette/Player_test.js"

search:
"apps/search/test/marionette/app_search_test.js"
"apps/search/test/marionette/places_search_test.js"

system:
"apps/system/test/marionette/desktop_notification_test.js"
"apps/system/test/marionette/edges_gesture_test.js"
"apps/system/test/marionette/fullscreen_statusbar_test.js"
"apps/system/test/marionette/notification_events_test.js"
"apps/system/test/marionette/notification_test.js"
"apps/system/test/marionette/notification_get_test.js"

We will fix them here.
Depends on: 950673
The version of marionette-apps we will use in Bug 950673 is v0.3.5.
Summary: Fix the failures of the marionette tests. → Fix the failing marionette tests
Sorry, it is also about homescreen app.

homescreen:
"apps/homescreen/test/marionette/install_bookmark_test.js"
Summary: Fix the failing marionette tests → Fix the failing marionette tests after using v0.3.5 marionette-apps
Depends on: 1011415
Summary: Fix the failing marionette tests after using v0.3.5 marionette-apps → [meta] Fix the failing marionette tests after using v0.3.5 marionette-apps
So why is the switchToApp logic different now? I see some apps with: 

client.switchToFrame();
client.apps.(url);

What is so broken about this?
Hi Kevin,

Currently, `switchToApp` method wait for the app's frame is on the `transition-state="opened"` state and the class list included `render`. It means the app is already rendered and show on the screen, and then the app is ready to be controlled by marionette. We could refer to Bug 1003788.

After we have the updated `switchToApp` method, we could not do the below things.
1. Do not use the `switchToApp` method to switch to keyboard frame.
2. Do not use the `switchToApp` method to switch to homescreen frame, when we already launched some app.

And the above two wrong cases is in the gaia code base.
For example, this line https://github.com/mozilla-b2g/gaia/blob/master/dev_apps/home2/test/marionette/bookmark_test.js#L52 should do `client.switchToFrame(home2Frame)` to instead of `client.apps.switchToApp(Home2.URL)`.

For "apps/calendar/test/marionette/create_event_test.js", "apps/calendar/test/marionette/day_view_test.js", "apps/calendar/test/marionette/month_view_test.js", "apps/calendar/test/marionette/toggle_calendar_test.js", this line https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/marionette/lib/calendar.js#L280 should to `client.switchToFrame(keyboardFrame)` to instead of `client.apps.switchToApp('app://keyboard.gaiamobile.org');`.

We need to fix the wrong usage of marionette-apps in gaia case by case.
Depends on: 1014426
Depends on: 1014436
Depends on: 1014446
Depends on: 1014469
Depends on: 1014475
I think this (and the patch to marionette-apps) are invalid and that the issue has more to do with the system app page loading semantics than it does correctness of integration tests...
I don't really see how there can be a problem with loading semantics if we can load apps, and the phone is functional. It seems like we just need to find a better fix for marionette-apps so we don't have to patch the tests.
See Also: → 1014820
No longer depends on: 950673
No longer depends on: 1003788
Blocks: 1003788
(In reply to Gareth Aye [:gaye] on PTO until 06/02 from comment #9)
> I think this (and the patch to marionette-apps) are invalid and that the
> issue has more to do with the system app page loading semantics than it does
> correctness of integration tests...

Well, system app is not existing to provide any support to a not-correctly-implemented test library. As somebody's note: there should be more communication made before people invented this library between  library author and system app owner.
Summary: [meta] Fix the failing marionette tests after using v0.3.5 marionette-apps → [meta] Fix the failing marionette tests after using v0.3.7 marionette-apps
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Mass modify - set status-b2g-v2.0 fixed for fixed bugs under vertical homescreen dependency tree.
You need to log in before you can comment on or make changes to this bug.