Closed Bug 1176882 Opened 9 years ago Closed 9 years ago

Jenkins Jobs failing within 5 minutes [Bitbar && MTV Labs]

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(firefox41 fixed)

RESOLVED FIXED
Tracking Status
firefox41 --- fixed

People

(Reporter: onelson, Assigned: martijn.martijn)

References

Details

Attachments

(4 files)

Description:
A recent regression for the automated tests appears to have broken all Jenkins jobs. Across Bitbar and MTV labs, all tests are failing within 5 minutes.

* http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.smoke/313/
* http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/248/

Traceback (From console output [in full in attached text doc]):
02:56:16 Traceback (most recent call last):
02:56:16   File "setup.py", line 57, in <module>
02:56:16     install_requires=deps)
02:56:16   File "/usr/local/lib/python2.7/distutils/core.py", line 151, in setup
02:56:16     dist.run_commands()
02:56:16   File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
02:56:16     self.run_command(cmd)
02:56:16   File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
02:56:16     cmd_obj.run()
02:56:16   File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/setuptools/command/develop.py", line 32, in run
02:56:16     self.install_for_development()
02:56:16   File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/setuptools/command/develop.py", line 132, in install_for_development
02:56:16     self.process_distribution(None, self.dist, not self.no_deps)
02:56:16   File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 710, in process_distribution
02:56:16     % e.args
02:56:16 TypeError: not enough arguments for format string
02:56:16 Build step 'Execute managed script' marked build as failure
02:56:16 [Testdroid] - Releasing device session
02:56:16 Archiving artifacts
02:56:16 [htmlpublisher] Archiving HTML reports...
02:56:16 [htmlpublisher] Archiving at BUILD level /var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/tests/python/gaia-ui-tests/report to /var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/builds/2015-06-23_02-50-15/htmlreports/HTML_Report
02:56:16 ERROR: Specified HTML directory '/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/tests/python/gaia-ui-tests/report' does not exist.
02:56:16 Recording test results


Environmental Variables:
--------------------------------------------------
Device: Flame 3.0
BuildID: 20150623010204
Gaia: 311c4e59936a407e64509f54fecb440d8a78e3c8
Gecko: be81b8d6fae9
Gonk: a4f6f31d1fe213ac935ca8ede7d05e47324101a4
Version: 41.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0
--------------------------------------------------
Loose Regression Window by looking at b2g-inbound
---------------------------------------------------

Last Working:
* http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.bitbar/1996/

First Broken:
* http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.bitbar/1997/
Flags: needinfo?(gmealer)
I guess this is a regression from bug 1176408, because the failure comes from  marionette-transport-0.5/setup.py.
Johnathan, can you take a look at this, please?
Blocks: 1176408
Flags: needinfo?(jgriffin)
Does one of you have a pushlog link for the given range? I don't really see that this should be marionette-transport related. Having all changesets between those builds would be good to have.
Flags: needinfo?(onelson)
Attached file consoleText
Here is the whole log of the error in a Jenkins job.
Hrm, locally, those changes don't give problems to me. I don't know this is setup on Jenkins, but apparently, there is some mismatch in packages?
Flags: needinfo?(dave.hunt)
As you can see from this log marionette-transport is correctly installed:

Installed /var/lib/jenkins/jobs/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_transport-0.5-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 57, in <module>
    install_requires=deps)

The traceback happens afterward. Can you check a working build what happens next there?
(In reply to Henrik Skupin (:whimboo) from comment #6)
> As you can see from this log marionette-transport is correctly installed:
> 
> Installed
> /var/lib/jenkins/jobs/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.
> bitbar/workspace/.env/lib/python2.7/site-packages/marionette_transport-0.5-
> py2.7.egg
> Traceback (most recent call last):
>   File "setup.py", line 57, in <module>
>     install_requires=deps)
> 
> The traceback happens afterward. Can you check a working build what happens
> next there?

This is from a working Jenkins job. As you can see, it doesn't go past marionette_transport-0.5 setup.
I think this is just a deps problem.

gaiatest is pinned to marionette-driver==0.7, which is pinned to marionette-transport==0.4, but for some reason the logs show marionette-transport 0.5 being loaded. 

Bumping the marionette-driver dep in gaiatest may resolve this.
Flags: needinfo?(jgriffin)
marionette-client is pulling in marionette-transport >= 0.4 which is probably causing this. We could either pin that, or remove it, and allow that package dependency to be managed via marionette-driver.
Flags: needinfo?(dave.hunt)
Ok, like this then?
Attachment #8625584 - Flags: review?(dave.hunt)
(In reply to Dave Hunt (:davehunt) from comment #9)
> marionette-client is pulling in marionette-transport >= 0.4 which is
> probably causing this. We could either pin that, or remove it, and allow
> that package dependency to be managed via marionette-driver.

That should be the better approach, yes. I don't think that you use marionette-transport directly in your gaia tests. :) And marionette-driver definitely knows it better which transport version is best for the moment.
Comment on attachment 8625584 [details] [diff] [review]
1176882.diff (for check-in)

This looks good, however I haven't tested it locally. It should be relatively trivial to reproduce the failure with a clean virtual environment.
Attachment #8625584 - Flags: review?(dave.hunt) → review+
I did test with a clean environment, but I get this error instead, doing python setup.py develop:
Installed /Users/mwargers/.virtualenvs/gaia-py-test/lib/python2.7/site-packages/marionette_transport-0.5-py2.7.egg
error: marionette-transport 0.5 is installed but marionette-transport==0.4 is required by set(['marionette-driver'])
Keywords: checkin-needed
Attachment #8625584 - Attachment description: 1176882.diff → 1176882.diff (for check-in)
You might also have to update the dependency for marionette-driver, where we also released a new version.
(In reply to Henrik Skupin (:whimboo) from comment #14)
> You might also have to update the dependency for marionette-driver, where we
> also released a new version.

Oh, I see what you mean. Yeah, could be, but if the patch fixes this problem, then I'm happy. I don't really know how this works.
Please see the patch on bug 1176408. marionette-driver has a strict dependency to marionette-transport. So only version 0.9 of driver will work with transport 0.5. Any other combination will cause the bustage as above.

Where do you have the dependencies defined? Any URL for us?
Martijn, would you mind to push this patch to try first? I would feel more comfortable seeing that tests would pass again.
Keywords: checkin-needed
(In reply to Henrik Skupin (:whimboo) from comment #16)
> Please see the patch on bug 1176408. marionette-driver has a strict
> dependency to marionette-transport. So only version 0.9 of driver will work
> with transport 0.5. Any other combination will cause the bustage as above.
> 
> Where do you have the dependencies defined? Any URL for us?

I don't know, isn't that what Dave Hunt was saying in comment 4? Marionette-client in testing/marionette/client/requirements.txt ?
(In reply to Henrik Skupin (:whimboo) from comment #17)
> Martijn, would you mind to push this patch to try first? I would feel more
> comfortable seeing that tests would pass again.

Well, these tests are not running on tryserver. They are run on Jenkins, I could do an adhoc run there, but I wouldn't know how to include the gecko part which would include the patch.
Johan, would you know how to do this? Or where you talking about some other tests than the Gaia UI tests on Jenkins?
Flags: needinfo?(jlorenzo)
Flags: needinfo?(hskupin)
This change makes sense so I think we should land regardless.
But wouldn't it also need a new release of marionette-client? Sounds like that this Jenkins CI is getting its packages from pypi.
Flags: needinfo?(hskupin)
(In reply to Jonathan Griffin (:jgriffin) from comment #20)
> This change makes sense so I think we should land regardless.

Re-adding checkin-needed keyword, so we can get the Jenkins jobs running again (currently, QAnalyst need to do those runs locally). 

(In reply to Henrik Skupin (:whimboo) from comment #21)
> But wouldn't it also need a new release of marionette-client? Sounds like
> that this Jenkins CI is getting its packages from pypi.

Could be, like I said I don't know enough of this subject. I hope to meet up with Dave Hunt this week to get more informed about this subject.
Keywords: checkin-needed
Well, I doubt that this patch is enough, and I think we still need the version bump too.
Yes, we'll need to bump and release, but we generally do that in a separate bug.
Ok, I created a new virtualenv. I applied the patch that removed the "marionette-transport >= 0.4" line in  testing/marionette/client/requirements.txt of my mozilla-central gecko build, then did python setup.py install there.
Then, I went to /Users/mwargers/b2g/gaia_clean/tests/python/gaia-ui-tests and tried to do python setup.py develop there, but got into problems with other dependancies ( error: marionette-driver 0.7 is installed but marionette-driver>=0.8 is required by set(['marionette-client']) ), so I changed the requirements.txt file there to use:
marionette_driver==0.7
marionette_client==0.13
After that, I did python setup.py develop again and then I didn't get any errors.
Flags: needinfo?(jlorenzo)
Inbound is closed or I'd land this myself.  Given the chaos of the work week, how important is this?  If it is, I can release a new version of marionette-client to pypi ahead of the landing.
It's not a big deal until Monday, at least.
FWIW, the commit message for your patch should be summarizing what it's doing, not restating the problem being solved by it.
Assignee: nobody → martijn.martijn
https://hg.mozilla.org/mozilla-central/rev/dc94ada04fa8
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
So this appears still to be a problem. Dave Hunt (from mail):
Probably best to reopen that bug and investigate - I suspect we need to pin an additional dependency.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I just reran a mozilla-central sanity job and it passed - perhaps the fix hadn't fully propogated on the previous build?
Flags: needinfo?(martijn.martijn)
Yeah, the Jenkins job seem to be running now again, too. Looks like this was eventually fixed by bug 1177513, I guess.
No longer blocks: 1177513
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Depends on: 1177513
Flags: needinfo?(martijn.martijn)
Resolution: --- → FIXED
Was keeping ni? to watch, issue doesn't appear to be coming back.
Flags: needinfo?(onelson)
Flags: needinfo?(gmealer)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: