Closed
Bug 1176882
Opened 9 years ago
Closed 9 years ago
Jenkins Jobs failing within 5 minutes [Bitbar && MTV Labs]
Categories
(Firefox OS Graveyard :: Gaia::UI Tests, defect)
Tracking
(firefox41 fixed)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox41 | --- | fixed |
People
(Reporter: onelson, Assigned: martijn.martijn)
References
Details
Attachments
(4 files)
Description: A recent regression for the automated tests appears to have broken all Jenkins jobs. Across Bitbar and MTV labs, all tests are failing within 5 minutes. * http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.smoke/313/ * http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/248/ Traceback (From console output [in full in attached text doc]): 02:56:16 Traceback (most recent call last): 02:56:16 File "setup.py", line 57, in <module> 02:56:16 install_requires=deps) 02:56:16 File "/usr/local/lib/python2.7/distutils/core.py", line 151, in setup 02:56:16 dist.run_commands() 02:56:16 File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands 02:56:16 self.run_command(cmd) 02:56:16 File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command 02:56:16 cmd_obj.run() 02:56:16 File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/setuptools/command/develop.py", line 32, in run 02:56:16 self.install_for_development() 02:56:16 File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/setuptools/command/develop.py", line 132, in install_for_development 02:56:16 self.process_distribution(None, self.dist, not self.no_deps) 02:56:16 File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 710, in process_distribution 02:56:16 % e.args 02:56:16 TypeError: not enough arguments for format string 02:56:16 Build step 'Execute managed script' marked build as failure 02:56:16 [Testdroid] - Releasing device session 02:56:16 Archiving artifacts 02:56:16 [htmlpublisher] Archiving HTML reports... 02:56:16 [htmlpublisher] Archiving at BUILD level /var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/tests/python/gaia-ui-tests/report to /var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/builds/2015-06-23_02-50-15/htmlreports/HTML_Report 02:56:16 ERROR: Specified HTML directory '/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/workspace/tests/python/gaia-ui-tests/report' does not exist. 02:56:16 Recording test results Environmental Variables: -------------------------------------------------- Device: Flame 3.0 BuildID: 20150623010204 Gaia: 311c4e59936a407e64509f54fecb440d8a78e3c8 Gecko: be81b8d6fae9 Gonk: a4f6f31d1fe213ac935ca8ede7d05e47324101a4 Version: 41.0a1 (3.0) Firmware Version: v18D-1 User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0 --------------------------------------------------
Reporter | ||
Comment 1•9 years ago
|
||
Loose Regression Window by looking at b2g-inbound --------------------------------------------------- Last Working: * http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.bitbar/1996/ First Broken: * http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.bitbar/1997/
Flags: needinfo?(gmealer)
Assignee | ||
Comment 2•9 years ago
|
||
I guess this is a regression from bug 1176408, because the failure comes from marionette-transport-0.5/setup.py. Johnathan, can you take a look at this, please?
Blocks: 1176408
Flags: needinfo?(jgriffin)
Comment 3•9 years ago
|
||
Does one of you have a pushlog link for the given range? I don't really see that this should be marionette-transport related. Having all changesets between those builds would be good to have.
Flags: needinfo?(onelson)
Assignee | ||
Comment 4•9 years ago
|
||
Here is the whole log of the error in a Jenkins job.
Assignee | ||
Comment 5•9 years ago
|
||
Hrm, locally, those changes don't give problems to me. I don't know this is setup on Jenkins, but apparently, there is some mismatch in packages?
Flags: needinfo?(dave.hunt)
Comment 6•9 years ago
|
||
As you can see from this log marionette-transport is correctly installed: Installed /var/lib/jenkins/jobs/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_transport-0.5-py2.7.egg Traceback (most recent call last): File "setup.py", line 57, in <module> install_requires=deps) The traceback happens afterward. Can you check a working build what happens next there?
Assignee | ||
Comment 7•9 years ago
|
||
(In reply to Henrik Skupin (:whimboo) from comment #6) > As you can see from this log marionette-transport is correctly installed: > > Installed > /var/lib/jenkins/jobs/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke. > bitbar/workspace/.env/lib/python2.7/site-packages/marionette_transport-0.5- > py2.7.egg > Traceback (most recent call last): > File "setup.py", line 57, in <module> > install_requires=deps) > > The traceback happens afterward. Can you check a working build what happens > next there? This is from a working Jenkins job. As you can see, it doesn't go past marionette_transport-0.5 setup.
Comment 8•9 years ago
|
||
I think this is just a deps problem. gaiatest is pinned to marionette-driver==0.7, which is pinned to marionette-transport==0.4, but for some reason the logs show marionette-transport 0.5 being loaded. Bumping the marionette-driver dep in gaiatest may resolve this.
Flags: needinfo?(jgriffin)
Comment 9•9 years ago
|
||
marionette-client is pulling in marionette-transport >= 0.4 which is probably causing this. We could either pin that, or remove it, and allow that package dependency to be managed via marionette-driver.
Flags: needinfo?(dave.hunt)
Assignee | ||
Comment 10•9 years ago
|
||
Ok, like this then?
Attachment #8625584 -
Flags: review?(dave.hunt)
Comment 11•9 years ago
|
||
(In reply to Dave Hunt (:davehunt) from comment #9) > marionette-client is pulling in marionette-transport >= 0.4 which is > probably causing this. We could either pin that, or remove it, and allow > that package dependency to be managed via marionette-driver. That should be the better approach, yes. I don't think that you use marionette-transport directly in your gaia tests. :) And marionette-driver definitely knows it better which transport version is best for the moment.
Comment 12•9 years ago
|
||
Comment on attachment 8625584 [details] [diff] [review] 1176882.diff (for check-in) This looks good, however I haven't tested it locally. It should be relatively trivial to reproduce the failure with a clean virtual environment.
Attachment #8625584 -
Flags: review?(dave.hunt) → review+
Assignee | ||
Comment 13•9 years ago
|
||
I did test with a clean environment, but I get this error instead, doing python setup.py develop: Installed /Users/mwargers/.virtualenvs/gaia-py-test/lib/python2.7/site-packages/marionette_transport-0.5-py2.7.egg error: marionette-transport 0.5 is installed but marionette-transport==0.4 is required by set(['marionette-driver'])
Keywords: checkin-needed
Assignee | ||
Updated•9 years ago
|
Attachment #8625584 -
Attachment description: 1176882.diff → 1176882.diff (for check-in)
Comment 14•9 years ago
|
||
You might also have to update the dependency for marionette-driver, where we also released a new version.
Assignee | ||
Comment 15•9 years ago
|
||
(In reply to Henrik Skupin (:whimboo) from comment #14) > You might also have to update the dependency for marionette-driver, where we > also released a new version. Oh, I see what you mean. Yeah, could be, but if the patch fixes this problem, then I'm happy. I don't really know how this works.
Comment 16•9 years ago
|
||
Please see the patch on bug 1176408. marionette-driver has a strict dependency to marionette-transport. So only version 0.9 of driver will work with transport 0.5. Any other combination will cause the bustage as above. Where do you have the dependencies defined? Any URL for us?
Comment 17•9 years ago
|
||
Martijn, would you mind to push this patch to try first? I would feel more comfortable seeing that tests would pass again.
Keywords: checkin-needed
Assignee | ||
Comment 18•9 years ago
|
||
(In reply to Henrik Skupin (:whimboo) from comment #16) > Please see the patch on bug 1176408. marionette-driver has a strict > dependency to marionette-transport. So only version 0.9 of driver will work > with transport 0.5. Any other combination will cause the bustage as above. > > Where do you have the dependencies defined? Any URL for us? I don't know, isn't that what Dave Hunt was saying in comment 4? Marionette-client in testing/marionette/client/requirements.txt ?
Assignee | ||
Comment 19•9 years ago
|
||
(In reply to Henrik Skupin (:whimboo) from comment #17) > Martijn, would you mind to push this patch to try first? I would feel more > comfortable seeing that tests would pass again. Well, these tests are not running on tryserver. They are run on Jenkins, I could do an adhoc run there, but I wouldn't know how to include the gecko part which would include the patch. Johan, would you know how to do this? Or where you talking about some other tests than the Gaia UI tests on Jenkins?
Flags: needinfo?(jlorenzo)
Flags: needinfo?(hskupin)
Comment 20•9 years ago
|
||
This change makes sense so I think we should land regardless.
Comment 21•9 years ago
|
||
But wouldn't it also need a new release of marionette-client? Sounds like that this Jenkins CI is getting its packages from pypi.
Flags: needinfo?(hskupin)
Assignee | ||
Comment 22•9 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #20) > This change makes sense so I think we should land regardless. Re-adding checkin-needed keyword, so we can get the Jenkins jobs running again (currently, QAnalyst need to do those runs locally). (In reply to Henrik Skupin (:whimboo) from comment #21) > But wouldn't it also need a new release of marionette-client? Sounds like > that this Jenkins CI is getting its packages from pypi. Could be, like I said I don't know enough of this subject. I hope to meet up with Dave Hunt this week to get more informed about this subject.
Keywords: checkin-needed
Comment 23•9 years ago
|
||
Well, I doubt that this patch is enough, and I think we still need the version bump too.
Comment 24•9 years ago
|
||
Yes, we'll need to bump and release, but we generally do that in a separate bug.
Assignee | ||
Comment 25•9 years ago
|
||
Ok, I created a new virtualenv. I applied the patch that removed the "marionette-transport >= 0.4" line in testing/marionette/client/requirements.txt of my mozilla-central gecko build, then did python setup.py install there. Then, I went to /Users/mwargers/b2g/gaia_clean/tests/python/gaia-ui-tests and tried to do python setup.py develop there, but got into problems with other dependancies ( error: marionette-driver 0.7 is installed but marionette-driver>=0.8 is required by set(['marionette-client']) ), so I changed the requirements.txt file there to use: marionette_driver==0.7 marionette_client==0.13 After that, I did python setup.py develop again and then I didn't get any errors.
Flags: needinfo?(jlorenzo)
Comment 26•9 years ago
|
||
Inbound is closed or I'd land this myself. Given the chaos of the work week, how important is this? If it is, I can release a new version of marionette-client to pypi ahead of the landing.
Assignee | ||
Comment 27•9 years ago
|
||
It's not a big deal until Monday, at least.
Comment 28•9 years ago
|
||
FWIW, the commit message for your patch should be summarizing what it's doing, not restating the problem being solved by it.
Assignee: nobody → martijn.martijn
Comment 29•9 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/dc94ada04fa8
Keywords: checkin-needed
Assignee | ||
Comment 31•9 years ago
|
||
So this appears still to be a problem. Dave Hunt (from mail): Probably best to reopen that bug and investigate - I suspect we need to pin an additional dependency.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 32•9 years ago
|
||
I just reran a mozilla-central sanity job and it passed - perhaps the fix hadn't fully propogated on the previous build?
Flags: needinfo?(martijn.martijn)
Assignee | ||
Comment 33•9 years ago
|
||
Yeah, the Jenkins job seem to be running now again, too. Looks like this was eventually fixed by bug 1177513, I guess.
Reporter | ||
Comment 34•9 years ago
|
||
Was keeping ni? to watch, issue doesn't appear to be coming back.
Flags: needinfo?(onelson)
Reporter | ||
Updated•9 years ago
|
Flags: needinfo?(gmealer)
You need to log in
before you can comment on or make changes to this bug.
Description
•