Closed Bug 1166431 Opened 9 years ago Closed 6 years ago

Create distinct log files for each call to BaseMarionetteTestRunner.run_tests

Categories

(Remote Protocol :: Marionette, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: whimboo, Unassigned, Mentored)

Details

(Keywords: pi-marionette-runner, Whiteboard: [lang=Python])

As I have seen right now while running our update tests in Jenkins, the CI marked the job as unstable. The reason for it is that we are generating invalid log files. This is happening because we are starting Marionette twice - first for direct updates and a second time for fallback updates.

In this case Marionette does not replace the log files but simply appends more data at the end. Especially for the xunit log files it will end up with multiple "<xml>" root elements.

David, any idea of how we could get this easily fixed? One possibility would be if we check for those command line options and if set, generate a backup of the log files. Or simply numerate them up. I think anything else is deeper in the Marionette client and not that important to get fixed.
Flags: needinfo?(dburns)
I keep thinking I have answered this and noticing I havent so I am sorry it has taken me so long to reply.

I suspect the issue is that somewhere we are creating a new reporter object when starting up the browser/session and that is why we are getting ourselves into bad place. I think it is just because restarts was never thought of in this respect and hopefully we can have a "global" report that we all use during a test run.
Flags: needinfo?(dburns)
Maja, with all of your latest work is this still a problem for e.g. xunit tests? If yes, maybe it's something we can still fix in Marionette proper in one of the next releases?
Flags: needinfo?(mjzffr)
It's still a problem. Let's fix it.
Assignee: hskupin → mjzffr
Blocks: 1210874
Component: Firefox UI Tests → Marionette
Flags: needinfo?(mjzffr)
Product: Mozilla QA → Testing
QA Contact: hskupin
Summary: Firefox UI update tests are generating invalid log files → Create distinct log files for each call to BaseMarionetteTestRunner.run_tests
:whimboo and I have concluded that this is no longer a high-priority change since firefox-update-tests are now run by a mozharness script so the mozharness logs are examined instead. As for the xml log, that was only used by a dashboard for mozmill-ci which is also falling out of use. 

In any case, I also discussed possible approaches to fix the issue. The approach I favour now is swap out the logger's handlers every time BaseMarionetteTestRunner.run_tests is called. In other words, the `logger` object has a list `handlers`. In this list, the StreamHandler objects are responsible for writing to files or stdout. We could add some logic to BaseMarionetteTestRunner.run_tests to flush and replace the appropriate StreamHandler objects whenever we want logging to go to new file(s). All the file names could have a date-stamp. This behaviour should be hidden behind a command-line option so that the default behaviour remains the same, that is, the default file names are reused for all log files so that we don't fill up the local workspace.
Assignee: mjzffr → nobody
No longer blocks: 1210874
Mentor: mjzffr
Status: ASSIGNED → NEW
Whiteboard: [lang=Python]
[mass update] Setting Harness bugs to all P3
Priority: -- → P3
I’m going to close this bug since the last update two years ago said it was no longer a high priority.  It is likely never going to become a high priority.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
It was a nice to have thing for the Firefox UI update tests. But those won't exist that long anymore.
We should make sure that when (if?) we move to a new test harness
based on pytest, it uses separate log files.
Product: Testing → Remote Protocol
You need to log in before you can comment on or make changes to this bug.