[:philipp] Reporter
	Description • 6 years ago

in the past couple of hours, the [@ EMPTY: no crashing thread identified; unknown error] signature is sky-rocketing: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=EMPTY%3A%20no%20crashing%20thread%20identified%3B%20unknown%20error#graphs

this started happening yesterday at exactly 19:00 UTC. the crash comments and sparse information in the reports indicate the signature is happening cross-platform and in many different circumstances, so i'd guess there is a problem processing the crashes by socorro (related to bug 1506669 or bug 1506635?).

	Rob Wu [:robwu]
	Comment 1 • 6 years ago

I can consistently reproduce this problem on Linux (stable 63 and Nightly 65):

1. Open browser content toolbox.
2. Run: Object.keys(tabs[0].content.Proxy.prototype);  // this crash is related to bug 1506736
3. Submit crash report.

Example of result: https://crash-stats.mozilla.com/report/index/1320578c-776c-4384-8cd9-3564a0181113

	Marcia Knous [:marcia]
	Comment 2 • 6 years ago

All of my Mac crashes are coming up with the Empty signature. I noticed it yesterday when trying to reproduce a web replay crash.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 3 • 6 years ago

Grabbing it to look into. Those crashes have this in the processor notes:

/stackwalk/stackwalker output failed in json: Expected object or value

I'll look at why mdsw is failing.

After figuring it out and fixing it, I'll reprocess affected crashes.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 5 • 6 years ago

I figured out the problem and I'm testing a fix now. I sent an email to the stability list to let everyone know where things are at.

If all goes well, I can have the fix in production in a couple of hours. I'll reprocess affected crashes after that.

	[github robot]
	Comment 6 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/c8e9c8d4a3e52ab07c8a04c9ed6259a0abb01fe4
fix bug 1506781: fix silent ujson errors

minidump-stackwalker was failing with exit code 134 because there was no
raw crash data to work on. There was no raw crash data to work on because
ujson and json handle parsing errors differently. Particularly, ujson
would shrug when trying to JSON serialize a configman DotDict. That's
really unhelpful.

This fixes the underlying issue (need to convert configman DotDict
to a dict before serializing to JSON), purges ujson from the codebase,
and fixes some tests that were testing what happens if the constants
of the universe were different which will never be the case.

https://github.com/mozilla-services/socorro/commit/398da01caffd3b5d63caffc471029715ba6eafce
Merge pull request #4709 from willkg/1506781-empty

fix bug 1506781: fix silent ujson errors

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 8 • 6 years ago

I'm reopening this and will close it after I've pushed the fix to prod and reprocessed affected crashes.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 9 • 6 years ago

I pushed the fix to prod and reprocessed 266k crashes. We should be all set now. Marking as FIXED.