[traceback] SymCacheErrorBadDebugFile: Unhandled bad debug file caused by: parsing error caused by: in section "inline origin record body"
Categories
(Eliot :: General, defect, P2)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
References
Details
Attachments
(2 files)
Sentry: https://sentry.io/organizations/mozilla/issues/3608028371/?query=is%3Aunresolved
Traceback:
SymCacheErrorBadDebugFile: bad debug file
caused by: parsing error
caused by: in section "inline origin record body" at
"INLINE_ORIGIN 579273"
^
,
in section "inline origin id" at
"INLINE_ORIGIN 579273"
^
,
expected whitespace at
"INLINE_ORIGIN 579273"
^
File "falcon/app.py", line 365, in falcon.app.App.__call__
responder(req, resp, **params)
File "markus/main.py", line 515, in _timer_decorator
return fun(*args, **kwargs)
File "eliot/symbolicate_resource.py", line 648, in on_post
results.append(self.symbolicate(stacks, modules, debug_stats))
File "eliot/symbolicate_resource.py", line 432, in symbolicate
self.get_symcache(module_info, debug_stats)
File "eliot/symbolicate_resource.py", line 350, in get_symcache
symcache = self.parse_sym_file(debug_filename, debug_id, sym_file)
File "markus/main.py", line 515, in _timer_decorator
return fun(*args, **kwargs)
File "eliot/symbolicate_resource.py", line 281, in parse_sym_file
return parse_sym_file(debug_filename, debug_id, data, self.tmpdir)
File "eliot/libsymbolic.py", line 129, in parse_sym_file
symcache = obj.make_symcache()
File "symbolic/debuginfo.py", line 133, in make_symcache
self._methodcall(lib.symbolic_symcache_from_object)
File "symbolic/utils.py", line 57, in _methodcall
return rustcall(func, self._get_objptr(), *args)
File "symbolic/utils.py", line 92, in rustcall
raise exc
I talked with gsvelto about this and we wrote up bug #1791785 that eventually led to a fix in dump_syms. However, we're still seeing the problem, so this bug is about manually fixing the problematic symbols files.
Assignee | ||
Comment 1•2 years ago
|
||
Yesterday, I went through the Sentry events for October and made this list:
- 441450BE1030C69946EC77871CA83C5D0 libglib-2.0.so.0
- E32113BBF16D446E4469E4E2FB57323B0 libgobject-2.0.so.0
- 435BCC7E34F1C98450F4BABE863BDBA80 libsoftokn3.so
- 9B39A7109EF1C7067EE75F075686EB5D0 libplc4.so
- 40A596D5CD5E58199912260DB81747880 libfreeblpriv3.so
- 2CBA64CC6D0E57AB7F6272B686B839D70 libplds4.so
They all fail to parse with symbolic because there's an INLINE_ORIGIN
line that has an index, but no name. Something like this:
INLINE_ORIGIN 3
Removing that line allows the files to be parsed and then everything seems fine after that.
I wrote some scripts to download files, let me fix them, verify that they parse, and then upload them which stomps on the problematic file.
https://symbols.mozilla.org/libgobject-2.0.so.0/E32113BBF16D446E4469E4E2FB57323B0/libgobject-2.0.so.0.sym
https://symbols.mozilla.org/libplds4.so/2CBA64CC6D0E57AB7F6272B686B839D70/libplds4.so.sym
https://symbols.mozilla.org/libfreeblpriv3.so/40A596D5CD5E58199912260DB81747880/libfreeblpriv3.so.sym
https://symbols.mozilla.org/libglib-2.0.so.0/441450BE1030C69946EC77871CA83C5D0/libglib-2.0.so.0.sym
https://symbols.mozilla.org/libsoftokn3.so/435BCC7E34F1C98450F4BABE863BDBA80/libsoftokn3.so.sym
https://symbols.mozilla.org/libplc4.so/9B39A7109EF1C7067EE75F075686EB5D0/libplc4.so.sym
New problematic file since yesterday:
- CF951B6F53184F670E0966A66CD0D9520 libxul.so
Assignee | ||
Comment 2•2 years ago
|
||
Gabriele: This recent problematic file is a libxul.so. Does the fact that it's not a system library suggest we've got other scenarios where the INLINE_ORIGIN
doesn't have a name?
Also, if I fix this file by removing the INLINE_ORIGIN
line, what should I do with INLINE
lines that reference it? Should I remove them, too? Should I just give the INLINE_ORIGIN
a name of "unknown" or something like that instead?
Assignee | ||
Comment 3•2 years ago
|
||
Gabriele urged us to fix dump_syms and not hand-fix sym files which makes sense especially if there were a lot of broken files and hand fixing took a while. I think in this case, I want Eliot to stop kicking up parse issues and the number of files to fix is small and it's pretty straight-forward to do.
The fix Gabriele did in dump_syms is here:
Instead of removing the INLINE_ORIGIN
lines, I should instead give them a name of <name omitted>
. I'll refix the ones I fixed before and fix the new one accordingly.
Assignee | ||
Comment 4•2 years ago
|
||
I re-added the INLINE_ORIGIN
lines I removed with the <name omitted>
name and uploaded all the fixed files.
I'll keep this open for a week to catch additional problems.
Assignee | ||
Comment 5•2 years ago
|
||
Assignee | ||
Comment 6•2 years ago
|
||
Comment 7•2 years ago
|
||
(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #2)
Gabriele: This recent problematic file is a libxul.so. Does the fact that it's not a system library suggest we've got other scenarios where the
INLINE_ORIGIN
doesn't have a name?
I suspect these are from one of Fedora builds for which we started dumping inlined functions like we do for our builds. Fedora splits the debug information of Firefox in two packages: one with the DWARF debug info but no strings, and one with Firefox sources. The DWARF info points to the sources and dump_syms doesn't support following those links, so we're often missing function names when dumping those builds.
Assignee | ||
Comment 8•2 years ago
|
||
I fixed:
- B68D6DF6915DDCBD20F569A412D6886E0 libssl3.so
Assignee | ||
Comment 9•2 years ago
|
||
I fixed:
- CDEB344D4860E40AB7E154F05189CC730 libnssutil3.so
Assignee | ||
Comment 10•2 years ago
•
|
||
I fixed:
- 015C5165CC511800A25E59423F0D5CB40 libxul.so
- 7792EBC12F9B97342563B21F4F0C0D4A0 firefox
Assignee | ||
Comment 11•2 years ago
|
||
I fixed:
- C1E3C105E3B1B521F24D8C37A0F3CF700 libnssckbi.so
Assignee | ||
Comment 12•2 years ago
|
||
I fixed:
- 3A4AF6D359B804DE5E124A02713562960 libcups.so.2
It's been a while since I fixed one of these, so I'm going to mark this FIXED and if any other issues come up, I'll continue to fix them manually.
Assignee | ||
Comment 13•1 year ago
|
||
Assignee | ||
Comment 14•1 year ago
|
||
Assignee | ||
Updated•1 year ago
|
Assignee | ||
Comment 15•1 year ago
|
||
I've fixed a bunch more files. The most recent one was due to the symbol being entirely whitespace. I opened up this issue to cover that:
https://github.com/mozilla/dump_syms/issues/543
That was fixed in dump_syms 2.2.0. Then I fixed:
- libxul.so 5AC6F470F832952B3246C5EADF3017AA0
Assignee | ||
Comment 16•3 months ago
|
||
I fixed glxtrace.so 63A0092F5FE8497FB1B7CC9E59E031AF0
Description
•