Currently, Tecken's download API takes a url path and then looks up that path in several AWS S3 buckets verbatim. It doesn't know anything about debug filenames, debug ids, code ids, structure of what's in the AWS S3 buckets, or how to make a hot granola treat for those saturday morning blues.

Wouldn't it be great if Tecken could find symbols by any of the following:

1. debug filename / debug id / symbol file
2. debug filename / code id / symbol file

Why?

Item 1 supports existing Breakpad-style symbol supplier which itself supports Microsoft Symbol Server supporting tools. We have this already.

Item 2 would be handy for bug #1689983 and other cases where for whatever reason we have a code id but no debug id. I think this requires us to collect and store more data at the time we handle the symbol upload. That enables Tecken to look up the (debug filename / code id) to get the debug id and then look that up in the AWS S3 buckets.

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #0)

2. debug filename / code id / symbol file

Note, this would have to be:

executable-or-library name / code id / symbol file

That's because we don't know what the debug name is like. If the debug ID is missing so is the debug filename.

Can you list some examples where the executable-or-library-name isn't the same as a debug filename?

Here's a few examples taken from 44aaa336-b46d-424f-aae7-cdfb70211221, some case differences (there's more):

• bcryptPrimitives.dll → bcryptprimitives.pdb
• kernel.appcore.dll → Kernel.Appcore.pdb
• uxtheme.dll → UxTheme.pdb
• IPHLPAPI.DLL → iphlpapi.pdb

Different names:

• msvcp140.dll → msvcp140.amd64.pdb
• VCRUNTIME140.dll → vcruntime140.amd64.pdb
• VCRUNTIME140_1.dll → vcruntime140_1.amd64.pdb

Ahh... right. I remember having to do some additional work in Eliot to capture the filename from the sym file.

I'm going to have to think about this. Maybe it'd be better to build a symbol query API.

https://bugzilla.mozilla.org/show_bug.cgi?id=1757890#c10

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment 8)

Maybe someone has a script that's using code_id/code_filename for downloading symbols? I know we talked about something like that a while back (or my memory is bad on this).

People opening minidumps in Visual Studio or windbg would do that. They'd try to fetch the DLL/EXE and use the code ID for that.

That's a pretty compelling use case for this bug and probably dictates how the API changes need to be.

Earlier this month, we moved Firefox Windows 7 and 8.1 users from release channel to ESR. That's caused a large, noticeable spike in the number of xul.dll/000000000000000000000000000000000 in the "Weekly report of modules with missing symbols in crash reports" emails. That email is generated from Socorro crash report data in BigQuery, which is generated by the stackwalker.

I'm going to focus on capturing the information we need for the lookup and adjusting things without changing the API. We can look at changes to the API later.

We've got debug id, debug filename, code id, and code filename. In order to make sure I can figure these values out correctly from the sym files, here's the sym files:

Windows: xul.pdb/46A0ADB3F299A70B4C4C44205044422E1/xul.sym

https://symbols.mozilla.org/uploads/files/file/49541198

MODULE windows arm64 46A0ADB3F299A70B4C4C44205044422E1 xul.pdb
INFO CODE_ID 64EC878F867C000 xul.dll
INFO GENERATOR mozilla/dump_syms 2.2.0

• debug filename: xul.sym -- computed from xul.pdb drop .pdb add .sym
• debug id: 46A0ADB3F299A70B4C4C44205044422E1
• code filename: xul.dll
• code id: 64EC878F867C000

Linux: libxul.so/9CEDAF00EDF6372A8DD3A5727D1306590/libxul.so.sym

https://symbols.mozilla.org/uploads/files/file/49540555

MODULE Linux x86_64 9CEDAF00EDF6372A8DD3A5727D1306590 libxul.so
INFO CODE_ID 00AFED9CF6ED2A378DD3A5727D130659E179A902
INFO GENERATOR mozilla/dump_syms 2.2.0

• debug filename: libxul.so.sym -- computed from libxul.so add .sym
• debug id: 9CEDAF00EDF6372A8DD3A5727D1306590
• code filename:
• code id: 00AFED9CF6ED2A378DD3A5727D130659E179A902

Note: linux modules don't have a code filename

Mac: XUL/4C4C44E655553144A1E39130609F119F0/XUL.sym

MODULE Mac x86_64 4C4C44E655553144A1E39130609F119F0 XUL
INFO CODE_ID 4C4C44E655553144A1E39130609F119F
INFO GENERATOR mozilla/dump_syms 2.2.0

• debug filename: XUL.sym computed from XUL add .sym
• debug id: 4C4C44E655553144A1E39130609F119F0
• code filename:
• code id: 4C4C44E655553144A1E39130609F119F

Note: macos modules don't have a code filename.

I think the plan is something like this:

1. I'll adjust the db to have some additional fields: debug filename, debug id, code filename, code id
2. Then I'll adjust the upload API view to capture debug filename, debug id, code filename, and code id for sym files
3. The stackwalker used to pass a code_filename and code_id via querystring params. I'm going to make sure that's still the case and then adjust the download API view in Tecken to do a lookup by code_filename and code_id to get the real debug id if the debug id is 000000000000000000000000000000000.

After that's landed and figured out, I'll figure out how to backfill the information for modules we're missing that are showing up in the missing symbols report and reprocess relevant crash reports.

Gabriele: Does that sound right to you?

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #6)

I think the plan is something like this:

1. I'll adjust the db to have some additional fields: debug filename, debug id, code filename, code id
2. Then I'll adjust the upload API view to capture debug filename, debug id, code filename, and code id for sym files
3. The stackwalker used to pass a code_filename and code_id via querystring params. I'm going to make sure that's still the case and then adjust the download API view in Tecken to do a lookup by code_filename and code_id to get the real debug id if the debug id is 000000000000000000000000000000000.

We use code_id and code_file, not code_filename. Maybe I got that wrong when I added this feature to the new stackwalker?

https://github.com/rust-minidump/rust-minidump/blob/7cda551f9f082f8ba5a286f931b5ad24e34861d9/breakpad-symbols/src/http.rs#L210-L211

Gabriele: Does that sound right to you?

Yes. Just one note: the code ID and filename don't really matter on Linux and macOS. Both operating systems don't really have a concept of a code ID and so Breakpad made it up; but we don't use it for anything (while on Windows we could use those bits instead of the debug ID/filename). You could index only the Windows ones if you want to.

The Tecken download API documents it as code_id and code_filename:

https://tecken.readthedocs.io/en/latest/download.html#head---DEBUG_FILENAME---DEBUG_ID---SYMBOL_FILE-

I don't know whether the code_id and code_file query string params are Mozilla-created or not. I don't have a preference as to whether we change the download API and docs in Tecken or update the stackwalker code. If you don't have a preference either, I'll look into whether anything else is using those query string parameters and if not, fix it in Tecken.

I'll dig into the breakpad sources to see if it comes from there.

The old stackwalker used code_file and code_id, so that's where this comes from. I haven't found instances of code_filename in the old stackwalker, breakpad or mozilla-central, maybe it was a typo in Tecken?

You're right--I documented it wrong in Tecken. That's great because that's an easy fix.

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #6)

large, noticeable spike in the number of xul.dll/000000000000000000000000000000000 in the "Weekly report of modules with missing symbols in crash reports" emails.

Hmm, why does this say xul.dll and not xul.pdb? I haven't seen these emails before, do these emails usually list the code name and not the debug name?

They're two different files related to the same module, so I think it's bookkeeping kinds of things. The code that generates the emails is here:

https://github.com/marco-c/missing_symbols/blob/master/modules-with-missing-symbols.ipynb

(Pretty sure that's the one.)

I messed up the details in comment #6.

It should be this to match the tecken download api:

Windows: xul.pdb/46A0ADB3F299A70B4C4C44205044422E1/xul.sym

https://symbols.mozilla.org/uploads/files/file/49541198

MODULE windows arm64 46A0ADB3F299A70B4C4C44205044422E1 xul.pdb
INFO CODE_ID 64EC878F867C000 xul.dll
INFO GENERATOR mozilla/dump_syms 2.2.0

• debug filename: xul.pdb
• debug id: 46A0ADB3F299A70B4C4C44205044422E1
• code file: xul.dll
• code id: 64EC878F867C000

When the stackwalker requests that module but has a debug id of 000000000000000000000000000000000, the url looks like this:

https://symbols.mozilla.org/xul.pdb/000000000000000000000000000000000/xul.sym?code_file=xul.dll&code_id=64EC878F867C000

The download API view will see the debug id 000000000000000000000000000000000 which will prompt it to do a lookup in the db using either the debug filename (xul.pdb) and code id (64EC878F867C000) or the code file (xul.dll) and code id (64EC878F867C000). I haven't decided which to do, yet. Probably the former (debug filename and code id).

Is it possible to for duplicate (debug filename / code id) pairs to exist in the world? If we hit one, how should we resolve it? Maybe pick the most recently uploaded which may be wrong, but at least it's stable?

gsvelto: I'm adding debug_filename, debug_id, code_file, and code_id fields to the db table. Any idea how many characters to allot for those?

debug_id (the version with no hyphens) is 33 characters. For example, 46A0ADB3F299A70B4C4C44205044422E1.

I don't know if there are limits on the other fields. The "spec" for sym files doesn't talk about maximum lengths anywhere. I haven't seen any documentation on how code_id is generated which would suggest how long it could be.

Any ideas on how much space to allot for the data we're getting?

willkg merged PR #2786: "bug 1746940: add FileUploadAdmin and UploadAdmin to Django admin" in d5fb2ef.

Gabriele said in Matrix:

debug_filename and code_file don't have limits, code_id is a 48-bit value IIRC, so it should never be more than 12 hexadecimal characters
I have to check in the documentation but off the top of my head it's that

I'll go with that.

I've got it working in my local dev environment. When a download is requested and the debug id is the bogus 000000000000000000000000000000000 value and code_file and code_id are specified in the query string, then the view code will look up code_file/code_id in the file upload table to find the correct debug_id. If it gets one, then it'll return an HTTP 302 redirect to the download API url with the correct debug id.

Here's an example exchange:

> GET /xul.pdb/000000000000000000000000000000000/xul.sym?code_id=64E130A115A30000&code_file=xul.dll HTTP/1.1
> Host: web:8000

< HTTP/1.1 302 Found
< Location: /xul.pdb/404B9729BE96C3CF4C4C44205044422E1/xul.sym?code_id=64E130A115A30000&code_file=xul.dll

> GET /xul.pdb/404B9729BE96C3CF4C4C44205044422E1/xul.sym?code_id=64E130A115A30000&code_file=xul.dll HTTP/1.1
> Host: web:8000

< HTTP/1.1 302 Found
< Location: http://localstack:4566/publicbucket/v1/xul.pdb/404B9729BE96C3CF4C4C44205044422E1/xul.sym

> GET /publicbucket/v1/xul.pdb/404B9729BE96C3CF4C4C44205044422E1/xul.sym HTTP/1.1
> Host: localstack:4566

< HTTP/1.1 200 
< Content-Length: 149120754
...

This involves a non-trivial db migration, so I have to land it in parts.

Part 1

1. create a PR:
   1. commit that disables run_migrations.sh from running during the deploy [1]
   2. commit that adds the migration that adds the sym debug fields to the fileupload table
2. review and land PR. that deploys it to stage.
3. write up DSRE issue for running cleanse_upload command and then running migrations in stage. run migrations.
4. deploy to prod.
5. write up DSRE issue for running cleanse_upload command and then migrations in prod. run migrations.

Part 2

1. create a PR with everything else
2. review and land PR. that deploys it to stage.
3. add dashboard graph for tecken.download_symbol_code_id_lookup.
4. test uploads and downloads on stage. test with socorro processor. make sure graph shows the right thing.
5. deploy to prod.
6. add dashboard graph for tecken.download_symbol_code_id_lookup.

[1] We might want to permanently disable running migrations during deploy since Mozilla Symbols Server production has some big tables and almost all migrations are rough, but that's a thing to think about some other day.

willkg merged PR #2788: "bug 1746940: migration for adding debug information to fileupload table (part 1)" in 92af4eb.

This is part 1 of the plan. It includes the commit that disables running migrations during the deploy and the migration to add the fields to the fileupload table.

I'll follow this through to production. Then we'll work on part 2.

willkg merged PR #2789: "bug 1746940: fix FileUpload admin; add fake data generator" in 510d292.

willkg merged PR #2791: "bug 1746940: debug_id lookup by code_file/code_id (part 2)" in e4ed26d.

I landed some minor fixes and the second part. I'll wait for stage to deploy the changes and then I'll test it on stage.

The changes deployed to stage. However, when I point my local Socorro at stage, it's still now getting symbol files for crash reports where the minidump is missing the debug id. It looks like the stackwalker sees the missing debug id and then doesn't even try to download the file. That's not what I thought was happening, so I'm a bit surprised.

I wrote up an issue in the rust-minidump tracker. Assuming I'm correct, they'll need to change how the stackwalker works.

https://github.com/rust-minidump/rust-minidump/issues/867

All my other testing worked fine:

1. uploaded sym files get parsed and the relevant information is added to the fileupload record for both Windows and non-Windows module symbol files
2. the symbol debug data shows up in the file page in the ui and the FileUpload admin page
3. the download API works and redirects to the download API url with the corrected debug id

We can deploy the rest of this.

When I was testing, I was testing with bp-421408bf-aa63-4e5e-8af9-8ee3a0230907.

That has xul.dll modules information like this:

{
    "base_addr": "0x69920000",
    "cert_subject": "Mozilla Corporation",
    "code_id": "64e782c570c4000",
    "corrupt_symbols": false,
    "debug_file": "",
    "debug_id": "000000000000000000000000000000000",
    "end_addr": "0x709e4000",
    "filename": "xul.dll",
    "loaded_symbols": false,
    "missing_symbols": true,
    "symbol_url": null,
    "version": "117.0.0.8636"
},

That has a bogus debug id, but also has no debug filename. So the stackwalker doesn't try to request a symbol file.

I looked at other crash reports where one of the modules in the stack has a debug_id of 000000000000000000000000000000000 and they all have an empty debug filename.

... which is what Gabriele said in comment #1.

Given that, what I was implementing as outlined in comment #6 and comment #14 won't work. We'll need to adjust the download API to also support:

/{code_file}/{code_id}/{sym_file}

Then we'll need to adjust the stackwalker to try that, too, instead of bailing if there's no debug filename or debug id.

I'll make that change and then open up an issue in rust-minidump to add support for it.

willkg merged PR #2792: "bug 1746940: minor fixes" in 23daea4.

willkg merged PR #2793: "bug 1746940: rework download API to support /code_file/code_id/sym_file" in 7614fbc.

I wrote up an issue in rust-minidump to add support for requesting the symbol file with /code_file/code_id/sym_file.

https://github.com/rust-minidump/rust-minidump/issues/870

willkg merged PR #2798: "bug 1746940: move info lookup to syminfo API endpoint" in 0a94b20.

This should fix the problem we were having when pushing this code to production. It takes the lookup out of the download API and moves it to a separate API where I can work on optimizing it in the production environment without it taking out production again.

Next steps:

1. test on stage
2. deploy to production, test in production, gather performance data, optimize the lookup and API
3. depending on performance
   1. if we can get it good enough, re-add the improved lookup to the download API
   2. if we can't get it good enough, leave it as separate API and change the plan for this bug

willkg merged PR #2799: "bug 1746940: fix metric keys so they're consistent with existing metrics" in c62180c.

willkg merged PR #2800: "bug 1746940: thread refresh_cache through api params" in f3b84ee.

Everything up to now was deployed to prod just now in bug #1854172.

willkg merged PR #2812: "bug 1746940: implement codeinfo lookup in download API" in 19b6773.

This re-implements the codeinfo lookup in the download API. It's more constrained on when it performs the lookup, it caches results, and the lookup uses indexes on the table. I think that'll alleviate the problems we had the first time around.

We'll test before and after timings on stage and see how timings are affected.

Timings before (0.4s - 0.8s):

2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650B24AF8362000/xul.sym,404,none,0.6887755393981934
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650B23E18A1B000/xul.sym,404,none,0.5132982730865479
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650C1C9080B1000/xul.sym,404,none,0.7136039733886719
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650C1CC092D3000/xul.sym,404,none,0.512908935546875
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650B23319467000/xul.sym,404,none,0.8286173343658447
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650B24A3EE56000/xul.sym,404,none,0.8235323429107666
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650B25737A25000/xul.sym,404,none,0.7039215564727783
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/xul.dll/650C1C8A919A000/xul.sym,404,none,0.49950623512268066
2023-10-04 10:31:51,ee97ec8dcc6ce7c3e46ba718fe793830fb407e3d,/try/jsapi-tests.exe/650C28161E78000/jsapi-tests.sym,404,none,0.490
21387100219727

Timings after (0.5s - 0.6s):

2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650B24AF8362000/xul.sym,302,/try/xul.pdb/18CF28149AB52
7FB4C4C44205044422E1/xul.sym?_refresh=1,0.5119500160217285
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650B23E18A1B000/xul.sym,302,/try/xul.pdb/37F7581DBA96E
81D4C4C44205044422E1/xul.sym?_refresh=1,0.5411970615386963
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650C1C9080B1000/xul.sym,302,/try/xul.pdb/42FBEF05F4484
D744C4C44205044422E1/xul.sym?_refresh=1,0.5016095638275146
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650C1CC092D3000/xul.sym,302,/try/xul.pdb/57B80E469B0E2
52D4C4C44205044422E1/xul.sym?_refresh=1,0.5481138229370117
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650B23319467000/xul.sym,302,/try/xul.pdb/699B1D693B39A
66A4C4C44205044422E1/xul.sym?_refresh=1,0.5255749225616455
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650B24A3EE56000/xul.sym,302,/try/xul.pdb/6B8753C71BD03
5D54C4C44205044422E1/xul.sym?_refresh=1,0.5238499641418457
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650B25737A25000/xul.sym,302,/try/xul.pdb/908C1D71C5212
4544C4C44205044422E1/xul.sym?_refresh=1,0.5849294662475586
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/xul.dll/650C1C8A919A000/xul.sym,302,/try/xul.pdb/B5F2F1863A506
4364C4C44205044422E1/xul.sym?_refresh=1,0.6173436641693115
2023-10-04 11:11:23,19b6773827e87ed2e7fddc4cc425208c7d365c6b,/try/jsapi-tests.exe/650C28161E78000/jsapi-tests.sym,302,/try/jsapi
-tests.pdb/DA78C0594BBA62FC4C4C44205044422E1/jsapi-tests.sym?_refresh=1,0.6146829128265381

Changes in timings seem fine--both sets are in roughly the same sub-1s range.

Redirects look good. Request for /try/xul.dll/650C1C8A919A000/xul.sym turns into /try/xul.pdb/B5F2F1863A506 4364C4C44205044422E1/xul.sym.

I think this is good to push to prod.

I deployed the codefile lookup changes to prod in bug #1856987. Everything looks good so far. Download API timings didn't change. I'll go back to working on rust-minidump changes next.

Great news, thanks!

This blocks S2 bugs, so it's S2.

willkg merged PR #10: "bug 1746940: update stackwalker to 3819edf (past v0.18.0)" in 070a26b.

willkg merged PR #11: "bug 1746940: update stackwalker to 1869d3a0" in e14858b.

That's the binary builds of rust-minidump that we use for Socorro. I'll update Socorro to use the new builds next.

willkg merged PR #6484: "bug 1746940: update stackwalker to v20231013.0 with codeinfo lookup" in 07e6954.

This deployed to stage just now.

On stage, we have crash bp-76175739-874a-494c-8d66-773a30231013 which was processed earlier today. The stack looks like:

0  xul.dll  xul.dll@0x24d742f  
1  xul.dll  xul.dll@0x24d7603  
2  xul.dll  xul.dll@0x24d7391  
3  xul.dll  xul.dll@0x197f893  
4  xul.dll  xul.dll@0x8f4840  
5  xul.dll  xul.dll@0xd0b7a2  
6  xul.dll  xul.dll@0x3641994  
7  xul.dll  xul.dll@0x2bfca3  
8  xul.dll  xul.dll@0x36132bd  
9  xul.dll  xul.dll@0x36126d0  

I reprocessed it just now after stage deployed the last of the code changes and now the stack is this:

0  xul.dll  mozilla::ipc::FatalError  ipc/glue/ProtocolUtils.cpp:180
1  xul.dll  mozilla::ipc::IProtocol::HandleFatalError  ipc/glue/ProtocolUtils.cpp:412
2  xul.dll  mozilla::ipc::IProtocol::FatalError  ipc/glue/ProtocolUtils.cpp:403
2  xul.dll  mozilla::ipc::PickleFatalError  ipc/glue/ProtocolUtils.cpp:157
3  xul.dll  IPC::MessageWriter::FatalError const  ipc/chromium/src/chrome/common/ipc_message_utils.h:116
3  xul.dll  IPC::MessageBufferWriter::MessageBufferWriter  ipc/chromium/src/chrome/common/ipc_message_utils.cc:32
4  xul.dll  IPC::ParamTraits<JSStructuredCloneData>::Write  ipc/glue/SerializedStructuredCloneBuffer.cpp:25
5  xul.dll  IPC::WriteParam  ipc/chromium/src/chrome/common/ipc_message_utils.h:441
5  xul.dll  IPC::ParamTraits<mozilla::SerializedStructuredCloneBuffer>::Write  ipc/glue/SerializedStructuredCloneBuffer.h:77
5  xul.dll  IPC::WriteParam  ipc/chromium/src/chrome/common/ipc_message_utils.h:441

I'm going to keep an eye on timings and other things for stage over the next few days and then if everything looks good, we'll deploy it to production.

I deployed the last part of this (the new stackwalker) in bug #1860841.

I reprocessed bp-58a11cc1-9c69-4f85-8fc8-7dedf0231024 and it did a codeinfo lookup and found symbols for xul.dll.

This is now working.

I have to update the Tecken download API docs. I want to do a retrospective. I'll do those this week.

willkg merged PR [mozilla-services/tecken]: bug 1746940: document codeinfo lookup in download API (#2827) in 5812a13.