Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Description • 6 years ago

Incoming crash reports for beta channel report a version like "62.0" and we need to convert that to something like "62.0b2" so that it can be queried and show up correctly.

Currently, the BetaVersionRule uses the VersionString API in the webapp to convert a (product, channel, version, buildid) into the actual version. The VersionString API queries the product_versions and product_version_builds tables to do this.

In the near future (bug #1366301), we're going to scrap all that in favor of a much simpler data model for product and version information and switch from scraping archive.mozilla.org directly to using Buildhub. Pretty sure we can rewrite the BetaVersionRule to hit Buildhub instead of the VersionString API in the webapp.

This bug covers rewriting the BetaVersionRule to use Buildhub.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 1 • 6 years ago

I'm going to make this block 1366301 for now. The reason is that if we can get this to work, it's one less requirement for the data we're getting from Buildhub and storing and that reduces the work required for bug #1366301.

Grabbing it to do soon.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 2 • 6 years ago

I think this lets us ditch the AuroraVersionFixitRule, too. DOUBLE-RAINBOW!

This is the first Buildhub thing, so I tossed Benson an email giving him a heads up.

	[github robot]
	Comment 4 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/c0aad6f54e8805c06d232c3a713ef4cc5c247c0c
fix bug 1501780: rewrite BetaVersionRule to use Buildhub

This removes the VersionString API which we no longer need since the
BetaVersionRule was the only user of it.

This removes the AuroraVersionFixitRule because we don't need it anymore.
That rule was in place because ftpscraper and the related stored procedures
didn't handle devedition aurora well causing the BetaVersionRule to muck
up version strings for b1 and b2. Buildhub doesn't have that problem so
we can get rid of it.

https://github.com/mozilla-services/socorro/commit/cc35e7c41c2b27db4b3ed373b2b9fa1c2b216e81
Merge pull request #4658 from willkg/1501780-betaversionrule

fix bug 1501780: rewrite BetaVersionRule to use Buildhub

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 5 • 6 years ago

This worked super except it's not working for devedition.

For example, this crash report reports "Firefox" as the product and "aurora" as the channel, but it's probably devedition:

https://crash-stats.allizom.org/report/index/f86e4eab-f319-4534-a209-c38f30181024

I think we need to adjust the rule to query Buildhub with "devedition" as the product instead of Firefox for aurora. Reopening to do that.

	[github robot]
	Comment 7 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/c88889ced2ef9c1b313fb8aca669997cb6328126
fix bug 1501780: fix BetaVersionRule lookups for aurora

https://github.com/mozilla-services/socorro/commit/8b6d08dddbf40189e54d89b288ada5d61937c74d
Merge pull request #4664 from willkg/1501780-fix-aurora

fix bug 1501780: fix BetaVersionRule lookups for aurora

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 8 • 6 years ago

Several things this did:

1. This alleviates the need (and maintenance of) the AuroraVersionFixitRule, so I removed it.

2. All aurora (which are devedition) crashes are coming up with the right version now.

3. This now fixes versions with rc version numbers. Pretty sure that's bug #1295963.


Before I push this to prod, I'll send an email to the stability mailing list since it's changing version numbers of things.

There's one regression (so I'm reopening).

Stage with the rewritten BetaVersionRule gets 63.0b0:

   https://crash-stats.allizom.org/report/index/b24d2c58-3b1c-411c-a85d-b10020181025

Prod gets 63.0b99:

   https://crash-stats.mozilla.com/report/index/b24d2c58-3b1c-411c-a85d-b10020181025

("firefox", "20181018182531", "beta") points to nothing in Buildhub, but if we switch the channel to "release", then it points to this:

   https://mozilla-services.github.io/buildhub/?q=20181018182531

That's a release build.

In the case where it's in the beta channel and the first lookup got nothing, should the processor do an additional lookup for the release channel and if that gives a hit, then add a b99 to the end?

Calixte, Marco: What do you want Socorro to do here?

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 9 • 6 years ago

I've got other things in the queue on stage that are waiting on this to go out, so I'll push this out now as is and we can fix the issue and reprocess later.

	[github robot]
	Comment 10 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/467e83bad0017b1a3767134e646e6449fbfa51d3
fix bug 1501780: fix b99 and reduce buildhub hits

ftpscraper would generate a product_versions record for X.0b99 which is a
"final beta". This fixes the BetaVersionRule to do a lookup for
(product, build id, beta) channel and if that returns nothing and the
version is possibly a final beta version, it will do a second lookup for
(product, build id, release) and if that returns something, it adds a b99
to the end of the version string.

Buildhub only stores build information for Firefox and Fennec, so we shouldn't
bother hitting it for other products. This fixes that.

This also splits up the monolithic "test_everything" into seprate tests.

https://github.com/mozilla-services/socorro/commit/9f8ed469a2c5d128d6c8926c4218f8a7b16e9043
Merge pull request #4670 from willkg/1501780-fix-99

fix bug 1501780: fix b99 and reduce buildhub hits

	[github robot]
	Comment 11 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/464863c0000483d1de7fa0c6b4fddf40591dc34a
fix bug 1501780: fix the rc case

Previously, we tossed on a b99, but it turns out that Buildhub had the right
version, but because we were doing a limit 1 and not sorting, it'd
intermittently come up with the right version and the wrong version. Yay!

This fixes that and ditches the whole b99 thing and now we're in a much better
place!

https://github.com/mozilla-services/socorro/commit/e0ec3845d206e1703656249517a0f09ac6880d49
Merge pull request #4672 from willkg/1501780-fix-rc

fix bug 1501780: fix the rc case

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 13 • 6 years ago

Marco, Calixte: I think I figured out a better solution. I'll test that out and ping you if I need help.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 14 • 6 years ago

I emailed stability. I think we're good here. Any issues that arise from this can be put in new bugs.