Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Description • 6 years ago

ftpscraper does a bunch of stuff to populate the product_versions table. We want to get rid of that table and all the related postgres bits, but we need to keep scraping archive.mozilla.org because Buildhub isn't quite there, yet.

The new scraper should:

1. run as a crontabber app

2. scrape archive.mozilla.org

3. save data to a single table that is backed by a Django model with all the information required to do BetaVersionRule lookups


This bug covers that rewrite.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 1 • 6 years ago

I've got parts of this done already. I'll just finish it up.

	[github robot]
	Comment 3 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/27f8813419ca143d91122bff1dfcdcf6452c29b8
bug 1506907: implement Product model

This creates a Django model for products, creates a data migration to move
product data from the old table to the new one, and reimplements things to
use the new table.

We have to leave the old table in place because it's a foreign key of
the old product_versions table and some other tables. All that will go
away in a future commit.

https://github.com/mozilla-services/socorro/commit/78e1197bfc5e361f3ab3b6d961694824d3465e49
Merge pull request #4712 from willkg/1506907-product

bug 1506907: implement Product model

	[github robot]
	Comment 5 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/e72264259b809e66473b5409c1064befe1f5c316
bug 1506907: fix admin regarding Product model

https://github.com/mozilla-services/socorro/commit/eed10000803238535368979690bf4a4b87d20a80
Merge pull request #4713 from willkg/1506907-admin-fix

bug 1506907: fix admin regarding Product model

	[github robot]
	Comment 8 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/7c703b3a27d1993f22d44b00328f38b764005bd0
bug 1506907: add ProductVersion table

This table has all the information for the processor's BetaVersionRule to
convert release_versions into version_strings.

https://github.com/mozilla-services/socorro/commit/e493f70baeeaa6292d8a68d004aaa3871a77d76d
fix bug 1506907: implement archivescraper

This implements ArchiveScraperCronApp which is a slimmed down minimal
version of ftpscraper. The goal is to populate the crashstats_productversion
table with the information the BetaVersionRule needs to do lookups
to convert release_version to version_string for Firefox and Fennec.

https://github.com/mozilla-services/socorro/commit/017945a677681b377325110a97af51f4e0d5fd4c
Merge pull request #4714 from willkg/1506907-archivescraper

fix bug 1506907: implement archivescraper