Will Kahn-Greene [:willkg] ET needinfo? me Reporter
	Description • 6 years ago

The processor is a FetchTransformSaveWithSeparateNewCrashSourceApp Socorro app which has a destination for saving processed crashes. That destination is entirely defined in configuration like this:

destination.storage0.benchmark_tag=PGBenchmarkWrite
destination.storage0.crashstorage_class=socorro.external.statsd.statsd_base.StatsdBenchmarkingWrapper
destination.storage0.statsd_prefix=processor.postgres
destination.storage0.transaction_executor_class=socorro.database.transaction_executor.TransactionExecutorWithInfiniteBackoff
destination.storage0.wrapped_crashstore=socorro.external.postgresql.crashstorage.PostgreSQLCrashStorage
destination.storage0.wrapped_object_class=socorro.external.postgresql.crashstorage.PostgreSQLCrashStorage
destination.storage1.active_list=save_raw_and_processed
destination.storage1.benchmark_tag=BotoBenchmarkWrite
destination.storage1.crashstorage_class=socorro.external.statsd.statsd_base.StatsdBenchmarkingWrapper
destination.storage1.statsd_prefix=processor.s3
destination.storage1.use_mapping_file=False
destination.storage1.wrapped_crashstore=socorro.external.boto.crashstorage.BotoS3CrashStorage
destination.storage1.wrapped_object_class=socorro.external.boto.crashstorage.BotoS3CrashStorage
destination.storage2.active_list=save_raw_and_processed
destination.storage2.benchmark_tag=BotoBenchmarkWrite
destination.storage2.crashstorage_class=socorro.external.statsd.statsd_base.StatsdBenchmarkingWrapper
destination.storage2.es_redactor.forbidden_keys=memory_report, upload_file_minidump_browser.json_dump, upload_file_minidump_flash1.json_dump, upload_file_minidump_flash2.json_dump
destination.storage2.statsd_prefix=processor.es
destination.storage2.use_mapping_file=False
destination.storage2.wrapped_crashstore=socorro.external.boto.crashstorage.BotoS3CrashStorage
destination.storage2.wrapped_object_class=socorro.external.es.crashstorage.ESCrashStorageRedactedJsonDump
...

The way that's configured is really tough. For example, it's hard to accurately and confidently change the position of two crash storage destinations. For example, it's hard to drop a crash storage destination in the middle of the list. It's hard to clean up crash storage configuration after changes like that time we changed the StatsdBenchmarkingWrapper configuration (wrapped_crashtore vs. wrapped_object_class which appears to still be in configuration).

This bug covers moving destination specification to code so that it's in a Python data structure in a Python module and versioned with everything else much like we've done with other things like the signature generation pipeline, processor rule sets, and other things.

	[github robot]
	Comment 2 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/3ac947b90f42f0587f92b715a5414476f772d6c3
Fix bug 1430860: Move destination config to python-based source.

This also replaces the storage_classes config with storage_namespaces to make
reading and changing the config much easier and less error-prone.

https://github.com/mozilla-services/socorro/commit/e115765dae7898070d1f77e6d23313c4a16834df
Merge pull request #4307 from Osmose/python-based-settings

Fix bug 1430860: Move destination config to python-based source.