Dustin J. Mitchell [:dustin] (he/him) Assignee
	Description • 9 years ago

This should be a minor config change, but will need to be tested.

	Dustin J. Mitchell [:dustin] (he/him) Assignee
	Comment 1 • 9 years ago

OK, I have this configured on the staging relengapi.  I've tested it out with badpenny tasks (which ordinarily ignore their results).

I also added a `run-celery.sh` that you can run on a celery node (from /data/www/relengapi) to run the celery command line.

Jordan, can you do a bit of stress-testing of the staging site, and we can take a look at the DB tables to see what they look like?

	Dustin J. Mitchell [:dustin] (he/him) Assignee
	Comment 2 • 9 years ago

BTW, all I changed was RELENGAPI_BACKEND="db+mysql://...".

It took me a while to figure out that celery tasks with ignore_results=True wouldn't even create the results table.  I had to hack in a task without that parameter set to get the results table to appear.

	Jordan Lund (:jlund)
	Comment 3 • 9 years ago

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> BTW, all I changed was RELENGAPI_BACKEND="db+mysql://...".
> 
> It took me a while to figure out that celery tasks with ignore_results=True
> wouldn't even create the results table.  I had to hack in a task without
> that parameter set to get the results table to appear.

archiver shouldn't need such a hack so we should get a clear idea of how well this works with some stress tests. I'll write a bash script to fire off a bunch of archiver_client.py calls with various branches/revs

	Jordan Lund (:jlund)
	Comment 4 • 9 years ago

staging results (pun intended) looked good.

however we discovered that the automagic cleanup (deletion) of result 'stored task tombstones' do not happen for database backends unless we:

1) flip on 'celery beat'
2) implement our own 'cleanup' via badpenny job
3) use a different backend like redis

context: http://docs.celeryproject.org/en/latest/configuration.html#std:setting-CELERY_TASK_RESULT_EXPIRES

	Dustin J. Mitchell [:dustin] (he/him) Assignee
	Comment 5 • 9 years ago

I just landed this, in hopes it will decrease the frequency of bug 1185098, where the Flask app is failing to connect to amqp to fetch results from the backend.  The MySQL backend may do a better job of reconnecting.

To revert: set CELERY_BACKEND='amqp' in settings.py.

We still need to set up the cleanup task using one of the approaches jordan gave in comment 4.

	Jordan Lund (:jlund)
	Comment 6 • 9 years ago

\o/ you can see the effect once this landed around 0930 PT

this has drastically reduced the number of errors like the following that we were hitting https://bugzilla.mozilla.org/show_bug.cgi?id=1185098#c0

	Dustin J. Mitchell [:dustin] (he/him) Assignee
	Comment 7 • 9 years ago

351 archiver tasks over the weekend, about 68% of which were try jobs.  And I can confirm there's no cleanup task running!

	Dustin J. Mitchell [:dustin] (he/him) Assignee
	Comment 8 • 9 years ago

I think we can just call the task directly from a badpenny task -- which will give better tracking, while still maintaining Celery's default behavior.  I'm asking to verify here:

https://github.com/celery/celery/pull/2728

	Dustin J. Mitchell [:dustin] (he/him) Assignee
	Comment 9 • 9 years ago

https://github.com/mozilla/build-relengapi/pull/312

	Dustin J. Mitchell [:dustin] (he/him) Assignee
	Comment 10 • 9 years ago

Pull req is in staging now and will soon be in prod.