Wayne Mery (:wsmwk) Reporter
	Description • 8 years ago

I have several whines. The problematic whine is set to run every hour against a query [1] to find bugs created in the past hour that I am not CC on.  (saved query named "new hangs and crashes in last hour")

Not only is the query being run every 15 minutes, but the first trigger isn't on the hour, and it's running *twice* every 15 minutes.

This started happening a few days ago

[1] https://bugzilla.mozilla.org/query.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=1h&chfieldto=Now&field0-0-0=short_desc&field0-0-1=keywords&field1-0-0=cc&field1-0-1=reporter&negate1=1&order=changeddate%2Cbug_status%2Cpriority%2Cassigned_to%2Cbug_id&product=MailNews%20Core&product=Thunderbird&query_format=advanced&query_type=advanced&type0-0-0=substring&type0-0-1=substring&type1-0-0=substring&type1-0-1=substring&value0-0-0=crash&value0-0-1=crash&value1-0-0=vseerror%40lehigh.edu&value1-0-1=vseerror%40lehigh.edu&known_name=crashes-new-notcc-1h

	Wayne Mery (:wsmwk) Reporter
	Comment 1 • 8 years ago

I have other queries that run daily. They don't have this issue.

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 2 • 8 years ago

that's odd. I wonder if it has to do with the UTC conversion somehow. I'm traveling today, but I'll see about debugging this further.

	Wayne Mery (:wsmwk) Reporter
	Comment 3 • 8 years ago

Sunday had nother occurrence. But today for bug 1274970 pushed 1, not 4 notices

	David Lawrence [:dkl]
	Comment 4 • 8 years ago

fubar, do you feel the UTC changes in cron config for whine.pl could be causing duplicates to be sent out intermittently? 
I can also check the DB to see what the timestamps are set to for that particular query.

dkl

	Kendall Libby [:fubar] (he/him)
	Comment 5 • 8 years ago

Weird. That looks like cron got confused and ran two jobs at a time, but the cron log doesn't show anything like that. The whine cron job runs every 15 minutes (*/15) so it wasn't modified for the UTC changes; it WAS disabled and then re-enabled, but that wouldn't cause intermittent issues.

	Wayne Mery (:wsmwk) Reporter
	Comment 6 • 8 years ago

I'm still seeing this

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 7 • 8 years ago

Time in whine.pl is touched by both perl's idea of time and mysql's idea. I feel this is a regression from the UTC conversion but so far I can't find any evidence to support that theory.

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 8 • 8 years ago

Wayne's run_next for the problematic whine is null. it appears to always be null. There are two other users with null run_next's, and I bet they are in the same boat.

SELECT 
    *
FROM
    whine_schedules AS s
        JOIN
    whine_events AS e ON s.eventid = e.id
WHERE
    run_next IS NULL
LIMIT 1000

The only way run_next can be null is if it gets set that null_schedule() in whine.pl.
That is only called by reset_timer() and only if reset_timer() has seen the same schedule more than once per run.

which means it's getting set more than once per run, which we already knew...

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 9 • 8 years ago

I bet this has something to do with it.

perl -MBugzilla -MBugzilla::Util -E 'say format_time("2016-06-07 03:32:00", "%Y-%m-%d %R")'

Running that on production yields:

2016-06-06 20:32

TIME TRAVEL

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 10 • 8 years ago

format_time() relies on the user's timezone unless an optional third argument is passed. 

This should be fixed upstream too.

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 11 • 8 years ago

for reference:

perl -MBugzilla -MBugzilla::Util -E 'say format_time("2016-06-07 03:32:00", "%Y-%m-%d %R", "UTC")'
2016-06-07 03:32

	David Lawrence [:dkl]
	Comment 12 • 8 years ago

Comment on attachment 8760534 [details] [diff] [review]
1274757_1.patch

Review of attachment 8760534 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl

	Dylan Hardison [:dylan] (he/him) Assignee
	Comment 13 • 8 years ago

To ssh://gitolite3@git.mozilla.org/webtools/bmo/bugzilla.git
   5af5f0a..8102e81  master -> master

	Wayne Mery (:wsmwk) Reporter
	Comment 14 • 8 years ago

thanks