Georg Fritzsche [:gfritzsche] Assignee
	Description • 9 years ago

We decided to go with a 5% sample of opt-out Telemetry usage from release.

Bug 1191912 takes care of the sample, but we still need to flip the pref for making Telemetry default to opt-in again on 41.

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 2 • 9 years ago

Moving part 3 from bug 1191912 here, to track the pref change uplifts in one bug.

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 3 • 9 years ago

Comment on attachment 8645787 [details] [diff] [review]
Make Telemetry opt-in for Firefox 41

Approval Request Comment
[Feature/regressing bug #]:
Unified Telemetry
[User impact if declined]:
We decided to not fully ship Unified Telemetry in 41 due to some unexplained discrepancies.
Instead we make Telemetry opt-in again, but enable it opt-out for 5% of users.
[Describe test coverage new/current, TreeHerder]:
Automation coverage, doing basic manual confirmation.
[Risks and why]:
Low risk at this point - the feature has been tested on the trains for a while now. The biggest remaining concern here is data quality, which doesn't affect clients.
[String/UUID change made/needed]:
None.

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 4 • 9 years ago

Comment on attachment 8652272 [details] [diff] [review]
Enable opt-out Telemetry sampling for Firefox 41

Approval Request Comment
[Feature/regressing bug #]:
Unified Telemetry
[User impact if declined]:
We decided to not fully ship Unified Telemetry in 41 due to some unexplained discrepancies.
Instead we make Telemetry opt-in again, but enable it opt-out for 5% of users.
[Describe test coverage new/current, TreeHerder]:
Automation coverage, doing basic manual confirmation.
[Risks and why]:
Low risk at this point - the feature has been tested on the trains for a while now. The biggest remaining concern here is data quality, which doesn't affect clients.
[String/UUID change made/needed]:
None.

	Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)
	Comment 5 • 9 years ago

Comment on attachment 8645787 [details] [diff] [review]
Make Telemetry opt-in for Firefox 41

UT is Opt in pref by default, seems safe. Beta41+.

	Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)
	Comment 6 • 9 years ago

Comment on attachment 8652272 [details] [diff] [review]
Enable opt-out Telemetry sampling for Firefox 41

This is an optout (will only apply for 5% of release population). Beta41+

	Ryan VanderMeulen [:RyanVM]
	Comment 7 • 9 years ago

https://hg.mozilla.org/releases/mozilla-beta/rev/efc43d513eee
https://hg.mozilla.org/releases/mozilla-beta/rev/5faa1eae6f68

	Andrei Vaida [:avaida]
	Comment 8 • 9 years ago

Tested the sampling feature against live servers on Firefox 41.0b5 (20150827142634) and Firefox 41.0b6 (20150831172306), using Windows 10 x64, Ubuntu 14.04 x64 and Mac OS X 10.10.4.

I've used various pref combinations to create specific conditions and triggered Telemetry and FHR pings to confirm that the expected behavior is met in all cases. In order to simulate client data originating from either the 5% or outside of the 5%, I've set specific clientIDs in state.json, per Georg's instructions.

Apart from two errors I've been seeing in the Browser Console, there were no issues encountered during testing. Please note that the following errors were thrown no matter the test conditions and across the platforms:
-- by Log.jsm:749:0
> 1441020571793   Toolkit.Telemetry   ERROR   TelemetryStorage::loadAbortedSessionPing
> - error removing ping: PingReadError JS Stack trace: 
> PingReadError@TelemetryStorage.jsm:80:15 < 
> TelemetryStorageImpl.loadPingFile<@TelemetryStorage.jsm:1440:13
-- by Log.jsm:749:0
> 1441086970653   Toolkit.Telemetry   ERROR   TelemetryReportingPolicy::get
> dataSubmissionPolicyNotifiedDate - Invalid date stored.

Georg, what's your take on these?

	Georg Fritzsche [:gfritzsche] Assignee
	Comment 9 • 9 years ago

(In reply to Andrei Vaida, QA [:avaida] from comment #8)
> Tested the sampling feature against live servers on Firefox 41.0b5
> (20150827142634) and Firefox 41.0b6 (20150831172306), using Windows 10 x64,
> Ubuntu 14.04 x64 and Mac OS X 10.10.4.
> 
> I've used various pref combinations to create specific conditions and
> triggered Telemetry and FHR pings to confirm that the expected behavior is
> met in all cases. In order to simulate client data originating from either
> the 5% or outside of the 5%, I've set specific clientIDs in state.json, per
> Georg's instructions.
> 
> Apart from two errors I've been seeing in the Browser Console, there were no
> issues encountered during testing.

Great to hear!

> Please note that the following errors
> were thrown no matter the test conditions and across the platforms:
> -- by Log.jsm:749:0
> > 1441020571793   Toolkit.Telemetry   ERROR   TelemetryStorage::loadAbortedSessionPing
> > - error removing ping: PingReadError JS Stack trace: 
> > PingReadError@TelemetryStorage.jsm:80:15 < 
> > TelemetryStorageImpl.loadPingFile<@TelemetryStorage.jsm:1440:13
> -- by Log.jsm:749:0

Known issue, just weird logging, covered by bug 1191825.

> > 1441086970653   Toolkit.Telemetry   ERROR   TelemetryReportingPolicy::get
> > dataSubmissionPolicyNotifiedDate - Invalid date stored.
> 
> Georg, what's your take on these?

Known issue, just weird logging, covered by bug 1191461.

	Andrei Vaida [:avaida]
	Comment 10 • 9 years ago

Thanks for clarifying those, Georg! This bug is verified fixed on Firefox 41, per Comment 8 and Comment 9.

	Ada [:adalucinet]
	Comment 11 • 9 years ago

Confirming this fix with 41.0.2 build 2 (20151014143721) too, across platforms [1].

[1] Windows 7 64-bit, Mac OS X 10.10.4 and Ubuntu 14.04 32-bit