Johann Hofmann [:johannh] Assignee
	Description • 6 years ago

We'd like to add telemetry for the new Content Blocking feature. pdol made a wish list here:

https://docs.google.com/document/d/1uUxoAh718myve0-DzVPRstYgMPsMdlcK1n_K05ke7J4/edit

Talking to telemetry people these goals seem pretty reasonable to attain.

	Johann Hofmann [:johannh] Assignee
	Comment 2 • 6 years ago

This probe was always supposed to measure TP enabled status for a session, not a window.

	Johann Hofmann [:johannh] Assignee
	Comment 4 • 6 years ago

This removes the old TRACKING_PROTECTION_EVENTS probe and replaces it with new
Telemetry events that record basic user interaction in the identity popup.

We are now measuring interaction with more elements than before, not just block/unblock.

We're also dropping the old way of measuring updates to onSecurityChange,
since with the recent changes to content blocking it would just record most page loads.

	Johann Hofmann [:johannh] Assignee
	Comment 6 • 6 years ago

Peter, Tanvi (and everyone else), can you please take a look at the Event measurements introduced in part 2 and part 3 and also check that the change we're making in part 1 (recording feature usage at startup instead of window open) is fine with you?

Thanks!

	:Gijs (he/him)
	Comment 8 • 6 years ago

Comment on attachment 9012116 [details]
Bug 1484251 - Part 1 - Record Tracking Protection enabled at startup, not at window open. r=Gijs

:Gijs (out Thu 27 - Sun 30 / 9;  he/him) has approved the revision.

	François Marier [:francois]
	Comment 9 • 6 years ago

Chris, do you still like to review all new Event probes?

	François Marier [:francois]
	Comment 10 • 6 years ago

Comment on attachment 9012124 [details]
request.md

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in Histograms.json, Scalars.yaml and Events.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Presumably pdol.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 2.

5) Is the data collection request for default-on or default-off?

Default ON for some of it, on all channels.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, telemetry alerts are sufficient.

	Peter Dolanjski [:pdol]
	Comment 11 • 6 years ago

(In reply to Johann Hofmann [:johannh] from comment #6)
> Peter, Tanvi (and everyone else), can you please take a look at the Event
> measurements introduced in part 2 and part 3 and also check that the change
> we're making in part 1 (recording feature usage at startup instead of window
> open) is fine with you?
> 
> Thanks!

This looks okay from my perspective.

	Chris H-C :chutten
	Comment 12 • 6 years ago

(In reply to François Marier [:francois] from comment #9)
> Chris, do you still like to review all new Event probes?

The important thing is checking that the volume of these events will be small enough, if they're always enabled. In this case they're tied to user interactions, which makes them likely to be low-volume.

Thanks for the heads-up!

	Chris H-C :chutten
	Comment 13 • 6 years ago

Looping in Sunah for any feedback she might have on taxonomy and event use.

	:Paolo Amadini
	Comment 14 • 6 years ago

In addition to the review comments, if you need to answer questions that require the data to be combined, you may want to write code that does that beforehand and submits also the combined flag, to simplify analysis later.

One example of these questions from the document might be "Of those who have hit Add Blocking, what percentage actually have Tracking Protection set to Always?"

	bugzilla
	Comment 15 • 6 years ago

In terms of volume since this is pre-release only I don't think there's going to be any issues with continual recording, though I think making enable/disable of the event category a pref seems to be emerging as a best practice and would be useful here in case we do end up wanting to turn it off out-of-band.

Regarding taxonomy, if adding dismissal without an action is possible it'd probably make sense to add that as well.

	Johann Hofmann [:johannh] Assignee
	Comment 16 • 6 years ago

(In reply to :Paolo Amadini from comment #14)
> In addition to the review comments, if you need to answer questions that
> require the data to be combined, you may want to write code that does that
> beforehand and submits also the combined flag, to simplify analysis later.
> 
> One example of these questions from the document might be "Of those who have
> hit Add Blocking, what percentage actually have Tracking Protection set to
> Always?"

Yes, talking to chutten it seems like it's a good idea to associate the event data with the current state of content blocking as extra fields. Thanks for raising that point.

(In reply to Sunah Suh (she/her) [:sunahsuh] from comment #15)
> In terms of volume since this is pre-release only I don't think there's
> going to be any issues with continual recording, though I think making
> enable/disable of the event category a pref seems to be emerging as a best
> practice and would be useful here in case we do end up wanting to turn it
> off out-of-band.

I'll add a pref, thanks!
 
> Regarding taxonomy, if adding dismissal without an action is possible it'd
> probably make sense to add that as well.

Sorry, I didn't really understand this, do you mean dismissal of the identity popup without interaction? It's an interesting idea but not possible without recording some intermediate value and this bug has already grown large enough that I'd prefer to defer this to a follow-up.

	Darkspirit
	Comment 17 • 6 years ago

Full TP would be #3 of all addons next week or so. What a success! https://data.firefox.com/dashboard/usage-behavior
Do you already have some Beta 63 numbers about the same growth? Has it been regressed e.g. by bug 1476224 or can it ride into release because growth numbers are the same or better? For the worst case, would you be able to show the Content Blocking panel, but keeping the easy to understand Full TP main menu toggle? (bug 1490388 comment 1) Thanks!

	Pulsebot
	Comment 18 • 6 years ago

Pushed by jhofmann@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8297dbe74765
Part 1 - Record Tracking Protection enabled at startup, not at window open. r=Gijs
https://hg.mozilla.org/integration/autoland/rev/734669014545
Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=Gijs
https://hg.mozilla.org/integration/autoland/rev/8794df889e3f
Part 3 - Use Telemetry events to record interactions in the identity popup. r=Gijs
https://hg.mozilla.org/integration/autoland/rev/29d9b2d2192d
Part 4 - Test updates for new content blocking telemetry. r=Gijs

	Narcis Beleuzu [:NarcisB]
	Comment 19 • 6 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/8297dbe74765
https://hg.mozilla.org/mozilla-central/rev/734669014545
https://hg.mozilla.org/mozilla-central/rev/8794df889e3f
https://hg.mozilla.org/mozilla-central/rev/29d9b2d2192d