Closed Bug 1109422 Opened 9 years ago Closed 9 years ago

AppUsage Metric collection enhancements for MonthlyActiveUsers computation

Categories

(Firefox OS Graveyard :: General, defect)

x86
macOS
defect
Not set
normal

Tracking

(blocking-b2g:2.2+, feature-b2g:2.2?, b2g-v2.2 fixed, b2g-master fixed)

RESOLVED FIXED
2.2 S6 (20feb)
blocking-b2g 2.2+
feature-b2g 2.2?
Tracking Status
b2g-v2.2 --- fixed
b2g-master --- fixed

People

(Reporter: rdandu, Assigned: marshall)

References

Details

(Keywords: verifyme)

Attachments

(3 files, 2 obsolete files)

Following enhancements for the App Usage metric collection:
1) Change AppUsageInfo to opt-out. This will offer the User Value of "Display of info" to the user to show the app usage in the country. 
2) Report less frequently (every 2 weeks), but store data per day on device
3) Add deviceinfo.product_model, deviceinfo.hardware, MCC, and MNC to AUM ping
Assignee: nobody → marshall
Summary: AppUsage Metric collection enhancements → AppUsage Metric collection enhancements for MonthlyActiveUsers computation
As part of Appusage collection, following are the related bugs on client, server, and legal:
1) Client: 1109422 AppUsage Metric collection enhancements for MonthlyActiveUsers computation
2) Server: 1109426 Server side Dashboard for FxOS User to view most used Apps in their country
3) Legal, Privacy; 1109429, Request for legal and privacy review of default on of AppUsage info for Monthly Active User computation
Marshall, can you share with us the possible engineering efforts on this bug? We would like to take it into consideration and see if we should add this into 2.2. We expect to see most engineering bugs to be landed before Jan 12th. The feature landing date is still 2/23. 

Thanks.
feature-b2g: --- → 2.2?
Flags: needinfo?(marshall)
See Also: → 1109426
QA Whiteboard: [2.2-feature-qa+]
QA Contact: slyu
(In reply to rdandu from comment #0)
> 2) Report less frequently (every 2 weeks), but store data per day on device

Is it possible to make this any more frequent, say weekly? I'm not sure how this fits in with data usage/payload size considerations, but we'd want to decrease the reporting lag due to the ping frequency as much as possible.

> 3) Add deviceinfo.product_model, deviceinfo.hardware, MCC, and MNC to AUM
> ping

I think we also need deviceinfo.os - doesn't look like this is there yet. But in general, wouldn't it make sense to collect all the same device and network measurements as in the FTU ping?
(In reply to Kevin Hu [:khu] from comment #2)
> Marshall, can you share with us the possible engineering efforts on this
> bug?

These changes should be accomplished within 1 sprint
Flags: needinfo?(marshall)
(In reply to dzeber from comment #3)
> (In reply to rdandu from comment #0)
> > 2) Report less frequently (every 2 weeks), but store data per day on device
> 
> Is it possible to make this any more frequent, say weekly? I'm not sure how
> this fits in with data usage/payload size considerations, but we'd want to
> decrease the reporting lag due to the ping frequency as much as possible.

As long as we retry the ping more frequently (say once every 12 hours) when it fails, I think bi-weekly as the baseline reporting frequency would be OK?

> > 3) Add deviceinfo.product_model, deviceinfo.hardware, MCC, and MNC to AUM
> > ping
> 
> I think we also need deviceinfo.os - doesn't look like this is there yet.
> But in general, wouldn't it make sense to collect all the same device and
> network measurements as in the FTU ping?

+1, I'll make sure they have the same data in this patch
(In reply to Marshall Culpepper [:marshall_law] from comment #4)
> (In reply to Kevin Hu [:khu] from comment #2)
> > Marshall, can you share with us the possible engineering efforts on this
> > bug?
> 
> These changes should be accomplished within 1 sprint

Thanks! Then, we can put this on in 2.2.
feature-b2g: 2.2? → 2.2+
> As long as we retry the ping more frequently (say once every 12 hours) when
> it fails, I think bi-weekly as the baseline reporting frequency would be OK?

I'm concerned about the amount of time until we can generate reports. If the ping only gets sent every two weeks, that means we won't have the total count of how many users we had on Jan 1 until Jan 15 at the earliest, as we have to wait up to two weeks to receive pings that include activity from Jan 1. On top of this, there will be some amount of lag caused by pings not getting sent as soon as they are ready because of network connectivity issues.

I'd like this ping to be sent as frequently as possible within the constraints of data usage/payload size - weekly or more if possible.
Flags: in-moztrap?(slyu)
@Eric: I'll add moztrap cases when the spec is finalized.
(In reply to Marshall Culpepper [:marshall_law] from comment #4)
> (In reply to Kevin Hu [:khu] from comment #2)
> > Marshall, can you share with us the possible engineering efforts on this
> > bug?
> 
> These changes should be accomplished within 1 sprint

Marshall: can you please update this bug with progress and an eta please.

Thanks
Hema
Flags: needinfo?(marshall)
Target Milestone: --- → 2.2 S4 (23jan)
Attached file MAU updates - WIP (obsolete) —
This patch is a WIP and changes the way we collect elapsed time in app usage metrics, and I'd like to get a sanity check before going much further with it..

Instead of keeping absolute start/stop times, we now batch app metrics by day, and keep a running total of elapsed time (which is measured/incremented by using relative performance timers)
Flags: needinfo?(marshall)
Attachment #8546912 - Flags: feedback?(etienne)
Attachment #8546912 - Flags: feedback?(dflanagan)
Comment on attachment 8546912 [details] [review]
MAU updates - WIP

I'm giving the - here because the way you track elapsed time means that batches of data will only be send after 2 weeks of having the phone on, not 2 weeks of total elapsed time.

I don't get why it is worth batching usage data by day, but if that is a real feature request from the metrics people, I guess I can't argue with it.  See github for what I think is a bug in your code related to the by-day stuff.

And I wonder if this feature has been vetted for privacy issues, especially if it is becoming opt-out intead of opt-in.  In particular do we really need the sim mnc value?
Attachment #8546912 - Flags: feedback?(dflanagan) → feedback-
I was confused about mnc. I just read more about it and understand that it is required with mcc. So ignore that last paragraph of the comment above.
Hi Marshall, 

I need your help designing test cases for this feature. Please correct me if my understanding is wrong:

Changes in this patch:
1. Added these fields:
  + 'app.update.channel': 'unknown',
  + 'developer.menu.enabled': false, // If true, data is probably an outlier
  + 'deviceinfo.hardware': 'unknown',
  + 'deviceinfo.os': 'unknown',
  + 'deviceinfo.product_model': 'unknown',
  + 'deviceinfo.software': 'unknown',
  + 'deviceinfo.update_channel': 'unknown'

2. Added simInfo
3. App usage elapsed time is batched by day, and uploaded bi-weekly.

Question: What will happen if I change the upload interval to less than one day (for faster manual testing)?
Flags: needinfo?(marshall)
2.2 was branched. Let me postpone it to later versions of FxOS. If we would like to land it to 2.2, please nominate it again, and also ask for landing approval from the release management team. Thanks.
feature-b2g: 2.2+ → 2.3?
The App Usage metric enhancement needs to get into 2.2, as it is essential to achieve the goal of tracking Monthly Active Users(MAU). Computing the MAU is crucial goal for FxOS, so please approve for 2.2.
blocking-b2g: --- → 2.2+
feature-b2g: 2.3? → 2.2?
Comment on attachment 8546912 [details] [review]
MAU updates - WIP

Flagging Alberto on this to get a fresh pair of eyes.
Attachment #8546912 - Flags: feedback?(etienne) → feedback?(apastor)
Added some comments in github.
(In reply to Shing Lyu [:slyu] from comment #13)
> Hi Marshall, 
> 
> I need your help designing test cases for this feature. Please correct me if
> my understanding is wrong:
> 
> Changes in this patch:
> 1. Added these fields:
>   + 'app.update.channel': 'unknown',
>   + 'developer.menu.enabled': false, // If true, data is probably an outlier
>   + 'deviceinfo.hardware': 'unknown',
>   + 'deviceinfo.os': 'unknown',
>   + 'deviceinfo.product_model': 'unknown',
>   + 'deviceinfo.software': 'unknown',
>   + 'deviceinfo.update_channel': 'unknown'

Yup, except 'app.update.channel' was added accidentally in this patch. It will be removed again in the updated patch going up shortly, and should be the same as 'deviceinfo.update_channel'.

'developer.menu.enabled' is not actually new (it was in the v1 payload as well)

> 2. Added simInfo

Yes, this will contain the same 'network' and 'icc' keys that are in the FTU ping:
https://wiki.mozilla.org/FirefoxOS/Metrics#Example_payload_and_URL

> 3. App usage elapsed time is batched by day, and uploaded bi-weekly.

Yes

> 
> Question: What will happen if I change the upload interval to less than one
> day (for faster manual testing)?

You should just see one batch of metrics for the day the test submitted to your test server (provided you aren't changing the phone's date in the middle, then you might actually see multiple batches)
Flags: needinfo?(marshall)
Attached file MAU updates - v1
I've incorporated some of the feedback from the first patch, updated the unit tests, and removed the relative timing changes so they can be vetted a little more thoroughly in a separate bug.
Attachment #8555340 - Flags: review?(dflanagan)
Attachment #8546912 - Attachment is obsolete: true
Attachment #8546912 - Flags: feedback?(apastor)
Comment on attachment 8555340 [details] [review]
MAU updates - v1

r+, but see my notes on github. I think there is one place you meant to write deviceInfo but wrote deviceinfo.  Also consider organizing the report by day first and then by app. It seems to me that that might be easier to deal with on the server side.
Attachment #8555340 - Flags: review?(dflanagan) → review+
Target Milestone: 2.2 S4 (23jan) → 2.2 S5 (6feb)
Depends on: 1037495
Comment on attachment 8555340 [details] [review]
MAU updates - v1

Hi Fernando,

I need to get additional review on the FTU screen changes that enable opt-out for AppUsageMetrics in 2.2. Here's the link to the specific commit in my pull request:

https://github.com/marshall/gaia/commit/00722269f5d559595fd2f79d9dd70310758af08c
Flags: needinfo?(fernando.campo)
Attachment #8555340 - Flags: review?(fernando.campo)
Francisco/Gregor,

Marshall is trying to wrap this up asap so we could at least have a round of testing before Shing takes off for Chinese new year holidays next week. If Fernando cannot get to this review in the next day, is there anyone else who could also do an additional review for this FTU changes for Marshall? 

Thanks
Hema
Flags: needinfo?(francisco)
Flags: needinfo?(anygregor)
(In reply to Hema Koka [:hema] from comment #23)
> Francisco/Gregor,
> 
> Marshall is trying to wrap this up asap so we could at least have a round of
> testing before Shing takes off for Chinese new year holidays next week. If
> Fernando cannot get to this review in the next day, is there anyone else who
> could also do an additional review for this FTU changes for Marshall? 
> 
> Thanks
> Hema

Sam Foster can help out.
Flags: needinfo?(anygregor)
I see that we are just waiting for Fernando Campo, will ping him offline to see if we can speedup the review.
Flags: needinfo?(francisco)
Comment on attachment 8555340 [details] [review]
MAU updates - v1

Hey guys, sorry for the delay, checkpoint plus couple of flights.

Code for FTU looks nice, just a couple of nits for code style, but completely optional, so good to go.
Flags: needinfo?(fernando.campo)
Attachment #8555340 - Flags: review?(fernando.campo) → review+
master - https://github.com/mozilla-b2g/gaia/commit/a8ab4a083908ed02950c76eea3c2ccb855b3a7d9
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Keywords: checkin-needed
Target Milestone: 2.2 S5 (6feb) → 2.2 S6 (20feb)
Please request Gaia v2.2 approval on this patch when you get a chance.
Flags: needinfo?(marshall)
Attachment #8564468 - Attachment is obsolete: true
Flags: needinfo?(marshall)
Comment on attachment 8566040 [details] [review]
[gaia] marshall:bug1109422_mauChanges_v2.2 > mozilla-b2g:v2.2

Monthly Active Users changes to AppUsageMetrics for v2.2

[Bug caused by] (feature/regressing bug #): Bug 1109422

[User impact] if declined: We will not be able to calculate MAU for FxOS

[Testing completed]: Unit tests, Shing has created also created a set of tests in mozTrap for this feature

[Risk to taking this patch] (and alternatives if risky): This changes the report interval of AppUsageMetrics from daily to every two weeks, which will cause more data to be stored on device. The data as I've measured so far is on order of Kilobytes

[String changes made]: None
Attachment #8566040 - Flags: approval-gaia-v2.2?(khu)
Keywords: verifyme
Attachment #8566040 - Flags: approval-gaia-v2.2?(khu) → approval-gaia-v2.2+
From a user advocacy point of view, data collection should be opt-in by default and not opt-out. In case we decide to go with opt-out by default, it should trigger a privacy review.
Flags: needinfo?(rdandu)
Yes, there is review in progress in the below bugs
1) Main legal/privacy bug https://bugzilla.mozilla.org/show_bug.cgi?id=1109429  
2) Automation Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1134445
Flags: needinfo?(rdandu)
Attached file MozTrap Cases
Flags: in-moztrap?(slyu) → in-moztrap+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: