Jim Mathies [:jimm] Reporter
	Description • 9 years ago

To assess the stability of a11y + e10s, we need a metric similar to our standard crashstats # of crashes / 100 ADI. Except I'd like to limit this to the pool of users who have a11y turned on.

https://crash-stats.mozilla.com/search/?product=Firefox&version=43.0a1&accessibility=__true__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
Robert - do you have any suggestions here? I can do queries but I'm not sure how we go about normalizing using ADI value.

	Robert Kaiser
	Comment 1 • 9 years ago

(In reply to Jim Mathies [:jimm] from comment #0)
> To assess the stability of a11y + e10s, we need a metric similar to our
> standard crashstats # of crashes / 100 ADI. Except I'd like to limit this to
> the pool of users who have a11y turned on.

As we do not have ADI numbers for those people, we cannot provide such stats.

> Robert - do you have any suggestions here? I can do queries but I'm not sure
> how we go about normalizing using ADI value.

See above. Also, Nightly is so noisy due to people checking in patches all the time that cause crashes that it's really hard to do good assessment of any data there.


That said, e10s is on by default on Nightly, so the amount of people who have it off is probably very small.

	David Bolter [:davidb] (NeedInfo me for attention)
	Comment 2 • 9 years ago

Does the A11Y_INSTANTIATED_FLAG (telemetry) help here?

	Jim Mathies [:jimm] Reporter
	Comment 3 • 9 years ago

(In reply to David Bolter [:davidb] from comment #2)
> Does the A11Y_INSTANTIATED_FLAG (telemetry) help here?

I don't think we can correlate the two since they are different data sets collected via different methods.

	Trevor Saunders (:tbsaunde)
	Comment 4 • 9 years ago

(In reply to Jim Mathies [:jimm] from comment #3)
> (In reply to David Bolter [:davidb] from comment #2)
> > Does the A11Y_INSTANTIATED_FLAG (telemetry) help here?
> 
> I don't think we can correlate the two since they are different data sets
> collected via different methods.

Well, by comparing the percent of the telemetry population using a11y to the percent of crashes where a11y is running you could get some kind of guess how much more a11y crashes.  It wouldn't be perfectly accurate because the telemetry number isn't a perfect sample of the population, but my statistics fu isn't strong enough to be able to quantify that.

	tapper82
	Comment 5 • 8 years ago

Hi how can you test for this wen if you have a11y on e10s is turned off at start-up. If i force e10s on then Nvda can not read anything in a firefox window.

	Jim Mathies [:jimm] Reporter
	Comment 6 • 8 years ago

Hey chris, can we calculate a crash rate comparison linux users who use accessibility?

https://telemetry.mozilla.org/new-pipeline/dist.html#!compare=e10sEnabled&cumulative=0&end_date=2016-05-27&keys=__none__!__none__!__none__&max_channel_version=beta%252F47&measure=A11Y_INSTANTIATED_FLAG&min_channel_version=null&os=Linux&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=0&trim=1&use_submission_date=0

The a11y team wants to figure out if they can safely roll out with e10s to release, and to do that, we need some way of knowing how stable the combination is.

	Chris H-C :chutten
	Comment 7 • 8 years ago

So, to clarify, you would like to measure something in a population:

1) On beta
2) Linux
3) A11y enabled

And then we want to compare the subpopulations based on E10s enabled?

Well, the size of the population of just 1 and 2 (Linux users on Beta) is 83,067 [1]

As you've linked [2] there is at most a 3.4% a11y enabled rate in that population, which translates to 2,833. 

This is beta, so 50% of that population is sacrosanct and we can't ask it for anything. But assuming we can use it as the control population, we're comparing the crashes of one population of about 1400 users to another population of about 1400 users.

I'm not sure at what rate a particular crash that only shows on beta+linux+a11y+e10s would have to occur in order to be visible in that sort of sample size and not just be counted as noise.

[1]: https://sql.telemetry.mozilla.org/queries/465
[2]: http://mzl.la/1RRSWg1

	Jim Mathies [:jimm] Reporter
	Comment 8 • 8 years ago

David, sounds like we don't have enough users to do a good comparison with.

	David Bolter [:davidb] (NeedInfo me for attention)
	Comment 9 • 8 years ago

Wow so few Linux users on beta. I guess we'll watch crash-stats as always and revisit this if needed.