+++ This bug was initially created as a clone of Bug #1649503 +++

Creating a new bug for this.

bug 1649503 comment 24

It looks like this bug isn't really fixed at least as of 84.0b7. I've run into this issue and trying to look at the most recent looked at project in Gitlab for example only shows "This feature requires browser localStorage support".

bug 1649503 comment 26

Same here, now that Firefox 84 has been released it broke our application that relies on storing ~ 1.6 MB of data in local storage. I managed to make the issue reproducible on a new profile using Firefox 84.0 using the following script running on a localhost server:
[...] read the linked comment

I think what is seen now is a different issue (or even multiple different issues?), which, if, I understand correctly, wasn't happening before ff 84, so it might be a regression from something that landed after ff 83.

https://bugzilla.mozilla.org/show_bug.cgi?id=1649503#c26 has details on one such issue.

Jan, do you have an idea what might have regressed this?

I seem to struggle with the same problem in firefox 84 for some domains the I get random ERROR DOMException: The quota has been exceeded. which brakes the site. These are different application hosted under the same top level domain from which I understand might be using shared local storage?

Using private modus of firefox or using chrome I do not encounter the same problem.

(In reply to Severin Wünsch from comment #2)

I seem to struggle with the same problem in firefox 84 for some domains the I get random ERROR DOMException: The quota has been exceeded. which brakes the site. These are different application hosted under the same top level domain from which I understand might be using shared local storage?

Using private modus of firefox or using chrome I do not encounter the same problem.

I don't believe this has been mentioned yet but for anyone affected by this, the workaround is simply to increase dom.storage.default_quota in about:config to a value high enough not to trigger this bug (the default value is 5 MiB).

(In reply to Nathan Monfils from comment #3)

I don't believe this has been mentioned yet but for anyone affected by this, the workaround is simply to increase dom.storage.default_quota in about:config to a value high enough not to trigger this bug (the default value is 5 MiB).

This indeed fixed the problem for me but I hope there is a solution for other users which does not require to change a firefox setting in about:config

(In reply to Severin Wünsch from comment #4)

(In reply to Nathan Monfils from comment #3)

I don't believe this has been mentioned yet but for anyone affected by this, the workaround is simply to increase dom.storage.default_quota in about:config to a value high enough not to trigger this bug (the default value is 5 MiB).

This indeed fixed the problem for me but I hope there is a solution for other users which does not require to change a firefox setting in about:config

Same here. If I increase from 5MiB to 10MiB is solves the issue but hard to tell your users to do that.

I cleared all cookies & site data, logged directly into my app I was still having the issue in all subdomains (even on the first one). Managed Cookies and Site data displays only 64KB across 4 subdomains.

On incognito mode I don't have this issue at all.

I came across this bug report while trying to troubleshoot issues in our web application. In my current Firefox profile, the app is storing around 6602 characters of data (at least that's the char count after converting the domain's whole localstorage to json and checking -- do you count key length as well? I did here...)

I'm not sure what is meant by "the default value" of dom.storage.default_quota being 5 MiB, but I am pretty sure that 6602 individual chars (even multibyte chars) is nowhere near 5 mebibytes.

Can someone clarify exactly what limitation my 6602 characters of local storage is running up against when I see this error message?

(In reply to jamie from comment #6)

Can someone clarify exactly what limitation my 6602 characters of local storage is running up against when I see this error message?

The bug seems to be twofold:

1. Everything from your public domain (abc.tld) counts against your quota, even if it is in a seemingly unrelated subdomain (e.g. my-app.example.com and intranet.example.com).
2. The quota is not recomputed properly, requiring a firefox restart after clearing your data on other subdomains of your public domain.

I troubleshot it here.

You can find what else is using up quota on your public domain by running a SQLite command against your local storage database as I did (with firefox closed, you will also likely need to add a WHERE clause to narrow down your search unless you are running on an empty profile):

~ $ sqlite3 ~/.mozilla/firefox/h14dzk29.test/webappsstore.sqlite 'SELECT key, originKey, length(value) AS v FROM webappsstore2 ORDER BY v DESC ;' -tabs

(In reply to Nathan Monfils from comment #7)

(In reply to jamie from comment #6)

Can someone clarify exactly what limitation my 6602 characters of local storage is running up against when I see this error message?

The bug seems to be twofold:

1. Everything from your public domain (abc.tld) counts against your quota, even if it is in a seemingly unrelated subdomain (e.g. my-app.example.com and intranet.example.com).
2. The quota is not recomputed properly, requiring a firefox restart after clearing your data on other subdomains of your public domain.

I troubleshot it here.

You can find what else is using up quota on your public domain by running a SQLite command against your local storage database as I did (with firefox closed, you will also likely need to add a WHERE clause to narrow down your search unless you are running on an empty profile):

~ $ sqlite3 ~/.mozilla/firefox/h14dzk29.test/webappsstore.sqlite 'SELECT key, originKey, length(value) AS v FROM webappsstore2 ORDER BY v DESC ;' -tabs

Thanks for that clarification and the tips. I hope they address the bug soon, because we may be facing huge numbers of client reports, and we don't have time to factor the localstorage use out of our code. I'm really hoping the cross-domain limitation is the main driver of the problem (test servers, staging, etc.), because that will lessen the likelihood our users will experience the problem.

(In reply to Nathan Monfils from comment #3)

I don't believe this has been mentioned yet but for anyone affected by this, the workaround is simply to increase dom.storage.default_quota in about:config to a value high enough not to trigger this bug (the default value is 5 MiB).

Was searching for a solution for this issue. This did the trick, thank you.

I gave some hints to Tom and he is going to take a closer look at this.

(In reply to Nathan Monfils from comment #7)

The bug seems to be twofold:

1. Everything from your public domain (abc.tld) counts against your quota, even if it is in a seemingly unrelated subdomain (e.g. my-app.example.com and intranet.example.com).
2. The quota is not recomputed properly, requiring a firefox restart after clearing your data on other subdomains of your public domain.

Thanks for that clarification. I noticed the problem few days ago, after FireFox auto-upgrade to 84 version.
Increasing 'dom.storage.default_quota' in 'about:config' solves the problem.
But we are using up to 1.5 KB strings to store in local storage, so I was wondering where 5 MB of storage has gone.

So I checked FireFox local site storage settings (https://support.mozilla.org/en-US/kb/storage?as=u&utm_source=inproduct&redirectslug=permission-store-data&redirectlocale=en-US) and found such lines:

www.our-domain.com: cookies: 17, Storage: 5.8 MB
my-app-server.our-domain.com: cookies: 0, Storage: 64.0 KB

After increasing 'dom.storage.default_quota' to value above 5800, our application https://my-app-server.our-domain.com starts working.
After decreasing 'dom.storage.default_quota' to value below 5800, our application https://my-app-server.our-domain.com stops working.

When I cleared storage for 'www.our-domain.com' and started FireFox (see "2. The quota is not recomputed properly, requiring a FireFox restart"), our application https://my-app-server.our-domain.com started working again.
This confirms that quota for local storage is counted for all 'xxx.our-domain.com' domains.

We had no such problems with FireFox before, and still not having with Chrome.

This test fails because two reasons.

• Group usage and limit check is run on the content process (and usage is got
  asynchronously).
• Add/Update operations are first collected, coalesced and then flushed after a
  short time on the parent process. (So that usage got from the parent process
  is possible to be old value)

(In reply to Nathan Monfils from comment #26)
Thanks for the script! It helps me to figure out a test quickly.

localStorage.removeItem('s');
let a = '';
for (let i = 0; i < 4000000; ++i)
    a = a + ' ';
localStorage.setItem('a', a);

1. Open two tabs on localhost and a.localhost: both of those scripts execute properly.

Background:

• Limit for a site/group and an origin: 5 * 1024 * 1024 bytes and this test writes 1+4000000 bytes in localStorage.setItem('a', a);

It's because ,says the tab for localhost is opened first, the changes made localhost haven't been updated to the parent process yet. So that when the tab for a.localhost is opened, it's notified that the group usage for localhost is still 0.

I guess if you open a.localhost a short time later, the second tab should receive QuotaExceededError.

2. Exit firefox completely and re-open the profile: the scripts still work fine

Because the old value is the same as the new value so that the check is skipped.
Note that localStorage.removeItem('s'); doesn't remove the value for key 'a'.

3. In the JS console of localhost, run localStorage.removeItem('a') (or localStorage.setItem('a', '')) and refresh the page: the script now fails due to Uncaught DOMException: The quota has been exceeded..

Note the localStorage.removeItem('a') (or localStorage.setItem('a', '') succeeds. It fails when it executes localStorage.setItem('a', a);.

Clearing local storage on localhost and a.localhost with localStorage.removeItem does not fix the issue, even though the localStorage is now completely empty (after closing firefox, sqlite3 ~/.mozilla/firefox/h14dzk29.test/webappsstore.sqlite 'SELECT key, originKey, length(value) AS v FROM webappsstore2 ORDER BY v DESC ;' -tabs is empty). Restarting firefox fixes the issue.

It's probably because the usage for the same site/group but the different origin is not updated to other origins. More specifically, after clearing, the tab for localhost thinks we still can 1+4000000 bytes usage for a.localhost and so does the other way around. So that the script fails to run on both tabs.

Restart forces them to get the updated usage so that the script works successfully again.

Overall, I think the issue is:

• The usage and check stay on the content process
  • The usage is got asynchronously and is updated at here (from my observation, it might be updated after operations (e.g. setItem) )
• It seems the changes for the usage for a group have never been updated to other processes.

The options in my mind at the moment are:

1. Move the check for usage to the parent process
2. Preload the group usage (see code) so that we get updated usage on each preload.
3. (a) Change AsyncGetUsage to SyncGetUsage to ensure usage is update-to-date before setItem
   (b) Ensure add/update operations execute immediately so that the sooner we can update the delta to other processes. (see here)
   (c) Update the delta for mUsage to the processes that loading the same group.
4. Remove the group limit checks for the old LocalStorage impl.

(1) is the real fix but it requires non-small changes and potentially performance impacts. I am not sure if we should do that given we are on the way to skip LSNG.
(2) only mitigate this issue. The timing thing (races between processes) can still be a problem. We only have small performance impacts on this one since we only need to query usage additionally (or I guess, we can even have cached usages on the parent process to make this faster).
(3) only mitigate this issue. The timing thing (races between processes) can still be a problem. And we still can have performance impacts (I guess this is less than (1)) since we change to SyncGetUsage and execute add/update immediately.
(4) is a workaround, but it causes us to expose the fill disk attack again since we remove the group limit.

I lean to option (2). Jan, what do you think?

Tom, can you please tell me what's the exact definition for "origin" in the context of local/session storage? The DevTools' Storage Inspector obviously refers to full domain of the current page, i.e. it distincts between localhost, a.localhost, and b.localhost.
Though from all I get here, the local and session storages are actually shared between subdomains.

I'll use your answer to create a bug report for the Storage Inspector to correct its display to show all data related to an origin. That will allow developers to see all saved data and correctly calculate the used quota (and at some point display the usage in the Storage Inspector, see bug 1349530).

Sebastian

Actually, shouldn't the APIs for getting items (i.e. window.localStorage.getItem() and window.sessionStorage.getItem()) follow the same definition for "origin" as the logic behind calculating the quota when setting them (i.e. window.localStorage.setItem() and window.sessionStorage.getItem())?

Sebastian

The old LS implementation has 5MB limits both for origin and eTLD+1 domain.
The 5MB limit for eTLD+1 domain was added to address http://www.filldisk.com/
The new LS implementation (LSNG) doesn't have the 5MB limit for eTLD+1 domain since the storage used by LSNG is able to evict the least recently used origins and it also has a 2GB limit for eTDL+1 domain.

(In reply to Sebastian Zartner [:sebo] from comment #14)

Tom, can you please tell me what's the exact definition for "origin" in the context of local/session storage? The DevTools' Storage Inspector obviously refers to full domain of the current page, i.e. it distincts between localhost, a.localhost, and b.localhost.

There are "origin" and "group".

Origin is composed by scheme, host, and port from URL
Definition for origin: https://html.spec.whatwg.org/multipage/origin.html#origin
MDN: https://developer.mozilla.org/en-US/docs/Glossary/origin

Group/Site is eTLD+1 domain.

For instance, a.example.com and example.com are two different origins but they are under the same group (example.com).

I think the behavior of devtools is correct. It should differentiate between localhost, a.localhost, and b.localhost since they are different origins.

Though from all I get here, the local and session storages are actually shared between subdomains.

SessionStorage isn't bound by quota limitation with an eTLD+1 domain, IIUC. It only has quota limitation for an origin.
LocalStorage is not shared between subdomains, but its usage and quota are shared between eTLD+1 domains.
For instance, the eTLD+1 domain is example.com and we have a.example.com and b.example.com.
After a.example.com runs localStorage.setItem(a,.), b.example.com should still get undefined from localStorage.getItem('a').

However, says, we have 5 MB limits for each eTLD+1 domain.
Before a.example.com runs localStorage.setItem(a,.), b.example.com has full 5 MB quota.
After that, b.example.com` has 5 MB - 2 btyes quota.

I'll use your answer to create a bug report for the Storage Inspector to correct its display to show all data related to an origin. That will allow developers to see all saved data and correctly calculate the used quota (and at some point display the usage in the Storage Inspector, see bug 1349530).

Thanks, and please let me know if you need helps from the Worker and Storage team. :)

(In reply to Sebastian Zartner [:sebo] from comment #15)

Actually, shouldn't the APIs for getting items (i.e. window.localStorage.getItem() and window.sessionStorage.getItem()) follow the same definition for "origin" as the logic behind calculating the quota when setting them (i.e. window.localStorage.setItem() and window.sessionStorage.getItem())?

That's a good point. At the moment, SessionStorage only has quota checks for origins and LocalStorage(old impl) has quota checks for origins and groups.

As Jan mentioned in comment #16, the group limit check is used to address the fill disk attacks. It's probably the reason why we don't have that for SessionStorage.

Thank you for the detailed info and the clarification of origin and eTLD+1! I already assumed that but wasn't sure. I've filed the bug 1685495 for the Storage Inspector improvement.

With the old implementation the issue is that developers get the quota exceeded exception even when the local storage of their website seemingly doesn't exceed it because the group quota calculation isn't exposed. Though it sounds that with LSNG the quota limitation to eTLD+1 domains generally won't be an issue anymore.

One last question, will the quota limit per origin be removed completely and only this 2 GB limit for eTLD+1 be applied or will the 5 MB limit for origins stay?

Sebastian

(In reply to Sebastian Zartner [:sebo] from comment #18)

One last question, will the quota limit per origin be removed completely and only this 2 GB limit for eTLD+1 be applied or will the 5 MB limit for origins stay?

The latter, localStorage is a synchronous API which is quite bad for performance in general. Since we can't drop support for localStorage, we have to mitigate the issues by preloading all the data into memory when a page is being loaded. If we allowed more than 5MB per origin, it would lead to memory footprint issues.

2GB for all *.domain.tld, and 5MB per *.domain.tld seems reasonable.

Do you have any ETA when this will be released?
We have an app that uses this functionality more apparently to users, and the latest FF breaks the app because of this defect. We're holding off deploying the app for now, if the timing is not in the too far future for now.

Decision on team meeting call is to raise the group limit for legacy localstorage to 25MB (multiply by 5) per Jan, Tom, and myself. This may entail splitting the pref and the constant.

Would you mind adding the reasoning on how you decided 5 times is a sufficient value? Trying to understand as 25 MB seems enough but I'd like to better understand to assess if this would work on our rather large intranet (with external websites on the same domain name). Thanks for all your work!

Of course! Just wanted to make sure to get our action items written down in the bug. The rationale is:

• There's a multi-process synchronization issue with group quota in legacy LocalStorage (LS) as expressed by the commit message for comment 11.
• We want to mitigate the impact of this synchronization issue with the recognition that the long term fix is LocalStorage NextGen (LSNG) which relaxes the group/domain constraints to the normal QuotaManager constraints.
• Accordingly, we're increasing the group limit for legacy LS as a short-to-medium-term mitigation until LSNG is used on (most if not all) Firefox release installs.
• The specific multiple of 5 is an arbitrary value driven by these factors:
  • Choosing a value closer to 1 is likely too limited a response.
  • Legacy LS data is stored in a single database which lacks any concept of storage pressure leading to the clearing of old/unused data or otherwise being aware of available disk space. Relaxing group limits for legacy LS all the way to 2GB could create a lot of new problems related to this which would potentially break Legacy LS across all origins, not just specific quota-exceeding ones.
  • This isn't intended to be a policy change, just a band-aid. The LSNG change is the policy change which is backed up by the necessary logic for data clearing on storage pressure and other heuristics.

Thanks!

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #21)

Decision on team meeting call is to raise the group limit for legacy localstorage to 25MB (multiply by 5) per Jan, Tom, and myself. This may entail splitting the pref and the constant.

We made another decision on the team today since we found that this issue is revealed by bug 1676410 and thus we decided to revert the changes back to the previous semi-broken behavior as short-term mitigation.

This patch:

• adds a new pref for site quota.
• sets 25 MB as the default site qutoa.
• rename LocalStorageManager::GetQuota() to LocalStorageManager::GetOriginQuota().
• adds LocalStorageManager::GetSiteQuota().
• updates LocalStorage quota tests.

A short update for this issue so far:

Background: Some users got QuotaExceededError since FF 84.
Reason: Legacy LocalStorage has a limit per site/group/domain and they reach that limit.

Why they didn't get QuotaExceededError before FF84?
It's mainly because:

• Usage tracking mechanism for legacy LocalStorage is not properly synced within processes.
• Incorrect site usage is got before bug 1676410.
  Bug 1676410 fixes the issue that legacy LocalStorage gets wrong site usage in some situations and thus reveals this issue.

Since we are on the way to ship LocalStorage NextGen (LSNG) and we want to provide a band-aid to release as soon as possible, we decided to:

• Revert changes in Bug 1676410.
• Change the default limit per site to 25 MB (was 5 MB which is the same value as the limit per origin)

Note that these only hide and mitigate the issue.

Backed out for xpcshell failures on test_archive.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/b8e452973bf4e5ecc227ca3d0338b85122a10501
Log link: https://treeherder.mozilla.org/logviewer?job_id=326713002&repo=autoland&lineNumber=6931

fully try push with two patches and disable LSNG: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8cd98722b098a0cabcdf6cff8633dff1fd7c6780

I(In reply to Narcis Beleuzu [:NarcisB] from comment #31)

Backed out for xpcshell failures on test_archive.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/b8e452973bf4e5ecc227ca3d0338b85122a10501
Log link: https://treeherder.mozilla.org/logviewer?job_id=326713002&repo=autoland&lineNumber=6931

test_archive.js enables LSNG whcih doesn't seem to be affected by the blackout patch.

I am somehow confused by the sequence. From this push, (Thu, Jan 14, 16:43:01), test_archive.js starts to fail.

The backout push mentioned by comment 31 was at here, (Thu, Jan 14, 18:41:13).

Isn't Bug 1685098 or bug 1683826 more likely to be the cause for the failure?

:ttung , yes. Seems to be caused by one of those 2 bugs.
Sorry for the confusion, I`ll re-land it.

This is hitting quite a few people, it seems.

https://hg.mozilla.org/mozilla-central/rev/a4b5ff1a57a1
https://hg.mozilla.org/mozilla-central/rev/a143bb08b3a1

Legacy LocalStorage checks if origins are in the same site by checking suffix
so that https://test2.example.com was expected to share site quota with
http://example.com and http://test1.example.com in the test.
However, in Fission, process isolation is based on SiteOrigin and that means
https://test2.example.com runs in one process and the other two run in another
process.

This patch only updates the test to ensure testing origins run in the same
process when Fission enables. The usage synchronization is still an issue and
that means Fission needs LSNG to be enabled.

So the patches D101726 and D101756 are supposed to fix this issue in the current release configuration (LSNG and Fission off). The remaining issue mentioned in comment 38 refers only to the (currently unsupported/untested) configuration having LSNG switched off but Fission on and can be safely ignored.

This needs a Beta uplift request ASAP if comment 36 is meant to say that we should ship this fix in 85. We're down to only the RC build at this point.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #40)

This needs a Beta uplift request ASAP if comment 36 is meant to say that we should ship this fix in 85. We're down to only the RC build at this point.

Yes, I'm just verifying the patches apply cleanly and running some basic tests.

Comment on attachment 9197040 [details]
Bug 1683299 - Restore semi-broken behaviour before bug 1676410 and bug 1676973; r=#dom-workers-and-storage-reviewers

Beta/Release Uplift Approval Request

• User impact if declined: Some users would still see quota exceeded errors leading to unusable localStorage for given origin(s) during entire FF session (the problem disappears after a restart).
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The first patch is rather a backout of bug 1676410 which restores a state that was used for many years, so it should be pretty safe.
  The other patch splits the 5MB limit we had for both the origin and the domain into 5MB for origin and 25MB for domain. Allowing a bit more data to be stored should be quite safe from the point of view of causing new regressions. The split of the limits can be easily undone by setting a pref.
• String changes made/needed: None

Comment on attachment 9197040 [details]
Bug 1683299 - Restore semi-broken behaviour before bug 1676410 and bug 1676973; r=#dom-workers-and-storage-reviewers

Approved for 85.0rc1.

https://hg.mozilla.org/releases/mozilla-beta/rev/716b5f48799e
https://hg.mozilla.org/releases/mozilla-beta/rev/854247b3a4eb

https://hg.mozilla.org/mozilla-central/rev/d557cd3ae0e0

Was this included in v85 that was fully released today (I only see mentions of it in being 85rc1) or aiming for 86? This is blocking 20k users of ours.

Yes, RC builds are what we intend to ship to end users. This fix should be present in the final 85 release per comments 44 & 45.