Open Bug 1385832 Opened 7 years ago Updated 2 months ago

getBytesInUse() is not supported in WebExtentions API

Categories

(WebExtensions :: Compatibility, defect, P3)

54 Branch
defect

Tracking

(Not tracked)

UNCONFIRMED

People

(Reporter: u580826, Unassigned)

Details

(Whiteboard: [storage])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
Build ID: 20170628075643

Steps to reproduce:

I used uBlock Origin 1.13.9.0 (dev channel) and noticed that "Storage used" in "Dashboard" of uBlock Origin shows "? bytes". I opened an issue and gorhill (author of the addon) explained that it's because Firefox doesn't support browser.storage.local.getBytesInUse(). The issue's url is here: https://github.com/gorhill/uBlock/issues/2812


Actual results:

uBlock Origin 1.13.9.0 (dev channel, webext-hybrid) doesn't show storage usage. Version 1.13.8 (stable channel, legacy addon) shows correctly.


Expected results:

Both addon of webext-hybrid and of legacy shows correct value. I would appreciate if you could implement the API.
Component: Untriaged → WebExtensions: Compatibility
Product: Firefox → Toolkit
Priority: -- → P3
Whiteboard: [storage]
In case this matters, uBlock Origin only uses the getBytesInUse(null, ...) version of the API, it is just interested in reporting the storage used as a whole.
Product: Toolkit → WebExtensions
The documentation is confusing on this topic. Looking at https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/local gives one the impression that getBytesInUse() is supported since it's listed as one of the methods and the browser compatibility table below shows Firefox support since v45. However, if you click on this method and go to https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/StorageArea/getBytesInUse you will see that Firefox doesn't support this method. I wish the Mozilla team would give a clear indication that this method is not supported from the parent page (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/local).

Any plans to add support for this method? It's preventing me from porting over my Chrome extension also.
For now you can accomplish the same thing by doing this:

  browser.storage.local.get(function(items) {
    console.log(JSON.stringify(items).length);
  });

I checked in firefox 65.0.1. browser.storage.local.getBytesInUse is still undefined.

Since the implementation is so trivial, maybe it's worth marking this as Good First Bug?

One question though (for your Tom), your implementation seems to be giving different numbers in Chrome:

browser.storage.sync.get().then(function(items) { console.log(JSON.stringify(items).length); });
2915
await browser.storage.sync.getBytesInUse()
2961

Any ideas why?

For the record, as part of bug 1634615 etc, browser.storage.sync does have this API. I'm not going to close this because IIUC, browser.storage.local still does not.

See also bug 1637166 comment #7 and https://wiki.developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/StorageArea/getBytesInUse

getBytesInUse
Firefox
Full support 78
Notes
Only supported by the sync storage area.

Can confirm that this is still an issue. I'm running into this for my extension Stock Inspector, which caches data to reduce user's network requests and data usage. I'd love to use this feature to show users the amount of cache storage used like ublock.

@Ray I've been studying this deeply and have a working formula now :), that you can use as workaround:

// Docs: "The maximum amount (in bytes) of data that can be stored in local storage, as measured by the JSON stringification of every value plus every key's length."
Object.entries(await browser.storage.sync.get()).map(([key, value]) => key.length + JSON.stringify(value).length).reduce((acc, x) => acc + x, 0)
// will give you same result as:
await browser.storage.sync.getBytesInUse()

But it will work also on "local" storage where the docs says the same thing.

Know that this will work correctly ONLY in Firefox. Chromium has bug where "<" symbol is stored escaped and takes more space out of your quota than it should:
https://bugs.chromium.org/p/chromium/issues/detail?id=1115239

(In reply to juraj.masiar from comment #9)

Object.entries(await browser.storage.sync.get()).map(([key, value]) => key.length + JSON.stringify(value).length).reduce((acc, x) => acc + x, 0)

As Javascript stores strings as UTF-16, and string.length property returns the number UTF-16 code units[1], wouldn't you have to multiply the value by 2 to get the number of bytes used?

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length#Description

(In reply to juraj.masiar from comment #9)

// Docs: "The maximum amount (in bytes) of data that can be stored in local storage, as measured by the JSON stringification of every value plus every key's length."

juraj, can you tell me where you found that quote? I do not find it in the docs I am viewing[1]

[1]https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/StorageArea/getBytesInUse

Does StorageArea.getBytesInUse use backend computation which is more efficient than the polyfill suggested by juraj.masiar? Or is it just a convenience method that is actually doing the same thing?

Flags: needinfo?(markh)

(In reply to JulianHofstadter from comment #10)

(In reply to juraj.masiar from comment #9)

Object.entries(await browser.storage.sync.get()).map(([key, value]) => key.length + JSON.stringify(value).length).reduce((acc, x) => acc + x, 0)

As Javascript stores strings as UTF-16, and string.length property returns the number UTF-16 code units[1], wouldn't you have to multiply the value by 2 to get the number of bytes used?

The data is stored to disk as JSON.

(In reply to JulianHofstadter from comment #11)

juraj, can you tell me where you found that quote? I do not find it in the docs I am viewing[1]

[1]https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/StorageArea/getBytesInUse

See the quota descriptions at https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync.

(In reply to JulianHofstadter from comment #12)

Does StorageArea.getBytesInUse use backend computation which is more efficient than the polyfill suggested by juraj.masiar? Or is it just a convenience method that is actually doing the same thing?

That's exactly what it does, because that's what chrome's spec says to do. We can't look at the bytes stored in the database column because that wouldn't conform to the spec. The implementation is at https://searchfox.org/mozilla-central/source/third_party/rust/webext-storage/src/api.rs#333

Flags: needinfo?(markh)

(In reply to Mark Hammond [:markh] [:mhammond] from comment #13)

The data is stored to disk as JSON.

As the JSON data can include strings, and in some cases the value passed to storage is only a string, I'm not clear how you would always end up with 1 byte per code unit, but I'll just have to take your word for it that taking the length property of the JSON.stringified object will give the total number of bytes.

Sorry, I missed the point - I think you are correct that the polyfill doesn't take this into account - while utf-8 is stored, that can be > 1 byte. So it probably should be utf-8 encoding after stringifying.

(In reply to Mark Hammond [:markh] [:mhammond] from comment #15)

while utf-8 is stored, that can be > 1 byte. So it probably should be utf-8 encoding after stringifying.

If I understand correctly, JS only deals in UTF-16 as far as strings go, and will convert UTF-8 to UTF-16. So it seems to me the strings you would be dealing with even after stringifying would be UTF-16, as far as length property goes.

Again, https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length#Description

A non-ascii character in a JS string will have a length of 1. If you utf-8 encode that you will get an array of (typically) 2 - which is how many bytes the character consumes on disk. JS uses utf-16 in memory, but that's not relevant here.

"\u00DF".length
1
(new TextEncoder("utf8")).encode("\u00DF").length
2

(In reply to Mark Hammond [:markh] [:mhammond] from comment #17)

A non-ascii character in a JS string will have a length of 1. If you utf-8 encode that you will get an array of (typically) 2 - which is how many bytes the character consumes on disk. JS uses utf-16 in memory, but that's not relevant here.

"\u00DF".length
1
(new TextEncoder("utf8")).encode("\u00DF").length
2

I see. So then back to the polyfill, in order to be accurate, would you need to do something like:

(new TextEncoder("utf8")).encode(key + JSON.stringify(value)).length

yes, that looks correct.

Thanks @JulianHofstadter for fixing my bug! I guess my "deep" study wasn't deep enough.
So something like this should yield correct results:

new TextEncoder().encode(
  Object.entries(await browser.storage.sync.get())
    .map(([key, value]) => key + JSON.stringify(value))
    .join('')
).length

Note that TextEncoder doesn't take encoding parameter anymore and support only UTF-8:
https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder

Severity: normal → S3

7 years later and browser.storage.local.getBytesInUse() is still not implemented. Come on devs... pretty please?

You need to log in before you can comment on or make changes to this bug.