Closed Bug 1269131 Opened 8 years ago Closed 3 years ago

sqlite takes a uses a lot of memory on firefox for android

Categories

(Firefox for Android Graveyard :: General, defect)

defect
Not set
normal

Tracking

(fennec+)

RESOLVED INCOMPLETE
Tracking Status
fennec + ---

People

(Reporter: bkelly, Unassigned)

Details

(Whiteboard: [MemShrink:P2])

On a fresh firefox for android 46 instance about:memory shows:

sqlite: 7.14 MB
  other: 6.87 MB
  webappsstore.sqlite: 0.10 MB
  cookies.sqlite: 0.07 MB
  permissions.sqlite: 0.07 MB
  signons.sqlite: 0.04 MB

On a fresh firefox 46 desktop instance on windows 10 I get:

│    └──1.95 MB (01.84%) -- sqlite
│       ├──0.87 MB (00.82%) ── other
│       ├──0.65 MB (00.62%) ++ places.sqlite
│       ├──0.20 MB (00.19%) ++ webappsstore.sqlite
│       ├──0.07 MB (00.07%) ++ cookies.sqlite
│       ├──0.07 MB (00.06%) ++ permissions.sqlite
│       ├──0.05 MB (00.04%) ++ content-prefs.sqlite
│       └──0.04 MB (00.03%) ++ 2918063365piupsah.sqlite

Why is sqlite:other taking ~7MB on android and less than 1MB on desktop?  Are we missing some tuning parameter on android?
All mozStorage memory reporter infrastructure uses SQLite-provided values.  "other" is the *global* sqlite3_memory_used() less what we counted up for each *mozStorage* connection.

The following possibilities for sources of other are:
* Other stuff in the same process using SQLite but not via mozStorage.  Note that SQLite defaults as compiled (which Fennec does not change at runtime AFAICT) will result in a max cache size of 2 megs per connection.
  * Fennec's SQLiteBridge and its consumers in the same process.  Maybe its BrowserDB stuff lives in the same process?  This is the best explanation for what's going on AFAICT since cache pages are the best explanation for that bulk.  While mozStorage initializes SQLite globally to not pre-allocate the page cache, each connection will grow its page cache up to 2MiB by default.  So 4 connections could explain ~8megs of use.
  * NSS can use SQLite.
  * The js-ctypes bindings to SQLite in sqlite_internal.js at https://dxr.mozilla.org/mozilla-central/source/toolkit/components/sqlite/sqlite_internal.js but DXR suggests no one uses them yet which is consistent with bug 853438 remaining unfixed.
* The SQLite per-connection lookaside allocations which should tally out to 64000=128*5000 byte allocation per connection and it doesn't look like we account for that in our memory reporters.  (They're just sqlite3Malloc calls which get tracked globally, and the per the lookaside constants at https://sqlite.org/c3ref/c_dbstatus_options.html there isn't really a way to get the lookaside block's size.  Noting that sqlite only supports one size, so it won't vary per connection)
* Some other SQLite mechanism, but both https://sqlite.org/malloc.html and some mild cold spelunking suggests there's nothing else we're missing.

Unrelated stuff that should not impact:
* MOZ_STORAGE_MEMORY: We don't define it for android which means we don't register a custom allocator which means SQLite memory allocations may be inefficient on android and result in wasted space.  But this just means that our memory reporters are under-reporting actual memory use.  See https://dxr.mozilla.org/mozilla-central/source/storage/mozStorageService.cpp#403 for more info.

In terms of actionable things, it's not clear there's an easy/clean way to get mozStorage to be able to know anything about SQLite databases opened by things that aren't mozStorage.  From brief code spelunking, it doesn't look like SQLite exposes an API to get the list of open connections which makes sense from a mutex/threading sanity perspective.  While the TelemetryVFS probably does end up getting used by any external consumers because we register it as the default, it only gets a sqlite3_file pointer, not a sqlite3 (connection) one.  So hacks like that are probably out of the question.  I think for accurate reporting we'd need Fennec's SQLite wrapper to call some life-cycle hooks exposed by mozStorage to let us track its connections.

Hacks to avoid confusion include:
* renaming "other" on android/fennec to something like "other non-mozStorage consumers"
* Hiding "other" on fennec.
Whiteboard: [MemShrink]
tracking-fennec: --- → ?
Whiteboard: [MemShrink] → [MemShrink:P2]
tracking-fennec: ? → +
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.