Girish Sharma [:Optimizer] Reporter
	Description • 10 years ago

These notifications should cover:

- New DB creation
- DB Deletion
- DB Schema Update
- New Object Store creation
- Object store deletion
- Change in Object store properties (keys, etc)
- New item added to an object store
- Item removed from an object store
- Item edited in an object store

This is required by Storage Inspector so as to make the IDB view live. (as in, get live updates of changes happening in db being viewed)

	Jonas Sicking (:sicking) No longer reading bugmail consistently
	Comment 1 • 10 years ago

I drafted up this API a while ago as IDBv2 feature to cover notifications for changes to stored data.


dictionary IDBObserverParams {
  any range = undefined; // key, IDBKeyRange or undefined
  boolean key = false;
  boolean value = false; // implies key=true
  boolean oldValue = false; // implies key=true

  // possible optimization
  // boolean dontExpandClear; // would notify about calls to clear() a
                                 single operation rather than as a bunch of individual deletes
}

interface IDBObjectStore {
  ...
  ChangeObserver observe(IDBObserverParams params);
}

interface ChangeObserver : EventTarget {
  attribute EventHandler onchange;
  void cancel();
}

dictionary IDBObserverChange {
  any key; // only set if key was requested

  any value; // only set if value was requested
  boolean removed;  // only set if value was requested

  any oldValue; // only set if oldValue was requested
  boolean added; // only set if oldValue was requested

  // IDBRange clearRange;
}

[Constructor(...)]
interface IDBChangeEvent : Event {
  readonly attribute sequence<IDBObserverChange> changes;
}



We could implement this as [ChromeOnly] for now until the spec gets something similar.

Additionally we'd need the ability to observe changes to DB schemas (i.e. list of objectstores and indexes), as well as creations and removals of DBs themselves.

Given that that's pretty rare, maybe simply having a nsIObserverService notification with something like

interface nsIIDBSchemaChangeNotification {
  DOMString origin;
  DOMString dbName;
  DOMString change; // "created", "deleted", or "updated"
}

On "change" you'd have to rescan the list of object stores and indexes (and update the version number of you show that), but that should be pretty quick given that there usually aren't that many and that we keep all the data in memory. So hopefully that's ok.

	Jonas Sicking (:sicking) No longer reading bugmail consistently
	Comment 2 • 10 years ago

hrm... One thing that is a problem though is that the above data observer feature requires holding the database open. Which if fine for a normal webpage, but probably a problem for devtools since it interferes with how the API behaves. As long as a database is held open, it's schema can't change and it can't be deleted.

Not sure how to solve that. I guess we could enable chrome to open a database "weakly" so that even if it holds the database open, the database's schema can change.

	Ben Turner (not reading bugmail, use the needinfo flag!)
	Comment 3 • 10 years ago

(In reply to Jonas Sicking (:sicking) from comment #2)
> hrm... One thing that is a problem 

We can do this with the versionchange event, no sweat.

	Jonas Sicking (:sicking) No longer reading bugmail consistently
	Comment 4 • 10 years ago

Bent points out that comment 2 isn't actually a problem.

The devtools code can simply listen to the versionchange event. When it fires, simply close the database (which will cancel all observers), and then reopen it immediately again.

That will allow the webpage to upgrade the database schema before the database is reopened again.

Once the database has been reopened, rescan the data to see if it has changed and attach new change observers. This should be pretty rare, and not too slow, so I think it's fine.

In fact, this solution could be used to get notified about schema changes and database deletions. I.e. the devtools code could keep an open handle to any database which it wants to know schema changes for. The only thing that we'd need to use an nsIObserverService notification for is database creations. Though arguably it's easy, and probably faster, to also notify about deletions and schema changes through nsIObserverService.

	Girish Sharma [:Optimizer] Reporter
	Comment 5 • 10 years ago

Me and Ben were discussing this over IRC.

 - The notifications can be batches. No need of sending notification for individual change.
 - We can start by having notification for the last three items in the comment 0 and then go bottoms up and implement others
 - After Ben's massive code change for storages, this will be rather easy to implement.

About the approach in comment 1 and 2, I would rather like an indexedDB independent notification system. i.e. the notification would itself have a frozen list of oldValue, newValue for the items that changed. This way, any observer to the notification would not have to open the DB all over again to get the changes.

For instance, if 2 idb items got added, 1 got deleted and 1 edited in a transaction, then the notification observer can simply send an array of 4 objects of following format

{oldValue: object, newValue: object, objectStore: "name", db: "name", origin: "hostname"}

for added items, oldValue will be null and for deleted, newValue will be null.
For schema changes in objectStore, the objectStore key will be null and for schema changes in db, both the objectStore and db keys will be null;

(old|new)Value will have different format in each of the three cases (item, objectStore, db) in which the format will be fixed for db and objectStore and can be a primitive or an object for item.

Jonas, Ben, what do you think about this approach ?

It might be a little heavy on performance, but we can enable it on docshell basis to only have these notifications for the pages with storage inspector opened.

	Girish Sharma [:Optimizer] Reporter
	Comment 6 • 10 years ago

Ben, what do you think of the above comment ? And how much work is it ?

	Jonas Sicking (:sicking) No longer reading bugmail consistently
	Comment 7 • 10 years ago

(In reply to Jonas Sicking (:sicking) from comment #1)
> I drafted up this API a while ago as IDBv2 feature to cover notifications
> for changes to stored data.
> 
> dictionary IDBObserverParams {
>   any range = undefined; // key, IDBKeyRange or undefined
>   boolean key = false;
>   boolean value = false; // implies key=true
>   boolean oldValue = false; // implies key=true
> 
>   // possible optimization
>   // boolean dontExpandClear; // would notify about calls to clear() a
>                                  single operation rather than as a bunch of individual deletes
> }

One thing that should be called out here is that if key=false, then this only ever generates a single IDBObserverChange when the observer is notified. I.e. when key=false, you're just told that "something" has changed in the requested range.

	Ben Turner (not reading bugmail, use the needinfo flag!)
	Comment 8 • 9 years ago

I believe Jan will be working on this in the future

	Jan Varga [:janv]
	Comment 9 • 9 years ago

(In reply to Ben Turner (not reading bugmail, use the needinfo flag!) from comment #8)
> I believe Jan will be working on this in the future

Yes, right after we finish QuotaManager and FileHandle on PBackground.

	dmurph
	Comment 10 • 9 years ago

Hey everyone!

We've (blink storage) been thinking about this as well, you can see our current thoughts here:
https://github.com/dmurph/indexed-db-observers

We talked to Ben Turner earlier in the summer, but I don't think he's still around. Do you have a roadmap for implementing this feature? I would love to chat in person (and I think I might be in the same building, so that should be easy).

	Jonas Sicking (:sicking) No longer reading bugmail consistently
	Comment 11 • 9 years ago

I can't speak for implementation plans (Jan has a better idea of when he'd have time), but I'm happy to discuss the API. Probably better to do that on the public-webapps list than in bugzilla or github though?

	dmurph
	Comment 12 • 9 years ago

Sounds great! I mostly wanted to meet just to flesh out all of the possible use cases so I wasn't missing anything.  That way the public-webapps discussion can be super focused.

	Jonas Sicking (:sicking) No longer reading bugmail consistently
	Comment 13 • 9 years ago

Meeting up sounds great. I'm located in SF so if you are also in the bay area then that would be really easy. Jan is in Slovakia, so harder to meet in person, but possibly over video.

But let's figure out these details over email instead. This email address is my normal one.

	Jan Varga [:janv]
	Comment 14 • 9 years ago

Yeah, email or video sounds good.

	Bevis Tseng[:bevis][:btseng](Exited)
	Comment 15 • 8 years ago

Will followup this once the spec in https://github.com/dmurph/indexed-db-observers is final.

	Olli Pettay [:smaug][bugs@pettay.fi]
	Comment 16 • 8 years ago

FYI, https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/alcPcPKzzkg

	Jukka Jylänki
	Comment 18 • 8 years ago

Dropping the games tracking label for this, games don't really care about listening to IndexedDB changes themselves since they fully implement IndexedDB in their code, rather than act as a man in the middle.

Instead developers are interested in bug 912562, which we want to track instead. (This bug is a prerequisite for 912562)

	Hsin-Yi Tsai (she/her) [:hsinyi]
	Comment 19 • 8 years ago

Moving btpp-* to P* (priority flag). Flag P3 to reflect the plan so far. Bevis, please feel free to adjust it.

	dmurph
	Comment 20 • 7 years ago

Hello! I'm comfortable with starting origin trials for the feature in chrome, as it's mostly done.

I do not have the key filtering implemented yet, but everything else is working.

	dmurph
	Comment 21 • 7 years ago

You can try it out using canary with the #enable-experimental-web-platform-features flag enabled.

Here is a small demo I whipped up:

https://dmurph.github.io/samples/indexed_db_observers/index.html

	Girish Sharma [:Optimizer] Reporter
	Comment 22 • 7 years ago

(In reply to dmurph from comment #20)
> Hello! I'm comfortable with starting origin trials for the feature in
> chrome, as it's mostly done.
> 
> I do not have the key filtering implemented yet, but everything else is
> working.

(In reply to dmurph from comment #21)
> You can try it out using canary with the
> #enable-experimental-web-platform-features flag enabled.
> 
> Here is a small demo I whipped up:
> 
> https://dmurph.github.io/samples/indexed_db_observers/index.html

Umm... this bug is for Firefox .. ?

	dmurph
	Comment 23 • 7 years ago

Ah, I didn't give much context. I've been collaborating with Mozilla (mostly Bevis) on the spec here: https://github.com/WICG/indexed-db-observers/issues

I'm basically giving an FYI - there is a working implementation of the spec now, so anyone can play around with it and evaluate it.

Sorry for the confusion. I'm a little excited about this spec change ;)

Daniel