Marcus Cavanaugh [:mcav] Assignee
	Description • 11 years ago

From https://github.com/mozilla-b2g/gaia-email-libs-and-more/pull/258#issuecomment-28451654:

UX action item bug to deal with the case where it's clear there are a metric ton of messages in the spool. Propose actions like arbitrarily only picking some finite number of them and putting the rest into the UIDL map so we ignore them and only loook for new messages.

	Marcus Cavanaugh [:mcav] Assignee
	Comment 1 • 11 years ago

As discussed with :asuth over IRC, here's the game plan:

1. Choose a reasonable maximum number of messages to fetch (i.e. 100); the rest are "overflow" messages.

2. Put overflow messages in the UIDL map as "already seen" so that we don't try to download old messages when we issue another sync for the latest messages.

3. Additionally, put overflow message UIDLs in a list of "old messages we haven't fetched yet".

4. In the UI, if there are messages in the overflow list, show the button to "load more". When users click that, trigger a sync to fetch a batch of those UIDLs. Repeated clicks on that would eventually trigger a download of all old messages.

5. Determine, either programmatically or manually, which direction new messages tend to appear in lists as returned by popular POP3 servers, and be sure we start fetching from that side of the list.

	Marcus Cavanaugh [:mcav] Assignee
	Comment 3 • 11 years ago

The new constant in syncbase, POP3_MAX_MESSAGES_PER_SYNC, sets the overflow limit. In addition to the unit test in `test_pop3_overflow_sync.js`, I manually tested it as follows:

1. Change syncbase.POP3_MAX_MESSAGES_PER_SYNC to 2.
2. make install-into-gaia
3. Sync some messages in the simulator/device, see that (a) only two messages get synced first; (b) the "sync more messages" button shows up, and (c) you can repeatedly sync two more messages until all messages have been downloaded.

I tested dovecot, gmail, and AOL mail; all of them return UIDLs in ascending order (oldest first). Accordingly, I flipped the sync logic in POP3 to sync newest messages first (thus overflow messages are the ones going pastwards). To make the unit tests continue to work, I had to make a patch for mail-fakeservers, which previously just blindly appended the messages; the patch makes the pop3 fakeserver return messages the same way other servers do.

I had some trouble with the logic for marking time regions as synced; you can see a "XXX" comment on line 535 of pop3/sync.js. I would think that we'd want to mark the oldest message date, but doing that caused the unit tests to fail. What I've put there now seems to work, but I don't understand it and I'm not sure what the correct solution is.

Right now, GELAM tests are green for me with the mail-fakeservers patch.

	Andrew Sutherland [:asuth] (he/him)
	Comment 4 • 11 years ago

Fabrice, this needs formal sheriff approval to land.  Please approve!

We believe this to be absolutely essential for POP3 support in v1.3 and likely constitutes a blocker.  Without this enhancement, if a user creates a POP3 account with, say, 1,000 or 10,000 messages, we will fetch all of them (well, envelope level details) during our initial synchronization, potentially filling up the device.  With this enhancement, we will only fetch the most recent 100 (or what probably constitutes the most recent.)  The user can then keep requesting 100 more messages at a time to be retrieved.

	[:fabrice] Fabrice Desré
	Comment 5 • 11 years ago

Andrew, feel free to merge.

	Andrew Sutherland [:asuth] (he/him)
	Comment 6 • 11 years ago

Setting needsinfo to track landing.

	Marcus Cavanaugh [:mcav] Assignee
	Comment 7 • 11 years ago

Landed.

mail-fakeservers: https://github.com/mozilla-b2g/mail-fakeservers/commit/dbf1a4eb8995b643a6635b5bbff65be6b4679aac

gelam: https://github.com/mozilla-b2g/gaia-email-libs-and-more/commit/fb921c4f813b905b2756a29d05014f92b91a24f0

gaia: https://github.com/mozilla-b2g/gaia/commit/1b8dd53b663a67d378560a50196994be3c8addb1