Closed Bug 812348 Opened 12 years ago Closed 2 years ago

Firefox Sync duplicating and scrambling bookmarks & folders

Categories

(Firefox :: Sync, defect)

Product:
Firefox
Component:
Sync
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: marty, Unassigned)

References

(Depends on 1 open bug)

Details

(Whiteboard: [sync:bookmarks])

Attachments

(2 files)

duplicated bookmarks
493.09 KB, image/png
Details
"Recursive" syncing. Algorithms are pretty well!
116.70 KB, image/png
Details 
The hierarchy of many hundreds of bookmark folders and bookmarks was not maintained after syncing with other devices. Many bookmarks and folders were moved to or duplicated under "Unsorted bookmarks".
Everything under "Bookmarks Menu" has been duplicated more than once. Many bookmarks formerly under folder structure were moved below all folders.
Adding a new device seems to scramble everything again.
Sync options are set to Merge
Firstly, Sync Options only affect the first sync. You can pretty much ignore that option entirely.

Secondly, what devices have you added, and in what order?

Thirdly, how many bookmarks do you have? Is it "a few hundred" or "six thousand"?

If you install the Places Maintenance addon:

  https://addons.mozilla.org/en-US/firefox/addon/places-maintenance/?src=api

you can run the "Statistics" preset and copy the results to this bug.
Flags: needinfo?(marty)
Thanks
I first added an Android phone, then an iPad, then a Windows Laptop. This was over many months time. The laptop was a few weeks ago.

A new development is that I tried Restoring from earlier (though avaialable backups didn't go back far enough), and that worked, though the next time I started Firefox the entire bookmarks sidebar and manager is empty. HOWEVER, autocomplete in the URL bar is working, so they may be somewhere but just not displaying?

The Statistics say 7810 bookmarks, but that would include the triplication (or more). Here are the Places stats:

> Integrity check
+ The database is sane
> Reindex
+ The database has been reindexed
> Coherence check
+ The database is coherent
> Statistics
Database size is 30720 KiB
user_version is 21
page_size is 32768
cache_size is 128
journal_mode is wal
synchronous is 1
History can store a maximum of 104858 unique pages
Table moz_places has 33866 records
Table moz_historyvisits has 44870 records
Table moz_inputhistory has 531 records
Table moz_bookmarks has 7810 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 0 records
Table sqlite_sequence has 0 records
Table moz_favicons has 2218 records
Table moz_anno_attributes has 11 records
Table moz_annos has 646 records
Table moz_items_annos has 32 records
Table sqlite_stat1 has 15 records
Table moz_hosts has 7108 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_bookmarks_roots_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index sqlite_autoindex_moz_hosts_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_uniqueindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
Index moz_favicons_guid_uniqueindex
Flags: needinfo?(marty)
(In reply to marty from comment #2)
> Thanks
> I first added an Android phone, then an iPad, then a Windows Laptop. This
> was over many months time. The laptop was a few weeks ago.

If you used a pre-release Android version (that is, if you were trying out Aurora or Beta prior to the release of Firefox 14 for Android), you might have been bitten by some early bug. That's life on the bleeding edge!

Remember also that Sync has persistent server data. It only takes one client, once, to muck things up and leave stuff on the server; next time another client downloads a bunch of data, it'll include something painful. Cause and effect can be very far apart.


If you have more than 5,000 bookmarks, or so many that Android kills the sync before we finish, then that would also cause a problem. See Bug 769476 Comment 18 for some details.


> A new development is that I tried Restoring from earlier (though avaialable
> backups didn't go back far enough), and that worked, though the next time I
> started Firefox the entire bookmarks sidebar and manager is empty. HOWEVER,
> autocomplete in the URL bar is working, so they may be somewhere but just
> not displaying?

The Awesomebar autocomplete runs against history, not just bookmarks. Doesn't sound related to this bug, but feel free to file one in Bookmarks for that issue.


> The Statistics say 7810 bookmarks, but that would include the triplication
> (or more).

Perhaps. If you just restored from earlier backups, then there would be no duplication, right?
Running Aurora 18.0a2 (2012-11-19).
I've had exactly the same problem for quite some time now. I have two devices (two Win7 PC's) set up, and every time this bug occurs, I have to use add-ons like CheckPlaces to scan for duplicates.
(Side note: Moving ~500 or more bookmarks often crashes the browser).

Stats report 25093 bookmarks (including the duplicates):
> Statistics
Database size is 20480 KiB
user_version is 21
page_size is 32768
cache_size is 128
journal_mode is wal
synchronous is 1
History can store a maximum of 104858 unique pages
Table moz_places has 24164 records
Table moz_historyvisits has 30 records
Table moz_inputhistory has 857 records
Table moz_bookmarks has 25093 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 2 records
Table sqlite_sequence has 1 records
Table moz_favicons has 4612 records
Table moz_annos has 7744 records
Table moz_anno_attributes has 11 records
Table moz_items_annos has 9169 records
Table sqlite_stat1 has 15 records
Table moz_hosts has 7362 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_bookmarks_roots_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index sqlite_autoindex_moz_hosts_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_uniqueindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
Index moz_favicons_guid_uniqueindex
(In reply to rctgamer3 from comment #4)
> Running Aurora 18.0a2 (2012-11-19).
> I've had exactly the same problem for quite some time now. I have two
> devices (two Win7 PC's) set up, and every time this bug occurs, I have to
> use add-ons like CheckPlaces to scan for duplicates.

How often have you noticed this?

Do you have any other devices connected to your Sync account, other than your Windows 7 PCs?
Flags: needinfo?(rctgamer3)
I did use Beta on my Android phone so an early bug might be partially responsible.
The 5000 mark limitation (or is it 5 minutes?)as explained in  Bug 769476 Comment 18 sounds very close to my symptoms.

The impact of this on my ability to work cannot be understated. If I did nothing else, it will take me days to reorganize these*. This limitation should be stated up front as a WARNING! 

*As of now, I have NO bookmarks. Zip, Zilch, as in EMPTY. I've tried restoring the earliest and the latest dates available, and still nothing. I'm hesitant to go any further without expert advise.
(In reply to Richard Newman [:rnewman] from comment #5)
> How often have you noticed this?
Whenever I'm sorting bookmarks (once every couple of weeks or so). I just checked both computers, and the folders are currently out of sync by a couple of hundred bookmarks.

> Do you have any other devices connected to your Sync account, other than
> your Windows 7 PCs?
No. I tried the Firefox Sync app (for iOS) app once (about 1.5 years ago or so), but I reset my account/key a couple of months after, so it might not even be related/relevant to this bug.
Flags: needinfo?(rctgamer3)
(In reply to rctgamer3 from comment #7)
> Whenever I'm sorting bookmarks (once every couple of weeks or so). I just
> checked both computers, and the folders are currently out of sync by a
> couple of hundred bookmarks.

Can you get a sync log from both devices? See

  https://philikon.wordpress.com/2011/06/13/how-to-file-a-good-sync-bug/

and be sure to turn on success logging.
Flags: needinfo?(rctgamer3)
Depends on: 814801
(In reply to marty from comment #6)
> I did use Beta on my Android phone so an early bug might be partially
> responsible.

Yup.

> The 5000 mark limitation (or is it 5 minutes?)as explained in  Bug 769476
> Comment 18 sounds very close to my symptoms.
> 
> The impact of this on my ability to work cannot be understated. If I did
> nothing else, it will take me days to reorganize these*. This limitation
> should be stated up front as a WARNING! 

I'm sorry you're frustrated. Sadly, software is imperfect, and not every edge case is correctly assessed in anything but hindsight.

My suggestions are:

* If your bookmarks are very precious, treat them as you would any other piece of precious data: back them up, in multiple locations, with dated backups. If losing your bookmarks would impact your ability to work, then you're vulnerable to bugs in your browser, in addons, to hard drive or OS failures, and even to bugs in your backup software. Right now you should be saying "well, that's annoying", then pulling a hard drive off the shelf and restoring an older Firefox profile, rather than manually reordering bookmarks.

* If you have lots of bookmarks, don't enable bookmark sync.

* Keep an eye on Bug 814331 (which aims to prevent others from ending up in your position) and Bug 814801 (which will eventually allow for arbitrary-size syncs).


> *As of now, I have NO bookmarks. Zip, Zilch, as in EMPTY. I've tried
> restoring the earliest and the latest dates available, and still nothing.
> I'm hesitant to go any further without expert advise.

That sounds like you have another bug. Try running the Places Maintenance addon, see if anything is wrong; if that reports nothing, look in the Error Console.
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Windows 7 → All
Hardware: x86_64 → All
Version: 16 Branch → unspecified
Well, in hindsight, I'd agree. Here's an idea, a selectable sync option- Primary Desktop, Others secondary. Delegates the desktop as the primary, transmitting one-way to secondary devices, except that any bookmarks made on secondary devices are placed in a separate "Mobile Bookmarks" folder and sent separately to all devices. This way only "Mobile bookmarks" would be vulnerable to corruption.

I did run the Places Maintenance routine, see the results copy above.
Where are Backup / Restore files stored and what do files names.extensions would they have? The restore menu shows 6 restore points, and I want to verify that the files still exist.

I now have an Unknown Sync Error message with an option to retry. Would this overwrite anything?

Thanks
(In reply to marty from comment #10)
> Well, in hindsight, I'd agree. Here's an idea, a selectable sync option-
> Primary Desktop, Others secondary.

You and I are two of the tiny minority of people who are able to understand that topology; it's not a viable user experience. We want to make Sync *less* complicated, not more.

Personally, I'd much rather concentrate on making bookmark sync work in every edge case, rather than spend time making sure a more complicated multi-directional scheme worked at all.


> I did run the Places Maintenance routine, see the results copy above.
> Where are Backup / Restore files stored and what do files names.extensions
> would they have? The restore menu shows 6 restore points, and I want to
> verify that the files still exist.

$profiledir/bookmarkbackups/

They're JSON blobs.

Note that -- if you're using Time Machine or a similar backup tool -- you can scroll back through time in this folder and see snapshots of your backups through time. Pretty awesome.


> I now have an Unknown Sync Error message with an option to retry. Would this
> overwrite anything?

That error is probably indicative of -- as I mentioned -- a problem with your local bookmarks database (e.g., file corruption of places.db). about:sync-log should include an error log. Retrying will probably do nothing of note -- same issue again.
I did find about 11 bookmarkbackup files dating back to 11/09/12.
Is it correct that restoring to any of them will not overwrite anything other than the current file, which at this point is empty anyway? Will that also restore the probably corrupted Places.db database? Is there another way to recover or debug the database?

I also found the sync-logs. There's one from 11/20 that is much larger than any other and contains a number of errors. Is there anything I can do with this information? Would it be helpful in any way to either recover from the situation or prevent another episode?

Thank you Richard,
Marty
(In reply to marty from comment #12)
> I did find about 11 bookmarkbackup files dating back to 11/09/12.
> Is it correct that restoring to any of them will not overwrite anything
> other than the current file, which at this point is empty anyway?

Correct.

If you want to be really paranoid, make a copy of your profile directory (<https://support.mozilla.org/en-US/kb/back-and-restore-information-firefox-profiles#w_backing-up-your-profile>) and all of those backups.


> Will that also restore the probably corrupted Places.db database?

That depends on what's wrong.

> Is there another way
> to recover or debug the database?

If you don't mind leaking your history and bookmarks, you can send it to one of us at Mozilla to take a look. If you do mind, then not really :D

> I also found the sync-logs. There's one from 11/20 that is much larger than
> any other and contains a number of errors. Is there anything I can do with
> this information? Would it be helpful in any way to either recover from the
> situation or prevent another episode?

It might tell us what's wrong, yes.

If you can zip up every error log in $profile/weave/logs, and send it to me, then I'd be happy to take a look.

Those logs will include some details about your Sync account (username, etc.), and potentially some information about your bookmark and history data.
Hi,
I'd be willing to provide you with those files for diagnostics, but here's a bit of weirdness that might be useful first- 

Selecting All Bookmarks or All History brings up the Library viewer, and both are completely blank. No matter if I try to Restore bookmarks from any available date or import from an HTML backup file, I get nothing. No errors, and no content.

What do you make of that?
(In reply to marty from comment #14)

> Selecting All Bookmarks or All History brings up the Library viewer, and
> both are completely blank. No matter if I try to Restore bookmarks from any
> available date or import from an HTML backup file, I get nothing. No errors,
> and no content.
> 
> What do you make of that?

As I suggested in Comment 9, it sounds like a corrupt Places database (either semantically or structurally).
Corresponding over email. Looks like the underlying problem for marty is Bug 670069.
Blocks: bookmarksync
Whiteboard: [sync:bookmarks]
No longer blocks: bookmarksync
Depends on: bookmarksync
Late reply:
I had one folder which every unsorted bookmark went in (not "Unsorted Bookmarks" but a folder with bookmarks i still need to sort). That folder became too large to sync (because of the sheer amount of duplicates caused by this bug, hence "payload too large" errors) and was probably blocking other bookmarks/folders from syncing. I moved 50% of the bookmarks in that folder to a second folder. That seemed to have worked, and I no longer get any errors.

Though I managed to fix it, I still don't know what exactly went wrong.
Flags: needinfo?(rctgamer3)
(In reply to rctgamer3 from comment #17)
> Late reply:
> I had one folder which every unsorted bookmark went in (not "Unsorted
> Bookmarks" but a folder with bookmarks i still need to sort). That folder
> became too large to sync

Do you know what that count was?

The theoretical limit that I know of is about 17,000 bookmarks per folder.
Component: Bookmarks & History → Firefox Sync: Backend
Product: Firefox → Mozilla Services
(In reply to Richard Newman [:rnewman] from comment #18)
> Do you know what that count was?
> 
> The theoretical limit that I know of is about 17,000 bookmarks per folder.

Problem's back. Weave/Sync Fails to sync one folder with 14527 bookmarks in it (payload too large).
(In reply to rctgamer3 from comment #19)

> Problem's back. Weave/Sync Fails to sync one folder with 14527 bookmarks in
> it (payload too large).

Not unexpected; that's in the ballpark I predicted.

The only solution, I'm afraid, is "don't do that". Android Sync will refuse to sync bookmarks at all if you have that many, and desktop will run into protocol limitations like maximum upload size. In this case, the record for that folder includes JSON like:

  children: ["abcdefghijkl", …]

which will be a 217,000 character array. That, plus other fields, plus encryption overhead, means that the individual record is simply too big.

This will not be fixed in Sync. Cross your fingers for PICL. You are a very, very unusual user.
I have run into this problem after having installed Firefox on my Android phone (Samsung Gamaxy S4, Android 4.2). Adding the phone to Sync duplicates hundreds of bookmarks and scrambles the heirarchy. Devices on Sync ar my desktop (XP SP3), my laptop (Windows 7), and the phone. Sync works just fine between the desktop and laptop. I tried restoring from backup and instructing Sync to overwrite data on the laptop and phone with that from my desktop, but synching from the phone still scrambled the bookmarks. And there is no setting on Firefox for Android to tell sync to overwrite local bookmarks. Data regarding my database:

> Integrity check
+ The database is sane
> Reindex
+ The database has been reindexed
> Orphans expiration
+ Database cleaned up
> Coherence check
+ The database is coherent
> Vacuum
Initial database size is 10240 KiB
+ The database has been vacuumed
Final database size is 10240 KiB
> Statistics
Database size is 10240 KiB
user_version is 21
page_size is 32768
cache_size is 128
journal_mode is wal
synchronous is 1
History can store a maximum of 80490 unique pages
Table moz_places has 1175 records
Table moz_historyvisits has 22 records
Table moz_inputhistory has 0 records
Table moz_hosts has 1029 records
Table moz_bookmarks has 2502 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 17 records
Table sqlite_sequence has 1 records
Table moz_favicons has 324 records
Table moz_anno_attributes has 11 records
Table moz_annos has 0 records
Table moz_items_annos has 179 records
Table sqlite_stat1 has 14 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_hosts_1
Index sqlite_autoindex_moz_bookmarks_roots_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_uniqueindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
(In reply to Michael A. Koenecke from comment #21)
> I have run into this problem after having installed Firefox on my Android
> phone (Samsung Gamaxy S4, Android 4.2). Adding the phone to Sync duplicates
> hundreds of bookmarks and scrambles the heirarchy. Devices on Sync ar my
> desktop (XP SP3), my laptop (Windows 7), and the phone. Sync works just fine
> between the desktop and laptop. I tried restoring from backup and
> instructing Sync to overwrite data on the laptop and phone with that from my
> desktop, but synching from the phone still scrambled the bookmarks.

We would need a log from the phone to find out what's happening here. The log would need to include the entire bookmark sync from start to finish.

http://160.twinql.com/how-to-file-a-good-android-sync-bug/

Feel free to send it to me privately.

With 2500 bookmarks, most likely something is (repeatedly) failing part-way through the sync -- timeout, malformed record, some kind of collision -- and the phone is attempting to recover and just making matters worse.

> there is no setting on Firefox for Android to tell sync to overwrite local
> bookmarks.

"Reset others" from desktop, as you said you did, will cause that to happen without having to do anything on the phone.
Flags: needinfo?(mike)
My apologies: I tried obtaining a log via running aLogCat during the first synch, but apparently aLogCat does not retain information long enough for it to be recorded. The alternative appeared to require installing an entire development toolkit for the sole purpose of getting one log, and I'm just a lawyer, not a programmer. After three or so attempts, after each of which I had to restore bookmarks completely, I gave up. I used an extension to remove duplicate bookmarks and manually fixed the file on Firefox in Windows. It seems to be fairly stable now. Doubtless it is still a problem, but I cannot come up with the information needed to solve it. Sorry.
Flags: needinfo?(mike)
(In reply to Michael A. Koenecke from comment #23)
> My apologies: I tried obtaining a log via running aLogCat during the first
> synch, but apparently aLogCat does not retain information long enough for it
> to be recorded. The alternative appeared to require installing an entire
> development toolkit for the sole purpose of getting one log, and I'm just a
> lawyer, not a programmer.

aLocRec doesn't, at least, but I understand if you're reluctant to spend the time.
Okay, I'll try restoring my bookmarks to the old set and see if I can duplicate the problem while running aLocRec. There had to be something in those bookmarks that made things go haywire: after putting in a lot of time cleaning up and reorganizing them, they now seem pretty stable. Though I now show *two* Samsung Galaxy S4 devices in my synched devices, which is odd.
I had this happen today and unlike the above poster, I have VERY few bookmarks, like under 50...

Sync was working fine between my desktop and laptop, but after syncing my galaxy s2 and nexus 7 it totally screwed with my bookmarks on my desktop computer, the "show all bookmarks" sidebar was showing many duplicates (duplicate bookmarks toolbar folder, bookmarks menu folder, history and tags duplicates were even showing in the sidebar). I also could not delete any of these folders, delete was greyed out. Also under "Unsorted bookmarks" there was an "all bookmarks" folder which seemed to have an unlimited hierarchy (under all bookmarks was all my bookmarks, including unsorted bookmarks, and under unsorted was all bookmarks again, and this went on forever).

My laptop wasn't showing the duplicated on the show all bookmarks sidebar, but it was showing the infinite "all bookmarks" rabit hole. I ended up unlinking all my devices, restoring my desktop and laptop from a bookmarks backup (and for some reason on the desktop I actually had to reset my profile, because restoring from the json backup and telling it to replace all existing bookmarks wasn't changing anything, but after resetting the profile and importing from the same backup it was fine...), and re-linking the desktop and laptop to my sync account. Haven't tried to resync my android devices yet because I'm afraid it will screw up my bookmarks again...
Maybe as a way forward this should block the meta bug 621584 Or possibly be considered as a duplicate of for instance bug 618403
Blocking the meta bug might help discovery, so doing that. It won't make this any more likely to move forward, though. Making bookmark sync reliable in all situations involves some kind of rewrite (e.g., Bug 814801).
Blocks: 621584
Attached image duplicated bookmarks — 
I still have this happen intermittently even with the new sync and therefore a new sync account, really quite annoying. It just decided to happen again today. Everything was fine, but all the sudden today all kinds of duplicate folders and weirdness as seen in attached screenshot.
and as before, it only seems to happen shortly after adding my android devices to sync...This is an insanely annoying bug, when it happens I have to disconnect from sync, remove all my bookmarks, and resync.
*Looks like just deleting and re-syncing doesn't work anymore either :(

I deleted my places.sqlite folder which got rid of all the weird duplicates, but upon signing into firefox sync they all returned, wonderful...
Nothing should be different with new Sync; that's just a change to the auth layer.

If you have lots of bookmarks, or a flaky network connection, or add/change bookmarks frequently, then you're at increased risk of seeing corruption, and there's not much you can do about it. I recommend not using bookmark sync if it's a problem for you.

Out of interest, how many bookmarks do you have, Brandon?
Flags: needinfo?(bwatkins)
It doesn't seem to matter if you have several thousand bookmarks as I do, or less than 50 as Brandon does. As Richard suggests, if bookmarks are important to you, do not use sync.

I'd like to repeat a suggestion I made when I first reported this problem 2-1/2 years ago that could be easier to implement and be more reliable:
As an alternative to live sync, I'd like my mobile devices to receive /update their list of bookmarks, passwords, etc. from my "Main" Firefox installation, which would be my Desktop.

As an additional option, any bookmarks, etc. that save to my mobile devices could be communicated to other devices via email, message, or other means, and be optionally added to the main lists by some method other than live sync. Once they are added to the main installation, they would be included in all other devices upon the next update.

That last option would facilitate the occasions when I want to save bookmarks on a mobile device, say when I'm traveling, but I don't necessarily want them  added permanently to my main bookmarks list.
@Richard Newman

I currently have ~175 bookmarks, but I've had this issue occur with as little as < 50 bookmarks in the past.

I generally don't have a lot of bookmarks, but I do add/remove stuff from unsorted bookmarks somewhat frequently.
Flags: needinfo?(bwatkins)
> I'd like to repeat a suggestion I made when I first reported this problem
> 2-1/2 years ago that could be easier to implement and be more reliable:

I'm afraid this suggestion is no easier to implement -- if we were going to the effort to build a second bookmarks channel, we'd just build a better bidirectional bookmark sync and make everyone happy, rather than the minority that wants unidirectional update with some kind of out-of-band addition.
Had this happen again today, and this time I didn't even have any mobile devices added to sync, only my windows desktop pc and my macbook air... Are there any plans yet to improve the reliability of sync? Its not acceptable for a sync feature to randomly freak out like this. I've had the same google account for years and have never had chrome's sync scramble my bookmarks around.. With firefox I end up having to delete my sync account and wiping my firefox bookmarks every few months :/ There really needs to be some kind of sanity/integrity checks implemented in sync to prevent this from happening.
(In reply to Brandon Watkins from comment #36)
> Had this happen again today, and this time I didn't even have any mobile
> devices added to sync, only my windows desktop pc and my macbook air...

The problem isn't limited to mobile devices. The fundamental problem with bookmark sync is that it's a structured format being exchanged without transactions or good conflict detection, combined with a reliance on heuristics: Bug 814801 has most of the details.


> Are there any plans yet to improve the reliability of sync?

There have been some proposals (e.g., <https://wiki.mozilla.org/User_Services/Sync#Technical_Materials>, "Storage research"), but I'm not aware of anyone prioritizing building a replacement for bookmarks sync specifically, or building a more fundamentally sound syncing solution in general. Sync 1.x sucks, but it's also "good enough", so in the big picture it's hard to justify the significant investment required to do better.


Doing bookmark sync truly correctly and scalably requires a better protocol -- equivalent to a shift from WebDAV or FTP to something like Git, which allows for incremental transmission of structures with conflict detection.

Improving reliability with the current protocol and record format isn't impossible, but involves performance tradeoffs and itself is very likely to introduce its own set of consistency/correctness bugs, particularly on desktop.

My feeling is that much of the problem comes from interrupted uploads, not downloads, and that's very difficult for a client to solve.

Furthermore, Sync as a whole doesn't track enough information to do three-way merges in the case of conflicts: see Bug 1098501 for a bug to do that specifically for passwords, which might generalize to other independent record types. That doesn't help with bookmarks, though.
(In reply to Richard Newman [:rnewman] from comment #37)

> Doing bookmark sync truly correctly and scalably…

Just to expand on this momentarily:

It's easy to suggest using a data format that isn't reliant on cross-record structure as a solution for this; cross-record structure is the main cause of problems. Right now every bookmark and every folder gets its own record, so moving a bookmark between folders means uploading three records. So why not put every bookmark in a single record? That way you get atomic uploads and downloads!

The problems with this are:

* Conflict detection and resolution become very important. Now you get an atomic blob containing every bookmark, and you have to resolve them. Collisions in a folder result in screwed up ordering or records being dumped in unsorted bookmarks.

* Scaling to large numbers of bookmarks becomes impossible. Consider Comment 2 (> 7000 bookmarks), Comment 19 (14,527 bookmarks in a single folder!).

* Efficiency/perf are dramatically impacted. Every bookmark addition would involve reuploading your entire bookmark bundle on one device, and redownloading the whole lot on every other device. Every bookmark change would be like a first sync is now -- that is, it'll hang your browser for multiple seconds.

* There would still be a ton of work to do. Bookmark sync uses your real live bookmark database as working storage as it applies records, and that needs to change.


The only real solution for this problem that I see is to allow for clients to maintain an equivalent of a shadow bookmarks DB (in order to handle shared parents), and for the server model to change to allow for incremental uploads with an atomic compare-and-swap.

It's probably possible to build this on top of the existing Sync storage service, with some difficulty.

The main point of this comment, though: simple solutions do not exist for this problem. Even the potential mitigations (e.g., stop treating Unsorted Bookmarks as a structured collection) come with their own snags.
Sync going crazy. Firefox 31.4.0esr
We have to determine a way for reproducing the bug, otherwise it won't go fixed.
I was able to fix my problem by deleting all places.sqlite* .

The downside is History goes away as well.
In my case problem was caused by Firefox for android (most probably, issue 769476).
After deleting firefox account on my android device, unusual effects was gone.
So for now I cannot sync with android, but it's better than have scrambled bookmarks on all devices.
(In reply to pedrogfrancisco from comment #40)
> We have to determine a way for reproducing the bug, otherwise it won't go
> fixed.

Unusually, no, we don't need steps-to-reproduce for this bug. Practical STR would be hard to define ("after nineteen milliseconds, unplug your ethernet cable…"), and theoretical STR are already described.

The shortcomings with bookmark sync are well understood, at least by me and a few others who have worked on the innards of Sync. They're fairly well outlined in this bug. There are a range of causes, from implementation decisions (direct record application) through to protocol limitations (lack of exclusionary primitives), which together allow consistency errors to occur.

Achieving reliability doesn't really entail fixing a bug in the traditional sense. It involves a total rewrite of bookmark syncing on desktop, a partial rewrite of bookmark syncing on Android, and some difficult tradeoffs (e.g., an increase in space usage).

It also implies some significant changes to the overall structure of Sync clients -- e.g., to allow for partial/incremental syncs.

This isn't really directly actionable by non-experts.

The iOS Sync client I'm about to build will be something of a test run for an architecture that I feel minimizes the weaknesses in Sync, and would allow for relatively robust bookmark syncing. If the fates align, we can use that as a model for desktop.
Richard Newman [:rnewman],
Maybe my bug is different. It's the second time I go check the filesize after noticing duplicate/triplicate/octuplate bookmarks.

Both times, places.sqlite is precisely 10,0 MB (10 485 760 bytes).

Cound something else be going on here?
Ok, places.sqlite is allocated in 10MB chunks. So ignore last comment.
Hello,
Because I have more than 15000 bookmarks, I can't use the Sync solution offered by Firefox to have my bookmarks on mobile.
However, going back to Chrome to have access to my bookmarks on mobile pains me a lot, as I have many privacy issues with Google.
As such, excuse me if my suggestions sound dumb (I'm a sys admin not a developer), but wouldn't it be possible for the 0,01% "poweruser" people like me to :
- run our own sync server and adjust the settings to something suited for our needs ? 
Having my own Nginx webserver allow me to adjust upload limit to 256MB if I want to, same for all other settings, so lots of issues solved, no ?
I read in an other bugzilla thread than storage would be an issue and who would paid for it. Well here, it's the actual user who pay for it, so problem solved... 
But I can't find how to sync firefox for android to my personnal sync server...
https://docs.services.mozilla.com/howtos/run-sync-1.5.html
- export in an html file and import manually for the first "sync", and then watch for the changes ?
(In reply to Y from comment #46)
> But I can't find how to sync firefox for android to my personnal sync
> server...
> https://docs.services.mozilla.com/howtos/run-sync-1.5.html
> - export in an html file and import manually for the first "sync", and then
> watch for the changes ?

You can do it by first installing this add-on: https://addons.mozilla.org/fr/android/addon/fxa-custom-server-addon
However be careful to check that the right URL is used when you sign in (like it is shown in one of the screenshots). The first time I tried it didn't work and my bookmarks were synced to the official server.
(In reply to Y from comment #46)

> - run our own sync server and adjust the settings to something suited for
> our needs ? 

The problem isn't really the server -- it's the clients. Much simpler would be a switch: "try to sync any number of bookmarks?".

The issue is that you'd need to guarantee that the clients you use all:

1. Have perfectly reliable network connections
2. Have very fast storage
3. Never make any mistakes.

If you're wrong, then you stand some chance of data corruption. The limit we picked is based on #2 with a little #1 mixed in.

There is a protocol limit, which is the maximum size for an individual record (Comment 20). I don't think that's relevant for most users, and there's an easy workaround: add a couple of folders.


> Having my own Nginx webserver allow me to adjust upload limit to 256MB if I
> want to

If you're thinking "can't I just upload and download all changed records in one go?" -- you'd need to change a few batching constants on each platform, and you'd also need to make sure you had tons of memory available on every device. I don't know offhand how much memory it would take to batch fifteen thousand bookmark records on Android, but I'd bet it would trigger a low-memory event on most phones.

More importantly, to be totally correct you'd need to ensure that clients never synced at the same time.

For most users, most of the time, bookmark sync works just fine. For users at the edges, it's fundamentally broken. It's a known problem. The algorithm I'm designing for Firefox on iOS should avoid most of them, and might one day make its way to desktop and Android.

> I read in an other bugzilla thread than storage would be an issue and who
> would paid for it. Well here, it's the actual user who pay for it, so
> problem solved... 

Storage is not the issue with this bug; it's a factor in what and how we might choose to sync, favicons being one example, but it didn't affect the decision on this limit. Reliability is the problem here.
This problem is far from being resolved I undergo exactly the same for some probably months.
Firefox was incredibly slow in response time.
And I discovered at least a quadruple copy of all of my bookmarks.
The size of probably more than 3000 bookmarks is now under 2K (HTML format)
Before the complete cleaning it was above 10K.
It is easy to say that a corruption occurred somewhere.
The fact is I have abandoned the SYNC system and REACTIVATE XMARKS which was still present has inactive plugin. You should take example on this plugin that never failed for years.
I decided to try the internal SYNC with these bad results.
Kindly,

Jo
5 years later, I have the same problem,   30K bookmarks, and they get completely scrambled both bookmarks and folders beyond recognition.  I consider bookmark sync to be a total failure, and will never use it until it gets the attention it deserves.
(In reply to Frederick Sieber from comment #51)
> 5 years later, I have the same problem,   30K bookmarks, and they get
> completely scrambled both bookmarks and folders beyond recognition.  I
> consider bookmark sync to be a total failure, and will never use it until it
> gets the attention it deserves.

I understand this is frustrating, but we have been working on making bookmark syncing more reliable (apparently not completely successfully) and are still working on that.

Could you please let us know what devices you have connected to your Sync account? ie, do you have Firefox for Android or iOS connected?
I was running on windows 10, using portable firefox 58,  two different portable installations to check out how well bookmark sync would work.   I created a new sync account, but used an existing profile with about 30K bookmarks.    I could have saved and restored the booksmarks to "purify" the profile bookmark files, but then I would lose all my favicons, another "feature" designed with the programmer in mind, but not the end user.  I could now try it again doing a save/restore first and use he addon checkmarks to restore the favicons.
Hi All,

What Ive noticed is that it affects me when I have the same bookmarks folder names on PC and mobile devices. Now I have changed the names of the folders and it looks ok for now. On PC and Mobile (Android) I use Firefox Beta.
Hello 
I am experiencing this problem, which is very frustrating. I edit the order of Bookmarks and the next day (after rebooting) the  bookmarks are haywire once again.
I am using Firefox Quantum 61.0.1 (64-bit) on a Windows 10 Home PC, Version 1803, build 17134.165
I also was using Firefox on a Mac Mini, but this recurring problem has made me start using Safari on my mac instead of Firefox!
(In reply to David Farbey from comment #55)
> Hello 
> I am experiencing this problem, which is very frustrating.

Hi David,
  Sorry to hear you are also having this issue. Do you have any mobile Firefoxes (eg, Firefox for Android or for iOS) connected and syncing with your account?
Component: Firefox Sync: Backend → Sync
Product: Cloud Services → Firefox
On Firefox 62.0.2 Stable on Windows 10.0.17134.320, I've been having this issue for several days. Neither restarting Firefox with add-ons disabled nor refreshing Firefox twice was any help. When I try to move a bookmark one place up which is where I always wanted it, it refuses. I can move bookmarks down just fine, but then as soon as I reopen my library, something gets scrambled again.
The two devices I use Firefox Sync with are my Dell Inspiron 7558 and iPhone 6s. An iPad will be a third device next year.
(In reply to Thomas James Seymour from comment #57)
> On Firefox 62.0.2 Stable on Windows 10.0.17134.320, I've been having this
> issue for several days. Neither restarting Firefox with add-ons disabled nor
> refreshing Firefox twice was any help. When I try to move a bookmark one
> place up which is where I always wanted it, it refuses. I can move bookmarks
> down just fine, but then as soon as I reopen my library, something gets
> scrambled again.

Does this happen immediately after moving the bookmark, or some time after? If immediately, it sounds unlikely to be directly related to sync. If it is a problem with the places database, refreshing may not fix it. Could you please visit about:support, and down near the bottom in the "Places Database" section, click the "Verify Integrity" button and paste the output here?
> Task: checkIntegrity
+ The places.sqlite database is sane
+ The favicons.sqlite database is sane
> Task: invalidateCaches
+ The caches have been invalidated
> Task: checkCoherence
+ The database is coherent
> Task: expire
+ Database cleaned up
> Task: originFrecencyStats
+ Recalculated origin frecency stats
> Task: vacuum
+ Initial database size is 10240KiB
+ The database has been vacuumed
+ Final database size is 10240KiB
> Task: stats
+ Places.sqlite size is 10240KiB
+ Favicons.sqlite size is 14912KiB
+ pragma_user_version is 52
+ pragma_page_size is 32768
+ pragma_cache_size is -2048
+ pragma_journal_mode is wal
+ pragma_synchronous is 1
+ History can store a maximum of 118977 unique pages
+ Table moz_origins has 3425 records
+ Table moz_places has 13816 records
+ Table moz_historyvisits has 17527 records
+ Table moz_inputhistory has 103 records
+ Table moz_bookmarks has 5267 records
+ Table moz_bookmarks_deleted has 0 records
+ Table moz_keywords has 63 records
+ Table sqlite_sequence has 1 records
+ Table moz_anno_attributes has 5 records
+ Table moz_annos has 1617 records
+ Table moz_items_annos has 1990 records
+ Table moz_meta has 7 records
+ Table sqlite_stat1 has 20 records
+ Index sqlite_autoindex_moz_origins_1
+ Index sqlite_autoindex_moz_inputhistory_1
+ Index sqlite_autoindex_moz_bookmarks_deleted_1
+ Index sqlite_autoindex_moz_keywords_1
+ Index sqlite_autoindex_moz_anno_attributes_1
+ Index moz_places_url_hashindex
+ Index moz_places_hostindex
+ Index moz_places_visitcount
+ Index moz_places_frecencyindex
+ Index moz_places_lastvisitdateindex
+ Index moz_places_guid_uniqueindex
+ Index moz_places_originidindex
+ Index moz_historyvisits_placedateindex
+ Index moz_historyvisits_fromindex
+ Index moz_historyvisits_dateindex
+ Index moz_bookmarks_itemindex
+ Index moz_bookmarks_parentindex
+ Index moz_bookmarks_itemlastmodifiedindex
+ Index moz_bookmarks_dateaddedindex
+ Index moz_bookmarks_guid_uniqueindex
+ Index moz_keywords_placepostdata_uniqueindex
+ Index moz_annos_placeattributeindex
+ Index moz_items_annos_itemattributeindex
> Task: _refreshUI
Flags: needinfo?(markh)
Does the problem still occur? If so, it might be worth attaching the most recent log file shown in about:sync-log.
Flags: needinfo?(markh)
1538088988110	Sync.Service	INFO	Loading Weave 1.64.0
1538088988115	Sync.Engine.Clients	DEBUG	Engine constructed
1538088988120	Sync.Engine.Clients	DEBUG	Resetting clients last sync time
1538088988179	Sync.Engine.Addons	DEBUG	Engine constructed
1538088988188	Sync.Engine.Addons	DEBUG	SyncEngine initialized: addons
1538088988190	Sync.AddonsReconciler	DEBUG	No data seen in loaded file: addonsreconciler
1538088988194	Sync.Engine.Forms	DEBUG	Engine constructed
1538088988200	Sync.Engine.Forms	DEBUG	SyncEngine initialized: forms
1538088988246	Sync.Engine.History	DEBUG	Engine constructed
1538088988254	Sync.Engine.History	DEBUG	SyncEngine initialized: history
1538088988261	Sync.Engine.Passwords	DEBUG	Engine constructed
1538088988268	Sync.Engine.Passwords	DEBUG	SyncEngine initialized: passwords
1538088988272	Sync.Engine.Prefs	DEBUG	Engine constructed
1538088988280	Sync.Engine.Prefs	DEBUG	SyncEngine initialized: prefs
1538088988290	Sync.Engine.Tabs	DEBUG	Engine constructed
1538088988296	Sync.Engine.Tabs	DEBUG	SyncEngine initialized: tabs
1538088988297	Sync.Engine.Tabs	DEBUG	Resetting tabs last sync time
1538088988301	Sync.Engine.Extension-Storage	DEBUG	Engine constructed
1538088988306	Sync.Engine.Extension-Storage	DEBUG	SyncEngine initialized: extension-storage
1538088988444	Sync.Engine.Bookmarks	DEBUG	Engine constructed
1538088988450	Sync.Engine.Bookmarks	DEBUG	SyncEngine initialized: bookmarks
1538088988451	Sync.Service	INFO	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0
1538088988456	Sync.AddonsReconciler	INFO	Registering as Add-on Manager listener.
1538088988456	Sync.AddonsReconciler	DEBUG	Adding change listener.
1538088988456	Sync.Engine.History.Tracker	INFO	Adding Places observer.
1538088988463	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent
1538088988746	Sync.AddonsReconciler	DEBUG	Add-on change: onUninstalling to gmp-gmpopenh264
1538088988746	Sync.AddonsReconciler	DEBUG	Ignoring onUninstalling for restartless add-on.
1538088988748	Sync.AddonsReconciler	DEBUG	Add-on change: onUninstalled to gmp-gmpopenh264
1538088988748	Sync.AddonsReconciler	INFO	Saving reconciler state to file: addonsreconciler
1538088988963	Sync.AddonsReconciler	DEBUG	Add-on change: onInstalling to gmp-gmpopenh264
1538088988963	Sync.AddonsReconciler	DEBUG	Ignoring onInstalling for restartless add-on.
1538088988963	Sync.AddonsReconciler	DEBUG	Add-on change: onInstalled to gmp-gmpopenh264
1538088988964	Sync.AddonsReconciler	DEBUG	Rectifying state for addon OpenH264 Video Codec provided by Cisco Systems, Inc. (version=1.7.1, id=gmp-gmpopenh264)
1538088988964	Sync.AddonsReconciler	DEBUG	Adding change because add-on not present locally: gmp-gmpopenh264
1538088988964	Sync.AddonsReconciler	INFO	Change recorded for gmp-gmpopenh264
1538088988964	Sync.Engine.Addons.Tracker	DEBUG	changeListener invoked: 1 gmp-gmpopenh264
1538088988965	Sync.Engine.Addons.Store	DEBUG	gmp-gmpopenh264 not syncable: type not in whitelist: plugin
1538088988965	Sync.Engine.Addons.Tracker	DEBUG	Ignoring change because add-on isn't syncable: gmp-gmpopenh264
1538088988967	Sync.AddonsReconciler	INFO	Saving reconciler state to file: addonsreconciler
1538088990717	Sync.AddonsReconciler	DEBUG	Add-on change: onUninstalling to gmp-widevinecdm
1538088990718	Sync.AddonsReconciler	DEBUG	Ignoring onUninstalling for restartless add-on.
1538088990719	Sync.AddonsReconciler	DEBUG	Add-on change: onUninstalled to gmp-widevinecdm
1538088990719	Sync.AddonsReconciler	INFO	Saving reconciler state to file: addonsreconciler
1538088990812	Sync.AddonsReconciler	DEBUG	Add-on change: onInstalling to gmp-widevinecdm
1538088990812	Sync.AddonsReconciler	DEBUG	Ignoring onInstalling for restartless add-on.
1538088990813	Sync.AddonsReconciler	DEBUG	Add-on change: onInstalled to gmp-widevinecdm
1538088990813	Sync.AddonsReconciler	DEBUG	Rectifying state for addon Widevine Content Decryption Module provided by Google Inc. (version=1.4.9.1088, id=gmp-widevinecdm)
1538088990813	Sync.AddonsReconciler	DEBUG	Adding change because add-on not present locally: gmp-widevinecdm
1538088990813	Sync.AddonsReconciler	INFO	Change recorded for gmp-widevinecdm
1538088990813	Sync.Engine.Addons.Tracker	DEBUG	changeListener invoked: 1 gmp-widevinecdm
1538088990813	Sync.Engine.Addons.Store	DEBUG	gmp-widevinecdm not syncable: type not in whitelist: plugin
1538088990813	Sync.Engine.Addons.Tracker	DEBUG	Ignoring change because add-on isn't syncable: gmp-widevinecdm
1538088990814	Sync.AddonsReconciler	INFO	Saving reconciler state to file: addonsreconciler
1538088993468	Sync.Service	DEBUG	User-Agent: Firefox/62.0.2 (Windows NT 10.0; Win64; x64) FxSync/1.64.0.20180920131237.desktop
1538088993469	Sync.Service	INFO	Starting sync at 2018-09-27 18:56:33 in browser session i2zhQ2jcLqXu
1538088993469	Sync.Service	DEBUG	In sync: should login.
1538088993470	Sync.Service	INFO	User logged in successfully - verifying login.
1538088993472	Sync.BrowserIDManager	DEBUG	unlockAndVerifyAuthState: user declined to unlock master-password
1538088993472	Sync.Status	DEBUG	Status.login: success.login => service.master_password_locked
1538088993472	Sync.Status	DEBUG	Status.service: success.status_ok => error.login.failed
1538088993472	Sync.Service	DEBUG	Fetching unlocked auth state returned service.master_password_locked
1538088993473	Sync.ErrorHandler	ERROR	Sync encountered a login error
1538088993474	Sync.SyncScheduler	DEBUG	Clearing sync triggers and the global score.
1538088993474	Sync.SyncScheduler	DEBUG	Couldn't log in: master password is locked.
1538088993474	Sync.SyncScheduler	DEBUG	Starting client-initiated backoff. Next sync in 900000 ms.
1538088993474	Sync.SyncScheduler	DEBUG	Next sync in 900000 ms. (why=client-backoff-schedule)
1538088993477	Sync.Service	DEBUG	Exception calling WrappedLock: Error: Login failed: service.master_password_locked (resource://services-sync/service.js:868:15) JS Stack trace: onNotify@service.js:868:15
1538088993477	Sync.Service	DEBUG	Not syncing: login returned false.
Flags: needinfo?(markh)
Do you have a master-password enabled? If so, maybe check the largest log you can find and ensure it doesn't have "service.master_password_locked", then attach it to this bug? I'm really just looking for evidence that the database appears corrupt. Another thing you could consider is taking a copy of your places.sqlite, then deleting it from your profile - it should end up rebuilt and continue to sync, and may solve your problem.
Flags: needinfo?(markh)

Hello! I have tried to reproduce the issue with firefox 97.0a1(2021-12-10) on Windows 10 unfortunately I wasn't able to reproduce the issue. Since there was no activity in the past few years I will mark this issue as RESOLVED->WORKSFORME

If the issue is still valid please feel free to reopen it.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: