Axel Hecht [:Pike] Reporter
	Description • 7 years ago

I've got an offered update for Daily, German, but it fails to apply:

Fuchsia:updates axelhecht$ pwd
/Users/axelhecht/Library/Caches/Thunderbird/updates
Fuchsia:updates axelhecht$ find . -type f
./Applications/Daily/active-update.xml
./Applications/Daily/updates/backup-update.log
./Applications/Daily/updates/last-update.log
./Applications/Daily/updates.xml
Fuchsia:updates axelhecht$ cat ./Applications/Daily/active-update.xml
<updates xmlns="http://www.mozilla.org/2005/app-update"/>Fuchsia:updates axelhecht$ 
Fuchsia:updates axelhecht$ cat ./Applications/Daily/updates/last-update.log
Performing a staged update
PATCH DIRECTORY /Users/axelhecht/Library/Caches/Thunderbird/updates/Applications/Daily/updates/0
INSTALLATION DIRECTORY /Applications/Daily.app
WORKING DIRECTORY /Users/axelhecht/Library/Caches/Thunderbird/updates/Applications/Daily/updates/0/Updated.app
DoUpdate: error extracting manifest file
calling QuitProgressUI

My suspicion is bug 641212, and the lack of a watershed like bug 1385856.

Which will be tough for l10n at least, as the thunderbird l10n nightlies failed for a while.

All that assuming that the lzma packaging change just applied to Thunderbird.

	Tom Prince [:tomprince] Assignee
	Comment 1 • 7 years ago

I think what needs to be done for this is:

0) Disable updates on nightly [DONE] (bhearsum did this https://aus5.mozilla.org/api/v1/releases/Thunderbird-comm-central-nightly-20170730030209)
1) Get xz installed on windows buildbots (probably via tooltool https://bugzilla.mozilla.org/show_bug.cgi?id=641212#c200)
2) Disable xz compression for one nightly build (probably by setting MAR_OLD_FORMAT in mozconfigs)
3) Create a watershed for updating pre-xz versions to the nightly build in (2).

We'll need to do the same for the betas and the releases as well.

Chris, does this seem reasonable?

	Wayne Mery (:wsmwk)
	Comment 2 • 7 years ago

not build config code, but I think it makes sense to file such bugs there so people watching that component are better informed.

	Tom Prince [:tomprince] Assignee
	Comment 3 • 7 years ago

https://github.com/mozilla-releng/ship-it/pull/184 should be merged once updates are turned back on.

	Tom Prince [:tomprince] Assignee
	Comment 5 • 7 years ago

We have proper builds for for the transition. I was planning to create these changes myself, but I don't yet have VPN access for access to balrog.

For all builds with buildid <= 20170731030207 (i.e. with old updater and packed with old format, update to 
Target buildID=20170804030207 (which has new updater but packed with old format). For all builds after 20170731030207 renable updates to latest nightly.

I've manually tested updating with linux x86_64 builds in en-US and fr, but it would probably be sensible to test this on a custom channel first.

	Kim Moir [:kmoir] ET
	Comment 6 • 7 years ago

What custom channel are you looking to test these on?

	Kim Moir [:kmoir] ET
	Comment 7 • 7 years ago

nthomas what channel do we usually use to test these sorts of updates on?  tom is suggesting the nightly channel but not sure that testing a production channel is the usual approach

kmoir: I'm not sure. I was thinking something like https://bugzilla.mozilla.org/show_bug.cgi?id=1366893#c20 . But I guess the failure mode here is that the update fails to apply, so perhaps testing directly on the nightly channel is fine.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 8 • 7 years ago

We'd use nightlytest for that if this was Firefox. Looks like there are no rules set up for Thunderbird nightlytest before now.

	Kim Moir [:kmoir] ET
	Comment 9 • 7 years ago

So I have set up the rules for releng approval on the nightlytest channel

https://aus4-admin.mozilla.org/rules/scheduled_changes

You can see two scheduled changes there for the nightlytest channel

Some notes.  There isn't a 
20170731030207 build in the the nightly database so I used this one instead
Thunderbird-comm-central-nightly-20170730030209

	Tom Prince [:tomprince] Assignee
	Comment 10 • 7 years ago

I see builds at https://archive.mozilla.org/pub/thunderbird/nightly/2017/07/2017-07-31-03-02-07-comm-central/ I'm curious why they aren't showing up in balrog.

	Kim Moir [:kmoir] ET
	Comment 11 • 7 years ago

If you look at the logs, it looks like there was a problem uploading to balrog

see

https://archive.mozilla.org/pub/thunderbird/nightly/2017/07/2017-07-31-03-02-07-comm-central/comm-central-win64-nightly-bm73-build1-build2.txt.gz

Executing: ['python.exe', 'c:/builds/moz2_slave/tb-c-cen-w64-ntly-000000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin.mozilla.org/api', '--username', 'balrog-tbirdbld', '-t', 'nightly', '--verbose', '--url-replacement', 'ftp.mozilla.org,download.cdn.mozilla.net', '--credentials-file', 'BuildSlaves.py']
attempt 1/10
retry: Calling <function update_dated at 0x02CEAB70> with args: (), kwargs: {}, attempt #1
Balrog request to https://aus4-admin.mozilla.org/api/releases/Thunderbird-comm-central-nightly-20170731030207
Data sent: None
Starting new HTTPS connection (1): aus4-admin.mozilla.org
"GET /api/releases/Thunderbird-comm-central-nightly-20170731030207 HTTP/1.1" 200 1750
REQUEST STATS: {"url": "https://aus4-admin.mozilla.org/api/releases/Thunderbird-comm-central-nightly-20170731030207", "timestamp": 1501500421.778, "method": "GET", "elapsed_secs": 2.501000165939331, "status_code": 200}
Balrog request to https://aus4-admin.mozilla.org/api/releases/Thunderbird-comm-central-nightly-20170731030207/builds/WINNT_x86_64-msvc/en-US
Data sent: None
Starting new HTTPS connection (1): aus4-admin.mozilla.org
"GET /api/releases/Thunderbird-comm-central-nightly-20170731030207/builds/WINNT_x86_64-msvc/en-US HTTP/1.1" 404 189
Caught HTTPError: {"status": 404, "type": "about:blank", "detail": "[\"Couldn't find locale identified by: Thunderbird-comm-central-nightly-20170731030207, WINNT_x86_64-msvc, en-US\"]", "title": "Not Found"}
REQUEST STATS: {"url": "https://aus4-admin.mozilla.org/api/releases/Thunderbird-comm-central-nightly-20170731030207/builds/WINNT_x86_64-msvc/en-US", "timestamp": 1501500423.717, "method": "GET", "elapsed_secs": 1.938999891281128, "status_code": 404}
Balrog request to https://aus4-admin.mozilla.org/api/releases/Thunderbird-comm-central-nightly-20170731030207
Data sent: None

	Tom Prince [:tomprince] Assignee
	Comment 12 • 7 years ago

There are now proper builds allowing transition from BZ2/SHA-1 updaters to LZMA/SHA-384 updaters. There are update currently available on the "nightlytest" channel. I've scheduled rule changes for tomorrow morning that will enable updates on nightly channel at a rate of 10%.

	Tom Prince [:tomprince] Assignee
	Comment 13 • 7 years ago

The update rate has been raised to 100% now.