Open Bug 1387485 Opened 7 years ago Updated 1 year ago

L10n repacks don't work on artifact builds

Categories

(Firefox Build System :: General, enhancement)

Product:
Firefox Build System
Component:
General
Type:
enhancement

Tracking

(firefox57 wontfix)

Tracking Flags:
Tracking Status
firefox57 --- wontfix

People

(Reporter: Pike, Unassigned)

References

Details

AFAICT, this is mostly because l10n repacks do all kinds of things in all kinds of ways, and the buildfaster back-end doesn't want to follow that.

I can understand that.

Hoping to make things less crazy in bug 1385227. Still far away, though.
Depends on: 1392256
The first thing that's breaking l10n repacks is that for the faster backend, we don't put the 

   JAR_MANIFEST := .../browser/locales/jar.mn

into backend.mk.

I struggle to understand how jar.mn processing works for faster builds, and how to best integrate this with repacks.

On the symptom side, we could add JAR_MANIFEST to backend.mk inside an ifdef IS_LANGUAGE_REPACK, and regular artifact builds wouldn't see them outside of actual repacks.

But I have no idea if that pill just kills the patient.
The faster make backend expects everything to be set at configure time. The "let's poke at a few make rules overriding some variables" scheme is not and won't be supported. L10n repacks need stop relying on make overrides.
We'll need a way to signal to the build system that we're doing a language pack, as long as the builds we do insist of creating platform-dependent l10n assets. As we're not going to create per-platform language packs.
It seems like it'd be straightforward to add something like `--enable-l10n-repack=ab-cd` and then use that in our repacks.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> It seems like it'd be straightforward to add something like
> `--enable-l10n-repack=ab-cd` and then use that in our repacks.

I'm kind of wanting to avoid a need for a configure run to do a l10n repack, or at least multiple configure runs for each locale switch. I know thats not entirely the case today, but it may preclude being able to do `--enable-l10n-repack=ab-cd`

But I don't yet have the build system improvements in my line of sight here, been focused on getting buildbot out of our way, so we can improve this for all platforms, and releases, all at once without caring as much about backwards compat with old system integrations.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> It seems like it'd be straightforward to add something like
> `--enable-l10n-repack=ab-cd` and then use that in our repacks.

How would that work out for engineers wanting to do a localized version of their own build for testing purposes?

Right now, they'd do ./mach package;./mach build installers-de and they'd have something testable where they expect it.
... worst bug to keep notes, but as we're keeping notes:

We also need to support multi-locale builds on mobile, and also on desktop in the near future.
Depends on: 1413737
No longer depends on: 1413737
Product: Core → Firefox Build System
Just for my own notes, the issue with FasterMake is that it doesn't handle l10n repacks, but it doesn't actively get in the way of the RecursiveMake system.  Yesterday I tried adding

ac_add_options --enable-artifact-builds
ac_add_options --build-backends="RecursiveMake","FasterMake"

to my mozconfig, and my (Android!) things appeared to work.  That combination of build-backends is exactly what non-artifact builds use -- see https://searchfox.org/mozilla-central/rev/8fa0b32c84f924c6809c690117dbd59591f79607/moz.configure#246 -- and that configuration should still support artifact builds.

Of course, since it's not the default, it's not well tested.

Just tried this to confirm my doc updates in bug 1601249, and it seems that this has regressed in the past two years. I didn't invest too much energy into the investigation on what might have regressed this.

Build breaks on requiring buildid.h, and that not being there.

 0:06.10 build/application.ini.stub
 0:06.22 build/automation.py.stub
 0:06.29 Traceback (most recent call last):
 0:06.29   File "/usr/local/Cellar/python@2/2.7.17/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
 0:06.29     "__main__", fname, loader, pkg_name)
 0:06.29   File "/usr/local/Cellar/python@2/2.7.17/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
 0:06.29     exec code in run_globals
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/action/file_generate.py", line 121, in <module>
 0:06.29     sys.exit(main(sys.argv[1:]))
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main
 0:06.29     ret = module.__dict__[method](output, *args.additional_arguments, **kwargs)
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/action/preprocessor.py", line 15, in generate
 0:06.29     pp.handleCommandLine(list(args), True)
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/preprocessor.py", line 509, in handleCommandLine
 0:06.29     self.processFile(input=input, output=out)
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/preprocessor.py", line 420, in processFile
 0:06.29     self.do_include(input, False)
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/preprocessor.py", line 834, in do_include
 0:06.29     self.handleLine(l)
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/preprocessor.py", line 583, in handleLine
 0:06.29     cmd(args)
 0:06.29   File "/src/central/mozilla-central/python/mozbuild/mozbuild/preprocessor.py", line 807, in do_include
 0:06.29     raise Preprocessor.Error(self, 'FILE_NOT_FOUND', str(args))
 0:06.29 mozbuild.preprocessor.Error: ('$SRCDIR/build/application.ini.in', 16, 'FILE_NOT_FOUND', '/src/central/mozilla-central/obj-artifact-firefox/buildid.h')
 0:06.30 make[4]: *** [.deps/application.ini.stub] Error 1
 0:06.30 make[4]: *** Waiting for unfinished jobs....

Just realized that with bug 1641184 being fixed, we might be able to get the build-faster backend to support l10n paths. It's just l10n or en-US now, instead of a fallback logic. So at the time that you know the locale, you can tell the full dependencies.

(In reply to Axel Hecht [:Pike] from comment #10)

Just realized that with bug 1641184 being fixed, we might be able to get the build-faster backend to support l10n paths. It's just l10n or en-US now, instead of a fallback logic. So at the time that you know the locale, you can tell the full dependencies.

I think this is probably true for the things declared in moz.build, but -- IIRC -- the FasterMake backend doesn't handle jar.mn files in a locale-aware manner at all: it packages a bunch of copies (links, really) into install manifests, which are determined at build-backend time. If the build did know the entire set of locales at build time (see Bug 1465901 and related tickets) this could be made to work, but right now it the build doesn't know enough to do that.

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.