Closed Bug 973274 Opened 10 years ago Closed 9 years ago

Install GStreamer 1.x on linux build slaves

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
All
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: alessandro.d, Assigned: rail)

References

Details

Attachments

(2 files)

gstreamer-headers.tar.xz
236.75 KB, application/octet-stream
Details
gstreamer-tooltool-manifest.diff
1.58 KB, patch
Details | Diff | Splinter Review 
gstreamer 1.x support has recently landed in m-c. Currently there's no way to test it on try as the build slaves don't have gstreamer 1.x packages. gstreamer 0.10.x and 1.x can be installed side to side, so that shouldn't be a problem.

The packages needed would be: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-bad, gst-libav. The latest gst version is 1.2.2, but any 1.x should be enough.
Component: Buildduty → Platform Support
QA Contact: armenzg → coop
Ping? Getting this fixed would make me this happy: http://imgur.com/p9CH1
This could be a bit problematic, since all of our linux slaves are a bit... "stable"...

AFAICT, CentOS 6 (as on the build slaves) still only has the ancient 0.10.29 in the latest version.
Ubuntu 12.04 (as on the test slaves) is still on 0.10.36. The more recent Ubuntu 13.04 has 1.x.

Nick, is there a plan to upgrade these any time soon? The test slaves at least -- missing headers at build time can be worked around. (I suspect the answer starts with "if it ain't broke...")
Failing that, is there something we can do for not-in-repo packages? (I suspect the answer ends with "...dependency hell")
No longer blocks: 947287
Flags: needinfo?(nthomas)
Summary: Install GStreamer 1.x on linux build machines → Install GStreamer 1.x on linux build and test slaves
I defer to coop on the scheduling question. It would probably help him if you were to describe the pros/cons of this upgrade, and what is unblocked by it.

Others in RelEng have experience with compiling packages where the OS repos don't have recent enough versions.
Flags: needinfo?(nthomas) → needinfo?(coop)
https://bugzilla.mozilla.org/show_bug.cgi?id=851290 anyone? :P (that bug has gst 1.2.2 integrated in gecko-dev).

Anyway, the gstreamer modules used by the ffox backend don't have that many dependencies. We would have to recompile liborc, glib, gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-bad, and gst-libav.
If a side-by-side install is indeed possible as suggested in comment #0, this will be less work.

Alessandro: would you be willing to take a loaner slave (build first, then test) to investigate the upgrade yourself? Keep notes of packages you need to install and env changes made and then we can hopefully get those added to puppet more easily.

I'm down two people in platform support at the moment, so that would be really helpful since you seem to have incentive here.

https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_a_slave
Flags: needinfo?(coop) → needinfo?(alessandro.d)
Alessandro: ping re: comment #5?
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → INCOMPLETE
Is there any plan to update the test slaves to a more recent distro?
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Can you install gstreamer-1.0 from the gstreamer PPA?

https://launchpad.net/~gstreamer-developers/+archive/ubuntu/ppa
Flags: needinfo?(coop)
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #7)
> Is there any plan to update the test slaves to a more recent distro?

Right now, no. That's a quarter-long project for multiple teams: releng, relops, ateam. 

Not saying it's not needed, or it won't be done, but there's no plan to bootstrap a new linux testing platform before 2015.

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #8)
> Can you install gstreamer-1.0 from the gstreamer PPA?
> 
> https://launchpad.net/~gstreamer-developers/+archive/ubuntu/ppa

It's not a simple question of installing a package and walking away. 

We need developer help to make sure the new package is working as intended for (at the very least) the builds you are concerned about. Releng can then run that modified configuration against other builds and/or tests to make sure it doesn't regress something elsewhere.

I offered a loaner slave to Alessandro in comment #5 but never heard back.
Flags: needinfo?(coop)
(In reply to Chris Cooper [:coop] from comment #9)
> (In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #7)
> > Is there any plan to update the test slaves to a more recent distro?
> 
> Right now, no. That's a quarter-long project for multiple teams: releng,
> relops, ateam. 

I suggest you schedule some time in Q1 2016.

> (In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #8)
> It's not a simple question of installing a package and walking away. 
> 
> We need developer help to make sure the new package is working as intended
> for (at the very least) the builds you are concerned about. Releng can then
> run that modified configuration against other builds and/or tests to make
> sure it doesn't regress something elsewhere.

gstreamer-1.x can be installed alongside gstreamer-0.10. The code in gecko is separate for each API version. Right now most of the distros ship 1.0 but we don't test the 1.0 code path at all.

> I offered a loaner slave to Alessandro in comment #5 but never heard back.

He has given up a lot of his own time to help us.
Flags: needinfo?(alessandro.d)
Does that mean gstreamer 1.0 won't be enabled by default before 2016?
In the last year mozilla has gone from being very interested in GStreamer and wanting to use it as the media abstraction for at least linux and mac, to wanting to replace GStreamer with stagefright in linux. All this without involving me at all in the decisions, which I guess is fair enough, but yeah forgive me if I've lost interest a bit :)

That said, i'm willing to help to setup 1.x on linux slaves. I just had a chat with simone in #developers who explained me a little how the slaves work. If someone can give me access to a centos 6.2 slave, I can build binaries for gst and deps. Then I would need some assistance to setup tooltool/mock envs to use them. How do I get ssh access?
(In reply to Alessandro Decina from comment #12)
> In the last year mozilla has gone from being very interested in GStreamer
> and wanting to use it as the media abstraction for at least linux and mac,
> to wanting to replace GStreamer with stagefright in linux. All this without
> involving me at all in the decisions, which I guess is fair enough, but yeah
> forgive me if I've lost interest a bit :)
> 
> That said, i'm willing to help to setup 1.x on linux slaves. I just had a
> chat with simone in #developers who explained me a little how the slaves
> work. If someone can give me access to a centos 6.2 slave, I can build
> binaries for gst and deps. Then I would need some assistance to setup
> tooltool/mock envs to use them. How do I get ssh access?

https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_a_slave
Any updates on this?  Is anyone currently working on a slave?  I'm guessing the gtk3 port would also like a more recent platform to build on.. 

I'm happy to help with an Ubuntu 12.04 slave (PPA option) or even setting up a new 14.04 slave if that's what we need to do..
(In reply to Bryan Quigley from comment #14)
> Any updates on this?  Is anyone currently working on a slave?  I'm guessing
> the gtk3 port would also like a more recent platform to build on.. 
> 
> I'm happy to help with an Ubuntu 12.04 slave (PPA option) or even setting up
> a new 14.04 slave if that's what we need to do..

I'm not aware of anyone working on this.
Depends on: 1122859
Bryan was able to create builds with gstreamer1: https://bugzilla.mozilla.org/show_bug.cgi?id=1122859#c9

There's still a lot of packaging work to do in order to clean up the install process. If the system package upgrades are required, we'll also need to validate other linux builds to make sure they still work.
I'll look at this bug this week.
Assignee: nobody → rail
I just looked at the PPA for packages we'd need https://launchpad.net/~gstreamer-developers/+archive/ubuntu/ppa/+packages?field.name_filter=&field.status_filter=published&field.series_filter=precise

3 of them are broken at the moment.

gst-plugins-{good,bad} require libvpx-dev (>= 1.1.0) which requires gcc-4.8 which not a thing in 12.04. I can try to use at https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test/+packages?field.name_filter=&field.status_filter=published&field.series_filter=precise, but it may take some time.

I haven't looked at the build side yet.
The dependency on gcc-4.8 was added to fix some solaris issues. I downgraded the requirement and libvpx built normally.

Then I found that gst-plugins-good1.0 requires libpulse 2.0+, but the default one for 12.04 is 1.1.

It sounds like it may be easier just to upgrade the test machines :)

$ ls *.deb | wc -l
86

I'll poke this again tomorrow or next week.
Also that PPA is abandoned, if anybody wants to take that over let me know. I'm not using Ubuntu anymore since many years, and once they started diverging too much from the Debian packages (and upstream tarballs...) I stopped updating it as I couldn't test the things anymore without installing Ubuntu anywhere.

Just upgrade to a less ancient Ubuntu version, or build the required packages manually :)
Upgrading gstreamer will upgrade a lot of system libraries required by it. This is almost similar to upgrading the reference platform to a fresher Ubuntu version, what would cause a fair amount of work.

I briefly talked to coop and jmaher about upgrading the reference platform and one of the ideas was upgrade and switch to docker images at the same time.

Per IRC chat with arr, we may start the upgrade in 2015q2.

I'll update the dependency when we have bugs on file.
Can we just install the gstreamer-1.0 headers so we can compile against them?
What about tests?
(In reply to Rail Aliiev [:rail] from comment #23)
> What about tests?

We aren't running any now.

We're not testing 1.0 now and it is a different code path from 0.10. We need to drop support for 0.10 and most of the Linux distros use 1.0 now anyway. Tests starting in 2015q2, I guess.
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #22)
> Can we just install the gstreamer-1.0 headers so we can compile against them?

This is the easiest thing we can do! :) I have a couple of follow up questions.

1) Do we need any libraries as well?

2) How to specify what headers to use? (parameters for mozconfig)

3) How to verify builds?
(In reply to Rail Aliiev [:rail] from comment #25)
> This is the easiest thing we can do! :) I have a couple of follow up
> questions.

It is far from ideal, but it is better than nothing.

> 1) Do we need any libraries as well?

I don't believe so.

> 2) How to specify what headers to use? (parameters for mozconfig)

I don't think we have a mechanism for that so we'll have to add one.

> 3) How to verify builds?

Gstreamer is dynamically linked if available. It just won't be available. We're already not testing gstreamer-1.0 so we won't be any worse off.
One more question: do we care about the *.pc files or having just the headers is enough?
I created a script which uses docker to create a tarball with headers from gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-libav packages (1.4.5):

https://github.com/rail/gstreamer-docker-builder

See attached tarball with generated headers.

If the attached headers are enough to generate a build, I can upload it to tooltool and we can use it on try to play with actual builds.
Ralph - can you see if these headers are sufficient?
Flags: needinfo?(giles)
Do you need anything else from me or can I return the build slave (1122859)?  Thanks!
(In reply to Bryan Quigley from comment #30)
> Do you need anything else from me or can I return the build slave (1122859)?
> Thanks!

I think you can return it. If the headers look OK, we can continue using try.
...And  thanks for your help!
Can anyone else verify that the attached archive with gstreamer headers is sufficient?
It does include all the files we include directly, so this should work fine.
Flags: needinfo?(giles)
Great! I'll upload the files and add some information on how to find them as a part of a build tomorrow.
If you apply this patch to your m-c tree, then you can access the headers in  $topsrcdir (for example $topsrcdir/gstreamer-1.0/gst/gstbuffer.h) in your mozconfigs.

Let me know if you need to add an extra directory to the file hierarchy in the archive to avoid header collisions using $topsrcdir. Maybe to something like $topsrcdir/gstreamer-headers/gstreamer-1.0/gst/gstbuffer.h.
Morphing the bug into build-only. The test part depends on OS upgrade. From what I've heard this can happen later this year.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 9 years ago
Resolution: --- → FIXED
Summary: Install GStreamer 1.x on linux build and test slaves → Install GStreamer 1.x on linux build slaves
Is there any ETA when this can actually happen?
The headers are available already, see comment #36. I'm not sure if there is a plan to switch to using them though.
Who would know about it? Without this switch, Firefox will be built against obsolete gstreamer. Practically all distros switched to 1.x and many already don't ship 0.10 at all.
(In reply to Shmerl from comment #40)
> Who would know about it? Without this switch, Firefox will be built against
> obsolete gstreamer. Practically all distros switched to 1.x and many already
> don't ship 0.10 at all.

I think you might want to ask in bug 947287 instead, considering that one is marked as closed.
(In reply to AnAkkk from comment #41)
> I think you might want to ask in bug 947287 instead, considering that one is
> marked as closed.

Well, bug 947287 references this one as a prerequisite (since it's about installing gstreamer 1.x on Linux build machines). But since this one is closed, there is not tracking point for this dependency. This bug really should not be closed yet, since it didn't happen.
Or may be there is some separate bug for the test part. Then it should be added as a dependency for #947287.
Depends on: 1212916
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: