Jitsi issue https://github.com/jitsi/jitsi-meet/issues/2835 depends on this one

I'm interested in finding out when this might get fixed.

This here is required for Jitsi Meet to work properly.
As such, as you had some whiteboard for [jitsi-meet], please assign that to this. (It is not, currently.)

I've involved in the discussion about how to implement it. I am taking the bug for now. Perhaps we can split this goal into several smaller tasks so it will be clearer what we are going to do.

(In reply to C.M.Chang[:chunmin] from comment #14)

I've involved in the discussion about how to implement it. I am taking the bug for now. Perhaps we can split this goal into several smaller tasks so it will be clearer what we are going to do.

Thank you!

Brief update from the discussion: Now our architecture (simplified version) is something like this (for microphone device). The goal is to implement this design.

Why many local audio streams can't be merged into a single stream ? Maybe this way could spare the bandwidth ? Even if you keep 2 channels (for stereo) into the single stream, would it allow the codec to achieve a few optimizations ?

(In reply to Jérôme from comment #18)

Why many local audio streams can't be merged into a single stream ?

They might not come from the same physical device, and so will drift with respect to each other. This is the main complexity in this bug, keep the low-latency, but reclock audio input devices.

To add a non-native input source in the MediaTrackGraph, we need a
input-only cubeb stream.

Depends on D116534

Add a AudioInputStream class that will generate audio input data from
one thread and it allows its user to read the data from another thread.

The following patches will add an abstract MediaTrack for the audio
input devices. The NativeInputTrack will be one implementation of it.
These MediaTracks holding the audio input data should be put together so
this patch moves the NativeInputTrack into individual .h and .cpp files,
which will also be the places implementing this new type of MediaTrack.

This patch addes a new DeviceInputTrack class, which is also the base
class for our NativeInputTrack class. The AudioInputTrack's
data-producer should always be DeviceInputTrack. We will implement
another subclass of DeviceInputTrack for non-native input device in the
following patches. As a result, the AudioInputTrack has no need to care
whether its requested device source is native or non-native.

This patch creates a new class, DeviceInputTrackManager, to manage all
the DeviceInputTrack. It helps to lower the complexity of
MediaTrackGraph and it is the only way to get access to the
DeviceInputTrack.

Depends on D118552

By the comment in AudioDriftCorrection: "The construction can happen
in any thread", the AudioDriftCorrection class is ought to be created
in any thread, but it's not true actually. The fact that ctor calls
Preferences::GetInt makes it can only be called on main-thread, which
is the thread Preferences::GetInt should be.

To make AudioDriftCorrection work as what it's expected, the
preference getter should be moved to the AudioDriftCorrection's caller
itself.

InitDataHolderIfNeeded is used to initialize an AudioDataBuffers but
there is no memory needed to be allocated and there is no need to wrap
the AudioDataBuffers by a Maybe.

Depends on D118552

AudioBufferInfo::Scope::All is a shortcut version of
static_cast<AudioBufferInfo::Scope>(AudioBufferInfo::Scope::Input | AudioBufferInfo::Scope::Output).

Depends on D122513

D116534 is probably solved by D122338 in bug 1725137

Add more debugging log showing device information

Depends on D118862

The patches can be tested here: https://chunminchang.github.io/playground/webrtc/multi_microphones.html.

The test page can open multiple AudioInputTrack with different devices, and the channel count can be reset anytime. The audio-processing including autoGainControl, echoCancellation and noiseSuppression can be set when the track are opened. Those audio-processing settings are used to set the AudioInputTrack to passthrough mode. It will lead to a different path when ProcessInput is called.

This comment should be deleted in Bug 1725810.

The NativeInputTrack should only care about input data. Other
non-data related members should be moved out of the class. This makes
NativeInputTrack become a simple class focusing on input data storing.

Depends on D122513

The NativeInputTrack should be the only track that can open or close the
audio stream.

Depends on D130142

Add a simple test demonstrating the usage of NativeInputTrack

Depends on D118242

This patch implements a NonNativeInputTrack class that inherits the
DeviceInputTrack class. The audio data in this class instance is
produced by a non-graph audio stream via NotifyInputData(), and is
consumed by the graph thread via ProcessInput().

Depends on D122297

Comment on attachment 9244448 [details]
Bug 1238038 - Remove useless comment

Revision D127605 was moved to bug 1741959. Setting attachment 9244448 [details] to obsolete.

Comment on attachment 9241255 [details]
Bug 1238038 - Add debugging logs

Revision D125626 was moved to bug 1741959. Setting attachment 9241255 [details] to obsolete.

Comment on attachment 9227313 [details]
Bug 1238038 - Move NativeInputTrack to individual files

Revision D117965 was moved to bug 1741959. Setting attachment 9227313 [details] to obsolete.

Comment on attachment 9236045 [details]
Bug 1238038 - Move audio data processing to ProcessInput

Revision D122513 was moved to bug 1741959. Setting attachment 9236045 [details] to obsolete.

Comment on attachment 9248980 [details]
Bug 1238038 - Add test for NativeInputTrack

Revision D130232 was moved to bug 1741959. Setting attachment 9248980 [details] to obsolete.

Comment on attachment 9235703 [details]
Bug 1238038 - Relax AudioDriftCorrection's thread limitation

Revision D122297 was moved to bug 1746893. Setting attachment 9235703 [details] to obsolete.

Separate common interfaces that will be used for NonNativeInputTrack
class, from NativeInputTrack, into a base class: DeviceInputTrack, where
the NonNativeInputTrack will be implemented in the next patch.

Depends on D137781

Depends on D130233

Depends on D116535

Comment on attachment 9262249 [details]
WIP: Bug 1238038 - Fix comment for include guard

Revision D137817 was moved to bug 1754250. Setting attachment 9262249 [details] to obsolete.

Comment on attachment 9262163 [details]
WIP: Bug 1238038 - Narrower down the thread limits

Revision D137781 was moved to bug 1754254. Setting attachment 9262163 [details] to obsolete.

Depends on D137911

Depends on D116535

Depends on D138189

This patch creates a class to manage all DeviceInputTracks in one place,
which simplify the code in the MediaTrackGraph.

Depends on D138726

The mInputDeviceID can be removed now since we can get the input-device
id via the DeviceInputTrackManager on the graph-thread side now.

Depends on D139444

Depends on D137910

Add a waitable event that will be fired when the state is changed, so
the user can wait for certain state of the cubeb stream.

Depends on D143100

Add a test to verify we can open multiple microphones

Depends on D139445

https://hg.mozilla.org/mozilla-central/rev/ed855828ee19
https://hg.mozilla.org/mozilla-central/rev/cee7aa3686a2
https://hg.mozilla.org/mozilla-central/rev/3e81a5bebba2
https://hg.mozilla.org/mozilla-central/rev/f2c23215c842
https://hg.mozilla.org/mozilla-central/rev/28f666109dfa
https://hg.mozilla.org/mozilla-central/rev/eff158481883
https://hg.mozilla.org/mozilla-central/rev/b66220158398
https://hg.mozilla.org/mozilla-central/rev/80575371d79f
https://hg.mozilla.org/mozilla-central/rev/8f7feefeda0c
https://hg.mozilla.org/mozilla-central/rev/a1e8aebd2e88
https://hg.mozilla.org/mozilla-central/rev/eb898ad02abd
https://hg.mozilla.org/mozilla-central/rev/2b2de9f71e7f

This should be for bug 1731141. The patches here remove the code that causes the crash.

Release Note Request (optional, but appreciated)
[Why is this notable]:
Audio input support improvements for web conferencing.
[Affects Firefox for Android]:
Yes
[Suggested wording]:
Firefox now allows users to use as many microphones as you want at the same time during video conferencing! The most important benefit is that you can easily switch your microphones at any time, if your conferencing service provider enables this flexibility.
[Links (documentation, blog post, etc)]:

jitsi enables this in PR 1988 🎉

Added to the final Fx101 relnotes.