MicrophoneGST

From Gnash Project Wiki

Jump to: navigation, search

Contents

GStreamer Microphone Implementation

Prior to Summerbash the Microphone ActionScript class was entirely unimplemented. An intern, Ben Limmer (l1m5), worked on a basic GStreamer implementation for the class. The code is fairly well documented in Doxygen format, however a bit more explanation will be provided on this page. It is also important to note that this code was partially adapted from another free and open source project, Cheese. This information is up-to-date as of trunk revision number 11412.

Users: If you're trying to figure out how to properly set the Microphone device in the gnashrc file, please run the findmicrophones program in the utilities subdirectory. This will tell you the line you need to add to properly select your device.

Pipeline Design

Gstreamer uses the concept of 'pipelines' to control audio coming through the device attached to the machine. This concept can be a bit confusing at first, and I recommend reading documentation about programming in Gstreamer before attacking anything Gstreamer based.

If you don't want to read the Gstreamer documentation, here's the gist:

  • GStreamer pipelines are the main workhorse of the implementation. You drop bins into the pipeline and when you start up the pipeline, all the elements inside the pipeline are put into motion. This is convenient for stopping and starting large amounts of complex elements.
  • You drop bins into pipelines. Bins contain elements which are like the building blocks in GStreamer. So the heierarchy is something like elements->bins->pipelines.
  • Elements are the small building blocks. For example, the raw microphone source is an element (in my implementation a pulsesrc).
  • Bins can have 'ghostpads' attached to them. A ghostpad can either be active or not active (which allows making and breaking links depending on the use of the application or the user's decision).
  • Take some time to go through the code to get a feel for Gstreamer's syntax.

If you're a visual person like I am, download this image. It's a visual representation of the pipeline utilized in the AudioInputGst.cpp source file.

In this image boxes with angular ends represent elements, large boxes represent bins and small boxes on the ends of bins represent ghostpads. Solid arrows represent links that are always present and dotted arrows represent links that can be optionally made.

  • Audio pipeline. It contains three bins:
    • Main Bin -- contains source bin and elements to be optionally connected to save or playback bin
    • Save Bin -- contains and encoder and muxer to save the audio input to an *.ogg file.
    • Display Bin -- contains a playback sink to playback the audio as it's being recorded.
  • The audio pipeline is what's started to begin recording audio.
  • Main bin. It contains one sub-bin:
    • Source Bin -- contains the information to capture from the selected audio input source, and run through a capabilities filter which keeps the data from a multitude of sources consistent. The data then runs through an rgvolume element which helps control the amount of gain given to the device as specified by the ActionScript call to Microphone.
    • Main Bin -- The source bin is always linked into a tee element in the main bin which splits the audio stream into two unique streams. These streams are then piped into two queues with ghostpads (called the playback queue and the save queue). Since these two elements have ghostpads on them, this is where links are made and broken depending on what the use of the application is. These queue are linked to either the playback bin, the save bin or both (both described below).
  • Save Bin.
    • The save bin has the elements that write the data out to an *.ogg file. It has a ghostpad before which can be optionally connected to the save queue in the main bin. If you call the makeAudioSourceSaveLink function, the link between the main bin and save bin is made. Inside the bin there are several elements.
      • Audio Convert -- converts the raw audio data into something that can be encoded by the next element.
      • Vorbis Encoder -- this element processes the data into something that the muxer can write out to a file.
      • Ogg muxer -- this element takes the encoded data and prepares it to be written to an *.ogg file
      • File sink -- this element saves the data from the muxer and writes it out to disk.
  • Playback bin
    • The playback bin has the elements that playback the audio being recorded. It currently uses the autoaudiosink element which allows the audio to be played back. This was mainly used for testing, and it is likely that as further implementation happens, the playback bin will need to be refactored to work properly with the flash stage.

Functions

  • For information about internal functions and how they work, please generate the doxygen documentation from the libmedia/gst/AudioInputGst.h file.

Known Issues

  • As of the latest gstreamer stable release (0.10), there is a known issue which affects audio playback and saving when the audio pipeline is started and stopped once. So if you, for example, make the connection between the save and the playback bins and start the pipeline, everything works as it should. But, if you stop the pipeline and then start it back up again, no audio is played back or recorded. This bug is documented here and has not yet been addressed by GStreamer developers.

Tips & Tricks

Here are a few things I learned about Gstreamer which might help GStreamer developers in the future.

  • Test things with gst-launch. Here are some examples of using the gst-launch command line. I found that when designing a pipeline, testing it standalone was always a good idea before taking the time to write the code. For example let's say I wanted to test my video display connections to make sure they would work. I could type the following command into the terminal window:
gst-launch audiotestsrc ! autovideosink

As you can see you type gst-launch and then the elements you want to have (they will automatically link) separated by exclamation points. I ran into several weird problems while coding up this implementation of the Microphone class that I easily found out were bugs in GStreamer, not in my code, by using this trick.

  • Easily find information about any and all elements in GStreamer. The terminal commmand "gst-inspect" is your friend. For example if you type "gst-inspect" you will get a list of all elements installed on your machine, based on the plugins you selected when installing GStreamer. You can also type things like "gst-inspect pulsesrc" which will list useful information about the pulsesrc (pulse audio) source element which can help you debug your pipeline design.