From Gnash Project Wiki

Jump to: navigation, search


GStreamer Camera Implementation

Prior to Summerbash the Camera ActionScript class was entirely unimplemented. An intern, Ben Limmer (l1m5), worked on a basic GStreamer implementation for the class. The code is fairly well documented in Doxygen format, however a bit more explanation will be provided on this page. It is also important to note that this code was partially adapted from another free and open source project, Cheese. This information is up-to-date as of trunk revision number 11412.

Users: If you're trying to figure out how to properly set the Webcam device in the gnashrc file, please run the findwebcams program in the utilities subdirectory. This will tell you the line you need to add to properly select your device.

Pipeline Design

Gstreamer uses the concept of 'pipelines' to control video coming through the device attached to the machine. This concept can be a bit confusing at first, and I recommend reading documentation about programming in Gstreamer before attacking anything Gstreamer based.

If you don't want to read the Gstreamer documentation, here's the gist:

  • GStreamer pipelines are the main workhorse of the implementation. You drop bins into the pipeline and when you start up the pipeline, all the elements inside the pipeline are put into motion. This is convenient for stopping and starting large amounts of complex elements.
  • You drop bins into pipelines. Bins contain elements which are like the building blocks in GStreamer. So the heierarchy is something like elements->bins->pipelines.
  • Elements are the small building blocks. For example, the raw webcam source is an element (in my implementation usually a v4l2src).
  • Bins can have 'ghostpads' attached to them. A ghostpad can either be active or not active (which allows making and breaking links depending on the use of the application or the user's decision).
  • Take some time to go through the code to get a feel for Gstreamer's syntax.

If you're a visual person like I am, download this image. It's a visual representation of the pipeline utilized in the VideoInputGst.cpp source file.

In this image boxes with angular ends represent elements, large boxes represent bins and small boxes on the ends of bins represent ghostpads. Solid arrows represent links that are always present and dotted arrows represent links that can be optionally made.

  • Video pipeline. It contains three bins:
    • Main Bin -- contains source bin and elements to be optionally connected to save or display bin
    • Save Bin -- contains and encoder and muxer to save the video input to an *.ogg file.
    • Display Bin -- contains a display sink to playback the video onscreen as it's being recorded.
  • The video pipeline is what's started to begin recording video.
  • Main bin. It contains one sub-bin:
    • Source Bin -- contains the information to capture from the selected video input source, and run through a capabilities filter which keeps the data from a multitude of sources consistent.
    • Main Bin -- The source bin is always linked into a tee element in the main bin which splits the video stream into two unique streams. These streams are then piped into two queues with ghostpads (called the display queue and the save queue). Since these two elements have ghostpads on them, this is where links are made and broken depending on what the use of the application is. These queue are linked to either the display bin, the save bin or both.
  • Save Bin.
    • The save bin has the elements that write the data out to an *.ogg file. It has a ghostpad before which can be optionally connected to the save queue in the main bin. If you call the makeVideoSourceSaveLink function, the link between the main bin and save bin is made. Inside the bin there are several elements.
      • Colorspace converter -- this element helps to convert the raw video coming from the camera into a colorspace which can be encoded later
      • Rate control -- this element forces the video stream to maintain a life-like rate. Without this element, the video can speed up or slow down on the fly and produce strange looking video.
      • Video scale -- this element scales down the video to the resolution specified when constructing the pipeline.
      • Theora encoder -- this element takes the now smoothed video stream and encodes it in a theora format (prepares for muxing).
      • Ogg muxer -- this element takes the encoded data and prepares it to be written to an *.ogg file
      • File sink -- this element saves the data from the muxer and writes it out to disk.
  • Display Bin.
    • The display bin has the elements that playback the video onscreen. It currently uses the autovideosink element which allows the video to be displayed onscreen. This was mainly used for testing, and it is likely that as further implementation happens, the display bin will need to be refactored to work properly with the flash stage.


  • For information about internal functions and how they work, please generate the doxygen documentation from the libmedia/gst/VideoInputGst.h file.

Tips & Tricks

Here are a few things I learned about Gstreamer which might help GStreamer developers in the future.

  • Test things with gst-launch. Here are some examples of using the gst-launch command line. I found that when designing a pipeline, testing it standalone was always a good idea before taking the time to write the code. For example let's say I wanted to test my video display connections to make sure they would work. I could type the following command into the terminal window:
gst-launch videotestsrc ! videoscale ! autovideosink

As you can see you type gst-launch and then the elements you want to have (they will automatically link) separated by exclamation points. I ran into several weird problems while coding up this implementation of the Camera class that I easily found out were bugs in GStreamer, not in my code, by using this trick.

  • Easily find information about any and all elements in GStreamer. The terminal commmand "gst-inspect" is your friend. For example if you type "gst-inspect" you will get a list of all elements installed on your machine, based on the plugins you selected when installing GStreamer. You can also type things like "gst-inspect v4l2src" which will list useful information about the v4l2src (video4linux2) source element which can help you debug your pipeline design.