Libmedia

From Gnash Project Wiki

Jump to: navigation, search

This page describes the media lib which gnash will shortly use to decode and play audio and video data.

Contents

Goals

The current code for decoding and playing audio/video data is at the moment in libbase, backend and server/asobj folders, but this new design will move all that code to a new folder called libmedia, and much of the code will be restructured to make it more modular and easier to understand, maintain and extend.

The current way to choose the media handler will also be abandoned in favor of a new plugin system where it can be decided at runtime what media handler should be used. The 2 media handlers which will be used are SDL and Gstreamer. Read more about this in a email posted to the gnash-dev mailing list some time ago: http://article.gmane.org/gmane.comp.gnu.gnash.devel2/2755.

Design

One of the most important things to get right is what "the buffer" is. In AS2 there is a NetStream.setBufferTime(<number>) method, and a readonly NetStream.bufferTime property, and this is defined as the amount of data (in seconds) to load before playbacks should start. The property NetStream.bufferLenght is the amount of data currently buffered, in seconds.

Whether the data in the buffer is decoded or not is not known, but it would make it easier to just grab a ready video or audio frame from the buffer, though it could mean using a lot of memory if the buffer is set to be long/big.

Besides the buffer mentioned above, the media file is downloaded to the a local storage as fast as possible, unless it is a rtmp streamed video, then (I assume) it is downloaded as needed to fill the buffer mentioned above.

For now the new design will assume that the above is what happens, and that the content of the buffer is decoded media. The current NetStream{Ffmpeg,Gst} and Sound{Gst,Ffmpeg,Mad} classes will be dropped, and the use of LoadThread will also be removed. Instead the NetStream and Sound class will be using a decoder class from libmedia which will be using FFMPEG/Mad or GST (depending on runtime parameters) to fill the buffer. The buffer will be a separate class consisting of two queues (as in the current implementation): one for audio, and one for video.

The buffer will be owned by the NetStream/Sound obejct. To make sure the buffer is filled we will need a thread to decode the media data as needed. We also need to download ("warm up") the media file (for non-rmtp video), and so far it seems this can be handled in one thread.

To get the audio played callbacks will be used, like it is already done in Ffmpeg, but this is something new for Gst. This should (hopefully) make it easier to control volume of entire MovieClips

The Decoders

There will (for now) be 2 decoders: MediaDecoderSdl and MediaDecoderGst. These decoders are based on the virtual class MediaDecoder. When decoding FLVs we will use the FLVParser (which will also be moved to libmedia) to parse the file into data packets, but for everything else they will use internal parsers.

We need to be able to handle event stuff, so we store the events that need to be handled inside the decoder, and then NetStream/Sound can ask for it on ::advance() or whenever it is appropriate, and notify the eventhandlers. Event handling is different from AS2 to AS3, so that is also needed to be taken into account.

MediaDecoderSdl

This decoder aims to be very modular so that many different libs can be used for decoding and handling the media data. When parsing we can either use FLVParser (for FLV), Mp3ParserMad (for mp3, if ffmpeg isn't available) or MediaParserFfmpeg (for all non-FLV), which will all have the virtual class as parent. For decoding audio we have the classes AudioDecoderMad, AudioDecoderFfmpeg, based on the virtual class AudioDecoder, and for decoding video we have VideoDecoderFfmpeg, with the parent class VideoDecoder. The AudioDecoder and VideoDecoder classes can also be used to decode embedded audio and video, reducing duplicated code.


MediaDecoderGst

Because Gst is build with pipelines, and not libs, MediaDecoderGst can't be modular in the same way MediaDecoderSdl is. For FLV video we will use the FLVParser, and feed the data into the pipeline using appsrc, and extract it with appsink. When decoding non-FLVs the gstgnashsrc element (from libbase, will be moved to libmedia) will be used to feed the Gstreamer decoding pipeline, but we will still use appsink to extract the data (if possible). appsrc and appsink is fairly new additions to the Gstreamer distribution, so as an option we will distribute it with gnash for optional compilation.


Other elements

NetConnection

To connect to the media file we use the NetConnection object, which creates the connection used. In the current implementation it uses LoadThread to get the information it is required to know.

FLVParser

FLVParser parses an FLV (surprise!!). In the current implementation it uses LoadThread to get data and information needed for parsing. Since LoadThread also "warms up" the file being downloaded using a thread, it could lead to noticeable periods of waiting for threads. In the new design one thread will be doing the downloading, parsing and decoding (for now anyway), which would improve on this. Since the new design aims for modularity, more parser classes will be made, and therefore the top-class MediaParser has been created, and FLVParser will be a sub class of this.

SoundHandlers

The soundhandlers will also be moved to libmedia, and will (where possible) use the new decoding classes. This will be easy for the SDL soundhandler, but for the GST soundhandler this might not be possible at all.

Current status of the new design/implementation

Libmedia is in the source tree now, and AudioDecoders are used by the SDL soundhandler for sound decoding, and when decoding embedded video frames (both Ffmpeg and GST) VideoDecoders are used.

A patch has been uploaded to the tracker which enables the use libmedia from Sound and NetStream.

A MediaDecoder class has been constructed, and subclass MediaDecoderSdl. A MediaBuffer class containing an audio and video queue, has also be created. Accessing the MediaBuffer is mutex protected.

So far playback for FLVs is now working for MediaDecoderSdl, using the new classes MediaParser, AudioDecoder and VideoDecoder, but non-FLV playback is not tested yet.

Work on MediaDecoderGst has started but only FLV-playing code, and it is does not work.

FLVParser and NetConnection has been changed to use tu_file instead of a LoadThread. The ownership of the tu_file is a bit tricky, since NetConnection could be told to changes its connection, but since closing the existing connection would mess things up, we now use a shared_ptr to make sure the connection stay around as long as used by either NetConnection, FLVParser or the decoder.

Still in progress:

So far the plugin based structure has not be worked on, but the plugin structure will most likely be postponed until the decoding and parsing classes has been tested throughly. So the new code will be introduced in the libmedia folder, and will be fully loaded at runtime, not on demand.