MediaCapabilities

From Gnash Project Wiki

Jump to: navigation, search

SWF application authors asked for a way to detect which media formats and codecs are supported by a player. We here try to define an ActionScript interface to query such capabilities so that it isn't likely to interfere with future compatibility and can be easily adopted by different free (or non-free) players.

Contents

Draft1

This is the draft interface proposed by John Gilmore:

Very drafty draft (from this, you already know I don't understand ActionScript syntax): In a system with Gnash, configured to use ffmpeg or gstreamer, in which the configured media library contains these decoders. (Depending on what other codec plugins are available, lots more will also be described in these arrays.)

 System.capabilities.videoMIMETypes ["video/ogg"] will be true.
 System.capabilities.audioMIMETypes ["audio/ogg"] will be true.
 System.capabilities.audioDecoderTypes ["vorbis"] will be true.
 System.capabilities.audioDecoderTypes ["speex"] will be true.
 System.capabilities.audioDecoderTypes ["flac"] will be true.
 System.capabilities.videoDecoderTypes ["theora"] will be true.

The codec names should generally be in lowercase, so people won't have to case-match them. (We may be stuck with random cases in the MIME types, like the above "video/3GP", but at least we can do better in our own interface.) Some will be nonintuitive, e.g. MP3 won't be called mp3 because it's real name is something else; mp3 is a high-speed collision between a container format and a codec.

Compendium 1 (by strk)

In *addition* to exposing capabilities, we may actually extend the NetStream interface to support callbacks on errors. That'd mean calling an ActionScrip method of the NetStream object used to signal something went wrong with decoding. This would account for unexpected errors, like the ones warned about by gstreamer guys.

It should be inspected first if such an interface is already available in the proprietary player. The NetStream-SquareTest.swf file in testsuite/ming-misc.all already tries to load an .ogg (in addition to an .flv). Doesn't expect any callback, but doesn't mean there would be none. Worth asking on flash coders mailing lists...

Feasibility within gnash

In order to support such interface from within gnash we'll need to inspect capabilities query of the underlying media handler: gstreamer and ffmpeg.

Gstreamer

This command describes the plugins available, but it isn't clear how to tell which of these are file formats, which are codecs, etc, except by parsing English descriptions (that might appear in other languages if the user doesn't speak English).

% gst-inspect -p
musepack:  musepackdec: Musepack decoder
faad:  faad: AAC audio decoder
dvdread:  dvdreadsrc: DVD Source
a52dec:  a52dec: ATSC A/52 audio decoder
wavpack:  wavpackenc: Wavpack audio encoder
wavpack:  wavpackdec: WavePack audio decoder
wavpack:  wavpackparse: WavePack parser
gsm:  gsmdec: GSM audio decoder
gsm:  gsmenc: GSM audio encoder
...
Total count: 111 plugins, 294 features

I hope there's a programmatic interface that's easier.

<gnu{-> I'm wondering how an app can figure out the list of supported media file formats and codecs available in the local system thru gstreamer.
<MikeS> gnu{-: that's usually the wrong thing to do.
<gnu{-> The idea is that we want to play a video, it's available at different URLs for different transcodings, and we want to pull down one that's locally supported.
<gnu{-> E.g. there's an Ogg Theora version, an H.264, an MP4; which one should we play?
<MikeS> But you can iterate over all available elements in the registry, and look at their template caps.
<gnu{-> Please point me to API doc on how to do that.
<MikeS> Look at the GstRegistry documentation
<gnu{-> (If there's a better thing to do in this circumstance, I'm all ears.)
<wtay> gnu{-, you can't know until you try to play the file
<gnu{-> I can't know what?
<wtay> gnu{-, and if theora is available, select that one
<wtay> gnu{-, if you can play a certain file
<wtay> gnu{-, you can guess by looking at available codecs and demuxers
<gnu{-> I know there'll be corner cases.  But the video files come with metadata saying what format & codec they use
<MikeS> gnu{-: if you know the container format and all of the codecs used inside it, you can make a good guess by checking for things that handle ALL of those, but it's a pain, and unreliable.
<gnu{-> Since the video files can be hundreds of megs, we don't want to start pulling them all down.
<MikeS> Your description of " there's an Ogg Theora version, an H.264, an MP4" is far from sufficient, do you have a lot more metadata than that?
<gnu{-> We can have more metadata if we have to.  
<gnu{-> The whole scheme is:  The Internet Archive has whole piles of video, and transcoding engines.
<gnu{-> They want to make those videos playable on anybody's machine, esp. using free formats.
<gnu{-> They offer download of all videos in all avail formats, but they also have a Flash-based player.
<gnu{-> The Flash player would like to stream down a version that actually plays, if the user hits the Play button.
<gnu{-> Which of five or six versions will actually play locally?  That's what we want the player to be able to decide.
<gnu{-> Gnash, the free Flash player, uses gstreamer (or ffmpeg) as its media back-end.
<gnu{-> So we want Gnash to be able to offer (to any Flash movie) a list of locally supported file formats and codecs.
<MikeS> ummm... so you're trying to do this in flash?
<gnu{-> The Archive's web server can feed the Flash movie a list of URLs and the corresponding metadata about their formats.
<gnu{-> Then the Flash movie can decide which one to stream.
<gnu{-> That's it...
<gnu{-> (If the Flash player notices it isn't running under gnash, it can fall back to FLV)
<MikeS> gnu{-: this sounds like a terrible idea to me, the official flash player supports a fixed list of formats, and for other things, isn't it better to NOT use flash?
<gnu{-> (which is supported inside the Adobe Flash player)
<gnu{-> The official, proprietary Flash player ONLY supports proprietary formats.  If we use those formats, it locks out everyone who chooses freedom.
<gnu{-> How do you suggest that we NOT use flash when displaying a web page in an arbitrary browser and seeking to play video in the page?
<gnu{-> The <video> tag isn't working in most browsers, and is buggy in the ones where it works (so far).
<twi_> oh, I was going to suggest the video tag and activex in ie :)
<twi_> don't opera safari and firefox support the video tag?
<MikeS> gnu{-: just link to the media file.
<gnu{-> firefox cvs does -- not a release.
<twi_> gnu{-: right
<gnu{-> We do offer links to all the media files.  But many people prefer an embedded player.  And many won't know how to pick WHICH media file will work for them.
<MikeS> gnu{-: well, your original question was about how to do this with gstreamer, which I answered.
<gnu{-> Here's an example:   www.archive.org /details/AlaskaAM1948
<MikeS> so, go do it if you still want to.
<gnu{-> MikeS:  Thank you, I will run down your suggestion.  But you or others said it's unreliable.
<MikeS> Yes indeed.
<wtay> gnu{-, it's as reliable as it can get
<gnu{-> OK, thanks.  If you can think of ways to evolve gst to be more reliable for this, please do evolve it!  Gnash will follow your lead.
<gnu{-> I've been assuming that the following metadata is sufficient to be pretty reliable:  Container format, audio codec, video codec.  Should we have more?
<MikeS> To be _reliable_? Yes, vastly more. 
<gnu{-> ...like...
<MikeS> Profiles, details of the muxing, variants, etc.
<MikeS> i.e. a thousand tiny details
<gnu{-> Sure; if your muxer plugin version 0.345 has a bug in how it handles Frobozz streams, then you need version numbers on everything too.
<gnu{-> But is there something more than container format, audio codec, video codec that will make us 5% more reliable?
<gnu{-> e.g. move us from playing 90% of the files reliably to playing 95% of them?
<wtay> gnu{-, gstreamer element version number, maybe
<gnu{-> Pls give me an example.  E.g. the Archive has a video file, it knows the container format (ogg), it knows the audio (vorbis), it knows the video (dirac), what gstreamer element version number would be associated with this file?
<gnu{-> The file may have been created with gstreamer (if transcoded), or maybe it was created by some other software.
<wtay> gnu{-, if your file plays with versions X,Y of the codec/demuxer combinations, then it'll play with those combinations
<wtay> gnu{-, if you don't try, it might not work
<gnu{-> So the idea is that the player can upload status to the website, i.e. "Playing file X worked with gst version X, plugins A (ver Y), B (ver Z), C (ver N)"?
<gnu{-> Then the next guy to come along can know that that'll work?
<gnu{-> It's pretty deep inspection, but if we have to do it, I guess it's possible.  Media is such hard, crappy stuff sometimes.
<wtay> yes, something like that. else you can't know because of bugs
<gnu{-> Thanks!
<wtay> or maybe only a non-funcional element is installed
<wtay> decoding aac with ffmpeg instead of faad will not work very well for some aac files
<gnu{-> You mean sometimes gst registry will advertise that it can support format X, but actually it's a codec wizard or something, that won't really decode format X?
<wtay> but it might work somewhat for others
<wtay> gnu{-, but I would for starters just check the codecs in the registry. with some blacklisting and ranks it might work good enough
<gnu{-> thx, will do!

FFMPEG

From command-line:

$ ffmpeg -formats

Example output:

File formats:
  E 3g2             3gp2 format
  E 3gp             3gp format
 D  4xm             4X Technologies format
 D  MTV             MTV format
 D  RoQ             Id RoQ format
 D  aac             ADTS AAC
 DE ac3             raw ac3
 ...
 Codecs:
 D V    4xm
 D V D  8bps
 D V    VMware video
  EV    libtheora
 D V    theora
 ...
 Note, the names of encoders and decoders dont always match, so there are
 several cases where the above table shows encoder only or decoder only entries
 even though both encoding and decoding are supported for example, the h263
 decoder corresponds to the h263 and h263p encoders, for file formats its even
 worse

Community comments

#ffmpeg on freenode - 2008 09 23
--------
< Dark_Shikari> well, H.264 is the best format efficiency-wise
< Dark_Shikari> so, if its supported, there's no reason to go for others
< Dark_Shikari> the only reason to support other formats is fallbacks for old versions of flash
< Dark_Shikari> which can be done trivially by version-checking the flash player