Expanding JavaFX's media support

Note: For those that don't want to read through the post and just want the patch for MKV support, you can grab it from this ticket, or here if you don't have a JIRA account.

Background

One of the predominant things lacking a "nice" approach in the Java world for years now has been good media support. Oh sure, we had JMF, but anyone who ever had the misfortune of using that will I'm sure understand why that never really took on. (Yes, it really was that bad.) A few other approaches came and went, most notably Java Media Components - but none ever made there way into core Java, and for a long time it became pretty de-facto knowledge that if you wanted any form of comprehensive media support in Java, you used a cross-platform native library, perhaps with a Java wrapper.

However, when JavaFX 2 came along we were provided with a new, baked in media framework that provided this functionality on the Java level! This is a massive step forward, sure it uses GStreamer underneath but that's not really an issue - the required libraries are baked into JavaFX, and the end user can just treat the corresponding MediaView as any other node in the scenegraph.

However, while this is much better than what we had previously, it's rather limited support at this stage:

7. Does JavaFX provide support for audio and video codecs?

JavaFX provides a common set of APIs that make it easy to include media playback within any JavaFX application. The media formats currently supported are the following:

• Audio: MP3; AIFF containing uncompressed PCM; WAV containing uncompressed PCM; MPEG-4 multimedia container with Advanced Audio Coding (AAC) audio
• Video: FLV containing VP6 video and MP3 audio; MPEG-4 multimedia container with H.264/AVC (Advanced Video Coding) video compression .

The format support is far from useless; for videos bundled with an application that can be in any format, mp4 with h264 and AAC is certainly a relatively standard option. Likewise, if you're writing an application that uploads videos and uses something like ffmpeg to convert them over to a standard format before displaying them with a JFX frontend somehow, this is also adequate.

However, there are many use cases where the current support is very restrictive indeed - certainly any general purpose media player is out of the question, as is (realistically) any application where you want user-selected video files to play with any degree of reliability. Two of the most common container formats are out whatever formats are inside them (MKV and the badly ageing AVI), AC3 audio support isn't there... well, anything that isn't in the above list isn't there. Which is a lot. Many of these will have been excluded for licensing reasons, though even many free ones that could be in there aren't (MKV, OGG, FLAC, etc.)

But as said already, the JavaFX media classes use GStreamer to do the heavy lifting, which has all these formats (and more) available to it through plugins. So now that the whole thing is open sourced, it should in theory be possible to rebuild JFX with more GStreamer plugins compiled in, right? Turns out it is - the following is an outline of the process of how I did it. I'll be adding support for the MKV container here, other plugins can no doubt be added in a similar way.

Note: I'm far from an authority on this subject, I'm not a JFX developer and I'm certainly not advocating that what I describe here is necessarily all accurate, or correct. It's merely what I've been able to work out from tracing through the source and from some helpful people on the openjfx-dev mailing list.

Setting up the build

You'll firstly need to check out the JFX repo using Mercurial:

hg clone http://hg.openjdk.java.net/openjfx/8u-dev/master/

You will of course need a Mercurial client. If you're using Windows and haven't got a Mercurial client installed already, I highly recommend TortoiseHg. The clone may take a while to complete (the repository is on the larger side!) so be patient.

When you've checked out the repository, you'll then need to make sure you have all the prerequisites you need to build successfully - for that, see the build page and make sure you have everything installed that you need (for Windows users, you must have Cygwin installed with the listed plugins.)

There's a couple of points you also must take note of if you're running Windows that aren't mentioned in the document however:

• If the DirectX SDK fails to install then it's almost definitely because of the issue described here - uninstall the relevant packages, then try the install again and it should go through without an issue.
• You must also have the samples from the Windows SDK installed.
• We want to compile the media module, which is disabled by default for timing reasons. In the root of the repo, copy gradle.properties.template to gradle.properties, then uncomment the line (delete the first hash) that says "#COMPILE_MEDIA = true".
• Things will be much easier if you add your gradle bin folder to PATH.

You can then fire up a shell, type "gradle sdk" and watch it attempt to build. Since you haven't made any changes at this point, all should go well and you should be presented with a "BUILD SUCCESSFUL" message after a while (the build will take a few minutes to complete.) If not, then go back and double check you've set everything up as per the instructions on the build page. Of course, if you're still stuck then feel free to leave a comment :)

Making the changes

At this point, there's two ways you can proceed - the first is to just grab the patch file, apply it to the repo, and then rebuild (run "gradle sdk" again.) All being well, the MKV container will then be supported in the resulting build. So if you want to just do that the easy way, you can skip the rest of this step.

However, in the interest of being as informative as possible to those that want to repeat the step with another plugin, I'll describe the necessary changes here in detail. Changes on both the Java and native layer are required, so we'll start with the Java layer.

Java layer

The bit of the Java layer that we're interested in is really just responsible for performing some basic checks on the file's type to determine if it has a hope of playing it. This is a relatively simple process, hopefully explained by the diagram below:

This means we have to add support in two places; we have to modify the filenameToContentType() method so it can work out the correct content type from the file name, and we then have to add it to the list of supported content types for the corresponding platform, GSTPlatform in this case (this is just an entry in the array.) Optionally we could also add knowledge of the signature to the fileSignatureToContentType() method, but that isn't strictly necessary since it's just used as a fallback if the type can't be worked out from the fileNameToContentType() method. In doing so you'll also need to add the extension and content type to the MediaUtils class:

public static final String CONTENT_TYPE_MKV = "video/x-matroska";

private static final String FILE_TYPE_MKV = "mkv";

The file extension is obvious, but the content type should be grabbed from the GStreamer defined types list here.

I won't go into huge amounts of detail on what exactly to change in the Java classes - it's pretty basic Java, and I'm assuming most of the people reading this will be Java programmers! You can of course look at the patch to see my exact changes. If you get stuck, feel free to leave a comment and I'll do my best to help.

When you've done this, rebuild JFX (run "gradle sdk") and try creating a media object to point to an MKV file. If the above has worked successfully, you'll get a different (native) error to the one you got before. This is good - it means the Java layer is letting the file pass down to be played in the native layer - now we need to enable it here.

Native Layer

We now need to grab the required plugin for GStreamer - in the case of the matroska plugin, this is in the "plugins-good" category, which means it's a well written and tested plugin that shouldn't pose distribution problems. The tarball of source for the plugins can be grabbed from here.

However, make sure when you're doing this that you grab the correct version of the plugins - JavaFX (for 8u20 and before at least) isn't built with the latest GStreamer. To find out what one, we can look at the GStreamer modifications that Oracle publish, a zip file containing modifications to the bits of GStreamer they've included. This shows the plugins being used are the 0.10.30 ones (for the good branch anyway), so go ahead, download the 0.10.30 good plugins and pull out the matroska one, and drop it in the relevant directory - in this case "modules/media/src/main/native/gstreamer/gstreamer-lite/gst-plugins-good/gst/".

Most of the modifications that Oracle make are fixes and performance improvements to the other plugins, though one required change is to the plugin loading system. In the main plugin C file, matroska.c in this case, you'll see an init function like the following:

GST_PLUGIN_DEFINE (GST_VERSION_MAJOR,
    GST_VERSION_MINOR,
    "matroska",
    "Matroska and WebM stream handling",
    plugin_init, VERSION, "LGPL", GST_PACKAGE_NAME, GST_PACKAGE_ORIGIN)

We don't need this function, so we can remove it - instead of removing it though, make sure you follow the convention of wrapping it in "#ifndef":

+#ifndef GSTREAMER_LITE
+GST_PLUGIN_DEFINE (GST_VERSION_MAJOR,
+    GST_VERSION_MINOR,
+    "matroska",
+    "Matroska and WebM stream handling",
+    plugin_init, VERSION, "LGPL", GST_PACKAGE_NAME, GST_PACKAGE_ORIGIN)
+#endif

This then makes it much easier to find where changes have been made later on, as well as preserving the original functionality if it's not being built in the JFX environment for whatever reason (in which case GSTREAMER_LITE won't be defined, so the above will execute.)

We also need to change the plugin_init function in a similar way, which is the following:

static gboolean
plugin_init (GstPlugin * plugin)

We want to make two changes here - we don't want it to be static, and we want the name to be more unique so it can be initialised alongside other plugins without any conflict. The convention appears to be "plugin_init_pluginname", so replace the above with the following:

#ifdef GSTREAMER_LITE
gboolean
plugin_init_matroska (GstPlugin * plugin)
#else // GSTREAMER_LITE
static gboolean
plugin_init (GstPlugin * plugin)
#endif // GSTREAMER_LITE

Again, the changes are wrapped in the appropriate tags to make it clear what we've changed.

That's all the changes for this file, but we need to add the method we've defined (plugin_init_matroska in this case) to the appropriate headers file, "modules/media/src/main/native/gstreamer/gstreamer-lite/projects/plugins/gstplugins-lite.h". Open it, and add:

gboolean plugin_init_matroska (GstPlugin * plugin);

...or whatever you called your function above to the list of method headers. We also need to make sure the function is called to initialise the plugin, so open up modules/media/src/main/native/gstreamer/gstreamer-lite/projects/plugins/gstplugins-lite.c and find the section where the plugins are initialised, it'll be a list in a big if statement looking something like this:

...

!plugin_init_aiff(plugin) ||
!plugin_init_app(plugin) ||
!plugin_init_audioparsers(plugin) ||

...

Then just add the call to your plugin in the same way:

...
!plugin_init_aiff(plugin) ||
!plugin_init_app(plugin) ||
!plugin_init_audioparsers(plugin) ||
!plugin_init_matroska(plugin) ||
...
We now need to update the JavaFX cpp bridging code to make it aware of the format, and create and return a pipeline for it. Let's start by defining the format in modules/media/src/main/native/jfxmedia/MediaManagement/MediaTypes.h - this is just a case of adding it to the list, making sure it's the same as you defined it in the Java code. So for this case, the following needs to be added:

#define CONTENT_TYPE_MKV    "video/x-matroska"
Next, open modules/media/src/main/native/jfxmedia/platform/gstreamer/GstPipelineFactory.cpp - this is where the pipeline creation actually takes place. There's a few things that need to be changed in here:

Find the pushback function calls, something like:
     m_ContentTypes.push_back(CONTENT_TYPE_MP4);
     m_ContentTypes.push_back(CONTENT_TYPE_M4A);
     m_ContentTypes.push_back(CONTENT_TYPE_M4V);
Then add your content type to it in the same way:
     m_ContentTypes.push_back(CONTENT_TYPE_MKV);

The next step is a bit less clear cut, you need to find the CreatePlayerPipeline function, and identify the part where your pipeline should be created. You'll see a general pattern here - the video container formats are dealt with within one if statement that sets up a video sink before creating the pipeline, and the audio formats are dealt with afterwards. So in the appropriate place, you need to follow the pattern to hook in and call a method to create your pipeline. For MKV, it seemed to make most sense to add it to the if/else block just after checking for mp4 files, so straight afterwards I added this:
    else if (CONTENT_TYPE_MKV == locator->GetContentType())
 {
  uRetCode = CreateMKVPipeline(pSource, pVideoSink, (CPipelineOptions*) pOptions, ppPipeline);
  if (ERROR_NONE != uRetCode)
               return uRetCode;
        }



Of course, you then need to write the function to return the pipeline, and again you can follow the pattern of similar ones for this. Once again, MKV is similar in the way it's handled to MP4 so I simply copied that function and made the appropriate changes thus producing:

uint32_t CGstPipelineFactory::CreateMKVPipeline(GstElement* source, GstElement* pVideoSink, CPipelineOptions* pOptions, CPipeline** ppPipeline)
{
#if TARGET_OS_WIN32
    return CreateAVPipeline(source, "matroskademux", "dshowwrapper", true, dshowwrapper", pVideoSink, pOptions, ppPipeline);
#elif TARGET_OS_MAC
    return CreateAVPipeline(source, "matroskademux", "audioconverter", false, avcdecoder", pVideoSink, pOptions, ppPipeline);
#elif TARGET_OS_LINUX
#if ENABLE_GST_FFMPEG
    return CreateAVPipeline(source, "matroskademux", "ffdec_aac", true,
                            "ffdec_h264", pVideoSink, pOptions, ppPipeline);
#else // ENABLE_GST_FFMPEG
    return CreateAVPipeline(source, "matroskademux", "avaudiodecoder", false, "avvideodecoder", pVideoSink, pOptions, ppPipeline);
#endif // ENABLE_GST_FFMPEG
#else
    return ERROR_PLATFORM_UNSUPPORTED;
#endif // TARGET_OS_WIN32
}



The only things I changed from the CreateMP4Pipeline were the name of the function, and the name of the demuxing plugin (matroskademux in this case) - everything else remains the same.


Of course, you'll now need to add the above function to the relevant header file, so in modules/media/src/main/native/jfxmedia/platform/gstreamer/GstPipelineFactory.h, add:

uint32_t    CreateMKVPipeline(GstElement* source, GstElement* videosink, CPipelineOptions* pOptions, CPipeline** ppPipeline);

...to the list of functions.





That should be all the changes you need to make to the native code, now we just need to ensure it's compiled and linked properly. So to start with, you'll need to update the plugins makefile, modules/media/src/main/native/gstreamer/projects/win/gstreamer-lite/Makefile.gstplugins - add the directory and all the c files in that appropriate directory. So my list of directories now looks (partly) something like:

...

gst-plugins-good/gst/spectrum/ \
gst-plugins-good/gst/wavparse/ \
gst-plugins-good/gst/matroska/ \
gstreamer/plugins/elements/ \
gstreamer/plugins/indexers/ \
...


And the list of files:


...
gst-plugins-good/gst/spectrum/gstspectrum.c \
gst-plugins-good/gst/wavparse/gstwavparse.c \
gst-plugins-good/gst/matroska/webm-mux.c \
gst-plugins-good/gst/matroska/matroska-parse.c \
gst-plugins-good/gst/matroska/matroska-mux.c \
gst-plugins-good/gst/matroska/matroska-ids.c \
gst-plugins-good/gst/matroska/matroska-demux.c \
gst-plugins-good/gst/matroska/matroska.c \
gst-plugins-good/gst/matroska/lzo.c \
gst-plugins-good/gst/matroska/ebml-write.c \
gst-plugins-good/gst/matroska/ebml-read.c \
gstreamer/plugins/elements/gstcapsfilter.c \
gstreamer/plugins/elements/gstelements.c \
...


The boldings are my addition (obviously the above is just an excerpt.)


Now you can try and build and see if you encounter any compilation errors (you could and arguably should of course, do this as you're going through as well.) I found odd things sometimes cropped up if I didn't do a clean (gradle clean) before re-building, so if something doesn't seem quite right, bear that in mind.


You shouldn't have any compilation errors at this point, but you almost certainly will have linker errors (probably starting with "Unresolved external symbol" or something similar, then referencing a function.) These functions will either start with g_ or gst_ (ignore the leading underscore.) The ones that start with gst need to be added to modules/media/src/main/native/gstreamer/projects/win/gstreamer-lite.def matching the format of the file already there. This means for each function, you need a new line in the format of:


function_name<TAB>@nextsequentialnumber<TAB>NONAME.

Obviously, function_name should be replaced with the function name, <TAB> should be replaced with an actual tab and nextsequentialnumber should be replaced with, you guessed it, the next sequential number. Apart from that you don't need to worry about ordering. For the matroska plugin, I had to add the following to the end of the file:

gst_byte_writer_free_and_get_buffer @184 NONAME
gst_byte_writer_free @185 NONAME
gst_byte_writer_new_with_size @186 NONAME

For the functions that start with "g_" instead, you need to add these in the exact same way to both the modules/media/src/main/native/gstreamer/3rd_party/glib/glib-2.28.8/build/win32/vs100/glib-lite.def and the modules/media/src/main/native/gstreamer/3rd_party/glib/glib-2.28.8/build/win32/vs100/glib-liteD.def files.


If you recompile and it now complains about an unresolved external in one of the def files itself, changes are the source that contains those functions isn't actually getting compiled (because it wasn't needed in any of the plugins already there.) For the Matroska plugin this is the case for the bytewriter functions mentioned above. These are in gstbytewriter.c, and a quick check in the makefile (modules/media/src/main/native/gstreamer/projects/win/gstreamer-lite/Makefile.gstreamer) indeed confirmed that it wasn't on the list, so that was promptly added:


gstreamer/libs/gst/base/gstbasetransform.c \
gstreamer/libs/gst/base/gstbytereader.c \
gstreamer/libs/gst/base/gstbytewriter.c \
gstreamer/libs/gst/base/gstcollectpads.c \
gstreamer/libs/gst/base/gstpushsrc.c \

And that's it - after performing those steps, doing a full build, then running against the built dll's and jfxrt.jar, media support for MKV (or whatever other format you've chosen) should just work!


Summary
These are relatively extensive instructions, but none are really major changes. Summarised, they're pretty much as follows:



Edit the fileNameToContentType() method to return the correct content type for your file name.
Optionally edit the fileSignatureToContentType() method to return the correct type for your file signature (this page may be of use here.)
Add the content type to the list of GSTPlatform's supported content types
Download the plugin file and drop it in the relevant directory
In the plugin's main file, rename the plugin_init function to something more unique, remove the static modifier
Initialise the plugin in gstplugins-lite.c
Define the media type in MediaTypes.h
In GSTPipelineFactory.cpp, call m_ContentTypes.push_back on the content type
In the same file, in the CreatePlayerPipeline function, add in a hook to call a function to create the pipeline for your content type
Create the above function (using the other ones in that file as a template)
Add the relevant files / directory to Makefile.gstplugins
Make sure all the files you rely on are being built, add any that aren't to Makefile.gstreamer
Add any gstreamer functions (gst) to gstreamer-lite.def
Add any glib functions (g) to glib-lite.def and glib-liteD.def


As said already, the above guide is my (little) experience simply with playing around and adding MKV support, so I can't guarantee it will be exactly the same for other plugins and formats. However, it should at least serve as a starting guide for those wanting to build much more comprehensive media support into JavaFX. There's many, many gstreamer plugins available, most of which will, in all likelihood never be included in JavaFX core because of licensing issues. But if you want to build your own version of JFX to distribute with your application, then building much more media support into it, as described above, is more than do-able.