Using VLCJ for video reliably with out of process players

This post is really a follow on to the last, so if you want a bit of background give that a read through. In it I basically talk about the various options I came to when trying to implement video support in Java, and how I felt the best way to go was using out of process players and VLCJ.

Before I get to the code, a warning - while this isn't stupidly hard, it's definitely not for beginners. If you're competent with Java in general you shouldn't have any trouble following - but this is not a cut / paste / forget the advanced stuff job! Eventually and depending on interest I may look into packaging it up into an easy to use library and distributing, but that day is not today!

Secondly, this code is literally cut straight out of Quelea and at the moment is:

• Largely uncommented


• From an unreleased version


• Not the best quality


• May contain Quelea specific bits that don't apply

Eventually, it'll all be refactored into beautiful OO-like niceness. But in terms of the concepts, it should be enough to get them across.

An overview

Essentially, how it works is by firing off a separate process executing all the native VLC code. The references to the standard output and standard input streams of the "other" process(es) are saved, and these are used for communication between the processes. (It could just as easily be done by sockets, shared memory magic, some generic RMI framework and so on. But this for me is a scalable solution that just works without any substantial libraries or potential problems with firewalls that you can get with sockets.)

There's a rudimentary protocol that maps commands sent between the streams to the actions the out of process player should take, or the values it returns. The API user doesn't need to worry about these, since it's all encapsulated in an easy to use class, RemotePlayer.

I'm using this approach to run 3 concurrent media players in Quelea, and I've had 0 VM blowouts thus far in all the trial runs I've done (which is a fair number!) The guy in charge of VLCJ also reports the same behaviour when using out of process players. However you accomplish it, this seems undoubtedly the way to go if you want to prevent your application from crashing.

The actual code bit

So, now actually onto the code. Where it all starts from the user's point of view is RemotePlayerFactory:

package org.quelea.video;

import com.sun.jna.Native;
import java.awt.Canvas;

public class RemotePlayerFactory {

    public static RemotePlayer getRemotePlayer(Canvas canvas) {
        try {
            long drawable = Native.getComponentID(canvas);
            StreamWrapper wrapper = startSecondJVM(drawable);
            final RemotePlayer player = new RemotePlayer(wrapper);
            Runtime.getRuntime().addShutdownHook(new Thread() {
                @Override
                public void run() {
                    player.close();
                }
            });
            return player;
        }
        catch (Exception ex) {
            throw new RuntimeException("Couldn't create remote player", ex);
        }
    }

    private static StreamWrapper startSecondJVM(long drawable) throws Exception {
        String separator = System.getProperty("file.separator");
        String classpath = System.getProperty("java.class.path");
        String path = System.getProperty("java.home")
                + separator + "bin" + separator + "java";
        ProcessBuilder processBuilder = new ProcessBuilder(path, "-cp", classpath, "-Djna.library.path=" + System.getProperty("jna.library.path"), OutOfProcessPlayer.class.getName(), Long.toString(drawable));
        Process process = processBuilder.start();
        return new StreamWrapper(process.getInputStream(), process.getOutputStream());
    }
}

The call that might be most unfamiliar initially is the JNA one - Native.getComponentId(). Every heavyweight component has an ID which is assigned at the OS level and used for drawing to a particular window. It's this component ID that we'll be passing to the separate process, which it will use to draw to the window owned by the parent process. Notice also the shutdown hook so the other VM is terminated when this one is (that's essentially what the close method does, more on that later.) It's not a foolproof approach but it's good just in case it doesn't get cleared up otherwise.

In terms of the startSecondJVM method, this is a pretty standard, cross-platform (as much as I care about anyway, Windows, MacOS and Linux should all be fine) method to start up a second JVM. It starts the OutOfProcessPlayer with the component ID as its argument. There's just a few differences - firstly, we copy over the classpath and JNA library path of this VM so it can execute the class in this project which executes native VLC code without any problems. Secondly, it captures the streams in a StreamWrapper object, which is just as follows:

package org.quelea.video;

import java.io.InputStream;
import java.io.OutputStream;

/**
 *
 * @author Michael
 */
public class StreamWrapper {

    private InputStream inputStream;
    private OutputStream outputStream;

    StreamWrapper(InputStream inputStream, OutputStream outputStream) {
        this.inputStream = inputStream;
        this.outputStream = outputStream;
    }

    public InputStream getInputStream() {
        return inputStream;
    }

    public OutputStream getOutputStream() {
        return outputStream;
    }
}

Nothing special here, it's literally just wrapping up the input and output streams.

Onto the two core classes here, RemotePlayer and OutOfProcessPlayer. As the name suggests, the latter is the one that sits out of process.

RemotePlayer.java:

package org.quelea.video;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;

/**
 * Controls an OutOfProcessPlayer via input / output process streams.
 * @author Michael
 */
public class RemotePlayer {

    private BufferedReader in;
    private BufferedWriter out;
    private boolean open;
    private boolean playing;
    private boolean paused;

    /**
     * Internal use only.
     */
    RemotePlayer(StreamWrapper wrapper) {
        out = new BufferedWriter(new OutputStreamWriter(wrapper.getOutputStream()));
        in = new BufferedReader(new InputStreamReader(wrapper.getInputStream()));
        playing = false;
        open = true;
    }

    private void writeOut(String command) {
        if (!open) {
            throw new IllegalArgumentException("This remote player has been closed!");
        }
        try {
            out.write(command + "\n");
            out.flush();
        }
        catch (IOException ex) {
            throw new RuntimeException("Couldn't perform operation", ex);
        }
    }

    private String getInput() {
        try {
            return in.readLine();
        }
        catch (IOException ex) {
            throw new RuntimeException("Couldn't perform operation", ex);
        }
    }

    public void load(String path) {
        writeOut("open " + path);
    }

    public void play() {
        writeOut("play");
        playing = true;
        paused = false;
    }

    public void pause() {
        if(!paused) {
            writeOut("pause");
            playing = false;
            paused = true;
        }
    }

    public void stop() {
        writeOut("stop");
        playing = false;
        paused = false;
    }

    public boolean isPlayable() {
        writeOut("playable?");
        return Boolean.parseBoolean(getInput());
    }

    public long getLength() {
        writeOut("length?");
        return Long.parseLong(getInput());
    }

    public long getTime() {
        writeOut("time?");
        return Long.parseLong(getInput());
    }

    public void setTime(long time) {
        writeOut("setTime " + time);
    }

    public boolean getMute() {
        writeOut("mute?");
        return Boolean.parseBoolean(getInput());
    }

    public void setMute(boolean mute) {
        writeOut("setMute " + mute);
    }

    /**
     * Terminate the OutOfProcessPlayer. MUST be called before closing, otherwise
     * the player won't quit!
     */
    public void close() {
        if (open) {
            writeOut("close");
            playing = false;
            open = false;
        }
    }

    /**
     * Determine whether the remote player is playing.
     * @return true if its playing, false otherwise.
     */
    public boolean isPlaying() {
        return playing;
    }
    
    /**
     * Determine whether the remote player is paused.
     * @return true if its paused, false otherwise.
     */
    public boolean isPaused() {
        return paused;
    }
    
}

OutOfProcessPlayer.java:

package org.quelea.video;

import com.sun.jna.NativeLibrary;
import com.sun.jna.Pointer;
import java.awt.Canvas;
import java.io.BufferedReader;
import java.io.File;
import java.io.InputStreamReader;
import java.io.PrintStream;
import org.quelea.utils.QueleaProperties;
import uk.co.caprica.vlcj.binding.LibVlcFactory;
import uk.co.caprica.vlcj.binding.internal.libvlc_media_player_t;
import uk.co.caprica.vlcj.player.embedded.EmbeddedMediaPlayer;
import uk.co.caprica.vlcj.player.embedded.linux.LinuxEmbeddedMediaPlayer;
import uk.co.caprica.vlcj.player.embedded.mac.MacEmbeddedMediaPlayer;
import uk.co.caprica.vlcj.player.embedded.windows.WindowsEmbeddedMediaPlayer;
import uk.co.caprica.vlcj.runtime.RuntimeUtil;

/**
 * Sits out of process so as not to crash the primary VM.
 * @author Michael
 */
public class OutOfProcessPlayer {

    public OutOfProcessPlayer(final long canvasId) throws Exception {

        //Lifted pretty much out of the VLCJ code
        EmbeddedMediaPlayer mediaPlayer;
        if (RuntimeUtil.isNix()) {
            mediaPlayer = new LinuxEmbeddedMediaPlayer(LibVlcFactory.factory().synchronise().log().create().libvlc_new(1, new String[]{"--no-video-title"}), null) {

                @Override
                protected void nativeSetVideoSurface(libvlc_media_player_t mediaPlayerInstance, Canvas videoSurface) {
                    libvlc.libvlc_media_player_set_xwindow(mediaPlayerInstance, (int) canvasId);
                }
            };
        }
        else if (RuntimeUtil.isWindows()) {
            mediaPlayer = new WindowsEmbeddedMediaPlayer(LibVlcFactory.factory().synchronise().log().create().libvlc_new(1, new String[]{"--no-video-title"}), null) {

                @Override
                protected void nativeSetVideoSurface(libvlc_media_player_t mediaPlayerInstance, Canvas videoSurface) {
                    Pointer ptr = Pointer.createConstant(canvasId);
                    libvlc.libvlc_media_player_set_hwnd(mediaPlayerInstance, ptr);
                }
            };
        }
        else if (RuntimeUtil.isMac()) {
            mediaPlayer = new MacEmbeddedMediaPlayer(LibVlcFactory.factory().synchronise().log().create().libvlc_new(2, new String[]{"--no-video-title", "--vout=macosx"}), null) {

                @Override
                protected void nativeSetVideoSurface(libvlc_media_player_t mediaPlayerInstance, Canvas videoSurface) {
                    Pointer ptr = Pointer.createConstant(canvasId);
                    libvlc.libvlc_media_player_set_nsobject(mediaPlayerInstance, ptr);
                }
            };
        }
        else {
            mediaPlayer = null;
            System.exit(1);
        }

        mediaPlayer.setVideoSurface(new Canvas());

        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        String inputLine;

        //Process the input - I know this isn't very OO but it works for now...
        while ((inputLine = in.readLine()) != null) {
            if (inputLine.startsWith("open ")) {
                inputLine = inputLine.substring("open ".length());
                mediaPlayer.prepareMedia(inputLine);
            }
            else if (inputLine.equalsIgnoreCase("play")) {
                mediaPlayer.play();
            }
            else if (inputLine.equalsIgnoreCase("pause")) {
                mediaPlayer.pause();
            }
            else if (inputLine.equalsIgnoreCase("stop")) {
                mediaPlayer.stop();
            }
            else if (inputLine.equalsIgnoreCase("playable?")) {
                System.out.println(mediaPlayer.isPlayable());
            }
            else if (inputLine.startsWith("setTime ")) {
                inputLine = inputLine.substring("setTime ".length());
                mediaPlayer.setTime(Long.parseLong(inputLine));
            }
            else if (inputLine.startsWith("setMute ")) {
                inputLine = inputLine.substring("setMute ".length());
                mediaPlayer.mute(Boolean.parseBoolean(inputLine));
            }
            else if (inputLine.equalsIgnoreCase("mute?")) {
                boolean mute = mediaPlayer.isMute();
                System.out.println(mute);
            }
            else if (inputLine.equalsIgnoreCase("length?")) {
                long length = mediaPlayer.getLength();
                System.out.println(length);
            }
            else if (inputLine.equalsIgnoreCase("time?")) {
                long time = mediaPlayer.getTime();
                System.out.println(time);
            }
            else if (inputLine.equalsIgnoreCase("close")) {
                System.exit(0);
            }
            else {
                System.out.println("unknown command: ." + inputLine + ".");
            }
        }
    }

    public static void main(String[] args) {
        //Next 3 lines Quelea specific
        File nativeDir = new File("lib/native");
        NativeLibrary.addSearchPath("libvlc", nativeDir.getAbsolutePath());
        NativeLibrary.addSearchPath("vlc", nativeDir.getAbsolutePath()); 

        PrintStream stream = null;
        try {
            stream = new PrintStream(new File(QueleaProperties.get().getQueleaUserHome(), "ooplog.txt"));
            System.setErr(stream); //This is important, need to direct error stream somewhere
            new OutOfProcessPlayer(Integer.parseInt(args[0]));
        }
        catch (Exception ex) {
            ex.printStackTrace();
        }
        finally {
            stream.close();
        }
    }
}

Hopefully, after you study the above two classes it should be pretty self-explanatory what's going on. The two classes talk to each other via the streams and react accordingly. The protocol there isn't complete - for now I've just completed what Quelea requires, though if I develop it into a library in its own right I'll complete the API obviously!

Conclusion

It's not trivial implementing out of process players in VLCJ, but neither is it ridiculously difficult and it does provide for trouble free playing if you get it right. The code above could definitely, and will almost definitely be improved - in terms of quality, completeness and error handling (there's nothing in there at the moment to cope with the external VM just disappearing or being closed externally, for instance.) And while communicating over streams like the above is reliable and doesn't need any big external libraries on top, it's not the fastest approach (more than good enough for my purpose however.)

However, as I already stated the purpose of this is to get the idea across, to provide some skeleton code for out of process VLCJ players, and to show that with a bit of work, it's entirely possible to get excellent video support in a Java application (even if it is done natively.)

I hope it proves useful to at least someone!