For the next version of my Twitter client (GraffiTwit) I wanted to add support for camera capture since GraffiTwit already provides a drawing interface. On Linux the GStreamer framework is used to build pipelines of media-handling components. Doing a single frame capture is a simple pipeline, like this one (on a desktop linux box) :

v4l2src ! ffmpegcolorspace ! pngenc ! filesink location=test.png
You can test it using the gst-launch tool from the gstreamer-tools package, or use it from a C program, with two further options :
  • create each element (v4l2src, ffmpegcolorspace, etc.) and chain them together in a bin to create the pipeline (see the hello_world exemple for details).
  • use the gst_parse() function to create a pipeline from the same syntax used with gst-launch.

The second method does not allow as much configuration as the first one but is sufficient when you only need to run a relatively simple pipeline.

Now we need a way to call gst_parse() from Mozilla. There is some ongoing work to add GStreamer based support to video and image tags, but since it has not landed yet, we need to find another solution. Here we have three options :

  • use nsIProcess to launch gst-launch. This is kind of ugly, and will likely fail as gst-launch is not installed by default on many linux distributions.
  • write a C++ XPCOM component to encapsulate GStreamer functionnality. Not difficult, but really not funny.
  • use js-ctypes to access the GStreamer library directly from javascript land.

js-ctypes is a Mozilla code module that makes it possible to call C-compatible foreign library functions from javaScript code. This means that you don't need to write your own XPCOM wrapper to access low-level code in your javascript code. There are some limitations on what kind of parameters you can send and receive, so that may or not fit your needs.

It our case, we need to access the following functions from libgstreamer-0.10.so : gst_init(), gst_parse_launch(), gst_element_set_state, gst_element_get_state().

Here's the javascript code needed :
Components.utils.import("resource://gre/modules/ctypes.jsm");

const GST_STATE_NULL                = 1;

const GST_STATE_PLAYING             = 4;

const GST_STATE_CHANGE_ASYNC        = 2;

let gstreamer = ctypes.open("libgstreamer-0.10.so.0");

let gst_init = gstreamer.declare("gst_init", ctypes.default_abi, ctypes.void_t, 

                                 ctypes.int32_t, ctypes.int32_t);

let gst_parse_launch = gstreamer.declare("gst_parse_launch", ctypes.default_abi, ctypes.int32_t,

                                         ctypes.string, ctypes.int32_t);

let gst_element_set_state = gstreamer.declare("gst_element_set_state", ctypes.default_abi, ctypes.int32_t,

                                              ctypes.int32_t, ctypes.int32_t);

let gst_element_get_state = gstreamer.declare("gst_element_get_state", ctypes.default_abi, ctypes.int32_t,

                                              ctypes.int32_t, ctypes.int32_t, ctypes.int32_t, ctypes.int64_t);

gst_init(0, 0);

let pipeline = gst_parse_launch("v4l2src ! ffmpegcolorspace ! pngenc ! filesink location=/tmp/test.png", 0);

let ret = gst_element_set_state(pipeline, GST_STATE_PLAYING);

if (ret == GST_STATE_CHANGE_ASYNC) { // wait for state change to happen

  ret = gst_element_get_state(pipeline, 0, 0, -1);

  gst_element_set_state(pipeline, GST_STATE_NULL);

  gstreamer.close();

}

And that's all ! Less than 21 lines of javascript.

When the API sends or expects a pointer to an element, we use an opaque int32_t datatype. This is a hack, but future version of js-ctypes are expected to have better type support.

Even if in our case this is not as good as having real support, this shows that js-ctypes is part of the hackability of the browser. For instance, you can use it in Jetpack...