Impedance Mismatch

jetpack goes mobile

Fabrice — Fri, 11 Dec 2009 16:18:00 +0100

Jetpack is a project from Mozilla Labs that aims to ease the development of Firefox add-ons using web technologies without the complexity of the classical add-on model. Jetpacks are lightweight extensions, perfect for simple tasks like enhancing the browsing experience on a site. They are very similar to Google Chrome extensions, only better ;-)

Currently available for Firefox and experimentally for Thunderbird, Jetpack is not yet ready for Firefox Mobile aka Fennec. So I took a look at adding support for Fennec in Jetpack since lightweight extensions seem a perfect fit for a mobile browser.

The port is not complete yet, because some UI features are unavailable in Fennec (sidebars and menus for instance). I only adapted the statusbar support, adding an area at the bottom of the tabstrip.

Guided Tour

Once you have this custom Jetpack version installed, you'll find the entry point to the configuration and Jetpack gallery in the Add-ons panel :

Choosing "Configure Jetpack" opens the about:jetpack page in a new tab. Clicking on the logo also leads to the gallery.

Let's now choose an add-on in the gallery :

Select "Let me Install this unreviewed Jetpack" and "INSTALL JETPACK" to install TagCounter :

Hooray !

You can also install Jetpacks published elsewhere, like this one (a bugzilla reading enhancer) :

Since this Jetpack comes from an untrusted source, you need to opt-in before installing :

Then we can check that our Jetpacks (called "Features") are available :

Of course we can now put them to use ! TagCounter added a small clickable icon in the status bar. Just select it to get a nice graphic in a new tab :

It worked !

The Bugzilla Feature doesn't add anything to the status bar : it listens to page loads and adds useful information on the fly :

Next steps

This is a work in progress, and a lot still needs to be done, both on the Fennec side and in Jetpack :

Lots of UI polish, like finding a way in Fennec UI to support sidebars. And the current status bar is not really touch-friendly yet.
Jetpack relies on FUEL (or on STEEL in Thunderbird) which is lacking in Fennec. I had to include a stripped down version implementing just what Jetpack needed.
Refactor some parts of Jetpack code to have a cleaner support of different browsers (the tab browser in Fennec is not the same as the one in Firefox).
I was surprised to find that Jetpack was not localized at all.

And of course, testing, testing and some more testing !

Using GStreamer with js-ctypes

Fabrice — Thu, 19 Nov 2009 15:08:00 +0100

For the next version of my Twitter client (GraffiTwit) I wanted to add support for camera capture since GraffiTwit already provides a drawing interface. On Linux the GStreamer framework is used to build pipelines of media-handling components. Doing a single frame capture is a simple pipeline, like this one (on a desktop linux box) :

v4l2src ! ffmpegcolorspace ! pngenc ! filesink location=test.png

You can test it using the gst-launch tool from the gstreamer-tools package, or use it from a C program, with two further options :

create each element (v4l2src, ffmpegcolorspace, etc.) and chain them together in a bin to create the pipeline (see the hello_world exemple for details).
use the gst_parse() function to create a pipeline from the same syntax used with gst-launch.

The second method does not allow as much configuration as the first one but is sufficient when you only need to run a relatively simple pipeline.

Now we need a way to call gst_parse() from Mozilla. There is some ongoing work to add GStreamer based support to video and image tags, but since it has not landed yet, we need to find another solution. Here we have three options :

use nsIProcess to launch gst-launch. This is kind of ugly, and will likely fail as gst-launch is not installed by default on many linux distributions.
write a C++ XPCOM component to encapsulate GStreamer functionnality. Not difficult, but really not funny.
use js-ctypes to access the GStreamer library directly from javascript land.

js-ctypes is a Mozilla code module that makes it possible to call C-compatible foreign library functions from javaScript code. This means that you don't need to write your own XPCOM wrapper to access low-level code in your javascript code. There are some limitations on what kind of parameters you can send and receive, so that may or not fit your needs.

It our case, we need to access the following functions from libgstreamer-0.10.so : gst_init(), gst_parse_launch(), gst_element_set_state, gst_element_get_state().

Here's the javascript code needed :

Components.utils.import("resource://gre/modules/ctypes.jsm");

const GST_STATE_NULL                = 1;

const GST_STATE_PLAYING             = 4;

const GST_STATE_CHANGE_ASYNC        = 2;

let gstreamer = ctypes.open("libgstreamer-0.10.so.0");

let gst_init = gstreamer.declare("gst_init", ctypes.default_abi, ctypes.void_t, 

                                 ctypes.int32_t, ctypes.int32_t);

let gst_parse_launch = gstreamer.declare("gst_parse_launch", ctypes.default_abi, ctypes.int32_t,

                                         ctypes.string, ctypes.int32_t);

let gst_element_set_state = gstreamer.declare("gst_element_set_state", ctypes.default_abi, ctypes.int32_t,

                                              ctypes.int32_t, ctypes.int32_t);

let gst_element_get_state = gstreamer.declare("gst_element_get_state", ctypes.default_abi, ctypes.int32_t,

                                              ctypes.int32_t, ctypes.int32_t, ctypes.int32_t, ctypes.int64_t);

gst_init(0, 0);

let pipeline = gst_parse_launch("v4l2src ! ffmpegcolorspace ! pngenc ! filesink location=/tmp/test.png", 0);

let ret = gst_element_set_state(pipeline, GST_STATE_PLAYING);

if (ret == GST_STATE_CHANGE_ASYNC) { // wait for state change to happen

  ret = gst_element_get_state(pipeline, 0, 0, -1);

  gst_element_set_state(pipeline, GST_STATE_NULL);

  gstreamer.close();

}

And that's all ! Less than 21 lines of javascript.

When the API sends or expects a pointer to an element, we use an opaque int32_t datatype. This is a hack, but future version of js-ctypes are expected to have better type support.

Even if in our case this is not as good as having real support, this shows that js-ctypes is part of the hackability of the browser. For instance, you can use it in Jetpack...

Welcome !

Fabrice — Wed, 02 Sep 2009 18:04:00 +0000

Hello world ;-)