Gstreamer C/C++

A brief introduction to developing a gstreamer application in C/C++ to run on the F-11.

For more complicated streaming setups, we often resort to a more robust gstreamer solution. To do this, we use the gstreamer C library, where we have full control over the streaming pipeline. The current gstreamer version we support is 1.16.3, which is the version that is shipped with Nvidia JetPack 5.1.2 by default. Upgrading gstreamer is possible, though we will only support the version that Nvidia supports.

Building With CMake

The easiest way to build gstreamer applications on the F-11 is to use cmake, which we heavily rely on in our own development. To do this, you can add the following few lines in your CMakeLists.txt file to find the required gstreamer libraries:

find_package(PkgConfig REQUIRED)
pkg_search_module(gstreamer REQUIRED IMPORTED_TARGET gstreamer-1.0>=1.4)
pkg_search_module(gstreamer-sdp REQUIRED IMPORTED_TARGET gstreamer-sdp-1.0>=1.4)
pkg_search_module(gstreamer-app REQUIRED IMPORTED_TARGET gstreamer-app-1.0>=1.4)
pkg_search_module(gstreamer-video REQUIRED IMPORTED_TARGET gstreamer-video-1.0>=1.4)

Then linking against the gstreamer libraries is simple:

target_link_libraries(<TARGET>
    PkgConfig::gstreamer
    PkgConfig::gstreamer-sdp
    PkgConfig::gstreamer-app
    PkgConfig::gstreamer-video
)

Sample Pipeline

A gstreamer application is responsible for data manipulation, which may represent audio or video data, through the use of a pipeline. A good, though often inadequate, analogy of a gstreamer pipeline is a water pipe. A gstreamer pipeline should be free of leaks, and data flows from sources to sinks.

Suppose we take the second pipeline:

gst-launch-1.0 rtspsrc latency=0 location=rtsp://<endpoint_ip>:<port>/<endpoint> ! rtph264depay ! rtspclientsink location=rtsp://0.0.0.0:8554/forward_endpoint

While this may be sufficient for some applications, we often run into situations where error handling must be handled gracefully so that the service is not halted. We can accomplish this in C with just a few steps. At a high level, we will do the following:

1. Initialize gstreamer

2. Create the pipeline elements and configure their settings

3. Link elements that can be statically linked

4. Set up handlers to dynamically link the rest

5. Play the pipeline

6. Listen to the message bus for any messages, warnings, or errors

7. Cleanup

Initialization

Every gstreamer application has to initialize the library, usually passing in command line arguments if they are present (or NULL otherwise).

gst_init (&argc, &argv);

Creating the pipeline elements

As we will see shortly, there are many callbacks involved with gstreamer. In these callbacks, it is useful to have access to our pipeline elements and any other data we may need. We therefore often organize everything we need into some data struct. In this example, the data struct will only consist of the pipeline elements, as well as the pipeline itself. In general, it may include more information such as certain logical states or references to hidden elements.

typedef struct _data_t {
    GstElement *pipeline;
    GstElement *source;
    GstElement *rtph264depay;
    GstElement *sink;
} data_t;

Now that we have laid out what our data is, we can start populating it by creating the corresponding elements. The most important element to create is the pipeline itself, which will have all of the elements as its children. We will use gst_pipeline_new and pass in the pipeline's name:

data_d data;
data.pipeline = gst_pipeline_new ("sample-pipeline");

To create the rtspsrc, rtph264depay, and rtspclientsink elements, we can use a predefined element factory that will produce these elements for us that we can reference by name. We accomplish this using gst_element_factory_make, where we pass in the factory type and the name of the element we will create:

data.source = gst_element_factory_make ("rtspsrc", "source");
data.rtph264depay = gst_element_factory_make ("rtph264depay", "rtph264depay");
data.sink = gst_element_factory_make ("rtspclientsink", "sink");

Should an element creation fail, the factory will return NULL. Each element should be checked to be non-NULL upon creation.

Lastly, we will need to set some properties of these elements, such as the URI of the RTSP server we will publish to. Since gstreamer is built on top of glib, object properties can be set with g_object_set:

g_object_set (data.source, "latency", 0, NULL);
g_object_set (data.source, "location", "rtsp://<endpoint_ip>:<port>/<endpoint>", NULL);
g_object_set (data.sink, "location", "rtsp://0.0.0.0:8554/forward_endpoint", NULL);

These properties directly reflect the properties specified in the second sample pipeline.

Lastly, these elements need to all be added to the pipeline object itself. This allows their states to be fully managed by the pipeline, and is required in order to link them. We can do this with gst_bin_add_many:

gst_bin_add_many (GST_BIN (data.pipeline), data.source, data.rtph264depay, data.sink, NULL);

Static Linking

In general, gstreamer elements have pads that can be connected to each other. Just like real pipes, the end of one can be connected to the start of another. The source pad of an element, which can be thought of as its output, can be connected to the sink of another element, which can be considered its input. Elements may have zero or more source pads and zero or more sink pads. When these pads are always present, we can use static linking to connect them to each other. We can think of linking as gluing together two pipes at a specific point. Just like pipes can only be connected if their interface matches up, pipeline elements can only be linked if their source and sink pads can agree on the connection type.

To check the pad availability of an element, we can use gst-inspect-1.0. For example, running gst-inspect-1.0 rtph264depay shows us the following:

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: avc
              alignment: au
      video/x-h264
          stream-format: byte-stream
              alignment: { (string)nal, (string)au }

Doing the same for the rtspclientsink gives us:

Pad Templates:
  SINK template: 'sink_%u'
    Availability: On request
    Capabilities:
      ANY
    Type: GstRtspClientSinkPad

There are two main things to look for, which are the availability and the capabilities of these pads. In order to link two pads, their capabilities have to be in agreement. In this case, the rtspclientsink has a sink pad that has the ANY capabilities, which means it is compatible with any other pad. We also see that the rtph264depay has a src, or source, pad that is Always available. This means it can be statically linked. The rtspclientsink doesn't have a single sink, but rather a sink template. This means that it may have multiple sinks, and that each sink is created On request. In this case, we can deduce that static linking of these two elements is possible, and we can accomplish this gst_element_link:

gst_element_link (data.rtph264depay, data.sink)

which should return TRUE on success (this is a gboolean, not to be confused with the standard true).

Dynamic Linking

When checking the pads of rtspsrc with gst-inspect-1.0, we get the following:

Pad Templates:
  SRC template: 'stream_%u'
    Availability: Sometimes
    Capabilities:
      application/x-rtp
      application/x-rdt

This is different than the previous section. This element connects to an RTSP server and starts streaming from it, but it will not create any stream pads (we can think of these as the source pads for now) until data actually flows through this element. No data will flow through this element until the pipeline is actually playing, which will not happen until we finish setting up the pipeline and setting its state to playing. Therefore, this stream pad will not exist until later on in our execution. Situations like these require dynamic pad linking, which can be done by setting up a new callback to handle this event.

g_signal_connect(data.source, "pad-added", G_CALLBACK(pad_added_handler), &data);

This line says that when our rtspsrc element finally adds a new stream pad, we will call our pad_added_handler. This handler will be responsible for linking this pad to the pipeline, and can be done as follows:

static void pad_added_handler(GstElement* src, GstPad* new_pad, data_t* data)
{
    GstPad *sink_pad = gst_element_get_static_pad(data->rtph264depay, "sink");
    GstPadLinkReturn ret; 
    GstCaps *new_pad_caps = NULL;

    g_print("Received new pad '%s' from '%s':\n", GST_PAD_NAME(new_pad), GST_ELEMENT_NAME(src));

    /* Check the new pad's name */
    if (!g_str_has_prefix(GST_PAD_NAME(new_pad), "recv_rtp_src_")) {
        g_print("  It is not the right pad.  Need recv_rtp_src_. Ignoring.\n");
        goto exit;
    }

    /* If our converter is already linked, we have nothing to do here */
    if (gst_pad_is_linked(sink_pad)) {
        g_print(" Sink pad from %s already linked. Ignoring.\n", GST_ELEMENT_NAME(src));
        goto exit;
    }

    /* Attempt the link */
    ret = gst_pad_link(new_pad, sink_pad);
    if (GST_PAD_LINK_FAILED(ret)) {
        g_print("Link failed.\n");
    } else {
        g_print("Link succeeded.\n");
    }
exit:
    /* Unreference the sink pad */
    gst_object_unref(sink_pad);
}

There are a lot of things that happen in this function, but there are only two main ideas that we need to cover. This function is only called when the rtspsrc elements adds a new pad, and this pad is available for us as a parameter. First, this function grabs the sink pad of the rtph264depay, and then it attempts to link these two pads directly. Note that previously we linked elements, but here we use gst_pad_link to link the specific pads. The rest of the function simply checks that we are looking at the correct pad, that it has not been linked yet, and cleans up.

Playing the Pipeline

Now that we have a pipeline that is built, linked, and ready for dynamic linking, we can actually play the pipeline. This will kickstart all our processes and should start pulling the stream from the endpoint.

GstStateChangeReturn ret = gst_element_set_state (data.pipeline, GST_STATE_PLAYING);

if (ret == GST_STATE_CHANGE_FAILURE) {
    g_print ("Unable to set the pipeline to the playing state.\n");
    gst_object_unref (data.pipeline);
    return -1;
}

Message Bus

GstBus *bus;
GstMessage *msg;

// 1
bus = gst_element_get_bus (data.pipeline);

while (true) {
    // 2
    msg =
        gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE,
        static_cast<GstMessageType>(GST_MESSAGE_ERROR));

    /* Parse message */
    if (msg != NULL) {
        GError *err;
        gchar *debug_info;

        switch (GST_MESSAGE_TYPE (msg)) {
            case GST_MESSAGE_ERROR: {
                // 3
                gst_message_parse_error (msg, &err, &debug_info);
                g_printerr ("Error received from element %s: %s\n", GST_OBJECT_NAME (msg->src), err->message);
                g_printerr ("Error code: %i\n", err->code);
                g_printerr ("Error domain: %i\n", err->domain);
                /* 
                 * Gracefully handle error if possible
                 */
                g_clear_error (&err);
                g_free (debug_info);
                break;
            }
            default: {
                g_print("Received message of type: %s\n", gst_message_type_get_name(GST_MESSAGE_TYPE(msg)));
                break;
            }
        }
        
        gst_message_unref (msg);
    }
}

With gstreamer, a pipeline has a message bus that transmits all of the messages of its children elements. In this case, all three of our pipeline elements will send their messages through this bus. We can easily listen to this bus by first referencing this bus with line 1. Next, we have to specify what type of messages we are interested in listening to. In our simple case, we will only listen to errors, though there are many message types to listen to. We continuously monitor the bus for new messages, and when we finally get a new message we can handle it. There are other methods of listening to messages, this is a very simple approach. In line 3, we finally see a new error message. We can parse this message to get more information, such as the type of error and the domain of the error, and we can then try to gracefully handle the error.

Cleaning Up

Lastly, we have to clean up the resources we used. In our case, we have the bus and the pipeline to clean up. Before cleaning up a pipeline element, we have to set its state to GST_STATE_NULL, as failing to do so will not allow us to clean up our resources.

gst_object_unref (bus);
gst_element_set_state (data.pipeline, GST_STATE_NULL);
gst_object_unref (data.pipeline);