# Gstreamer C/C++ For more complicated streaming setups, we often resort to a more robust gstreamer solution. To do this, we use the gstreamer C library, where we have full control over the streaming pipeline. The current gstreamer version we support is 1.23, which is the version that is shipped with Nvidia JetPack 6.0 by default. Upgrading gstreamer is possible, though we will only support the version that Nvidia supports. ## Building With CMake The easiest way to build gstreamer applications on the F-11 is to use cmake, which we heavily rely on in our own development. To do this, you can add the following few lines in your `CMakeLists.txt` file to find the required gstreamer libraries: ``` find_package(PkgConfig REQUIRED) pkg_search_module(gstreamer REQUIRED IMPORTED_TARGET gstreamer-1.0>=1.4) pkg_search_module(gstreamer-sdp REQUIRED IMPORTED_TARGET gstreamer-sdp-1.0>=1.4) pkg_search_module(gstreamer-app REQUIRED IMPORTED_TARGET gstreamer-app-1.0>=1.4) pkg_search_module(gstreamer-video REQUIRED IMPORTED_TARGET gstreamer-video-1.0>=1.4) ``` Then linking against the gstreamer libraries is simple: ``` target_link_libraries( PkgConfig::gstreamer PkgConfig::gstreamer-sdp PkgConfig::gstreamer-app PkgConfig::gstreamer-video ) ``` ## Sample Pipeline A gstreamer application is responsible for data manipulation, which may represent audio or video data, through the use of a *pipeline.* A good, though often inadequate, analogy of a gstreamer pipeline is a water pipe. A gstreamer pipeline should be free of leaks, and data flows from sources to sinks. Suppose we take the [second pipeline](/quickstart/video-streaming.md#example-pipelines): ``` gst-launch-1.0 rtspsrc latency=0 location=rtsp://:/ ! rtph265depay ! rtspclientsink location=rtsp://0.0.0.0:8554/forward_endpoint ``` While this may be sufficient for some applications, we often run into situations where error handling must be handled gracefully so that the service is not halted. We can accomplish this in C with just a few steps. At a high level, we will do the following: 1. Initialize gstreamer 2. Create the pipeline elements and configure their settings 3. Link elements that can be statically linked 4. Set up handlers to dynamically link the rest 5. Play the pipeline 6. Listen to the message bus for any messages, warnings, or errors 7. Cleanup ### Initialization Every gstreamer application has to initialize the library, usually passing in command line arguments if they are present (or `NULL` otherwise). ``` gst_init (&argc, &argv); ``` ### Creating the pipeline elements As we will see shortly, there are many callbacks involved with gstreamer. In these callbacks, it is useful to have access to our pipeline elements and any other data we may need. We therefore often organize everything we need into some data struct. In this example, the data struct will only consist of the pipeline elements, as well as the pipeline itself. In general, it may include more information such as certain logical states or references to hidden elements. ``` typedef struct _data_t { GstElement *pipeline; GstElement *source; GstElement *rtph265depay; GstElement *sink; } data_t; ``` Now that we have laid out what our data is, we can start populating it by creating the corresponding elements. The most important element to create is the pipeline itself, which will have all of the elements as its children. We will use [gst\_pipeline\_new](https://gstreamer.freedesktop.org/documentation/gstreamer/gstpipeline.html?gi-language=c#gst_pipeline_new) and pass in the pipeline's name: ``` data_d data; data.pipeline = gst_pipeline_new ("sample-pipeline"); ``` To create the *rtspsrc, rtph265depay,* and *rtspclientsink* elements, we can use a predefined element factory that will produce these elements for us that we can reference by name. We accomplish this using [gst\_element\_factory\_make](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelementfactory.html?gi-language=c#gst_element_factory_make), where we pass in the factory type and the name of the element we will create: ``` data.source = gst_element_factory_make ("rtspsrc", "source"); data.rtph265depay = gst_element_factory_make ("rtph265depay", "rtph265depay"); data.sink = gst_element_factory_make ("rtspclientsink", "sink"); ``` Should an element creation fail, the factory will return `NULL`. Each element should be checked to be non-NULL upon creation. Lastly, we will need to set some properties of these elements, such as the URI of the RTSP server we will publish to. Since gstreamer is built on top of [glib](https://docs.gtk.org/glib/), object properties can be set with [g\_object\_set](https://docs.gtk.org/gobject/method.Object.set.html): ``` g_object_set (data.source, "latency", 0, NULL); g_object_set (data.source, "location", "rtsp://:/", NULL); g_object_set (data.sink, "location", "rtsp://0.0.0.0:8554/forward_endpoint", NULL); ``` These properties directly reflect the properties specified in the second sample pipeline. Lastly, these elements need to all be added to the pipeline object itself. This allows their states to be fully managed by the pipeline, and is required in order to link them. We can do this with [gst\_bin\_add\_many](https://gstreamer.freedesktop.org/documentation/gstreamer/gstbin.html?gi-language=c#gst_bin_add_many): ``` gst_bin_add_many (GST_BIN (data.pipeline), data.source, data.rtph265depay, data.sink, NULL); ``` ### Static Linking In general, gstreamer elements have pads that can be connected to each other. Just like real pipes, the end of one can be connected to the start of another. The source pad of an element, which can be thought of as its *output*, can be connected to the sink of another element, which can be considered its *input*. Elements may have zero or more source pads and zero or more sink pads. When these pads are *always* present, we can use static linking to connect them to each other. We can think of linking as gluing together two pipes at a specific point. Just like pipes can only be connected if their interface matches up, pipeline elements can only be linked if their source and sink pads can agree on the connection type. To check the pad availability of an element, we can use `gst-inspect-1.0`. For example, running `gst-inspect-1.0 rtph265depay` shows us the following: ``` Pad Templates: SRC template: 'src' Availability: Always Capabilities: video/x-h265 stream-format: avc alignment: au video/x-h265 stream-format: byte-stream alignment: { (string)nal, (string)au } ``` Doing the same for the *rtspclientsink* gives us: ``` Pad Templates: SINK template: 'sink_%u' Availability: On request Capabilities: ANY Type: GstRtspClientSinkPad ``` There are two main things to look for, which are the availability and the capabilities of these pads. In order to link two pads, their capabilities have to be in agreement. In this case, the *rtspclientsink* has a sink pad that has the `ANY` capabilities, which means it is compatible with any other pad. We also see that the *rtph265depay* has a *src,* or source, pad that is *Always* available. This means it can be statically linked. The *rtspclientsink* doesn't have a single sink, but rather a sink template. This means that it may have multiple sinks, and that each sink is created *On request*. In this case, we can deduce that static linking of these two elements is possible, and we can accomplish this [gst\_element\_link](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c#gst_element_link): ``` gst_element_link (data.rtph265depay, data.sink) ``` which should return `TRUE` on success (this is a `gboolean`, not to be confused with the standard `true`). ### Dynamic Linking When checking the pads of *rtspsrc* with `gst-inspect-1.0`, we get the following: ``` Pad Templates: SRC template: 'stream_%u' Availability: Sometimes Capabilities: application/x-rtp application/x-rdt ``` This is different than the previous section. This element connects to an RTSP server and starts streaming from it, but it will not create any *stream* pads (we can think of these as the *source* pads for now) until data actually flows through this element. No data will flow through this element until the pipeline is actually playing, which will not happen until we finish setting up the pipeline and setting its state to playing. Therefore, this stream pad will not exist until later on in our execution. Situations like these require dynamic pad linking, which can be done by setting up a new callback to handle this event. ``` g_signal_connect(data.source, "pad-added", G_CALLBACK(pad_added_handler), &data); ``` This line says that when our *rtspsrc* element finally adds a new stream pad, we will call our *pad\_added\_handler*. This handler will be responsible for linking this pad to the pipeline, and can be done as follows: ``` static void pad_added_handler(GstElement* src, GstPad* new_pad, data_t* data) { GstPad *sink_pad = gst_element_get_static_pad(data->rtph265depay, "sink"); GstPadLinkReturn ret; GstCaps *new_pad_caps = NULL; g_print("Received new pad '%s' from '%s':\n", GST_PAD_NAME(new_pad), GST_ELEMENT_NAME(src)); /* Check the new pad's name */ if (!g_str_has_prefix(GST_PAD_NAME(new_pad), "recv_rtp_src_")) { g_print(" It is not the right pad. Need recv_rtp_src_. Ignoring.\n"); goto exit; } /* If our converter is already linked, we have nothing to do here */ if (gst_pad_is_linked(sink_pad)) { g_print(" Sink pad from %s already linked. Ignoring.\n", GST_ELEMENT_NAME(src)); goto exit; } /* Attempt the link */ ret = gst_pad_link(new_pad, sink_pad); if (GST_PAD_LINK_FAILED(ret)) { g_print("Link failed.\n"); } else { g_print("Link succeeded.\n"); } exit: /* Unreference the sink pad */ gst_object_unref(sink_pad); } ``` There are a lot of things that happen in this function, but there are only two main ideas that we need to cover. This function is only called when the *rtspsrc* elements adds a new pad, and this pad is available for us as a parameter. First, this function grabs the sink pad of the *rtph265depay*, and then it attempts to link these two pads directly. Note that previously we linked elements, but here we use [gst\_pad\_link](https://gstreamer.freedesktop.org/documentation/gstreamer/gstpad.html?gi-language=c#gst_pad_link) to link the specific pads. The rest of the function simply checks that we are looking at the correct pad, that it has not been linked yet, and cleans up. ### Playing the Pipeline Now that we have a pipeline that is built, linked, and ready for dynamic linking, we can actually play the pipeline. This will kickstart all our processes and should start pulling the stream from the endpoint. ``` GstStateChangeReturn ret = gst_element_set_state (data.pipeline, GST_STATE_PLAYING); if (ret == GST_STATE_CHANGE_FAILURE) { g_print ("Unable to set the pipeline to the playing state.\n"); gst_object_unref (data.pipeline); return -1; } ``` ### Message Bus ``` GstBus *bus; GstMessage *msg; // 1 bus = gst_element_get_bus (data.pipeline); while (true) { // 2 msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, static_cast(GST_MESSAGE_ERROR)); /* Parse message */ if (msg != NULL) { GError *err; gchar *debug_info; switch (GST_MESSAGE_TYPE (msg)) { case GST_MESSAGE_ERROR: { // 3 gst_message_parse_error (msg, &err, &debug_info); g_printerr ("Error received from element %s: %s\n", GST_OBJECT_NAME (msg->src), err->message); g_printerr ("Error code: %i\n", err->code); g_printerr ("Error domain: %i\n", err->domain); /* * Gracefully handle error if possible */ g_clear_error (&err); g_free (debug_info); break; } default: { g_print("Received message of type: %s\n", gst_message_type_get_name(GST_MESSAGE_TYPE(msg))); break; } } gst_message_unref (msg); } } ``` With gstreamer, a pipeline has a message bus that transmits all of the messages of its children elements. In this case, all three of our pipeline elements will send their messages through this bus. We can easily listen to this bus by first referencing this bus with line 1. Next, we have to specify what type of messages we are interested in listening to. In our simple case, we will only listen to errors, though there are many [message types](https://gstreamer.freedesktop.org/documentation/additional/design/messages.html?gi-language=c#message-types) to listen to. We continuously monitor the bus for new messages, and when we finally get a new message we can handle it. There are other methods of listening to messages, this is a very simple approach. In line 3, we finally see a new error message. We can parse this message to get more information, such as the type of error and the domain of the error, and we can then try to gracefully handle the error. ### Cleaning Up Lastly, we have to clean up the resources we used. In our case, we have the bus and the pipeline to clean up. Before cleaning up a pipeline element, we have to set its state to `GST_STATE_NULL`, as failing to do so will not allow us to clean up our resources. ``` gst_object_unref (bus); gst_element_set_state (data.pipeline, GST_STATE_NULL); gst_object_unref (data.pipeline); ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.flybydev.com/quickstart/video-streaming/gstreamer-c-c++.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.