Interactive Data Exploration for Computer Vision Projects with Rerun

tools for data visualization and exploration have allowed me to iterate way quicker on my computer vision projects, especially when the problems I am faced with are not straightforward and I need to make algorithm or design decisions based on dynamic time-varying signals. These signals can often be challenging to analyze by just looking at rapidly changing numbers plotted on the screen or saved in a table.

While working on some of these problems, I explored the built-in interactive elements from OpenCV, but apart from some sliders the options there are very limited, especially when trying to integrate some animated plots. There’s some hacky ways to get dynamic plots from matplotlib into OpenCV, which I explored in this post.

I also explored different UI frameworks like tkinter, which I used in my last project for a sentiment analysis visualization. I built a custom component that allowed me to display dynamic frames. However it still didn’t really feel like the right tool for the task, especially when trying to work with interactive plots.

And then I stumbled upon Rerun. Every once in a while I discover a tool or framework that really excites me, and this was exactly one of those times. Rerun is an open source tool for visualizing data typically found in the robotics domain ranging from simple time series data and static images to complex 3D point clouds, video streams or other types of multi-modal data. The demos look really impressive and the setup and code samples are so simple, I was immediately hooked.

So I decided to rework my ball tracking demo from a previous project and plot the data using rerun. Let me show you how easy it is to use and create interactive applications!

Quickstart

You can install rerun in any of your python projects using pip or uv:

pip install rerun-sdk
uv add rerun-sdk

You can start the viewer after installing the sdk by simply running it from the command line:

rerun

The viewer will be your main window where your experiments will be shown. You can leave it open or close it between your experiments.

To instantiate a rerun viewer from a python script, you need to spawn an instance with an experiment name:

import rerun as rr

rr.init("ball_tracking", spawn=True)

Ball Tracking Demo

Rerun experiment recordings can be saved to and loaded from .rrd recording files. You can download the recording file for the ball tracking demo from here. Press Ctrl + O or select Open... in the menu on the top left of the rerun viewer and load the downloaded recording file.

You will see the ball tracking demo playback once and then the video stops. At the bottom of the viewer, you have the timeline. You can scrub through the timeline by clicking and dragging the handle.

These recording files only contain the data of the experiment including the video, its annotations and the time series of the tracking. The layout of the viewer is stored separately in a .rbl blueprint file. Download the blueprint file for the demo here and open it on top of the existing data file.

Now we have a slightly better overview with the position, velocity and acceleration plots separated and the video prominently centered.

In the video frame you can click on the annotations and in the left Blueprint panel you can hide or show them individually.

Time Series Plots

To analyze a specific plot, you can click on the expand view button at the top right of any window, for example the position plot.

This is a TimeSeriesView. This view can be used to plot data in a 2D chart with the x-axis representing the time domain, in our case the discrete frame index of the video. In this ball tracking demo, we iterate over the video frames in a loop, where we can set the frame index of our timeline explicitly.

frame_index = 0

while True:
    ret, frame = cap.read()
    if not ret:
        break

    frame_index += 1
    if frame_index >= num_frames:
        break

    rr.set_time("frame_idx", sequence=frame_index)

To create the plot for the position, you need to log a Scalar value for the position of the tracked ball at every frame index. In this case after we calculate the position we can simply log it to rerun:

rr.log("ball/position_y", rr.Scalars(pos))

To configure the style of this plot, we also need to log something to the same entity path (ball/position_y), but since the style doesn’t change we can log it once before the loop and supply a static argument.

rr.log(
    "ball/position_y",
    rr.SeriesLines(colors=[0, 128, 255], names="pos y"),
    static=True,
)

To define the axis range that’s visible per default, we need to specify a layout component.

view_pos = rrb.TimeSeriesView(
    origin="ball/position_y",
    axis_y=rrb.ScalarAxis(range=(0, 700)),
)
layout = rrb.Blueprint(view_pos)
rr.send_blueprint(layout)

Video Stream

Similarly we can create a view for the video frames by logging the image to rerun. Since rerun expects an RGB images but OpenCV ues BGR for its color channel ordering, we need to convert the frames from BGR to RGB before passing them to rerun.

frame_rgb: np.ndarray = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
rr.log("frame", rr.Image(frame_rgb))

To add annotations to the image view we need to log spatial elements to a sub path of the specified entity path. For example, we can draw the center of the tracked ball:

rr.log(
    "frame/points",
    rr.Points2D(
        positions=[center],
        colors=[0, 0, 255],
        radii=4.0,
    ),
)

To include the video frame view in the layout, we can use a Spatial2DView node:

view_frame = rrb.Spatial2DView(origin="frame")

Then we can stack the plot from before vertically with the frame view by using a Vertical layout component:

layout = rrb.Blueprint(
    rrb.Vertical(
        view_pos,
        view_frame,
        row_shares=[0.33],
    ),
)
rr.send_blueprint(layout)

The row shares specifies how much each of the rows occupies in percentages. We can omit the second row share entry for the frame view since the shares have to add up to 1.

Limitations

While working on this project, I ran into some limitations of Rerun. In the original project I visualized a prediction of the trajectory at every timestep, but this is currently not possible in a time series view. Also the layouts and configuration of the plotted data is limited, for example there’s no built-in way to draw a circle without fill. But since the project is very actively being developed, there’s a good chance that some of these might be possible in the future.

Conclusion

The developer experience with this tool as a computer vision engineer is extremely nice, the user interface loads almost instantly and the timeline scrubbing can be incredibly helpful for understanding or debugging signals in time series plots or in videos. I will definitely keep using and exploring this project and can only recommend you to try it out for yourself!

---

For more details and the full implementation, checkout the source code of this project under src/ball_tracking/trajectory_rerun.py.

https://github.com/trflorian/ball-tracking-live-plot

---

All visualizations in this post were created by the author.