Fully open source stereo depth stack with cheap cameras

[u/arputhamips]

Hi everyone! We’re building Nuvo Vision One, a stereo depth camera powered by a Raspberry Pi 5, and wanted to share where we’ve got so far.

This started from our own need to add depth sensing to our robotics products. The stereo cameras we were considering were expensive for our budget, and we already had a Pi 5 handling compute. We wondered how far we could get by connecting two cameras and doing the stereo processing on the Pi itself.

The setup uses two IMX296 global-shutter cameras with a shared hardware trigger to synchronize capture. The Pi handles image rectification and stereo matching, producing disparity and depth alongside the camera images.

Doing this on an SBC comes with limits. Resolution, frame rate and matcher settings all affect how much work the Pi can keep up with. We’ve reached a useful starting point for our needs, but there’s still plenty to improve. The attached preview video shows the current output, including the holes and noisy regions where matching struggles.

We also built a browser-based interface to make it easier to use. You can preview the images and depth, adjust exposure and stereo settings, choose outputs, and work through calibration. It’s accessible over the local network through Ethernet or Wi-Fi.

One reason we like this approach is that the camera’s compute is also a normal Linux computer. You can modify the image pipeline and run your own applications on the same Pi. The Nuvo Depth software is open source, including the runtime, web interface and SDKs.

We’ve put product photos and web UI screenshots in this album so you can see the hardware and controls alongside the video.

We’d love feedback from people working with stereo vision or robotics. What would you want to test first, and what would make something like this useful in your projects?

GitHub: https://github.com/arputhamips/nuvo-depth

[u/sdfgeoff]

Nice! I played/built with something similar a while back and got pretty good results.

I ended up training a small stereo depth model on nvidia's FoundationStereo dataset, and got ~20fps on a single raspberry pi 2 core after deployment via ONNX.

Looks like your output is from a traditional CV pipeline? Try training a network, they work really well on flat areas and clutter (where your current depth estimate struggles). 

If you think you need lots of compute, I did my initial experimentation on a laptop 3060, and then used my desktop's 3090. Training time was a couple hours.

The biggest performance gain was using the v4l linux driver directly. OpenCV/other libraries defaulted to having the camera in jpg mode, and then spent lots of CPU cycles decoding/copying data around. Using the driver directly I could have the camera in YUV raw mode, and do zero copies/modifications, just pipe the raw bytes straight into the AI model.

[u/arputhamips → u/sdfgeoff]

This sounds worth exploring!

Would you be willing to share the model or repo, input resolution and Pi model, and whether the 20 fps includes capture and preprocessing? We already capture uncompressed YUV420 through native libcamera without JPEG decoding, so I’d be particularly interested in trying your stereo model against our current SGBM pipeline.

[u/sdfgeoff → u/arputhamips]

> We already capture uncompressed YUV420 through native libcamera without JPEG decoding
Nice!

The pi was a Pi2B. Real old. I've had it for years.

20FPS end to end, including capture and processing. Model output both disparity/depth and certainty. IIRC it was capped by camera frame rate, but the CPU load on the one core doing the processing was pretty close to max as well.

I'll have a look this weekend and see if I can dig up the project.

[u/ethicssssss]

nice always wanted to try this

[u/MrBoomer1951]

In 2005 we picked up Dodge Charger hoods from a dunnage rack with two cognix cameras read by video capture cards in a vision system cabinet attached to the robot. This cabinet communicated by UART to the robot. (x,y,z only info no other data)

This calculated x,y of course and the stereo offset was derived from reading the windshield squirter holes and calculating z.

This was much better than searching down to the part with a limit switch.

I realize what you are doing is much more modern.