Basis Robotics

uvtarget - a helper for bundling python in CMake

2025-07-06T00:00:00+00:00

I’ve released today a tool uvtarget - used to link together multiple python projects into one workspace with a single lockfile, with CMake. It’s available here on GitHub.

This took a few weeks of various evenings and weekend afternoons to make, which feels crazy for something that’s not even 500 lines of code. Difficulty was mainly knowing what lines to write and figuring out how uv wanted things to be handled. There were a lot of small cases around making sure the environment CMake was being invoked in didn’t leak into the compilation environment, as well as handling subtle differences between project, workspace and virtual environment.

FAQ

Q. Why?

A. Basis uses CMake as a lowest common denominator build system. I’m still in the process of integrating Python into basis. I had two problems: (1) I need to be able to pull in arbitrary Python versions into the build environment. (2) I need to easily package up the build environment into some kind of install space, and make sure everything comes along for the ride. Uv is an easy win for this, it just needs some work to integrate with CMake, especially for workflows that end up calling sudo make install.¹.

Q. Why create a workspace?

A. In a robotics context, one might have several related python projects that need installed into the same image, that are being developed in parallel. Regardless of the structure we give the projects (one package, multiple packages, etc), if we run tests and simulations on one set of dependencies, we need to also ship the codebase on that same set of dependenies.

Q. Why do we need a meta pyproject?

A. One really can’t have multiple lock files for one environment, so this means one pyproject (1:1 relationship). The alternative is multiple environments, but in my experience this is far more trouble than its worth.

Q. Why autogenerate?

A. For now, basis works on a single CMakeList source tree. Adding additional libraries or units that contain python shouldn’t require doing anything more than adding them to your CMake tree - managing an additional parallel tree of dependencies isn’t great. Along with that, we shouldn’t inherit some other library’s lockfile - we should take the constraints but not the pin. Autogeneration can easily be opted out of, if you prefer to do things the other way around (manually track the libraries going into the venv).

Q. Will this work with ROS?

A. Yes, it should - with some caveats. Instructions are for catkin tools. Under catkin_make this should actually work “perfectly” but I really don’t recommend it, catkin tools is faster and has better isolation. I don’t know the best way to do this under colcon, but it should also be possible (one might want).

Create a package pybase or similar, inside that package point at each of your other python packages with uv_add_pyproject(../other_package) or with whatever path resolution you prefer.
When you call uv_initialize make sure you use UNMANAGED_PYPROJECT_FILE
If other packages depend on having a working python environment (due to needing to use an installed package), make sure they gave catkin depends on pybase.
Inside the venv that uvtarget creates, install a .pth file pointing at each directory containing ROS python that needs to be imported into the venv. (this can be done in CMake)
When you source catkin’s setup.bash, call export PYTHONPATH="" && source build/path_to_pybase/.env/bin/activate to get an environment with your packages and not the system packages. YMMV, you may need to recompile things like moveit or tf2.

Q. What about (compiled) extension modules?

A. I haven’t tried creating extension modules yet - but this should work fine. You may or may not need to add a dependency on uv_sync to your extension builds, otherwise the sync may fail. Happy to work with you for a solution if you’re interested.

How does it work?

cmake …

Near the root of your project, uv_initialize(...) is called, setting up your environment’s settings. This sets up some variables in CMake and sets up a hook to run at the end of the CMake generation step. uv venv --python $version is called to create the virtual environment for development.

Each time you call uv_add_pyproject or uv_add_dev_dependency, the arguments are stored off for later.

Finally, when the deferred hook is called, we create a pyproject depending on all of the earlier declared dependencies.

make

The cmake step adds up a build target that calls uv sync with the correct environment and arguments to sync the environment to the workspace. We call this on every build, uv is so fast that you don’t notice in the cases where nothing has changed. An “optimization” would be to only call uv sync when a pyproject has changed, but that’s just duplicating the logic uv has internally.

make install

We call uv export with a bunch of flags to generate a requirements.txt for just the dependencies. Then uv build to build wheels for each workspace member. Finally, we install them in one big swoop with uv pip install.

There are a bunch of options that could be supported for installation that aren’t currently implemented. Wheel+requirements only export, installing to a non virtual environment, exporting to an installer, etc. The wheel+requirements flag is probably the most useful for alternative workflows.

Where are you going from here?

With uvtarget

Nothing immediately planned, but happy to accept feature requests and PRs.

With basis

I’m finally on to working on subinterpreter support, allowing for loading arbitrary numbers of Python modules in one process, along side C++ (and someday Rust).

I really should fix the directory ownership in basis to not use root owned installs. It’s unneeded, but OTOH it’s nice to at least support it as an option. ↩

Single process, multiple interpreters, no GIL contention - pre-Python3.12

2025-05-26T00:00:00+00:00

First off…

What’s up with Basis?

Dunno. I’ve spent the past few months working for money rather than for “free”¹. I feel like I’m working more hours for them as I was on Basis, but the distribution is a little bit different. I also had a 35 year old Marshall Amp land in my lap², so I’ve been learning guitar. That being said: I still like working with the framework, and I’ll probably keep using it for my own projects. If nobody uses it, oh well. I need to update the website and license to reflect the current state of things, but there’s no hurry. In any case I’m going to use this as a space to write about some of the robotics/C++/tech stuff I deal with. I don’t think Thomas will mind, anyhow. I have some fun stuff from my day job about how running stat on a certain directory can cause all networking (really, anything using hard IRQs) to hang for dozens of millseconds.

After driving to South Bay and back several times in a week, I let my mind wander a bit for how I would make Python work in Basis. I came up with a pretty good scheme for the serialization (probably can get away with just reserializing once for other languages), but there’s one big problem - how would I make stuffing multiple interpreters into one process work? Using subinterpreters would probably work, but restrict use to 3.12+. While I think a lot of people will want to use the latest and greatest Python, in practice it’s easy to get stuck on an older version due to dependencies. Particularly, I’d like to import rospy message definitions without having to worry about mucking around with PYTHONPATH to manually add them in. Surely there has to be a way of doing this, right?

A quick rundown on Basis’s architecture

Basis loads everything up into the same process space, unless you opt out.³ My not so secret belief is that “shared memory” is actually a bit of a scam, and that you can go even faster if you put everything into the same process space and share shared_ptr around. True zero copy without needing to manage shared memory segments. In my mind, a robot should probably have around 3 or 4 processes running “robotics” code. One for your perception pipeline, one for your planning pipeline, one for your low level controls/safety, and one for everything else dealing with telemetry, visualization, etc. ROS1 can do this with nodelets, though the API for them is poor. ROS2 does this with “components” - this actually looks like a pretty neat API, good on them. One thing Basis can do that others cannot, as far as I can tell is run without a Coordinator (ROS Master, if you lean that way) if you have only one process in your launch file.

flowchart TB
    Coordinator-->p1
    Coordinator-->p2
    
    subgraph p1 [process 1]
        direction TB
        realsense_capture
        perception_scene
    end

    subgraph p2 [process 2]
        direction TB
        ll_controls
    end

^{(side note: I’m noticing now that these diagrams are cut off on Safari - if you have ideas why, lemme know)}

This presents a bit of a problem for Python, though. Python (pre 3.12) does not like being run as multiple instances in one process. It just straight out doesn’t work - everything runs on the same runtime, you can’t call Py_Initialize again to get a second runtime. You can run a separate interpreter using Py_NewInterpreter but this is mainly for getting a new environment⁴, all interpreters still hold the same GIL. For a large python robotics codebase, this would mean horrible contention between interpreters.

I was able to “fix” this, on Python3.8. As far as I know, nobody has ever done this before, but would be very interested in prior art on this. (I found this gist saying it sorta worked, but broke with multithreading). I found another post on HN from girfan/Gohar Chaudhry that mentions doing the same. I’ve reached out to him to see if he was able to get a fully working solution or not.

Disclaimer: I’m not a Python expert, nor a Linux expert⁵, and I’m especially not a Python internals expert. I recommend not trying to do anything in this post, except for educational purposes.

Disclaimer 2: This is a somewhat meandering post, if you’re interested in the actual way to do this, go to the TLDR

The long road to a solution

Getting it to work, sorta

I started off trying just some simple test code in a Unit using pybind11 was essentially.

  pybind11::scoped_interpreter py;
  pybind11::print("Hello, World!");

Sure enough, running one copy printed Hello, World! but loading the Unit twice via a launch file

---
# this launch file loads up the same Unit (shared object) twice, with two different names and sets of arguments
units:
  py_a:
    unit: pybind_test
    args:
      pub: True
  py_b:
    unit: pybind_test
    args:
      pub: False

got me this error

[12255.044380416] [launch] [info] Started thread with unit /opt/basis/unit/pybind_test.unit.so
terminate called after throwing an instance of 'std::runtime_error'
  what():  The interpreter is already running

This is entirely expected, and documented in both pybind and Python.

The initial “fix”

Basis Units are loaded via dlopen, essentially as plugins. If you’ve never seen a call to dlopen before, it looks like this.

// For now - need to use RTLD_GLOBAL to allow different inproc transports to communicate
void *handle = dlopen(path.c_str(), RTLD_NOW | RTLD_GLOBAL );

This call takes a path to a shared object and loads it into a handle that one can pull symbols from.

If you’re interested in how dlopen is used and how to load an arbitrary function for a shared object, expand this

Basis is shutting down

2025-01-08T00:00:00+00:00

Basis as a company is shutting down. Thomas and I gave it a good shot but - we have no funding, and after our release, not one person tried our software.

Good software, bad company

Building backwards

What happened? Basically, what we have are the pretty good bones of a distributed framework, but there’s no way we can see this working economically. I built everything backwards - I had an idea that I wanted to build, built it, and then tried to get others interested. That’s fine for OSS (I’m happy I built it, and I’ll use it), but not great for sustainting a business.

One of the VCs we talked to specifically asked where our plan for building a community was. At the time I didn’t get it (why would I waste runway on that?! who would join a community for a thing they couldn’t try?), but now I see that he was looking for some form of validation - I’d assumed that releasing the framework would get people to try it, rather than paving the way for success first. Conventional startup wisdom seems to be to find PMF first, then build. We did the reverse. (Matt! You were right, we just didn’t get it at the time.)

Focusing on the wrong thing

Late in the process, we realized that we were somewhat barking up the wrong tree. We’d focused very much on deterministic replay and testability as the framework’s key features. In our experience looking at larger robotics companies, these are important to have. However - our initial customers wouldn’t care about those things. They care about how quickly they can get off the ground and get features out the door.

There’s a reason people use ROS - it’s really great for getting your robot prototyped. We could have focused our “marketing” on doing that, but even better - this was an area we are better than ROS in, in several ways (not having to use a custom build system, easily supporting new platforms, abstracting away system software concerns from your roboticists, etc). This was another bit of feedback we got from a VC - they recognized that ROS is a problem, but didn’t see how believe it was slowing down their robotics companies. We should have swapped our marketing much sooner, especially as we got several comments on release that “something easier get started with than ROS” was a great idea.

Difficult to capture initial customers

There’s another problem here - who are our initial customers? We’d had two ideal companies in mind: fresh companies with not much code, and companies stuck on ROS1 but not yet moved to ROS2. In retrospect, these are difficult customers to find.

The first group isn’t super loud about existing, and moreover, they are busy and unlikely to want to bet their company on new tech, unless they really know what they want (in our entire time of searching, we found one of these companies, who is/was still a promising lead as of this post, but see the next section for how this doesn’t help…).

The latter group is just too damn hard to capture. Ignoring the fact they have a lower tolerance for missing features, they more than likely have started to swap to ROS2 anyhow. They won’t want to roll back that move, and aren’t likely to want to immediately move a second time after making the jump. We talked with a few companies still on ROS1 and “happily” there - if fit their needs, they were fine with eating the maintinence cost of a fork, and didn’t need anything we could provide. Beyond all this, it’s a huge ask to get companies to just try our stuff out. Robots are pretty specialized, and we don’t have a deep enough back cataloge of modules to be able to just toss some drivers and a planning algorithm together and get a quick prototype working for a new user wanting to try things. We probably could have done better here, and made more compelling demos.

Economics

Lastly - this whole thing never actually made sense from an economic perspective. The whole idea was that companies should be willing to pay for a better middleware. So far…I haven’t seen any evidence of it, though. People are willing to pay for a lot of things that have free alternatives, and willing to pay for robotics software, too. Foxglove appears to be succeeding in the visualization/data management space. For some reason, people are willing to throw millions a year at Applied Intuition for simulation.

Why can’t middleware be the same? We theorized that using Basis could save at least a system software hire, that has to be worth something, right? Turns out, not so much. I have some theories around this, but I’m guessing it’s just too scary to move to a new framework, even for free. We had one friend in the industry suggest that companies might pay $1MM/year for a performance focused framework (I wish it were true, but nobody has come up to me with promises saying they would). Our best plan was to get a little bit of money short term in (discounted) contract integration work, and then make it up years down the line with some sort of game-engine-like fee structure. This still has a few big holes:

How do we structure the contract/license so that customers are protected from Basis Robotics having bad behavior down the line? Putting on my customer hat, I’d be very afraid of basing my business on someone else’s platform - they can screw me later and there’s nothing I can do! This happens a fair amount in the mobile space w/Apple and Google. Those companies deal with it, but smaller developers often feel the pain.
How is pricing scaled? If it’s based on revenue, we lose out on all money from the next Cruise or other company that has a ton of money, but makes zero revenue. If it’s based on robot count or developer count, it’s very hard to make a one size fits all contract. This sucks from the point of view of having clear pricing.

Even if we find a price structure and contract that fits, the math on profit is grim. And given we don’t have evidence people will pay in the short term, that rules out bootstrapping.

What’s happening to the framework?

Basis as a piece of software will continue to exist, as a hobby/Open Source project. I’ll make the license plain Apache 2.0, and open the determinsitic_replay repo that makes deterministic playback possible (and eventually pull it forward to be compatible with main, integrate it within the basis repo, etc, etc). I’ll still work on my little test robot, for fun. The next thing to do is to integrate the LIDAR I bought and put some SLAM algo on top - there’s a few out there, but once the SLAM Handbook releases Part 2, I might try to do my own.

Why didn’t we…

Get Angel funding?

Angel funding won’t solve the user problem. Again, not a single person tried our software after we released it. Maybe we could have done so and paid a marketing genius to help us out. Beyond that, if we’re going to get funding I want enough that I can draw a small salary to pay rent. Before that it’s not really worth it, if we had enough traction I’d much rather have borrowed/gotten investment from friends/family. The numbers are low enough for angel funding that I can make it back in a short time at a FAANG or AI company to make them whole, if we borrowed and busted.

Pivot?

We could certainly pivot to making a robot, instead. I have a few ideas (putting it out there - some sort of Foundational Model based handyman robot with interchangable hands/fingers will come some day, I think). I think there’s also space for improvement for robotics deployment and telemetry (though the space isn’t as empty as middleware was). But I’m really loathe to spend another three to six months grinding and fundraising (again) - my savings are drained (they were low already due to some poor choices with stock options from Embark Trucks). We aren’t going to get kicked out of our apartment or anything - my wife has savings and income, but I need a good reason to start dipping into her savings as well.

Build on top of ROS?

I don’t want to. I don’t like their development model: everything is way too distributed, designed by committee, it’s impossible to know who really owns what or what is good quality. My PR to tf2 removing extra Matrix->Quaternion conversions still hasn’t merged…. Beyond that, I don’t think the user story really makes sense. The static analysis and replay benefits that Basis has won’t work for things like MoveIt or Nav2 - they are too large, I’d never convince them to use my bindings on top, and the API surface is too large to cover as a fork. It might have been a solid strategy to start with (one of the YC partners asked why we weren’t), but at this point - too late. Maybe someday I’ll go back and add a ROS backend to the code generator.

Integrate AI?

This would have probably been a solid play to get VC money, if I were more dishonest. I did wrack my brain for how AI could help, and at best I could come up with is “declaring all of each unit’s inputs and outputs makes it easier for AIs to understand the codebase”, which is true for humans as well. I think there is probably a need for a good production framework for working with Foundational Models, but I’d need to invest in more hardware and pay a platform to train models on (as well as skilling up on modern AI, which I suppose I’ll have to do eventually, anyhow).

Conclusion

I learned a hell of a lot out of this. It was even more stressful than I expected, and we never even had money change hands. I went from being super nervous at every pitch to being pretty relaxed and happy to talk about what we were building. Got a whole lot better at networking at conventions, gave my first talk in front of other companies at an event. I got to set up the bones of a company from scratch. I got to get the idea that’s been knocking around my head for the past three or four years out of it and into code. I feel lucky that I had the opportunity for all of this and I’m not unhappy for trying.

Given the opportunity, I’d do it again.

What’s next?

It sounds like Thomas is taking some time off - he deserves it. It’s been a fun time, and I’m glad to have had him at my side.

I’m now open to contract and full time work. You can see the entire corpus of what I’ve worked on over the past 7 months on our github (please keep in mind that normally I’d do a bit better with tests/docs!). I’d love to stay in robotics but open to other areas as well. Please reach out at kyle@basisrobotics.tech

Building a Robot on Basis 02 - Software

2024-11-24T00:00:00+00:00

I’ve spent the past few weeks working on a small robot to both be able to give demos with and exercise our code. This post is about the current software architecture for the robot.

Part 01 - Hardware
Part 02 - Software (You’re here!)
Part 03 - tf2 support and LiDAR

After getting the hardware working, I moved on to the software. This required a few small changes to the core framework (mostly fixing CMake technicalities), but nothing crazy.

I can now move the robot around with an wireless controller! The left stick and bumpers control the wheels and the right stick controls the servos.

The Architecture:

---
config:
  flowchart:
    nodeSpacing: 5
    subGraphTitleMargin:
      bottom: 10
  defaultRenderer: elk
  elk:
      mergeEdges: True
---
graph LR
  %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
  subgraph unit_/freenove/rpi_freenove_mecanum_driver["/freenove/rpi_freenove_mecanum_driver"]
    handler_/freenove/rpi_freenove_mecanum_driver::Update[["Update()
100Hz"]]
  end

  subgraph unit_/freenove/rpi_freenove_servo_driver["/freenove/rpi_freenove_servo_driver"]
    handler_/freenove/rpi_freenove_servo_driver::Update[["Update()
100Hz"]]
  end

  subgraph unit_/freenove/rpi_libcamera_driver["/freenove/rpi_libcamera_driver"]
    handler_/freenove/rpi_libcamera_driver::OnCameraImage[["OnCameraImage()
30Hz"]]
  end

  subgraph unit_/freenove/joystick_driver["/freenove/joystick_driver"]
    handler_/freenove/joystick_driver::Tick[["Tick()
20Hz"]]
  end

  subgraph unit_/foxglove/foxglove["/foxglove/foxglove"]
    handler_/foxglove/foxglove:::hidden
  end
handler_/freenove/rpi_freenove_servo_driver::Update::/servo/1/request_degrees:::hidden x--/servo/1/request_degrees--> handler_/freenove/rpi_freenove_servo_driver::Update
handler_/freenove/rpi_freenove_servo_driver::Update::/servo/0/request_degrees:::hidden x--/servo/0/request_degrees--> handler_/freenove/rpi_freenove_servo_driver::Update
handler_/freenove/joystick_driver::Tick --/user_inputs--> handler_/freenove/rpi_freenove_mecanum_driver::Update
handler_/freenove/joystick_driver::Tick --/user_inputs--> handler_/freenove/rpi_freenove_servo_driver::Update
handler_/freenove/rpi_libcamera_driver::OnCameraImage --/camera/rgb--x /camera/rgb::handler_/freenove/rpi_libcamera_driver::OnCameraImage:::hidden
handler_/freenove/rpi_freenove_servo_driver::Update --/servo/1/current_degrees--x /servo/1/current_degrees::handler_/freenove/rpi_freenove_servo_driver::Update:::hidden
handler_/freenove/rpi_freenove_servo_driver::Update --/servo/0/current_degrees--x /servo/0/current_degrees::handler_/freenove/rpi_freenove_servo_driver::Update:::hidden
handler_/freenove/rpi_freenove_mecanum_driver::Update --/motor_state--x /motor_state::handler_/freenove/rpi_freenove_mecanum_driver::Update:::hidden

This is a pretty straightforward architecture, for now. We run joystick input, allowing it to control both the servos the camera is mounted on as well as the wheels. Later, we’ll move the joystick input to the wheels to instead be an input to some sort of planning stick.

This graph was generated with basis launch --mermaid - it does a dry run, outputting information about the launch in mermaid. This is really useful - I can copy/paste directly into a blog post or github markdown document. The PR for this will be merged soon.

The code

rpi_libcamera_driver

args:
  device:
    type: string
    help: The device to capture from
    optional: True
  
  width:
    type: uint32_t
    default: 1280
  height:
    type: uint32_t
    default: 720
  topic_namespace:
    type: string
    default: /camera

threading_model:
  single
cpp_includes:
  - foxglove/RawImage.pb.h
  - image_conversion.h

handlers:
  OnCameraImage:
    sync:
      # All inputs are required (of which there are technically none)
      type: all
      # Run at 30fps
      rate: 0.03333333333
    outputs:
      {{args.topic_namespace}}/rgb:
        type: protobuf:foxglove.RawImage
        inproc_type: image_conversion::Image

This unit implements libcamera support. libcamera on raspberry pi does have a v4l2 interface, I sadly wasn’t able to get it working. There’s nothing special about this code, you can see it here. It’s based off of the libcamera tutorial in their docs. The only odd point was that libcamera::formats::RGB888 appears to be BGR - I didn’t bother to track down why, I instead just swapped to requesting BGR.

Note: it’s not perfect that we run this unit at a fixed 30hz - ideally, this unit (and other driver-like units) can update freely on a thread, and publish at will. This requires a change to Basis (sync: type: external), which will be made in the next month.

joystick_driver

threading_model:
  single
cpp_includes:
  - basis_robot_input.pb.h
handlers:
  Tick:
    sync:
      type: all
      rate: 0.05
    outputs:
      /user_inputs:
        type: protobuf:basis::robot::input::InputState

Again, very straightforward. I initially implemented this using ioctl and then switched to libevdev. This really simplified controller access. See here.

rpi_freenove_servo_driver

args:
  i2c_device:
    type: string
    default: "/dev/i2c-1"
  address:
    type: int32_t
    default: 0x40
  default_angle_0:
    type: float
    help: the angle to start at
    default: 0
  default_angle_1:
    type: float
    help: the angle to start at
    default: 0
threading_model:
  single
cpp_includes: 
  - google/protobuf/wrappers.pb.h
  - basis_robot_input.pb.h

handlers:
  Update:
    sync:
      type: all
      # Run at 100fps
      rate: 0.01
    inputs:
      # All three inputs here are optional - we will run at 100hz regardless of the messages we get in
      /user_inputs:
        type: protobuf:basis::robot::input::InputState
        optional: True
        cached: True
      /servo/1/request_degrees:
        type: protobuf:google::protobuf::DoubleValue
        optional: True
      /servo/0/request_degrees:
        type: protobuf:google::protobuf::DoubleValue
        optional: True
    outputs:
      "/servo/0/current_degrees":
        type: protobuf:google::protobuf::DoubleValue
      "/servo/1/current_degrees":
        type: protobuf:google::protobuf::DoubleValue

Finally - a little bit of complexity. This unit runs at 100hz, and picks up any inputs that were published since the last tick. We cache /user_inputs - it can run at a lower rate, and we’re okay with reusing the last joystick input for 5 ticks - nobody will notice the difference.

Doing things the hard way

The initial version of the servo looked something like this:

---
config:
  flowchart:
    nodeSpacing: 5
    subGraphTitleMargin:
      bottom: 10
  defaultRenderer: elk
  elk:
      mergeEdges: True
---
graph LR
  %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
  subgraph unit_/freenove/rpi_freenove_servo_driver["/freenove/rpi_freenove_servo_driver"]
    handler_/freenove/rpi_freenove_servo_driver::OnInputs[["OnInputs()"]]
    handler_/freenove/rpi_freenove_servo_driver::RequestState0[["RequestState0()"]]
    handler_/freenove/rpi_freenove_servo_driver::RequestState1[["RequestState1()"]]
    handler_/freenove/rpi_freenove_servo_driver::Update[["Update()
100Hz"]]
  end
handler_/freenove/rpi_freenove_servo_driver::RequestState1::/servo/1/request_degrees:::hidden x--/servo/1/request_degrees--> handler_/freenove/rpi_freenove_servo_driver::RequestState1
handler_/freenove/rpi_freenove_servo_driver::RequestState0::/servo/0/request_degrees:::hidden x--/servo/0/request_degrees--> handler_/freenove/rpi_freenove_servo_driver::RequestState0
handler_/freenove/rpi_freenove_servo_driver::OnInputs::/user_inputs:::hidden x--/user_inputs--> handler_/freenove/rpi_freenove_servo_driver::OnInputs
handler_/freenove/rpi_freenove_servo_driver::Update --/servo/1/current_degrees--x /servo/1/current_degrees::handler_/freenove/rpi_freenove_servo_driver::Update:::hidden
handler_/freenove/rpi_freenove_servo_driver::Update --/servo/0/current_degrees--x /servo/0/current_degrees::handler_/freenove/rpi_freenove_servo_driver::Update:::hidden

This required storing each input to the unit and is less performant than letting the framework handle it.

Optional and Cached

With optional and cached, the code is straightforward. optional lets a handler run without the tagged input. cached keeps an input around for future executions of the handler. You can use it to work around differences in publish rates while still having a single handler. In this example we use it for input, but another use might be for something like loading a map and publishing it at a low rate. The less often messages need published, the better.

---
config:
  flowchart:
    nodeSpacing: 5
    subGraphTitleMargin:
      bottom: 10
  defaultRenderer: elk
  elk:
      mergeEdges: True
---
graph LR
  %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
  subgraph unit_/freenove/rpi_freenove_servo_driver["/freenove/rpi_freenove_servo_driver"]
    handler_/freenove/rpi_freenove_servo_driver::Update[["Update()
100Hz"]]
  end
handler_/freenove/rpi_freenove_servo_driver::Update::/user_inputs:::hidden x--/user_inputs--> handler_/freenove/rpi_freenove_servo_driver::Update
handler_/freenove/rpi_freenove_servo_driver::Update::/servo/1/request_degrees:::hidden x--/servo/1/request_degrees--> handler_/freenove/rpi_freenove_servo_driver::Update
handler_/freenove/rpi_freenove_servo_driver::Update::/servo/0/request_degrees:::hidden x--/servo/0/request_degrees--> handler_/freenove/rpi_freenove_servo_driver::Update
handler_/freenove/rpi_freenove_servo_driver::Update --/servo/1/current_degrees--x /servo/1/current_degrees::handler_/freenove/rpi_freenove_servo_driver::Update:::hidden
handler_/freenove/rpi_freenove_servo_driver::Update --/servo/0/current_degrees--x /servo/0/current_degrees::handler_/freenove/rpi_freenove_servo_driver::Update:::hidden

I’ll show the code here…

class rpi_freenove_servo_driver : public unit::rpi_freenove_servo_driver::Base {
public:
  rpi_freenove_servo_driver(const Args& args, const std::optional<std::string_view>& name_override = {});

  virtual unit::rpi_freenove_servo_driver::Update::Output
  Update(const unit::rpi_freenove_servo_driver::Update::Input &input) override;

  PiPCA9685::PCA9685 pca;

  static inline constexpr size_t NUM_SERVOS = 2;
  std::array<double, NUM_SERVOS> current_state = {};
  std::array<double, NUM_SERVOS> requested_state = {};
};

The declaration for our Unit is nothing special. We store the PCA9685 interface, and store both the current state and requested state for each servo.

Update::Output rpi_freenove_servo_driver::Update(const Update::Input& input) {
  const float t = basis::core::MonotonicTime::Now().ToSeconds();

  if(input.servo_0_request_degrees) {
    requested_state[0] = input.servo_0_request_degrees->value();
  }
  if(input.servo_1_request_degrees) {
    requested_state[1] = input.servo_1_request_degrees->value();
  }

  if(input.user_inputs && !input.user_inputs->joysticks().empty()) {
    // If we have a joystick connected, use it
    constexpr float MAX_JOYSTICK_DEGREES_SEC = 180.0f;
    // Get the update rate for this handler
    const auto duration = handlers["Update"]->rate_duration; 

    // TODO: move to config
    constexpr size_t AXIS_IDXES[2] = {2, 5};

    const auto& joystick = input.user_inputs->joysticks()[0];
    for(int i = 0; i < NUM_SERVOS; i++) {
      const float delta = joystick.axes()[AXIS_IDXES[i]] * MAX_JOYSTICK_DEGREES_SEC * duration->ToSeconds();
      requested_state[i] = std::clamp(requested_state[i] - delta, -70.0, 70.0);
    }
  }
  else {
    // Otherwise, rotate back and forth
    // this logic will eventually get moved out to a separate unit
    requested_state[0] = (sin(t * 2.0)) * 70.0;
    requested_state[1] = (sin(t * 3.1)) * 60.0;
  }

  std::array<std::shared_ptr<google::protobuf::DoubleValue>, NUM_SERVOS> outputs;
  for(int i = 0; i < NUM_SERVOS; i++) {
    // TODO: we will eventually implement smoothing here
    current_state[i] = requested_state[i];
    outputs[i] = std::make_shared<google::protobuf::DoubleValue>();
    outputs[i]->set_value(current_state[i]);
    float ms = DegressToPWMMS(current_state[i]);
  
    pca.set_pwm_ms(8 + i, ms);
  }

  // Magic - convert from our array output to our output type
  return std::apply([](auto&&... args) { return Update::Output{args...}; }, std::tuple_cat(outputs));
}

Notice - we don’t have to store any messages, we don’t have to write any synchronizer code. Very straightforward.

rpi_freenove_mecanum_driver

args:
  i2c_device:
    type: string
    default: "/dev/i2c-1"
  address:
    type: int32_t
    default: 0x40
threading_model:
  single
cpp_includes: 
  - google/protobuf/wrappers.pb.h
  - basis_robot_input.pb.h
  - basis_robot_state.pb.h

handlers:
  Update:
    sync:
      type: all
      # Run at 100fps
      rate: 0.01
    
    inputs:
      /user_inputs:
        type: protobuf:basis::robot::input::InputState
        optional: True
        cached: True

    outputs:
      /motor_state:
        type: protobuf:basis::robot::state::MotorState

Same story as rpi_freenove_servo_driver. The only code of note is this:

std::array<float, 4> XYTtoWheels(float x, float y, float theta) {
  constexpr float MAX_SPEED = 10.0f; // TODO: units
  // simple enough?
  // https://robotics.stackexchange.com/questions/20088/how-to-drive-mecanum-wheels-robot-code-or-algorithm
  // This isn't quite what we want, I think the /3.0 is bad
  return {
    -MAX_SPEED * (y+x-theta),
    MAX_SPEED * (y-x-theta), // Note: the hardware for this device has one reversed motor
    -MAX_SPEED * (y+x+theta),
    -MAX_SPEED * (y-x+theta),
  };
}

Mecanum wheel control is really simple. This function takes in the x/y joystick input and the sum of the triggers (theta), and outputs the power to each motor to satisfy those inputs.

Final thoughts

This was pretty simple to do - helped of course by the availability of other libraries out there. I’m looking forward to getting LiDAR working - and then either SLAM (with an IMU?) or a simple planning+controls stack.

Building a Robot on Basis 01 - Hardware

2024-11-22T00:00:00+00:00

I’ve spent the past few weeks working on a small robot to both be able to give demos with and exercise our code. This is a quick post on the hardware I bought, and what’s worked/not worked.

Part 01 - Hardware (You’re here!)
Part 02 - Software
Part 03 - tf2 support and LiDAR

The hardware:

FreeNove 4WD Smart Car Kit w/ Mecanum Wheels - this is the main body and hardware for the robot
Raspberry Pi 5 8GB
Raspberry Pi 5 Active Cooler
Raspberry Pi 5 SSD Hat
Corsair MP600 CORE MICRO 1TB NVMe SSD
Mean Well RSP-75-7.5
Various other electrical bits and pieces
SLAMTEC RPLIDAR (Future)

All in all, I think I’ve spent a bit over $500 dollars on this project.

FreeNove 4WD Smart Car Kit w/ Mecanum Wheels

The good:

It all mostly worked
Support was responsive after I burned out a servo
Mecanum wheels are really cool, and the algo to drive them is very simple
It didn’t blow up when I hooked in power backwards
It uses standard hardware with good driver support (standard raspberry pi camera, PCA9685 for servos/wheels)

The bad:

The test code doesn’t actually run on rpi-5 by default, you have to comment out a broken import (I should make a PR fixing this)
The front left motor is wired backwards - this isn’t fatal, more on this in the software post
It’s not compatible with cooler and/or SSD hat by default - this is fixable, but annoying
The camera servo mount is a little wobbly
The camera cable is very flimsy, I nicked it and had to buy a new one
There’s no way to run off of wall power, you have to buy batteries separately if you buy off of Amazon, and the batteries are hard to get out with damaging. I fixed this with some soldering.
The nuts and bolts have a tendency to work themselves loose - I may go and reassemble it later with loctite blue on the threads
I’ve already burnt out a servo - I don’t think it was my fault, but I’m not sure why this happened.

Would I recommend this kit? Yes! This is the real life robotics experience and matches my professional experience. I’m not a hardware guy, but just being a bit handy I was able to assemble it without much difficulty.

Raspberry Pi 5, HAT woes

The pros:

It’s a Pi!
Driver support is good

The cons:

The wifi on this thing is horrendously bad (this might be fixable with some Linux magic)
The CSI cable connector tends to come loose (this might not be the Pi’s fault, “loose camera cable” is a common robotics woe)

I actually bought and assembled the Pi first. As I was intending on doing development directly on device, I went and got the cooler and SSD. Probably worth the money!

Overclocking the thing tended to result in instability. I will likely try and set up distcc or a cross-compilation workflow in the future to build off-device to speed things up - but compilation time isn’t bad by any means for a codebase of this size.

Integrating with the FreeNove board was…difficult.

If you don’t have any HATs, this is likely super easy. If you do, it’s really sad - both the cooler and the SSD HAT block things. There are third party SSD boards that don’t sit on top - if you’re going to use this setup, it might be easier to get one of those, and not use active cooling.

If you do want to use this setup, here’s how I fixed it.

My solution:

Get a 1 to 2 GPIO expansion, use it instead of the extender that comes with the SSD HAT
Run a cable in between the expansion and the FreeNove board

I bought a cable that was too big and trimmed it down on the robot side, and bent pins on the expansion side, protected with electical tape. Do not do this. Go and find the proper sized cable (one with a 2x4 Pin DuPont connector on each end looks correct).

Power

The next issue was power - with only running the Pi on batteries, I got maybe a bit under an hour’s worth of power. Not great, not bad - but even if replacing the batteries wasn’t so difficult, if I’m heads down I don’t want to stop and kill my docker container, workflow, etc. After doing some research (asking EY), I settled on a Mean Well power supply. I need ~8V and >= 10A (supposedly), which put a lot of restrictions on using more consumer focused power supplies.

The Mean Well works great. The exposed mains power on top doesn’t make me too happy, but nothing some electrical tape can’t fix in the short term (and a cover in the long term). I bought a standard appliance cord to connect the wall power, and some 18/2 power cable for the DC side.

Some lessons:

Soldering XT30 connectors is harder that it seemed - I ended up ditching them for cheaper crimping connectors. Skill issue, likely.
If you’re soldering directly onto some materials, it helps to sand them.

I’m likely going to pay someone to redo this for me in the future - what I have will work, it’s just not pretty. I’d love for a solution that lets the batteries sit in the holder and charge while also running the robot, but that’s beyond my expertise at the moment.

LiDAR

I haven’t yet hooked the LiDAR in, just ran it off of USB on my PC. It looks good. I’ll probably run it off of USB in the short term on the robot, but I’ll eventually properly hook up to the GPIO ports directly. The price is reasonable enough, but it is only a 2D LIDAR. I think in the future I want to a Unitree L1 LiDAR, but I’m a little nervous about power and compute requirements.

Final thoughts

Hardware is both easier and tougher than I expected. As it turns out, You can just do things. Any time I ran into an issue, it was mostly just a trip to Ace Hardware to fix it.

(Fixing) Determinism in Robotics Testing

2024-09-02T00:00:00+00:00

Here at Basis, we’re building a production/testing focused robotics framework. Along those lines, determinism when testing is one of our primary goals. This article is focused on how basis can not only achieve determinism in tests, but use it to run your tests lightning fast.

Here’s a preview:

`replay`	`deterministic_replay`

Background

Determinism in robotics is an ongoing problem, both for runtime and for testing. This post aims to show a solution for testing/simulation.

The big issues come down to this:

ROS (and other robotics frameworks) don’t have any way of running deterministically, even in testing mode.
Code running at different speeds between the robot and your development workstation is a problem
Code running at a different speed in CI and on the robot is a huge problem, made worse by either cheapening out on CI hardware or running with too much parallelism in an effort to speed up test times.
At a low level, running the same code on the same system will behave differently due to transport layer and scheduling nondeterminism.

As a result of the above, making integration tests for robots sucks, and the tooling that does exist kinda sucks as well. An integration test running in CI might look like this:

Launch some process orchestrator (ROS Master, etc)
Launch some subset of your robot with a custom launch file
- if you’re lucky, someone’s added a test flag to your main launch file
- if you’re unlucky, there’s a separate launch file for tests that might be out of date to what your robot actually runs
Replay some recorded data at a slower than realtime speed to try and dodge transport/scheduling nondeterminism
Wait extra long because you’re running at slower than realtime speed.
Your test fails CI.
Cross fingers that the cleanup for the test actually kills all the processes the test launched.
Go repeat the above on your development desktop.
The test succeeds (or maybe randomnly doesn’t succeed).
Rerun the test in CI, waiting even longer.
The test succeeds.
Throw your hands up in the air, give up, and merge your code, blaming the simulation team if it fails again in main.

In the end, this just trains the engineering team to ignore test results, and discourages creating larger/more complex tests and simulations. When a test fails, is it due to the framework or due to the actual robotics code? Especially in safety critical environments, it’s important to rule out flakes.

Basis is aiming for tests to give the same result, every time. Other types of determinism will also come later (code running the same time on replay as it did at runtime, for example).

Sources of nondeterminism

Nondeterminism can come from a wide variety of sources.

The current main focus of efforts creating basis (and the focus of this blog post) is nondeterminism from the transport layer and scheduling related nondeterminism.

Nondeterminism at the transport layer includes nondeterminism in the network stack, the order sockets are processed in, what happens when two messages come in at the same moment, that sort of thing.

Scheduling nondeterminism encompasses threading related woes, performance differences between test executions, etc.

A non-exhaustive list of other sources of nondeterminism:

calls to random()
use of the system clock
use of networked resources
data races
compiler flags
cosmic rays
pointer comparisons

(I’ve seen all of these in test environements before.)

How can we fix this?

A few rules:

All code to be run deterministically happens in response to a message or a timer (in basis terms, everything lives inside Handlers)
No side channel communication between Handlers. Use a topic - Basis supports runtime only topics (raw C++ structs), so use them.
- This isn’t strictly true - Handlers in the same Unit will share a C++ class, setting a variable is a side channel. This is mostly an issue with complex Units that allow parallel execution of Handlers. We will (eventually) provide tools to help with this case.
(Along with 2) Robotics code doesn’t know about what a Subscriber or Publisher are. They take messages on topics as inputs and output messages on topics as outputs.
- This doesn’t stop tooling that does care about these concepts, it just means those tools can’t be run deterministically as part of a test.
- This does mean that externally triggered code (sensor drivers) won’t be determinisitic, but that’s not a problem for most tests.
All Handlers have associated metadata describing their inputs, outputs, execution conditions.
All Units (think: ROS Node) can be loaded dynamically and contain the metadata
Code inside Handlers is determinisitic (duh).

From these rules, one can build a scheduler that looks at the requested Units to be run, looks at the contents of any data to be replayed, and appropriately invokes each Handler in the correct (and deterministic) order, with the correct data. Basis is built from the ground up to support this use case.

Demonstration

Let’s show how basis can help with determinism. First we’ll record some data from a live run. Then we’ll run a replay test on that data and find issues with the testing process. Finally, we’ll show deterministic_replay at work, fixing those issues.

Recording some data

Here’s an example of a basis Unit that wants to do some work. This Unit will take in a single message, block for 100ms (doing fake “work”) and then exit, allowing other callbacks to run.

threading_model:
  single
cpp_includes:
  - simple_pub_sub.pb.h
handlers:
  OnChatter:
    sync:
      type: all
    inputs:
      /chatter:
        type: protobuf:StringMessage

OnChatter::Output simple_sub::OnChatter(const OnChatter::Input &input) {
  // Convert protobuf nanoseconds into a basis timestamp
  auto send_stamp = basis::core::MonotonicTime::FromNanoseconds(input.chatter->send_stamp());
  BASIS_LOG_INFO("OnChatter: {} {}", input.chatter->message(), send_stamp.ToSeconds());

  // Calculate delay between "now" and when the message was sent
  basis::core::Duration delay = input.time - send_stamp;
  if (delay > basis::core::Duration::FromSeconds(0.2)) {
    BASIS_LOG_WARN("/chatter delayed by {:.2f}s - queueing has occured", delay.ToSeconds());
  }
  constexpr int work_time_ms = 2000;
  BASIS_LOG_INFO("Doing {} ms worth of work", work_time_ms);
  std::this_thread::sleep_for(std::chrono::milliseconds(work_time_ms));

  return OnChatter::Output();
}

Running this unit along with another unit to produce on /chatter at 1Hz will give a console output resembling this

[124998.122650826] [launch] [info] Recording (async) to /tmp/demo_124998.122377076.mcap
[124998.126956742] [launch] [info] Running process with 2 units
[124998.130765492] [launch] [info] Started thread with unit /opt/basis/unit/simple_sub.unit.so
[124998.134928909] [launch] [info] Started thread with unit /opt/basis/unit/simple_pub.unit.so
[124999.137999076] [/simple_pub] [info] PublishAt1Hz
[124999.138074076] [/simple_sub] [info] OnChatter: Hello, world! 124999.138065618
[124999.138080410] [/simple_sub] [info] Doing 100 ms worth of work
[125000.136988785] [/simple_pub] [info] PublishAt1Hz
[125000.137143077] [/simple_sub] [info] OnChatter: Hello, world! 125000.137124493
[125000.137151452] [/simple_sub] [info] Doing 100 ms worth of work
[125001.136178202] [/simple_pub] [info] PublishAt1Hz
[125001.136319869] [/simple_sub] [info] OnChatter: Hello, world! 125001.136305744
[125001.136327452] [/simple_sub] [info] Doing 100 ms worth of work
[125002.137870703] [/simple_pub] [info] PublishAt1Hz
[125002.138054536] [/simple_sub] [info] OnChatter: Hello, world! 125002.137991619
...

Replaying the data

Great, but now let’s pretend this was data recorded on a robot. We might want to run replay /tmp/demo_124998.122377076.mcap, along with simple_sub. If the hardware is similar between the two environments, we might see something like this:

[125458.039238910] [launch] [info] Running process with 1 units
[125458.040486243] [launch] [info] Started thread with unit /opt/basis/unit/simple_sub.unit.so
[124998.307061284] [launch] [info] simple_sub detected playback restart, restarting...

(note the time jump here, we got a new simulated time step)

[124998.977061284] [launch] [info] Running process with 1 units
[124998.977061284] [launch] [info] Started thread with unit /opt/basis/unit/simple_sub.unit.so
[125001.137061284] [/simple_sub] [info] OnChatter: Hello, world! 125001.137061284
[125001.137061284] [/simple_sub] [info] Doing 100 ms worth of work
[125002.147061284] [/simple_sub] [info] OnChatter: Hello, world! 125002.147061284
[125002.147061284] [/simple_sub] [info] Doing 100 ms worth of work
[125003.147061284] [/simple_sub] [info] OnChatter: Hello, world! 125003.147061284
[125003.147061284] [/simple_sub] [info] Doing 100 ms worth of work
...

Problem 1 - missing data, nondeterminism.

Rerunning this a few times, we see Problem #1 - we sometimes miss the first message of the test, and get an output:

[125002.147061284] [/simple_sub] [info] OnChatter: Hello, world! 125002.147061284

A quick check with mcap-cli reveals…

basis@aee413836118:/basis/demos/simple_pub_sub/build$ ~/mcap-linux-arm64 cat --json /tmp/demo_124998.122377076.mcap --topics /chatter
{"topic":"/chatter","sequence":0,"log_time":124999.138044993,"publish_time":124999.138044993,"data":{"sendStamp":"124999137959660", "message":"Hello, world!"}}
{"topic":"/chatter","sequence":0,"log_time":125000.137092327,"publish_time":125000.137092327,"data":{"sendStamp":"125000136919993", "message":"Hello, world!"}}
{"topic":"/chatter","sequence":0,"log_time":125001.136257244,"publish_time":125001.136257244,"data":{"sendStamp":"125001136105286", "message":"Hello, world!"}}
...

Several messages are missing from the test. Even our most complete replay was missing 124999 and 125000.

Problem 2 - performance differences can mean nondeterminism

Now what if instead, our testing environment was slower (overloaded CI, different/no GPU, less cores). In this scenario, let’s pretend that the work takes 2000 ms to run, and set constexpr int work_time_ms = 2000;.

Let’s run that same replay test.

[125002.147061284] [/simple_sub] [info] OnChatter: Hello, world! 125002.137789828
[125002.147061284] [/simple_sub] [info] Doing 2000 ms worth of work
[125004.147061284] [/simple_sub] [info] OnChatter: Hello, world! 125003.137948037
[125004.147061284] [/simple_sub] [warning] /chatter delayed by 1.01s - queueing has occured
[125004.147061284] [/simple_sub] [info] Doing 2000 ms worth of work
[125006.147061284] [/simple_sub] [info] OnChatter: Hello, world! 125004.13672012
[125006.147061284] [/simple_sub] [warning] /chatter delayed by 2.01s - queueing has occured
[125006.147061284] [/simple_sub] [info] Doing 2000 ms worth of work
[125008.147061284] [/simple_sub] [info] OnChatter: Hello, world! 125005.136193412
[125008.147061284] [/simple_sub] [warning] /chatter delayed by 3.01s - queueing has occured
[125008.147061284] [/simple_sub] [info] Doing 2000 ms worth of work

Look at that - rather than the second message being processed at 125003.137948037, it’s processed at 125004.147061284, a second later than expected (or exactly as one would expect for adding an additional second of blocking…). The delay will only continue to go up, and the queues backing the pub/sub system generally won’t be configured for infinite space, resulting in dropped messages and incorrect timings.

If the original runtime looked like this:

(Apologies for the default LucidChart colorscheme)

With our artifical slowdown, it now looks like this:

Note the queueing - we can’t process more messages while OnChatter is executing, but the replay system doesn’t know this and keeps publishing messages.

Problem 3 - integration tests are slow, clunky, and don’t pair well with unit testing frameworks

If each message takes 100ms to process, and we replay 10 messages, how long would you expect this test to take? With a regular integration test: 10 seconds, for 1 second worth of work.

Let’s say our CI hardware is even faster, able to process a message in 10ms - the test still takes 10 seconds to run, for 100ms worth of work.

Along with this, how would we integrate this test with gtest? We’d have to build some sort of framework that manages forking or shelling out to launch each part of it, with timeouts, management for grandchildren processes, etc.

If we wanted to programmatically create messages (avoiding the use of serialized data on disk), or create a single message and check the output of the system after each step, how would we do it? With ROS you’d have two choices:

Stand up a ROS Master, launch the nodes you care about, initialize your test as a ROS node, publish the message, and spin until you (maybe) get a response. (Don’t forget to turn off paralellism for your tests!)
Break apart your ROS Nodes into libraries, manually shuffling messages between the business logic for each node (good luck keeping this in sync with the launch files)

The solution

Let’s rerun this with basis’s deterministic replayer.

deterministic_replay /tmp/demo_124998.122377076.mcap /basis/demos/simple_pub_sub/launch_single_process.yaml --disable_unit demo:simple_pub
[129000.980036858] [replayer] [info] Found unit /simple_sub simple_sub at /opt/basis/unit/simple_sub.unit.so
[129000.982984441] [replayer] [info] Starting deterministic playback...
[129000.984189191] [replayer] [info] replaying topic /chatter
[129000.984319524] [replayer] [info] replaying topic /log
[129000.984376316] [replayer] [info] Beginning replay at 124998.127061284
[129000.985144524] [replayer] [info] Initialized unit /simple_sub
[124999.238044993] [/simple_sub] [info] OnChatter: Hello, world! 124999.13795966
[124999.238044993] [/simple_sub] [info] Doing 2000 ms worth of work
[125000.237092327] [/simple_sub] [info] OnChatter: Hello, world! 125000.136919993
[125000.237092327] [/simple_sub] [info] Doing 2000 ms worth of work
[125001.236257244] [/simple_sub] [info] OnChatter: Hello, world! 125001.136105286

Beautiful - we even caught the first few messages that were dropped before. The scheduler now waits on Handlers that are expected to be complete (in terms of simulated time) but aren’t yet finished (in realtime).

We can see that we don’t do any more work (such as replaying messages) while OnChatter is executing, as the Handler is only supposed to take 100ms of simulation time. It sucks that we’re running twice as slow as realtime, but it’s better than running with inaccurate messages.

Problem 1, Problem 2 - solved.

As for Problem 3… remember that preview gif from the beginning of the article?

Let’s turn the work time all the way down to 10ms. Because deterministic_replay knows when code should run, it also knows when code isn’t running. We can run the code as fast as our CPU will let us, ignoring the fact that the replay data publishes at 1Hz. This enables integration tests that run lightning fast.

(Other data replay systems usually have some form of rate multiplier command - but this is tough to tune for all conditions).

`replay`	`deterministic_replay`

Everything can be run in a single process for test mode, meaning a coordinator (master) process is only needed if communication with visualizers or other tooling is required (not required for unit testing). One could link deterministic_replay into a unit test, then run a launch file (or even a programatically created list of Units), push in messages, query outputted messages and inner states of nodes, modify in flight messages, etc. No extra processes needed.

Even further

With this power, we can do more:

Find out what happens if one of our messages arrives later than expected, or test different timing related error paths. How well have you actually tested your degraded states around timing?
The default behavior of deterministic_replay is to always rerun events that happen at the same time in the same order. What if instead we reversed it? Or randomized it with a seed? With message send time jitter randomization added on top, one could long tail test for timing related bugs.
Test out “what if” situations that may be expensive to implement. Let’s say your Perception Lead says he can speed up the perception stack by 15% with a quarter’s worth of work and two engineers. It sounds good on paper, but will a faster perception stack actually lead to better robot performance? Before doing the quarter’s worth of work, run your integration test suite with the promised timings, instead.
lldb -- deterministic_replayer ... just works, and can pause all units, properly, without having to mess with fork modes.

Hasn’t this been built yet?

This sort of tooling has been asked for before:

Unanswered answers.ros.org post
ROS1’s documentation: “For nodes like turtlesim, where minor timing changes in when command messages are processed can subtly alter behavior, the user should not expect perfectly mimicked behavior.”
Various other ROS answers posts
Engineers internal to companies integrating with Gazebo/Applied Intuition/internal tools/etc.

Various groups have tried approaching it:

A cool deck on ROS runtime determinism from BOSCH (runtime determinism isn’t covered here, but basis does make this easier)
Flow framework https://github.com/ZebraDevs/flow_ros?tab=readme-ov-file (This appears to be mostly a intermediary between ROS publishers and robotics business logic - I approve, good architecture. But where’s the deterministic replay?)
ROS2 DEF - A bolt on system for ROS2 from Ulm University. A valiant effort, but abandoned and incomplete. Can’t fully work due to ROS2 architecture.
Various large robotics companies having to implement this internally to various levels of completeness, using various strategies. Unfortunately these are going to be very company specific and they aren’t likely to publish or sell them.

Funnily enough, Applied Intuition completely sidesteps the issue, saying that determinism in ADAS is good, but not giving any answers for how to achieve it when integrating.