uvtarget - a helper for bundling python in CMake

I’ve released today a tool uvtarget - used to link together multiple python projects into one workspace with a single lockfile, with CMake. It’s available here on GitHub.

This took a few weeks of various evenings and weekend afternoons to make, which feels crazy for something that’s not even 500 lines of code. Difficulty was mainly knowing what lines to write and figuring out how uv wanted things to be handled. There were a lot of small cases around making sure the environment CMake was being invoked in didn’t leak into the compilation environment, as well as handling subtle differences between project, workspace and virtual environment.

FAQ

Q. Why?

A. Basis uses CMake as a lowest common denominator build system. I’m still in the process of integrating Python into basis. I had two problems: (1) I need to be able to pull in arbitrary Python versions into the build environment. (2) I need to easily package up the build environment into some kind of install space, and make sure everything comes along for the ride. Uv is an easy win for this, it just needs some work to integrate with CMake, especially for workflows that end up calling sudo make install.¹.

Q. Why create a workspace?

A. In a robotics context, one might have several related python projects that need installed into the same image, that are being developed in parallel. Regardless of the structure we give the projects (one package, multiple packages, etc), if we run tests and simulations on one set of dependencies, we need to also ship the codebase on that same set of dependenies.

Q. Why do we need a meta pyproject?

A. One really can’t have multiple lock files for one environment, so this means one pyproject (1:1 relationship). The alternative is multiple environments, but in my experience this is far more trouble than its worth.

Q. Why autogenerate?

A. For now, basis works on a single CMakeList source tree. Adding additional libraries or units that contain python shouldn’t require doing anything more than adding them to your CMake tree - managing an additional parallel tree of dependencies isn’t great. Along with that, we shouldn’t inherit some other library’s lockfile - we should take the constraints but not the pin. Autogeneration can easily be opted out of, if you prefer to do things the other way around (manually track the libraries going into the venv).

Q. Will this work with ROS?

A. Yes, it should - with some caveats. Instructions are for catkin tools. Under catkin_make this should actually work “perfectly” but I really don’t recommend it, catkin tools is faster and has better isolation. I don’t know the best way to do this under colcon, but it should also be possible (one might want).

• Create a package pybase or similar, inside that package point at each of your other python packages with uv_add_pyproject(../other_package) or with whatever path resolution you prefer.
• When you call uv_initialize make sure you use UNMANAGED_PYPROJECT_FILE
• If other packages depend on having a working python environment (due to needing to use an installed package), make sure they gave catkin depends on pybase.
• Inside the venv that uvtarget creates, install a .pth file pointing at each directory containing ROS python that needs to be imported into the venv. (this can be done in CMake)
• When you source catkin’s setup.bash, call export PYTHONPATH="" && source build/path_to_pybase/.env/bin/activate to get an environment with your packages and not the system packages. YMMV, you may need to recompile things like moveit or tf2.

Q. What about (compiled) extension modules?

A. I haven’t tried creating extension modules yet - but this should work fine. You may or may not need to add a dependency on uv_sync to your extension builds, otherwise the sync may fail. Happy to work with you for a solution if you’re interested.

How does it work?

cmake …

Near the root of your project, uv_initialize(...) is called, setting up your environment’s settings. This sets up some variables in CMake and sets up a hook to run at the end of the CMake generation step. uv venv --python $version is called to create the virtual environment for development.

Each time you call uv_add_pyproject or uv_add_dev_dependency, the arguments are stored off for later.

Finally, when the deferred hook is called, we create a pyproject depending on all of the earlier declared dependencies.

make

The cmake step adds up a build target that calls uv sync with the correct environment and arguments to sync the environment to the workspace. We call this on every build, uv is so fast that you don’t notice in the cases where nothing has changed. An “optimization” would be to only call uv sync when a pyproject has changed, but that’s just duplicating the logic uv has internally.

make install

We call uv export with a bunch of flags to generate a requirements.txt for just the dependencies. Then uv build to build wheels for each workspace member. Finally, we install them in one big swoop with uv pip install.

There are a bunch of options that could be supported for installation that aren’t currently implemented. Wheel+requirements only export, installing to a non virtual environment, exporting to an installer, etc. The wheel+requirements flag is probably the most useful for alternative workflows.

Where are you going from here?

With uvtarget

Nothing immediately planned, but happy to accept feature requests and PRs.

With basis

I’m finally on to working on subinterpreter support, allowing for loading arbitrary numbers of Python modules in one process, along side C++ (and someday Rust).