Handling dependencies with CMake

Over the years, I have seen mainly two strategies (or variants of them) being used to handle dependencies in CMake projects. Because I only like one of them, this post is about explaining why my preferred strategy is better.

TL;DR: I strongly believe that a CMake project should use find_package instead of add_subdirectory. Feel free to try building this Proxygen example or this gRPC example before reading if you want to.

First strategy: find_package

The first strategy is about realising that CMake is not a package manager, and therefore delegating package management to the user (who can install the dependency using their preferred way). This is typically the favoured solution when starting a small project.

For instance:

cmake_minimum_required(VERSION 3.10.2)

project(hello_world)

find_package(OpenSSL REQUIRED)

add_executable(hello_world
    main.cpp
)

target_link_libraries(hello_world
    OpenSSL::SSL
)

Here, we do rely on find_package to “find” OpenSSL, following some rules (documented here), which implies that the user installed OpenSSL before (for instance using apt install). If OpenSSL is not to be found, CMake will complain during the configure step.

“Well, apt install libssl-dev is nice, but what if apt does not provide my dependency, or not the version of the dependency I want?”, you ask. That is where I believe the second strategy becomes (wrongfully) popular.

Second strategy: add_subdirectory

As a project grows, there often comes a need for a dependency (or a version of it) that is not provided by the system package manager. At this point it is not enough to just apt install <dependency>. There are many ways to handle that, but it seems like adding the dependency as a git submodule and building it inside the project with add_subdirectory has become popular.

As an example, say we do not want to use GoogleTest (libgtest) from the system. We can add it as a git submodule, say gtest/, and use it from CMake:

cmake_minimum_required(VERSION 3.10.2)

project(hello_world)

add_executable(hello_world
    main.cpp
)

add_subdirectory(gtest)

target_include_directories(hello_world
    SYSTEM
    PRIVATE ${PROJECT_SOURCE_DIR}/gtest/googletest/include
    )

target_link_libraries(hello_world
    gtest
    gtest_main
    )

There are alternatives to the git submodule, such as CMake’s own FetchContent and of course wrappers on top of that like CPM. But that is not fundamentally different; at the end of the day, this strategy builds the dependency with add_subdirectory.

I argue that handling dependencies with add_subdirectory is flawed:

  1. What happens if your dependency does not support CMake? Say it uses Autotools, Meson or a custom build system instead? OpenSSL and Boost do not support CMake, for instance.
  2. How does it work with transitive dependencies? Say your project runs add_subdirectory(libA) and add_subdirectory(libB), but libA also runs add_subdirectory(libB). What happens then? Which version of libB is being used?
  3. Every clean build (e.g. in the CI) has to recompile all your dependencies. Isn’t it nicer (and potentially much faster) when the library can just be installed somewhere (or copied from a cache) once and for all?
  4. You force the consumers of your library to use your strategy. What if I want to use your library, but I do not want you to include OpenSSL through add_subdirectory(openssl) (maybe I want to use LibreSSL)?

The find_package strategy does not have those problems. So why are so many projects using add_subdirectory? I believe that the answer is that it feels simpler.

Delegate, but assist

Ok, so delegating the package management to the user (i.e. using find_package) is nicer, but “copying” the dependencies inside the project (i.e. using add_subdirectory) is simpler.

I believe that the solution is to use find_package, but to provide helpers if necessary, so that we get the best of both worlds: those who know what they are doing can handle dependencies the way they like, and those who don’t can run the helper script.

But before getting to the helper script, I would like to review different ways one can handle dependencies.

Using the system package manager

As mentioned above, the easiest way to get a dependency is to install it globally on the system using the system package manager. For instance, apt install libssl-dev for the corresponding find_package(OpenSSL).

The problem is that you may rely on a library that is not provided by the system package manager. Another issue arises when you need to cross-compile your library (and its dependencies), because typically the system package manager only provides packages for the host architecture.

Manually building and installing globally

Instead of using the system package manager (which may not have your dependency), you can install the dependency manually.

For instance, for a CMake library:

cmake -Bbuild -S.
sudo cmake --build build --target install

Using Autotools, it could look like this:

./configure
make
sudo make install

And with Meson:

meson builddir && cd builddir
meson compile
meson install

That should result in the dependency being installed on your system, similar to what the system package manager would do. Except that the package manager cannot know about this, and it could create conflicts. I don’t like this way myself (and I would discourage the use of it), but I admittedly used to do it years ago, when I was blindly following instructions saying stuff like “now run make && sudo make install”. I still see a lot of READMEs doing that, so I’m pretty sure it is still used extensively.

Manually building and installing locally

Less known is the fact that one can install a project locally, by specifying an install path in the CMake configure step:

cmake -DCMAKE_INSTALL_PREFIX=/path/to/local/install -Bbuild -S.
cmake --build build --target install

Using Autotools, we can specify the installation prefix at configure time:

./configure --prefix=/path/to/local/install
make
make install

Similarly with Meson, we can use --prefix in the configure step:

meson builddir && cd builddir
meson configure --prefix /path/to/local/install
meson compile
meson install

The trick with locally-installed dependencies is that we need to inform CMake about the installation path (how would find_package guess otherwise?). This is done using CMAKE_PREFIX_PATH. So when you build your project (the lines above built the dependency), you have to specify this path in the CMake configure step:

cmake -DCMAKE_PREFIX_PATH=/path/to/local/install -Bbuild -S.
cmake --build build

Let me repeat that:

  • Build the dependency with -DCMAKE_INSTALL_PREFIX (CMake) or --prefix (Autoconf, Meson) to specify where it should be installed.
  • Build the project with -DCMAKE_PREFIX_PATH to tell CMake where it can find the dependencies.

Spoiler alert: this is my favourite, because I can keep the dependencies per project, and it’s super easy to clean them (to me it feels close to using a venv in Python). Also it is very convenient when cross-compiling. For these reasons, that is what the helper script will leverage.

Using another package manager

Instead of relying on your system package manager, there are alternatives out there, like Conan. I am not familiar with them, but as long as they are compatible with the find_package syntax, I don’t see a problem with them. For instance Conan provides a find_package_generator.

The one requirement I have for such a package manager is that it should be totally optional. For instance the find_package_generator of Conan is compatible with CMAKE_PREFIX_PATH. As a user, I want to have the choice to use Conan (and set CMAKE_PREFIX_PATH accordingly) or to use whatever system I want (even build the dependencies manually).

I am not a huge fan of Hunter because it requires changes in the CMakeLists. But again, as long as it is made optional and I am free to handle the dependencies my way, I am fine with it.

Finally, the problem with “third party” package managers is that you may need to maintain your own repository (if your dependencies are not available in the official repo). This is not only costly, but you cannot really expect users of your library to rely on your package manager and your repository. Which brings us to my main point again: use a package manager as a helper if you want to, but make it optional: your CMakeLists should not know about it at all, and instead rely on find_package.

The helper subproject

As we have seen above, the goal is to use find_package() for all the dependencies in our CMakeLists, and to let the user make sure that they are found (by using a package manager or building and installing them manually). But nothing prevents us from providing a helper script that fetches, builds and installs all our dependencies locally.

I have been mentioning a script, but the way I do it is actually more of a CMake subproject. I like it because it can be cross-platform, but the point is really that it builds and installs dependencies locally somewhere.

Using the helper

Let’s first see how the helper is being used:

# Run the helper script, installing the dependencies in `./dependencies/install`
cmake -DCMAKE_INSTALL_PREFIX=dependencies/install -Bdependencies/build -Sdependencies
cmake --build dependencies/build

# Build the project, telling `find_package()` where to find dependencies
cmake -DCMAKE_PREFIX_PATH=$(pwd)/dependencies/install -Bbuild -S.
cmake --build build

For a user of your project, that’s all it takes: running two CMake projects:

  1. The first one builds the dependencies and installs them in a location chosen by the user (here ./dependencies/install).
  2. The second one builds the project, using the dependencies installed in step 1 (specified with CMAKE_PREFIX_PATH).

From the user perspective, is that really more difficult than handling git submodules? I don’t think so.

Writing the helper

The helper script builds and installs all our dependencies locally by leveraging an old CMake feature: ExternalProject.

It is extremely easy; the syntax looks like this (there are more options nicely described in the documentation):

cmake_minimum_required(VERSION 3.10.2)

project(my-dependency)
include(ExternalProject)

ExternalProject_add(
    <dependency name>
    URL <link to sources>
    PREFIX <build directory for this dependency>
    CONFIGURE_COMMAND <configure command>
    BUILD_COMMAND <build command>
    CMAKE_ARGS <args passed to the cmake commands (if relevant)>
    )

The beauty of it is that while it uses CMake to build and install the dependency, it does not require the final project to use CMake. You could use this to install dependencies locally, and then use them from a Meson or Autotools-based project! It also means that instead of using CMake with ExternalProject_add, you are free to write your helper in shell, or python, or whatever you want; the whole point is that this is completely transparent to your main CMake project.

Let’s see a few examples below (with many more available here).

A CMake dependency: re2
cmake_minimum_required(VERSION 3.1)

project(external-re2)
include(ExternalProject)

list(APPEND CMAKE_ARGS
    "-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_INSTALL_PREFIX}"
    "-DCMAKE_POSITION_INDEPENDENT_CODE=ON"
    "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
    "-DBUILD_SHARED_LIBS=OFF"
    )

ExternalProject_add(
    re2
    URL https://github.com/google/re2/archive/2022-04-01.tar.gz
    PREFIX re2
    CMAKE_ARGS "${CMAKE_ARGS}"
    )

Feel free to try building it with:

cmake -DCMAKE_INSTALL_PREFIX=install -Bbuild -S.
cmake --build build

The build step will run ExternalProject_Add which will fetch, configure, build and install re2 into ./install. See how I passed CMAKE_INSTALL_PREFIX through CMAKE_ARGS? CMAKE_ARGS is also an opportunity to pass all the CMake options you want for this dependency, e.g. -DBUILD_TESTING=OFF, -DBUILD_SHARED_LIBS=OFF, -DSOME_OPTION=ON, you name it.

Also note the use of URL to download a tarball containing the sources. This can be replaced by GIT_REPOSITORY and GIT_TAG (see for instance jsoncpp here).

An Autotools dependency: libsodium
cmake_minimum_required(VERSION 3.10.2)

project(external-sodium)
include(ExternalProject)

ExternalProject_add(
    sodium
    URL https://download.libsodium.org/libsodium/releases/libsodium-1.0.18-stable.tar.gz
    PREFIX sodium
    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=${CMAKE_INSTALL_PREFIX}
    BUILD_COMMAND make
    )

It can be run with the same commands as above:

cmake -DCMAKE_INSTALL_PREFIX=install -Bbuild -S.
cmake --build build

And it will result in libsodium being installed in ./install. Only this time it will have used Autotools (see the CONFIGURE_COMMAND and the BUILD_COMMAND).

A custom build system: boost

ExternalProject is very versatile, as can be seen when building Boost using their custom build system:

cmake_minimum_required(VERSION 3.10.2)

project(external-boost)
include(ExternalProject)

ExternalProject_add(
    boost
    URL https://boostorg.jfrog.io/artifactory/main/release/1.80.0/source/boost_1_80_0.tar.gz
    PREFIX boost
    CONFIGURE_COMMAND <SOURCE_DIR>/bootstrap.sh --prefix=${CMAKE_INSTALL_PREFIX}
    BUILD_COMMAND <SOURCE_DIR>/b2
    BUILD_IN_SOURCE TRUE
    INSTALL_COMMAND <SOURCE_DIR>/b2 install
    )
All together

The examples above show how to build one dependency using ExternalProject, and all that remains now is to group them into one, by just calling all of them from one parent CMakeLists:

cmake_minimum_required(VERSION 3.10.2)

project(dependencies)

add_subdirectory(re2)
add_subdirectory(sodium)
add_subdirectory(boost)

One more note: in the case of transitive dependencies, we need to install the dependencies in the right order and let new ones know about the install path. So typically the dependencies are built like this (when there are transitive dependencies):

cmake -DCMAKE_PREFIX_PATH=$(pwd)/install -DCMAKE_INSTALL_PREFIX=install -Bbuild -S.
cmake --build build

This just means that dependencies will be installed into ./install (as per CMAKE_INSTALL_PREFIX), and CMake will look for dependencies in $(pwd)/install (as per CMAKE_PREFIX_PATH) when it encounters a find_package instruction.

Conclusion

CMake is not a package manager, and therefore a proper CMake project should delegate the package management to the user. The way to do this is to use find_package for dependencies (and not add_subdirectory, which has many downsides).

Optionally, one can provide a helper script to build and install the dependencies locally, as suggested in this post. Of course, it can be done differently (as long as it works with find_package through CMAKE_PREFIX_PATH). At Facebook, for instance, they seem to have their way.

I wrote a small project illustrating the helper strategy here (using Facebook/Proxygen because it has a few transitive dependencies), and another one using gRPC. I also provide more dependency scripts here for inspiration.

Annex

What if the dependency does not play well with find_package?

If the dependency is a CMake project, it should support find_package. In case it does not, consider contributing it, or at least ask the maintainer to add it. If they don’t want to, and you can’t contribute it, maybe it is a good time to wonder whether you want to depend on that library, really.

This said, what about dependencies that are not CMake projects? It does make sense for a non-CMake library to not provide a CMake package configuration file (<PackageName>Config.cmake) indeed. Fortunately that usually happens in one of the following two scenarios:

  1. The library supports pkg-config. The good news is that CMake supports pkg-config, too (I like to hide that into a CMake Find Module, but that’s a longer story). For instance, gstreamer uses Meson, but supports pkg-config:

    find_package(PkgConfig REQUIRED)
pkg_search_module(GST REQUIRED gstreamer-1.0>=1.4

  2. The library is so popular that CMake provides a Find Module. That’s the case for OpenSSL and Boost, for instance; they do not support CMake, but find_package(OpenSSL) and find_package(Boost) both works because CMake supports them.

Worst case, you can always write a Find Module manually, but I never really had to myself (except to hide a call to pkg_search_module, as mentioned above).

What if I actually need add_subdirectory?

There is exactly one reason I can think of where one would want to add a dependency using add_subdirectory, and that’s when the author of the main project is also the author of the dependency, and wants to develop both in parallel. In that case, I would still prefer ExternalProject_add (i.e. using the helper script) which supports local paths. However, if your IDE really needs add_subdirectory, then you can still support both, like gRPC does (though I personally find this slightly convoluted).