Acarg

My email setup with Aerc and Git

Wed, 13 Dec 2023 00:19:36 +0100

Motivation

Ever since GMail in the early 2000s¹ I have been interacting with emails through the browser, in a webmail. It may sound stupid, but I only recently realised that I was missing out on some pretty cool features of email, such as:

Offline access: Webmails require to be online. Not only is it convenient to read/write emails offline, but it is also faster because there is no need to re-download the emails every time.
No lock-in: Without owning the domain, it is hard to change provider because it means changing the address and updating it everywhere. Email makes it easy to own the domain, separating the address from the provider.
Easy backup: Now that I fetch my emails and keep them offline, I have a mail/ folder that I can easily backup.
The right UI for the job: On my phone, I want an graphical app that saves my recent emails so that I can access my cinema ticket offline. On my desktop, I want to backup all my emails, and I want an easy way to forward patches from a mailing list into git (I like TUIs like Aerc or Mutt). Sometimes I want to access my emails from a device I don’t own, and the webmail is convenient for that.

I therefore spent some time setting up an email environment that works well for me. Everyone is different, so what is ergonomic for me may not be for others. But it doesn’t hurt to share my setup and - who knows - it may help someone, someday. You will also find other resources describing similar setups below.

My setup

I am using the following programs:

Aerc as my TUI email client (there are others like Mutt). It integrates well with the git email workflow.
Git with the email workflow, sending patches with git send-email.
isync/mbsync² to synchronize my mailbox (it downloads my emails into a mail/ folder that Aerc accesses).
OpenSMTPD as my SMTP relay, which queues the emails I send while I am offline and sends them later. An alternative would have been msmtp (which is “just” an SMTP client and may therefore be more lightweight) together with the msmtpq script (this setup is described here).

Also note that I am currently using Fastmail as a provider (different providers may require slightly different configurations). I know that JMAP is a thing, but I haven’t found an alternative to isync/mbsync for JMAP yet.

Aerc

Aerc can be used “standalone” (without isync/mbsync and OpenSMTPD), with a simple config like this:

# ./config/aerc/accounts.conf

[Fastmail]
source = imaps://my-user%40my-domain.com:my-password@imap.fastmail.com:993
outgoing = smtp://my-user%40my-domain.com:my-password@smtp.fastmail.com:587
from = My User <my-user@my-domain.com>

Where:

my-user@my-domain.com is my address and Fastmail username.
my-password is the token I got from Fastmail.

But in this setup, Aerc will only work online: you will not have access to the emails when offline, and you will not be able to send emails either.

That is why we will configure isync/mbsync to fetch and store the emails on disk (here in ~/mail/), such that Aerc can access them. We can therefore update the source in Aerc:

source = maildir://~/mail

In order to send emails while we are offline, we will send them with OpenSMTPD which will queue them and retry sending them when we are back online. We make Aerc leverage this by setting the outgoing param to OpenSMTPD’s sendmail executable:

outgoing = /usr/sbin/sendmail

The final ./config/aerc/accounts.conf looks like this:

# ./config/aerc/accounts.conf

[Fastmail]
source = maildir://~/mail
outgoing = /usr/sbin/sendmail
default = INBOX
from = My User <my-user@my-domain.com>
copy-to = Sent
check-mail-cmd = 'mbsync -c ~/.config/isync/mbsync.conf primary'
check-mail = 1m

source: the mail folder (synchronized with Fastmail by isync/mbsync).
outgoing: the sendmail executable that passes the email to OpenSMTPD.
check-mail-cmd: the command run by Aerc to check for new emails. Note that it will only be called while Aerc is running. If you want to fetch emails regularly independently from Aerc, consider running the mbsync command in a cron job.
check-mail: we tell Aerc to run check-mail-cmd every minute.

Git

Just like for Aerc, we can configure Git to send emails without OpenSMTPD (see this guide for more details):

# ~/.gitconfig

[user]
 email = my-user@my-domain.com
 name = My User
[sendemail]
 smtpserver = smtp.fastmail.com
 smtpuser = my-user@my-domain.com
 smtpencryption = ssl
 smtpserverport = 465
 smtppass = my-password

But again, sending emails will only work online. The good news is that with OpenSMTPD configured, we can just drop the whole [sendemail] section and the emails will be handled by it:

# ~/.gitconfig

[user]
 email = my-user@my-domain.com
 name = My User

Now git send-email will work offline, too, with the outgoing emails being queued in OpenSMTPD.

isync/mbsync

We have configured Aerc to fetch emails from ~/mail/ above, now we need isync/mbsync to actually put them there. Here is my ~/.config/isync/mbsync.conf:

# ~/.config/isync/mbsync.conf

CopyArrivalDate yes
Create Near
Expunge Both

IMAPAccount fastmail
Host imap.fastmail.com
User my-user@my-domain.com
Pass my-password
SSLType IMAPS

MaildirStore local
Path ~/mail/
Inbox ~/mail/INBOX
SubFolders Verbatim

IMAPStore fastmail
Account fastmail

Channel primary
Far :fastmail:
Near :local:
Patterns *

A few notes here:

User replaces the smtpuser we had in .gitconfig and the user we had in accounts.conf.
Pass replaces the smtppass we had in .gitconfig and the password we had in accounts.conf.
I do have issues with Fastmail and CopyArrivalDate yes because isync fails to correctly parse the dates with a 1-digit day (so it fails at the beginning of each month). It turns out that Fastmail (Postfix actually, which is used by Fastmail) is fine and isync is the one parsing incorrectly. There is a fix upstream but it hasn’t made it to a release yet. In my case I compile isync from sources for that reason.

With this configuration, we can synchronize with the Fastmail server by running the following command manually:

mbsync -c ~/.config/isync/mbsync.conf primary

But that’s just for testing, I usually don’t call it directly. Remember: this is the command that Aerc will run every minute as part of the check-mail-cmd mechanism (see the Aerc setup above).

OpenSMTPD

With isync/mbsync, Aerc will fetch and store the emails on the disk for offline use. Now we will setup OpenSMTPD as a relay to send messages, so that they get queued and sent when an Internet connection is available.

Note that OpenSMTPD runs as a service and therefore has to be enabled. Changing the default smtpd.conf to support my use-case was pretty straightforward (it is even described in the “examples” section of the smtpd.conf manpage). First, I created a new /etc/smtpd/secrets file with the following content:

# /etc/smtpd/secrets

me my-user@my-domain.com:my-password

Then I had to add the following two lines to /etc/smtpd/smtpd.conf in order to enable the relay:

# /etc/smtpd/smtpd.conf

table secrets file:/etc/smtpd/secrets

[...]

action "relay" relay host smtp+tls://me@smtp.fastmail.com:587 auth <secrets>

And that was it! The final /etc/smtpd/smtpd.conf file looks like this (I highlighted the two lines I had to add to the default file):

# /etc/smtpd/smtpd.conf

# This is the smtpd server system-wide configuration file.
# See smtpd.conf(5) for more information.

table aliases file:/etc/smtpd/aliases
table secrets file:/etc/smtpd/secrets

# To accept external mail, add something like: listen on eth0
#
listen on lo

action "local" maildir alias <aliases>
action "relay" relay host smtp+tls://me@smtp.fastmail.com:587 auth <secrets>

# Uncomment the following to accept external mail for domain "example.org"
#
# match from any for domain "example.org" action "local"
match for local action "local"
match from local for any action "relay"

Conclusion

We have configured isync/mbsync to synchronise a local folder (~/mail) with the remote mail server, and OpenSMTPD to relay emails sent with /usr/sbin/sendmail (also to the mail server). This enables Aerc to show the emails and both Aerc and Git to send emails - even when offline.

I really like this setup for my “normal” email conversations, but I believe it really shines when used with the git email workflow where Aerc and Git can be used (possibly offline) to review and send patches. And if you do not know about the git email workflow, I recommend you read some of these resources.

Annex: other resources

In the process of setting up my system, I found the following resources very helpful:

Drew Devault’s post using Aerc, isync/mbsync and postfix
Bence Ferdinandy’s very detailed post using Aerc, isync/mbsync, msmtp(q), notmuch and even more (for authentication, automation and address book).
Sean Hammond’s post explaining how to use isync/mbsync with Fastmail and GMail, and using Mutt.
Aerc wiki describing a setup with Aerc, isync/mbsync and notmuch for ProtonMail.

Or was it even Hotmail, before that? ↩︎
Note that I write “isync/mbsync” everywhere because the tool is called “isync”, but the command to call it is “mbsync”. ↩︎

CMake superbuild

Fri, 10 Mar 2023 00:37:56 +0100

A superbuild is an approach in which a build system not only builds the main project, but also its dependencies, as part of the same build process. This post builds on top of the previous one and explores ways to achieve this with CMake.

TL;DR: In my opinion, a superbuild should act as a wrapper around a project and not impose any specific dependency management approach on users. However, with CMake I find that it introduces unnecessary indirections and overhead, and therefore superbuilds should not be used. I provide a small example of a CMake superbuild here.

You may already have a superbuild

If you manage your dependencies with add_subdirectory, it is already a superbuild. But it has downsides, as explained in the previous post. What if a dependency does not support CMake? What if the user wants to use a package manager instead of building the dependencies? Not to mention that a clean build will require rebuilding all the dependencies.

The find_package strategy, however, is not a superbuild itself (that’s the point, after all: it should delegate the dependency management). But it can be used in a superbuild; let’s see how.

Treat your project like a dependency

The idea is simple: write a script that will build the dependencies (in the right order), and then the main project. At the point where the main project is being built, CMake will find the dependencies that were just built with find_package. Because it stays completely out of the main project, it can be written in any language: shell, Python, or CMake.

With a shell script

Let us consider a simple example where our main project (main) has two dependencies: depA (using Automake) and depB (using CMake). Our main project depends on both, and depB depends on depA.

The superbuild shell script (let’s call it build.sh) would live in the parent directory w.r.t. main, like so:

.
├── build.sh
└── main
 └── CMakeLists.txt

Note that main is a completely standard, standalone CMake project. Assuming that the dependencies are installed on the system, it could be built normally (from the parent directory) with:

cmake -Bmain/build -Smain
cmake --build main/build

For a superbuild, we want to build the dependencies instead of relying on the system libraries. So we want build.sh to:

Fetch and build the dependencies
Build the main project

Let’s first fetch the dependencies in order to better visualize the directory tree. The script could start like this:

#!/bin/sh

git clone https://git-repo-1.com/depA
git clone https://git-repo-2.com/depB

Which would result in the following tree:

.
├── build.sh
├── depA
│   └── configure
│   └── Makefile
├── depB
│   └── CMakeLists.txt
└── main
 └── CMakeLists.txt

Then it would need to build the projects in order, using the right tooling (note how depA uses Automake):

#!/bin/sh

git clone https://example.com/depA
git clone https://example.com/depB

# Build depA and install it in ./install
pushd depA
./configure --prefix ../install
make install
popd

# Build depB and install it in ./install, too
# Note that depB depends on depA, so we point CMAKE_PREFIX_PATH
# to ./install (where depA is installed)
cmake -DCMAKE_PREFIX_PATH=$(pwd)/install -DCMAKE_INSTALL_PREFIX=install -BdepB/build -SdepB
cmake --build depB/build --target install

# Build our main project, also pointing CMAKE_PREFIX_PATH to ./install
cmake -DCMAKE_PREFIX_PATH=$(pwd)/install -Bmain/build -Smain
cmake --build main/build

That’s it, we have our superbuild! Without changing anything in the main CMakeLists.txt of the main project. Now the user can either build ./main/CMakeLists.txt like a normal CMake project (that will try to find the dependencies on the system), or call ./build.sh that will build and install the dependencies locally (in ./install) and will then build the main project with them.

Though it works, this shell script is probably not cross-platform. If we used CMake instead, then it would be better in that regard¹. Let’s see how we can leverage our CMake helpers from the previous post.

With CMake

We can achieve the same result as the build.sh above with a CMake script. It will look like this:

.
├── CMakeLists.txt
└── main
 └── CMakeLists.txt

I find it important to realise that ./CMakeLists.txt and ./main/CMakeLists.txt are two different CMake projects (just like build.sh and ./main/CMakeLists.txt were two separate things)! In other words, ./CMakeLists.txt will not call add_subdirectory(main). That is not the typical setup (with consequences that we will discuss later): usually, there is a root CMakeLists that calls sub-CMakeLists with add_subdirectory, making a tree where all CMakeLists belong to the same project. Here we really have two different CMake projects: the superbuild, and the main project. But the superbuild will build multiple separate projects (i.e. the main project and its dependencies).

The superbuild CMakeLists (i.e. ./CMakeLists.txt) could look like this:

cmake_minimum_required(VERSION 3.10.2)

project(superbuild)

ExternalProject_Add(depA
 PREFIX depA
 GIT_REPOSITORY https://example.com/depA
 CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=${CMAKE_INSTALL_PREFIX}
 BUILD_COMMAND ${CMAKE_MAKE_PROGRAM}
 INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install
)

ExternalProject_Add(depB
 PREFIX depA
 GIT_REPOSITORY https://example.com/depB
 CMAKE_CACHE_ARGS
 -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
 -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}
 DEPENDS depA
)

ExternalProject_Add(main
 PREFIX main
 URL ${CMAKE_CURRENT_SOURCE_DIR}/main
 CMAKE_CACHE_ARGS
 -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
 -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}
 DEPENDS depA depB
)

And it would be called like so:

cmake -DCMAKE_INSTALL_PREFIX=install -DCMAKE_PREFIX_PATH=$(pwd)/install -Bbuild -S.
cmake --build build

This is now equivalent to the superbuild setup we had with build.sh, except that instead of a shell script, we now use a CMake script. Again, the user can choose to run the superbuild or to directly build the main project (and use the dependencies installed on the system).

A real-life example of a one-file superbuild can be found here in the gRPC examples.

With CMake and helper scripts

Both superbuild scripts above look quite simple, but they don’t involved many dependencies, platforms and options. In reality, that file will tend to grow and become harder to maintain. That is why I would advise using a structure similar to my previous post, but adapted for the superbuild. With the CMake superbuild script, it looks like this:

.
├── CMakeLists.txt
├── dependencies
│   ├── depA
│   │   ├── Makefile
│   │   └── configure
│   └── depB
│   └── CMakeLists.txt
└── main
 └── CMakeLists.txt

The superbuild CMakeLists may now look like this:

cmake_minimum_required(VERSION 3.10.2)

project(superbuild)

add_subdirectory(dependencies)

ExternalProject_Add(main
 PREFIX main
 URL ${CMAKE_CURRENT_SOURCE_DIR}/main
 CMAKE_CACHE_ARGS
 -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
 -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}
 DEPENDS depA depB
)

Now the superbuild calls the dependencies project with add_subdirectory(), which in turn will call each dependency (also with add_subdirectory()), which will eventually call ExternalProject_Add() for each dependency. And finally it will call ExternalProject_Add() for the main project.

I do believe that this is the best structure for a superbuild, because:

It doesn’t pollute the main project, which remains a standalone CMake project that can be built manually with the system libraries. You can even have your CI test that building with the system libraries keeps working.
It is just a wrapper around the dependencies, which can still be built without the superbuild scheme (again, the CI can test it).
It doesn’t use any exotic/advanced CMake features (and ExternalProject_Add has been supported for many years, it is virtually always available).
The superbuild script does not have to be written with CMake. It can be done with shell scripts (by adapting the build.sh example above) or any language you want.

At the beginning of this post, I advised against superbuilds. Let’s now look at the disadvantages of this design².

Disadvantages

A superbuild is an abstraction on top of the build, and the goal of an abstraction is to make something either easier or faster to use. In this case, it tries to make it easier for users to build the project. But it comes at a cost.

Abstractions reduce flexibility

By definition, an abstraction cannot offer all the features of the lower-level interface. Therefore, choices have to be made for every feature that gets abstracted away. For instance with the helper scripts, the user can decide where the built dependencies should be installed (with CMAKE_INSTALL_PREFIX). They can choose to install the dependencies in some place, and the main project in another place (for instance in order to cache the dependencies instead of rebuilding them over and over).

If we want to keep this flexibility in the superbuild, we will probably have to add an option like -DDEPENDENCIES_INSTALL_PREFIX and document it. It may add complexity: when -DCMAKE_INSTALL_PREFIX is specified alone, should it impact both the main project and the dependencies, or only the main project? Or should we add another option, e.g. -DMAIN_INSTALL_PREFIX? But then what do we do with -DCMAKE_INSTALL_PREFIX?

For every option we may want to add to the superbuild, we will have to think about how it impacts the dependencies, how it impacts the main project, and how to forward the option properly. And it is likely that from time to time, some user of the superbuild will want to add an option in order to access a lower-level feature that got abstracted away. And as a maintainer, you will have to decide whether you want to keep adding features (which will progressively destroy your abstraction) or to refuse to do it (which may piss off the users). That is a situation I personally do not like to be in.

Abstractions need to be maintained

Adding an abstraction should - as much as possible - not prevent advanced users from using the lower-level interface. That is why we use find_package instead of add_subdirectory in the first place: a package maintainer (e.g. for a Linux distribution) will always want to use the system libraries. And as we saw above, a user may want to build the dependencies separately instead of using the superbuild.

Our design allows all of that, but it means that we have to maintain three ways of building the project (system libraries, helper scripts, superbuild). This is extra work, and maintainers usually have enough to do already.

Abstractions keep users ignorant

It is not always a bad thing; I could not use my computer without abstractions. But it is a tradeoff, and it should not be abused. In my experience with superbuilds, I have seen users recompiling all the dependencies every time they needed a clean build in the main project, and then complaining that the build “took forever”. And I honestly could not blame them, because the build system was not obvious to those unfamiliar with CMake. Whereas without the superbuild script, it is suddenly obvious that the CMakeLists.txt at the root is the main project, and that ./dependencies/CMakeLists.txt is for the dependencies.

Beginners still have to learn how to call CMake, but they are C++ developers: they will have to learn that anyway. I may as well teach them how to do it with my project rather than putting effort into maintaining a superbuild abstraction.

Conclusion

I have designed and used superbuilds myself, and I used to think that it was a good idea. But my experience is that they are usually² an unnecessary abstraction that requires maintenance work and risks making the build system convoluted.

I did my best to explain how I believe it should be done, but my opinion is that it should not be done. Instead I would recommend delegating the dependency management to the user (by using the system libraries with find_package), possibly providing helper scripts if there is a need to build the dependencies from source (I typically do that when I need to cross-compile for Android or iOS).

In my experience, it still requires some care, e.g. when applying patches in ExternalProject_Add on Windows. But other than that it works quite well. ↩︎
At least with CMake. ↩︎ ↩︎

Handling dependencies with CMake

Sun, 10 Jul 2022 22:13:47 +0200

Over the years, I have seen mainly two strategies (or variants of them) being used to handle dependencies in CMake projects. Because I only like one of them, this post is about explaining why my preferred strategy is better.

TL;DR: I strongly believe that a CMake project should use find_package instead of add_subdirectory. Feel free to try building this Proxygen example or this gRPC example before reading if you want to.

First strategy: `find_package`

The first strategy is about realising that CMake is not a package manager, and therefore delegating package management to the user (who can install the dependency using their preferred way). This is typically the favoured solution when starting a small project.

For instance:

cmake_minimum_required(VERSION 3.10.2)

project(hello_world)

find_package(OpenSSL REQUIRED)

add_executable(hello_world
 main.cpp
)

target_link_libraries(hello_world
 OpenSSL::SSL
)

Here, we do rely on find_package to “find” OpenSSL, following some rules (documented here), which implies that the user installed OpenSSL before (for instance using apt install). If OpenSSL is not to be found, CMake will complain during the configure step.

“Well, apt install libssl-dev is nice, but what if apt does not provide my dependency, or not the version of the dependency I want?”, you ask. That is where I believe the second strategy becomes (wrongfully) popular.

Second strategy: `add_subdirectory`

As a project grows, there often comes a need for a dependency (or a version of it) that is not provided by the system package manager. At this point it is not enough to just apt install <dependency>. There are many ways to handle that, but it seems like adding the dependency as a git submodule and building it inside the project with add_subdirectory has become popular.

As an example, say we do not want to use GoogleTest (libgtest) from the system. We can add it as a git submodule, say gtest/, and use it from CMake:

cmake_minimum_required(VERSION 3.10.2)

project(hello_world)

add_executable(hello_world
 main.cpp
)

add_subdirectory(gtest)

target_include_directories(hello_world
 SYSTEM
 PRIVATE ${PROJECT_SOURCE_DIR}/gtest/googletest/include
 )

target_link_libraries(hello_world
 gtest
 gtest_main
 )

There are alternatives to the git submodule, such as CMake’s own FetchContent and of course wrappers on top of that like CPM. But that is not fundamentally different; at the end of the day, this strategy builds the dependency with add_subdirectory.

I argue that handling dependencies with add_subdirectory is flawed:

What happens if your dependency does not support CMake? Say it uses Autotools, Meson or a custom build system instead? OpenSSL and Boost do not support CMake, for instance.
How does it work with transitive dependencies? Say your project runs add_subdirectory(libA) and add_subdirectory(libB), but libA also runs add_subdirectory(libB). What happens then? Which version of libB is being used?
Every clean build (e.g. in the CI) has to recompile all your dependencies. Isn’t it nicer (and potentially much faster) when the library can just be installed somewhere (or copied from a cache) once and for all?
You force the consumers of your library to use your strategy. What if I want to use your library, but I do not want you to include OpenSSL through add_subdirectory(openssl) (maybe I want to use LibreSSL)?

The find_package strategy does not have those problems. So why are so many projects using add_subdirectory? I believe that the answer is that it feels simpler.

Delegate, but assist

Ok, so delegating the package management to the user (i.e. using find_package) is nicer, but “copying” the dependencies inside the project (i.e. using add_subdirectory) is simpler.

I believe that the solution is to use find_package, but to provide helpers if necessary, so that we get the best of both worlds: those who know what they are doing can handle dependencies the way they like, and those who don’t can run the helper script.

But before getting to the helper script, I would like to review different ways one can handle dependencies.

Using the system package manager

As mentioned above, the easiest way to get a dependency is to install it globally on the system using the system package manager. For instance, apt install libssl-dev for the corresponding find_package(OpenSSL).

The problem is that you may rely on a library that is not provided by the system package manager. Another issue arises when you need to cross-compile your library (and its dependencies), because typically the system package manager only provides packages for the host architecture.

Manually building and installing globally

Instead of using the system package manager (which may not have your dependency), you can install the dependency manually.

For instance, for a CMake library:

cmake -Bbuild -S.
sudo cmake --build build --target install

Using Autotools, it could look like this:

./configure
make
sudo make install

And with Meson:

meson builddir && cd builddir
meson compile
meson install

That should result in the dependency being installed on your system, similar to what the system package manager would do. Except that the package manager cannot know about this, and it could create conflicts. I don’t like this way myself (and I would discourage the use of it), but I admittedly used to do it years ago, when I was blindly following instructions saying stuff like “now run make && sudo make install”. I still see a lot of READMEs doing that, so I’m pretty sure it is still used extensively.

Manually building and installing locally

Less known is the fact that one can install a project locally, by specifying an install path in the CMake configure step:

cmake -DCMAKE_INSTALL_PREFIX=/path/to/local/install -Bbuild -S.
cmake --build build --target install

Using Autotools, we can specify the installation prefix at configure time:

./configure --prefix=/path/to/local/install
make
make install

Similarly with Meson, we can use --prefix in the configure step:

meson builddir && cd builddir
meson configure --prefix /path/to/local/install
meson compile
meson install

The trick with locally-installed dependencies is that we need to inform CMake about the installation path (how would find_package guess otherwise?). This is done using CMAKE_PREFIX_PATH. So when you build your project (the lines above built the dependency), you have to specify this path in the CMake configure step:

cmake -DCMAKE_PREFIX_PATH=/path/to/local/install -Bbuild -S.
cmake --build build

Let me repeat that:

Build the dependency with -DCMAKE_INSTALL_PREFIX (CMake) or --prefix (Autoconf, Meson) to specify where it should be installed.
Build the project with -DCMAKE_PREFIX_PATH to tell CMake where it can find the dependencies.

Spoiler alert: this is my favourite, because I can keep the dependencies per project, and it’s super easy to clean them (to me it feels close to using a venv in Python). Also it is very convenient when cross-compiling. For these reasons, that is what the helper script will leverage.

Using another package manager

Instead of relying on your system package manager, there are alternatives out there, like Conan. I am not familiar with them, but as long as they are compatible with the find_package syntax, I don’t see a problem with them. For instance Conan provides a find_package_generator.

The one requirement I have for such a package manager is that it should be totally optional. For instance the find_package_generator of Conan is compatible with CMAKE_PREFIX_PATH. As a user, I want to have the choice to use Conan (and set CMAKE_PREFIX_PATH accordingly) or to use whatever system I want (even build the dependencies manually).

I am not a huge fan of Hunter because it requires changes in the CMakeLists. But again, as long as it is made optional and I am free to handle the dependencies my way, I am fine with it.

Finally, the problem with “third party” package managers is that you may need to maintain your own repository (if your dependencies are not available in the official repo). This is not only costly, but you cannot really expect users of your library to rely on your package manager and your repository. Which brings us to my main point again: use a package manager as a helper if you want to, but make it optional: your CMakeLists should not know about it at all, and instead rely on find_package.

The helper subproject

As we have seen above, the goal is to use find_package() for all the dependencies in our CMakeLists, and to let the user make sure that they are found (by using a package manager or building and installing them manually). But nothing prevents us from providing a helper script that fetches, builds and installs all our dependencies locally.

I have been mentioning a script, but the way I do it is actually more of a CMake subproject. I like it because it can be cross-platform, but the point is really that it builds and installs dependencies locally somewhere.

Using the helper

Let’s first see how the helper is being used:

# Run the helper script, installing the dependencies in `./dependencies/install`
cmake -DCMAKE_INSTALL_PREFIX=dependencies/install -Bdependencies/build -Sdependencies
cmake --build dependencies/build

# Build the project, telling `find_package()` where to find dependencies
cmake -DCMAKE_PREFIX_PATH=$(pwd)/dependencies/install -Bbuild -S.
cmake --build build

For a user of your project, that’s all it takes: running two CMake projects:

The first one builds the dependencies and installs them in a location chosen by the user (here ./dependencies/install).
The second one builds the project, using the dependencies installed in step 1 (specified with CMAKE_PREFIX_PATH).

From the user perspective, is that really more difficult than handling git submodules? I don’t think so.

Writing the helper

The helper script builds and installs all our dependencies locally by leveraging an old CMake feature: ExternalProject.

It is extremely easy; the syntax looks like this (there are more options nicely described in the documentation):

cmake_minimum_required(VERSION 3.10.2)

project(my-dependency)
include(ExternalProject)

ExternalProject_add(
 <dependency name>
 URL <link to sources>
 PREFIX <build directory for this dependency>
 CONFIGURE_COMMAND <configure command>
 BUILD_COMMAND <build command>
 CMAKE_ARGS <args passed to the cmake commands (if relevant)>
 )

The beauty of it is that while it uses CMake to build and install the dependency, it does not require the final project to use CMake. You could use this to install dependencies locally, and then use them from a Meson or Autotools-based project! It also means that instead of using CMake with ExternalProject_add, you are free to write your helper in shell, or python, or whatever you want; the whole point is that this is completely transparent to your main CMake project.

Let’s see a few examples below (with many more available here).

A CMake dependency: re2

cmake_minimum_required(VERSION 3.1)

project(external-re2)
include(ExternalProject)

list(APPEND CMAKE_ARGS
 "-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_INSTALL_PREFIX}"
 "-DCMAKE_POSITION_INDEPENDENT_CODE=ON"
 "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
 "-DBUILD_SHARED_LIBS=OFF"
 )

ExternalProject_add(
 re2
 URL https://github.com/google/re2/archive/2022-04-01.tar.gz
 PREFIX re2
 CMAKE_ARGS "${CMAKE_ARGS}"
 )

Feel free to try building it with:

cmake -DCMAKE_INSTALL_PREFIX=install -Bbuild -S.
cmake --build build

The build step will run ExternalProject_Add which will fetch, configure, build and install re2 into ./install. See how I passed CMAKE_INSTALL_PREFIX through CMAKE_ARGS? CMAKE_ARGS is also an opportunity to pass all the CMake options you want for this dependency, e.g. -DBUILD_TESTING=OFF, -DBUILD_SHARED_LIBS=OFF, -DSOME_OPTION=ON, you name it.

Also note the use of URL to download a tarball containing the sources. This can be replaced by GIT_REPOSITORY and GIT_TAG (see for instance jsoncpp here).

An Autotools dependency: libsodium

cmake_minimum_required(VERSION 3.10.2)

project(external-sodium)
include(ExternalProject)

ExternalProject_add(
 sodium
 URL https://download.libsodium.org/libsodium/releases/libsodium-1.0.18-stable.tar.gz
 PREFIX sodium
 CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=${CMAKE_INSTALL_PREFIX}
 BUILD_COMMAND make
 )

It can be run with the same commands as above:

cmake -DCMAKE_INSTALL_PREFIX=install -Bbuild -S.
cmake --build build

And it will result in libsodium being installed in ./install. Only this time it will have used Autotools (see the CONFIGURE_COMMAND and the BUILD_COMMAND).

A custom build system: boost

ExternalProject is very versatile, as can be seen when building Boost using their custom build system:

cmake_minimum_required(VERSION 3.10.2)

project(external-boost)
include(ExternalProject)

ExternalProject_add(
 boost
 URL https://boostorg.jfrog.io/artifactory/main/release/1.80.0/source/boost_1_80_0.tar.gz
 PREFIX boost
 CONFIGURE_COMMAND <SOURCE_DIR>/bootstrap.sh --prefix=${CMAKE_INSTALL_PREFIX}
 BUILD_COMMAND <SOURCE_DIR>/b2
 BUILD_IN_SOURCE TRUE
 INSTALL_COMMAND <SOURCE_DIR>/b2 install
 )

All together

The examples above show how to build one dependency using ExternalProject, and all that remains now is to group them into one, by just calling all of them from one parent CMakeLists:

cmake_minimum_required(VERSION 3.10.2)

project(dependencies)

add_subdirectory(re2)
add_subdirectory(sodium)
add_subdirectory(boost)

One more note: in the case of transitive dependencies, we need to install the dependencies in the right order and let new ones know about the install path. So typically the dependencies are built like this (when there are transitive dependencies):

cmake -DCMAKE_PREFIX_PATH=$(pwd)/install -DCMAKE_INSTALL_PREFIX=install -Bbuild -S.
cmake --build build

This just means that dependencies will be installed into ./install (as per CMAKE_INSTALL_PREFIX), and CMake will look for dependencies in $(pwd)/install (as per CMAKE_PREFIX_PATH) when it encounters a find_package instruction.

Conclusion

CMake is not a package manager, and therefore a proper CMake project should delegate the package management to the user. The way to do this is to use find_package for dependencies (and not add_subdirectory, which has many downsides).

Optionally, one can provide a helper script to build and install the dependencies locally, as suggested in this post. Of course, it can be done differently (as long as it works with find_package through CMAKE_PREFIX_PATH). At Facebook, for instance, they seem to have their way.

I wrote a small project illustrating the helper strategy here (using Facebook/Proxygen because it has a few transitive dependencies), and another one using gRPC. I also provide more dependency scripts here for inspiration.

Annex

What if the dependency does not play well with `find_package`?

If the dependency is a CMake project, it should support find_package. In case it does not, consider contributing it, or at least ask the maintainer to add it. If they don’t want to, and you can’t contribute it, maybe it is a good time to wonder whether you want to depend on that library, really.

This said, what about dependencies that are not CMake projects? It does make sense for a non-CMake library to not provide a CMake package configuration file (<PackageName>Config.cmake) indeed. Fortunately that usually happens in one of the following two scenarios:

The library supports pkg-config. The good news is that CMake supports pkg-config, too (I like to hide that into a CMake Find Module, but that’s a longer story). For instance, gstreamer uses Meson, but supports pkg-config:
```
find_package(PkgConfig REQUIRED)
pkg_search_module(GST REQUIRED gstreamer-1.0>=1.4
```
The library is so popular that CMake provides a Find Module. That’s the case for OpenSSL and Boost, for instance; they do not support CMake, but find_package(OpenSSL) and find_package(Boost) both works because CMake supports them.

Worst case, you can always write a Find Module manually, but I never really had to myself (except to hide a call to pkg_search_module, as mentioned above).

What if I actually need `add_subdirectory`?

There is exactly one reason I can think of where one would want to add a dependency using add_subdirectory, and that’s when the author of the main project is also the author of the dependency, and wants to develop both in parallel. In that case, I would still prefer ExternalProject_add (i.e. using the helper script) which supports local paths. However, if your IDE really needs add_subdirectory, then you can still support both, like gRPC does (though I personally find this slightly convoluted).