2026-06-06 - How am I supposed to run my own tools from source?
The problem
In modern Linux, the software environment is largely controlled by the
distro’s package manager. Tools in /usr/bin, libraries in
/usr/lib, config files in /etc – all these locations have content
that is placed there by packages that comprise the OS installation on
the system.
So where am I supposed to put tools I build myself?
Replacing tools delivered in packages is a bad idea. If you replace,
for example, /usr/bin/rsync with a version you built yourself, it
will cause problems. Tools that check the integrity of installed
packages will detect that the file content is not a match for the one
delivered by the package that put it there. If the distro puts out a
new version of the package, the file will probably get overwritten
with a new one from the package. Merely adding to a location like
/usr/bin, even if not overwriting existing files, is still a risk
because some future update might bring in a package that overwrites
the locally-added file.
The obvious next place to consider is /usr/local. In many ways,
this seems like the right location. But over the years, I have seen
systems with things in /usr/local that I didn’t put there myself,
leaving me with the sense that I don’t completely control it. Maybe
some of this was things distributed outside the package manager
ecosystem, like proprietary tools installed by running shell scripts
that unpack files. That has left me a little uncertain about how much
I can just do whatever I want in /usr/local.
That leads us to /opt. Though it has been around for long time, it
is a somewhat newer concept; I first encountered it when HP-UX was
shifting from a BSD-style layout to one based on System V. It does
not seem to have been widely utilized, which I think makes it great as
a place to put things with reasonable confidence that they won’t
clash. It also is more conducive to having multiple versions of the
same thing side-by-side, because you can compartmentalize into trees
like /opt/rsync-v3.2.7, /opt/rsync-v3.3.0, and so on. Each of
these trees has its own set of the usual subdirectories – bin,
lib, and so on. This achieves better separation then /usr/local,
where different tools and versions all land together in
/usr/local/bin and its neighbours.
The problem with having a bunch of /opt/package/bin directories is
that they aren’t in the default search path. You can try to change
the default search path, but good luck getting that change to stick
system-wide. And attempting to change that value for all processes
every time a new version lands in /opt is unwieldy, and feels wrong.
On the other hand, /usr/local/bin usually is in the default search
path, with higher precedence than /usr/bin, which makes a very
strong argument for relying on it to resolve to our locally-built
tools.
A tentative solution
I think the most manageable way to build add-on tools from source and
install them so that they supersede distro-provided tools will be to
build them for installation into subdirectories of /opt, and then
use symlinks in /usr/local/bin to make those versions appear in the
path. Anything that uses the search path to resolve commands will
find them in /usr/local, while using subdirectories of /opt allows
new versions to be dropped in immutably and reversibly.
The problem of compiled-in paths
Many tools get their install path baked in at build time. This
prevents them from being freely relocatable. You often can’t take
something that was built with --prefix=/usr/local and just move
everything to the corresponding locations under /opt/packagename and
have it work. That is because programs often use resources that they
open at runtime, like data files and configuration. These paths often
are fixed at build time based on the prefix value, and if the files
aren’t at the expected location, the program doesn’t work right.
That’s why my strategy is to build the package for deployment under
/opt, telling it that’s the location at build time. (“Build” may
not literally be compilation; in some cases it’s something like making
a Python virtual environment. The same baked-in-paths issue still
applies.) When the program runs, it can find whatever things it
expects under the path in /opt that it was configured with.
/usr/local/bin is just a way to insert things into the search path,
but the things that it points to will know themselves as living under
/opt. This should work in almost every case, with the possible
exception of programs that try to parse argv[0] and operate relative
to that. I think this kind of behaviour is rare.
The other problem of compiled-in paths
If tools provided by the distro run supporting commands by their base
name, using the search path to resolve them to actual programs, then
the add-on versions in /usr/local will get used. However, there may
be tools provided by the distro that are hard-coded to run things out
of /usr/bin. In that case, they will get the distro’s own version,
rather than superseding it with the one higher in the search path.
Which behaviour is correct? It depends. If there is a strict revlock
between distro-provided tools, we might want things in /usr/bin to
call other things in /usr/bin. If we are simply trying to globally
replace a tool with our own version, we would prefer it always use the
search path. What the tools actually do and what we want may not
always align, and that’s just something we have to live with and be on
the lookout for. As much as possible, when injecting our own version
via /usr/local, it’s probably best to avoid installing the
corresponding package via the distro, though this may not always be
possible because of recursive dependencies.
What I’m not considering
Another strategy would be to build my own packages that I can install with the package manager. I don’t want to do this. Every time I have tried to do something like this, it has not gone well. Some projects come with configuration to build common distro packages, and some don’t. For the ones that don’t, I would have to construct my own scaffolding to produce packages. For the ones that do, it’s still another stage of the build process that I have to figure out how to run and get through and then test the result. Then I have to maintain these packages somewhere, and maybe sign them or otherwise mark them as being trustworthy. And I have sometimes seen distro major version upgrades eject everything that didn’t seem to come from the distro, leading to more work to try to recover the damage after an upgrade.
What I want is to simply copy files into place and use them. I want
package installation to be cp -r and uninstallation to be rm -r.
Package managers impose more complexity than I want, and demand more
effort to conform to their way of doing things than I want to give.
Conclusion
I’m going to build for /opt, and invoke via linkage in
/usr/local/bin and see how it goes. I’m looking forward to seeing
how it feels to get back to a way of doing things that I enjoyed
before about 2005 when package management took over everything.