2. Important Information

This chapter is used to explain some of the policies used throughout the book, to introduce important concepts and to explain some issues you may see with some of the included packages.

2.1 Notes on Building Software

Those people who have built an LFS system may be aware of the general principles of downloading and unpacking software. Some of that information is repeated here for those new to building their own software.

Each set of installation instructions contains a URL from which you can download the package. The patches; however, are stored on the LFS servers and are available via HTTP. These are referenced as needed in the installation instructions.

While you can keep the source files anywhere you like, we assume that you have unpacked the package and changed into the directory created by the unpacking process (the ‘build’ directory). We also assume you have uncompressed any required patches and they are in the directory immediately above the ‘build’ directory.

We can not emphasize strongly enough that you should start from a clean source tree each time. This means that if you have had an error during configuration or compilation, it’s usually best to delete the source tree and re-unpack it before trying again. This obviously doesn’t apply if you’re an advanced user used to hacking Makefiles and C code, but if in doubt, start from a clean tree.

Building Software as an Unprivileged (non-root) User

The golden rule of Unix System Administration is to use your superpowers only when necessary. Hence, BLFS recommends that you build software as an unprivileged user and only become the root user when installing the software. This philosophy is followed in all the packages in this book. Unless otherwise specified, all instructions should be executed as an unprivileged user. The book will advise you on instructions that need root privileges.

Unpacking the Software

If a file is in .tar format and compressed, it is unpacked by running one of the following commands:

tar -xvf filename.tar.gz
tar -xvf filename.tgz
tar -xvf filename.tar.Z
tar -xvf filename.tar.bz2

Note

You may omit using the v parameter in the commands shown above and below if you wish to suppress the verbose listing of all the files in the archive as they are extracted. This can help speed up the extraction as well as make any errors produced during the extraction more obvious to you.

You can also use a slightly different method:

bzcat filename.tar.bz2 | tar -xv

Finally, you sometimes need to be able to unpack patches which are generally not in .tar format. The best way to do this is to copy the patch file to the parent of the ‘build’ directory and then run one of the following commands depending on whether the file is a .gz or .bz2 file:

gunzip -v patchname.gz
bunzip2 -v patchname.bz2

Verifying File Integrity

Generally, to verify that the downloaded file is complete, many package maintainers also distribute md5sums of the files. To verify the md5sum of the downloaded files, download both the file and the corresponding md5sum file to the same directory (preferably from different on-line locations), and (assuming file.md5sum is the md5sum file downloaded) run the following command:

md5sum -c file.md5sum

If there are any errors, they will be reported. Note that the BLFS book includes md5sums for all the source files also. To use the BLFS supplied md5sums, you can create a file.md5sum (place the md5sum data and the exact name of the downloaded file on the same line of a file, separated by white space) and run the command shown above. Alternately, simply run the command shown below and compare the output to the md5sum data shown in the BLFS book.

md5sum <name_of_downloaded_file>

MD5 is not cryptographically secure, so the md5sums are only provided for detecting unmalicious changes to the file content. For example, an error or truncation introduced during network transfer, or a “stealth” update to the package from the upstream (updating the content of a released tarball instead of making a new release properly).

There is no “100%” secure way to make sure the genuity of the source files. Assuming the upstream is managing their website correctly (the private key is not leaked and the domain is not hijacked), and the trust anchors have been set up correctly using make-ca-1.12 on the BLFS system, we can reasonably trust download URLs to the upstream official website with https protocol. Note that BLFS book itself is published on a website with https, so you should already have some confidence in https protocol or you wouldn’t trust the book content.

If the package is downloaded from an unofficial location (for example a local mirror), checksums generated by cryptographically secure digest algorithms (for example SHA256) can be used to verify the genuity of the package. Download the checksum file from the upstream official website (or somewhere you can trust) and compare the checksum of the package from unofficial location with it. For example, SHA256 checksum can be checked with the command:

Note

If the checksum and the package are downloaded from the same untrusted location, you won’t gain security enhancement by verifying the package with the checksum. The attacker can fake the checksum as well as compromising the package itself.

sha256sum -c file.sha256sum

If GnuPG-2.4.0 is installed, you can also verify the genuity of the package with a GPG signature. Import the upstream GPG public key with:

gpg --recv-key keyID

keyID should be replaced with the key ID from somewhere you can trust (for example, copy it from the upstream official website using https). Now you can verify the signature with:

gpg --recv-key file.sig file

The advantage of GnuPG signature is, once you imported a public key which can be trusted, you can download both the package and its signature from the same unofficial location and verify them with the public key. So you won’t need to connect to the official upstream website to retrieve a checksum for each new release. You only need to update the public key if it’s expired or revoked.

Creating Log Files During Installation

For larger packages, it is convenient to create log files instead of staring at the screen hoping to catch a particular error or warning. Log files are also useful for debugging and keeping records. The following command allows you to create an installation log. Replace <command> with the command you intend to execute.

( <command> 2>&1 | tee compile.log && exit $PIPESTATUS )

2>&1 redirects error messages to the same location as standard output. The tee command allows viewing of the output while logging the results to a file. The parentheses around the command run the entire command in a subshell and finally the exit $PIPESTATUS command ensures the result of the <command> is returned as the result and not the result of the tee command.

Using Multiple Processors

For many modern systems with multiple processors (or cores) the compilation time for a package can be reduced by performing a “parallel make” by either setting an environment variable or telling the make program how many processors are available. For instance, a Core2Duo can support two simultaneous processes with:

export MAKEFLAGS='-j2'

or just building with:

make -j2

If you have applied the optional sed when building ninja in LFS, you can use:

export NINJAJOBS=2

when a package uses ninja, or just:

ninja -j2

but for ninja, the default number of jobs is +2, where is the number of processors available, so that using the above commands is rather for limiting the number of jobs (see below for why this could be necessary).

Generally the number of processes should not exceed the number of cores supported by the CPU. To list the processors on your system, issue: grep processor /proc/cpuinfo.

In some cases, using multiple processes may result in a ‘race’ condition where the success of the build depends on the order of the commands run by the make program. For instance, if an executable needs File A and File B, attempting to link the program before one of the dependent components is available will result in a failure. This condition usually arises because the upstream developer has not properly designated all the prerequisites needed to accomplish a step in the Makefile.

If this occurs, the best way to proceed is to drop back to a single processor build. Adding ‘-j1’ to a make command will override the similar setting in the MAKEFLAGS environment variable.

Note

When running the package tests or the install portion of the package build process, we do not recommend using an option greater than ‘-j1’ unless specified otherwise. The installation procedures or checks have not been validated using parallel procedures and may fail with issues that are difficult to debug.

Important

Another problem may occur with modern CPU’s, which have a lot of cores. Each job started consumes memory, and if the sum of the needed memory for each job exceeds the available memory, you may encounter either an OOM (Out of Memory) kernel interrupt or intense swapping that will slow the build beyond reasonable limits.

Some compilations with g++ may consume up to 2.5 GB of memory, so to be safe, you should restrict the number of jobs to (Total Memory in GB)/2.5, at least for big packages such as LLVM, WebKitGtk, QtWebEngine, or libreoffice.

Automated Building Procedures

There are times when automating the building of a package can come in handy. Everyone has their own reasons for wanting to automate building, and everyone goes about it in their own way. Creating Makefiles, Bash scripts, Perl scripts or simply a list of commands used to cut and paste are just some of the methods you can use to automate building BLFS packages. Detailing how and providing examples of the many ways you can automate the building of packages is beyond the scope of this section. This section will expose you to using file redirection and the yes command to help provide ideas on how to automate your builds.

File Redirection to Automate Input

You will find times throughout your BLFS journey when you will come across a package that has a command prompting you for information. This information might be configuration details, a directory path, or a response to a license agreement. This can present a challenge to automate the building of that package. Occasionally, you will be prompted for different information in a series of questions. One method to automate this type of scenario requires putting the desired responses in a file and using redirection so that the program uses the data in the file as the answers to the questions.

Building the CUPS package is a good example of how redirecting a file as input to prompts can help you automate the build. If you run the test suite, you are asked to respond to a series of questions regarding the type of test to run and if you have any auxiliary programs the test can use. You can create a file with your responses, one response per line, and use a command similar to the one shown below to automate running the test suite:

make check < ../cups-1.1.23-testsuite_parms

This effectively makes the test suite use the responses in the file as the input to the questions. Occasionally you may end up doing a bit of trial and error determining the exact format of your input file for some things, but once figured out and documented you can use this to automate building the package.

Using yes to Automate Input

Sometimes you will only need to provide one response, or provide the same response to many prompts. For these instances, the yes command works really well. The yes command can be used to provide a response (the same one) to one or more instances of questions. It can be used to simulate pressing just the Enter key, entering the Y key or entering a string of text. Perhaps the easiest way to show its use is in an example.

First, create a short Bash script by entering the following commands:

cat > blfs-yes-test1 << "EOF"
#!/bin/bash

echo -n -e "\n\nPlease type something (or nothing) and press Enter ---> "

read A_STRING

if test "$A_STRING" = ""; then A_STRING="Just the Enter key was pressed"
else A_STRING="You entered '$A_STRING'"
fi

echo -e "\n\n$A_STRING\n\n"
EOF
chmod 755 blfs-yes-test1

Now run the script by issuing ./blfs-yes-test1 from the command line. It will wait for a response, which can be anything (or nothing) followed by the Enter key. After entering something, the result will be echoed to the screen. Now use the yes command to automate the entering of a response:

yes | ./blfs-yes-test1

Notice that piping yes by itself to the script results in y being passed to the script. Now try it with a string of text:

yes 'This is some text' | ./blfs-yes-test1

The exact string was used as the response to the script. Finally, try it using an empty (null) string:

yes '' | ./blfs-yes-test1

Notice this results in passing just the press of the Enter key to the script. This is useful for times when the default answer to the prompt is sufficient. This syntax is used in the Net-tools instructions to accept all the defaults to the many prompts during the configuration step. You may now remove the test script, if desired.

File Redirection to Automate Output

In order to automate the building of some packages, especially those that require you to read a license agreement one page at a time, requires using a method that avoids having to press a key to display each page. Redirecting the output to a file can be used in these instances to assist with the automation. The previous section on this page touched on creating log files of the build output. The redirection method shown there used the tee command to redirect output to a file while also displaying the output to the screen. Here, the output will only be sent to a file.

Again, the easiest way to demonstrate the technique is to show an example. First, issue the command:

ls -l /usr/bin | more

Of course, you’ll be required to view the output one page at a time because the more filter was used. Now try the same command, but this time redirect the output to a file. The special file /dev/null can be used instead of the filename shown, but you will have no log file to examine:

ls -l /usr/bin | more > redirect_test.log 2>&1

Notice that this time the command immediately returned to the shell prompt without having to page through the output. You may now remove the log file.

The last example will use the yes command in combination with output redirection to bypass having to page through the output and then provide a y to a prompt. This technique could be used in instances when otherwise you would have to page through the output of a file (such as a license agreement) and then answer the question of “do you accept the above?”. For this example, another short Bash script is required:

cat > blfs-yes-test2 << "EOF"
#!/bin/bash

ls -l /usr/bin | more

echo -n -e "\n\nDid you enjoy reading this? (y,n) "

read A_STRING

if test "$A_STRING" = "y"; then A_STRING="You entered the 'y' key"
else A_STRING="You did NOT enter the 'y' key"
fi

echo -e "\n\n$A_STRING\n\n"
EOF
chmod 755 blfs-yes-test2

This script can be used to simulate a program that requires you to read a license agreement, then respond appropriately to accept the agreement before the program will install anything. First, run the script without any automation techniques by issuing ./blfs-yes-test2.

Now issue the following command which uses two automation techniques, making it suitable for use in an automated build script:

yes | ./blfs-yes-test2 > blfs-yes-test2.log 2>&1

If desired, issue tail blfs-yes-test2.log to see the end of the paged output, and confirmation that y was passed through to the script. Once satisfied that it works as it should, you may remove the script and log file.

Finally, keep in mind that there are many ways to automate and/or script the build commands. There is not a single “correct” way to do it. Your imagination is the only limit.

Dependencies

For each package described, BLFS lists the known dependencies. These are listed under several headings, whose meaning is as follows:

Using the Most Current Package Sources

On occasion you may run into a situation in the book when a package will not build or work properly. Though the Editors attempt to ensure that every package in the book builds and works properly, sometimes a package has been overlooked or was not tested with this particular version of BLFS.

If you discover that a package will not build or work properly, you should see if there is a more current version of the package. Typically this means you go to the maintainer’s web site and download the most current tarball and attempt to build the package. If you cannot determine the maintainer’s web site by looking at the download URLs, use Google and query the package’s name. For example, in the Google search bar type: ‘package_name download’ (omit the quotes) or something similar. Sometimes typing: ‘package_name home page’ will result in you finding the maintainer’s web site.

Stripping One More Time

In LFS, stripping of debugging symbols and unneeded symbol table entries was discussed a couple of times. When building BLFS packages, there are generally no special instructions that discuss stripping again. Stripping can be done while installing a package, or afterwards.

Stripping while Installing a Package

There are several ways to strip executables installed by a package. They depend on the build system used (see below the section about build systems), so only some generalities can be listed here:

Note

The following methods using the feature of a building system (autotools, meson, or cmake) will not strip static libraries if any is installed. Fortunately there are not too many static libraries in BLFS, and a static library can always be stripped safely by running strip –strip-unneeded on it manually.

Stripping Installed Executables

The strip utility changes files in place, which may break anything using it if it is loaded in memory. Note that if a file is in use but just removed from the disk (i.e. not overwritten nor modified), this is not a problem since the kernel can use “deleted” files. Look at /proc/*/maps and it is likely that you’ll see some (deleted) entries. The mv just removes the destination file from the directory but does not touch its content, so that it satisfies the condition for the kernel to use the old (deleted) file. The script below is just an example. It should be run as the root user:

cat > /usr/sbin/strip-all.sh << "EOF"
#!/usr/bin/bash

if [ $EUID -ne 0 ]; then
  echo "Need to be root"
  exit 1
fi

{ find /usr/lib -type f -name '*.so*' ! -name '*dbg'
  find /usr/lib -type f -name '*.a'
  find /usr/{bin,sbin,libexec} -type f
} |  while read file; do
       if ! readelf -h $file >/dev/null 2>&1; then continue; fi
       if file $file | grep --quiet --invert-match 'not stripped'; then continue; fi

       cp --preserve $file    ${file}.tmp
       strip --strip-unneeded ${file}.tmp
       mv ${file}.tmp $file
     done
EOF
chmod 744 /usr/sbin/strip-all.sh

If you install programs in other directories such as /opt or /usr/local, you may want to strip the files there too. Just add other directories to scan in the compound list of find commands between the braces.

For more information on stripping, see https://www.technovelty.org/linux/stripping-shared-libraries.html.

Working with different build systems

There are now three different build systems in common use for converting C or C++ source code into compiled programs or libraries and their details (particularly, finding out about available options and their default values) differ. It may be easiest to understand the issues caused by some choices (typically slow execution or unexpected use of, or omission of, optimizatons) by starting with the CFLAGS and CXXFLAGS environment variables. There are also some programs which use rust.

Most LFS and BLFS builders are probably aware of the basics of CFLAGS and CXXFLAGS for altering how a program is compiled. Typically, some form of optimization is used by upstream developers (-O2 or -O3), sometimes with the creation of debug symbols (-g), as defaults.

If there are contradictory flags (e.g. multiple different -O values), the last value will be used. Sometimes this means that flags specified in environment variables will be picked up before values hardcoded in the Makefile, and therefore ignored. For example, where a user specifies ‘-O2’ and that is followed by ‘-O3’ the build will use ‘-O3’.

There are various other things which can be passed in CFLAGS or CXXFLAGS, such as forcing compilation for a specific microarchitecture (e.g. -march=amdfam10, -march=native) or specifying a specific standard for C or C++ (-std=c++17 for example). But one thing which has now come to light is that programmers might include debug assertions in their code, expecting them to be disabled in releases by using -DNDEBUG. Specifically, if Mesa-22.3.5 is built with these assertions enabled, some activities such as loading levels of games can take extremely long times, even on high-class video cards.

Autotools with Make

This combination is often described as ‘CMMI’ (configure, make, make install) and is used here to also cover the few packages which have a configure script that is not generated by autotools.

Sometimes running ./configure –help will produce useful options about switches which might be used. At other times, after looking at the output from configure you may need to look at the details of the script to find out what it was actually searching for.

Many configure scripts will pick up any CFLAGS or CXXFLAGS from the environment, but CMMI packages vary about how these will be mixed with any flags which would otherwise be used (variously: ignored, used to replace the programmer’s suggestion, used before the programmer’s suggestion, or used after the programmer’s suggestion).

In most CMMI packages, running ‘make’ will list each command and run it, interspersed with any warnings. But some packages try to be ‘silent’ and only show which file they are compiling or linking instead of showing the command line. If you need to inspect the command, either because of an error, or just to see what options and flags are being used, adding ‘V=1’ to the make invocation may help.

CMake

CMake works in a very different way, and it has two backends which can be used on BLFS: ‘make’ and ‘ninja’. The default backend is make, but ninja can be faster on large packages with multiple processors. To use ninja, specify ‘-G Ninja’ in the cmake command. However, there are some packages which create fatal errors in their ninja files but build successfully using the default of Unix Makefiles.

The hardest part of using CMake is knowing what options you might wish to specify. The only way to get a list of what the package knows about is to run cmake -LAH and look at the output for that default configuration.

Perhaps the most-important thing about CMake is that it has a variety of CMAKE_BUILD_TYPE values, and these affect the flags. The default is that this is not set and no flags are generated. Any CFLAGS or CXXFLAGS in the environment will be used. If the programmer has coded any debug assertions, those will be enabled unless -DNDEBUG is used. The following CMAKE_BUILD_TYPE values will generate the flags shown, and these will come after any flags in the environment and therefore take precedence.

Value Flags
Debug -g
Release -O3 -DNDEBUG
RelWithDebInfo -O2 -g -DNDEBUG
MinSizeRel -Os -DNDEBUG

CMake tries to produce quiet builds. To see the details of the commands which are being run, use make VERBOSE=1 or ninja -v.

By default, CMake treats file installation differently from the other build systems: if a file already exists and is not newer than a file that would overwrite it, then the file is not installed. This may be a problem if a user wants to record which file belongs to a package, either using LD_PRELOAD, or by listing files newer than a timestamp. The default can be changed by setting the variable CMAKE_INSTALL_ALWAYS to 1 in the environment, for example by export‘ing it.

Meson

Meson has some similarities to CMake, but many differences. To get details of the defines that you may wish to change you can look at meson_options.txt which is usually in the top-level directory.

If you have already configured the package by running meson and now wish to change one or more settings, you can either remove the build directory, recreate it, and use the altered options, or within the build directory run meson configure, e.g. to set an option:

meson configure -D<some_option>=true

If you do that, the file meson-private/cmd_line.txt will show the last commands which were used.

Meson provides the following buildtype values, and the flags they enable come after any flags supplied in the environment and therefore take precedence.

Although the ‘release’ buildtype is described as enabling -DNDEBUG, and all CMake Release builds pass that, it has so far only been observed (in verbose builds) for Mesa-22.3.5. That suggests that it might only be used when there are debug assertions present.

The -DNDEBUG flag can also be provided by passing -Db_ndebug=true.

To see the details of the commands which are being run in a package using meson, use ‘ninja -v’.

Rustc and Cargo

Most released rustc programs are provided as crates (source tarballs) which will query a server to check current versions of dependencies and then download them as necessary. These packages are built using cargo –release. In theory, you can manipulate the RUSTFLAGS to change the optimize-level (default is 3, like -O3, e.g. -Copt-level=3) or to force it to build for the machine it is being compiled on, using -Ctarget-cpu=native but in practice this seems to make no significant difference.

If you find an interesting rustc program which is only provided as unpackaged source, you should at least specify RUSTFLAGS=-Copt-level=2 otherwise it will do an unoptimized compile with debug info and run much slower.

The rust developers seem to assume that everyone will compile on a machine dedicated to producing builds, so by default all CPUs are used. This can often be worked around, either by exporting CARGO_BUILD_JOBS= or passing --jobs to cargo. For compiling rustc itself, specifying --jobs on invocations of x.py (together with the `CARGO_BUILD_JOBS` environment variable, which looks like a "belt and braces" approach but seems to be necessary) mostly works. The exception is running the tests when building rustc, some of them will nevertheless use all online CPUs, at least as of rustc-1.42.0.

Optimizing the build

Many people will prefer to optimize compiles as they see fit, by providing CFLAGS or CXXFLAGS. For an introduction to the options available with gcc and g++ see https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html and https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html and info gcc.

Some packages default to ‘-O2 -g’, others to ‘-O3 -g’, and if CFLAGS or CXXFLAGS are supplied they might be added to the package’s defaults, replace the package’s defaults, or even be ignored. There are details on some desktop packages which were mostly current in April 2019 at https://www.linuxfromscratch.org/~ken/tuning/ - in particular, README.txt, tuning-1-packages-and-notes.txt, and tuning-notes-2B.txt. The particular thing to remember is that if you want to try some of the more interesting flags you may need to force verbose builds to confirm what is being used.

Clearly, if you are optimizing your own program you can spend time to profile it and perhaps recode some of it if it is too slow. But for building a whole system that approach is impractical. In general, -O3 usually produces faster programs than -O2. Specifying -march=native is also beneficial, but means that you cannot move the binaries to an incompatible machine - this can also apply to newer machines, not just to older machines. For example programs compiled for ‘amdfam10’ run on old Phenoms, Kaveris, and Ryzens : but programs compiled for a Kaveri will not run on a Ryzen because certain op-codes are not present. Similarly, if you build for a Haswell not everything will run on a SandyBridge.

There are also various other options which some people claim are beneficial. At worst, you get to recompile and test, and then discover that in your usage the options do not provide a benefit.

If building Perl or Python modules, or Qt packages which use qmake, in general the CFLAGS and CXXFLAGS used are those which were used by those ‘parent’ packages.

Options for hardening the build

Even on desktop systems, there are still a lot of exploitable vulnerabilities. For many of these, the attack comes via javascript in a browser. Often, a series of vulnerabilities are used to gain access to data (or sometimes to pwn, i.e. own, the machine and install rootkits). Most commercial distros will apply various hardening measures.

In the past, there was Hardened LFS where gcc (a much older version) was forced to use hardening (with options to turn some of it off on a per-package basis). The current LFS and BLFS books are carrying forward a part of its spirit by enabling PIE (-fPIE -pie) and SSP (-fstack-protector-strong) as the defaults for GCC and clang. What is being covered here is different - first you have to make sure that the package is indeed using your added flags and not over-riding them.

For hardening options which are reasonably cheap, there is some discussion in the ‘tuning’ link above (occasionally, one or more of these options might be inappropriate for a package). These options are -D_FORTIFY_SOURCE=2 and (for C++) -D_GLIBCXX_ASSERTIONS. On modern machines these should only have a little impact on how fast things run, and often they will not be noticeable.

The main distros use much more, such as RELRO (Relocation Read Only) and perhaps -fstack-clash-protection. You may also encounter the so-called “userspace retpoline” (-mindirect-branch=thunk etc.) which is the equivalent of the spectre mitigations applied to the linux kernel in late 2018. The kernel mitigations caused a lot of complaints about lost performance, if you have a production server you might wish to consider testing that, along with the other available options, to see if performance is still sufficient.

Whilst gcc has many hardening options, clang/LLVM’s strengths lie elsewhere. Some options which gcc provides are said to be less effective in clang/LLVM.

2.2 The /usr Versus /usr/local Debate

Should I install XXX in /usr or /usr/local?

This is a question without an obvious answer for an LFS based system.

In traditional Unix systems, /usr usually contains files that come with the system distribution, and the /usr/local tree is free for the local administrator to manage. The only really hard and fast rule is that Unix distributions should not touch /usr/local, except perhaps to create the basic directories within it.

With Linux distributions like Red Hat, Debian, etc., a possible rule is that /usr is managed by the distribution’s package system and /usr/local is not. This way the package manager’s database knows about every file within /usr.

LFS users build their own system and so deciding where the system ends and local files begin is not straightforward. So the choice should be made in order to make things easier to administer. There are several reasons for dividing files between /usr and /usr/local.

Some people ask why not use your own directory tree, e.g., /usr/site, rather than /usr/local?

There is nothing stopping you, many sites do make their own trees, however it makes installing new software more difficult. Automatic installers often look for dependencies in /usr and /usr/local, and if the file it is looking for is in /usr/site instead, the installer will probably fail unless you specifically tell it where to look.

What is the BLFS position on this?

All of the BLFS instructions install programs in /usr with optional instructions to install into /opt for some specific packages.

2.3 Optional Patches

As you follow the various sections in the book, you will observe that the book occasionally includes patches that are required for a successful and secure installation of the packages. The general policy of the book is to include patches that fall in one of the following criteria:

Fixes a compilation problem.

Fixes a security problem.

Fixes a broken functionality.

In short, the book only includes patches that are either required or recommended. There is a Patches subproject which hosts various patches (including the patches referenced in the books) to enable you to configure your LFS the way you like it.

2.4 BLFS Systemd Units

The BLFS Systemd Units package contains the systemd unit files that are used throughout the book.

Package Information

Download: https://www.linuxfromscratch.org/blfs/downloads/11.3-systemd/blfs-systemd-units-20220720.tar.xz

The BLFS Systemd Units package will be used throughout the BLFS book for systemd unit files. Each systemd unit has a separate install target. It is recommended that you keep the package source directory around until completion of your BLFS system. When a systemd unit is requested from BLFS Systemd Units, simply change to the directory, and as the root user, execute the given make install- command. This command installs the systemd unit to its proper location (along with any auxiliary configuration scripts) and also enables it by default.

Note

It is advisable to peruse each systemd unit before installation to determine whether the installed files meet your needs.

2.5 About Libtool Archive (.la) files

Files with a .la extension

In LFS and BLFS, many packages use a internally shipped libtool copy to build on a variety of Unix platforms. This includes platforms such as AIX, Solaris, IRIX, HP-UX, and Cygwin as well as Linux. The origins of this tool are quite dated. It was intended to manage libraries on systems with less advanced capabilities than a modern Linux system.

On a Linux system, libtool specific files are generally unneeded. Normally libraries are specified in the build process during the link phase. Since a linux system uses the Executable and Linkable Format (ELF) for executables and dynamic libraries, information needed to complete the task is embedded in the files. Both the linker and the program loader can query the appropriate files and properly link or execute the program.

Static libraries are rarely used in LFS and BLFS. And, nowadays most packages store the information needed for linking against a static library into a .pc file, instead of relying on libtool. A pkg-config –static –libs command will output the sufficient flags for the linker to link against a static library without any libtool magic.

The problem is that libtool usually creates one or more text files for package libraries called libtool archives. These small files have a “.la” extension and contain information that is similar to that embedded in the libraries or pkg-config files. When building a package that uses libtool, the process automatically looks for these files. Sometimes a .la file can contains the name or path of a static library used during build but not installed, then the build process will break because the .la file refers to something nonexistent on the system. Similarly, if a package is updated and no longer uses the .la file, then the build process can break with the old .la files.

The solution is to remove the .la files. However there is a catch. Some packages, such as ImageMagick-7.1.0-61, use a libtool function, lt_dlopen, to load libraries as needed during execution and resolve their dependencies at run time. In this case, the .la files should remain.

The script below, removes all unneeded .la files and saves them in a directory, /var/local/la-files by default, not in the normal library path. It also searches all pkg-config files (.pc) for embedded references to .la files and fixes them to be conventional library references needed when an application or library is built. It can be run as needed to clean up the directories that may be causing problems.

cat > /usr/sbin/remove-la-files.sh << "EOF"
#!/bin/bash

# /usr/sbin/remove-la-files.sh
# Written for Beyond Linux From Scratch
# by Bruce Dubbs <bdubbs@linuxfromscratch.org>

# Make sure we are running with root privs
if test "${EUID}" -ne 0; then
    echo "Error: $(basename ${0}) must be run as the root user! Exiting..."
    exit 1
fi

# Make sure PKG_CONFIG_PATH is set if discarded by sudo
source /etc/profile

OLD_LA_DIR=/var/local/la-files

mkdir -p $OLD_LA_DIR

# Only search directories in /opt, but not symlinks to directories
OPTDIRS=$(find /opt -mindepth 1 -maxdepth 1 -type d)

# Move any found .la files to a directory out of the way
find /usr/lib $OPTDIRS -name "*.la" ! -path "/usr/lib/ImageMagick*" \
  -exec mv -fv {} $OLD_LA_DIR \;
###############

# Fix any .pc files that may have .la references

STD_PC_PATH='/usr/lib/pkgconfig
             /usr/share/pkgconfig
             /usr/local/lib/pkgconfig
             /usr/local/share/pkgconfig'

# For each directory that can have .pc files
for d in $(echo $PKG_CONFIG_PATH | tr : ' ') $STD_PC_PATH; do

  # For each pc file
  for pc in $d/*.pc ; do
    if [ $pc == "$d/*.pc" ]; then continue; fi

    # Check each word in a line with a .la reference
    for word in $(grep '\.la' $pc); do
      if $(echo $word | grep -q '.la$' ); then
        mkdir -p $d/la-backup
        cp -fv  $pc $d/la-backup

        basename=$(basename $word )
        libref=$(echo $basename|sed -e 's/^lib/-l/' -e 's/\.la$//')

        # Fix the .pc file
        sed -i "s:$word:$libref:" $pc
      fi
    done
  done
done

EOF

chmod +x /usr/sbin/remove-la-files.sh

User Notes: https://wiki.linuxfromscratch.org/blfs/wiki/la-files

2.6 Libraries: Static or shared?

Libraries: Static or shared?

The original libraries were simply an archive of routines from which the required routines were extracted and linked into the executable program. These are described as static libraries, with names of the form libfoo.a on UNIX-like operating systems. On some old operating systems they are the only type available.

On almost all Linux platforms there are also “shared” (or equivalently “dynamic”) libraries (with names of the form libfoo.so) – one copy of the library is loaded into virtual memory, and shared by all the programs which call any of its functions. This is space efficient.

In the past, essential programs such as a shell were often linked statically so that some form of minimal recovery system would exist even if shared libraries, such as libc.so, became damaged (e.g. moved to lost+found after fsck following an unclean shutdown). Nowadays, most people use an alternative system install or a USB stick if they have to recover. Journaling filesystems also reduce the likelihood of this sort of problem.

Within the book, there are various places where configure switches such as --disable-static are employed, and other places where the possibility of using system versions of libraries instead of the versions included within another package is discussed. The main reason for this is to simplify updates of libraries.

If a package is linked to a dynamic library, updating to a newer library version is automatic once the newer library is installed and the program is (re)started (provided the library major version is unchanged, e.g. going from libfoo.so.2.0 to libfoo.so.2.1. Going to libfoo.so.3 will require recompilation – ldd can be used to find which programs use the old version). If a program is linked to a static library, the program always has to be recompiled. If you know which programs are linked to a particular static library, this is merely an annoyance. But usually you will not know which programs to recompile.

One way to identify when a static library is used, is to deal with it at the end of the installation of every package. Write a script to find all the static libraries in /usr/lib or wherever you are installing to, and either move them to another directory so that they are no longer found by the linker, or rename them so that libfoo.a becomes e.g. libfoo.a.hidden. The static library can then be temporarily restored if it is ever needed, and the package needing it can be identified. This shouldn’t be done blindly since many libraries only exist in a static version. For example, some libraries from the glibc and gcc packages should always be present on the system (libc_nonshared.a, libg.a, libpthread_nonshared.a, libssp_nonshared.a, libsupc++.a as of glibc-2.36 and gcc-12.2).

If you use this approach, you may discover that more packages than you were expecting use a static library. That was the case with nettle-2.4 in its default static-only configuration: It was required by GnuTLS-3.0.19, but also linked into package(s) which used GnuTLS, such as glib-networking-2.32.3.

Many packages put some of their common functions into a static library which is only used by the programs within the package and, crucially, the library is not installed as a standalone library. These internal libraries are not a problem – if the package has to be rebuilt to fix a bug or vulnerability, nothing else is linked to them.

When BLFS mentions system libraries, it means shared versions of libraries. Some packages such as Firefox-102.8.0 and ghostscript-10.00.0 bundle many other libraries in their build tree. The version they ship is often older than the version used in the system, so it may contain bugs – sometimes developers go to the trouble of fixing bugs in their included libraries, other times they do not.

Sometimes, deciding to use system libraries is an easy decision. Other times it may require you to alter the system version (e.g. for libpng-1.6.39 if used for Firefox-102.8.0). Occasionally, a package ships an old library and can no longer link to the current version, but can link to an older version. In this case, BLFS will usually just use the shipped version. Sometimes the included library is no longer developed separately, or its upstream is now the same as the package’s upstream and you have no other packages which will use it. In those cases, you’ll be lead to use the included library even if you usually prefer to use system libraries.

User Notes: https://wiki.linuxfromscratch.org/blfs/wiki/libraries

This page contains information about locale related problems and issues. In the following paragraphs you’ll find a generic overview of things that can come up when configuring your system for various locales. Many (but not all) existing locale related problems can be classified and fall under one of the headings below. The severity ratings below use the following criteria:

If there is a known workaround for a specific package, it will appear on that package’s page. For the most recent information about locale related issues for individual packages, check the User Notes in the BLFS Wiki.

The Needed Encoding is Not a Valid Option in the Program

Severity: Critical

Some programs require the user to specify the character encoding for their input or output data and present only a limited choice of encodings. This is the case for the -X option in Enscript-1.6.6, the -input-charset option in unpatched Cdrtools-3.02a09, and the character sets offered for display in the menu of Links-2.28. If the required encoding is not in the list, the program usually becomes completely unusable. For non-interactive programs, it may be possible to work around this by converting the document to a supported input character set before submitting to the program.

A solution to this type of problem is to implement the necessary support for the missing encoding as a patch to the original program or to find a replacement.

The Program Assumes the Locale-Based Encoding of External Documents

Severity: High for non-text documents, low for text documents

Some programs, nano-7.2 or JOE-4.6 for example, assume that documents are always in the encoding implied by the current locale. While this assumption may be valid for the user-created documents, it is not safe for external ones. When this assumption fails, non-ASCII characters are displayed incorrectly, and the document may become unreadable.

If the external document is entirely text based, it can be converted to the current locale encoding using the iconv program.

For documents that are not text-based, this is not possible. In fact, the assumption made in the program may be completely invalid for documents where the Microsoft Windows operating system has set de facto standards. An example of this problem is ID3v1 tags in MP3 files (see the BLFS Wiki ID3v1Coding page for more details). For these cases, the only solution is to find a replacement program that doesn’t have the issue (e.g., one that will allow you to specify the assumed document encoding).

Among BLFS packages, this problem applies to nano-7.2, JOE-4.6, and all media players except Audacious-4.2.

Another problem in this category is when someone cannot read the documents you’ve sent them because their operating system is set up to handle character encodings differently. This can happen often when the other person is using Microsoft Windows, which only provides one character encoding for a given country. For example, this causes problems with UTF-8 encoded TeX documents created in Linux. On Windows, most applications will assume that these documents have been created using the default Windows 8-bit encoding.

In extreme cases, Windows encoding compatibility issues may be solved only by running Windows programs under Wine.

The Program Uses or Creates Filenames in the Wrong Encoding

Severity: Critical

The POSIX standard mandates that the filename encoding is the encoding implied by the current LC_CTYPE locale category. This information is well-hidden on the page which specifies the behavior of Tar and Cpio programs. Some programs get it wrong by default (or simply don’t have enough information to get it right). The result is that they create filenames which are not subsequently shown correctly by ls, or they refuse to accept filenames that ls shows properly. For the GLib-2.74.5 library, the problem can be corrected by setting the G_FILENAME_ENCODING environment variable to the special “@locale” value. Glib2 based programs that don’t respect that environment variable are buggy.

The Zip-3.0 and UnZip-6.0 have this problem because they hard-code the expected filename encoding. UnZip contains a hard-coded conversion table between the CP850 (DOS) and ISO-8859-1 (UNIX) encodings and uses this table when extracting archives created under DOS or Microsoft Windows. However, this assumption only works for those in the US and not for anyone using a UTF-8 locale. Non-ASCII characters will be mangled in the extracted filenames.

The general rule for avoiding this class of problems is to avoid installing broken programs. If this is impossible, the convmv command-line tool can be used to fix filenames created by these broken programs, or intentionally mangle the existing filenames to meet the broken expectations of such programs.

In other cases, a similar problem is caused by importing filenames from a system using a different locale with a tool that is not locale-aware (e.g., OpenSSH-9.2p1). In order to avoid mangling non-ASCII characters when transferring files to a system with a different locale, any of the following methods can be used:

The last four methods work because the filenames are automatically converted from the sender’s locale to UNICODE and stored or sent in this form. They are then transparently converted from UNICODE to the recipient’s locale encoding.

The Program Breaks Multibyte Characters or Doesn’t Count Character Cells Correctly

Severity: High or critical

Many programs were written in an older era where multibyte locales were not common. Such programs assume that C “char” data type, which is one byte, can be used to store single characters. Further, they assume that any sequence of characters is a valid string and that every character occupies a single character cell. Such assumptions completely break in UTF-8 locales. The visible manifestation is that the program truncates strings prematurely (i.e., at 80 bytes instead of 80 characters). Terminal-based programs don’t place the cursor correctly on the screen, don’t react to the “Backspace” key by erasing one character, and leave junk characters around when updating the screen, usually turning the screen into a complete mess.

Fixing this kind of problems is a tedious task from a programmer’s point of view, like all other cases of retrofitting new concepts into the old flawed design. In this case, one has to redesign all data structures in order to accommodate to the fact that a complete character may span a variable number of “char”s (or switch to wchar_t and convert as needed). Also, for every call to the “strlen” and similar functions, find out whether a number of bytes, a number of characters, or the width of the string was really meant. Sometimes it is faster to write a program with the same functionality from scratch.

Among BLFS packages, this problem applies to xine-ui-0.99.14 and all the shells.

The Package Installs Manual Pages in Incorrect or Non-Displayable Encoding

Severity: Low

LFS expects that manual pages are in the language-specific (usually 8-bit) encoding, as specified on the LFS Man DB page. However, some packages install translated manual pages in UTF-8 encoding (e.g., Shadow, already dealt with), or manual pages in languages not in the table. Not all BLFS packages have been audited for conformance with the requirements put in LFS (the large majority have been checked, and fixes placed in the book for packages known to install non-conforming manual pages). If you find a manual page installed by any of BLFS packages that is obviously in the wrong encoding, please remove or convert it as needed, and report this to BLFS team as a bug.

You can easily check your system for any non-conforming manual pages by copying the following short shell script to some accessible location,

#!/bin/sh
# Begin checkman.sh
# Usage: find /usr/share/man -type f | xargs checkman.sh
for a in "$@"
do
    # echo "Checking $a..."
    # Pure-ASCII manual page (possibly except comments) is OK
    grep -v '.\\"' "$a" | iconv -f US-ASCII -t US-ASCII >/dev/null 2>&1 \
        && continue
    # Non-UTF-8 manual page is OK
    iconv -f UTF-8 -t UTF-8 "$a" >/dev/null 2>&1 || continue
    # Found a UTF-8 manual page, bad.
    echo "UTF-8 manual page: $a" >&2
done
# End checkman.sh

and then issuing the following command (modify the command below if the checkman.sh script is not in your PATH environment variable):

find /usr/share/man -type f | xargs checkman.sh

Note that if you have manual pages installed in any location other than /usr/share/man (e.g., /usr/local/share/man), you must modify the above command to include this additional location.

2.8 Going Beyond BLFS

The packages that are installed in this book are only the tip of the iceberg. We hope that the experience you gained with the LFS book and the BLFS book will give you the background needed to compile, install and configure packages that are not included in this book.

When you want to install a package to a location other than /, or /usr, you are installing outside the default environment settings on most machines. The following examples should assist you in determining how to correct this situation. The examples cover the complete range of settings that may need updating, but they are not all needed in every situation.

If you are in search of a package that is not in the book, the following are different ways you can search for the desired package.

Some general hints on handling new packages:

Tip

If you have found a package that is only available in .deb or .rpm format, there are two small scripts, rpm2targz and deb2targz that are available at https://anduin.linuxfromscratch.org/BLFS/extras/deb2targz.tar.bz2 and https://anduin.linuxfromscratch.org/BLFS/extras/rpm2targz.tar.bz2 to convert the archives into a simple tar.gz format.

You may also find an rpm2cpio script useful. The Perl version in the linux kernel archives at https://lore.kernel.org/all/20021016121842.GA2292@ncsu.edu/2-rpm2cpio works for most source rpms. The rpm2targz script will use an rpm2cpio script or binary if one is on your path. Note that rpm2cpio will unpack a source rpm in the current directory, giving a tarball, a spec file, and perhaps patches or other files.