[libc][docs] Update docs to reflect new headergen (#102381)

Since new headergen is now the default for building LLVM-libc, the docs
need to be updated to reflect that. While I was editing those docs, I
took a quick pass at updating other out-of-date pages.
This commit is contained in:
Michael Jones 2024-08-21 10:50:39 -07:00 committed by GitHub
parent c975dc1da0
commit b89fef8f67
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
15 changed files with 156 additions and 244 deletions

View File

@ -38,13 +38,6 @@ The libc can be built and tested in two different modes:
$> ninja libc-integration-tests
#. API verification test - See :ref:`api_test` for more information about
the API test. It can be run by the command:
.. code-block:: sh
$> ninja libc-api-test
Building with VSCode
====================

View File

@ -4,7 +4,7 @@
Contributing to the libc Project
================================
LLVM's libc is being developed as part of the LLVM project so contributions
LLVM-libc is being developed as part of the LLVM project so contributions
to the libc project should also follow the general LLVM
`contribution guidelines <https://llvm.org/docs/Contributing.html>`_. Below is
a list of open projects that one can start with:
@ -31,24 +31,12 @@ a list of open projects that one can start with:
directory. So, a simple but mechanical project would be to move the parts
following the old styles to the new style.
#. **Integrating with the rest of the LLVM project** - There are two parts to
this project:
#. One is about adding CMake facilities to optionally link the libc's overlay
static archive (see :ref:`overlay_mode`) with other LLVM tools/executables.
#. The other is about putting plumbing in place to release the overlay static
archive (see :ref:`overlay_mode`) as part of the LLVM binary releases.
#. **Implement Linux syscall wrappers** - A large portion of the POSIX API can
be implemented as syscall wrappers on Linux. A good number have already been
implemented but many more are yet to be implemented. So, a project of medium
complexity would be to implement syscall wrappers which have not yet been
implemented.
#. **Add a better random number generator** - The current random number
generator has a very small range. This has to be improved or switched over
to a fast random number generator with a large range.
#. **Update the clang-tidy lint rules and use them in the build and/or CI** -
Currently, the :ref:`clang_tidy_checks` have gone stale and are mostly unused
by the developers and on the CI builders. This project is about updating

View File

@ -1,25 +0,0 @@
.. _api_test:
========
API Test
========
.. warning::
This page is severely out of date. Much of the information it contains may be
incorrect. Please only remove this warning once the page has been updated.
The implementation of libc-project is unique because our public C header files
are generated using information from ground truth captured in TableGen files.
Unit tests only exercise the internal C++ implementations and don't ensure the
headers were generated by the build system and that the generated header files
contain the expected declarations and definitions. A simple solution is to have
contributors write an integration test for each individual function as a C
program; however, this would place a large burden on contributors and duplicates
some effort from the unit tests.
Instead we automate the generation of what we call as an API test. This API test
ensures that public facing symbols are visible, that the header files are
generated as expected, and that each libc function has the correct function
prototype as specified by the standards. The API test cmake rules are located in
``test/src/CMakeLists.txt``. The source file for the API test is generated in
``<build directory>/projects/libc/test/src/public_api_test.cpp``

View File

@ -1,11 +0,0 @@
The ground truth of standards
=============================
Like any modern libc, LLVM libc also supports a wide number of standards and
extensions. To avoid developing headers, wrappers and sources in a disjointed
fashion, LLVM libc employs ground truth files. These files live under the
``spec`` directory and list ground truth corresponding the ISO C standard, the
POSIX extension standard, etc. For example, the path to the ground truth file
for the ISO C standard is ``spec/stdc.td``. Tools like the header generator
(described in the header generation document), docs generator, etc. use the
ground truth files to generate headers, docs etc.

View File

@ -1,3 +1,5 @@
.. _header_generation:
Generating Public and Internal headers
======================================

View File

@ -15,10 +15,7 @@ Navigate to the links below for information on the respective topics:
config_options
clang_tidy_checks
fuzzing
ground_truth_specification
header_generation
implementation_standard
undefined_behavior
printf_behavior
api_test
mechanics_of_public_api

View File

@ -1,29 +0,0 @@
The mechanics of the ``public_api`` command
===========================================
The build system, in combination with the header generation mechanism,
facilitates the fine grained ability to pick and choose the public API one wants
to expose on their platform. The public header files are always generated from
the corresponding ``.h.def`` files. A header generation command ``%%public_api``
is listed in these files. In the generated header file, the header generator
replaces this command with the public API relevant for the target platform.
Under the hood
--------------
When the header generator sees the ``%%public_api`` command, it looks up the
API config file for the platform in the path ``config/<platform>/api.td``.
The API config file lists two kinds of items:
1. The list of standards from which the public entities available on the platform
are derived from.
2. For each header file exposed on the platform, the list of public members
provided in that header file.
Note that, the header generator only learns the names of the public entities
from the header config file (the 2nd item from above.) The exact manner in which
the entities are to be declared is got from the standards (the 1st item from
above.)
See the ground truth document for more information on how the standards are
formally listed in LLVM libc using LLVM table-gen files.

View File

@ -14,9 +14,10 @@ directories::
- docs
- examples
- fuzzing
- hdr
- include
- lib
- spec
- newhdrgen
- src
- startup
- test
@ -62,6 +63,14 @@ The directory structure within this directory mirrors the directory structure
of the top-level ``libc`` directory itself. For more details, see
:doc:`fuzzing`.
The ``hdr`` directory
---------------------
This directory contains proxy headers which are included from the files in the
src directory. These proxy headers either include our internal type or macro
definitions, or the system's type or macro definitions, depending on if we are
in fullbuild or overlay mode.
The ``include`` directory
-------------------------
@ -80,13 +89,14 @@ The ``lib`` directory
This directory contains a ``CMakeLists.txt`` file listing the targets for the
public libraries ``libc.a``, ``libm.a`` etc.
The ``spec`` directory
----------------------
The ``newhdrgen`` directory
---------------------------
This directory contains the specifications for the types, macros, and entrypoint
functions. These definitions come from the various standards and extensions
LLVM-libc supports, and they are used along with the ``*.h.def`` files and the
config files to generate the headers for fullbuild mode.
This directory contains the sources and specifications for the types, macros
and entrypoint functions. These definitions are organized in the ``yaml``
subdirectory and match the organization of the ``*.h.def`` files. This folder
also contains the python sources for new headergen, which is what generates the
headers.
The ``src`` directory
---------------------

View File

@ -8,35 +8,33 @@ Full Cross Build
:depth: 1
:local:
.. note::
Fullbuild requires running headergen, which is a python program that depends on
pyyaml. The minimum versions are listed on the :ref:`header_generation`
page, as well as additional information.
In this document, we will present recipes to cross build the full libc. When we
say *cross build* a full libc, we mean that we will build the full libc for a
target system which is not the same as the system on which the libc is being
built. For example, you could be building for a bare metal aarch64 *target* on a
Linux x86_64 *host*.
There are three main recipes to cross build the full libc. Each one serves a
There are two main recipes to cross build the full libc. Each one serves a
different use case. Below is a short description of these recipes to help users
pick the recipe that best suites their needs and contexts.
* **Standalone cross build** - Using this recipe one can build the libc using a
compiler of their choice. One should use this recipe if their compiler can
build for the host as well as the target.
* **Runtimes cross build** - In this recipe, one will have to first build the
libc build tools for the host separately and then use those build tools to
build the libc. Users can use the compiler of their choice to build the
libc build tools as well as the libc. One should use this recipe if they
have to use a host compiler to build the build tools for the host and then
use a target compiler (which is different from the host compiler) to build
the libc.
* **Bootstrap cross build** - In this recipe, one will build the ``clang``
compiler and the libc build tools for the host first, and then use them to
build the libc for the target. Unlike with the runtimes build recipe, the
user does not have explicitly build ``clang`` and other libc build tools.
build the libc for the target. Unlike with the standalone build recipe, the
user does not have explicitly build ``clang`` and other build tools.
They get built automatically before building the libc. One should use this
recipe if they intend use the built ``clang`` and the libc as part of their
toolchain for the target.
The following sections present the three recipes in detail.
The following sections present the two recipes in detail.
Standalone cross build
======================
@ -61,9 +59,9 @@ Below is the CMake command to configure the standalone crossbuild of the libc.
$> cd build
$> C_COMPILER=<C compiler> # For example "clang"
$> CXX_COMPILER=<C++ compiler> # For example "clang++"
$> cmake ../llvm \
$> cmake ../runtimes \
-G Ninja \
-DLLVM_ENABLE_PROJECTS=libc \
-DLLVM_ENABLE_RUNTIMES=libc \
-DCMAKE_C_COMPILER=$C_COMPILER \
-DCMAKE_CXX_COMPILER=$CXX_COMPILER \
-DLLVM_LIBC_FULL_BUILD=ON \
@ -72,8 +70,8 @@ Below is the CMake command to configure the standalone crossbuild of the libc.
We will go over the special options passed to the ``cmake`` command above.
* **Enabled Projects** - Since we want to build the libc project, we list
``libc`` as the enabled project.
* **Enabled Runtimes** - Since we want to build LLVM-libc, we list
``libc`` as the enabled runtime.
* **The full build option** - Since we want to build the full libc, we pass
``-DLLVM_LIBC_FULL_BUILD=ON``.
* **The target triple** - This is the target triple of the target for which
@ -94,88 +92,6 @@ The above ``ninja`` command will build the libc static archives ``libc.a`` and
``libm.a`` for the target specified with ``-DLIBC_TARGET_TRIPLE`` in the CMake
configure step.
.. _runtimes_cross_build:
Runtimes cross build
====================
The *runtimes cross build* is very similar to the standalone crossbuild but the
user will have to first build the libc build tools for the host separately. One
should use this recipe if they want to use a different host and target compiler.
Note that the libc build tools MUST be in sync with the libc. That is, the
libc build tools and the libc, both should be built from the same source
revision. At the time of this writing, there is only one libc build tool that
has to be built separately. It is done as follows:
.. code-block:: sh
$> cd llvm-project # The llvm-project checkout
$> mkdir build-libc-tools # A different build directory for the build tools
$> cd build-libc-tools
$> HOST_C_COMPILER=<C compiler for the host> # For example "clang"
$> HOST_CXX_COMPILER=<C++ compiler for the host> # For example "clang++"
$> cmake ../llvm \
-G Ninja \
-DLLVM_ENABLE_PROJECTS=libc \
-DCMAKE_C_COMPILER=$HOST_C_COMPILER \
-DCMAKE_CXX_COMPILER=$HOST_CXX_COMPILER \
-DLLVM_LIBC_FULL_BUILD=ON \
-DCMAKE_BUILD_TYPE=Debug # User can choose to use "Release" build type
$> ninja libc-hdrgen
The above commands should build a binary named ``libc-hdrgen``. Copy this binary
to a directory of your choice.
CMake configure step
--------------------
After copying the ``libc-hdrgen`` binary to say ``/path/to/libc-hdrgen``,
configure the libc build using the following command:
.. code-block:: sh
$> cd llvm-project # The llvm-project checkout
$> mkdir build
$> cd build
$> TARGET_C_COMPILER=<C compiler for the target>
$> TARGET_CXX_COMPILER=<C++ compiler for the target>
$> HDRGEN=</path/to/libc-hdrgen>
$> TARGET_TRIPLE=<Your target triple>
$> cmake ../runtimes \
-G Ninja \
-DLLVM_ENABLE_RUNTIMES=libc \
-DCMAKE_C_COMPILER=$TARGET_C_COMPILER \
-DCMAKE_CXX_COMPILER=$TARGET_CXX_COMPILER \
-DLLVM_LIBC_FULL_BUILD=ON \
-DLIBC_HDRGEN_EXE=$HDRGEN \
-DLIBC_TARGET_TRIPLE=$TARGET_TRIPLE \
-DCMAKE_BUILD_TYPE=Debug # User can choose to use "Release" build type
Note the differences in the above cmake command versus the one used in the
CMake configure step of the standalone build recipe:
* Instead of listing ``libc`` in ``LLVM_ENABLED_PROJECTS``, we list it in
``LLVM_ENABLED_RUNTIMES``.
* Instead of using ``llvm-project/llvm`` as the root CMake source directory,
we use ``llvm-project/runtimes`` as the root CMake source directory.
* The path to the ``libc-hdrgen`` binary built earlier is specified with
``-DLIBC_HDRGEN_EXE=/path/to/libc-hdrgen``.
Build step
----------
The build step in the runtimes build recipe is exactly the same as that of
the standalone build recipe:
.. code-block:: sh
$> ninja libc libm
As with the standalone build recipe, the above ninja command will build the
libc static archives for the target specified with ``-DLIBC_TARGET_TRIPLE`` in
the CMake configure step.
Bootstrap cross build
=====================
@ -203,8 +119,7 @@ CMake configure step
-DLLVM_RUNTIME_TARGETS=$TARGET_TRIPLE \
-DCMAKE_BUILD_TYPE=Debug
Note how the above cmake command differs from the one used in the other two
recipes:
Note how the above cmake command differs from the one used in the other recipe:
* ``clang`` is listed in ``-DLLVM_ENABLE_PROJECTS`` and ``libc`` is
listed in ``-DLLVM_ENABLE_RUNTIMES``.
@ -214,7 +129,7 @@ recipes:
Build step
----------
The build step is similar to the other two recipes:
The build step is similar to the other recipe:
.. code-block:: sh

View File

@ -8,17 +8,90 @@ Full Host Build
:depth: 1
:local:
.. note::
Fullbuild requires running headergen, which is a python program that depends on
pyyaml. The minimum versions are listed on the :ref:`header_generation`
page, as well as additional information.
In this document, we will present a recipe to build the full libc for the host.
When we say *build the libc for the host*, the goal is to build the libc for
the same system on which the libc is being built. Also, we will take this
opportunity to demonstrate how one can set up a *sysroot* (see the documentation
the same system on which the libc is being built. First, we will explain how to
build for developing LLVM-libc, then we will explain how to build LLVM-libc as
part of a complete toolchain.
Configure the build for development
===================================
Below is the list of commands for a simple recipe to build LLVM-libc for
development. In this we've set the Ninja generator, set the build type to
"Debug", and enabled the Scudo allocator. This build also enables generating the
documentation and verbose cmake logging, which are useful development features.
.. note::
if your build fails with an error saying the compiler can't find
``<asm/unistd.h>`` or similar then you're probably missing the symlink from
``/usr/include/asm`` to ``/usr/include/<HOST TRIPLE>/asm``. Installing the
``gcc-multilib`` package creates this symlink, or you can do it manually with
this command:
``sudo ln -s /usr/include/<HOST TRIPLE>/asm /usr/include/asm``
(your host triple will probably be similar to ``x86_64-linux-gnu``)
.. code-block:: sh
$> cd llvm-project # The llvm-project checkout
$> mkdir build
$> cd build
$> cmake ../runtimes \
-G Ninja \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DLLVM_ENABLE_RUNTIMES="libc;compiler-rt" \
-DLLVM_LIBC_FULL_BUILD=ON \
-DCMAKE_BUILD_TYPE=Debug \
-DLLVM_LIBC_INCLUDE_SCUDO=ON \
-DCOMPILER_RT_BUILD_SCUDO_STANDALONE_WITH_LLVM_LIBC=ON \
-DCOMPILER_RT_BUILD_GWP_ASAN=OFF \
-DCOMPILER_RT_SCUDO_STANDALONE_BUILD_SHARED=OFF \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DLLVM_ENABLE_SPHINX=ON -DLIBC_INCLUDE_DOCS=ON \
-DLIBC_CMAKE_VERBOSE_LOGGING=ON
Build and test
==============
After configuring the build with the above ``cmake`` command, one can build test
libc with the following command:
.. code-block:: sh
$> ninja libc libm check-libc
To build the docs run this command:
.. code-block:: sh
$> ninja docs-libc-html
To run a specific test, use the following:
.. code-block:: sh
$> ninja libc.test.src.<HEADER>.<FUNCTION>_test.__unit__
$> ninja libc.test.src.ctype.isalpha_test.__unit__ # EXAMPLE
Configure the complete toolchain build
======================================
For a complete toolchain we recommend creating a *sysroot* (see the documentation
of the ``--sysroot`` option here:
`<https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html>`_) which includes
not only the components of LLVM's libc, but also a full LLVM only toolchain
consisting of the `clang <https://clang.llvm.org/>`_ compiler, the
`lld <https://lld.llvm.org/>`_ linker and the
`compiler-rt <https://compiler-rt.llvm.org/>`_ runtime libraries. LLVM's libc is
not yet complete enough to allow using and linking a C++ application against
`compiler-rt <https://compiler-rt.llvm.org/>`_ runtime libraries. LLVM-libc is
not quite complete enough to allow using and linking a C++ application against
a C++ standard library (like libc++). Hence, we do not include
`libc++ <https://libcxx.llvm.org/>`_ in the sysroot.
@ -26,9 +99,6 @@ a C++ standard library (like libc++). Hence, we do not include
`libc++ <https://libcxx.llvm.org/>`_, libcxx-abi and libunwind in the
LLVM only toolchain and use them to build and link C++ applications.
Configure the full libc build
===============================
Below is the list of commands for a simple recipe to build and install the
libc components along with other components of an LLVM only toolchain. In this
we've set the Ninja generator, enabled a full compiler suite, set the build
@ -43,6 +113,7 @@ to use the freshly built lld and compiler-rt.
this command:
``sudo ln -s /usr/include/<TARGET TRIPLE>/asm /usr/include/asm``
.. TODO: Move from projects to runtimes for libc, compiler-rt
.. code-block:: sh
$> cd llvm-project # The llvm-project checkout
@ -51,7 +122,7 @@ to use the freshly built lld and compiler-rt.
$> SYSROOT=/path/to/sysroot # Remember to set this!
$> cmake ../llvm \
-G Ninja \
-DLLVM_ENABLE_PROJECTS="clang;libc;lld;compiler-rt" \
-DLLVM_ENABLE_PROJECTS="clang;lld;libc;compiler-rt" \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \

View File

@ -4,6 +4,11 @@
Fullbuild Mode
==============
.. note::
Fullbuild requires running headergen, which is a python program that depends on
pyyaml. The minimum versions are listed on the :ref:`header_generation`
page, as well as additional information.
The *fullbuild* mode of LLVM's libc is the mode in which it is to be used as
the only libc (as opposed to the :ref:`overlay_mode` in which it is used along
with the system libc.) In order to use it as the only libc, one will have to

View File

@ -63,9 +63,13 @@ targeting the default host environment as well.
Runtimes cross build
--------------------
.. note::
These instructions need to be updated for new headergen. They may be
inaccurate.
For users wanting more direct control over the build process, the build steps
can be done manually instead. This build closely follows the instructions in the
:ref:`main documentation<runtimes_cross_build>` but is specialized for the GPU
:ref:`main documentation<full_cross_build>` but is specialized for the GPU
build. We follow the same steps to first build the libc tools and a suitable
compiler. These tools must all be up-to-date with the libc source.

View File

@ -2,14 +2,16 @@
The LLVM C Library
==================
.. warning::
The libc is not complete. If you need a fully functioning C library right
now, you should continue to use your standard system libraries.
.. note::
LLVM-libc is not fully complete right now. Some programs may fail to build due
to missing functions (especially C++ ones). If you would like to help us
finish LLVM-libc, check out "Contributing to the libc project" in the sidebar
or ask on discord.
Introduction
============
The libc aspires to a unique place in the software ecosystem. The goals are:
LLVM-libc aspires to a unique place in the software ecosystem. The goals are:
- Fully compliant with current C standards (C17 and upcoming C2x) and POSIX.
- Easily decomposed and embedded: Supplement or replace system C library
@ -32,8 +34,9 @@ The libc aspires to a unique place in the software ecosystem. The goals are:
Platform Support
================
Most development is currently targeting x86_64 and aarch64 on Linux. Several
functions in the libc have been tested on Windows. The Fuchsia platform is
Most development is currently targeting Linux on x86_64, aarch64, arm, and
RISC-V. Embedded/baremetal targets are supported on arm and RISC-V, and Windows
and MacOS have limited support (may be broken). The Fuchsia platform is
slowly replacing functions from its bundled libc with functions from this
project.
@ -41,7 +44,7 @@ ABI Compatibility
=================
The libc is written to be ABI independent. Interfaces are generated using
LLVM's tablegen, so supporting arbitrary ABIs is possible. In it's initial
headergen, so supporting arbitrary ABIs is possible. In it's initial
stages there is no ABI stability in any form.
.. toctree::

View File

@ -28,18 +28,18 @@ Also, if users choose to mix more than one libc with the system libc, then
the name ``libllvmlibc.a`` makes it absolutely clear that it is the static
archive of LLVM's libc.
Building the static archive with libc as a normal LLVM project
--------------------------------------------------------------
Building LLVM-libc as a standalone runtime
------------------------------------------
We can treat the ``libc`` project as any other normal LLVM project and perform
the CMake configure step as follows:
We can treat the ``libc`` project like any other normal LLVM runtime library by
building it with the following cmake command:
.. code-block:: sh
$> cd llvm-project # The llvm-project checkout
$> mkdir build
$> cd build
$> cmake ../llvm -G Ninja -DLLVM_ENABLE_RUNTIMES="libc" \
$> cmake ../runtimes -G Ninja -DLLVM_ENABLE_RUNTIMES="libc" \
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_BUILD_TYPE=<Debug|Release> \ # Select build type
-DCMAKE_INSTALL_PREFIX=<Your prefix of choice> # Optional
@ -50,24 +50,29 @@ Next, build the libc:
$> ninja libc
The build step will build the static archive the in the directory
``build/projects/libc/lib``. Notice that the above CMake configure step also
specified an install prefix. This is optional, but if one uses it, then they
can follow up the build step with an install step:
Then, run the tests:
.. code-block:: sh
$> ninja install-llvmlibc
$> ninja check-libc
The build step will build the static archive the in the directory
``build/projects/libc/lib``. Notice that the above CMake configure step also
specified an install prefix. This is optional, but it's used, then the following
command will install the static archive to the install path:
.. code-block:: sh
$> ninja install-libc
Building the static archive as part of the bootstrap build
----------------------------------------------------------
The bootstrap build is a build mode in which runtime components like libc++,
libcxx-abi, libc etc. are built using the ToT clang. The idea is that this build
produces an in-sync toolchain of compiler + runtime libraries. Such a synchrony
is not essential for the libc but can one still build the overlay static archive
as part of the bootstrap build if one wants to. The first step is to configure
appropriately:
produces an in-sync toolchain of compiler + runtime libraries. This ensures that
LLVM-libc has access to the latest clang features, which should provide the best
performance possible.
.. code-block:: sh
@ -77,14 +82,13 @@ appropriately:
-DCMAKE_BUILD_TYPE=<Debug|Release> \ # Select build type
-DCMAKE_INSTALL_PREFIX=<Your prefix of choice> # Optional
The build and install steps are similar to the those used when configured
as a normal project. Note that the build step takes much longer this time
as ``clang`` will be built before building ``libllvmlibc.a``.
The build and install steps are the same as above, but the build step will take
much longer since ``clang`` will be built before building ``libllvmlibc.a``.
.. code-block:: sh
$> ninja libc
$> ninja install-llvmlibc
$> ninja check-libc
Using the overlay static archive
================================

View File

@ -43,21 +43,6 @@ have their own config directory.
config directory for Fuchsia as the bring up is being done in the Fuchsia
source tree.
The api.td file
---------------
If the :ref:`fullbuild_mode` is to be supported on the new operating system,
then a file named ``api.td`` should be added in its config directory. It is
written in the
`LLVM tablegen language <https://llvm.org/docs/TableGen/ProgRef.html>`_.
It lists all the relevant macros and type definitions we want in the
public libc header files. See the existing Linux
`api.td <https://github.com/llvm/llvm-project/blob/main/libc/config/linux/api.td>`_
file as an example to prepare the ``api.td`` file for the new operating system.
.. note:: In future, LLVM tablegen will be replaced with a different DSL to list
config information.
Architecture Subdirectory
=========================