Adds a JSON generator backend to emit mapped information as JSON. This will enable a better testing format for upcoming changes. It can also potentially serve to feed our other backend generators in the future, like Mustache which already serializes information to JSON before emitting as HTML.
This patch contains functionality to emit classes and provides most of the basis of the generator.
This patch re-enables -ftime-trace support in clang-doc. Initial support
in #97644 was reverted, and never relanded. This patch adds back the
command line option, and leverages the RAII tracing infrastructure more
thoroughly.
This patch refactors the clang-doc tool to replace manual error handling
(e.g., checking error codes and calling exit()) with llvm::ExitOnError.
Fixes#140085
The error message:
/usr/bin/ld: CMakeFiles/clang-doc.dir/ClangDocMain.cpp.o: undefined refe
rence to symbol '_Z20getMustacheHtmlFilesN4llvm9StringRefERN5clang3doc15
ClangDocContextE' /usr/bin/ld: /work/kparzysz/git/llvm.org/b/x86/./lib/l
ibclangDocSupport.so.21.0git: error adding symbols: DSO missing from com
mand line
collect2: error: ld returned 1 exit status
make[2]: *** [tools/clang/tools/extra/clang-doc/tool/CMakeFiles/clang-do
c.dir/build.make:107: bin/clang-doc] Error 1
This patch adds a command line option and enables the Mustache template
HTML backend. This allows users to use the new, more flexible templates
over the old and cumbersome HTML output. Split from #133161.
Co-authored-by: Peter Chou <peter.chou@mail.utoronto.ca>
This patch adds or fills in some helper functions related to template
setup when initializing the mustache backend. It was split from #133161.
Co-authored-by: Peter Chou <peter.chou@mail.utoronto.ca>
This patch adds the various assets used with the HTML Mustache backend.
This includes templates for a variety of different language constructs,
as well as the CSS. Split from #133161.
Co-authored-by: Peter Chou <peter.chou@mail.utoronto.ca>
Additionally moving the declarations from Generator.cpp
prevents misc-use-internal-linkage warnings from clang-tidy.
We fix those here too, since the static functions are part
of the same diagnostic
This PR adds a new command-line option that allows users to specify the
prefix used for line-based anchors in repository URLs. Different
repository interfaces use different formats for line anchors (GitHub
uses `#L123`, googlesource uses `#123`, etc.). This option enables users
to customize the line prefix to match their repository platform without
requiring hard-coded values for each service.
Fixes#59814
Currently, when we set URLs from JS, we set them only using the protocol
and host locations. This works fine when docs are served from the base
directory of the site, but if you want to nest it under another
directory, our JS fails to set the correct path, leading to broken
links.
This patch adds a --base option to specify the path prefix to use, which
is set in the generated index_json.js file. index.json can then fill in
the prefix appropriately when generating links in a browser. This flag
has no effect for non HTML output.
Given an index hosted at: www.docs.com/base_directory/index.html
we used to generate the following link:
www.docs.com/file.html
Using --base base_directory we now generate:
www.docs.com/base_directory/file.html
This allows such links to work when hosting pages without using a custom
index.js.
Fixes https://github.com/llvm/llvm-project/issues/97507#96809 introduce non-determinstic behaviour in clang-doc which caused
the output to be randomly ordered which lead to randomly failing test.
This patch modify clang-doc to behave deterministically again by sorting
the output by location or USR.
this patches adds ftime-tracing option for clang-doc which was what I
used to profile the performance problems we had with clang-doc
generating the llvm docs.
- ftime-trace produces a time profiler json file called
clang-doc-tracing.json
Reapply https://github.com/llvm/llvm-project/pull/94717
Adds a new option --asset which allows users to specified the asset
folder for the html output of clang-doc.
This patch adds a better test for --asset option + fixes bug where
clang-doc assumes that user supplied js file is assume to be index.js
Adds e2e tests for clang-doc, which serves as useful test against
regression in the html output. It also serves as a helpful tool for code
review as we can see the diffs to clang-docs output
Closes https://github.com/llvm/llvm-project/issues/93925
While we copy the asset files, like index.js, into the correct location
in the install step, tests do not have access to those resources in the
build directory.
This patch copies the contents of the clang-doc/assets directory into
the build folder, so that they can be used in testing.
Pull Request: https://github.com/llvm/llvm-project/pull/95185
The base class llvm::ThreadPoolInterface will be renamed
llvm::ThreadPool in a subsequent commit.
This is a breaking change: clients who use to create a ThreadPool must
now create a DefaultThreadPool instead.
Fixes a warning about a potentially unsupported template argument
deduction by explicitly specifying the template type in std::lock_guard.
Patch By: brettw
Differential Revision: https://reviews.llvm.org/D138961
Previously file naming and directory layout was handled on a per Info
object basis by ClangDocMain and the generators blindly wrote to the
files given. This means all generators must use the same file layout and
caused problems where multiple objects mapped to the same file. The
object collision problem happens most easily with template
specializations because the template parameters are not part of the
"name".
This patch moves the responsibility for output file organization to the
generators. Currently HTML and MD use the same structure as before. But
they now collect all objects that map to a given file and combine them,
avoiding the corruption problems.
Converts the YAML generator to naming files based on USR in one
directory. This is easier for downstream tools to manage and avoids the
naming problems with template specializations. Since this change
requires backward-incompatible output changes to referenced files anyway
(since each one is now an array), this is a good time to introduce this
change.
Differential Revision: https://reviews.llvm.org/D138073
This reverts commit f8a469fc572778d05b72f34a772082cf3abd3cda.
The test in single-file-public.cpp breaks on Mac, due to an unknown
regextype in the find command.
Previously file naming and directory layout was handled on a per Info
object basis by ClangDocMain and the generators blindly wrote to the
files given. This means all generators must use the same file layout and
caused problems where multiple objects mapped to the same file. The
object collision problem happens most easily with template
specializations because the template parameters are not part of the
"name".
This patch moves the responsibility for output file organization to the
generators. Currently HTML and MD use the same structure as before. But
they now collect all objects that map to a given file and combine them,
avoiding the corruption problems.
Converts the YAML generator to naming files based on USR in one
directory. This is easier for downstream tools to manage and avoids
the naming problems with template specializations. Since this change
requires backward-incompatible output changes to referenced files anyway
(since each one is now an array), this is a good time to introduce this
change.
Differential Revision: https://reviews.llvm.org/D138073
When using `clang-doc --format=html` it will crash on startup because of an assertion doing a self-assignment of a `SmallString`. This patch removes the self-assignment by creating an intermediate copy.
Reviewed By: paulkirth
Differential Revision: https://reviews.llvm.org/D131793
This is the original patch in my GNUInstallDirs series, now last to merge as the final piece!
It arose as a new draft of D28234. I initially did the unorthodox thing of pushing to that when I wasn't the original author, but since I ended up
- Using `GNUInstallDirs`, rather than mimicking it, as the original author was hesitant to do but others requested.
- Converting all the packages, not just LLVM, effecting many more projects than LLVM itself.
I figured it was time to make a new revision.
I have used this patch series (and many back-ports) as the basis of https://github.com/NixOS/nixpkgs/pull/111487 for my distro (NixOS), which was merged last spring (2021). It looked like people were generally on board in D28234, but I make note of this here in case extra motivation is useful.
---
As pointed out in the original issue, a central tension is that LLVM already has some partial support for these sorts of things. Variables like `COMPILER_RT_INSTALL_PATH` have already been dealt with. Variables like `LLVM_LIBDIR_SUFFIX` however, will require further work, so that we may use `CMAKE_INSTALL_LIBDIR`.
These remaining items will be addressed in further patches. What is here is now rote and so we should get it out of the way before dealing more intricately with the remainder.
Reviewed By: #libunwind, #libc, #libc_abi, compnerd
Differential Revision: https://reviews.llvm.org/D99484
This is the original patch in my GNUInstallDirs series, now last to merge as the final piece!
It arose as a new draft of D28234. I initially did the unorthodox thing of pushing to that when I wasn't the original author, but since I ended up
- Using `GNUInstallDirs`, rather than mimicking it, as the original author was hesitant to do but others requested.
- Converting all the packages, not just LLVM, effecting many more projects than LLVM itself.
I figured it was time to make a new revision.
I have used this patch series (and many back-ports) as the basis of https://github.com/NixOS/nixpkgs/pull/111487 for my distro (NixOS), which was merged last spring (2021). It looked like people were generally on board in D28234, but I make note of this here in case extra motivation is useful.
---
As pointed out in the original issue, a central tension is that LLVM already has some partial support for these sorts of things. Variables like `COMPILER_RT_INSTALL_PATH` have already been dealt with. Variables like `LLVM_LIBDIR_SUFFIX` however, will require further work, so that we may use `CMAKE_INSTALL_LIBDIR`.
These remaining items will be addressed in further patches. What is here is now rote and so we should get it out of the way before dealing more intricately with the remainder.
Reviewed By: #libunwind, #libc, #libc_abi, compnerd
Differential Revision: https://reviews.llvm.org/D99484
This is the original patch in my GNUInstallDirs series, now last to merge as the final piece!
It arose as a new draft of D28234. I initially did the unorthodox thing of pushing to that when I wasn't the original author, but since I ended up
- Using `GNUInstallDirs`, rather than mimicking it, as the original author was hesitant to do but others requested.
- Converting all the packages, not just LLVM, effecting many more projects than LLVM itself.
I figured it was time to make a new revision.
I have used this patch series (and many back-ports) as the basis of https://github.com/NixOS/nixpkgs/pull/111487 for my distro (NixOS), which was merged last spring (2021). It looked like people were generally on board in D28234, but I make note of this here in case extra motivation is useful.
---
As pointed out in the original issue, a central tension is that LLVM already has some partial support for these sorts of things. Variables like `COMPILER_RT_INSTALL_PATH` have already been dealt with. Variables like `LLVM_LIBDIR_SUFFIX` however, will require further work, so that we may use `CMAKE_INSTALL_LIBDIR`.
These remaining items will be addressed in further patches. What is here is now rote and so we should get it out of the way before dealing more intricately with the remainder.
Reviewed By: #libunwind, #libc, #libc_abi, compnerd
Differential Revision: https://reviews.llvm.org/D99484
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
