Adds a JSON generator backend to emit mapped information as JSON. This will enable a better testing format for upcoming changes. It can also potentially serve to feed our other backend generators in the future, like Mustache which already serializes information to JSON before emitting as HTML.
This patch contains functionality to emit classes and provides most of the basis of the generator.
This patch re-enables -ftime-trace support in clang-doc. Initial support
in #97644 was reverted, and never relanded. This patch adds back the
command line option, and leverages the RAII tracing infrastructure more
thoroughly.
This patch refactors the clang-doc tool to replace manual error handling
(e.g., checking error codes and calling exit()) with llvm::ExitOnError.
Fixes#140085
This patch adds a command line option and enables the Mustache template
HTML backend. This allows users to use the new, more flexible templates
over the old and cumbersome HTML output. Split from #133161.
Co-authored-by: Peter Chou <peter.chou@mail.utoronto.ca>
Additionally moving the declarations from Generator.cpp
prevents misc-use-internal-linkage warnings from clang-tidy.
We fix those here too, since the static functions are part
of the same diagnostic
This PR adds a new command-line option that allows users to specify the
prefix used for line-based anchors in repository URLs. Different
repository interfaces use different formats for line anchors (GitHub
uses `#L123`, googlesource uses `#123`, etc.). This option enables users
to customize the line prefix to match their repository platform without
requiring hard-coded values for each service.
Fixes#59814
Currently, when we set URLs from JS, we set them only using the protocol
and host locations. This works fine when docs are served from the base
directory of the site, but if you want to nest it under another
directory, our JS fails to set the correct path, leading to broken
links.
This patch adds a --base option to specify the path prefix to use, which
is set in the generated index_json.js file. index.json can then fill in
the prefix appropriately when generating links in a browser. This flag
has no effect for non HTML output.
Given an index hosted at: www.docs.com/base_directory/index.html
we used to generate the following link:
www.docs.com/file.html
Using --base base_directory we now generate:
www.docs.com/base_directory/file.html
This allows such links to work when hosting pages without using a custom
index.js.
Fixes https://github.com/llvm/llvm-project/issues/97507#96809 introduce non-determinstic behaviour in clang-doc which caused
the output to be randomly ordered which lead to randomly failing test.
This patch modify clang-doc to behave deterministically again by sorting
the output by location or USR.
this patches adds ftime-tracing option for clang-doc which was what I
used to profile the performance problems we had with clang-doc
generating the llvm docs.
- ftime-trace produces a time profiler json file called
clang-doc-tracing.json
Reapply https://github.com/llvm/llvm-project/pull/94717
Adds a new option --asset which allows users to specified the asset
folder for the html output of clang-doc.
This patch adds a better test for --asset option + fixes bug where
clang-doc assumes that user supplied js file is assume to be index.js
Adds e2e tests for clang-doc, which serves as useful test against
regression in the html output. It also serves as a helpful tool for code
review as we can see the diffs to clang-docs output
Closes https://github.com/llvm/llvm-project/issues/93925
The base class llvm::ThreadPoolInterface will be renamed
llvm::ThreadPool in a subsequent commit.
This is a breaking change: clients who use to create a ThreadPool must
now create a DefaultThreadPool instead.
Fixes a warning about a potentially unsupported template argument
deduction by explicitly specifying the template type in std::lock_guard.
Patch By: brettw
Differential Revision: https://reviews.llvm.org/D138961
Previously file naming and directory layout was handled on a per Info
object basis by ClangDocMain and the generators blindly wrote to the
files given. This means all generators must use the same file layout and
caused problems where multiple objects mapped to the same file. The
object collision problem happens most easily with template
specializations because the template parameters are not part of the
"name".
This patch moves the responsibility for output file organization to the
generators. Currently HTML and MD use the same structure as before. But
they now collect all objects that map to a given file and combine them,
avoiding the corruption problems.
Converts the YAML generator to naming files based on USR in one
directory. This is easier for downstream tools to manage and avoids the
naming problems with template specializations. Since this change
requires backward-incompatible output changes to referenced files anyway
(since each one is now an array), this is a good time to introduce this
change.
Differential Revision: https://reviews.llvm.org/D138073
This reverts commit f8a469fc572778d05b72f34a772082cf3abd3cda.
The test in single-file-public.cpp breaks on Mac, due to an unknown
regextype in the find command.
Previously file naming and directory layout was handled on a per Info
object basis by ClangDocMain and the generators blindly wrote to the
files given. This means all generators must use the same file layout and
caused problems where multiple objects mapped to the same file. The
object collision problem happens most easily with template
specializations because the template parameters are not part of the
"name".
This patch moves the responsibility for output file organization to the
generators. Currently HTML and MD use the same structure as before. But
they now collect all objects that map to a given file and combine them,
avoiding the corruption problems.
Converts the YAML generator to naming files based on USR in one
directory. This is easier for downstream tools to manage and avoids
the naming problems with template specializations. Since this change
requires backward-incompatible output changes to referenced files anyway
(since each one is now an array), this is a good time to introduce this
change.
Differential Revision: https://reviews.llvm.org/D138073
When using `clang-doc --format=html` it will crash on startup because of an assertion doing a self-assignment of a `SmallString`. This patch removes the self-assignment by creating an intermediate copy.
Reviewed By: paulkirth
Differential Revision: https://reviews.llvm.org/D131793
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This change has two components. The moves the generated file
for a namespace to the directory named after the namespace in
a file named 'index.<format>'. This greatly improves the browsing
experience since the index page is shown by default for a directory.
The second improves the markdown output by adding the links to the
referenced pages for children objects and the link back to the source
code.
Patch By: Clayton
Differential Revision: https://reviews.llvm.org/D72954
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
This reverts commit 3f76260dc0674cc0acb25f550a0f0c594cf537ea.
Breaks at least these tests on Windows:
Clang :: Driver/clang-offload-bundler.c
Clang :: Driver/clang-offload-wrapper.c
"Bad block found.\n" -> "bad block found"
The lower cased form with no full stop or newline is more common in LLVM
tools.
Reviewed By: juliehockett
Differential Revision: https://reviews.llvm.org/D66783
llvm-svn: 370155
The new design includes a header (contains the project name), a main section, and a footer.
The main section is divided into three subsections. Left, middle, right. The left section contains the general index, the middle contains the info's data, and the right contains the index for the info's content.
The CSS has been updated.
A flag --project-name is added.
The Attributes attribute of the TagNode struct is now a vector of pairs because these attributes should be rendered in the insertion order.
The functions (cpp and js) that converts an Index tree structure into HTML were slightly modified; the first ul tag created is now a ol tag. The inner lists are still ul.
Differential Revision: https://reviews.llvm.org/D66353
llvm-svn: 369139
The value, if any, of --source-root flag was not being used.
This has been fixed and the logic was moved to the ClangDocContext
contructor.
Differential revision: https://reviews.llvm.org/D66268
llvm-svn: 369065
Two command line options have been added to clang-doc.
--repository=<string> - URL of repository that hosts code; used for links to definition locations.
--source-root=<string> - Directory where processed files are stored. Links to definition locations will only be generated if the file is in this dir.
If the file is in the source-root and a repository options is passed;
a link to the source code will be rendered by the HTML generator.
Differential Revision: https://reviews.llvm.org/D65483
llvm-svn: 368460
Idx in ClangDocContext instance was being modified by multiple threads
causing a seg fault.
A mutex is added to avoid this.
Differential Revision: https://reviews.llvm.org/D65915
llvm-svn: 368313
Reduce phase has been parallelized and a execution time was reduced by
60% with this.
The reading of bitcode (bitcode -> Info) was moved to this segment of
code parallelized so it now happens just before reducing.
Differential Revision: https://reviews.llvm.org/D65628
llvm-svn: 368206
An index structure is created while generating the output file for each
info. This structure is parsed to JSON and written to a file in the
output directory. The html for the index is not rendered by clang-doc. A
Javascript file is included in the output directory, this will the JSON
file and insert HTML elements into the file.
Differential Revision: https://reviews.llvm.org/D65690
llvm-svn: 368070
