Contributor’s Guide
This page contains information for contributors to the MrDocs project. It is intended to provide an overview of the codebase and the process of adding new features.
Codebase Overview
The MrDocs codebase is divided into several modules:
This section provides an overview of each module and how they interact with each other in the MrDocs codebase.
Parsing options
MrDocs options affect the behavior of the compilation database, how symbols are extracted, and how the documentation is generated. They are parsed from the command line and configuration file.
The main entry point of MrDocs is the DoGenerateAction function in src/tool/GenerateAction.cpp.
It loads the options, creates the compilation database, and runs the extraction and generation steps.
The options formed from a combination of command line arguments and configuration file settings.
Command Line Options
Command line and common options are defined in src/tool/ToolArgs.hpp.
The ToolArgs class uses the llvm::cl library to define and parse the command line arguments.
Configuration File
Common options are defined in mrdocs/Config.hpp.
The Config class represents all public options that could be defined in a configuration file.
It also provides a representation plugins can use to access public options from the command line or configuration file.
The function clang::mrdocs::loadConfig is also provided to parse all public options from a YAML configuration file.
Internally, MrDocs uses the derived clang::mrdocs::ConfigImpl class (src/lib/Lib/ConfigImpl.hpp) to also store the private representation of parsed options, such as filters.
Finalizing Options
Common options are stored in the Config class, while the ToolArgs class stores common options and the command line options.
For instance, the config option can only be set from the command line, as it would be illogical to expect the location of the configuration file to be defined in the configuration file itself.
On the other hand, the output option can be set from both the command line and the configuration file so that the user can define a default output location in the configuration file.
Thus, after the command line and configuration file options are parsed, they are finalized in the DoGenerateAction function by calling ToolArgs::apply, which overrides the configuration file options in Config with the command line options, when applicable.
As a last step, DoGenerateAction converts the public Config settings into a ConfigImpl object, which is used by the rest of the program with the parsed options.
Extracting Symbols
At this stage, the clang frontend is used to parse the source code and generate an AST.
The AST information is extracted and stored in a Corpus object (mrdocs/Corpus.hpp).
Compilation Database
The second step in DoGenerateAction is to create a CompilationDatabase object, so we can extract symbols from its source files.
There are multiple possible sources for this file according to the configuration options: the file might be read directly from the path specified in the options, or it might be generated by MrDocs from build scripts.
Whatever the source, a derived MrDocsCompilationDatabase object (lib/Lib/MrDocsCompilationDatabase.hpp) is created to represent the compilation database.
The difference between the original CompilationDatabase and the MrDocsCompilationDatabase is that the latter includes a number of pre-processing steps to filter and transform compilation commands.
For each compilation command:
-
Command line arguments are adjusted
-
Warnings are supressed
-
Additional defines are added
-
Implicit include directories are added
-
Unrecognized arguments are removed
-
-
Paths are normalized
-
Non C++ files are filtered
Info Nodes
MrDocs represents each C++ symbol or construct as an Info node (mrdocs/Metadata/Info.hpp).
MrDocs currently defines the following Info nodes:
| Name | Description | Declaration |
|---|---|---|
|
The symbol is a namespace |
|
|
The symbol is a record (class or struct) |
|
|
The symbol is a function |
|
|
The symbol is an enum |
|
|
The symbol is a typedef |
|
|
The symbol is a variable |
|
|
The symbol is a field |
|
|
The symbol is a template specialization |
|
|
The symbol is a friend declaration |
|
|
The symbol is an enumerator |
|
|
The symbol is a deduction guide |
|
|
The symbol is a namespace alias |
|
|
The symbol is a using declaration |
|
|
The symbol is a concept |
|
Info can not only represent direct AST symbols but also C++ constructs that need to be inferred from these symbols.
Nodes in the first category will typically be created in the initial extraction step, and nodes in the second category will be created in the finalization step.
When defining a new Info type, it is important to consider how this type will be supported in all other modules of the codebase, including the AST visitor, generators, tests, and the documentation.
The script .github/check_info_nodes_support.sh will attempt to infer whether most of these features have been implemented for each node type.
Clang LibTooling
MrDocs uses Clang to extract Info objects from the C++ AST.
Clang offers two interfaces to access the C++ AST: the LibClang and LibTooling libraries.
MrDocs uses the latter, as it provides full control over the AST traversal process at the cost of an unstable API.
In LibTooling, once we have a Compilation Database, we can create a ClangTool object to run the Clang frontend on a set of source files.
clang::tooling::ClangTool Tool(compilationDatabase, sourceFiles);
newFrontendActionFactory<clang::SyntaxOnlyAction> actionFactory();
return Tool.run(actionFactory.get());The clang::tooling::ClangTool::run method takes a clang::tooling::ToolAction object that defines how to process the AST.
The action object that usually comes from a clang::tooling::FrontendActionFactory.
In the example above, the SyntaxOnlyAction is used to parse the source code and generate the AST without any further processing.
In MrDocs, this process happens in clang::mrdocs::CorpusImpl::build (src/lib/Lib/CorpusImpl.cpp), where we call Tool.run for each object in the database with our custom ASTAction action and ASTActionFactory factory (src/lib/AST/ASTVisitor.cpp).
AST Traversal
While ASTAction is the entry point for processing the AST, the real work is done by the ASTVisitor class.
As the AST is generated, it is traversed by the ASTVisitor class.
The entry point of this class is ASTVisitor::build, which recursively calls ASTVisitor::traverseDecl for the root clang::TranslationUnitDecl node of the translation unit.
During the AST traversal stage, the complete AST generated by the clang frontend is walked beginning with this root TranslationUnitDecl node.
Each clang node is converted into a mrdocs::Info node, which is then stored with any relevant information in a mrdocs::Corpus object.
USR Generation
It is during this stage that USRs (universal symbol references) are generated and hashed with SHA1 to form the 160 bit SymbolID for an entity.
Except for built-in types, all entities referenced in the corpus will be traversed and be assigned a SymbolID; including those from the standard library.
This is necessary to generate the full interface for user-defined types.
Finalizing the Corpus
After running the AST traversal on all translation units, CorpusImpl::build contains finalization steps for the Corpus object.
At this point, we process C++ constructs that are not directly represented in the AST.
The first finalization step happens in CorpusImpl::build (src/lib/Lib/CorpusImpl.cpp), where the Info objects from a single translation unit
are merged into a map containing the merged results from all other TUs.
The merging step is necessary as there may be multiple identical definitions of the same entity.
For instance, this represents the case where a function is declared at different points in the code base and might have different attributes or comments.
At this step, the doc comments are also finalized.
Each Info object has a pointer to its Javadoc object (mrdocs/Metadata/Javadoc.hpp), which is a representation of the documentation comments.
After AST traversal and Info merging, the result is stored as a map of Info objects indexed by their respective SymbolID.
A second finalization step is then performed in clang::mrdocs::finalize (src/lib/Metadata/Finalize.cpp), where any references to SymbolID objects that don’t exist are removed.
This is necessary because the AST traversal will generate references to entities that should be filtered and are not present in the corpus.
At this point, the Corpus object contains representations of all entities in the code base and further semantic C++ constructs that are not directly represented in the AST can be inferred.
Generators
Documentation generators may traverse this structure by calling Corpus::traverse with a Corpus::Visitor derived visitor and the SymbolID of the entity to visit (e.g. the global namespace).
Documentation generators are responsible for traversing the corpus and generating documentation in the desired format.
The API for documentation generators is defined in mrdocs/Generator.hpp.
Directory Layout
The MrDocs codebase is organized as follows:
include/—The main include directory
This directory contains the public headers for the MrDocs library.
-
include/mrdocs/—The core library headers-
include/mrdocs/ADT—Data Structures -
include/mrdocs/Dom—The Document Object Model for Abstract Trees -
include/mrdocs/Metadata—Infonodes and metadata classes -
include/mrdocs/Support—Various utility classes
-
src/—The main source directory
This directory contains the source code for the MrDocs library and private headers.
-
src/lib/—The core library-
src/lib/AST/—The AST traversal code -
src/lib/Dom/—The Document Object Model for Abstract Trees -
src/lib/Gen/—Generators -
src/lib/Lib/—The core library classes -
src/lib/Metadata/—Infonodes and metadata classes -
src/lib/Support/—Various utility classes
-
-
src/test/—The test directory -
src/test_suite/—The library used for testing -
src/tool/—The main program
share/—Shared resources
This directory contains shared resources for the documentation generators and utilities for developers.
Its subdirectories are installed in the share directory of the installation.
-
share/—Shared resources for the documentation generators -
share/cmake/—CMake modules to generate the documentation -
share/gdb/—GDB pretty printers -
share/mrdocs/—Shared resources for the documentation generators
docs—Documentation
This directory contains the documentation for the MrDocs project. The documentation is written in AsciiDoc and can be built using the Antora tool.
-
docs/—Documentation configuration files and scripts-
docs/modules/—The documentation asciidoc files -
docs/extensions—Antora extensions for the documentation
-
third-party/—Helpers for third-party libraries
This directory contains build scripts and configuration files for third-party libraries.
-
third-party/—Third-party libraries-
third-party/llvm/—CMake Presets for LLVM -
third-party/duktape/—CMake scripts for Duktape -
third-party/lua/—A bundled Lua interpreter
-
Coding Standards
Paths
The AST visitor and metadata all use forward slashes to represent file pathnames, even on Windows. This is so the generated reference documentation does not vary based on the platform.
Exceptions
Errors thrown by the program should always have type Exception.
Objects of this type are capable of transporting an Error object.
This is important for the scripting to work; exceptions are used to propagate errors from library code to scripts and back to the invoking code.
For exceptional cases, these thrown exceptions should be uncaught.
The tool installs an uncaught exception handler that prints a stack trace and exits the process immediately.
Testing
All new features should be accompanied by tests.
The mrdocs-test target is used to run the test suites.
This target has its entry point in src/test/TestMain.cpp, which can take two paths:
-
Golden testing: When input paths are provided to the test executable via the command line, the test suite will run the
DoTestAction()that iterates all files intest-filescomparing the input source files with the expected XML output files. -
Unit testing: When no input paths are provided, all unit tests will be run via
unit_test_main(), defined by the our test-suite library insrc/test_suite/test_suite.cpp.
The fixtures for golden testing are defined in test-files/golden-tests, where files in each directory have the following format:
-
mrdocs.yml: Basic configuration options for all files in this directory. -
<filename>.cpp: The input source file to extract symbols from. -
<filename>.xml: The expected XML output file generated with the XML generator. -
<filename>.bad.xml: The test output file generated when the test fails. -
<filename>.yml: Extra configuration options for this specific file.
Contributing
If you find a bug or have a feature request, please open an issue on the MrDocs GitHub repository: https://github.com/cppalliance/mrdocs/issues
If you would like to contribute a feature or bug fix, please open a pull request on the MrDocs GitHub repository: https://github.com/cppalliance/mrdocs/pulls
If you would like to discuss a feature or bug fix before opening a pull request, discussing happen in the #mrdocs channel on the Cpplang Slack: https://cpplang.slack.com/