Script Generators

A script generator is the second kind of Mr.Docs extension driven by scripts. This covers the cases where customizing Handlebars templates is not enough to express the output format. A generator hands the whole emit to the script instead.

A generator is written and loaded exactly like a corpus transform: a .js or .lua script under extensions/ in an addon directory, handed the same ctx. It declares itself with mrdocs.register_generator(id, fn) instead of mrdocs.register_transform. The id is the name you select on the command line with --generator=<id>; a registered generator takes precedence over a built-in of the same name.

Outputting the corpus

The smallest useful generator hands the corpus straight to JSON.stringify. A complete, runnable example lives at examples/generators/script-driven/json/. Given this input:

simple.cpp
/// A vector in the Euclidean plane.
struct Vector
{
    /// The length (magnitude) of the vector.
    double length() const;
};

In very few lines of code, this extension declares a json generator that writes every symbol to a single reference.json:

addons/extensions/json.js
mrdocs.register_generator("json", function(ctx) {
  const indent = ctx.params.indent || 0;
  ctx.output.write(
    "reference.json",
    JSON.stringify(ctx.corpus.symbols, null, indent));
});

Select it with --generator=json. Because ctx.corpus.symbols is already plain data, the whole generator is one JSON.stringify call, and the shape is yours: map or filter the array, or read ctx.params to add options, before writing it. A script generator takes precedence over a built-in of the same name, so this replaces the built-in JSON generator, which is the same idea implemented in C++.

The output is an array of every symbol, each a full DOM object as seen by the templates, with parent/child relationships expressed as id references. Its first entry, the global namespace, begins (the file continues with one object per symbol):

reference.json (excerpt)
[
  {
    "$meta": {
      "type": "NamespaceSymbol",
      "bases": [
        "SymbolCommonBase",
        "Symbol"
      ]
    },
    "anchor": "index",
    "loc": {
      "$meta": {
        "type": "SourceInfo",
        "bases": []
      },
      "loc": []
    },
    "kind": "namespace",
    "id": "4ZrjxJnU1LA5xSyrWMNuXTvSYKwt",
    "extraction": "regular",
    "isCopyFromInherited": false,
    "attributes": [],
    "isInline": false,
    "isAnonymous": false,
    "usingDirectives": [],
    "members": {
      "$meta": {
        "type": "NamespaceTranche",
        "bases": []
      },
      "namespaces": [],
      "namespaceAliases": [],
      "typedefs": [],
      "records": [
        "43E6qXkmkkfeWwCdMLUnhvUC5tmQ"
      ],
      "enums": [],
      "functions": [],
      "variables": [],
      "concepts": [],
      "guides": [],
      "usings": [],
      "macros": []
    }
  },

Filtering the corpus

Consider that you need a JSON file representing a search index for your reference documentation, in a format supported by your documentation framework. A complete, runnable example lives at examples/generators/script-driven/search-index/. Given this input:

simple.cpp
/// A vector in the Euclidean plane.
struct Vector
{
    /** The length (magnitude) of the vector.

        @return The Euclidean length.
    */
    double length() const;

    /** Scale the vector componentwise.

        @param sx Factor applied to the x component.
        @param sy Factor applied to the y component.
    */
    void scale(double sx, double sy);
};

the extension declares a search-index generator that emits a single search-index.json aggregating every symbol, an artifact no per-page generator can produce:

addons/extensions/search_index.lua
local url_for

mrdocs.register_generator("search-index", function(ctx)
  local entries = {}
  for _, sym in ipairs(ctx.corpus.symbols) do
    local name = sym.name or ""
    if name ~= "" then
      local entry = {}
      entry.name = name
      entry.url = url_for(ctx.corpus, sym)
      entries[#entries + 1] = entry
    end
  end
  ctx.output.write("search-index.json", ctx.stringify(entries))
end)

--- A symbol's page URL: the chain of anchors from the global namespace down to
--- the symbol, joined by "/", plus ".html". Walk up through `parent` (a symbol
--- id) with `corpus.get`; the global namespace has no parent, so it contributes
--- no path segment.
--- @param corpus table `ctx.corpus`, used to resolve each `parent` id.
--- @param sym table The symbol to build a URL for.
--- @return string url The symbol's page URL.
function url_for(corpus, sym)
  local parts = {}
  local cur = sym
  while cur and cur.parent do
    table.insert(parts, 1, cur.anchor)
    cur = corpus.get(cur.parent)
  end
  return table.concat(parts, "/") .. ".html"
end

Select it with --generator=search-index; it writes search-index.json into the output directory: one entry per named symbol.

search-index.json
[
    {
        "name": "Vector",
        "url": "Vector.html"
    },
    {
        "name": "length",
        "url": "Vector/length.html"
    },
    {
        "name": "scale",
        "url": "Vector/scale.html"
    }
]

Table of Contents

A table of contents or a navigation list over the whole API is the same idea as the search index: an artifact aggregated across every symbol, but it emits a nested list instead of JSON objects. Deriving it from the corpus prevents a project from manually maintaining a separate nav partial and re-syncing it whenever the API changes. A complete, runnable example lives at examples/generators/script-driven/toc/:

  • Generator

  • Input

  • Config

addons/extensions/toc.js
mrdocs.register_generator("toc", function(ctx) {
  const roots = buildRoots(ctx.corpus);
  ctx.output.write("toc.md", markdown(roots, 0));
  ctx.output.write("toc.adoc", asciidoc(roots, 0));
  ctx.output.write("toc.html", html(roots));
});

/**
 * Build the top-level navigation nodes for the whole corpus: the page-level
 * children of the global namespace (the one symbol with no `parent`), each
 * expanded into a node tree.
 * @param {object} corpus - `ctx.corpus`.
 * @returns {object[]} The root nav nodes.
 */
function buildRoots(corpus) {
  let root = null;
  for (let i = 0; i < corpus.symbols.length; ++i) {
    if (!corpus.symbols[i].parent) {
      root = corpus.symbols[i];
      break;
    }
  }
  const roots = [];
  const topIds = root ? childIds(root) : [];
  for (let j = 0; j < topIds.length; ++j) {
    const top = corpus.get(topIds[j]);
    if (top && top.name) {
      roots.push(node(corpus, top));
    }
  }
  return roots;
}

// Page-level child categories of a namespace, in the order a nav should list
// them. Record members (methods, fields) live on the record's own page, so the
// walk descends through namespaces only and treats everything else as a leaf.
const CATEGORIES = [
  "namespaces", "records", "enums", "functions",
  "variables", "typedefs", "concepts", "usings", "macros"
];

/**
 * The ids of a namespace's page-level children, across every category in
 * `CATEGORIES`. Anything that is not a namespace is a leaf in the navigation
 * tree, so this returns an empty array for it.
 * @param {object} sym - The symbol whose children to collect.
 * @returns {string[]} The child symbol ids, in category order.
 */
function childIds(sym) {
  const ids = [];
  if (sym.kind !== "namespace" || !sym.members) {
    return ids;
  }
  for (let c = 0; c < CATEGORIES.length; ++c) {
    const list = sym.members[CATEGORIES[c]] || [];
    for (let k = 0; k < list.length; ++k) {
      ids.push(list[k]);
    }
  }
  return ids;
}

/**
 * Build a plain `{ name, url, children }` node for a symbol, recursing through
 * its page-level children. Only namespaces have children; every other symbol
 * is a leaf linking to its own page. Unnamed children are skipped.
 * @param {object} corpus - `ctx.corpus`.
 * @param {object} sym - The symbol to convert.
 * @returns {{name: string, url: string, children: object[]}} The nav node.
 */
function node(corpus, sym) {
  const children = [];
  const ids = childIds(sym);
  for (let i = 0; i < ids.length; ++i) {
    const child = corpus.get(ids[i]);
    if (child && child.name) {
      children.push(node(corpus, child));
    }
  }
  return { name: sym.name, url: urlFor(corpus, sym), children: children };
}

/**
 * The page URL for a symbol: the chain of anchors from the global namespace
 * down to the symbol, joined by "/", plus ".html" (the shape the HTML
 * generator emits).
 * @param {object} corpus - `ctx.corpus`, used to resolve each `parent` id.
 * @param {object} sym - The symbol to build a URL for.
 * @returns {string} The symbol's page URL.
 */
function urlFor(corpus, sym) {
  const parts = [];
  let cur = sym;
  while (cur && cur.parent) {
    parts.unshift(cur.anchor);
    cur = corpus.get(cur.parent);
  }
  return parts.join("/") + ".html";
}

/**
 * Render nav nodes as a nested Markdown bullet list.
 * @param {object[]} nodes - Nav nodes.
 * @param {number} depth - Indentation depth, 0 at the top level.
 * @returns {string} The Markdown list.
 */
function markdown(nodes, depth) {
  let out = "";
  for (let i = 0; i < nodes.length; ++i) {
    const n = nodes[i];
    out += "  ".repeat(depth) + "- [" + n.name + "](" + n.url + ")\n";
    out += markdown(n.children, depth + 1);
  }
  return out;
}

/**
 * Render nav nodes as a nested AsciiDoc list of `xref` links.
 * @param {object[]} nodes - Nav nodes.
 * @param {number} depth - Nesting depth, 0 at the top level (one `*` per level).
 * @returns {string} The AsciiDoc list.
 */
function asciidoc(nodes, depth) {
  let out = "";
  for (let i = 0; i < nodes.length; ++i) {
    const n = nodes[i];
    out += "*".repeat(depth + 1) + " xref:" + n.url + "[" + n.name + "]\n";
    out += asciidoc(n.children, depth + 1);
  }
  return out;
}

/**
 * Render nav nodes as a nested HTML `<ul>` list. Returns an empty string for
 * an empty list, so leaf nodes emit no child `<ul>`.
 * @param {object[]} nodes - Nav nodes.
 * @returns {string} The HTML list.
 */
function html(nodes) {
  if (nodes.length === 0) {
    return "";
  }
  let out = "<ul>\n";
  for (let i = 0; i < nodes.length; ++i) {
    const n = nodes[i];
    out += "<li><a href=\"" + n.url + "\">" + n.name + "</a>" + html(n.children) + "</li>\n";
  }
  out += "</ul>\n";
  return out;
}
simple.cpp
/// Two-dimensional geometry primitives.
namespace geo {

/// A point in the plane.
struct Point
{
    /// The x coordinate.
    double x;

    /// The y coordinate.
    double y;

    /** The distance to another point.

        @param other The point to measure to.
        @return The Euclidean distance between the two points.
    */
    double distance_to(Point other) const;
};

/// How two shapes relate spatially.
enum class Relation
{
    disjoint,    ///< No shared points.
    touching,    ///< A shared boundary only.
    overlapping  ///< Shared interior points.
};

/** The midpoint of two points.

    @param a The first point.
    @param b The second point.
    @return The point halfway between `a` and `b`.
*/
Point midpoint(Point a, Point b);

/// Coordinate-system helpers.
namespace coord {

/** Convert polar coordinates to a Cartesian point.

    @param r The radius.
    @param theta The angle in radians.
    @return The equivalent Cartesian point.
*/
Point from_polar(double r, double theta);

}

}
mrdocs.yml
addons-supplemental:
  - addons
generator: toc
multipage: false
show-namespaces: false
warn-if-undocumented: false
source-root: .
input:
  - .

The generator walks the namespace tree once, then serializes it to three formats from that single walk, so the nav for each output stays in step. The corpus is a tree: the global namespace is the one symbol with no parent, and a namespace lists its children in members, grouped by category (namespaces, records, enums, functions, …​). The walk descends through namespaces and treats every record, enum, and function as a leaf linking to its own page. Select it with --generator=toc; it writes one file per format:

  • Markdown

  • AsciiDoc

  • HTML

toc.md
- [geo](geo.html)
  - [coord](geo/coord.html)
    - [from_polar](geo/coord/from_polar.html)
  - [Point](geo/Point.html)
  - [Relation](geo/Relation.html)
  - [midpoint](geo/midpoint.html)
toc.adoc
* xref:geo.html[geo]
** xref:geo/coord.html[coord]
*** xref:geo/coord/from_polar.html[from_polar]
** xref:geo/Point.html[Point]
** xref:geo/Relation.html[Relation]
** xref:geo/midpoint.html[midpoint]
toc.html
<ul>
<li><a href="geo.html">geo</a><ul>
<li><a href="geo/coord.html">coord</a><ul>
<li><a href="geo/coord/from_polar.html">from_polar</a></li>
</ul>
</li>
<li><a href="geo/Point.html">Point</a></li>
<li><a href="geo/Relation.html">Relation</a></li>
<li><a href="geo/midpoint.html">midpoint</a></li>
</ul>
</li>
</ul>

Reflection Generators

A custom generator does not have to emit documentation. It could be used for any task related to the project’s public API and even write back to the files used to generate the Corpus. For instance, Boost.Describe adds compile-time reflection to C++ types, but its BOOST_DESCRIBE_STRUCT and BOOST_DESCRIBE_ENUM annotations are written by hand. The corpus already knows every type and its members, so the same walk that produces docs can emit those annotations to keep them in sync with the code. A complete, runnable example lives at examples/generators/script-driven/describe/:

  • Generator

  • Input

  • Config

addons/extensions/describe.js
mrdocs.register_generator("describe", function(ctx) {
  const grouped = groupByHeader(ctx.corpus);
  for (let i = 0; i < grouped.order.length; ++i) {
    const header = grouped.order[i];
    const group = grouped.byHeader[header];
    ctx.output.write(sidecarName(header), renderSidecar(header, group.includes, group.order, group.byNs));
  }
});

/**
 * Group the describe annotation of every record and enum in the corpus, first
 * by the header it is defined in and then by its enclosing namespace, keeping
 * first-seen order at every level. Each header group also collects the base
 * sidecars its types depend on, minus its own.
 * @param {object} corpus - `ctx.corpus`.
 * @returns {{order: string[], byHeader: Object<string, {order: string[], byNs: Object<string, string[]>, includes: string[]}>}}
 *   The headers in first-seen order and, for each, its namespaces, their
 *   annotation lines, and the base sidecars to include.
 */
function groupByHeader(corpus) {
  const order = [];
  const byHeader = {};
  for (let i = 0; i < corpus.symbols.length; ++i) {
    const sym = corpus.symbols[i];
    const line = describeMacro(corpus, sym);
    if (line === null) {
      continue;
    }
    const header = sourceHeaderOf(sym);
    if (!header) {
      continue;
    }
    if (!(header in byHeader)) {
      byHeader[header] = { order: [], byNs: {}, includes: [] };
      order.push(header);
    }
    const group = byHeader[header];
    const ns = namespaceOf(corpus, sym);
    if (!(ns in group.byNs)) {
      group.byNs[ns] = [];
      group.order.push(ns);
    }
    group.byNs[ns].push(line);

    const self = sidecarName(header);
    const bases = baseSidecars(corpus, sym);
    for (let s = 0; s < bases.length; ++s) {
      if (bases[s] !== self && group.includes.indexOf(bases[s]) === -1) {
        group.includes.push(bases[s]);
      }
    }
  }
  return { order: order, byHeader: byHeader };
}

/**
 * The Boost.Describe annotation for a symbol: BOOST_DESCRIBE_STRUCT for a
 * record (its base classes and own public members) or BOOST_DESCRIBE_ENUM for
 * an enum (its enumerators). Returns null for any other kind of symbol.
 * @param {object} corpus - `ctx.corpus`.
 * @param {object} sym - The symbol to annotate.
 * @returns {string|null} The macro invocation, or null if `sym` is not a
 *   record or enum.
 */
function describeMacro(corpus, sym) {
  if (sym.kind === "record") {
    const bases = [];
    const baseList = sym.bases || [];
    for (let b = 0; b < baseList.length; ++b) {
      const name = baseList[b].type && baseList[b].type.name && baseList[b].type.name.identifier;
      if (name) {
        bases.push(name);
      }
    }
    const members = ownMemberNames(corpus, sym, "variables").concat(ownMemberNames(corpus, sym, "functions"));
    return "BOOST_DESCRIBE_STRUCT(" + sym.name + ", (" + bases.join(", ") + "), (" + members.join(", ") + "))";
  }
  if (sym.kind === "enum") {
    const values = [];
    const constants = sym.constants || [];
    for (let c = 0; c < constants.length; ++c) {
      const value = corpus.get(constants[c]);
      if (value && value.name) {
        values.push(value.name);
      }
    }
    return "BOOST_DESCRIBE_ENUM(" + sym.name + ", " + values.join(", ") + ")";
  }
  return null;
}

/**
 * The names of a record's own public members in `category`. The interface also
 * carries inherited members; Boost.Describe learns those from the base list, so
 * only members declared on this record (their `parent` is the record) are kept.
 * @param {object} corpus - `ctx.corpus`.
 * @param {object} record - The record symbol.
 * @param {string} category - The interface bucket, "variables" or "functions".
 * @returns {string[]} The own public member names, in declaration order.
 */
function ownMemberNames(corpus, record, category) {
  const names = [];
  const list = record.interface.public[category] || [];
  for (let i = 0; i < list.length; ++i) {
    const member = corpus.get(list[i]);
    if (member && member.name && member.parent === record.id) {
      names.push(member.name);
    }
  }
  return names;
}

/**
 * The fully-qualified enclosing namespace of a symbol ("a::b"), or "" when it
 * sits at global scope. A BOOST_DESCRIBE_STRUCT must appear in the type's own
 * namespace so the unqualified name resolves, so entries are grouped by this.
 * @param {object} corpus - `ctx.corpus`, used to walk `parent` ids.
 * @param {object} sym - The symbol whose namespace to compute.
 * @returns {string} The enclosing namespace, "::"-joined, or "" at global scope.
 */
function namespaceOf(corpus, sym) {
  const parts = [];
  let cur = sym.parent ? corpus.get(sym.parent) : null;
  while (cur && cur.name !== undefined && cur.kind === "namespace") {
    parts.unshift(cur.name);
    cur = cur.parent ? corpus.get(cur.parent) : null;
  }
  return parts.join("::");
}

/**
 * The header a symbol is defined in, taken from its definition location, or ""
 * when the location is unknown. Sidecars are grouped by this so each one can
 * include exactly the header whose types it annotates.
 * @param {object} sym - The symbol whose defining header to read.
 * @returns {string} The header path, or "" if unknown.
 */
function sourceHeaderOf(sym) {
  return (sym.loc && sym.loc.defLoc && sym.loc.defLoc.shortPath) || "";
}

/**
 * The sidecar file name for a header: "foo.hpp" becomes "foo.described.hpp", and
 * a header with any other (or no) extension gets ".described.hpp" appended.
 * Including it gives the described version of that header's types.
 * @param {string} header - The source header path.
 * @returns {string} The sidecar path.
 */
function sidecarName(header) {
  const dot = header.lastIndexOf(".");
  const slash = header.lastIndexOf("/");
  const stem = dot > slash ? header.slice(0, dot) : header;
  return stem + ".described.hpp";
}

/**
 * The sidecars of a record's described base classes: for each base that
 * resolves to a record or enum this generator also describes, the sidecar of
 * the header that base is defined in. Reflecting the record's inherited members
 * needs those base descriptors present.
 * @param {object} corpus - `ctx.corpus`.
 * @param {object} sym - The record whose bases to resolve.
 * @returns {string[]} The base sidecar paths, in base order.
 */
function baseSidecars(corpus, sym) {
  const sidecars = [];
  const baseList = sym.bases || [];
  for (let b = 0; b < baseList.length; ++b) {
    const id = baseList[b].type && baseList[b].type.name && baseList[b].type.name.id;
    const base = id ? corpus.get(id) : null;
    if (base && describeMacro(corpus, base) !== null) {
      const header = sourceHeaderOf(base);
      if (header) {
        sidecars.push(sidecarName(header));
      }
    }
  }
  return sidecars;
}

/**
 * Assemble one sidecar header: an include guard, a do-not-edit banner, an
 * include of the source header and of any base sidecars, the Boost.Describe
 * include, and each namespace's annotations wrapped in a `namespace { ... }`
 * block (a global-scope group, keyed by "", is emitted unwrapped).
 * @param {string} header - The source header this sidecar annotates.
 * @param {string[]} includes - Base sidecar paths to include first.
 * @param {string[]} nsOrder - Namespaces in emission order.
 * @param {Object<string, string[]>} byNs - Annotation lines per namespace.
 * @returns {string} The sidecar header text.
 */
function renderSidecar(header, includes, nsOrder, byNs) {
  const guard = guardName(sidecarName(header));
  let out =
    "#ifndef " + guard + "\n" +
    "#define " + guard + "\n\n" +
    "// Generated by MrDocs from documented types. Do not edit by hand.\n" +
    "//\n" +
    "// Include this to gain Boost.Describe reflection over the types declared\n" +
    "// in " + header + ".\n\n" +
    "#include \"" + header + "\"\n";
  for (let i = 0; i < includes.length; ++i) {
    out += "#include \"" + includes[i] + "\"\n";
  }
  out += "#include <boost/describe.hpp>\n\n";
  for (let g = 0; g < nsOrder.length; ++g) {
    const ns = nsOrder[g];
    if (ns) {
      out += "namespace " + ns + " {\n\n";
    }
    out += byNs[ns].join("\n") + "\n";
    if (ns) {
      out += "\n}  // namespace " + ns + "\n";
    }
    out += "\n";
  }
  out += "#endif  // " + guard + "\n";
  return out;
}

/**
 * An include-guard macro for a sidecar path: uppercased, every run of
 * non-alphanumeric characters collapsed to a single "_", with a leading "_"
 * added if the result would start with a digit.
 * @param {string} sidecar - The sidecar path.
 * @returns {string} The guard macro name.
 */
function guardName(sidecar) {
  let guard = sidecar.toUpperCase().replace(/[^A-Z0-9]+/g, "_");
  if (/^[0-9]/.test(guard)) {
    guard = "_" + guard;
  }
  return guard;
}
  • circle.hpp

  • shape.hpp

  • vec2.hpp

  • level.hpp

#ifndef APP_CIRCLE_HPP
#define APP_CIRCLE_HPP

#include "shape.hpp"

namespace app {

/// A circle.
struct Circle : Shape
{
    /// The radius.
    double radius;

    /// The enclosed area.
    double area() const;
};

} // namespace app

#endif
#ifndef APP_SHAPE_HPP
#define APP_SHAPE_HPP

namespace app {

/// A named shape.
struct Shape
{
    /// The display name.
    const char* name;
};

} // namespace app

#endif
#ifndef APP_VEC2_HPP
#define APP_VEC2_HPP

namespace app {

/// A 2-D vector.
struct Vec2
{
    /// The x component.
    double x;
    /// The y component.
    double y;

    /// The Euclidean length.
    double length() const;
};

} // namespace app

#endif
#ifndef APP_LEVEL_HPP
#define APP_LEVEL_HPP

namespace app {

/// Logging severity.
enum class Level
{
    debug,    ///< Verbose tracing.
    info,     ///< Normal operation.
    warning,  ///< Something looks wrong.
    error     ///< A failure occurred.
};

} // namespace app

#endif
mrdocs.yml
addons-supplemental:
  - addons
generator: describe
multipage: false
show-namespaces: false
warn-if-undocumented: false
source-root: include
input:
  - include

A non-intrusive BOOST_DESCRIBE_STRUCT has to sit at the type’s namespace scope with the type and its bases complete, so the extension emits one sidecar header per input header. In the example, for circle.hpp it writes circle.described.hpp, which includes circle.hpp and then the annotations for the types that header declares, including the annotations for the base class. A project can include the sidecar wherever it wants reflection over that header’s types. Select it with --generator=describe:

Generated sidecars
  • circle.described.hpp

  • shape.described.hpp

  • vec2.described.hpp

  • level.described.hpp

#ifndef CIRCLE_DESCRIBED_HPP
#define CIRCLE_DESCRIBED_HPP

// Generated by MrDocs from documented types. Do not edit by hand.
//
// Include this to gain Boost.Describe reflection over the types declared
// in circle.hpp.

#include "circle.hpp"
#include "shape.described.hpp"
#include <boost/describe.hpp>

namespace app {

BOOST_DESCRIBE_STRUCT(Circle, (Shape), (radius, area))

}  // namespace app

#endif  // CIRCLE_DESCRIBED_HPP
#ifndef SHAPE_DESCRIBED_HPP
#define SHAPE_DESCRIBED_HPP

// Generated by MrDocs from documented types. Do not edit by hand.
//
// Include this to gain Boost.Describe reflection over the types declared
// in shape.hpp.

#include "shape.hpp"
#include <boost/describe.hpp>

namespace app {

BOOST_DESCRIBE_STRUCT(Shape, (), (name))

}  // namespace app

#endif  // SHAPE_DESCRIBED_HPP
#ifndef VEC2_DESCRIBED_HPP
#define VEC2_DESCRIBED_HPP

// Generated by MrDocs from documented types. Do not edit by hand.
//
// Include this to gain Boost.Describe reflection over the types declared
// in vec2.hpp.

#include "vec2.hpp"
#include <boost/describe.hpp>

namespace app {

BOOST_DESCRIBE_STRUCT(Vec2, (), (x, y, length))

}  // namespace app

#endif  // VEC2_DESCRIBED_HPP
#ifndef LEVEL_DESCRIBED_HPP
#define LEVEL_DESCRIBED_HPP

// Generated by MrDocs from documented types. Do not edit by hand.
//
// Include this to gain Boost.Describe reflection over the types declared
// in level.hpp.

#include "level.hpp"
#include <boost/describe.hpp>

namespace app {

BOOST_DESCRIBE_ENUM(Level, debug, info, warning, error)

}  // namespace app

#endif  // LEVEL_DESCRIBED_HPP

Notice the level of customization provided by extensions here. There are so many ways to achieve this that no tool could ever ship it as a built-in feature. A script generator provides you with the corpus, and the project can specify the final customization step.

Round-trip extensions

An extension can register a transform and a generator together, so one addon covers both directions of the same job. For instance, a common task in C++ projects is keeping documentation examples and standalone snippet files in sync:

  • Import example snippets from test or example files into the documentation comments when they are complex enough to be kept in a separate file, typically because they need special setup or teardown code to compile and run.

  • Exporting snippets from documentation comments to testable source files when the snippets are short enough to be written inline.

In the extension:

  • A corpus transform script can read example files and inject their snippets into the matching symbol’s docs.

  • A generator script can write every @code example out to a file the build can compile, so an example in a comment is broken.

An extension that implements both can keep the two in sync: the generator exports snippets from the docs to files if they weren’t t included by the transform function.

  • Extension

  • Headers

  • Example file

  • Config

addons/extensions/snippets.lua
local read_file, header_of, stem_of, line_of, dedent, extract_snippets
local to_plain, code_blocks, indent_block, unique_name, symbol_function
local fingerprint, mark_imported, was_imported

-- Fingerprints of the snippets the transform imported, kept so the generator can
-- skip re-exporting an example that already lives in a compiled file. This is
-- module-level state shared between the two callbacks: the transform fills it,
-- the generator reads it.
local imported = {}

-- Import: for each symbol with no example of its own, append the snippets tagged
-- with its name in `<source>/<header-stem>.cpp` to its documentation, after any
-- description it already carries, and remember them so export skips them.
mrdocs.register_transform("snippets", function(ctx)
  local root = (ctx.params and ctx.params.source) or "example-code"
  local files = {}
  for _, sym in ipairs(ctx.corpus.symbols) do
    if sym.name ~= nil and #code_blocks(sym) == 0 then
      local header = header_of(sym)
      if header then
        local path = root .. "/" .. stem_of(header) .. ".cpp"
        if files[path] == nil then
          files[path] = read_file(path) or false
        end
        local text = files[path]
        if text then
          local snippets = extract_snippets(text, sym.name)
          if #snippets > 0 then
            if not sym.doc then
              sym.doc = {}
            end
            -- A script assigns a fresh block list rather than mutating one in
            -- place, so rebuild the description and append the imported snippets.
            local doc = {}
            for _, b in ipairs(sym.doc.document or {}) do
              doc[#doc + 1] = to_plain(b)
            end
            -- MrDocs heads an inline @code example with an "Example"/"Examples"
            -- heading; add the same before the imported snippets so they read
            -- the same as inline ones.
            local label = (#snippets == 1) and "Example" or "Examples"
            doc[#doc + 1] = {
              kind = "heading",
              children = { { kind = "text", literal = label } },
            }
            for _, snippet in ipairs(snippets) do
              doc[#doc + 1] = { kind = "code", literal = snippet }
              mark_imported(snippet, header)
            end
            sym.doc.document = doc
          end
        end
      end
    end
  end
end)

-- Export: group every symbol's inline @code examples by header and write one
-- runnable test file per header under `<output>/<header-stem>.cpp`, one function
-- per symbol plus a `main` that calls them all.
mrdocs.register_generator("snippets", function(ctx)
  local out_dir = (ctx.params and ctx.params.output) or "exported"
  local groups = {}
  local order = {}
  for _, sym in ipairs(ctx.corpus.symbols) do
    if sym.name ~= nil then
      local header = header_of(sym)
      if header then
        -- Export only the snippets written inline in the header; the ones the
        -- transform pulled from example files are already tested where they live.
        local blocks = {}
        for _, literal in ipairs(code_blocks(sym)) do
          if not was_imported(literal, header) then
            blocks[#blocks + 1] = literal
          end
        end
        if #blocks > 0 then
          if not groups[header] then
            groups[header] = {}
            order[#order + 1] = header
          end
          local g = groups[header]
          g[#g + 1] = { name = sym.name, line = line_of(sym), literals = blocks }
        end
      end
    end
  end
  for _, header in ipairs(order) do
    local syms = groups[header]
    table.sort(syms, function(a, b) return a.line < b.line end)
    local seen = {}
    local parts = { "#include \"" .. header .. "\"\n#include <cassert>\n" }
    local calls = {}
    for _, sym in ipairs(syms) do
      local fn, def = symbol_function(sym.name, sym.literals, seen)
      parts[#parts + 1] = def
      calls[#calls + 1] = "    " .. fn .. "();"
    end
    parts[#parts + 1] = "int main()\n{\n" .. table.concat(calls, "\n") .. "\n}\n"
    ctx.output.write(out_dir .. "/" .. stem_of(header) .. ".cpp", table.concat(parts, "\n"))
  end
end)

--- The code-block literals in a symbol's documentation, in order. A `@code`
--- example, whether written inline or imported, is a `code` block in the
--- symbol's `document`.
--- @param sym table The symbol to read.
--- @return string[] The code literals.
function code_blocks(sym)
  local out = {}
  if sym.doc then
    for _, b in ipairs(sym.doc.document or {}) do
      if b.kind == "code" then
        out[#out + 1] = b.literal
      end
    end
  end
  return out
end

--- The header a symbol lives in, as a short path. A defined entity carries a
--- definition location; one that is only declared (like these functions) has
--- an empty `defLoc`, so fall back to the first declaration in `loc`. Symbols
--- are grouped by this so each header maps to one input and one output example
--- file.
--- @param sym table The symbol whose header to read.
--- @return string|nil header The header path, or nil if unknown.
function header_of(sym)
  local info = sym.loc
  if not info then
    return nil
  end
  local def = info.defLoc
  if def and def.shortPath and #def.shortPath > 0 then
    return def.shortPath
  end
  for _, decl in ipairs(info.loc or {}) do
    if decl.shortPath and #decl.shortPath > 0 then
      return decl.shortPath
    end
  end
  return nil
end

--- The file stem of a header path: `arithmetic.hpp` -> `arithmetic`, so a
--- header maps to a `<stem>.cpp` example file next to it in each directory.
--- @param header string The header path.
--- @return string stem The stem, extension and directory removed.
function stem_of(header)
  local base = header:match("([^/]+)$") or header
  return base:gsub("%.[^.]*$", "")
end

--- Read a file and return its contents, or nil when the file does not exist.
--- @param path string The file to read.
--- @return string|nil content The contents, or nil if absent.
function read_file(path)
  local f = io.open(path, "r")
  if not f then
    return nil
  end
  local content = f:read("*a")
  f:close()
  return content
end

--- Extract every snippet tagged `name` from an example file's text, in order.
--- A snippet is the dedented lines between `//[<name>` and the next `//]`, so a
--- symbol documented by several regions collects several snippets.
--- @param text string The example file's contents.
--- @param name string The snippet tag, a symbol name.
--- @return string[] The snippets, one per tagged region.
function extract_snippets(text, name)
  local snippets = {}
  local body = nil
  for line in (text .. "\n"):gmatch("(.-)\n") do
    if body then
      if line:match("^%s*//%]%s*$") then
        snippets[#snippets + 1] = table.concat(dedent(body), "\n")
        body = nil
      else
        body[#body + 1] = line
      end
    elseif line:match("^%s*//%[" .. name .. "%s*$") then
      body = {}
    end
  end
  return snippets
end

--- Strip the common leading indentation from a list of lines, so an extracted
--- snippet reads as top-level code rather than carrying the indentation it had
--- inside its enclosing function.
--- @param body_lines string[] The snippet's lines.
--- @return string[] The lines with the shared indent removed.
function dedent(body_lines)
  local indent = nil
  for _, line in ipairs(body_lines) do
    if line:match("%S") then
      local lead = #line - #(line:gsub("^%s+", ""))
      if indent == nil or lead < indent then
        indent = lead
      end
    end
  end
  if not indent or indent == 0 then
    return body_lines
  end
  local out = {}
  for _, line in ipairs(body_lines) do
    out[#out + 1] = line:sub(indent + 1)
  end
  return out
end

--- Deep-copy a documentation value into plain Lua tables. A block read back
--- from the corpus is a live proxy that the engine will not accept as a fresh
--- value, so rebuilding a symbol's `document` means converting the blocks it
--- already has, dropping the internal `$meta` key and any methods.
--- @param value any A DOM value: a primitive, an array, or an object.
--- @return any A plain copy safe to assign back to `document`.
function to_plain(value)
  local t = type(value)
  if t ~= "table" and t ~= "userdata" then
    return value
  end
  if value[1] ~= nil then
    local arr = {}
    local i = 1
    while value[i] ~= nil do
      arr[i] = to_plain(value[i])
      i = i + 1
    end
    return arr
  end
  local obj = {}
  for k, v in pairs(value) do
    if type(k) == "string" and k ~= "$meta" and type(v) ~= "function" then
      obj[k] = to_plain(v)
    end
  end
  return obj
end

--- Record that a snippet was imported from an example file.
--- @param literal string The imported snippet.
--- @param header string The header its symbol belongs to.
function mark_imported(literal, header)
  imported[fingerprint(literal, header)] = true
end

--- A fingerprint for a snippet: the header it belongs to, its byte size, and a
--- hash of its text, joined into one key. Import and export both know a symbol's
--- header and its snippet text, so both compute the same key for the same
--- snippet, without either side storing the text itself.
--- @param literal string The snippet code.
--- @param header string The header the snippet's symbol belongs to.
--- @return string key The fingerprint.
function fingerprint(literal, header)
  local hash = 5381
  for i = 1, #literal do
    hash = (hash * 33 + literal:byte(i)) % 2147483648
  end
  return header .. "|" .. #literal .. "|" .. hash
end

--- Whether a snippet was imported, and so should not be exported again: an
--- imported example is already a tested file, so writing it back out would only
--- test it twice.
--- @param literal string The snippet.
--- @param header string The header its symbol belongs to.
--- @return boolean True when the snippet came from an example file.
function was_imported(literal, header)
  return imported[fingerprint(literal, header)] == true
end

--- The source line a symbol appears on, used to order exported snippets the way
--- they read in the header. Falls back to the first declaration, like
--- `header_of`, and to 0 when unknown.
--- @param sym table The symbol whose line to read.
--- @return number line The 1-based line number, or 0 if unknown.
function line_of(sym)
  local info = sym.loc
  if not info then
    return 0
  end
  if info.defLoc and (info.defLoc.lineNumber or 0) > 0 then
    return info.defLoc.lineNumber
  end
  for _, decl in ipairs(info.loc or {}) do
    if (decl.lineNumber or 0) > 0 then
      return decl.lineNumber
    end
  end
  return 0
end

--- The test function for a symbol: `void <name>_snippet() { ... }`. A single
--- snippet is the body; several are each placed in their own block scope so
--- their declarations do not collide.
--- @param sym_name string The symbol name, used for the function name.
--- @param literals string[] The symbol's snippets, in order.
--- @param seen table<string, boolean> Function names already used in this file.
--- @return string fn The function name.
--- @return string def The function definition.
function symbol_function(sym_name, literals, seen)
  local fn = unique_name(sym_name .. "_snippet", seen)
  local body
  if #literals == 1 then
    body = indent_block(literals[1], 4)
  else
    local scopes = {}
    for _, literal in ipairs(literals) do
      scopes[#scopes + 1] = "    {\n" .. indent_block(literal, 8) .. "\n    }"
    end
    body = table.concat(scopes, "\n")
  end
  return fn, "void " .. fn .. "()\n{\n" .. body .. "\n}\n"
end

--- A function name derived from `base`, made unique within `seen` by appending
--- a counter, so two symbols with the same name never define the same function.
--- @param base string The preferred name.
--- @param seen table<string, boolean> Names already used in this file.
--- @return string The unique name.
function unique_name(base, seen)
  local name = base
  local n = 1
  while seen[name] do
    n = n + 1
    name = base .. "_" .. n
  end
  seen[name] = true
  return name
end

--- Indent every non-empty line of a snippet by `n` spaces.
--- @param literal string The snippet code.
--- @param n number The number of leading spaces.
--- @return string The indented snippet.
function indent_block(literal, n)
  local pad = string.rep(" ", n)
  local out = {}
  for line in (literal .. "\n"):gmatch("(.-)\n") do
    out[#out + 1] = (#line > 0) and (pad .. line) or ""
  end
  return table.concat(out, "\n")
end
  • arithmetic.hpp

  • text.hpp

#ifndef APP_ARITHMETIC_HPP
#define APP_ARITHMETIC_HPP

namespace app {

/** Add two integers.

    Returns the arithmetic sum of the two arguments. The example lives in this
    comment and is exported to a file the build compiles, so it never falls out
    of date.

    @par Example

    @code
    int s = app::add(2, 3);
    assert(s == 5);
    @endcode
*/
int add(int a, int b);

/** Multiply two integers.

    Returns the product of the two arguments. Its examples are maintained in a
    separate file.
*/
int multiply(int a, int b);

/** Subtract the second integer from the first.

    Returns `a - b`. Its example is maintained in a separate file.
*/
int subtract(int a, int b);

} // namespace app

#endif
#ifndef APP_TEXT_HPP
#define APP_TEXT_HPP

namespace app {

/** The number of characters before the terminating null.

    Counts the bytes up to, but not including, the null terminator.

    @par Example

    @code
    assert(app::length("abc") == 3);
    @endcode
*/
int length(char const* s);

/** True when the string has no characters.

    Only the empty string is blank; any other string, including whitespace, is
    not.

    @par Examples

    @code
    assert(app::is_empty(""));
    @endcode

    @code
    assert(!app::is_empty("abc"));
    @endcode
*/
bool is_empty(char const* s);

} // namespace app

#endif
example-code/arithmetic.cpp
#include "arithmetic.hpp"
#include <cassert>

void examples()
{
    //[multiply
    int p = app::multiply(4, 5);
    assert(p == 20);
    //]

    //[multiply
    int q = app::multiply(-2, 3);
    assert(q == -6);
    //]

    //[subtract
    int d = app::subtract(9, 4);
    assert(d == 5);
    //]
}
mrdocs.yml
addons-supplemental:
  - addons
generator:
  - adoc
  - snippets
multipage: false
show-namespaces: false
warn-if-undocumented: false
source-root: include
input:
  - include
transform-options:
  snippets:
    source: example-code
generator-options:
  snippets:
    output: exported

In the input, add keeps its example inline in the header comment, while multiply and subtract import theirs from a separate example file. The transform reads its inputs from transform-options.snippets.source, and the generator writes the inline snippets to generator-options.snippets.output. The rendered page then shows every example regardless of where it came from:

Preview
app::add

Add two integers.

Synopsis

Declared in <arithmetic.hpp>

int
add(
    int a,
    int b);
Description

Returns the arithmetic sum of the two arguments. The example lives in this comment and is exported to a file the build compiles, so it never falls out of date.

Example
int s = app::add(2, 3);
assert(s == 5);
app::is_empty

True when the string has no characters.

Synopsis

Declared in <text.hpp>

bool
is_empty(char const* s);
Description

Only the empty string is blank; any other string, including whitespace, is not.

Examples
assert(app::is_empty(""));
assert(!app::is_empty("abc"));
app::length

The number of characters before the terminating null.

Synopsis

Declared in <text.hpp>

int
length(char const* s);
Description

Counts the bytes up to, but not including, the null terminator.

Example
assert(app::length("abc") == 3);
app::multiply

Multiply two integers.

Synopsis

Declared in <arithmetic.hpp>

int
multiply(
    int a,
    int b);
Description

Returns the product of the two arguments. Its examples are maintained in a separate file.

Examples
int p = app::multiply(4, 5);
assert(p == 20);
int q = app::multiply(-2, 3);
assert(q == -6);
app::subtract

Subtract the second integer from the first.

Synopsis

Declared in <arithmetic.hpp>

int
subtract(
    int a,
    int b);
Description

Returns a ‐ b. Its example is maintained in a separate file.

Example
int d = app::subtract(9, 4);
assert(d == 5);

Export runs comments into files, and writes complete translation units, one per header. Each file includes its header and a small prelude, gives every symbol a function that runs its snippets (each in its own scope when there is more than one), and ends with a main that calls them all, so the file both compiles and runs as a test. It exports only the examples written inline in the headers; ones the transform imported are left out, since they already live in a file the build compiles. exported/arithmetic.cpp therefore holds only add, not the imported multiply and subtract:

Exported test files
  • exported/arithmetic.cpp

  • exported/text.cpp

#include "arithmetic.hpp"
#include <cassert>

void add_snippet()
{
    int s = app::add(2, 3);
    assert(s == 5);
}

int main()
{
    add_snippet();
}
#include "text.hpp"
#include <cassert>

void length_snippet()
{
    assert(app::length("abc") == 3);
}

void is_empty_snippet()
{
    {
        assert(app::is_empty(""));
    }
    {
        assert(!app::is_empty("abc"));
    }
}

int main()
{
    length_snippet();
    is_empty_snippet();
}

Knowing which examples to skip is where the two halves of the extension meet. The transform function records a fingerprint of each snippet, and the generator reads that same table to skip them.

This extension can also compose with the ghostwriter extension. A typical problem with technical writers is that their examples are only tested locally, so they can drift and break as the API moves. This extension would ensure these import snippets are alo tested as inline scripts.