Preprocessors - mdBook Documentation
Description: Create book from markdown files. Like Gitbook but implemented in Rust

## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Preprocessors

A preprocessor is simply a bit of code which gets run immediately after the book is loaded and before it gets rendered, allowing you to update and mutate the book. Possible use cases are:

- Creating custom helpers like `{{#include /path/to/file.md}}`
- Substituting in latex-style expressions (`$$ \frac{1}{3} $$`) with their mathjax equivalents

See Configuring Preprocessors for more information about using preprocessors.

## Hooking into MDBook

MDBook uses a fairly simple mechanism for discovering third party plugins. A new table is added to `book.toml` (e.g. `[preprocessor.foo]` for the `foo` preprocessor) and then `mdbook` will try to invoke the `mdbook-foo` program as part of the build process.

Once the preprocessor has been defined and the build process starts, mdBook executes the command defined in the `preprocessor.foo.command` key twice. The first time it runs the preprocessor to determine if it supports the given renderer. mdBook passes two arguments to the process: the first argument is the string `supports` and the second argument is the renderer name. The preprocessor should exit with a status code 0 if it supports the given renderer, or return a non-zero exit code if it does not.

If the preprocessor supports the renderer, then mdbook runs it a second time, passing JSON data into stdin. The JSON consists of an array of `[context, book]` where `context` is the serialized object `PreprocessorContext` and `book` is a `Book` object containing the content of the book.

The preprocessor should return the JSON format of the `Book` object to stdout, with any modifications it wishes to perform.

The easiest way to get started is by creating your own implementation of the `Preprocessor` trait (e.g. in `lib.rs`) and then creating a shell binary which translates inputs to the correct `Preprocessor` method. For convenience, there is an example no-op preprocessor in the `examples/` directory which can easily be adapted for other preprocessors.

Example no-op preprocessor

    // nop-preprocessors.rs

    //! A basic example of a preprocessor that does nothing.

    use crate::nop_lib::Nop;
    use clap::{Arg, ArgMatches, Command};
    use mdbook_preprocessor::book::Book;
    use mdbook_preprocessor::errors::Result;
    use mdbook_preprocessor::{Preprocessor, PreprocessorContext};
    use semver::{Version, VersionReq};
    use std::io;
    use std::process;

    fn make_app() -> Command {
        Command::new("nop-preprocessor")
            .about("A mdbook preprocessor which does precisely nothing")
            .subcommand(
                Command::new("supports")
                    .arg(Arg::new("renderer").required(true))
                    .about("Check whether a renderer is supported by this preprocessor"),
            )
    }

    fn main() {
        let matches = make_app().get_matches();

        // Users will want to construct their own preprocessor here
        let preprocessor = Nop::new();

        if let Some(sub_args) = matches.subcommand_matches("supports") {
            handle_supports(&preprocessor, sub_args);
        } else if let Err(e) = handle_preprocessing(&preprocessor) {
            eprintln!("{e:?}");
            process::exit(1);
        }
    }

    fn handle_preprocessing(pre: &dyn Preprocessor) -> Result<()> {
        let (ctx, book) = mdbook_preprocessor::parse_input(io::stdin())?;

        let book_version = Version::parse(&ctx.mdbook_version)?;
        let version_req = VersionReq::parse(mdbook_preprocessor::MDBOOK_VERSION)?;

        if !version_req.matches(&book_version) {
            eprintln!(
                "Warning: The {} plugin was built against version {} of mdbook, \
                 but we're being called from version {}",
                pre.name(),
                mdbook_preprocessor::MDBOOK_VERSION,
                ctx.mdbook_version
            );
        }

        let processed_book = pre.run(&ctx, book)?;
        serde_json::to_writer(io::stdout(), &processed_book)?;

        Ok(())
    }

    fn handle_supports(pre: &dyn Preprocessor, sub_args: &ArgMatches) -> ! {
        let renderer = sub_args
            .get_one::<String>("renderer")
            .expect("Required argument");
        let supported = pre.supports_renderer(renderer).unwrap();

        // Signal whether the renderer is supported by exiting with 1 or 0.
        if supported {
            process::exit(0);
        } else {
            process::exit(1);
        }
    }

    /// The actual implementation of the `Nop` preprocessor. This would usually go
    /// in your main `lib.rs` file.
    #[allow(unreachable_pub, reason = "wouldn't be a problem in a proper lib.rs")]
    mod nop_lib {
        use super::*;

        /// A no-op preprocessor.
        pub struct Nop;

        impl Nop {
            pub fn new() -> Nop {
                Nop
            }
        }

        impl Preprocessor for Nop {
            fn name(&self) -> &str {
                "nop-preprocessor"
            }

            fn run(&self, ctx: &PreprocessorContext, book: Book) -> Result<Book> {
                // In testing we want to tell the preprocessor to blow up by setting a
                // particular config value
                match ctx
                    .config
                    .get::<bool>("preprocessor.nop-preprocessor.blow-up")
                {
                    Ok(Some(true)) => anyhow::bail!("Boom!!1!"),
                    Ok(_) => {}
                    Err(e) => anyhow::bail!("expect bool for blow-up: {e}"),
                }

                // we *are* a no-op preprocessor after all
                Ok(book)
            }

            fn supports_renderer(&self, renderer: &str) -> Result<bool> {
                Ok(renderer != "not-supported")
            }
        }

        #[cfg(test)]
        mod test {
            use super::*;

            #[test]
            fn nop_preprocessor_run() {
                let input_json = r##"[
                    {
                        "root": "/path/to/book",
                        "config": {
                            "book": {
                                "authors": ["AUTHOR"],
                                "language": "en",
                                "src": "src",
                                "title": "TITLE"
                            },
                            "preprocessor": {
                                "nop": {}
                            }
                        },
                        "renderer": "html",
                        "mdbook_version": "0.4.21"
                    },
                    {
                        "items": [
                            {
                                "Chapter": {
                                    "name": "Chapter 1",
                                    "content": "# Chapter 1\n",
                                    "number": [1],
                                    "sub_items": [],
                                    "path": "chapter_1.md",
                                    "source_path": "chapter_1.md",
                                    "parent_names": []
                                }
                            }
                        ]
                    }
                ]"##;
                let input_json = input_json.as_bytes();

                let (ctx, book) = mdbook_preprocessor::parse_input(input_json).unwrap();
                let expected_book = book.clone();
                let result = Nop::new().run(&ctx, book);
                assert!(result.is_ok());

                // The nop-preprocessor should not have made any changes to the book content.
                let actual_book = result.unwrap();
                assert_eq!(actual_book, expected_book);
            }
        }
    }

## Hints for implementing a preprocessor

By pulling in `mdbook-preprocessor` as a library, preprocessors can have access to the existing infrastructure for dealing with books.

For example, a custom preprocessor could use the `parse_input()` function to deserialize the JSON written to `stdin`. Then each chapter of the `Book` can be mutated in-place via `Book::for_each_mut()`, and then written to `stdout` with the `serde_json` crate.

Chapters can be accessed either directly (by recursively iterating over chapters) or via the `Book::for_each_mut()` convenience method.

The `chapter.content` is just a string which happens to be markdown. While it’s entirely possible to use regular expressions or do a manual find & replace, you’ll probably want to process the input into something more computer-friendly. The `mdbook-markdown` crate exposes the `pulldown-cmark` crate used by mdBook to parse Markdown. The `pulldown-cmark-to-cmark` crate can be used to translate events back into markdown text.

The following code block shows how to remove all emphasis from markdown, without accidentally breaking the document.

    //! This is a demonstration of an mdBook preprocessor which parses markdown
    //! and removes any instances of emphasis.

    use mdbook_preprocessor::book::{Book, Chapter};
    use mdbook_preprocessor::errors::Result;
    use mdbook_preprocessor::{Preprocessor, PreprocessorContext};
    use pulldown_cmark::{Event, Parser, Tag, TagEnd};
    use std::io;

    fn main() {
        let mut args = std::env::args().skip(1);
        match args.next().as_deref() {
            Some("supports") => {
                // Supports all renderers.
                return;
            }
            Some(arg) => {
                eprintln!("unknown argument: {arg}");
                std::process::exit(1);
            }
            None => {}
        }

        if let Err(e) = handle_preprocessing() {
            eprintln!("{e}");
            std::process::exit(1);
        }
    }

    struct RemoveEmphasis;

    impl Preprocessor for RemoveEmphasis {
        fn name(&self) -> &str {
            "remove-emphasis"
        }

        fn run(&self, _ctx: &PreprocessorContext, mut book: Book) -> Result<Book> {
            let mut total = 0;
            book.for_each_chapter_mut(|ch| match remove_emphasis(&mut total, ch) {
                Ok(s) => ch.content = s,
                Err(e) => eprintln!("failed to process chapter: {e:?}"),
            });
            eprintln!("removed {total} emphasis");
            Ok(book)
        }
    }

    fn remove_emphasis(num_removed_items: &mut usize, chapter: &mut Chapter) -> Result<String> {
        let mut buf = String::with_capacity(chapter.content.len());

        let events = Parser::new(&chapter.content).filter(|e| match e {
            Event::Start(Tag::Emphasis) | Event::Start(Tag::Strong) => {
                *num_removed_items += 1;
                false
            }
            Event::End(TagEnd::Emphasis) | Event::End(TagEnd::Strong) => false,
            _ => true,
        });

        Ok(pulldown_cmark_to_cmark::cmark(events, &mut buf).map(|_| buf)?)
    }

    pub fn handle_preprocessing() -> Result<()> {
        let pre = RemoveEmphasis;
        let (ctx, book) = mdbook_preprocessor::parse_input(io::stdin())?;

        let processed_book = pre.run(&ctx, book)?;
        serde_json::to_writer(io::stdout(), &processed_book)?;

        Ok(())
    }

Take a look at the full example source for more details.

## Implementing a preprocessor with a different language

The fact that mdBook utilizes stdin and stdout to communicate with the preprocessors makes it easy to implement them in a language other than Rust. The following code shows how to implement a simple preprocessor in Python, which will modify the content of the first chapter. The example below follows the configuration shown above with `preprocessor.foo.command` actually pointing to a Python script.

    import json
    import sys

    if __name__ == '__main__':
        if len(sys.argv) > 1: # we check if we received any argument
            if sys.argv[1] == "supports":
                # then we are good to return an exit status code of 0, since the other argument will just be the renderer's name
                sys.exit(0)

        # load both the context and the book representations from stdin
        context, book = json.load(sys.stdin)
        # and now, we can just modify the content of the first chapter
        book['items'][0]['Chapter']['content'] = '# Hello'
        # we are done with the book's modification, we can just print it to stdout,
        print(json.dumps(book))