Sphinx and Markdown around the world in 2021

2021-12-20T00:00:00Z

Read the Docs has been committed to improving the accessibility and user experience of Sphinx since the start, and that includes the markup language in which the documentation is written. Years ago, after carefully listening to users, we announced recommonmark to help bridging the immense popularity of Markdown with the powerful capabilities of Sphinx. (It is now deprecated in favor of MyST - keep reading to know more.)

It is no surprise that Markdown is in such demand: thanks in large part to the huge popularity of GitHub, Markdown is nowadays the most widely used markup language in open-source projects, ahead of reStructuredText, the second in the list, by an order of magnitude.

This year, thanks to the CZI Grant we received last year and the effort of other communities, especially the Executable Book Project, we have devoted lots of resources to help consolidate Markdown as a compelling markup language for Sphinx.

Writing Markdown in Sphinx

To that end, we recognized the potential of MyST as a successor of recommonmark, and deprecated the latter in favor of MyST-Parser. To start writing Markdown in Sphinx, you need to follow three steps:

Install MyST-Parser: pip install myst-parser
Add myst_parser to the list of Sphinx extensions:

extensions = [
    # ...other extensions
    "myst_parser",
]

Start writing content in *.md files!

You can find more information in the MyST-Parser documentation.

Talks and workshops at conferences

Luckily the migration to MyST-Parser was trivial for most projects, but we still wanted to spread the word about the best practices for creating Sphinx projects on Read the Docs using MyST Markdown. We set the goal of presenting at every large scientific Python event out there, and all our proposals were accepted [1]! Throughout 2021, we were present in several events:

We conducted a networking session and a sprint about documentation at SciPy US, and in that event we were kindly invited to the sktime documentation sprint.
We delivered a 90 minutes tutorial called “Document your scientific project with Markdown, Sphinx, and Read the Docs” at PyData Global, and we repeated a 120 minute version in Spanish language at the SciPy Latin America.
And we also gave shorter talks at more broadly focused events: PyCon Latam and PyCon Spain.

We have generated workshop material that everybody is welcome to use:

A main repository containing some introductory slides about Sphinx, and detailed notes of every step of the tutorial.
A supporting repository with a tiny Python library that gets used during the tutorial, inspired by the one used in the official Sphinx tutorial we authored with the help of the Sphinx community.

And finally, we have also developed an interactive web application to translate small bits of reStructuredText to MyST Markdown.

[1]	We also have submitted a proposal for SciPy India, which was postponed to January 2022. EuroSciPy and SciPy Japan didn’t happen in 2021.

Future plans

We are excited to see that the Executable Books team has released myst-docutils, which should help docutils parse CommonMark directly and possibly pave the way for Sphinx to parse Markdown without the need of third party extensions. Hopefully that will also make it easier for Sphinx to showcase Markdown content in its documentation more prominently, and users will be able to naturally pick the markup language that best suits their needs, be it reST or MyST.

—

Considering using Read the Docs for your next Sphinx, Jupyter Book, or MkDocs project? Check out our documentation to get started!

Read the Docs & Sphinx now support Commonmark

2015-10-16T00:00:00Z

Read the Docs is built on top of Sphinx, which has always relied on reStructuredText as an input mechanism. We have long heard from folks that they want to write documentation in Markdown, as well as RST.

Today we are announcing that this is now possible! With the standardization of Markdown into Commonmark, we have the ability to support a markup language with a proper spec. recommonmark is the bridge that allows Commonmark to be used inside Sphinx. This allows you to use both RST and Commonmark inside of your Sphinx project.

Get started

We have documented how to get started with Commonmark inside of Sphinx. You simply:

pip install recommonmark

Then in your conf.py:

from recommonmark.parser import CommonMarkParser

source_parsers = {'.md': CommonMarkParser}

source_suffix = ['.rst', '.md']

This allows you to write both .md and .rst files inside of the same project.

Example Project

You can see this in action in my sphinx-markdown-test repo on GitHub. You can also see a rendered version hosted on Read the Docs.

How it works

The way this project works is that it allows Commonmark files to be parsed into docutils nodes. That means that everything downstream of the parser works the same as if the content came from RST. This allows the content to slot into Sphinx and other RST based tools without much effort.

There are some caveats where certain Sphinx & docutils directives depend on the internal structure of RST. We would like to eventually build a bridge from Commonmark to the Sphinx directives, which would give a lot of the power of Sphinx to the Commonmark content.

We are looking at ways to handle this in the future, and hope to eventually bring the full power of RST and Sphinx into the Commonmark ecosystem.

Limitations

It should be noted that Commonmark doesn’t support a lot of the concepts that RST lets you represent. In particular, there is no standardized way in Commonmark to represent inline or block levels constructs. So things like the toctree directive and :ref: markup don’t have an analog.

There is a proposed draft to the Commonmark spec to allow a similar syntax. We are investigating adding support to this in recommonmark, but we don’t want to implement a standard prematurely.

Intended Usage

We think that Sphinx’s power to reference code and other programming concepts is quite powerful. However, all content doesn’t need this ability. When you’re writing content that just needs to have basic text formatting and links, Commonmark is a great option for this.

We imagine that API reference documentation will continue to be authored in RST for quite some time. Also index pages and other reference heavy content will continue to be RST. FAQ’s, blog posts, and other less reference heavy content is a great candidate for writing in Commonmark.

Feedback

This is a new release, so there are likely some missing pieces to our integration. Go ahead and try it out. Please file features ideas and bug reports on the recommonmark bug tracker.

Read the Docs Blog - Posts tagged markdown