Read the Docs Blog

Read the Docs newsletter - March 2022

2022-03-10T00:00:00Z

War in Ukraine and what it means for Read the Docs

2022-03-07T00:00:00Z

With news surrounding the invasion of Ukraine evolving rapidly, we felt it was necessary to provide an update to our users and customers.

At Read the Docs, we are outraged and saddened by the invasion of Ukraine and we condemn this act of violence as wrong and unlawful. We are monitoring the situation in Europe and how it relates to our employees, customers, and our services to the open source world.

Read the Docs is a very small company, with only seven employees, and fortunately none of them are currently in harm’s way. Our impact on the open source ecosystem, however, is bigger than our size, and we have users all over the world, including in Ukraine and Russia.

Impact on our services

We are continuing to monitor news that could potentially affect our users, including sanctions applied to the region.

Read the Docs is currently partially blocked in Russia by the Russian government. While we are working to resolve this, it does not appear that this block is related to war in Ukraine.

Other than this government block and potential effects from financial sanctions, our services are not impacted. Read the Docs is hosted and incorporated in the United States, and so any further changes to US sanctions would apply to our services.

What we are doing and what you can do

As the situation develops, we will be transparent about any actions we take. We take our role of helping the open source community with documentation seriously and we aim to be a platform for all developers.

Over the coming days, our advertising arm EthicalAds will be running a series of community ads aimed at raising money for the United Nations Crisis Relief Fund and other missions aimed at humanitarian aid for Ukraine. If you can donate to help provide humanitarian aid, please do.

We are hoping that peace is restored as quickly as possible.

Deprecation of the git:// protocol on GitHub

2022-03-01T00:00:00Z

Last year, GitHub announced the deprecation of the unsecured Git protocol due to security reasons. This change will be made permanent on March 15, 2022.

At Read the Docs we found around 900 projects using a Git protocol URL (git://github.com/user/project) to clone their projects. To save time for our users, we have migrated those to use the HTTPS cloning URL instead (https://github.com/user/project).

But there are other places where projects may be using a Git protocol URL, like in submodules or dependencies. You’ll need to migrate those to use a supported protocol in order for your builds to keep working after March 15, 2022. In most cases, this means changing URLs that start with git:// to start with https:// instead. If you have doubts, please check the documentation of the tools you are using, for example:

Note that this applies only to repositories hosted on GitHub, repositories using providers that support the Git protocol won’t be affected.

Read the Docs tries to keep users informed about deprecations and breaking changes that may impact projects. To receive future updates like this, subscribe to our newsletter.

Read the Docs newsletter - February 2022

2022-02-08T00:00:00Z

Welcome to the latest edition of our monthly newsletter, where we share the most relevant updates around Read the Docs, offer a summary of new features we shipped during the previous month, and share what we’ll be focusing on in the near future.

Company updates

We have mostly finished migrating Read the Docs for Business users to Cloudflare for SSL. There are lots of interesting features this will enable, so stay tuned for updates there.
We’re sad to announce that Juan Luis has moved on from Read the Docs as our developer advocate. The work he did was vital towards getting our CZI grant mostly finished, and we thank him for his time spent bettering the RTD, Sphinx, and docs community.
On a related note, we’re going to be hiring again soon to fill another position. It will be a bit different and likely a product-focused Python development position. If you’re interested, please let us know.

New features

In January we continued to work on refactors and internal changes. Among the major user-facing changes:

We fixed a bug in Bitbucket that didn’t allow us to properly sync user permissions. This resulted in a few support requests, but has now been resolved.
We improved our ability to mark projects as non-spam, so that we can validate a project isn’t spam and then make sure it doesn’t get flagged by our automated system.

You can always see the latest changes to our platforms in our Read the Docs Changelog.

Upcoming features

Cancelling a build is a long requested feature, and we’re getting close to implementing it.
We’re looking at tracking 404 pages in our project analytics, so that projects can easily fix up missing or outdated page links.
We are hoping to launch our revamped landing pages this month, which will give our front page a much needed refresh.
We are working to define a policy for canonical docs and cloned versions, so that we can more easily remove outdated docs for projects.
We’re working to investigate supporting a CDN on Read the Docs for Business, which will be an exciting new feature for our users there.
We’re looking at how to support multiple readthedocs.yml files in a single git repo.

Possible issues

We continue to see a good pace of development on the Sphinx project, with them adding support for docutils 0.18, and removing jQuery in a major refactor of their Javascript. We’re working to ensure that our theme and other parts of the ecosystem don’t break with these changes, and trying to be procative to address any possible issues. That said, there will likely still be some issues that sneak through with the release of Sphinx 4.5 (docutils upgrade) and Sphinx 5.0 (JS refactor).

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

Sphinx 4.4 release and other ecosystem news

2022-01-26T00:00:00Z

In this post we spread the word about the most relevant news of the Sphinx ecosystem of the past weeks, including Sphinx itself as well as extensions and themes developed by the community.

Sphinx 4.4 release

Sphinx 4.4 was released on January 17th with numerous changes, including:

Better control of intersphinx references with the :external: role

One of the power features of Sphinx is the ability to include cross references across projects, using intersphinx. By default, Sphinx transparently attempts to locate objects in external projects if they are not found in the current one, which is useful but also can become confusing in some cases. The new external role, coupled with the intersphinx_disabled_reftypes configuration introduced in Sphinx 4.3, enables documentation writers to control when intersphinx references should be used.

You can use the new :external: role as follows:

External reference: :external:py:class:`zipfile.ZipFile`.
External reference with constrained domain lookup: :external+python:py:class:`zipfile.ZipFile`.

See more information in the official documentation.

New asynchronous methods to load JavaScript files

The method sphinx.application.Sphinx.add_js_file() now accepts a new loading_method parameter that can either be "async" or "defer", which results in the script being loaded with the async or defer HTML attributes respectively.

Proper error messages when autosummary does not find a package

Sphinx ships the autosummary extension to automatically generate API pages of a Python library. However, autosummary used to raise misleading error messages if the target library failed to import, which resulted in user frustration. This problem has now been fixed and autosummary properly points out the reason of the ImportError.

You can read the complete Sphinx 4.4.0 release notes online.

Using Sphinx 4.4 on Read the Docs

Sphinx 4.4 is supported on Read the Docs. By default, RTD will install the latest version for you, as described in our documentation. If you are pinning your dependencies to improve the reproducibility of your builds, you can change the pinned version to sphinx==4.4.0 in either your requirements.txt or environment.yml.

Sphinx themes and extensions

Several other extensions and themes saw new releases recently, including:

pydata-sphinx-theme 0.8

pydata-sphinx-theme is a Sphinx theme based on Bootstrap CSS developed by the PyData community, The Read the Docs team collaborated with the developers to make some of the new features of 0.8 behave correctly in our platform, in particular the custom version switcher. As Chris Holdgraf summarizes in this Twitter thread, the new version brings other interesting additions, like better depth control of the left sidebar and custom SVG icons.

sphinx-codeautolink 0.9

sphinx-codeautolink is a Sphinx extension to automatically include links inside code blocks. The 0.9 version now links Python builtin objects if intersphinx is properly configured and has several logging improvements.

Demo of sphinx-codeautolink

sphinxcontrib-constdata 1.0

sphinxcontrib-constadata is a new Sphinx extension that allows documentation writers to easily load data stored in CSV, JSON, and YAML files. For example, it can be used to load UI labels (button labels, menu selection labels) from external files instead of hardcoding them in the documentation for better maintenance, for example:

Choose menu item :constdata:label:`menu.yaml?FileSaveAs`.

Upcoming

The Executable Books Project team is working on several exciting things around MyST, including the upcoming MyST-Parser 0.17 release and direct integration with Jupyter notebooks.

Preview of MyST integrated in Jupyter notebooks.

We are excited about seeing new and old Sphinx extensions being developed by the community, and we thank the Sphinx maintainers for their excellent work.

Considering using Read the Docs for your next Sphinx? Check out our documentation to get started!

Read the Docs newsletter - January 2022

2022-01-12T00:00:00Z

Company highlights

We are now managing custom domains for our corporate users using Cloudflare SSL for SaaS, which will remove the manual work that was needed on our side and make the process of setting up a custom domain almost instantaneous. It also will allow us to offer a CDN much easier in the future.
We improved our spam classification system, and started blocking the dashboard on spammy projects. We are happy to report that the traffic to spammy projects has already reduced significantly.

New features

December and January have been slow months, and we mainly worked on internal changes and documentation improvements. Among the major user-facing changes:

We changed the Google Analytics from a persistent 30 day cookie to a session cookie that is deleted when the browser is closed. This will result in less data collected from users.
We split the developer documentation from the user documentation to avoid confusion and made our Read the Docs for Business docs more visible.
We expanded our user documentation with more examples on how to use MyST Markdown and a guide to migrate from reST to MyST (more about our committment to support MyST in our blog).

Thanks to our external contributor Çağatay Yiğit Şahin.

You can always see the latest changes to our platforms in our Read the Docs Changelog.

Upcoming features

Ana will continue working on the new landing pages, polishing the copywriting along with Juan Luis and translating the mockups to Pelican templates.
Anthony will continue working on frontend scaffolding and finance bits.
Eric will continue to focus on business and sales efforts, and performing code review.
Juan Luis will work with Ana in the new landing pages, and with Manuel in a new process to migrate users from legacy or deprecated configurations.
Manuel will wrap up the migration to Django 3, improve the queryability of the project configuration in our database, and refactor our task queue to improve its robustness.
Santos will add support for multiple .readthedocs.yaml files per repository.

Possible issues

The Python packaging ecosystem is evolving rapidly to accomodate for modern standards that will make our life easier, and in particular setuptools has been introducing some breaking changes in recent times. We anticipated this sort of breakage for our users so we pinned the setuptools version in late November. However, a small number of projects did not have this version cap active because of a bug on our side and experienced failing builds after a new setuptools release. We quickly fixed the problem and offered a range of possible solutions to affected projects.

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

Sphinx and Markdown around the world in 2021

2021-12-20T00:00:00Z

Read the Docs has been committed to improving the accessibility and user experience of Sphinx since the start, and that includes the markup language in which the documentation is written. Years ago, after carefully listening to users, we announced recommonmark to help bridging the immense popularity of Markdown with the powerful capabilities of Sphinx. (It is now deprecated in favor of MyST - keep reading to know more.)

It is no surprise that Markdown is in such demand: thanks in large part to the huge popularity of GitHub, Markdown is nowadays the most widely used markup language in open-source projects, ahead of reStructuredText, the second in the list, by an order of magnitude.

This year, thanks to the CZI Grant we received last year and the effort of other communities, especially the Executable Book Project, we have devoted lots of resources to help consolidate Markdown as a compelling markup language for Sphinx.

Writing Markdown in Sphinx

To that end, we recognized the potential of MyST as a successor of recommonmark, and deprecated the latter in favor of MyST-Parser. To start writing Markdown in Sphinx, you need to follow three steps:

Install MyST-Parser: pip install myst-parser
Add myst_parser to the list of Sphinx extensions:

extensions = [
    # ...other extensions
    "myst_parser",
]

Start writing content in *.md files!

You can find more information in the MyST-Parser documentation.

Talks and workshops at conferences

Luckily the migration to MyST-Parser was trivial for most projects, but we still wanted to spread the word about the best practices for creating Sphinx projects on Read the Docs using MyST Markdown. We set the goal of presenting at every large scientific Python event out there, and all our proposals were accepted [1]! Throughout 2021, we were present in several events:

We conducted a networking session and a sprint about documentation at SciPy US, and in that event we were kindly invited to the sktime documentation sprint.
We delivered a 90 minutes tutorial called “Document your scientific project with Markdown, Sphinx, and Read the Docs” at PyData Global, and we repeated a 120 minute version in Spanish language at the SciPy Latin America.
And we also gave shorter talks at more broadly focused events: PyCon Latam and PyCon Spain.

We have generated workshop material that everybody is welcome to use:

A main repository containing some introductory slides about Sphinx, and detailed notes of every step of the tutorial.
A supporting repository with a tiny Python library that gets used during the tutorial, inspired by the one used in the official Sphinx tutorial we authored with the help of the Sphinx community.

And finally, we have also developed an interactive web application to translate small bits of reStructuredText to MyST Markdown.

[1]	We also have submitted a proposal for SciPy India, which was postponed to January 2022. EuroSciPy and SciPy Japan didn’t happen in 2021.

Future plans

We are excited to see that the Executable Books team has released myst-docutils, which should help docutils parse CommonMark directly and possibly pave the way for Sphinx to parse Markdown without the need of third party extensions. Hopefully that will also make it easier for Sphinx to showcase Markdown content in its documentation more prominently, and users will be able to naturally pick the markup language that best suits their needs, be it reST or MyST.

—

Considering using Read the Docs for your next Sphinx, Jupyter Book, or MkDocs project? Check out our documentation to get started!

Read the Docs newsletter - December 2021

2021-12-14T00:00:00Z

Company highlights

We successfully deployed mitigation measures against spam, and we are happy to report that the amount of abusive projects has dramatically decreased.
We started tracking usage of different parts of our application in a more systematic fashion which will enable us to make better product design decisions going forward.

New features

We revamped our support for webhooks: now they have a dedicated section in the project settings, you can customize the payload, and you can inspect the status of each webhook delivery. Interested in receiving build notifications on Slack or Discord? Now you can!

Revamped webhooks

On Read the Docs for Business, we improved our security audit logs to show information from all the organization according to its plan.

Organization audit logs

We expanded our documentation to describe how document projects with Jupyter Book and how to use Poetry for dependency management.

Thanks to our external contributor Rok Roškar.

You can always see the latest changes to our platforms in our Read the Docs Changelog.

Upcoming features

With the Christmas holidays coming up, we will have a few slow weeks ahead.

Ana will keep working on the complete redesign of our community site, which is already making good progress.
Anthony will work with Ana on the structure of our new community site and document our upcoming new user interface.
Eric will continue working on our commercial SSL provisioning and CDN with Santos.
Juan Luis will continue promoting our Embed API as well as wrap up his work on the Sphinx tutorial.
Manuel will adjust the new logging and spam fighting systems, and continue making progress on the new metrics infrastructure.
Santos will do the final bits of CDN work on our commercial site, finish moving our development documentation to a separate project, and continue unifying our commercial and community codebases.

Possible issues

Last week we accidentally banned a small number of legitimate users and they saw their projects temporarily blocked. As soon as we noticed this we apologized to the affected users and rolled back the ban, and the documentation is now serving normally.

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

Announcing Embed API v3 and sphinx-hoverxref 1.0

2021-12-07T00:00:00Z

We are thrilled to announce the availability of Read the Docs Embed API v3, along with its official client, sphinx-hoverxref 1.0. This work has been possible in part thanks to the the CZI grant we received.

The value of content embedding

As we wrote in our first blog post about sphinx-hoverxref, one of the most powerful features of Sphinx is the possibility of creating cross references to other documentation projects. However, a reader finding several links in a technical documentation might need to open several browser tabs to fully understand the context, resulting in a lot of friction in the form of context switching.

One way of mitigating the problem is to show a small preview in the form of tooltip or modal with the contents of the link. This is where the Embed API and sphinx-hoverxref come in.

sphinx-hoverxref is a Sphinx extension that addresses this problem by adding tooltips on the cross references of the HTML documentation with the content of the linked section. To do so, it inserts a small JavaScript file that calls the Read the Docs Embed API when the user hovers any link. Back in June we announced intersphinx support in sphinx-hoverxref 0.6b1, which allowed users to include modals and tooltips also for cross references to other projects.

sphinx-hoverxref displaying a tooltip including an equation

Some big projects already using it include Tweepy, a Python client for Twitter, and Scrapy, a framework for crawling websites.

Embed API v3 and sphinx-hoverxref 1.0

After some months of work, we are excited to publish v3 of our Embed API, and with it, version 1.0 of sphinx-hoverxref.

Among other things, the new versions allow you to embed content from pages hosted outside Read the Docs. For security reasons, we have an allowlist of such external domains that currently includes Python, NumPy, SciPy, SymPy, and we would like to invite the community to suggest more domains to add.

Other additions and fixes include:

Support for JS and CSS assets in Sphinx 4.1+.
Support for glossary / term and sphinxcontrib-bibtex.
Avoid showing the browser built-in tooltip for links that have the extension enabled.

Finally, version 3 of the Embed API paves the way for supporting non-Sphinx projects in the future.

Embedding content with sphinx-hoverxref

To use sphinx-hoverxref in your Read the Docs project, declare it as part of your dependencies:

requirements.txt

sphinx==4.3.1
sphinx_rtd_theme==1.0.0
sphinx-hoverxref==1.0.0

And add it to your list of Sphinx extensions:

conf.py

extensions = [
    # ...other extensions
    "hoverxref.extension",
]

To enable the extension on all :ref: of your documentation, set the hoverxref_auto_ref to True:

conf.py

hoverxref_auto_ref = True

And you will start seeing tooltips on your internal references. Apart from :ref: roles, you can also enable tooltips on:

Note

sphinx-hoverxref includes the JavaScript embed client in the HTML assets, but it is not yet available as a standalone library that can be reused with standard frontend packaging tools. If you would like to see this happening, let us know.

Calling the Embed API directly

As explained in our embedding guide, you can call the API directly from any HTTP client:

$ curl -s "https://readthedocs.org/api/v3/embed/\
> ?url=https://docs.readthedocs.io/en/latest/features.html\
> %23read-the-docs-features" | python -m json.tool
{
  "url": "https://docs.readthedocs.io/en/latest/features.html#read-the-docs-features",
  "fragment": "read-the-docs-features",
  "content": "<div class=\"section\" id=\"read-the-docs-features\">\n<h1>Read the Docs ...",
  "external": false
}

Or visually explore the query in the web interface instead.

The reference documentation includes more detail about the parameters and return values of the API.

Try it out!

We would like to invite the community to try out these features and send us feedback. With the help of our users, we will keep moving towards a more cohesive documentation ecosystem of interlinked Python projects.

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

Read the Docs ❤️ Jupyter Book

2021-11-11T00:00:00Z

We are proud to announce that now Jupyter Book projects are supported on Read the Docs!

Both Read the Docs and The Executable Book Project, the folks behind Jupyter Book, share a common passion for documentation, and we have been collaborating on various topics for some time already. For example, we started promoting MyST in favor of our recommonmark back in April this year, and we wrote a guide on using Jupyter notebook with Sphinx that benefitted a lot from their feedback.

Now we are happy to announce that Jupyter Book itself is supported on Read the Docs, after some thorough discussion about the possible implementations and thanks in large part to the Jupyter Book developers, who addressed all the minor incompatibilities that prevented this from happening.

What is Jupyter Book?

According to its own documentation,

Jupyter Book is an open source project for building beautiful, publication-quality books and documents from computational material.

Jupyter Book is the flagship product of the Executable Book Project, an international collaboration between several universities and software projects seeking to build open source tools that facilitate publishing computational narratives using the Jupyter ecosystem.

Even though Jupyter Book is much younger than Read the Docs as a project, it has become very popular in recent times among scientific software developers and educators. It enables users to write publication-quality content in Markdown thanks to MyST (a Markdown dialect compatible with Sphinx we started promoting back in April this year), use Jupyter notebooks to author content thanks to MyST-NB (featured in our Jupyter on Sphinx guide), easily add interactivity thanks to Thebe, and much more.

Why a change was needed

Read the Docs supports two documentation generation systems Sphinx and MkDocs. Adding extra systems is difficult with the current codebase, because it requires lots of effort to match all the features currently supported by the existing ones.

On the other hand, even though Jupyter Book leverages Sphinx “for almost everything that it does”, it purposefully hides some of the Sphinx implementation details from the user to create a more user friendly experience. One of the consequences of this is that the assumptions that Read the Docs makes to build the documentation of Sphinx projects don’t hold: in particular, Jupyter Book uses a declarative configuration file _config.yml that gets translated on the fly to the Sphinx dynamic configuration usually stored in conf.py. As a result, Jupyter Book projects could not be hosted on Read the Docs - until now!

How to deploy a Jupyter Book project to Read the Docs

With the current development version of Jupyter Book, you can now export a Sphinx conf.py configuration from the Jupyter Book declarative _config.yml and save it to disk, which allows Read the Docs to build the documentation from it like any other Sphinx project.

The challenge then becomes keeping both configuration files synchronized, since every update to _config.yml or new Jupyter Book release can potentially produce changes in the conf.py. As described in the official documentation, users can either manually export the configuration running a command at will, or set up an automation to do it on every change.

To see this in action, have a look at this example project that contains the bare minimum to make the demo book work on Read the Docs.

We are excited that this is now possible and look forward to seeing more projects built with Jupyter Book!

—

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!