Read the Docs Blog - Posts by Anthony Johnson

Build errors with docutils 0.18

2021-11-02T00:00:00Z

Starting about a week ago, some users started reporting new errors with their project builds. In most cases, these errors appeared out of nowhere and are usually rather cryptic errors referencing Sphinx and docutils.

So, what is happening?

These errors are due to an incompatibility bewteen old versions of Sphinx and one of its dependencies, docutils. You probably have not directly interacted with docutils, but this is the library that provides reStructuredText parsing to both Sphinx and to the Python standard library documentation tooling.

There are a number of errors that users have reported due to this incompatibility, but the most common error is:

TypeError: 'generator' object is not subscriptable

If your project’s builds suddenly start failing with the above error message, or other cryptic errors that don’t seem related to your project, your project is most likely encountering this bug.

Why is this happening?

The underlying cause for this error is using an outdated version of Sphinx (version < 3.0) with the latest release of docutils (version 0.18, released October 26). This latest release of docutils is no longer comptible with Sphinx versions 1.x and 2.x, however these versions do not specify a upper limit to the version of docutils installed, and so will install the latest release by default.

The reason so many projects use an old version of Sphinx is that, before October 2020, Read the Docs had a more strict pinning policy and Sphinx 1.8 was the default version. New projects are not affected.

Fixing the error

We have proposed to the Sphinx and docutils maintainers a solution that doesn’t involve manual intervention. However, for now we suggest you manually resolve this issue for any of your projects that have encountered the bug.

There are two ways to resolve this issue:

Upgrade Sphinx to version 3 or later

We recommend this option for most users. If your project doesn’t specifically still require Sphinx 1.x or 2.x, your project can most likely be upgraded, without issues, to a modern version of Sphinx. As a bonus, you’ll be able to use the latest Sphinx releases and extensions that only support modern Sphinx releases.

It is worth noting that one reason why you might still be using an old version of Sphinx is if you generate API documentation for Python 2 compatible code. In this case, you’ll have more work ahead of you in order to upgrade, as Sphinx version 3 and 4 only support Python 3.

Downgrade docutils

This is the next best option if you can’t immediately upgrade Sphinx. If you are familiar with Python packaging, the best specifier to use is docutils<0.18.

Fixing the error on Read the Docs

If you aren’t already, you should add a configuration file for your project, so that you can specify additional dependencies to install at build time. There are a few supported methods for installation, depending on how you normally install dependencies for your project. If you are not currently specifying any dependencies on Read the Docs, you most likely will want to use the python.install.requirements configuration option.

This is how the reference .readthedocs.yaml and docs/requirements.txt files would look like:

.readthedocs.yaml

version: 2

python:
  install:
  - requirements: docs/requirements.txt

docs/requirements.txt

docutils<0.18

If you still experience problems, feel free to open an issue.

Theme release 1.0.0rc1

2021-08-23T00:00:00Z

Update: Version 1.0.0 has been released!

The 1.0.0 version of sphinx_rtd_theme was released Sept 13, 2021. You can install the latest version with:

$ pip install "sphinx-rtd-theme~=1.0.0"

Alternatively, you can upgrade the version installed using:

$ pip install --upgrade sphinx-rtd-theme

We are closing in on the next official release of sphinx_rtd_theme, and we’re happy to announce that release candidate version 1.0.0rc1 was released to PyPI last week! Project maintainers can contribute to the final release by testing the release candidate and raising any potential bugs on our issue tracker.

Version 1.0.0 adds support for Sphinx 4, fixes several issues stemming from docutils 0.17, and contains several backwards-incompatible changes.

Backwards incompatible changes

Users need to be aware of several backwards incompatible changes before upgrading:

Support dropped for old dependencies: Support is removed for Sphinx versions 1.6 or older, and for Python versions 3.0 to 3.3. Theme developers no longer test using these versions and compatibility is not guaranteed for these versions.
HTML4 support will be removed: Support for the Sphinx HTML4 writer will be deprecated in release 2.0. A deprecation warning was added to alert users still using the HTML4 writer.
Installation from source will be deprecated soon: Tentatively scheduled for release 3.0, installation from source will no longer be a supported installation method. Currently, users are installing directly from our GitHub repository, which complicates development of static assets. A deprecation warning was added to alert users of the upcoming change.

Highlights

Here are some of the most significant additions included in release 1.0.0:

Support for new dependencies: The theme now supports Sphinx 4 and docutils 0.17, both the latest in each respective release series.
New accessibility features: Also added were a number of accessibility updates, including better keyboard navigation support and more descriptive navigation for screen readers.
Added 4 new translations: The theme is close to being almost fully translated in 14 different languages. This will provide project maintainers with localized navigation elements and accessibility content.

The theme roadmap has more information on upcoming releases, and backwards incompatible changes for each. A full list of changes included in this release is available in the theme changelog.

Testing

The 1.0.0rc1 release is a pre-release package, meaning it will not install by default as the most recent release. Project maintainers will need to specify the release version explicitly to install the release candidate.

In your package’s extra_requirements, or a separate requirements.txt file, you can pin this dependency with:

sphinx_rtd_theme~=1.0.0rc1

If you notice any new, unwanted behavior, feel free to open an issue on our issue tracker.

Upcoming changes

We have several upcoming releases planned, covered in more depth in the theme roadmap.

Our next official release, currently targeted for October or November, will be version 1.1. This release will contain additional bug fixes and will be the last release supporting Sphinx versions less than 3.0. The next planed release will be version 2.0, in early 2022.

The 2.0 release will be dropping support for a number of old dependencies, including Sphinx versions < 3.0, Sphinx HTML4 writer support, and Internet Explorer 11.

As the theme moves away from supporting direct installation through our GitHub repository, we want to preserve development previews of theme releases. In addition to several more focused releases on the theme roadmap, we will also be periodically publishing development releases to PyPI. This will be the prescribed method to use unreleased theme changes.

Thanks!

Special thanks to contributor Blendify for keeping up with support for fresh dependencies, and new contributor nienn for helping with bug fixes and the heavy amount of testing required for this release!

Defaulting New Projects to Python 3

2019-02-12T00:00:00Z

New projects that are just getting started with Read the Docs will now use Python 3 by default. While it is still possible to configure your project to use Python 2.7 with our configuration file, we think it’s important to help push the Python ecosystem towards adopting Python 3.

Our default Python version is currently Python 3.7. Projects can also select Python versions 3.6 and 3.5 using our default build image. We will eventually remove support for building projects with Python versions 3.3 and 3.4, however it is still possible to select a build image with support for either version.

To select a specific version of Python, other than our default, you can use our configuration file to specify a Python version, using the python.version configuration option.

If your project does require Python version 3.3 or 3.4, you can only select these versions if you also specify stable build image for your project’s build.image (legacy) configuration option. As we release new versions of our build image, this stable image will eventually lose support for Python 3.3 and 3.4, so it’s suggested that your project upgrade to a supported version of Python.

Incoming Webhook Deprecations

2019-01-21T00:00:00Z

In the coming weeks and months, Read the Docs will be moving some projects away from our legacy incoming webhooks, towards our per-project webhook integrations.

Our legacy incoming webhooks were our first attempt at allowing providers like GitHub to automatically trigger builds on for projects on Read the Docs. These webhooks lacked a number of security features, and so, about two years ago, we replaced these with per-project webhook integrations instead. We added a number of features to per-project webhook integrations at the time, and we stopped new projects from using the old incoming webhooks.

It’s possible that your project is still using these deprecated incoming webhooks if your project was configured on Read the Docs more than two years ago. If your project does not have an webhook integrations under the Integrations section of your project’s admin dashboard, you might be using these legacy incoming webhooks.

In order to continue building automatically, projects should reconfigure their repository to use a new project webhook integration instead. For more information, see our docs on Integration Creation.

There are two important dates to note here:

January 31st, 2019

This is the date that GitHub will stop sending notifications through their GitHub Services. Your project might be relying on GitHub Services if your GitHub repository is using the ReadTheDocs GitHub Service and does not have another webhook to Read the Docs configured.

April 1st, 2019

After this date, Read the Docs will stop accepting incoming webhook notifications for the remaining legacy incoming webhooks. You might be using one of these incoming webhooks if your repository has a webhook pointing to any of the following URLs:

We will be sending periodic notifications over the next few weeks to the owners of any projects that are still configured to use these deprecated incoming webhooks.

Python 3.6 Support

2018-03-29T00:00:00Z

A long time back, we wrote about started testing a new build image that uses pyenv to support multiple versions of Python. Until recently, we were selectively opting projects in to help test the new image, but at the beginning of the year, we added a configuration option to allow projects to opt in to using the new image before we make it the default build image.

In the near future, this build image will be the default build image, but for now, you can manually opt your project in using our YAML configuration file.

What you need

In order to use Python 3.6, you need to select our latest build image, and you need to select to use Python 3.6. The build image is controlled with the build.image (legacy) setting in the YAML config, and you can set the Python version with the python.version setting.

Your project’s readthedocs.yml file should contain:

build:
    image: latest

python:
    version: 3.6

If you run in to any problems building, you might also need to wipe your version for now. We plan to have some of the issues switching Python versions and build images resolved before making a move to a new build image.

As always, if you notice anything strange with your build, feel free to raise an issue on our issue tracker.

PyCon 2017 in Review

2017-05-31T00:00:00Z

Things are finally getting back to normal for us after another very busy PyCon. It’s always wonderful seeing old friends and getting a chance to meet new friends from the community. PyCon provides a great outlet to talk to others in our community about the problems we all face as open source developers and open source companies – like funding, sustainability, and building community. We are sad to see PyCon leave Portland, but luckily PyCascades will soon be filling the void left in Cascadia after PyCon moves on.

This year, we announced official sprints on Read the Docs during the sprint week following PyCon. We focused our sprint efforts on code cleanup, in order to avoid the problems on-boarding new contributors we normally face as a large project.

The sprints went surprisingly well this year. We were able to enlist the help of several new contributors during sprints, and folks didn’t waste any time diving into the code. It was a large relief to finally spend some time addressing code cleanup and removal. Contributors were able to create and merge 38 pull requests over the week, and we’re still wrapping up work on several other pull requests. Our work focused on several areas in particular:

Linting strictness: When we started a serious effort to lint our codebase, we decided to address linting incrementally. We currently do a primary pass with low strictness, and a second pass of higher strictness only on select paths. During the sprints, contributors were able to pick off and focus on a single Django application. Cumulatively, contributors were able to fix linting issues across all of our applications and we’re very close to again raising our strictness application-wide.
Code removal: Several contributors moved on to hunting down and removing some unused code and dependencies. Having started with linting and testing problems, these contributors seemed well primed to understanding more complex parts of our code. We were able to remove several unused legacy pieces of code and some unused dependencies as a result.
Python 3: Contributor gthank started porting Read the Docs to Python 3 before the sprints, and was able to drop by our sprint to debug some problems with porting. We’ve been wanting to port to Python 3 for years now, but unfortunately, we could never justify this work as the core team. This pull request is still a work in progress, but we anxiously await being able to make the switch to Python 3.

As the Read the Docs core team is small, we can’t easily justify using our time to focus on long standing issues with code quality, so the PyCon sprints were a perfect opportunity to focus on this type of clean up. Our codebase is complex and unwieldy to new contributors, so focusing on linting and testing allowed contributors to jump in without having to spend the better part of a day setting up our full stack.

We were very gracious to have all of this help. A big thanks once again to all of our new contributors who stuck around and contributed:

PyCon has now wrapped up, but our next stop this summer is DjangoCon 2017 in Spokane, WA. We look forward to seeing some more of you there and hopefully working with some of you to sprint on Read the Docs again!

Release for May 12, 2017

2017-05-12T00:00:00Z

Yesterday, we rolled out improved webhook management for projects, and several bug fixes around our upgrade to Sphinx 1.5.

Webhook management

We’ve been slowly making upgrades to our webhook management page. Projects that set up new webhooks will see a list of webhooks that we have configured, including HTTP exchanges that we encounter from each remote webhook.

Some of the other improvements to our webhook management include a new generic webhook endpoint, the ability to create arbitrary webhooks, and the ability to now re-establish webhooks on a per-provider basis.

You can play with the new features by selecting a project, opening the project admin dashboard, and selecting Integrations. If your project is relatively new, webhook integrations will be created for existing remote webhooks. If your project has been configured for a long time, or you are using our old webhook endpoints, such as http://readthedocs.org/github, you won’t notice anything new on the integrations dashboard page. You can still manually create a webhook and configure your repository to point to the new Read the Docs webhook endpoint however.

Bug fixes

Following the release of Sphinx 1.5, we set this version as the default version for Sphinx projects on Read the Docs. This in turn caused an issue with our override of the Sphinx search mechanism. We currently override this to provide our own search indexes, however the patch to this file was incompatible with Sphinx 1.5.

Due to this incompatibility, if a search query did not return anything from our indexes, but did match the Sphinx internal index, the links would have been incorrect.

We are now patching this search mechanism regardless of which version of Sphinx is used, and the broken links are no longer returned in the search results.

Problems?

If you encounter any problems with any of the recent changes, feel free to open up a bug report on our issue tracker.

Build Image Upgrades

2017-02-14T00:00:00Z

Starting this week, we’ll be deploying a new default image for our documentation build environments. This image will change the default Python versions that are supported.

We aren’t expecting any issues to arise from this change, but be sure to raise any issues on our issue tracker if you notice any strange behavior. The new build image supports Python 2.7 and Python 3.5, dropping support for Python 3.4. If you require access to Python 3.4, we suggest you sign up for access to our next beta image.

The next image for our build process is now being tested. This image will support multiple versions of Python: 2.7, 3.3, 3.4, 3.5, and 3.6. Instead of relying on the distribution’s versions of Python, multiple versions are instead installed from source, using pyenv.

If you’d like beta access to this build image for your projects, you can sign up here:

Update

Users are now able to opt in to a build image that support Python 3.6, signing up for beta access is no longer required. For more information, see our blog post on Python 3.6 Support

We are slowly moving project configuration to our YAML configuration file, readthedocs.yml. Access to specific versions of Python will only be available using this configuration method, by specifying a python.version element.

Securing Subdomains

2016-04-27T00:00:00Z

Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard.

Changes to provide security against broader threats have been in place for a while, however there are still a few scenarios that can only be addressed by migrating to a separate domain.

We implemented session hijacking detection and took precautions to limit cookie usage, but there are still a number of scenarios utilizing XSS and CSRF attacks that we aren’t able to protect against while hosting documentation from subdomains on the readthedocs.org domain. Moving documentation hosting to a separate domain will provide more complete isolation between the two user interfaces.

Projects will automatically be redirected, and this redirect will remain in place for the foreseeable future. Still, you should plan on updating links to your documentation after the new domain goes live.

If you notice any problems with the changes, feel free to open an issue on our issue tracker: http://github.com/rtfd/readthedocs.org/issues. If you do notice any security issues, contact us at security@readthedocs.org with more information.

Keep on documenting, The Read the Docs Team

Read the Docs Awarded Mozilla Open Source Support Grant

2015-12-10T00:00:00Z

Several months back, Mozilla launched a new initiative – the MOSS program – to provide financial support to the open source software projects it relies on. Mozilla allocated $1 million to the MOSS fund to provide grants for up to 10 projects matching the program’s criteria.

Read the Docs is among the first round of awards made for the MOSS program. Our proposed grant, for $48,000, is to build a separate instance that integrates with the Python Package Index’s upcoming website, Warehouse. This integration will provide automatic API reference documentation upon package release, with authentication tied to PyPI and simple configuration inside the distribution. API reference documentation on every release will be the starting point of this work, prose documentation generation will be more difficult here, as not all packages use Sphinx or rST for documentation.

Having consistent API reference documentation across all released distributions will be an enormous benefit to the Python ecosystem. API reference documentation will be found in a single space, with version specificity, and will be able to be tightly inter-linked across projects. This is hopefully the start of work that could benefit other communities as well, by putting more emphasis on usable documentation and best documentation practices.

We are currently just into the planning stage for this work. We hope to have an official roadmap planned out, and a detailed breakdown of the work involved, in the coming weeks. This post will be followed up with a more detailed look at our plan and the work required. Work on this grant will hopefully begin early in the new year and will be spaced out throughout the year.

Funding open source software communities is still a difficult proposition for most companies. We appreciate Mozilla’s stance and commitment to open source and can only hope more companies take note and learn how to put value on the health of the communities they rely on.