Read the Docs Blog - Posts from Portland, Oregon

Theme release 1.0.0rc1

2021-08-23T00:00:00Z

Update: Version 1.0.0 has been released!

The 1.0.0 version of sphinx_rtd_theme was released Sept 13, 2021. You can install the latest version with:

$ pip install "sphinx-rtd-theme~=1.0.0"

Alternatively, you can upgrade the version installed using:

$ pip install --upgrade sphinx-rtd-theme

We are closing in on the next official release of sphinx_rtd_theme, and we’re happy to announce that release candidate version 1.0.0rc1 was released to PyPI last week! Project maintainers can contribute to the final release by testing the release candidate and raising any potential bugs on our issue tracker.

Version 1.0.0 adds support for Sphinx 4, fixes several issues stemming from docutils 0.17, and contains several backwards-incompatible changes.

Backwards incompatible changes

Users need to be aware of several backwards incompatible changes before upgrading:

Support dropped for old dependencies: Support is removed for Sphinx versions 1.6 or older, and for Python versions 3.0 to 3.3. Theme developers no longer test using these versions and compatibility is not guaranteed for these versions.
HTML4 support will be removed: Support for the Sphinx HTML4 writer will be deprecated in release 2.0. A deprecation warning was added to alert users still using the HTML4 writer.
Installation from source will be deprecated soon: Tentatively scheduled for release 3.0, installation from source will no longer be a supported installation method. Currently, users are installing directly from our GitHub repository, which complicates development of static assets. A deprecation warning was added to alert users of the upcoming change.

Highlights

Here are some of the most significant additions included in release 1.0.0:

Support for new dependencies: The theme now supports Sphinx 4 and docutils 0.17, both the latest in each respective release series.
New accessibility features: Also added were a number of accessibility updates, including better keyboard navigation support and more descriptive navigation for screen readers.
Added 4 new translations: The theme is close to being almost fully translated in 14 different languages. This will provide project maintainers with localized navigation elements and accessibility content.

The theme roadmap has more information on upcoming releases, and backwards incompatible changes for each. A full list of changes included in this release is available in the theme changelog.

Testing

The 1.0.0rc1 release is a pre-release package, meaning it will not install by default as the most recent release. Project maintainers will need to specify the release version explicitly to install the release candidate.

In your package’s extra_requirements, or a separate requirements.txt file, you can pin this dependency with:

sphinx_rtd_theme~=1.0.0rc1

If you notice any new, unwanted behavior, feel free to open an issue on our issue tracker.

Upcoming changes

We have several upcoming releases planned, covered in more depth in the theme roadmap.

Our next official release, currently targeted for October or November, will be version 1.1. This release will contain additional bug fixes and will be the last release supporting Sphinx versions less than 3.0. The next planed release will be version 2.0, in early 2022.

The 2.0 release will be dropping support for a number of old dependencies, including Sphinx versions < 3.0, Sphinx HTML4 writer support, and Internet Explorer 11.

As the theme moves away from supporting direct installation through our GitHub repository, we want to preserve development previews of theme releases. In addition to several more focused releases on the theme roadmap, we will also be periodically publishing development releases to PyPI. This will be the prescribed method to use unreleased theme changes.

Thanks!

Special thanks to contributor Blendify for keeping up with support for fresh dependencies, and new contributor nienn for helping with bug fixes and the heavy amount of testing required for this release!

Read the Docs 2020 Stats

2021-03-17T00:00:00Z

2020 was a rough year for everyone, including our team. We managed to make it through, and continue to have 5 folks working full-time to make Read the Docs better for you.

We are going into 2021 with a new grant, which will require us to do some hiring. We also launched our EthicalAds network, which is bringing our approach to sustainability to the tech community as a whole.

As we have for the past 7 years, we are proud to share our 2020 stats post.

Note

You can always see our stats for the last 30 days.

Our posts from 2013, 2014, 2015, 2016, 2017, 2018, and 2019 are also available.

Page Views

Our stats:

580 Million Page Views (from 465 million)
156 Million Unique Visitors (from 120 million)

These numbers are not completely accurate, since we implemented Do Not Track support our analytics pageview figures understate our actual traffic. We estimate that around 6% of users are not counted here. We also don’t count all the users who block Google Analytics with a browser extension.

Site Stats

The stats, in total numbers:

240,000 projects (from 200,000)
430,000 users (from 400,000)
2,480,063 builds (from 10 million total last year)

We have been battling spam quite heavily again this year, so these numbers are a bit skewed. We do notice a uptick in usage across the site, both in terms of support and traffic.

Community

This year, we had:

14 people who committed code to the main repository (from 21)
3636 commits (from 4793)
532 issues - 104 open, 428 closed (from 609)

We have been working to move user support issues to email from GitHub, which likely addresses the reduction in ticket numbers. We are also working to expand our ecosystem of projects, so the commits to the main repository are down. We expect across our theme, extensions, and other projects commits are up this year, and we’re appreciative of all the work people contribute.

Funding

$234,000 advertising (from $249,000)
$30,000 Gold users and donations (from $26,600)
5 paid staff (same as last year)
We have additional revenue from our commercial offering at readthedocs.com, but aren’t including that in our community funding overview

With the launch of EthicalAds, we are excited to continue to grow our advertising business, and expand it to the larger community. We are also seeing continued growth of our paid product at https://readthedocs.com, which is allowing us to fund further growth of the Read the Docs codebase for all our users.

Conclusion

We continue to work to improve Read the Docs for our users. We’re excited for the growth we expect in 2021, and hope that you’ll join us for the next chapter of our story.

The Read the Docs Team

Announcing Chan Zuckerberg Initiative Grant to Expand the Interoperability of Scientific Documentation

2020-11-19T00:00:00Z

We’re excited to announce that Read the Docs has received a $200,000 grant from the Chan Zuckerberg Initiative’s Essential Open Source for Science (EOSS) program. Read the Docs is the largest open source documentation hosting platform in the world. We provide hosting for many scientific software packages, including some that have received EOSS funding in the past. You can read more about this round of grants in the official announcement.

Our grant has two parts:

Part 1 allows us to develop new software to improve the interoperability of scientific documentation.
Part 2 allows us do advocacy work around the importance and value of documentation in the scientific community.

We’re excited about the chance to work with the scientific community to improve the overall experience of writing and reading documentation. We plan to do a lot of outreach to various community members, so that we can ensure we’re building tools that provide the most value. We also plan to write content that will hopefully make the process of documenting scientific code easier.

Part 1: improve the interoperability of scientific documentation

This work is something that we’ve been doing as a side project for some time now. We have an extension to Sphinx that we call sphinx-hoverxref. It provides hovering tooltips for Sphinx projects when hover over a link. You might be familiar with this functionality on GitHub or Wikipedia – this project adds that functionality to any project hosted on Read the Docs. You can see a live demo on a fork of the Python documentation.

The goal of the grant will be to expand this functionality by doing a few things:

Improving and documenting the backend Embed API on Read the Docs, allowing documentation content to be embedded anywhere on the web.
Building a JavaScript client for the Embed API, which includes tooltips showing the content on any website.
Integrating this JS client and backend API together in a new version of sphinx-hoverxref.
Extending sphinx-hoverxref to work across multiple Read the Docs sites, allowing content hovers across the entire scientific documentation ecosystem.
If time allows, also extending the backend Embed API to support Sphinx documentation hosted outside of Read the Docs, allowing even more interoperability between documentation.

We are planning to contract a frontend developer to help with this work. The Python work will likely be carried out by our existing team, funded via the grant. This is because of the complexities involved with the Read the Docs and Sphinx integration, it will be more cost effective to have people already familiar with the codebases on these tasks.

Part 2: Advocacy work around documentation in the scientific community

We know that documentation is a pain point for scientific projects. There is a lack of resources in the community, and we feel that we can be a central clearing house for better documentation tutorials and best practices. This part of the funding will be about growing appreciation and knowledge of documentation in the scientific community in general, and advocating for time to be spent on crafting good documentation with the best tooling.

Part of this work would be letting people know about the work in Part 1, promoting the tooling to make better documentation in the Scientific Python ecosystem. Beyond that, we will also promote tutorials and best practices around documentation for Scientific users, leading to more resources being available and better documentation in the community.

The high-level roadmap for this part of the grant is:

Improve documentation and promote the work in Part 1.
Adapt the Guides section of the Read the Docs documentation into a more full-featured Education section, along with standardizing and editing all existing guides.
Expand our existing Sphinx Tutorial, hopefully adding it upstream to Sphinx as the official tutorial
Write additional documentation guides focused on the scientific community, based on feedback from people in the community. This will likely include tutorials on integrating Jupyter with Sphinx, Jupyterbook, and other pain points that users have.
If time allows, we would also love to work on curating some of the documentation resources at Write the Docs, specifically topics that are relevant to scientific projects.

As part of this work, we plan to contract someone in a technical writer/developer evangelist role. Someone with knowledge of the scientific community would be a great benefit, given that we plan to seek feedback from many scientific projects on the priority of tasks we work on.

Looking forward

We are starting to look for contractors who would be a good fit for the work. People who care about documentation and are already connected with the scientific community are a big plus. We hope to post proper job descriptions for these roles in the near future, but wanted to note that we’re looking in our initial announcement.

This is the largest grant that we’ve ever received, and we are working hard to make sure it’s successful. We appreciate the trust that the Chan Zuckerberg Initiative has placed in us, and we will work hard to make sure the results of this grant will be adopted across the scientific ecosystem.

If you want to keep up to date with this work, you can follow along on GitHub or subscribe to our blog for updates.

Shipping a CDN on Read the Docs Community

2020-05-18T00:00:00Z

You might have noticed that our Read the Docs Community site has gotten faster in the past few weeks. How much faster likely depends on how far away you live from Virginia, which is where our servers have traditionally lived.

We have recently enabled a CDN on all Read the Docs Community sites, generously sponsored by CloudFlare. This post will talk a bit more about how we implemented this, and why we’re excited about it.

We are also offering the CDN option to our Read the Docs for Business users on the Enterprise plan, you can reach out to us if you’re interested.

Hosting thousands of domains is hard

Traditionally the largest problem that we’ve had with rolling out features to all of our documentation sites is scale. We host thousands of custom domains for our users, and any solution needs to work across all of them. This has presented a number of issues in the past, but CDN is one of the most complicated.

To make a CDN function across all our sites, every data center that the CDN has needs to work with all our custom domains. Imagine we have 5,000 domains and there are 100 data centers across the world, this means that 5,000 * 100 = 500,000 different endpoints need to be configured for this to work at scale.

We are lucky that CloudFlare has the global scale to be able to donate us this service. Specifically, their SSL for SAAS service is what we’re using for both SSL and CDN across all our custom domains. This allows us to offload the complexity to CloudFlare, and only focus on our integration.

Implementation

One of the coolest things that CloudFlare’s CDN offers is something called Cache Tags. This lets us add a Cache-Tag header to all the documentation that we serve, and invalidate the cache using just that tag.

An example, when you load our docs, we will return a header:

GET https://docs.readthedocs.io/en/latest/
...
Cache-Tag: docs,docs-latest

We return a tag that matches both the $project, and the $project-$version. This allows us to invalidate the cache for a specific version, or across the entire project.

As you know, cache invalidation is one of the harder problems, so we take a pretty conservative approach. We invalidate the cache in the following scenarios:

Project cache on Project or Domain save
Version cache on documentation builds for specific versions

The client code is quite simple, a single HTTP request to Cloudflare’s API with a list of tags to clear.

Outcomes

There are two important outcomes from this work:

Page loads are much faster for users around the world
Our server load has gone down quite a lot because we handle fewer requests

The biggest winner here is our users. Docs are faster for everyone, and we are looking at implementing additional features into our documentation hosting code now that we have reduced load.

We hope that Read the Docs Community has gotten noticeably faster, and that in the near future it will continue to get better with the new features that this enables.

Read the Docs 2019 Stats

2020-05-14T00:00:00Z

2019 was another good year for Read the Docs. We continue to have a team of 5 folks working on the project, and we’ve rolled out a number of new features for the year.

Here are our stats for the past year, which we’ve published since 2013. This is part of our effort to be transparent in our organization, as well as our source code.

Note

You can always see our stats for the last 30 days.

Our posts from 2013, 2014, 2015, 2016, 2017, and 2018 are also available.

Page Views

Our stats:

465 Million Page Views (from 400M)
120 Million Unique Visitors (from 77M)

These numbers are not completely accurate, since we implemented Do Not Track support last year, our analytics pageview figures understate our actual traffic. We estimate that around 6% of users are not counted here. We also don’t count all the users who block Google Analytics with a browser extension.

Site Stats

The stats, in total numbers:

200,000 projects (from 100k)
400,000 users (from 150k)
10,188,182 builds (new this year)

We have been battling spam quite heavily again this year, so these numbers are a bit skewed. We do notice a uptick in usage across the site, both in terms of support and traffic.

Community

This year, we had:

21 people who committed code to the main repository (from 28)
4793 commits (from 3774)
609 issues - 80 open, 529 closed (from 797)

We have continued to have a similar team working on the project this year, but have actually made more commits. We moved our support queue from issues to email, which likely accounts for the lower issue count. We’re quite happy with our closed issue stats this year, which are much better than previous years.

Funding

$275,000 in revenue from readthedocs.org (from 294,000)
$249,000 from advertising (from $278,000)
$26,600 from Gold users (from $16,000)
We have additional revenue from our commercial offering at readthedocs.com, but aren’t including that in our community funding overview

Our advertising has been pretty steady for 2019. The small drop in the year-over-year comparison we account to the fact that ad blockers happened halfway through 2018, meaning that the comparison is 15% lower in terms of our total traffic.

We continue to be hosted by Azure, and we’re using a lot of their nicer feature to keep our costs down. You can find some posts on our blog about this.

Conclusion

We continue to keep working to improve the documentation ecosystem in the software industry. We’re excited about a number of new features that we’re working on this year, and expect to continue to be a resource you can depend on for a long time to come :)

Read the Docs Offsite 2019

2019-06-20T00:00:00Z

The Read the Docs team just finished our first offsite ever in April of 2019. We all gathered together in person for the first time, and talked about the future of the project.

A picture will show this better than I can:

The team at the house we rented in Aruba

The full list of attendees, which is the full-time staff, from left to right:

David Fischer
Manuel Kaufmann
Eric Holscher
Anthony Johnson
Santos Gallegos

Goals

Our biggest goal was building a shared understanding of the vision for the project. Being fully remote, it’s often hard to communicate at a deep level about mission and vision. Being together in person for a week gave us the space to understand each other and our views better.

The other large goal was to build a roadmap for the next 3-6 months for the project. We have often had smaller roadmaps, but never had the chance to discuss all the problems that we encounter with the project, and then prioritize them.

Format

We used the following format:

Before the offsite, we created a Trello board with a list of topics to discuss. This was all the major feature ideas, issues, and concerns that came up over the previous couple months.
For the first 3 days, we went through the list and discussed each topic. We took notes (almost 20 pages) of these discussions, and mostly tried to build a shared understanding on a path forward.
The 4th day we broke each section down into action items, and chunked them into 1-3 smaller sections of work. We called these “v1”, “v2”, and “v3” to break out the stages where each project could be shipped and have impact on our users.
The 5th day we took all the tasks, then sized and prioritized them. We broke things out into 4 piles representing months going forward, and tried to balance the tasks so each month had a similar amount of work in it. We then turned this into a Trello roadmap board.

This worked quite well for us. It gave us space to talk through all the various topics we had, but also gave concrete next steps to move forward on our tasks.

Outcomes

The most valuable outcome is something I said at the offsite:

Before, it felt like we were 5 contributors working on an open source project. Now, it feels like we’re members of a team.

For each member of the team, there was someone else they had never met in person before this. Having all met in person will make it much easier to collaborate online going foward, and to feel like we are working towards a goal that we all share.

In terms of technical output, the roadmap we have established will make us much more productive in our work going foward. The entire team now has vision around the roadmap items, and understands the tasks other people are working on much better. This has already lead to a much better ability to collaborate together.

We are hoping to do another offsite in 2020, and if we do our jobs right, hopefully we’ll have another teammate or two.

Defaulting New Projects to Python 3

2019-02-12T00:00:00Z

New projects that are just getting started with Read the Docs will now use Python 3 by default. While it is still possible to configure your project to use Python 2.7 with our configuration file, we think it’s important to help push the Python ecosystem towards adopting Python 3.

Our default Python version is currently Python 3.7. Projects can also select Python versions 3.6 and 3.5 using our default build image. We will eventually remove support for building projects with Python versions 3.3 and 3.4, however it is still possible to select a build image with support for either version.

To select a specific version of Python, other than our default, you can use our configuration file to specify a Python version, using the python.version configuration option.

If your project does require Python version 3.3 or 3.4, you can only select these versions if you also specify stable build image for your project’s build.image (legacy) configuration option. As we release new versions of our build image, this stable image will eventually lose support for Python 3.3 and 3.4, so it’s suggested that your project upgrade to a supported version of Python.

Incoming Webhook Deprecations

2019-01-21T00:00:00Z

In the coming weeks and months, Read the Docs will be moving some projects away from our legacy incoming webhooks, towards our per-project webhook integrations.

Our legacy incoming webhooks were our first attempt at allowing providers like GitHub to automatically trigger builds on for projects on Read the Docs. These webhooks lacked a number of security features, and so, about two years ago, we replaced these with per-project webhook integrations instead. We added a number of features to per-project webhook integrations at the time, and we stopped new projects from using the old incoming webhooks.

It’s possible that your project is still using these deprecated incoming webhooks if your project was configured on Read the Docs more than two years ago. If your project does not have an webhook integrations under the Integrations section of your project’s admin dashboard, you might be using these legacy incoming webhooks.

In order to continue building automatically, projects should reconfigure their repository to use a new project webhook integration instead. For more information, see our docs on Integration Creation.

There are two important dates to note here:

January 31st, 2019

This is the date that GitHub will stop sending notifications through their GitHub Services. Your project might be relying on GitHub Services if your GitHub repository is using the ReadTheDocs GitHub Service and does not have another webhook to Read the Docs configured.

April 1st, 2019

After this date, Read the Docs will stop accepting incoming webhook notifications for the remaining legacy incoming webhooks. You might be using one of these incoming webhooks if your repository has a webhook pointing to any of the following URLs:

We will be sending periodic notifications over the next few weeks to the owners of any projects that are still configured to use these deprecated incoming webhooks.

Read the Docs 2018 Stats

2019-01-08T00:00:00Z

2018 was another good year for Read the Docs. We’ve settled into a sustainability model that is working for us, and have a team of 5 folks working full-time on the project.

Here are our stats for the past year, which we’ve published for the past 6 years. This is part of our effort to be transparent in our organization, as well as our source code.

Note

You can always see our stats for the last 30 days.

Our posts from 2013, 2014, 2015, 2016, and 2017 are also available.

Page Views

Our stats:

400 Million Page Views (up from 338M, 18%)
77 Million Unique Visitors (up from 75M, 3%)

These numbers are not completely accurate, since we implemented Do Not Track support in May, our analytics pageview figures understate our actual traffic. We estimate that around 6% of users are not counted here. We also don’t count all the users who block Google Analytics with a browser extension.

Site Stats

The stats, in total numbers:

100,000 projects (up from 77k, 30%)
155,000 users (up from 103k, 50%)

We have been battling spam quite heavily again this year, so these numbers are a bit skewed.

Community

This year, we had:

28 people who committed code to the main repository (up from 23, 22%)
3774 commits (up from 1058, 257%)
797 issues - 189 open, 608 closed (up from 513, 55%)

You can see the number of commits and issues has gone up quite a bit. This is largely because of the additional staffing we have had on the project, allowing us to be more active to both our users and to release more features.

Funding

$294,000 in revenue from readthedocs.org (up from $218,000, 35%)
$278,000 from advertising (up 59%)
$16,000 from Gold users (up 16%)
We have additional revenue from our commercial offering at readthedocs.com, but aren’t including that in our community funding overview

The biggest impact to our revenue this year was our ads being blocked. This happened in May, and caused an initial 32% reduction in our ad revenue. We were added to the Acceptable Ads list later in the month, which regained around half of our revenue. So for the second half of the year, our ad revenue was down about 15%.

We are also exploring sharing revenue with our documentation authors. There are currently 3 projects enrolled in a revenue share, and we’re hoping to expand that in 2019 to better support open source projects.

The project is now on firm financial footing. We have a cash reserve that will allow us to pay all of our staff for more than 3 months in the event of a loss of revenue, which allows us to be more flexible with our revenue going forward. This funding allowed us to expand from 4 to 5 full-time folks working on the project.

We also moved hosting to Azure this year in August from Rackspace, who both have sponsored our hosting.

Conclusion

This year we’ve continued to invest in our ethical advertising approach, growing our team by one member and building lots of new features. It feels like the project is progressing quite well, and we’re committing way more code and dealing with a much larger number of user issues.

It feels like success to be able to properly support our community by improving the product, while creating a business model that we can be proud of.

Ethical Advertising Works

2018-04-16T00:00:00Z

It has been two years since we first launched ads on Read the Docs. We figured it was time to report on the results that we’ve seen, and say thanks to those who have helped us along the way.

The Results

To put it simply:

Ethical Advertising works.

In Q1 2018, our estimated revenue from advertising is $75,000. This is enough to fund our small team of 4 people to work on Read the Docs full-time.

When we started building our ethical advertising model, we didn’t know how it would work out. We didn’t know much about the ad industry, and we were doing things different than everyone else.

We are happy to report that you can build advertising that is a win/win/win for all parties:

Read the Docs wins because we generate revenue, allowing us to sustain our operations
Advertisers win because they are able to reach our audience of developers, helping to grow their business.
Users win because their privacy is maintained, and they continue to have access to our service.

Ethical advertising on Read the Docs has not been without its challenges, which we will cover in future posts.

Tracking users isn’t required

Along the way we’ve been trying to prove a simple thesis:

We can have successful advertising without tracking our users. So much of the trouble facing the ad industry these days is because of this massive databases of user data.

We have been able to show that we can show ads without knowing exactly who our users are. The content they are reading is enough to show them targeted ads, without knowing anything about them. Specifically:

We do not let advertisers run 3rd party code on Read the Docs. Ad images and javascript is all hosted by us.
We do not provide advertisers with data dumps, except aggregated reports on the clicks and views their ads got.
We target our ads based only on geography and the details of the page being visited, not on a profile of the user.

We’re now taking a look at what we’ve learned, and how we might help more infrastructure projects reach sustainability with an ethical advertising approach.

Towards an Ethical Ad Network

We have shown that an advertising business model that respects users can also sustain critical open source infrastructure. We hope that we can act as an example for other projects that depend on ads. You can make money without giving away your users data.

Developers have the power to shift the ad industry to respect user privacy. We must start with our own projects, leading by example. We need to have all the ads in the open source ecosystem respect user privacy, and then we can start to apply pressure to other parts of the tech industry.

We realize that not every open source project can do all the work that we’ve done. We want to explore how we can build an Ethical Ad Network, where user data isn’t sent to us, but each project doesn’t need to do ad sales.

Our ethical ad network is still in the planning phases, but if you know a project that might be interested in being a beta tester, please get in touch.

Thanks

I’d like to thank all of companies who have believed in our ethical ads initiative. You can only sell something if people buy it, and I’d like to give credit to the folks who have believed in our concept:

Twilio
Sentry
Rollbar
Stream
Intel
Data Dog
Mongo
Hosted Graphite
Level 12
Pyup
Triplebyte
Linode
Digital Impact Alliance
Digital Ocean
Exoscale
Nginx
Odoo
CircleCI
Cherry Servers
Uniregistry
OSCON

Want your company to be on the list? You can get more information on our advertising page.