Read the Docs Blog - Posts tagged feature

Announcing Embed API v3 and sphinx-hoverxref 1.0

2021-12-07T00:00:00Z

We are thrilled to announce the availability of Read the Docs Embed API v3, along with its official client, sphinx-hoverxref 1.0. This work has been possible in part thanks to the the CZI grant we received.

The value of content embedding

As we wrote in our first blog post about sphinx-hoverxref, one of the most powerful features of Sphinx is the possibility of creating cross references to other documentation projects. However, a reader finding several links in a technical documentation might need to open several browser tabs to fully understand the context, resulting in a lot of friction in the form of context switching.

One way of mitigating the problem is to show a small preview in the form of tooltip or modal with the contents of the link. This is where the Embed API and sphinx-hoverxref come in.

sphinx-hoverxref is a Sphinx extension that addresses this problem by adding tooltips on the cross references of the HTML documentation with the content of the linked section. To do so, it inserts a small JavaScript file that calls the Read the Docs Embed API when the user hovers any link. Back in June we announced intersphinx support in sphinx-hoverxref 0.6b1, which allowed users to include modals and tooltips also for cross references to other projects.

sphinx-hoverxref displaying a tooltip including an equation

Some big projects already using it include Tweepy, a Python client for Twitter, and Scrapy, a framework for crawling websites.

Embed API v3 and sphinx-hoverxref 1.0

After some months of work, we are excited to publish v3 of our Embed API, and with it, version 1.0 of sphinx-hoverxref.

Among other things, the new versions allow you to embed content from pages hosted outside Read the Docs. For security reasons, we have an allowlist of such external domains that currently includes Python, NumPy, SciPy, SymPy, and we would like to invite the community to suggest more domains to add.

Other additions and fixes include:

Support for JS and CSS assets in Sphinx 4.1+.
Support for glossary / term and sphinxcontrib-bibtex.
Avoid showing the browser built-in tooltip for links that have the extension enabled.

Finally, version 3 of the Embed API paves the way for supporting non-Sphinx projects in the future.

Embedding content with sphinx-hoverxref

To use sphinx-hoverxref in your Read the Docs project, declare it as part of your dependencies:

requirements.txt

sphinx==4.3.1
sphinx_rtd_theme==1.0.0
sphinx-hoverxref==1.0.0

And add it to your list of Sphinx extensions:

conf.py

extensions = [
    # ...other extensions
    "hoverxref.extension",
]

To enable the extension on all :ref: of your documentation, set the hoverxref_auto_ref to True:

conf.py

hoverxref_auto_ref = True

And you will start seeing tooltips on your internal references. Apart from :ref: roles, you can also enable tooltips on:

Note

sphinx-hoverxref includes the JavaScript embed client in the HTML assets, but it is not yet available as a standalone library that can be reused with standard frontend packaging tools. If you would like to see this happening, let us know.

Calling the Embed API directly

As explained in our embedding guide, you can call the API directly from any HTTP client:

$ curl -s "https://readthedocs.org/api/v3/embed/\
> ?url=https://docs.readthedocs.io/en/latest/features.html\
> %23read-the-docs-features" | python -m json.tool
{
  "url": "https://docs.readthedocs.io/en/latest/features.html#read-the-docs-features",
  "fragment": "read-the-docs-features",
  "content": "<div class=\"section\" id=\"read-the-docs-features\">\n<h1>Read the Docs ...",
  "external": false
}

Or visually explore the query in the web interface instead.

The reference documentation includes more detail about the parameters and return values of the API.

Try it out!

We would like to invite the community to try out these features and send us feedback. With the help of our users, we will keep moving towards a more cohesive documentation ecosystem of interlinked Python projects.

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

Read the Docs ❤️ Jupyter Book

2021-11-11T00:00:00Z

We are proud to announce that now Jupyter Book projects are supported on Read the Docs!

Both Read the Docs and The Executable Book Project, the folks behind Jupyter Book, share a common passion for documentation, and we have been collaborating on various topics for some time already. For example, we started promoting MyST in favor of our recommonmark back in April this year, and we wrote a guide on using Jupyter notebook with Sphinx that benefitted a lot from their feedback.

Now we are happy to announce that Jupyter Book itself is supported on Read the Docs, after some thorough discussion about the possible implementations and thanks in large part to the Jupyter Book developers, who addressed all the minor incompatibilities that prevented this from happening.

What is Jupyter Book?

According to its own documentation,

Jupyter Book is an open source project for building beautiful, publication-quality books and documents from computational material.

Jupyter Book is the flagship product of the Executable Book Project, an international collaboration between several universities and software projects seeking to build open source tools that facilitate publishing computational narratives using the Jupyter ecosystem.

Even though Jupyter Book is much younger than Read the Docs as a project, it has become very popular in recent times among scientific software developers and educators. It enables users to write publication-quality content in Markdown thanks to MyST (a Markdown dialect compatible with Sphinx we started promoting back in April this year), use Jupyter notebooks to author content thanks to MyST-NB (featured in our Jupyter on Sphinx guide), easily add interactivity thanks to Thebe, and much more.

Why a change was needed

Read the Docs supports two documentation generation systems Sphinx and MkDocs. Adding extra systems is difficult with the current codebase, because it requires lots of effort to match all the features currently supported by the existing ones.

On the other hand, even though Jupyter Book leverages Sphinx “for almost everything that it does”, it purposefully hides some of the Sphinx implementation details from the user to create a more user friendly experience. One of the consequences of this is that the assumptions that Read the Docs makes to build the documentation of Sphinx projects don’t hold: in particular, Jupyter Book uses a declarative configuration file _config.yml that gets translated on the fly to the Sphinx dynamic configuration usually stored in conf.py. As a result, Jupyter Book projects could not be hosted on Read the Docs - until now!

How to deploy a Jupyter Book project to Read the Docs

With the current development version of Jupyter Book, you can now export a Sphinx conf.py configuration from the Jupyter Book declarative _config.yml and save it to disk, which allows Read the Docs to build the documentation from it like any other Sphinx project.

The challenge then becomes keeping both configuration files synchronized, since every update to _config.yml or new Jupyter Book release can potentially produce changes in the conf.py. As described in the official documentation, users can either manually export the configuration running a command at will, or set up an automation to do it on every change.

To see this in action, have a look at this example project that contains the bare minimum to make the demo book work on Read the Docs.

We are excited that this is now possible and look forward to seeing more projects built with Jupyter Book!

—

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

Ubuntu 20.04, Python 3.10, and support for Node, Rust, and Go

2021-10-14T00:00:00Z

We are excited to announce that now Read the Docs users can use a newer build specification in their projects that will change the base image to one based on Ubuntu 20.04, ship the recently released Python 3.10, and allow users to easily specify the version of Node.js, Rust, and Go. This feature has been a long time in the making, and we think it will simplify the configuration of many projects.

Previous situation

The Docker images used by our builders were based on Ubuntu 18.04. Recently, we added a new feature to install custom system packages, which allowed many projects to have better control of their build process without having to use conda to manage non-Python dependencies.

However, this Ubuntu version was released three years ago and, even though it was good enough for most projects, we have been receiving requests to upgrade certain system packages over the years. Since these are managed through the operating system package manager, and we do not provide a way to add custom Personal Package Archives, such upgrades were not possible.

Many Python projects use a secondary programming language for a variety of reasons. Established compiled languages like C, C++, and FORTRAN have readily available compilers that will work for the majority of use cases. However, other younger technologies move at a faster pace, and developers often want specific versions of the toolchains that cannot be installed with the system package manager. This affected projects that combine backend and frontend code that need Node.js to compile their JavaScript sources, such as Jupyter extensions, and libraries with performance-critical code written in Rust or Go.

We have often suggested specifying custom versions of non-Python dependencies using conda, to specify custom versions of non-Python dependencies, but this requires learning another tool that might be unfamiliar for some projects.

New `build` YAML configuration

To overcome all these problems, we have added a new configuration value, build.tools, that receive a dictionary of toolchain versions. This new configuration requires specifying the base image name in build.os, currently ubuntu-20.04. Our configuration documentation contains a simple example:

version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-20.04
  tools:
    python: "3.9"
    # You can also specify other tool versions:
    # nodejs: "16"
    # rust: "1.55"
    # golang: "1.17"

And you can draw inspiration from some community projects that are using this feature already:

cryptography, the reference Python library for cryptographic algorithms, can now specify the Rust version required to build the package.
parsel, a Python library to manipulate HTML and XML using XPath and CSS selectors, leverages the new specification to build their documentation on the just released Python 3.10.
Thebe, a Jupyter-powered tool to make HTML pages interactive, needs Node.js to build the latest version of the code.

Do you think this feature will be useful for you as well? Give it a try and let us know.

Remember you can always see the latest changes to our platforms in our Read the Docs Changelog.

—

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

Install custom operating system packages (apt)

2021-05-19T00:00:00Z

We are thrilled to announce that now Read the Docs users can declare custom operating system packages in their project configuration that will get installed in our Ubuntu-based builders using apt. This has been a long awaited feature, and we think it will simplify the configuration of many projects, especially scientific ones.

Previous solutions

The Ubuntu images used by our builders contain lots of preinstalled system packages that we ship to all the projects to make the most common use cases possible. This includes compilers, development headers of common libraries, and others.

However, on one hand this makes our images way bigger than necessary, and it is still not enough to solve 100 % of the use cases. Fortunately, Read the Docs has supported the conda package manager for a long time already, and this has allowed users with special needs to add any non-Python libraries they needed. Still, interoperability between conda and pip is not perfect, and for users that are not used to conda this could feel like a hack.

New `apt_packages` configuration

To overcome all these problems, we have added a new configuration value, build.apt_packages, that receive a list of APT packages that will be installed in our Ubuntu-based images. Our configuration documentation contains a simple example:

version: 2

build:
  image: latest
  apt_packages:
    - libclang
    - cmake

(Notice that at the moment our images are based Ubuntu 18.04, this will change in the near future)

And you can draw inspiration from some community projects that are using this feature already:

geoserver-rest, a Python library to interact with GeoServer, uses apt_packages to install some GDAL and pycurl dependencies.
UCX-Py, a Python interface for the low-level networking library UCX, installs extra dependencies to temporarily work around upstream issues.

We are happy that this feature is already being used minutes after being deployed and are looking forward to seeing more projects make use of it.

Remember you can always see the latest changes to our platforms in our Read the Docs Changelog and Ethical Ad Server Changelog.

Considering using Read the Docs for your next Sphinx or MkDocs project? Check out our documentation to get started!

API v3 is now stable

2021-02-16T00:00:00Z

We are excited to announce that our API v3 has reached a stable release, and is now available for all Read the Docs users. Since we announced the API v3 beta, we have been adding extra functionality and bug-fixing minor issues based on user feedback.

The new API v3 is not a fully replacement (yet!) of API v2, but we highly recommend using API v3 for all the new integrations. API v2 will be deprecated soon, though we don’t have a firm timeline for deprecation. We will alert users with our plans when we do.

New features

We’re excited about the following actions that API v3 makes possible:

Import new projects
Activate a version
Trigger builds
Check build status
Manage redirects

API v3 allows you to easily manage common tasks on your Read the Docs project and organization. These include:

Setting up documentation for a new project
Automating the release of a new version
Pausing a release if you detect a failed build
Automatically create redirects for changes files

If you want to know more about it, please check out the full APIv3 documentation.

Help us make it better

We always love to hear from users about how they are using the API. We want to help you manage your documentation in the easiest and most efficient way possible. Please, feel free to open an issue in our issue tracker if there is a use case we are not covering.

Pull Request Builders available for all users

2021-01-27T00:00:00Z

We’re excited to announce that Pull Request building is now available for all Read the Docs users. We have been working on this feature for over a year, and having it available for all our users is a major milestone.

This feature allows users to confirm documentation builds correctly for all of their commits, not just ones merged into branches that are activated on Read the Docs. This moves documentation into your continuous integration pipeline, and improves the workflow for everyone working on documentation.

GitHub Build Status Reporting for Pull Requests

You can read more about this new feature in our previous blog post announcing it and in our documentation for the feature.

Serving docs for every branch

Along with building and validating each branch on your repository, we also host a rendered version of the documentation. This allows you to see exactly how the docs will look when they are deployed.

Building the documentation on Read the Docs during the development process also helps test integration with Read the Docs. The entire build process is run for each commit, so this will ensure that builds work as expected when you’re ready to merge into production.

These previews will stay live for 90 days from the time you merge or close the Pull Request, allowing you to finish any outstanding work on the task.

Limitations

Due to the increased demand for build resources, the Pull Request builders have the following limitations:

Read the Docs search is not enabled on these versions
Only HTML versions of the documentation are built (not PDF, ePub, etc.)
Documentation builds are concurrency limited. Read the Docs for Business concurrency varies based on your plan, and Read the Docs Community are limited to 2 concurrent builds.

We are working to address these limitations over time, but given the increase in scale with these builds it will take some time.

Give us feedback

We always love to hear from our users about the product. If you have feedback on this feature of any others, you can always reach us at hello@readthedocs.org.

Automation Rules

2020-05-04T00:00:00Z

A time ago we introduced a new feature to help users to automate some tasks on Read the Docs. Automation rules.

If you manage a project with several versions, you may have noticed that Read the Docs doesn’t always activate your new versions [1]. If you require to do any action over a new version, you’ll need to log in your Read the Docs account and manually do so.

With automation rules, you can choose what do to after a new version is created. To add a rule, you only need to choose a pattern and an action. After that, every new tag or branch that matches the pattern will trigger the action.

Currently, we only have two actions available:

Activate version
Set version as default

For the pattern, you can select:

All versions
The ones that match SemVer
Or you can write your own using a regular expression

With automation rules you can do things like:

Activate new tags only
Activate only branches that start with v
Activate tags and branches that belong to the 1.x release
Set as default tags that have the -stable or -release suffix
Activate all tags except those containing the --nightly suffix

Find more information and examples in our documentation.

What other tasks you would like to automate? Let us know on a GitHub issue!

[1]	We activate and build new versions automatically only if you follow PEP 440, and the new version is greater than the current stable version.

Announcing API v3 Beta

2019-08-27T00:00:00Z

In the last months, we have been working on making our API better. Considering the limitations of our current REST API v2, we decided to make a bigger step forward and create a new API v3, putting the focus on the use cases we heard about from existing users.

Compared to API v2, our new API v3 has some big differences that make it more user-friendly and useful.

These differences includes:

API access requires authentication
Token based API authentication
Read and write
Resource lookup can use project’s slug
Wider resource coverage

Some examples of new queries and actions that will be possible with the API are:

If you want to know more about it, please check out the full APIv3 documentation.

Invite-only beta feature

APIv3 is currently in an invite-only beta state on our community site, https://readthedocs.org. In case you want to test it and give us feedback on it, please email us to get early access.

Based on the feedback we receive from you, we plan to add more features to cover most of the common use cases. Please, help us with your feedback to make the new APIv3 even better!

GSOC 2019: Autobuild Documentation for Pull Requests

2019-08-14T00:00:00Z

Building documentation for pull requests is one of the most requested features of Read the Docs. Similar to how a continuous integration system runs a test suite on every pull request, this change would build the documentation for each pull request and send build status notification to the providers’ Status API (e.g. Github Status API). This will let users check if the documentation build passed and also how the documentation looks before merging it to master.

As a student of Google Summer of Code (GSoC) 2019, I (Maksudul Haque) was tasked with building this feature. The main goal of my project was to make it possible to build documentation whenever a pull request was created, and send build status notification to the Providers’ Status API.

To see this feature in action you can click on one of the pull requests on readthedocs.org repository.

GitHub Build Status Reporting for Pull Requests

Google Summer of Code

Google Summer of Code is a global program where students work with an open source organization on a 3 month programming project. Autobuild Documentation for Pull Requests was one of the project ideas on Read the Docs Project Ideas for Google Summer of Code 2019. I applied for the project and got accepted.

I started working on this project since May 2019. All of my work on this project can be seen in the Pull Request Builder Board.

Background

Many of our users wanted a way to visualize the documentation update that was made in a pull request. They also want to know whether the documentation build will pass before merging the pull request to master. This would allow users to have more confidence on the pull request and make the pull request less likely to break the documentation after merging. So, to achieve this Read the Docs core team selected Autobuild Documentation for Pull Requests as one of the projects of Google Summer of Code 2019.

Pull Request Builder Features

Currently I have implemented these features and working on more upcoming features. Some of the major features are as follows:

Creating External Versions: We create an external version when we receive a pull request webhook event for a project from GitHub and trigger a build for that version. External versions are short-lived versions that are separate from the project’s main documentation.
Synchronizing External Versions: Whenever there is a new commit on the pull request, we synchronize the external version for that pull request and trigger a new documentation build with the latest changes.
Deleting External Versions: Whenever the pull request is closed or merged, we delete the external Version associated with that pull request.
Warning Banner for pull request documentation: While building documentation for pull requests we add a warning banner at the top of those documentations to let the users know that this documentation was generated for pull requests and is not the main documentation for the project.

We send build status reports to the status API of the provider (e.g. GitHub). When a build is triggered for a pull request we send build pending notification with the build URL and after the build has finished we send success notification if the build succeeded without any error or failure notification if the build failed. By going to the build URL provided in the status report users can view the build steps and also see the documentation generated by that build.

Currently, we only support GitHub and planning to extend to GitLab and BitBucket.

Getting Started

Building documentation for pull requests is currently in Beta testing and only supports Github repositories. If you want to dive in and enable this feature for your project you can email us with the URL of your ReadtheDocs project.

Future Improvements

We are planing to extend this feature to other platforms such as GitLab and BitBucket. We are also planning to make this feature more customizable through our configuration file (.readthedocs.yaml). If you have any improvements or features in mind for building documentation for Pull requests we would love to know about it. Please feel free to let us know by submitting an issue on GitHub.

Contributors Wanted

As Read the Docs is an open source project backed by a small team of developers, most of them are busy to keep things up and running only. Therefore, its quite hard for them to take time to implement new features. We would like to get more contributors to improve this feature. If you know a bit of Django and Python and interested to improve this feature you are always welcome to contribute. If you need any support to start contributing, you can get in touch with me or any other member of Read the Docs team. You can find all of us at #readthedocs freenode IRC channel or readthedocs gitter channel. I am saadmk11 at IRC and @saadmk11 at gitter.

Conclusion

To conclude, I would like to say this was a much needed feature for Read the Docs and also its users. This feature will improve our platform and make it a true Continuous Documentation platform. I think that many users will benefit from this feature. We will keep making improvements along the way for a better user experience.

GSOC 2019: Improved Search Results and Search As You Type

2019-08-13T00:00:00Z

Giving users the ability to easily find the information that they are looking for has always been important for Read the Docs. This year, I, Vaibhav Gupta, took the opportunity provided by Google Summer of Code to improve the search. The main goals of my GSoC project were:

to make a Sphinx extension to provide “search you type” experience to the users.
to add support for multiple hits per search result.
to improve the UI/UX around code search.
to add support for search analytics.

Search as you type feature is live on our docs. You can go through these links to experience it:

Google Summer of Code

Google Summer of Code is a global program where students work with an open source organization on a 3-month programming project. I got accepted in the program and was excited to spend my summer with Read the Docs.

I have worked full time during the GSoC coding period and implemented various features. All of my search related work can be seen in the In-Doc Search UI Project Board.

Background

The search code was originally contributed by Rob Hudson, back in 2013 and then improved upon by other contributors. It was greatly improved and upgraded by Safwan Rahman during GSoC’18. Continuing on the same path, I have implemented some new features on top of the existing search backend.

New Features

During the GSoC period, I have worked on the following features:

In-Doc Search UI: readthedocs-sphinx-search is a sphinx extension that adds a clean and minimal search UI to your docs. It supports “search as you type” feature. So, it is now possible to get instant results without being redirected to any other page. Read the docs here.

In-Doc Search UI Demo

Multiple Hits Per Search Result: This is one of the highly requested features. We now support search results from the sections of the docs, clicking on which will take you to that particular section and not just to the top of the result page.
Code Search: We now support code search. If you want to search a particular function or an API endpoint – you can just type your query and you will find it in the results. Eg: api/v3/ or module.function.
Search Analytics: We now have support for search analytics. These analytics makes it easy to know what the users are looking for in your documentation. You can see these analytics in your project admin dashboard. Currently, this feature is in beta state and is available under a feature flag. We plan to make this available for everyone soon. If you want to test this feature out and help giving us feedback, please contact us via GitHub issues.

Search analytics demo dashboard

What Next?

We don’t intend to stop just yet. We are planning to work on some more cool features in the near future, some of which are:

Search Facets: Facets can be used to make search more accurate. For example: In Celery docs, facets can be used to search inside Kombu docs for “serializers”, like subproject: kombu serializers. (readthedocs/readthedocs.org#5966)
Search Results Ordered By Most Viewed Pages: It would be much more useful if the most viewed pages are shown first in the search results. (readthedocs/readthedocs.org#5968)
Search Inside Sections: It would be good if users have the option to get the search results from a particular section of the documentation. For example: Getting results from only Guides of our documentation. (readthedocs/readthedocs-sphinx-search#23).

Contributors Wanted

As Read the Docs is an open source project backed by a small team of developers, most of them are busy just keeping the site up and running. Therefore, it’s quite hard for them to take time to implement new features. If you know some bit of Django or Python and Elasticsearch, you can contribute to the search functionality of Read the Docs. If you need any support to start contributing, you can get in touch with me or any member of Read the Docs team. You can find all of us at #readthedocs freenode IRC channel or readthedocs gitter channel. I am dojutsu-user at IRC and @dojutsu-user at gitter.

Conclusion

These new features will make it much easier to find the relevant information in the docs. There are an infinite number of ways it can be improved and I believe we can compete with major search engines in terms of documentation searching. We don’t need superhero or coding guru, just need people who understand Python, Django and Elasticsearch and have some time to write some code for us. You are a Superhero to us if you can lend your time and effort to improve Read the Docs.

Read the Docs Blog - Posts tagged feature

Announcing Embed API v3 and sphinx-hoverxref 1.0

The value of content embedding

Embed API v3 and sphinx-hoverxref 1.0

Embedding content with sphinx-hoverxref

Calling the Embed API directly

Try it out!

Read the Docs ❤️ Jupyter Book

What is Jupyter Book?

Why a change was needed

How to deploy a Jupyter Book project to Read the Docs

Ubuntu 20.04, Python 3.10, and support for Node, Rust, and Go

Previous situation

New build YAML configuration

Install custom operating system packages (apt)

Previous solutions

New apt_packages configuration

API v3 is now stable

New features

Help us make it better

Pull Request Builders available for all users

Serving docs for every branch

Limitations

Give us feedback

Automation Rules

Announcing API v3 Beta

Invite-only beta feature

GSOC 2019: Autobuild Documentation for Pull Requests

Google Summer of Code

Background

Pull Request Builder Features

Getting Started

Future Improvements

Contributors Wanted

Conclusion

GSOC 2019: Improved Search Results and Search As You Type

Google Summer of Code

Background

New Features

What Next?

Contributors Wanted

Conclusion

New `build` YAML configuration

New `apt_packages` configuration