Read the Docs Blog - Posts tagged gsoc

GSOC 2019: Autobuild Documentation for Pull Requests

2019-08-14T00:00:00Z

Building documentation for pull requests is one of the most requested features of Read the Docs. Similar to how a continuous integration system runs a test suite on every pull request, this change would build the documentation for each pull request and send build status notification to the providers’ Status API (e.g. Github Status API). This will let users check if the documentation build passed and also how the documentation looks before merging it to master.

As a student of Google Summer of Code (GSoC) 2019, I (Maksudul Haque) was tasked with building this feature. The main goal of my project was to make it possible to build documentation whenever a pull request was created, and send build status notification to the Providers’ Status API.

To see this feature in action you can click on one of the pull requests on readthedocs.org repository.

GitHub Build Status Reporting for Pull Requests

Google Summer of Code

Google Summer of Code is a global program where students work with an open source organization on a 3 month programming project. Autobuild Documentation for Pull Requests was one of the project ideas on Read the Docs Project Ideas for Google Summer of Code 2019. I applied for the project and got accepted.

I started working on this project since May 2019. All of my work on this project can be seen in the Pull Request Builder Board.

Background

Many of our users wanted a way to visualize the documentation update that was made in a pull request. They also want to know whether the documentation build will pass before merging the pull request to master. This would allow users to have more confidence on the pull request and make the pull request less likely to break the documentation after merging. So, to achieve this Read the Docs core team selected Autobuild Documentation for Pull Requests as one of the projects of Google Summer of Code 2019.

Pull Request Builder Features

Currently I have implemented these features and working on more upcoming features. Some of the major features are as follows:

Creating External Versions: We create an external version when we receive a pull request webhook event for a project from GitHub and trigger a build for that version. External versions are short-lived versions that are separate from the project’s main documentation.
Synchronizing External Versions: Whenever there is a new commit on the pull request, we synchronize the external version for that pull request and trigger a new documentation build with the latest changes.
Deleting External Versions: Whenever the pull request is closed or merged, we delete the external Version associated with that pull request.
Warning Banner for pull request documentation: While building documentation for pull requests we add a warning banner at the top of those documentations to let the users know that this documentation was generated for pull requests and is not the main documentation for the project.

We send build status reports to the status API of the provider (e.g. GitHub). When a build is triggered for a pull request we send build pending notification with the build URL and after the build has finished we send success notification if the build succeeded without any error or failure notification if the build failed. By going to the build URL provided in the status report users can view the build steps and also see the documentation generated by that build.

Currently, we only support GitHub and planning to extend to GitLab and BitBucket.

Getting Started

Building documentation for pull requests is currently in Beta testing and only supports Github repositories. If you want to dive in and enable this feature for your project you can email us with the URL of your ReadtheDocs project.

Future Improvements

We are planing to extend this feature to other platforms such as GitLab and BitBucket. We are also planning to make this feature more customizable through our configuration file (.readthedocs.yaml). If you have any improvements or features in mind for building documentation for Pull requests we would love to know about it. Please feel free to let us know by submitting an issue on GitHub.

Contributors Wanted

As Read the Docs is an open source project backed by a small team of developers, most of them are busy to keep things up and running only. Therefore, its quite hard for them to take time to implement new features. We would like to get more contributors to improve this feature. If you know a bit of Django and Python and interested to improve this feature you are always welcome to contribute. If you need any support to start contributing, you can get in touch with me or any other member of Read the Docs team. You can find all of us at #readthedocs freenode IRC channel or readthedocs gitter channel. I am saadmk11 at IRC and @saadmk11 at gitter.

Conclusion

To conclude, I would like to say this was a much needed feature for Read the Docs and also its users. This feature will improve our platform and make it a true Continuous Documentation platform. I think that many users will benefit from this feature. We will keep making improvements along the way for a better user experience.

GSOC 2019: Improved Search Results and Search As You Type

2019-08-13T00:00:00Z

Giving users the ability to easily find the information that they are looking for has always been important for Read the Docs. This year, I, Vaibhav Gupta, took the opportunity provided by Google Summer of Code to improve the search. The main goals of my GSoC project were:

to make a Sphinx extension to provide “search you type” experience to the users.
to add support for multiple hits per search result.
to improve the UI/UX around code search.
to add support for search analytics.

Search as you type feature is live on our docs. You can go through these links to experience it:

Google Summer of Code

Google Summer of Code is a global program where students work with an open source organization on a 3-month programming project. I got accepted in the program and was excited to spend my summer with Read the Docs.

I have worked full time during the GSoC coding period and implemented various features. All of my search related work can be seen in the In-Doc Search UI Project Board.

Background

The search code was originally contributed by Rob Hudson, back in 2013 and then improved upon by other contributors. It was greatly improved and upgraded by Safwan Rahman during GSoC’18. Continuing on the same path, I have implemented some new features on top of the existing search backend.

New Features

During the GSoC period, I have worked on the following features:

In-Doc Search UI: readthedocs-sphinx-search is a sphinx extension that adds a clean and minimal search UI to your docs. It supports “search as you type” feature. So, it is now possible to get instant results without being redirected to any other page. Read the docs here.

In-Doc Search UI Demo

Multiple Hits Per Search Result: This is one of the highly requested features. We now support search results from the sections of the docs, clicking on which will take you to that particular section and not just to the top of the result page.
Code Search: We now support code search. If you want to search a particular function or an API endpoint – you can just type your query and you will find it in the results. Eg: api/v3/ or module.function.
Search Analytics: We now have support for search analytics. These analytics makes it easy to know what the users are looking for in your documentation. You can see these analytics in your project admin dashboard. Currently, this feature is in beta state and is available under a feature flag. We plan to make this available for everyone soon. If you want to test this feature out and help giving us feedback, please contact us via GitHub issues.

Search analytics demo dashboard

What Next?

We don’t intend to stop just yet. We are planning to work on some more cool features in the near future, some of which are:

Search Facets: Facets can be used to make search more accurate. For example: In Celery docs, facets can be used to search inside Kombu docs for “serializers”, like subproject: kombu serializers. (readthedocs/readthedocs.org#5966)
Search Results Ordered By Most Viewed Pages: It would be much more useful if the most viewed pages are shown first in the search results. (readthedocs/readthedocs.org#5968)
Search Inside Sections: It would be good if users have the option to get the search results from a particular section of the documentation. For example: Getting results from only Guides of our documentation. (readthedocs/readthedocs-sphinx-search#23).

Contributors Wanted

As Read the Docs is an open source project backed by a small team of developers, most of them are busy just keeping the site up and running. Therefore, it’s quite hard for them to take time to implement new features. If you know some bit of Django or Python and Elasticsearch, you can contribute to the search functionality of Read the Docs. If you need any support to start contributing, you can get in touch with me or any member of Read the Docs team. You can find all of us at #readthedocs freenode IRC channel or readthedocs gitter channel. I am dojutsu-user at IRC and @dojutsu-user at gitter.

Conclusion

These new features will make it much easier to find the relevant information in the docs. There are an infinite number of ways it can be improved and I believe we can compete with major search engines in terms of documentation searching. We don’t need superhero or coding guru, just need people who understand Python, Django and Elasticsearch and have some time to write some code for us. You are a Superhero to us if you can lend your time and effort to improve Read the Docs.

Improved Search

2019-02-11T00:00:00Z

Have you ever struggled with a poorly documented software project? What about a well documented project but you can’t find the right section inside the docs? The Read the Docs core team has realized the importance of good search for documentation and got me to take the challenge as a Google Summer of Code student. The main goal of my GSoC project was to refactor the search code together with upgrading the backend search engine, as well as adding more features to our search functionality like exact match search, case insensitive search, search as you type, suggestions and more.

Google Summer of Code

Google Summer of Code is a global program where students work with an open source organization on a 3 month programming project. The core team of Read the Docs proposed some Project Ideas, one of them was Refactor & improve our search code. I (Safwan Rahman) was keen to get my hands dirty with Elasticsearch and grasped the opportunity to do so by applying for this project and I got accepted.

I have worked full time for from April to August to upgrade the whole codebase to compatible with Elasticsearch 6.x and also implemented various features like:

Together with this, we are planning more features like the following:

All of my search related work can be seen in the Search Project Board.

Background

Search is a vital part of any documentation hosting platform, so people can get the information they need. As a documentation hosting platform, the same rules apply to Read the Docs. Because of having a small core team, the search functionality of Read the Docs has lagged behind for quite a while now. Initially the search code was voluntarily contributed by Rob Hudson, back in 2013 and then improved by other contributors. The search infrastructure was already outdated as Read the Docs were using Elasticsearch 1.3.x which was already reached its End of Life in 2016. Therefore, Upgrading the search infrastructure was badly needed.

Built in Search vs Read the Docs Search

Both Sphinx and MkDocs already have built in search functionality. But the features are very limited. At Read the Docs, we have felt the limitations and therefore we index the documentations in our Elasticsearch index so that we can provide better search experience like:

Search across multiple projects
Advanced query syntax
Search inside subprojects
Improved search result order
Public Search API (Documentation pending)

Currently, we do not index MkDocs documents with Elasticsearch, but any kind of help is welcome.

New Features

In the 4 months of full time work, I have implemented these features along with many bug fixes. Some of the major features are as following:

Exact Matching Search: Exact matching is one of our most highly requested features for Read the Docs. Now you can search for exact word in documentation by having your query inside quotation (""). So if you search for "Here is foo" (with the quotation), you will get all the documentation where the full Here is foo phrase exists.
Simple Query String Syntax support: Now you can use Simple Query String Syntax for searching. For example, if you search with Mozilla +-Firefox, you will get documentation where the Mozilla word is present, but Firefox word is not present. For more information, you can look at the Simple Query String Syntax documentation.
Case Insensitive Search: The search is now case insensitive. So if you search for Foo, you will get all the documentation which have one of either Foo, FOO, foo or fOO.
Improved Result order: The result order is improved dramatically. Therefore if you Search with Foo Bar, you will first see the documentation which have both Foo Bar, then you will see other documentation which have either Foo or Bar
Auto Removing from Search Index: In the past, if any page got removed from documentation, the page was still available in documentation. But from now on, the page will be removed automatically from the search index as soon as its no longer in the documentation.
Zero Downtime Indexing: As search is an important part of documentation, there will be no downtime while we reindex our Elasticsearch Index. So the search will be much more reliable than before.

Code Improvements

Code quality is very important in development world, specially in open source. As I have rewritten the search functionality from scratch, the code quality is improved in many ways like test coverage and documentation. So its easy for any contributor to start working on the search functionality

Contributors Wanted

As Read the Docs is an open source project backed by a small team of developers, most of them are busy to keep things up and running only. Therefore, its quite hard for them to take time to implement new features. If you know some bit of Django or Python and Elasticsearch, you can contribute into the search functionality of Read the Docs. If you need any support to start contributing, you can get in touch with me or any member of Read the Docs team. You can find all of us at #readthedocs freenode IRC channel or readthedocs gitter channel. I am safwan at IRC and @safwanrahman at gitter.

Conclusion

To conclude, I must say that the Search improvement in Read the Docs was very necessary and I’m glad I could improve it in such a short amount of time. There are an infinite number of ways it can be improved and I believe we can compete with major search engines in terms of documentation searching. Due to the constraints of only working for three months, a number of compelling features were left out such as Search as You Type and Autocomplete and Code Search functionality. Moreover, proper documentation is needed for the search architecture. I have tried to write test cases for most of the scenario, but because of time constrains, a lot of code is out of test coverage.

I strongly hope that we will get the left behind work done within a short amount of time. This can be done easily if we get more contributors donate their time for improving Read the Docs. We don’t need superhero or coding guru, just need people who understand Python, Django and Elasticsearch and have some time to write some code for us. You are a Superhero to us if you can lend your time and effort to improve Read the Docs.