Read the Docs Blog - Posted in 2016

Announcing pydoc.io beta

2016-11-17T00:00:00Z

Today we’re excited to announce our latest project: https://pydoc.io. This work was made possible by the MOSS Grant from Mozilla. Thanks to Mozilla for funding our time building this wonderful community resource.

About pydoc.io

Running Read the Docs, we’ve always been proud of the documentation tooling that the Python community has. We prioritize prose over API documentation listings, and generally have a high standard of documentation in our projects.

There are a specific set of use cases that API reference documentation support, and the Python community doesn’t support them well. Tools like http://godoc.org and http://rubydoc.info provide these for others languages, and we now introduce http://pydoc.io as the Python community’s version of that.

What is it?

Pydoc.io will eventually auto-generate API reference documentation for every package on PyPI. Our beta release will support a limited set of packages that we are using for testing. Over time we’ll expand the tooling to automatically support generating docs for any package posted on PyPI.

Reference documentation is wonderful for folks who already know how a library works, and simply want to use it. It’s also great for navigating and understanding the internals of a piece of code. Giving them simple access to nicely rendered API documentation supports using and understanding libraries in a nicer way than reading code. We’re happy to build out these tools, along with the wonderful prose tools we’ve always had with Sphinx itself.

Our progress and some context is tracked in issue 1957.

The tools

This project is built on top of Sphinx at it’s core. On top of that, we’ve layered a few different tools:

sphinx-autoapi which is the tool that takes Python source files and turns them into rST.
- Internally, this uses the pydocstyle AST parser, which we use for a parse-only representation of the Python files.
pydoc.io is a Django application that is the actual web front-end and documentation building code.
sphinx-apitheme is the Sphinx theme that powers the actual API listing pages.

The biggest piece to call out here is the parse-only documentation generation. Sphinx’s autodoc module, and most Python documentation generators rely on importing the code to get docstrings. As we’ve learned with Read the Docs, this is quite impractical for a large set of libraries, including the scientific ecosystem that heavily relies on C libraries. Building and supporting a parse-only API tool makes doc builds faster and more reliable.

We have built on top of the pydocstyle AST, but it looks like we’ll likely have to build out our own AST traversal to retrieve all docstrings and construct declarations that we need to output. We have looked into some of the other AST options in Python, but haven’t decided on what direction we want to go when building our own.

We have put a good deal of work into all these different pieces, but there is still a lot of work to be done.

Give us feedback

This is a very beta release. We have developed a few different sets of tools, and would love your feedback. In particular:

Give us examples of API documentation patterns that you like. We’d like to incorporate powerful patterns into our theme.
Don’t give us feedback on specific design issues with the theme, we’re still working on it!
Let us know if you have ideas about how we can improve the general usability of the documentation
Let us know features that you wish the site had

Thanks

Thanks again to Mozilla, who funded this work. It wouldn’t have been possible without them, and we’re happy they funded the initial development. We’re always looking at ways to make the ongoing maintenance sustainable, so please reach out to us if you’d like to help with our efforts!

Running ads with Sentry

2016-10-24T00:00:00Z

We’re excited to announce our first partner in our Ethical Advertising campaign: Sentry.

Sentry was a natural first partner, both because they have a long history supporting and contributing to the open source community, and because they also strive to support open source while building a sustainable business around their project.

Sentry has built a sustainable business built around a 100% open source project, with all components of their stack available under a developer (and company) friendly license. The company also continue to support the community beyond its own contributions. It’s backed projects like Django, the Django REST framework, Vue.js, and Webpack, as well as providing Sentry free for all open source projects.

We are excited to be working with companies that see the value in open source and treating our users with respect. Making a commitment to do advertising in an ethical manner allows us to feel good about serving ads, and allows users to know their privacy isn’t being violated.

Sentry is a product that allows developers to capture and debug their production errors. It allows you to proactively respond to issues on your site, keeping both your users and your ops team happy. We’ve been using Sentry on our projects here at Read the Docs for a while now, and it’s a great way to handle errors in our production Django environment. We wholeheartedly recommend their product, and thank them for their sponsorship.

Securing Subdomains

2016-04-27T00:00:00Z

Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard.

Changes to provide security against broader threats have been in place for a while, however there are still a few scenarios that can only be addressed by migrating to a separate domain.

We implemented session hijacking detection and took precautions to limit cookie usage, but there are still a number of scenarios utilizing XSS and CSRF attacks that we aren’t able to protect against while hosting documentation from subdomains on the readthedocs.org domain. Moving documentation hosting to a separate domain will provide more complete isolation between the two user interfaces.

Projects will automatically be redirected, and this redirect will remain in place for the foreseeable future. Still, you should plan on updating links to your documentation after the new domain goes live.

If you notice any problems with the changes, feel free to open an issue on our issue tracker: http://github.com/rtfd/readthedocs.org/issues. If you do notice any security issues, contact us at security@readthedocs.org with more information.

Keep on documenting, The Read the Docs Team

Advertising on Read the Docs Community Sites

2016-03-09T00:00:00Z

On Monday, we enabled an ad space on documentation pages hosted on readthedocs.org. We have tested the ad space in the past, and had followed up with a discussion of the implications of advertising. The space was meant to be totally unobtrusive to the browsing and usage of documentation pages, and doesn’t enable third-party tracking of users in any way. This is all in an effort to help raise funds to support continued support and maintenance on the project, and ensure free and open documentation stays available.

Sustainability

Read the Docs is a massive project, and requires a large amount of work to keep running and support. We receive support requests and must tend to the operations of the servers on a daily basis. Neither of these tasks lend to volunteer contributions, they both require dedicated support roles for such a large site. I’ve personally been wearing a pager in support of the project for over 4 years, all on a volunteer basis.

Though we have done multiple donation drives, and have enabled other ways for users to support us, these only supply a limited amount of funding. Our users are the last ones we want to charge, they are other open source projects who aren’t paid themselves! This puts us in a hard situation. Selling a small amount of documentation reader attention allows us to sustainably support Read the Docs without affecting authors.

No Tracking

We firmly believe that the major issue with the advertising industry isn’t advertising, but tracking of users and third-party scripts. To address this, we don’t allow any tracking of our users by third-party scripts.

We do record the number of impressions each image gets from our server logs, and we track clicks with Google Analytics, which only we have access to.

Opting Out

We have added multiple ways to opt out of the advertising on Read the Docs. Inside the Admin > Advertising section of every project, you can see options to control or remove advertising on your project. These options include:

Supporting us financially with Read the Docs Gold.
Supporting us with your time by contributing to the project.
Moving to our paid product over at readthedocs.com.
Opting out without doing any of the above. This will make us a little sad, but we understand not everyone has the means to contribute back.

The Future

In a perfect world we wouldn’t include advertising on our pages. If you are at a company or another institution that would like to fund operations, support, and development of the project, please contact us!

For the foreseeable future, supporting the Read the Docs community site with advertising is our most viable model. We hope that you understand our thoughts here, and we’re always happy to have feedback or to field new ideas that might allow us to not require this in the future!

Read the Docs 2015 Stats

2016-01-15T00:00:00Z

2015 has been another great year for Read the Docs. We’ve addressed some long-standing issues like not having Markdown support, and built a number of wonderful tools for the documentation community.

Note

You can always see our stats for the last 30 days.

Our posts from 2013 and 2014 are also available.

Community

This year, we had:

27 people who committed code (-41% from last year)
1931 commits (+54%)
800 issues - 630 closed, 170 open (+60%)

We have a much lower number of people who committed to the code-base this year, while increasing the number of commits and issues by over 50%. This is mainly because we worked with Gregor for three months after our fundraiser.

The lack of committers is sad though, and I hope that we can get more people involved with the project over time. It really is unsustainable at the current level of community involvement. All of our operations and user support tasks are done by vanishingly small number of volunteers, with no new growth in that group of folks.

Interested in contributing? We have documentation on contributing on the site. You can also contribute financially at our sustainability page or by buying a monthly Gold subscription.

Page Views

Our stats:

170 Million Page Views (+55%)
38 Million Unique Visitors (+58%)

Page view growth has been steady again this year. It’s great to see so many people reading documentation that we are hosting.

We remain one of the largest sites on the internet, and that doesn’t account for traffic to CNAME’s, which is another 40% of our traffic.

Site Stats

The stats, in total numbers:

28000 projects (+85%)
39001 users (+75%)

Read the Docs has some high profile projects that push a lot of traffic. There are however thousands of smaller libraries and projects that fill out that full range of documentation that we host. We are happy to host documentation for all open source projects, and are glad the community finds the services useful.

Funding

Our hosting costs are sponsored by Rackspace, which is fantastically generous of them.

Development on Read the Docs is funded by the community. We launched a Sustainabilty Campaign earlier this year that funded work for 3 months. It was great to have help with our support workflow.

We are launching a new user-supported funding model this year, and we hope to be able to bring someone on to work on support in a more permanent basis.

Conclusion

2015 has been a year of mixed result. We have continued growing and serving more traffic. However, our contributor base shrank, and we haven’t been doing a good job of supporting users in the second half of the year.

We hope that with renewed funding we’ll be able to better support users in 2016.