Read the Docs Blog - Posts tagged report

State of the Docs

2015-09-09T00:00:00Z

August has historical significance for Read the Docs, so it seemed fitting to wrap up August by taking a moment to step back and reflect on what we’ve done in the last year.

What makes August so significant for us?

For starters, this August saw the 5th birthday of Read the Docs! We’re still humbled by the project’s success, and the important role it continues to play for old and new users alike.

This growth was becoming difficult to manage without focused support given back to the community. Though the project has always had core maintainers, for Read the Docs to continue growing as a community resource, we knew it required dedicated attention.

This is the reason why, in August of last year, Eric and I set out to create a company around Read the Docs – another significant bookmark in Read the Docs’ history. We planned to build the company to fund the ongoing support efforts of the community site and to help accelerate its growth. At this time last year, we had just formed Read the Docs, Inc, we were fresh into the PIE accelerator program, and we were planning how to bring our documentation values to companies that weren’t open source.

The last year

Our primary goal has always been to drive the development of our community site, with hopes that it would one day be self-sustaining. We made a number of decisions and changes to work towards this goal in the last year:

Treat the community site as a separate entity

In order to be independently sustainable, the community site needs income of its own. The service is large enough now that dedicated staff is a necessity. There needs to be someone to drive development, handle support requests, and be accountable for keeping everything running.

Currently, all money coming into Read the Docs is treated as income into the business. We do track income to the community site separately, so we can budget for the community site separately from running the rest of the business.

We aren’t yet to the point where this split is critical. There are no employees of the business yet, and we are only able to pay ourselves for a small portion of the work we perform for the community site.

We will need to track this explicitly as we grow however, as the community site will need to rely on donations for the foreseeable future. Those donations should not go towards developing the for-profit side directly. Development of the community site will still be a primary function of income into the for-profit side.

Eventually, the community site could be structured completely separate, to provide a more explicit split. It could probably be structured as a 501c, This doesn’t make financial sense yet.

Our support fundraiser

In an effort to fund development and maintenance on the community site, we decided to call on our community for financial support back in June. This was the first time we have asked the community for contributions. Our goal was to fund at least a portion of the time spent developing and supporting the community site.

Thanks to all the contributors from our community, we raised our goal of $24,000, and were able to fund the time of multiple engineers working steadily on Read the Docs for the last few months.

We already outlined our costs and our plans behind our June fundraiser in a previous post. We will be following up the last few months of work with an overall review of the fundraiser and the work we performed in the coming weeks.

Triaging

Hopefully, our support responsiveness has been more apparent. One hurdle in bringing on more engineers, and engaging the community more, was the lack of formal ticketing and open decision making. Gregor, among many other changes, laid out a formal triage process that has been a major influence in keeping an organized and healthy support queue.

We’ve definitely found that the more support engagement we provide the community, the more support comes out of the woodwork. This is all part of supporting a healthy community however, as we have learned.

Stability

Read the Docs has “grown organically”, which is to say it has grown fast, unwieldy, and in many directions. With more proper focus on code quality, including clean up efforts, refactoring, and long due upgrades, Read the Docs has shed considerable technical debt in the last couple months. This work has already paid off in improving stability and predictability of the project.

Ads

We ran a few tests with advertising on built documentation in the last few months. We discussed at length how we felt about ads and determined our own problems with the standard implementation of ads is not the ads themselves, but the breach of privacy they signify.

Because of that, we established that a requirement of us ever showing ads is to not use third party tracking. In fact, we look to remove all third party tracking from built documentation as soon as we can.

To put ads into a clearer context, revenue from in-house ads fully cover our costs each month, without asking the community for any further support. It probably shouldn’t be surprising that advertising revenue is an easier income source than seeking donations. We certainly feel Read the Docs provides more value than the ads it would serve though. Our last fundraiser reflected the value the community puts on the site, and we hope the next fundraiser can be as successful.

Should we ever use ads as a primary revenue source for the community site, we have established some rules:

We will provide our ads in-house, not through any service that establishes third party tracking
We will not sell any data about our users
We will always prefer ad sponsorship from community members
Users that support us through donations or sponsorship are opt out of ads

Feedback regarding ads has been positive, hopefully our intentions with ads has been mostly apparent. We’ll be discussing our thoughts on advertising in more depth soon, if you have any feedback.

Gold

We’ve begun working on a “gold” status feature for users that want to support us each month. This will be an optional, recurring subscription to Read the Docs. The subscription will provide some additional features for users or projects, and some advanced project features might suggest a gold subscription for projects that make use of them.

This idea is still evolving, we’ll be asking for more feedback as this feature matures.

The next year

So, let us now set some expectations for the next year forward:

Transparency and feedback

Operating in the open has been an important tenant to our company since formation. We believe whole-heartedly in the trust and candid feedback it provides, and we have pledged to keep openness and transparency at the core of the company.

Part of the structure that has guided us as a company has been providing updates to those that we are accountable to. During our time at PIE, this was to our mentors and our peers. Now, we want to be accountable to you, our community.

We would like to continue the periodic updates that started with the fundraiser, and we plan on releasing updates, on a quarterly basis, that share more insights into the company. We will continue to expose our goals, problems, and decisions to you as we grow and we hope you can provide us with your feedback.

Continued work towards stability: We still have some technical debt that will take a while to pay off, but ensuring stability is paramount to having a service we can all rely on.
Support responsiveness: We will adhere as much as we can to the processes that we established around our support channels. We hope that these processes will also allow others in the community to easily fill these roles should we need the help.

As for our technical goals, outside our roadmap and continued cleanup and maintenance, here is what we have on our roadmap:

User experience: With components becoming more stable, and a generally more stable code base, we are shifting focus towards user experience – the pieces most obvious to users. We’ve made some gains here in the past few months, but have a ways to go still.
Easier documentation build tools: Sphinx is an incredibly powerful tool, but for new users this can be overwhelming. We are developing tooling to ease this process for new users and provide a solid footing for the majority of use cases. This will also help to make the build process more predictable, as this would mimic our build backend processes closely.
More reference doc language support: We have been working towards wider language support for generating API reference documentation. Most recently, we worked with Microsoft to develop support for .NET language support in Sphinx. This allows authors to use existing .NET tooling to generate reStructuredText API reference documentation.

Let us know

If you have any feedback or opinions on any of this, continue to let us know. We’re happy to have the feedback.

Report - July 2015

2015-09-09T00:00:00Z

August was a hectic month for us. So busy, in fact, we never wrote this update. We dropped the ball on getting this report out on time, but hope to keep on top of future updates.

Goals

July was a month of stability fixes, with some substantial changes to our infrastructure. First, here’s an update on some of the goals we had set with our last monthly update:

Improve Github/Bitbucket integration

This got pushed to next month, we didn’t focus on this as much as we had hoped.

Test internal analytics

We have a basic testbed for internal analytics set up, but haven’t gone as far as pushing this integration into production. We still need to test production load and what the dataset would look like over the course of a month.

This is on hold for the moment.

We also set some additional goals near the beginning of the month:

Finish rolling out a new stack

Our servers were all reasonably aged and have not been re-provisioned in a long while. We set the goal to move all configuration to configuration management and be able to reproduce servers in a predictable manner.

We were able to finish moving all provisioning to configuration management and rolled out all new servers, including some new servers.

Improve monitoring

We have hit a lot of common cases that we should be tracking as part of our standard operations notifications. We set the goal to improve our notification and debugging capabilities.

With fresh provisioning, we now also have a programmatic approach towards setting up system checks and notifications, and have expanded our reporting a great deal.

Update Details

Ops

Fresh web servers

Last month, we rolled out fresh build servers, strictly using configuration management to provision servers. This month, we continued by moving web server provision to our configuration management and re-rolling all of our web servers.

Fresh database servers and migration

We also rolled new database servers, and took the opportunity to upgrade our database’s PostgreSQL instance and perform some needed schema changes.

We decided to perform the migration late one evening, when our then primary database server was included in an outage of some Rackspace infrastructure. The outage resulted in some extended downtime, and we were not immediately able to migrate data from the database server to the fresh instance. The outage resolved early in the morning and migration wrapped shortly after.

The new instance performed better than expected, once we made it live. The fresh instance had more memory and increased bandwidth for disk operations. Along with a reindex of a table with slow queries, we noticed significant reduction in query times and a great improvement of stability.

Improved monitoring

We started to catch some edge cases that were not previously monitored and improved greatly on our monitoring and graphing.

Dev

Django 1.8 upgrade: Our previous version of Django was on the now unsupported 1.6 branch. With 1.8 released, we felt it was a great chance to finally upgrade our dependencies.
Triage process: We have formalized our triage process for development. This will guide how we make decisions on our issue tracker.

Linting: Linting has not been performed on the codebase for a long while. We felt this was important to get back to in order to ensure code quality.
Stable version improvements: There has been some buggy behavior behind the stable version detection. We worked here to clean up some of the broken behavior.
Fix slugging on versions: There have been a number of error cases that have surfaced around version slugging recently. This works to fix some logic errors on edge cases.

For more information on our development orchestration, visit or subscribe to our public Trello board.

Issues

In the month of July:

Read the Docs saw 80 new issues

50 of those issues have been closed

Additionally, 35 outstanding issues were closed out

Feel free to raise any questions you have about any of these changes on our issue tracker, or on our Freenode IRC channel, #readthedocs.

Goals for Next Month

Improve Github/Bitbucket integration: We have pushed this up to next month, we still need to address some failures as part of Github/Bitbucket integration.
Better Sphinx experience for authoring: We can streamline a lot of the experience around using Sphinx, and at the same time, make more reproduceable builds, locally, for authors. This would involve mimicking more of what we preform in our build backend for local authoring.
More code cleanup: We started some linting efforts to bring code quality up, we will continue deeper and more thorough linting efforts.
More features to user gold subscription: We would like to have more users on a recurring subscription, which is a great step towards sustainability. We should be surfacing features and reminding users to donate if they are using some of those features.

Report - August 2015

2015-09-09T00:00:00Z

August was a busy month for us. Write the Docs Europe was at the end of the month, and our focus has been on getting readthedocs.com into a public beta state. We’re nearing the end of the 3 month period that we budgeted for as part of our fundraiser and have a few more tasks to push out.

Goals

First, an update of the goals we set for this month:

Improve Github/Bitbucket integration: We fixed a very common bug caused by the way implemented syncing to the Github API. There are still some user experience issues we would like to address.
Better Sphinx experience for authoring: We started to refactor parts of our build backend and discussed what the optimal tool for authors would be. We don’t have a clear sense of where to split up our build backend code and what is extraneous to authors yet, however.
More code cleanup: Linting cleanup progressed, and we are on to discussing stricter linting efforts.
More features to user gold subscription: We have added the ability to “bless” projects with a gold subscription, but are still working on the user experience behind reminding users to donate via gold subscriptions, nor are we explaining gold to users yet.

Update Details

Build backend refactor

The builder backend code was refactored and now all build commands executed are tracked and associated with builds. This surfaced a number of hidden commands and a few failure conditions that were being disregarded.

This also removes the confusing magic PDF build that gets triggered from a normal build object.

Build output page

With the build refactor tracking all commands separately, we can now show output of all the commands executed. Before this output was split into output and error output streams, and it wasn’t always easy to distinguish failures. The build output page will now also live update through the API as changes become available.

Improved Github Syncing

Our initial attempt at Github integration updated from the Github API in-process, which could result in a timeout when the API didn’t return quick enough. Syncing against the API was moved to our task queue, with an API to monitor changes to the task.

Started work on more automated build toolchain

We have plans to make the process for building Sphinx documentation more automated for authors. We started laying out plans for what that tool would look like and how it would function.

For more information on our development orchestration, visit or subscribe to our public Trello board.

Issues

In the month of August:

Read the Docs saw 55 new issues

36 of those issues have been closed

Additionally, 59 outstanding issues were closed out

Feel free to raise any questions you have about any of these changes on our issue tracker, or on our Freenode IRC channel, #readthedocs.

Goals for Next Month

Improve gold subscriptions more: With the fundraiser funds nearly allocated, we need to begin budgeting for the next few months, and will need to start bringing in income. We want to improve the user experience around donating, as well as present more reminders to donate to active users.
Increase income for the community site: With better user experience around donating, we hope to launch another effort to get more users and companies to donate.
Improve import user experience: There are still some gritty parts to the import process that should be streamlined. Some of the major Github bugs have been squashed, but a larger UX improvement is still required.

Report - June 2015

2015-07-01T00:00:00Z

With the Write the Docs conference wrapped up in May, we shifted our focus back to Read the Docs for the remainder of May and June. Following our post on contract positions available with Read the Docs, we hired two contractors to work with us and focus on support and stability for readthedocs.org. Gregor Müllegger will be working on support and development, and Andrew Kreps has helped us on building and improving our operations infrastructure.

Updates

Read the Docs had the following major updates in June:

Ops

Elastic Search support was upgraded and our search cluster is now hosted on Found.

We greatly improved the Read the Docs infrastructure, including monitoring and alerting, as well as server environment consistency. Using Nagios and Graphite, as well as Salt, for this task.

Static assets were migrated to use a CDN, provided by MaxCDN.

We did a fresh provision of our build cluster using Salt. Previously, there were slight disparities between the servers, which surfaced some issues during builds.

Fixed Python 3 build environments with new virtualenv and pip.

Changed our queueing strategy to not be tied to a single specific build machine.

Dev

Output formats such as PDF and ePub are now configurable, but continue to default to on.

Added the ability to wipe the Build environment to the Versions page.

We started to address issues with stable version detection

Fixed issues related to version name slugging.

In order to move away from third party analytics, we have spun up an instance of Piwik. We plan on testing this with our production traffic in the coming months.

Fixed “Edit on GitHub” links for stable versions

General refactoring and cleanup of the codebase, and added test coverage.

For more information on our development orchestration, visit or subscribe to our public Trello board.

Issues

We have started to formalize on a ticket triage process and have more focus on performing project management in the open. This is a very welcome change to our existing workflow and hope to continue improving on it.

In the month of June:

Read the Docs saw 50 new issues in the month of June

26 of those issues have been closed

The remaining 24 have been triaged but remain unresolved

Additionally, 19 outstanding issues were closed out

Next Month

Once work is wrapped up on version detection cleanup, we hope to start improving our Github/Bitbucket integration and user experience issues around deleting old projects and versions from Read the Docs.

We will also be testing internal analytics in the coming months. Our stance is that user data, for both our users and for readers of hosted documentation, should not be handed over to third party services like Google Analytics. We hope to replace external analytics with an internally hosted service, so that your data remains with us.

Feel free to raise any questions you have about any of these changes on our issue tracker, or on our Freenode IRC channel, #readthedocs.