I’d like to identify ways in which Automattic can release more code as open source by default.
In the WordPress.com repository, there’s a fair amount of code in use that’s never seen the light of day. Bits and pieces of this code would probably be useful to other people, and subsequently be improved as more developers read, implemented, and found new uses for it.
Open source is a unique and tremendously important phenomenon because it enables people to create more economic value than they could with previous collaborative frameworks. It’s “one of the most important ideas of our generation.” Automattic as a company believes this too; on the first page of our internal company documentation, Matt Mullenweg says, “I’m fine with releasing basically any code on WordPress.com that isn’t our password files.”
However, we don’t release as much code as we could be releasing, nor do we go about many our projects in the open source way we admire. I get the sense others in the community may agree.
Part of the challenge, as I perceive it, is that we (Automattic) build new functionality for WordPress.com, are lazy in our implementation, and it then becomes extra work at the end to “release” the functionality as a plugin. People have limited time in the day, understandably, and the last step doesn’t get done.
Last month, I wrote an internal blog post identifying four ways I’d like to see Automattic improve:
- Releasing (or mirroring) as much of the WordPress.com codebase as possible.
- Building community around parts of WordPress.com infrastructure, both in terms of people who use it and people who develop for it.
- Incorporating more third-party code into our codebase, and actively contribute back to it.
- Modeling best practices for open source projects, including support and documentation.
There was good conversation in the comments and, in somewhat typical fashion, not a lot of clarity on the takeaways.
Personally, I’d like to get as much WordPress.com code into a repository that VIP developers can check out and use in their development environments. Hell, it even means they can submit patches for fixes, coincidentally as Erick Hitter just did while I’m writing this post. I’d like to move our existing, private shared plugins repository into a WordPress.com public plugins repository which would also contain code like the shortcodes we’ve implemented on WordPress.com.
Communication is the lifeblood to open source, and the only remedy to poor communication is over-communication. For an open source project, discussion should be public, ticketing should be public, and the changelog should be public. This obviously isn’t to say that everyone should have an equal voice — rather, everyone should have an equal opportunity to participate. Making public WordPress.com’s internal code would be a start, but that would just be the start.
What other approaches do you think we could take?
If you haven’t already, I’d highly encourage you to read Tom Preston-Warner’s “Open Source (Almost) Everything“. It’s legit.
16 Comments
Long-term any externally useful code will be encompassed by Jetpack, and I’d rather put any effort toward getting things into Jetpack rather than the intrinsically-interesting but user-poor goal of just opening up our internal repos. I agree that shortcodes would be a great place to work on feature parity and code unification.
I also think it’s important to be able to have candid discussions in tickets and commit messages without the pretense and scrutiny caused by publicity. Privacy is also important when we’re working on things that are unannounced, particularly with partners like Twitter or Facebook. Sometimes we make observations on third parties that are useful for our work but I wouldn’t publish. There are also, of course, security and anti-spam contexts where having our code and commits be open would be counter-productive. We have 51,000+ existing commits that would need to be inspected, scrubbed, or just thrown out (starting over again and losing valuable history we’ve built up). We’d have to split things up or contort the ways we work so much I think it’d ultimately nullify many of the benefits.
It’s much better to just start something new with the full intention and expectation that it’s public.
Context is extremely important for communications, just as you would have a different conversation at your family dinner table versus a ball game with your friends versus asking a question to POTUS at a Town Hall. The comments you get on this from Automatticians won’t be as good as the internal P2 ones because it’s essentially a public stage, and it’d be awkward to have a meaningful debate on an internal issue in public. (I’m not going to, either, after this comment.) It’s important for keeping conversation high-bandwidth that we have the luxury of a protected context, which we’ve evolved an extremely sophisticated way of doing.
Bearing the security concerns in mind, I can really appreciate the point you’re making. I have no doubt that there are many code snippets that would prove invaluable to both VIP and non-VIP developers alike.
I’ve encounter the situation regularly with the VIP work I’m doing, where transitioning from the open-source WordPress side to the black box of WP.com/VIP impedes efficient development. I find the transition most jarring because I’m accustomed to being able to read a function definition and trace back its constituent parts both as part of my development and learning processes. Perhaps the first step is to move more code into the VIP plugins repo, where authentication is still required, before making it available to the general public.
PS, thanks for the shoutout. Always happy to help.
As I mentioned on our internal discussion, I think writing (most) new code with the thinking that it will be immediately open-sourced is the way to go and then put it into Jetpack or release it as a snippet if it’s not full-blown code.
Open-sourcing much of our existing code just isn’t practical for a wide variety of reasons (hacky, one-off, not self-contained, etc.)
There are a lot of good *internal*-facing reasons for not releasing everything, many of which have been mentioned above. Hacky, not-self-contained, security, context sensitivity, would take too much work to maintain/support, etc. These reasons are “internal-facing” because they have to do with the way that Automattic manages its resources, where “resources” is broadly understood as labor, money, reputation, and so on.
From an *external* point of view, though – what’s best for the community of developers and users – I think the case is different. Matt calls the release-by-default strategy “intrinsically-interesting but user-poor”. I’m not really convinced. For one thing, Daniel’s suggestion in this post doesn’t seem to be “turn everything into a plugin in the .org repo”, but is instead something like “let’s let a particular circle of external devs have a look at our hacks”. The meaning of “user” in the former context is very different from its meaning in the latter context. The intended “user” for Daniel’s suggestion would be developers who would, presumably, know the difference between what’s production-ready and what’s not.
Moreover, and more importantly, the Jetpack-or-nothing strategy eschews many of the benefits of open source development communities. To wit: A couple of weeks ago I had a weird WordPress problem that required a custom hack. After I was done, I took an extra five minutes to write a docblock, throw it into a Github gist, and tweet about it – the message was something along the lines of “Use at your own risk”. Within a day, another member of the community had decided that my snippet was useful and important enough to turn into a plugin. He cleaned up my hack, abstracted it, built a UI, packaged it as a proper plugin, and put it in the wordpress.org repo. Of course, this doesn’t happen in all (or even most) cases. But if you commit to only releasing “finished” code that you’re ready to support as an end-product, this kind of thing will *never* happen. In so doing, you preclude the generative possibilities latent in a community of developers.
In the end, Automattic’s strategy (not *just* Automattic, of course – the point applies equally to anyone who is doing private client work) should be made based on a careful weighing of internal *and* external factors. It’s not obvious to me that there’s a clear right or wrong in every case. But I do think that this is a situation where what’s best for the community may not always be in line with what’s best for Automattic, which is a tension that should at least be recognized. (Plug! I wrote a post on a sorta-related topic just last week: http://teleogistic.net/2012/05/the-patronage-model-for-free-software-freelancers/)
Thanks for the provocation, Daniel!
That’s not true of everything though. Case in point: I just released a snippet (class) the other day: https://github.com/Automattic/WordPress-Zendesk-API
Automattic is extremely good at open sourcing it’s stuff (stating the obvious).
Off the top of my head, the only thing I can think of which I would like to see open sourced is the shortcodes, and a few of the widgets.
I had to replicate a bunch of the WordPress.com shortcode functionality a year ago which was a pain in the neck for me.
I think that “Open Sourcing” of private code is an inferior approach to writing it to be Open Source from the start. You make different decisions.
What I’d like is for more future Automattic projects to be designed to be Open Source. Even if not completely. There are ways of designing things so that the base code (the framework) is Open Source, but the things you privately build on top of it don’t have to be. Doing things this way will likely lead you to design better frameworks that will not only be externally useful, but will serve the company better. Underscores is a great example.
I believe this is the best approach. If you want the code to be open, have it open by default. I recommend considering moving your development to Github, which facilitates this change. While you can still have private repos, it makes being open by default much easier.
Reblogged this on Joachim Kudish and commented:
Great article by my coworker Daniel Bachhuver on how Automattic can open source more of its work.
I liked this comment by Mark Jaquith, which summarizes pretty well how I feel about it personally:
“What I’d like is for more future Automattic projects to be designed to be Open Source. Even if not completely. There are ways of designing things so that the base code (the framework) is Open Source, but the things you privately build on top of it don’t have to be. Doing things this way will likely lead you to design better frameworks that will not only be externally useful, but will serve the company better. Underscores is a great example.”
I think it would be great to open up the VIP lobby/plugins. Would love to be able to point people there for tools/best practices.
Totally agree. Some of the examples are specific to VIP, and others would benefit from being public documentation.