Using Pirate Metrics to measure success of open source projects

No readers like this yet.
"open" over an ocean
Image by:

Patrick Nouhailler. Modified by Opensource.com. CC BY-SA 2.0.

So you started your own open source project. You had a great idea, you actually took the time and energy to create a project on GitHub, and you even went so far as to set the right frameworks in place to allow other people to contribute to your community. Awesome!

So how is that going? Is the project succeeding? Can you see improvements from week to week? Are you getting closer to your goal? Do you have a goal? If these questions make you shuffle around awkwardly in your seat, read on. If you answered all of the questions confidently, read on anyway, because you might find yourself challenging some of your previous assumptions.

As a community manager of a relatively young open source project, I want to share some of our experiences and methodologies we use for measuring the success of our project. These ideas and techniques are a work in progress. Hopefully you can pick up some ideas from hearing what we do, and better yet, share some of your new ideas with us.

Is it all about contributions?

Here's a fact. Most open source projects have very few contributors. Donnie Berkholz wrote a great blog post analyzing about 50,000 open source projects with active communities. According to his analysis, "The vast majority of projects are tiny, having just one or a few contributors." Consider these statistics:

  • 87% of projects have five or fewer committers per year.
  • Merely 1% of projects have 50 or more committers per year, and a scant 0.1% have 200 or more.

So, assuming you're in the 99% of projects that has less than 50 committers a year, you might want to have other metrics to measure whether your project is on a good trajectory.

AARR! Introducing Pirate Metrics

Users engage with your project on many different levels. There's the guy who might have heard about your project while reading a blog post and stopped there. And then there's the crazy developer who's contributing code to your community three times a day. The point is that in between those two cases there are multiple levels of engagement.

Pirate Metrics is a simple and effective framework that helps you measure the success of your project at each significant level of user engagement. Pirate Metrics speaks about five general levels of user engagement:

At each level of engagement, you carry out activities to encourage user engagement and to move them to the next level. An important part of this process is understanding which activities are effective in encouraging more engagement.

Adopting Pirate Metrics to measure open source success

Pirate Metrics would typically be used for business startup initiatives that generally have the goal of increasing revenue. However, there is no rule that says revenue means money. Maybe the ultimate payment is code contribution to your software or becoming an active member of your community.

My team at CloudSlang adopted these Pirate Metric definitions to the world of open source, and we use these metrics to measure our success at each level of engagement. So, here are examples for how we define success at each level of engagement:

Notice the activities that are feeding into the engagement funnel. When we publish a blog post or present something at a meetup, we are interested in understanding the impact of these activities on user engagement. For example, a cool blog post that mentions your product may lead to a lot of unique site visits (acquisition), but those same visitors may be less interested in downloading your product (activation). On the other hand, when you present your project at a very relevant online meetup, you may have fewer website visits (acquisition), but a much higher percentage of downloads of your product (activation).

Measure and learn

Each week, we look at multiple metrics in each level of engagement and document the data (e.g., number of downloads). We document the unique outreach activities that we did during that same week (e.g., blog posts or new releases) in parallel. This way, we are able to actively measure ourselves each week and learn about which activities are effective in increasing engagement in our project.

Please share your feedback

Like I said, this methodology is a work in progress. I'd be really interested in hearing how other open source projects approach this subject. Please leave your feedback in the comments section, and together we can create a new generation of open source pirates.

Tags
Business
Community management
Gaby Fachler
User profile image.
Gaby Fachler is the open source community manager for the CloudSlang project (http://cloudslang.io). He is also an experienced IT Project Manager responsible for execution and governance of software deliveries across multiple domains and geographies.
More about me

1 Comment

Avatar
Ron McFarland | June 16, 2016
No readers like this yet.

That was a very interesting article on funneling contributors. It follows a very similar trend in sales when salesmen are looking for new customers. You have to contact a certain number of people just to find one person that is interested in you (your product or the project you're working on)

Two important points:

One should never give up in his efforts to find contributors (or customers). They can come from the strangest places that you could have never dream of initially.

Second, even though in the end you have only one, two or three people that do all the real work, the problem is you don't know who those people are when you start. Sometimes major contributors themselves don't know how much they can contribute until everyone starts working together. Then, as things progress, they explode with energy and contribution.

It is great that you have weekly review meetings to confirm if you are on track or not. I have found doing sales training that some people work very hard looking for customers using a poorly developed customer list (They work very hard for few sales.) Others have a good customer list, but are too lazy to contact anyone. (They too generate few sales.) It is ideal to continue to review the contact list used, the method of contacting people and to confirm that people are actually being contacted.

The key is to continue to explore. One thing I've learned about open organizations and open sourcing is the value of building a quality community take time but it can achieve things that no one never thought possible.

Related Content

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.