Community Success Metrics

Martyn Davies (00:00): So this is our second Twitter spaces session for this week. First one was on Monday with Ben from Notion which went really well. This one is our second one, I'm very pleased to be able to host it. I'm Martyn, I'm the developer relations lead Orbit, the mission control for your community. And today, we've got a really amazing lineup of folks from the developer and community relations world with us. I've got a few questions that I'm going to kind of kick things off with. But we will have opportunity for you to ask questions as well both in Twitter spaces. And if you jump into the Discord, then you can also ask questions in there as well. I've got it in front of me.

Martyn Davies (00:42): So I will ask questions on your behalf if you post them in the Discord. But you can also ask questions directly in here as well. So we'll be able to make time we've got just under an hour of time. Although this is a pretty meaty topic that we're going to be going over. I think that there'll be plentiful opportunities for folks to ask it and have their questions answered by the great group of people that we've got with us today. So our panelists, if you could really call them that in Twitter spaces. I don't really know what the terminology should really be. But we'll go with panelists for now. Today are Mary Thengvall, Jeremy Meiss, Joe Nash, and Matthew Revell. We were also going to have Tessa Kriesel as well. But unfortunately, Tessa isn't feeling well, she has a migraine, so isn't going to be able to join us today.

Martyn Davies (01:31): But, Tessa, I know, you're going to listen back to this at some point. So we do hope you feel better soon. And yeah, I'm going to hand it over to our esteemed panelists for just a second to let them introduce themselves real quick. Mostly, so that you can get an understanding a little bit of what it is that they sound like. So when they're answering questions a little bit later on, you know who it is that's actually speaking. We'll do it in this order. We'll go Mary first, then Jeremy, then Joe, and then Matthew. So Mary, do you want to kick it off with a quick intro?

Mary Thengvall (02:01): Sure thing. Hi, everybody, I am the director of developer relations at a company called Camunda. We are an open source process automation company. And I lead the team there. We've got a team of developer advocates, we've got a team of developer experience professionals, as well as a team of community managers. I've been working with a variety of developer communities for 12 years now, I guess. And really just love digging into how we can best enable and serve those communities and trying to figure out the best ways to make them successful, not only with our products, but also in their careers and with whatever products they happen to be engaging with.

Jeremy Meiss (02:42): Hi, I'm Jeremy Meiss. The second of the panelists or the speakers or the spacers, I guess, maybe spacers might be the term. And I'm the director of DevRel at CircleCI. We're a continuous integration delivery platform. I've been doing DevRel and community for probably 10, 12 years now, depending on how you want to classify that and look at it. But so on the team at CircleCI, we have a team about six of us from Dev advocates, to program managers, to community managers spread in different regions, EMEA and APAC and in the Americas. And really excited to be here. And my first Twitter space, so let's do it.

Joe Nash (03:28): Hello, everyone. My name is Joe. I'm currently a developer educator at Twilio working on the Twilio quest team. I'm here mostly because my previous work I was a freelance community and DevRel consultant with my little firm called Interhacktive. I've been doing community focused ever work, particularly with education communities for about six years now, starting out and majorly cackling and kind of culminating in looking after the global student community over GitHub education. Super excited to be here. This is also my first Twitter space, I had to reinstall the app just for this. So yeah, excited to be here.

Matthew Revell (04:05): Hello, I'm Matthew Revell. I work at Hoopy. We are a developer relations and development, marketing consultancy. I'm really excited to talk about this because my passion, if you want to call it that is how we can move developer relations forward as a discipline. And do more to prove the value of it. Over the, I guess, 15 years or so that I've been working in developed communities, it's been really good to see how various people many of them on... Well, all of them on the panel tonight, but others as well, who have helped to push that forward. And I'm really keen to take part in the discussion tonight. So thank you for having me.

Martyn Davies (04:45): No problem. Thank you for being here. Thank you, everyone for being here. And Jeremy, we are adopting spaces instead of panelists from now on. So thank you, and this may well be the beginning of your whole new term that you've just coined.

Jeremy Meiss (04:59): I got to be on the bleeding edge of something in my life. So I'll take it.

Martyn Davies (05:03): Congratulations, you've set the new bar. Cool. So look, we're going to be discussing what makes good community success metrics, what makes a useful community success metrics? This in my experience is definitely one of the most important and without doubt, the most debated topic in the world of Developer Relations, and community. There's a ton that's been written about this kind of stuff, there's a ton that has been said about this kind of stuff. There's still a lot of movement within this kind of area around what it is that you actually do report, on how you look at metrics, how you talk about this stuff, how you convince people with data, how you capture data, what you do with it. It's a huge topic, we could go on well beyond the hour, but we will try to be as condensed and useful as possible. For those of you listening. So I thought we should probably kick it off with a little clarity question, if you don't mind. And that would be your thoughts spaces on what is a useful success metric?

Mary Thengvall (06:02): You're kicking it off with I think the biggest question there is. It's a big one for us to tackle first and foremost. I think for me, I take a somewhat broader look at it. And I like to start instead of starting with what metrics am I tracking? I want to start with what questions am I asking? Because I can track hundreds of different metrics, depending on what we're looking at from GitHub, to Discourse to Twitter, to Slack to everything else. But if I don't know why I'm tracking those particular metrics, or what questions I'm trying to find the answer to, I'm only pulling additional numbers, and it's not really going to get me the answers that I'm looking for.

Jeremy Meiss (06:45): Yeah. I was going to say like, my first answer was going to be depends. Because depending on where you are, depending on what stage you're at there's going to be a lot of different metrics there. And like Mary said, that it's almost the wrong question. And I think if you go down, just go, "Hey, what are the metrics I want to have?" I think you're in danger of missing the point for why you have metrics. I think a really good book around that is crap. I want to say it's death of metrics. But that's not what it's called. Anyways, I'll figure it out. I'll share it out. But I think you have to like Mary said, flip it around, come the back way. What answers I'm trying to solve?

Matthew Revell (07:36): I think the important thing is to build on that is that there is no one reason for doing developed communities. And there's even some argument as to what is... I don't want to get into this old question of what is developer relations? But the truth is that we lump a lot of different activities and different goals under the same banner. And so I think you need to come back to what is the reason that someone is paying for this committee to happen on the one side. And then measurements naturally fall out of that. But on the other side is what is it that the participants have come here to do? Because as Joe is very fond of saying, I'll leave him say this, but communities aren't audiences, they're not the same thing. So you have to have that shared common goal.

Matthew Revell (08:26): So what is it that your sponsor, whoever that is, is expecting to happen? And what is it this participants are expecting to happen? And then there are other metrics that we can kind of come out with that may be more generally applicable. But ultimately, the things that matter, we come back to those two questions.

Martyn Davies (08:47): Yeah. I think that's really good, generally. I just kind of want to jump back in quickly on one of the things that Mary mentioned there about, like what questions am I looking to answer? I think that's probably key for pretty much everyone that's here. But actually figuring out those questions in the first place is kind of tricky. When you've been doing this, and this is for all of you, when you have been doing this, where have you started when trying to figure out what those questions are?

Joe Nash (09:14): So I think one of the things that's really interesting about the question, you asked Martyn, what are useful success metrics? And then what Mary went on to say, is like, ultimately thinking about the reason we track metrics is to make sure that we're doing the right activities, make sure we're doing the right things, and that those things are working in the ways we want them to work and that we're able to report that up. Right? So to come to the point that Matthew set me up for, thank you Mathew. When we're building communities, like we ultimately to remember that, like people gather for a reason, right? And like, we need to ask ourselves, why have these people gathered here? What is it that they are coming together to form a community of and so in making sure that we are serving those needs and that we are tracking metrics that help us know that we're serving those needs.

Joe Nash (10:00): I think that is like the first, where I start with these trying to form these questions is like asking why are the community member's there? And that's kind of where I think we tend to get into the tricky place of audience versus community where we often want the members to be there to be part of our community and to consume our message rather than something that's driving from them. So that's where I tend to start looking for the questions like what's brought these people together?

Jeremy Meiss (10:25): Yeah. And I think once you figured out what's kind of brought them together, it's also kind of coming and looking at those data points, the things that you're trying to figure out how it's got to track up, to fulfill some thing that especially if this is a business community. If this is a product community, something that's part of a brand, you've got to make sure that those metrics that you've identified for success, track up to something else that the business has. Because if it doesn't, then you're eventually going to get the questions of well, why do we have this thing then? Because there has to be some value that you bring back, and it can't be nebulous like, "Well, we get everybody here to talk together." Like you've got to be able to show how it helps the bottom line for the company. And answer that question before it's even asked.

Mary Thengvall (11:20): Yeah. I'll second that. Because for me, anytime I start figuring out what metrics do I want to track? The first questions that I asked myself is, what are the goals of the company right now? Because if I'm not aligning our team goals with the company goals, then like Jeremy just said the question is going to come back, well, what value you actually bringing to the company? And so the better I can understand hey, the company is interested in awareness, or the company is interested in adoption, or in ARR, or whatever it is that they're focused on for that quarter or a year, then I can go back to my team and go great, the company is super focused on new customers and increasing the creating revenue. In order for that revenue to increase, and for people to sign up, they need to be enabled to sign up, right.

Mary Thengvall (12:11): So maybe we focus more on onboarding. And so we focus on adoption metrics, or time to value metrics or things like that, that encourage additional adoption, which then ideally ramps up the ARR. And so we can then go and say, "Look, we reduce the amount of time that it takes for someone to get onboarded, we're getting great feedback about the additional resources that we've created, people are then referring other people from the community to those resources. We have a direct impact on the amount of money that we're bringing in this year, over last year because of the work that we're doing." And so while we might not be the ones that are selling the product, or directly contributing to that bottom line metric at the end of the day, and so then I'm going to be tracking the things that show, what's the value that we're bringing back to the company. And how does that also impact the community? Because we have to keep both of those things in mind when we're tracking metrics as well.

Martyn Davies (13:13): Yeah, for sure. I think one of the things that resonated with me there, it's definitely that aspect of the different types of metrics and the granularity, a lot of it, that stems very far away from more traditional funnel metrics to a certain degree. How do you ensure when you're trying to figure out what things to look at that you don't roll too much towards straight up vanity metrics? So how do you tell the difference between something that's going to be a vanity metric and going to be more tangible, that's actually going to help tell a story inside of your organization?

Matthew Revell (13:49): Again, I think it depends on why does the community exists? I think there are broad archetypes of community, right? There's the contributor community, the open source, there's the let's try and build the excitement around this to drive adoption. There's the more mature communities that are there to enable developers to perhaps be more productive with a product or technology. But I think a lot of what we talk about in DevRel certainly around the community is more of those end user rather than contribute to communities. And so I think that, to me, that there are three very broad things that you're measuring, it's to be very crude about it, is how many people are coming in? How many people are staying and for how long? And then ultimately because nothing else worked with this, how satisfied are the people who are coming in that they're achieving what they came to achieve? How happy are they? And again, then everything else falls out of that I think.

Martyn Davies (14:54): Let's move on a little bit towards kind of setting targets around some of this stuff because I know that certainly for me and a lot of people that I've spoken to. And certainly when you're beginning to do this when a community is relatively new, or when you're setting goals for the first time, putting targets on stuff is super hard, especially when you don't have data to back up any of the reasoning necessarily. What advice or thoughts or tips do you have around? And how you might approach this, when you're setting a metric for the very first time? How do you put a target on it?

Jeremy Meiss (15:28): I think you start small. And this is where you've got to be in good communication with your leadership, and really make sure that they understand that until you've really figured out what your community is capable of, or what your community is capable of? How are they able to grow? And what are the things that really contribute to that growth? I think you have to start small, and look at a quarter by quarter basis and see what's the change and keep challenging yourself more. I know that one thing we thought we were going to be, that 10% growth in one area was going to be a really big, like we looked at the number and thought, "Man, that's going to be really hard." And then within a month we gone 300% over that. We then fully understand what was in our community, or what we could achieve with that. So starting small, I think is important to get you engaged that, and then from there, you have a baseline, and it's important to get to those baselines quickly. And then you build from there.

Mary Thengvall (16:34): I definitely agree. It's important to establish the baselines. I think something to keep in mind too, that a lot of us, myself included, forget at times when we're doing like, quarterly planning or yearly planning is that depending on the goal of your community, you might not be aiming for growth, right? So like, if your goal for your online community is a place for people to engage and share projects that they're working on and things but the primary use that it has currently is for people to get community support. Those are two very different things. And so there's a certain point at which you have to say, "Okay, how much do we want to grow in this area? And what sacrifice does that make over here?" And is it reasonable to say, "We're going to keep increasing the number of engaged users 10%, quarter over quarter, or however much year over year? Or do we want to focus on instead of new users, the amount of engagement that those users have on a day by day or month over month basis?"

Mary Thengvall (17:42): And I think that's part of going back to the vanity metrics question as well that like, there's times when tracking vanity metrics makes sense. And not necessarily from a just a, "Hey, we gained this many more users this month." But it gives you a better way of tracking patterns, and then investigating further where if you can say, "Hey, we've had a steady growth pattern of 10% more engaged members every month, or active members every month, and then suddenly, we had a spike of 20%. What happened on May 12th, that caused that spike?" And so I tend to track those vanity metrics as a way to see what do I need to look deeper into, to better understand what's actually happening in the community. And I might not ever report that 10% month over month growth to my CTO.

Mary Thengvall (18:34): But I'm going to keep an eye on that to know, "Hey, we had a major spike every April and every October, when we have new releases that go out, that makes a lot of sense." So if the spike next April isn't as big, I want to know not because it's necessarily a failure, but because maybe the website changed. So the way that we're tracking, that metric changed, or something else changed on the back end, that I need to investigate further, and figure out what we can do differently in the future.

Matthew Revell (19:00): To build on what you said there Mary as well, I think it's sometimes we're a little bit tough on ourselves in community DevRel around, saying that things are vanity metrics, when actually, they're not purely vanity, they do have value, as you described there. And one thing I would point to, I know that the funnel is a very imperfect model for what we're doing. But certainly in sales, I use it and you can bet the your colleagues and sales will not consider people going into the top of the funnel, the leads that they collect, they weren't going to consider that to be a vanity metric. They'll consider that to be an increased surface area for potential sales. And I think that's what we should look at community size as. So a lot of these things that we talk of is vanity metrics, actually they do have value as Mary said.

Joe Nash (19:47): And one other thing, digging a bit to what I think is coming out a couple of things here is, when you're setting these metrics, particularly what the scale is that you're trying to achieve, like what the actual defining numbers, I think it's really important to consider where that metric is ending up and how, what systems is being evaluated as part of like. What is the metric framework or the strategic framework that your company uses. And how does that influence the way they look at numbers? For example, like companies that are driven by OKRs, if they're following the old googly version of OKRs, part of that is you might just set numbers that you don't necessarily hit, right? And so you're meant to be ambitious.

Joe Nash (20:23): And there are other frameworks that have different takes on to what extent you come close to numbers, or what the numbers actually represent what they mean. And so to Mary's point, there are metrics that we share, and are part of why the business goals that other people want to be seeing, we'll be using to track results. And then there are metrics that we use to calibrate our own work. And knowing which is which, and where that number is going to end up is really important in saying that initial number and making sure that as Matthew say, we don't end up with a stick that will be of helps with.

Jeremy Meiss (20:57): Yeah. And I'd say like you when the first time also, to kind of even expand on that is that like the first time or six times that you present those metrics, especially to on the vertical up to executives, making sure that you are providing context, and explaining what this is, and why it matters. Because if you don't say that, then you just throw things out, you leave that open for the recipient of that metric to draw their own conclusion. So making sure that they know why you're tracking this, and what decisions you're going to make out of it. If you're going to put it on a slide, you need to be able to show what decision you're going to make out of it. Because if not, you're going to find yourself getting tied into a metric that you didn't intend to be the all seeing eye of your community. And that's not going to go well.

Matthew Revell (21:59): Yeah. Amara Graham talks about success metrics is narratives, you need to tell a story. Numbers in their raw form a kind of uninteresting, you have to tell a story to put them in contact.

Jeremy Meiss (22:11): Yeah. And that's why I try and go for like, really push us to the what kind of impact are we having? Things that we can show that there's a result that comes out of it within our developer community, not just a number. And so for us specifically, like we're paying attention to how many returning members over quarter. How many are active and how many return because that's showing some viability within our community. And that shows some activity that's actually happening, then that way we can show that there's actually things going on.

Mary Thengvall (22:47): Yeah, and I'll double down on that storytelling piece, right? That like, the numbers are great, but being able to then shake that into the narrative that says, "Here's the success, here's what we're doing." And not just, "Here's the work output, or here's the number of engaged members, or here's the number of conferences that we're at." But here's why this matters. Here's the impact that it had on the company, here's the impact that it had on the community, and continuing to have those conversations internally is huge. I had a client of mine years ago asked me like, "Well, how much time should I spend talking to people internally, as well as talking to people externally?"

Mary Thengvall (23:25): I was like, "You probably 50/50. Because if you're not having those conversations internally about, here's what we're doing, and here's the impact that we have, and you're only talking to folks externally, then people internally aren't going to understand the value of what you're doing." And so for me, it's far more than just tracking the actual metrics. And it's far more about how do I take those metrics and use them to flesh out this story of the value that we bring to the company as well as the community.

Martyn Davies (23:54): I'm keen to dig in on a couple of things that came out there. In fact, there was a question that I was going to ask you all that you've basically all just answered, which is great about the communication to especially senior leadership, and how you form that narrative in there. So thank you for pre emptying that one, I appreciate that. Engagement was mentioned in there a couple of times. And I've spoken with a lot of people in this world of DevRel and community about engagement in the past. And it's sometimes seen to be a bit of a tricky one. But to measure effectively, especially when it comes to things like events or just generally things that are can sometimes seem a little bit fluffy, as a metric. How have you seen engagement like measured effectively? How have you done it or how have you seen other companies approach that measurement of engagement?

Jeremy Meiss (24:46): I would say like to just kick that one off. I think it comes down to what is your community about? What is your product? To use a term there like, what does it do? If you're an API, solely an API, then you're going to have the engagement level could look something like, how many people are actually doing something with your API? How many have gotten to that? They've gotten their first 200. And then now they're engaging with your tool. It also can be if you have a library that gets used with other things, how often is that library exists in GitHub repos, or the videos that you do around like a tutorial? How long are they watching? How many people have viewed it? How long are they watching, like those types of engagement can kind of tell you how they're interacting with your product, your tool, your community, like all of those pieces, and it's shaped very much by I think, your product. Why your community exists?

Joe Nash (25:54): Sure. I think to kind of, I guess, abstract that a little bit. When it comes to like community design, I like to think of communities as essentially like, systems of incentive loops, right? Where you have a series of activities that you want community members to perform because business value, or based on these of the product, the community members have a certain amount of value that they hope to get from the community. That's why they're showing up. And the loop that they get, the thing that captures engagement, is them performing activities and returning the reward, like receiving the reward for it is called mechanical sounds. I don't mean that literally in the fact of like, "Hey, do x thing, receive piece of swag," although that could literally be it. But what we're looking for is activities being performed. And not to shell on the Orbit community camp thing, that's part of why we love the orbit model is like, it kind of encourages that line of thinking very well in laying out the activities and assigning values to them.

Joe Nash (26:49): But if you look at things that way, as an incentive system, it's far easier to see what engagement means and where engagement is because you're looking for people to be performing the activities that you want and engaging with the rewards that they're able to get from performing those activities in a meaningful way. And it's also much easier to find where things are falling down where you're losing engagement, when you're looking at each individual part of your community as a system of incentive loops because you start to see which activities are falling behind. Is that a case of the incentive is not strong enough for the cost of that activity to that member.

Joe Nash (27:25): And you get into these kind of interesting, calculations that you can do around each part of your community. So GitHub, Jeremy mentioned GitHub repos. GitHub repos are a really interesting place to see this because you have some valuable activities that are open pull request is a much higher energy thing, and shows inherently much more engagement than storing repository. And so having an idea of the relative costs of different activities that you want members to perform is very useful in tracking engagement in that way.

Mary Thengvall (27:55): I'll add one more quick thing on here. Carrie Melissa Jones has a great way of addressing engagement and that for her frameworks, people being engaged, they're either engaged with your product because of the purpose, or you're engaged in your community because of the purpose behind it or because of the people in it. And so it's given me a very different framework to use when I'm thinking about engagement, and the types of things that we're doing in order to keep people engaged. And so it's either they're connected because they feel really great about giving back to the community in our case, or contributing to the open source repos, or things like that. Or because they've really connected to the other people in that community. And so it's caused us to kind of think differently about how do we amplify the work of the community, which increases that purpose and that drive for people to be engaged because they're seeing the reward and the outcome of contributing in that way.

Mary Thengvall (28:52): Or making sure that people are better connected to each other, and not just two people at Camunda. Because the more connected that they are to each other, the more likely they are to stick around, the more likely they are to be further engaged in the future. And so it gives kind of an interesting way to think about those engagement statistics in more than just, "Hey, is this person is still here?"

Martyn Davies (29:12): I'll go right through to a question from the Discord. Brian also asks, when you're starting out, does it make sense to start with a few experiments and see what may be useful to measure after you've started? Or should you always start with a base set of metrics and let those evolve?

Mary Thengvall (29:33): I'll jump in real quick. I love experiments. I was actually just talking to you a Martin, who's on my team this morning about like, the awesome thing about trying a whole bunch of different things, is you can very easily go, "That didn't work. Stop doing that, this one looks like it might be. Let's dig into that further." And so it gives you the flexibility to quickly be able to say, "Okay, we're going to try this. If this happens, we'll keep going. If it doesn't will pull back." And so rather than saying build up this huge infrastructure where you can track all of the metrics, and then kind of let the answers come to you, it gives you the ability to really be more flexible. And then once you figure it out, "Okay, think this is the direction that we want to head in," you can build out the infrastructure to support those metrics, rather than spending a lot of time building out the infrastructure and then deciding, "We only really need to track four of these."

Martyn Davies (30:32): Oh, sorry, go ahead.

Joe Nash (30:34): I was going to say questions, the rest of the panel kind of based on that is like, how do you know that a metric isn't working? Because that seems to me like, it'd be very easy to fall into the trap of like, "Hey, this metric isn't telling the narrative that we want to tell, the narrative that we think is happening." And following down that false path, rather than when the metric may actually be showing that something's not working. So I'd love to hear a little bit, Mary about how you validate the negative results and some of those experiments?

Mary Thengvall (31:07): Sure, so part of that is knowing or going back to the questions that you're asking from the start, right? So not going in with, necessarily, "Hey, we think this is the answer," but saying, "What's the type of community we want to be serving? And let's look at the metrics that indicate where are people coming from?" How are they finding us. Of those people who are finding us who's sticking around? Maybe that's going to inform who our core community is right now. And then based on that, saying, great, now let's go track metrics for these other types of 10 generally related communities, or from these other platforms to see if that backs up the initial findings.

Mary Thengvall (31:49): And if it doesn't, then we go back to the questions again, and go, "Great. Okay, let's ask this in a slightly different manner. Let's track it again, let's see if that changes the answers, or changes the findings." And so it's less about trying to find metrics that fit the story that you already have in your mind, and more about asking the questions that you want to find answers to, and then seeing where that leads you.

Jeremy Meiss (32:12): Go ahead, Matthew.

Matthew Revell (32:14): I was going to very quickly say that, there is often not much cost in tracking numbers. So you can take the data in and decide which ones you're going to tell a story around. So you can track the metrics, but just not necessarily report all of them.

Jeremy Meiss (32:31): Yeah. And I think the answer my answer to you, Joe would be like, it's important to get all the data. Kind of what Matthew said, is like you have all the data there, get somebody, also maybe within your data team to kind of sit down and help you see, if there's maybe some patterns that you're missing because unless you really have a lot of training around that analytical piece, you might be missing something. So I think it's important there, but I think it's also important to crowdsource some of that too. Talk with other community managers. You're not in a bubble, there are plenty of community managers and endeavor professionals and in such that have been doing this for a while. And so hop on a call, or if you're local, and we're not suffering from a pandemic, grab a bite, or a pint and sit and bounce some ideas off. See what's worked for them.

Jeremy Meiss (33:34): And especially those that might be in a similar industry because I think we tend to get ourselves so stuck into, this is our community, and we miss a lot of the collaborative opportunities that exist outside of ourselves. And we're not, I think I can probably pretty clearly and with a lot of confidence here say that like, there's nothing new under the sun, really, with community management. There's different ways of looking at things. But chances are, there's probably somebody out there that's tried it, or is currently trying something that you're thinking about. So use that and use it as an opportunity to grow and use that as extra data to inform whether or not that's a metric that is working for your community.

Martyn Davies (34:24): Super, thank you. And Brian, thanks very much for your question. Those of you in the Twitter space, as well as in the Discord. We're going to jump into questions from all of you in a minute. So if you want to start getting the minds working about what you're going to ask, your time is coming soon. One other question, I think before that, for me, would be that I've seen as well a lot of teams going through the process of setting metrics, doing this definition, communicating the definition, and then really not coming back to it until say the end of the quarter and then going, "Oh, you know what, we should probably check out we're doing with this." That's not the best way to approach it. I'm keen to hear about how often you are coming back around on the data with your teams. Not necessarily senior leadership, but with the people that you're working with directly. How often are you sitting down and discussing these metrics, talking about them, revisiting them, resetting them? How does that look?

Joe Nash (35:23): Just from my perspective, I think there'll be an element of stratification to some of the things that he said, like what Mary's point about vanity metrics. There are definitely numbers that are useful to keep a track on almost day to day in terms of like them providing a red signal, green signal, are we moving in the right direction. So a good example of that in some things that I check pretty regularly with teams is engagement, and number of messages sent in community spaces, for example. That's one where it's useful to look at it day to day because there are micro optimizations you can make on a day to day, to lift those numbers, if they're looking down. But then number of new community members is one that is probably a longer range one where like growth is something that you can look at, on a bit of a distance.

Joe Nash (36:13): I think the ones that tend to be really dangerous, and definitely get to the last minute are the ones were to Jeremy's point earlier, where you have to collaborate with other teams. And unfortunate, I think that tends to be the ones where it's proving the most business value. So where you work out the metrics that track the transition points from your community members to points further down the funnel, points further along the sales process. Those tracking on those metrics, and keeping tabs on them often requires checking in with other systems within the company, or other members within the company. And so those tend to fall off the radar really quickly. And that's where I think the danger lies. So working out some kind of cadence with those teams, getting them into regular meetings, if you do bring them into a team meeting once a month, that kind of thing, is super useful to make sure they don't disappear.

Jeremy Meiss (37:05): This may be popular or unpopular, but my answer is weekly. I have one on ones with my team, each member of my team, and we have within our organization, what we call vital few. And those are the things that tell us whether or not we're being successful or not. And so we have a weekly department meeting, and then I have one on ones with my team. And we're able and while that seems like a lot, especially from like a department level. So that's like all of marketing, what actually it has enabled us to do is, we're able to see these metrics, these vital few on a very regular basis. And it's easier to plan and make adjustments when you're looking at it on a regular cadence like that, than to do it on every other week or a monthly basis. Because then especially if you're looking at from a quarter perspective, that means if that on the first month of that quarter, you see, "Oh, this isn't performing the way we want."

Jeremy Meiss (38:09): And then you start making some changes, and then it doesn't come until month two, and then you're down to only one more month to hopefully make a change. Or to bring it forward of saying, "Hey, I need some help from another area of my company." Or like Joe's saying some other team. If you're waiting over a longer cadence, that means that you end up being to that other team in fire mode. Whereas if you are looking at it on a shorter timeframes, you're able to see like Mary said, a little bit ago around like, you can make those small adjustments. You can see, "Okay, maybe this isn't working, what do we need to do?" And you can make those small adjustments and see them on a shorter timeframe than to just say, "We'll get together once a month and look at this." I think that can be really dangerous, especially if you're in an area where you want to get a lot of buy-in from above.

Mary Thengvall (39:05): Yeah. I'll second some of what everybody has said, I think. Especially when you're doing experiments, tracking early and often weekly, or more regularly than that, if you're trying to iterate quickly, is extremely important. Because like Jeremy said, You can't only track once a month and expect to try three or four different experiments in a quarter when you're only checking in on the metrics two or three times max. I think once you have some established metrics, there can be some metrics that you track that if you're tracking them more often than once a month, you'll actually miss some of those broader patterns and start to lose some of the forest for the trees. And so there's some things that you'll want to track monthly or maybe even quarterly to see, "Hey, how is growth progressing?" Or we like I mentioned earlier, we always start the spike in April, we always spike in October. Hey, now we're also starting to see you download spikes after our events or things like that.

Mary Thengvall (40:03): Whereas if I were just looking at week over week, I'd look at it and go "What's happening here? Or we're completely lower than we were last week. That's terrible." It's like, well, no, we're just coming back down from what's happening. So I think to a certain extent, and it depends, it's actually something that I am working on with my team. There's some things that we track on a monthly basis. But a lot of what we're communicating is asynchronous. And part of that is because we're a global team. We've got people across the states, we've got people throughout Europe, and then New Zealand as well. And part of that also is people are working on so many different projects that having a single time that we're all reporting on metrics once a week would take up the entirety of a team meeting. And so making sure that we're putting the information out there for people to be able to access it, if they're interested or when they need to is great. And then I surface that backup to our CTO for the things that we're tracking more broadly as a company.

Martyn Davies (41:01): All right. Are there any questions in the Twitter space that anyone wants to ask? Maybe not. That's okay. Got other questions from the Discord. There's a question. This is an interesting question I'm going to ask anyway. Ariadne asked, are there metrics that can help identify how many lurkers are in your community? And later clarified a little bit extra for that, like who is sort of online but not really contributing, as opposed to technically being part of the community but not engaged at all.

Joe Nash (41:33): I want to start with my favorite community number of all time, which is the 1% rule, that inequality of participation rule. I'm sure pretty much everyone in this chat has probably heard a variation of it. But if you haven't, it's the idea that 1% of your community will create the content 9% will engage with it, and 90% will lurk. This is a pretty often quoted thing with some research find that particularly from communities of practice. So in general, it's useful to keep in mind that in any online community, you're expected to have an incredibly high percentage of lurkers as to how you identify those in metrics, that's going to come down on kind of a platform by platform basis. Like Discourse, for example, is a platform that once you get access to the server insights provides a lot of data natively about how many lurkers you have.

Joe Nash (42:21): And so really like the metrics you have available to track lurkers, it's going to depend on the technology that you're using, and where your community is actually, like gathering what your community spaces are. But it's more or less most platforms nowadays, will be able to tell you, like your monthly active users, or which users are inactive, in some form. Even Slack will tell you which users are inactive. So it's pretty common that you'll be able to track that within your platform.

Jeremy Meiss (42:44): Yeah. And this Discourse, I always have to make sure I say it right because there's too many disc things out there. But Discourse also gives you kind of a view, if you're using that as your forum platform, it gives you a view on how many people are logged in? How many people are not, that are still viewing your content. And so that's one way that you can kind of see where that maybe conversion level percentage, you have logged in users versus logged out. And then mixing that with how many people actually reply is one way that you can kind of see that movement within Discourse reports that can kind of help you see who are the lurkers and who aren't. And again, those that have fallen off the line, that I've used that in the past, and that definitely is helpful to kind of see who are those that are dropping off that you probably want to. And that's why I feel like I need to say this.

Jeremy Meiss (43:44): That's a normal life cycle of community as well. And I think we don't give ourselves enough credit or enough space, it's not credit. Enough space to realize that there are people that will join your community on day one. And they joins primarily because they're looking for some problem to solve. Once they've solved that problem, they may never come back, and there's probably nothing you can do to get them back. And there also will be those that have been very engaged for six months, and then after a year, they're no longer engaged and they never come back. And there's a lot of reasons for that. And so I think it's important for us to understand that life cycle community and not be so hard on ourselves with it.

Martyn Davies (44:25): That's a very definitive truth that Jeremy, but a good one to point out at this stage. I think that there's so often a lot of stress almost, that's caused by not necessarily managing to bring back every single person ever into the community. I think hearing that okay, is a fantastic band aid for many of us. Thank you for that. So I'm going to throw it back open to the Twitter spaces again, if anybody wants to ask a question there. Jump up right now.

Jeremy Meiss (44:54): Is anyone awake?

Martyn Davies (44:57): No, it's late here in Europe. So it could well be that most people logged in. And they're falling asleep with their phone next to them on the bed. But there's a question from Discord from Brian, who asked the question earlier, he's back again, with a question that's been getting a lot of emoji love. So I feel like it definitely has to be asked. And Brian asks, can everyone talk about their most successful metric that they can remember off the top of their head? Why did it work? And what was the context it worked in?

Mary Thengvall (45:25): I'll jump in here first. So this was an interesting one that I think some of you already know about. I mentioned it in my book under a slightly different name, I call it warm handoffs in the book, but I call it DevRel qualified leads these days. And when I was working with a client, they were really struggling with people internally understanding what they were doing, the value that they're bringing. But they're making all of these fantastic introductions from the community back into the company and facilitating a lot of great conversations and connections. And so I encourage them to take that and put an actual metric against it. And DevRel qualified leads was the name that we came up with, to kind of piggyback off of marketing qualified leads, and I've got a whole blog post about it if you want to know more. But that changed the whole way that the company viewed them.

Mary Thengvall (46:14): And within a couple months, they literally went from the company questioning whether or not the team should still exist, to other teams advocating on their behalf that like you can't dissolve the DevRel team because they're our main pipeline for new recruits, or they're our main pipeline for feedback, product feedback, or for marketing content, or all of these things. And it absolutely transformed the way that they were seen internally, and it wasn't that they were doing anything different. It's just they were actually putting words to the actions that they had been doing all along, and making sure that people were aware of the impact of those actions.

Matthew Revell (46:58): I'll jump in. For me, it's in the context I've been working in, it's been a mixture of revenue attributable to the program and leads, which isn't a very, I guess, fashionable thing to say. But in a startup environment where community has been funded because it will have people hope, executives hope, and effects on the bottom line, then often revenue is a really good thing. Attributable revenue is a really good thing to show to people to carry on having them fund what you're doing. And then you can do all the other stuff that you know is important. But sometimes going back to those narratives sometimes you've got to tell the story. That means you can carry on doing the things you want, while other people get the things that they want out of the activity.

Joe Nash (47:54): I'll jump in from the perspective of someone who often works with communities where the path to sales is very long waiting in education. On the Education team, we started to track basically the usage of our particular audience on the product writ large. So this is really useful when your community that you're looking after is clearly identifiable or separate from the rest of the audience. So looking at, for example the general pool of GitHub users, we know who the student users are. And so the metric that kind of unlocked lots of things for us was looking at the number of contributions from the student community. And that was really interesting for a bunch of reasons.

Joe Nash (48:34): For one, us starting to pay attention to that number, unlocked some things throughout the company, for example, a regular slump, those recurring year on year, we were able to identify as being caused by students going on holiday, which was super great. When you cause a slump, that makes half the company panic, it turns out to be your audience going on vacation. And also labeled just to save for the first time that like, "Hey, this percentage of overall use of the product in this particular use of the product that the company considers valuable, is tied to the work of our audience." So that was super useful for us. And also, just while I'm speaking, for education and stuff as well, being able to say around the free market, does that use a convert to non education plans is always to be useful.

Jeremy Meiss (49:17): Yeah. To kind of close that out. I think for us, it's always been returning users, and in what percentage of them of the total number of active users are returning because for us, if we can raise that up, I'm going to use correct grammar here. If we can raise that percentage up, that means that we're getting more and more people that care about what we're doing and care about our products that are getting involved, they're engaging. And so if we can keep that going quarter over quarter, showing more and more returning members, while still seeing our new members grow. So that's showing some viability within our community. And so that's been one that we've consistently used and across companies that I've been a part of over the years. Like that continues to stand out for us, at least as one that really tells me whether or not we're being successful and helps inform a lot of the decisions that I have. And that I make over the coming quarters and months. So thank you.

Martyn Davies (50:21): Fantastic. Thank you. And Jeremy, I know you need to drop off right now. So I'm going to say thanks very much to you for being here. I really appreciate your time. Really appreciate your thoughts. And Jeremy will be available in the Discord, he's there pretty much all the time. So if you've got other questions that you want to ask Jeremy, specifically, you can jump over there and find him to have a conversation with.

Jeremy Meiss (50:41): Absolutely. Thanks for having me.

Martyn Davies (50:43): No problem. I guess we're going to wrap up. We've got a final question here in the Twitter space is from [Nasha 00:50:47]. What's your question?

Nasha (50:49): Hello, and thank you for having me. So yeah, basically, I started working as a developer advocate, three months ago, and this was new for me this opportunity came through my Twitter audience. And I've been reading Mary's book. And this has been helping me a lot on how to find metrics, how to... Yeah. It really helped me a lot with the job. But basically, the question that I had for you was like, how to find communities and how to engage in those different communities that relate to your product. But for example, like we can work, we're having integrations with a product, like oh, with an integration with Firebase. How to find those communities and then engage with those communities without looking like you're spamming because you want to share something valuable because you have build something great with your product, and you want to share that with that community. But how to do it without being spammy, how to engage in an organic way.

Mary Thengvall (51:50): I'll hop in there first. I would say you always have to listen and observe, before you're jumping into recommend your product. The quickest way to get people to not pay attention to what you're saying is to speak up every time someone says, "Hey, does anyone have a solution for this?" And you go, "Oh, this is your solution. This is totally what you should use." And you've never contributed to that community before. Right? So I think really, digging in, understanding the questions people are asking, contributing in other ways first, and then as you see questions start to come up. And after you've established yourself as a part of that community, you can start saying, "Oh, hey, I happen to work for such and such a company, we have this product that might be helpful. It might be useful way for you to get started." Because then you've already established yourself as reliable and an engaged member of the community. And you're not coming in only with the intent of selling or promoting in that community.

Nasha (52:52): Thank you. Thank you very much. Mary, it's very exciting for me to get an answer from you because I've been reading your book, and I'm halfway through it. And so it's very exciting to listen to their advice.

Mary Thengvall (53:05): I'm so glad it's been helpful. Thanks.

Martyn Davies (53:08): Thanks very much for your question there. That is our time. Unfortunately, we've already gone over the top of the hour. So we are going to have to wrap it up. Thank you so much. to Mary, Joe, Jeremy and Matthew for joining us for this is a huge topic. And I'm sure we could do at least another hour of questions and thoughts on this. If you do want to jump in and continue the conversation, feel free to head over and join the Discord channel the Orbit Discord. You can find it @discord.orbit.love. And there's a full running commentary of this entire session in there, if you want to look back at any of the notes that have been taken up to this point. As well as any of the notes that have been taken out from any of the sessions that we've had so far this week.

Martyn Davies (53:57): If you're keen on hearing more about what we've got going on this week, as part of community camp, you can find out about the other sessions that are going to be taking place over the rest of this week @communitycamp.com. And whilst you there, if you fancy taking a little bit of time to fill out our big community tools survey, we're running this right now to gather information on the tools that are being used to build and manage communities. We'd really love your input. So if you head over to communitycamp.com you'll find a survey there. And your reward for taking such a survey will be that we will send you a copy of the finished report that we make. So that's it. Thank you very much everybody. Really appreciate you being here. Thank you for listening. Have yourselves a great rest of the day, a great evening or if you are further east than I am right now a great morning. See you soon.

Nasha (54:46): Thank you for having me. Bye-bye.