😵 What are the right metrics to track? How do we know if a solution is successful, what should be the target? How do we learn from it?

These are the most common questions I got asked as the digital analytics manager but identifying the right metrics is more than just a google search for KPIs. Let's think of an example, suppose you work at an e-commerce company and have just released a re-branded home page with new layouts and information. Would you think that conversion rate is one of your primary metrics? well, it depends:

• What is the purpose of your home page?
• Can visitors add to cart straight from the home page?
• What changes were done in design and capabilities and how did the team make those decisions? (eg what data was available, user research, A/B testing, etc)
• Where are the visitors to your home page coming from and at what stage of the purchase funnel are they likely to be?
• How is conversion calculated? does it require a specific definition where only certain pages or visitor cohorts are included in the calculation?
• Who will be the stakeholders reviewing your metrics? Is it the company leadership that might just be looking at your channel's dollar contribution to the overall company revenue? is it the e-commerce director who might be interested in the performance of the site and how this particular feature is impacting it? will it be the designers interested on the performance of a new component? or maybe your engineers wanting to know if there are technical or accessibility bugs?

The best way I can contribute is by being involved in the early stages so that decisions are informed by data and, early on, I know the reason why a project is being prioritized and data used during the discovery phase to inform this prioritization; alternatively, especially for new features or designs, understand the learning objective and develop metrics that will help us achieve it. This ideal situation will be dependent on the company and team data culture and maturity so in the cases where I am engaged until a feature has been released or is close to being finalized, I can approach it in two different ways:

• If I am familiar with the project or it is not highly complex, the project brief and wireframes/mocks will allow me to get up to speed and only ask a few clarifying questions
• If I haven't heard of the project, is complex and/or does not have sufficient documentation, I would talk directly with product and, if needed, design & engineering

I love to coach people who are interested in self-serving data and developing their own metrics and have found that getting started is the hardest part, which is why, in partnership with a Product Manager from my team, we developed a template that provides people with a helpful structure to get closer to the answers themselves! Once they have gone through this template, our conversations are a lot more efficient and we can dive deeper into identifying the actual metrics, what to call them, what would be the definition within the context, whether we can start tracking or launch or create a ticket to instrument the data collection, identifying the event name and corresponding properties as per taxonomy, etc.

There were to main additional wins from this template:

• It allowed me to have a starting point to facilitate interactive workshops to support teams without a dedicated product analyst who were getting started with analytics; I transformed it into a figjam board
• The first product launched using this template became in itself a template for any new product launches. For the second product launch, the metrics were fairly similar and only small adjustments were required; we even replicated the dashboard in Amplitude Analytics and modified the queries!

Breaking it down

The template allows the requestor to consider what their expectations are for deliverables, deadlines, and in what capacity they require the analyst to be involved. It also helps the analyst understand the tasks to be completed and align on the deadline based on their capacity.

In this section, I will break down each of the components from the template with the explanation of why that information was asked for.

‍Introduction section

Discuss roles & responsibilities, project objectives and desired outcomes, and discuss measurement plan objectives and strategy.

RACI Matrix

I really like the RACI framework to clearly identify the involvement of each member/stakeholder. I do find that its sometimes awkward to clarify in a live conversation when someone is trying to take ownership of a project but they are only being consulted so having it written down before the live conversation is really helpful! This section will identify the involvement required by the analyst and who are the stakeholders to whom tasks should be communicated; in other words, make sure no work is being duplicated, everyone that needs to be in the know is included, and no one is taking over someone else's expected tasks to complete.

Project background

This is a summary of the project and respective documentation. While this is critical information to review, I personally like the answer to this as a way to assess the reliability of the project. For example, if no documentation exists, that is a red flag for me! how will anyone know what the objectives are? what business goals is it aligned to? what user pain point is it solving? if it was a request from leadership, what data was used to inform this decision or, since it was likely a "gut feeling", what is their expectation for results?

Ideally, there would have been key data identified during the discovery phase which will very likely become part of the success metrics and/or provide baselines to compare against performance.

Measurement plan objectives

The requestor will have a general idea of some metrics that are important to them, if we are in an e-commerce website, then it is very likely tied to achieving sales targets. To get to the true measure of success, I like to ask what metrics they think are important (which ones will they actually be looking into) and ask why its important to them. By asking why, perhaps more than once, I can better identify which are the metrics that are directly tied to the expected release and that we have control over so the team can pivot based on performance. We might still need to track sales, but it is a very general metric that is influenced by too many factors that it would make it hard for the team to quickly understand what piece of the puzzle needs to be adjusted.

Strategy

How is the project objective being solved with the presented solution? Unlike the project background where all the decisions were made, this question gets the requestor to think through the solution to be implemented and opens it up for discussion on what other potential solutions were identified and how they arrived at a decision. You'd be surprised how many times this question has uncovered alternate solutions, A/B testing ideas or future iteration milestones!

Build measurement plan section

This is where we put it all together.

Tactics

What are the actions that we need to know whether they succeeded or failed? Note that some of these might be categorized based on the reporting audience; for example, if you are reporting to executive leadership, then what is the general result to track that will give them enough information without overwhelming them with details they are not aware of? There might be another subset of your audience who might be the multi-disciplinary core team working on the project that will want to know the general performance of the project and some of them will want further detail based on what they had the most influence in.

Metrics

For each of the actions identified, what exactly do we need to know for each of them? these will be the metrics to track. At this stage, I also like to identify if data instrumentation will be required, what type of data we will be collecting and where it will be accessed.

Segments

Let's say that you launched a new product, would conversion rate for the whole site matter? it does but not for the purposes of this measurement plan where we are trying to identify the success of a particular project. If the product launch was to increase the sales of a particular category, one segment to calculate the conversion rate would be those visitors who are prospects for that product category. Or let's say this product had a soft launch phase were only newsletter subscribers have access to an early purchase, a segment would be those who include the appropriate UTM parameters.

Segments allows you to narrow down your view and easily identify changes that could otherwise be missed in a wider scope. It also provides a better performance overview for those visitors or users that you had in mind when designing the solution.

‍

Reporting expectations section

This is the time to clarify the tasks to be delivered, reporting periods, format, communication frequency and medium, involvement, etc.

‍

💡 Being helpful isn't about giving someone a satisfactory answer, is about identifying the opportunities and collaborating with others to achieve results.