Do Your Product Manager Interviews Miss These Critical Analytical Skills?

This post is part of The Ultimate Guide to Product Manager Interview Questions series.

You’ve always heard that you should hire the smartest product managers you can.

Some even say you should only hire people that are smarter than you.

But what does all this really mean?

Should you be trying to give candidates a verbal IQ test? Or should you be testing more practical problem solving?

What are the right questions to ask?

It is true, of course … you should hire the smartest product managers you can. And doing that well starts with a purpose-built interview.

The Analytical & Cognitive Skills Interview

The purpose of this interview is to evaluate problem solving and critical thinking skills. A product manager needs to be extremely smart, and this interview gives you the framework to effectively evaluate raw intelligence, creativity and analytical abilities.

Your goal should be to evaluate the candidate’s intellectual horsepower not in the context of esoteric puzzles and brainteasers, but in the context of common and relevant business problems.

This interview should take 45-60 minutes to complete. The key sections of this interview are:

Market Sizing & Problem Solving: 15-20 minutes
Business Model Exploration: 15-20 minutes
Metrics / Data Analysis & Problem Solving: 15-20 minutes

Market Sizing & Problem Solving – High Level Thinking

Product managers need to be able to quickly make assumptions and generate back-of-the-envelope estimates. Hypothetical market sizing is one of the most common ways of assessing this skill.

Good answers require a candidate to communicate key assumptions and keep everything straight while quickly doing the math in his or her head.

Interview questions you could ask include (you should only ask one):

What is the total annual revenue of all gas stations in the United States?

What is the average weekly revenue of a typical movie theater in San Francisco?

How many people travel from San Francisco to New York every day by commercial airline?

These type of market sizing case interviews are used heavily by top management consulting firms like McKinsey, Bain and BCG.

Here is a 9 minute sample interview video from Bain’s London office, where you can see how an interviewer leads a candidate through a market sizing case and how she analyzes his responses at the conclusion of the interview:

Business Model Exploration – Case

Winning in business is just as much about great business models as it is about features, technologies and value propositions. Unfortunately, not all product manager interviews focus as much on the former as they do on the latter.

There is a lot more to a great product than just it’s value propositions and features. Alex Osterwalder, author of the Business Model Generation and creator of the Business Model Canvas, talks about four levels of strategic mastery:

Level 0 – The Oblivious: Focuses on the product and value propositions alone instead of the value proposition AND the business model.
Level 1 – The Beginners: Uses the business model canvas as a checklist.
Level 2 – The Masters: Outcompetes others with a superior business model where each business model element reinforces each other. Examples are Nintendo Wii, Nespresso and Dell.
Level 3 – The Invincible: Continuously disrupts themselves while their existing business models are still successful.

To be most successful, you do not want your product managers to be playing at Level 0 or Level 1. They need to be comfortable iterating on the whole product and not have a toolbox limited only to features, technologies and value propositions.

In this section, you will evaluate the frameworks and processes a candidate uses to analyze a potential business. What you are looking for is familiarity with a wide variety of business models, a good framework for analysis, and creativity and divergent thinking in coming up with solutions.

A good framework for early stage startups is Ash Maurya’s Lean Canvas. A good framework for more mature businesses is the Business Model Canvas. Either of these could be complemented with the Eliminate-Reduce-Raise-Create grid of Blue Ocean Strategy to further develop interesting and disruptive ideas.

For a junior role, use a white board to analyze and iterate on business model prototypes together. Consider the analysis of either existing businesses or potential new businesses, depending on the stage of your company and what the product manager’s role will be. Have the candidate walk you through his or her process for analyzing issues and making decisions.

As an example, here is a video of Alex Osterwalder at Stanford using the Business Model Canvas to do a 5 minute analysis of the Nestle Nespresso business model:

For a more senior role, the candidate should be able to synthesize a framework and lead a discussion without help. Provide a more open ended scenario like:

Based on what you know, how would you identify and validate the best business model for us to pursue for one of our products?

Another good option for a startup specific interview question is:

Imagine you just received a $5 million Series A investment from Andreessen Horowitz. Describe your process to validate the proposed business model and to get the company to product/market fit.

This scenario not only allows you to test a candidate’s business model exploration and prototyping skills, but also tests for big picture thinking and comfort level with uncertainty.

As the candidate walks you through a business model design and prototyping process, the key areas you want to get clarification on are:

Revenue model
Gross margin model
Operating model
Working capital model
Investment model

These five elements were highlighted in Getting to Plan B: Breaking Through to a Better Business Model by John Mullins of London Business School and Randy Komisar of Kleiner Perkins Caufield & Byers.

For each of these elements, ask a candidate:

Talk to me about the [revenue, gross margin, operating, working capital, or investment] model.

Why did you choose this model?

Are there alternatives that might provide a better competitive advantage?

Here is a 14 minute video of John Mullins giving examples of companies with innovative approaches to each of the five elements:

Metrics / Data Analysis & Problem Solving – Detail Thinking

Smart product managers use data effectively to make better decisions. Ask about data analysis using an open ended question like:

Tell me about how you use data to make decisions.

What you’re hoping to hear is the candidate talk about dashboards and how to use actionable metrics instead of vanity metrics in the analysis and measurement of success. If the candidate does not get here alone, prompt him or her more directly:

Can you take one of our products and talk about how you would build a dashboard for it?

What metrics would you track? Why?

If you’re an early stage startup and haven’t released a product yet, pick a product the candidate is familiar with.

You want to hear a candidate talk about lifetime value, per-customer metrics, events, funnel analysis and cohort metrics.

Ask the candidate to step you through a funnel analysis on the whiteboard:

Can you walk me through how you might do a funnel analysis for one of our products?

Rapidly and clearly doing this exercise will tell you a lot about the candidate’s ability to think on-the-fly and how smart he or she is about data analysis.

Your ideal is a candidate who can talk about actionable metrics in the same way Neil Patel, Co-Founder of Kissmetrics does.

Did you like this article? Subscribe now to get content like this delivered free to your inbox. Learn more about what I do: https://andyjagoe.com/services/

This post is part of The Ultimate Guide to Product Manager Interview Questions series.

P.S. A lot of people have asked for book recommendations to further prepare for a product manager interview. Here are some great ones:

Photo by João Trindade