hypothesis testing in product management

Shipping Your Product in Iterations: A Guide to Hypothesis Testing

Glancing at the App Store on any phone will reveal that most installed apps have had updates released within the last week. Software products today are shipped in iterations to validate assumptions and hypotheses about what makes the product experience better for users.

By Kumara Raghavendra

Kumara has successfully delivered high-impact products in various industries ranging from eCommerce, healthcare, travel, and ride-hailing.

PREVIOUSLY AT

A look at the App Store on any phone will reveal that most installed apps have had updates released within the last week. A website visit after a few weeks might show some changes in the layout, user experience, or copy.

Today, software is shipped in iterations to validate assumptions and the product hypothesis about what makes a better user experience. At any given time, companies like booking.com (where I worked before) run hundreds of A/B tests on their sites for this very purpose.

For applications delivered over the internet, there is no need to decide on the look of a product 12-18 months in advance, and then build and eventually ship it. Instead, it is perfectly practical to release small changes that deliver value to users as they are being implemented, removing the need to make assumptions about user preferences and ideal solutions—for every assumption and hypothesis can be validated by designing a test to isolate the effect of each change.

In addition to delivering continuous value through improvements, this approach allows a product team to gather continuous feedback from users and then course-correct as needed. Creating and testing hypotheses every couple of weeks is a cheaper and easier way to build a course-correcting and iterative approach to creating product value .

What Is Hypothesis Testing in Product Management?

While shipping a feature to users, it is imperative to validate assumptions about design and features in order to understand their impact in the real world.

This validation is traditionally done through product hypothesis testing , during which the experimenter outlines a hypothesis for a change and then defines success. For instance, if a data product manager at Amazon has a hypothesis that showing bigger product images will raise conversion rates, then success is defined by higher conversion rates.

One of the key aspects of hypothesis testing is the isolation of different variables in the product experience in order to be able to attribute success (or failure) to the changes made. So, if our Amazon product manager had a further hypothesis that showing customer reviews right next to product images would improve conversion, it would not be possible to test both hypotheses at the same time. Doing so would result in failure to properly attribute causes and effects; therefore, the two changes must be isolated and tested individually.

Thus, product decisions on features should be backed by hypothesis testing to validate the performance of features.

Different Types of Hypothesis Testing

A/b testing.

One of the most common use cases to achieve hypothesis validation is randomized A/B testing, in which a change or feature is released at random to one-half of users (A) and withheld from the other half (B). Returning to the hypothesis of bigger product images improving conversion on Amazon, one-half of users will be shown the change, while the other half will see the website as it was before. The conversion will then be measured for each group (A and B) and compared. In case of a significant uplift in conversion for the group shown bigger product images, the conclusion would be that the original hypothesis was correct, and the change can be rolled out to all users.

Multivariate Testing

Ideally, each variable should be isolated and tested separately so as to conclusively attribute changes. However, such a sequential approach to testing can be very slow, especially when there are several versions to test. To continue with the example, in the hypothesis that bigger product images lead to higher conversion rates on Amazon, “bigger” is subjective, and several versions of “bigger” (e.g., 1.1x, 1.3x, and 1.5x) might need to be tested.

Instead of testing such cases sequentially, a multivariate test can be adopted, in which users are not split in half but into multiple variants. For instance, four groups (A, B, C, D) are made up of 25% of users each, where A-group users will not see any change, whereas those in variants B, C, and D will see images bigger by 1.1x, 1.3x, and 1.5x, respectively. In this test, multiple variants are simultaneously tested against the current version of the product in order to identify the best variant.

Before/After Testing

Sometimes, it is not possible to split the users in half (or into multiple variants) as there might be network effects in place. For example, if the test involves determining whether one logic for formulating surge prices on Uber is better than another, the drivers cannot be divided into different variants, as the logic takes into account the demand and supply mismatch of the entire city. In such cases, a test will have to compare the effects before the change and after the change in order to arrive at a conclusion.

However, the constraint here is the inability to isolate the effects of seasonality and externality that can differently affect the test and control periods. Suppose a change to the logic that determines surge pricing on Uber is made at time t , such that logic A is used before and logic B is used after. While the effects before and after time t can be compared, there is no guarantee that the effects are solely due to the change in logic. There could have been a difference in demand or other factors between the two time periods that resulted in a difference between the two.

Time-based On/Off Testing

The downsides of before/after testing can be overcome to a large extent by deploying time-based on/off testing, in which the change is introduced to all users for a certain period of time, turned off for an equal period of time, and then repeated for a longer duration.

For example, in the Uber use case, the change can be shown to drivers on Monday, withdrawn on Tuesday, shown again on Wednesday, and so on.

While this method doesn’t fully remove the effects of seasonality and externality, it does reduce them significantly, making such tests more robust.

Test Design

Choosing the right test for the use case at hand is an essential step in validating a hypothesis in the quickest and most robust way. Once the choice is made, the details of the test design can be outlined.

The test design is simply a coherent outline of:

The hypothesis to be tested: Showing users bigger product images will lead them to purchase more products.
Success metrics for the test: Customer conversion
Decision-making criteria for the test: The test validates the hypothesis that users in the variant show a higher conversion rate than those in the control group.
Metrics that need to be instrumented to learn from the test: Customer conversion, clicks on product images

In the case of the product hypothesis example that bigger product images will lead to improved conversion on Amazon, the success metric is conversion and the decision criteria is an improvement in conversion.

After the right test is chosen and designed, and the success criteria and metrics are identified, the results must be analyzed. To do that, some statistical concepts are necessary.

When running tests, it is important to ensure that the two variants picked for the test (A and B) do not have a bias with respect to the success metric. For instance, if the variant that sees the bigger images already has a higher conversion than the variant that doesn’t see the change, then the test is biased and can lead to wrong conclusions.

In order to ensure no bias in sampling, one can observe the mean and variance for the success metric before the change is introduced.

Significance and Power

Once a difference between the two variants is observed, it is important to conclude that the change observed is an actual effect and not a random one. This can be done by computing the significance of the change in the success metric.

In layman’s terms, significance measures the frequency with which the test shows that bigger images lead to higher conversion when they actually don’t. Power measures the frequency with which the test tells us that bigger images lead to higher conversion when they actually do.

So, tests need to have a high value of power and a low value of significance for more accurate results.

While an in-depth exploration of the statistical concepts involved in product management hypothesis testing is out of scope here, the following actions are recommended to enhance knowledge on this front:

Data analysts and data engineers are usually adept at identifying the right test designs and can guide product managers, so make sure to utilize their expertise early in the process.
There are numerous online courses on hypothesis testing, A/B testing, and related statistical concepts, such as Udemy , Udacity , and Coursera .
Using tools such as Google’s Firebase and Optimizely can make the process easier thanks to a large amount of out-of-the-box capabilities for running the right tests.

Using Hypothesis Testing for Successful Product Management

In order to continuously deliver value to users, it is imperative to test various hypotheses, for the purpose of which several types of product hypothesis testing can be employed. Each hypothesis needs to have an accompanying test design, as described above, in order to conclusively validate or invalidate it.

This approach helps to quantify the value delivered by new changes and features, bring focus to the most valuable features, and deliver incremental iterations.

How to Conduct Remote User Interviews [Infographic]
A/B Testing UX for Component-based Frameworks
Building an AI Product? Maximize Value With an Implementation Framework

Understanding the basics

What is a product hypothesis.

A product hypothesis is an assumption that some improvement in the product will bring an increase in important metrics like revenue or product usage statistics.

What are the three required parts of a hypothesis?

The three required parts of a hypothesis are the assumption, the condition, and the prediction.

Why do we do A/B testing?

We do A/B testing to make sure that any improvement in the product increases our tracked metrics.

What is A/B testing used for?

A/B testing is used to check if our product improvements create the desired change in metrics.

What is A/B testing and multivariate testing?

A/B testing and multivariate testing are types of hypothesis testing. A/B testing checks how important metrics change with and without a single change in the product. Multivariate testing can track multiple variations of the same product improvement.

Kumara Raghavendra

Dubai, United Arab Emirates

Member since August 6, 2019

About the author

World-class articles, delivered weekly.

Subscription implies consent to our privacy policy

Toptal Product Managers

Artificial Intelligence Product Managers
Blockchain Product Managers
Business Systems Analysts
Cloud Product Managers
Data Science Product Managers
Digital Marketing Product Managers
Digital Product Managers
Directors of Product
eCommerce Product Managers
Enterprise Product Managers
Enterprise Resource Planning Product Managers
Freelance Product Managers
Interim CPOs
Jira Product Managers
Kanban Product Managers
Lean Product Managers
Mobile Product Managers
Product Consultants
Product Development Managers
Product Owners
Product Portfolio Managers
Product Strategy Consultants
Product Tour Consultants
Robotic Process Automation Product Managers
Robotics Product Managers
SaaS Product Managers
Salesforce Product Managers
Scrum Product Owner Contractors
Web Product Managers
View More Freelance Product Managers

Join the Toptal ® community.

Product Management

How to Generate and Validate Product Hypotheses

What is a product hypothesis.

A hypothesis is a testable statement that predicts the relationship between two or more variables. In product development, we generate hypotheses to validate assumptions about customer behavior, market needs, or the potential impact of product changes. These experimental efforts help us refine the user experience and get closer to finding a product-market fit.

Product hypotheses are a key element of data-driven product development and decision-making. Testing them enables us to solve problems more efficiently and remove our own biases from the solutions we put forward.

Here’s an example: ‘If we improve the page load speed on our website (variable 1), then we will increase the number of signups by 15% (variable 2).’ So if we improve the page load speed, and the number of signups increases, then our hypothesis has been proven. If the number did not increase significantly (or not at all), then our hypothesis has been disproven.

In general, product managers are constantly creating and testing hypotheses. But in the context of new product development , hypothesis generation/testing occurs during the validation stage, right after idea screening .

Now before we go any further, let’s get one thing straight: What’s the difference between an idea and a hypothesis?

Idea vs hypothesis

Innovation expert Michael Schrage makes this distinction between hypotheses and ideas – unlike an idea, a hypothesis comes with built-in accountability. “But what’s the accountability for a good idea?” Schrage asks. “The fact that a lot of people think it’s a good idea? That’s a popularity contest.” So, not only should a hypothesis be tested, but by its very nature, it can be tested.

At Railsware, we’ve built our product development services on the careful selection, prioritization, and validation of ideas. Here’s how we distinguish between ideas and hypotheses:

Idea: A creative suggestion about how we might exploit a gap in the market, add value to an existing product, or bring attention to our product. Crucially, an idea is just a thought. It can form the basis of a hypothesis but it is not necessarily expected to be proven or disproven.

We should get an interview with the CEO of our company published on TechCrunch.
Why don’t we redesign our website?
The Coupler.io team should create video tutorials on how to export data from different apps, and publish them on YouTube.
Why not add a new ‘email templates’ feature to our Mailtrap product?

Hypothesis: A way of framing an idea or assumption so that it is testable, specific, and aligns with our wider product/team/organizational goals.

Examples:

If we add a new ‘email templates’ feature to Mailtrap, we’ll see an increase in active usage of our email-sending API.
Creating relevant video tutorials and uploading them to YouTube will lead to an increase in Coupler.io signups.
If we publish an interview with our CEO on TechCrunch, 500 people will visit our website and 10 of them will install our product.

Now, it’s worth mentioning that not all hypotheses require testing . Sometimes, the process of creating hypotheses is just an exercise in critical thinking. And the simple act of analyzing your statement tells whether you should run an experiment or not. Remember: testing isn’t mandatory, but your hypotheses should always be inherently testable.

Let’s consider the TechCrunch article example again. In that hypothesis, we expect 500 readers to visit our product website, and a 2% conversion rate of those unique visitors to product users i.e. 10 people. But is that marginal increase worth all the effort? Conducting an interview with our CEO, creating the content, and collaborating with the TechCrunch content team – all of these tasks take time (and money) to execute. And by formulating that hypothesis, we can clearly see that in this case, the drawbacks (efforts) outweigh the benefits. So, no need to test it.

In a similar vein, a hypothesis statement can be a tool to prioritize your activities based on impact. We typically use the following criteria:

The quality of impact
The size of the impact
The probability of impact

This lets us organize our efforts according to their potential outcomes – not the coolness of the idea, its popularity among the team, etc.

Now that we’ve established what a product hypothesis is, let’s discuss how to create one.

Start with a problem statement

Before you jump into product hypothesis generation, we highly recommend formulating a problem statement. This is a short, concise description of the issue you are trying to solve. It helps teams stay on track as they formalize the hypothesis and design the product experiments. It can also be shared with stakeholders to ensure that everyone is on the same page.

The statement can be worded however you like, as long as it’s actionable, specific, and based on data-driven insights or research. It should clearly outline the problem or opportunity you want to address.

Here’s an example: Our bounce rate is high (more than 90%) and we are struggling to convert website visitors into actual users. How might we improve site performance to boost our conversion rate?

How to generate product hypotheses

Now let’s explore some common, everyday scenarios that lead to product hypothesis generation. For our teams here at Railsware, it’s when:

There’s a problem with an unclear root cause e.g. a sudden drop in one part of the onboarding funnel. We identify these issues by checking our product metrics or reviewing customer complaints.
We are running ideation sessions on how to reach our goals (increase MRR, increase the number of users invited to an account, etc.)
We are exploring growth opportunities e.g. changing a pricing plan, making product improvements , breaking into a new market.
We receive customer feedback. For example, some users have complained about difficulties setting up a workspace within the product. So, we build a hypothesis on how to help them with the setup.

BRIDGES framework for ideation

When we are tackling a complex problem or looking for ways to grow the product, our teams use BRIDGeS – a robust decision-making and ideation framework. BRIDGeS makes our product discovery sessions more efficient. It lets us dive deep into the context of our problem so that we can develop targeted solutions worthy of testing.

Between 2-8 stakeholders take part in a BRIDGeS session. The ideation sessions are usually led by a product manager and can include other subject matter experts such as developers, designers, data analysts, or marketing specialists. You can use a virtual whiteboard such as Figjam or Miro (see our Figma template ) to record each colored note.

In the first half of a BRIDGeS session, participants examine the Benefits, Risks, Issues, and Goals of their subject in the ‘Problem Space.’ A subject is anything that is being described or dealt with; for instance, Coupler.io’s growth opportunities. Benefits are the value that a future solution can bring, Risks are potential issues they might face, Issues are their existing problems, and Goals are what the subject hopes to gain from the future solution. Each descriptor should have a designated color.

After we have broken down the problem using each of these descriptors, we move into the Solution Space. This is where we develop solution variations based on all of the benefits/risks/issues identified in the Problem Space (see the Uber case study for an in-depth example).

In the Solution Space, we start prioritizing those solutions and deciding which ones are worthy of further exploration outside of the framework – via product hypothesis formulation and testing, for example. At the very least, after the session, we will have a list of epics and nested tasks ready to add to our product roadmap.

How to write a product hypothesis statement

Across organizations, product hypothesis statements might vary in their subject, tone, and precise wording. But some elements never change. As we mentioned earlier, a hypothesis statement must always have two or more variables and a connecting factor.

1. Identify variables

Since these components form the bulk of a hypothesis statement, let’s start with a brief definition.

First of all, variables in a hypothesis statement can be split into two camps: dependent and independent. Without getting too theoretical, we can describe the independent variable as the cause, and the dependent variable as the effect . So in the Mailtrap example we mentioned earlier, the ‘add email templates feature’ is the cause i.e. the element we want to manipulate. Meanwhile, ‘increased usage of email sending API’ is the effect i.e the element we will observe.

Independent variables can be any change you plan to make to your product. For example, tweaking some landing page copy, adding a chatbot to the homepage, or enhancing the search bar filter functionality.

Dependent variables are usually metrics. Here are a few that we often test in product development:

Number of sign-ups
Number of purchases
Activation rate (activation signals differ from product to product)
Number of specific plans purchased
Feature usage (API activation, for example)
Number of active users

Bear in mind that your concept or desired change can be measured with different metrics. Make sure that your variables are well-defined, and be deliberate in how you measure your concepts so that there’s no room for misinterpretation or ambiguity.

For example, in the hypothesis ‘Users drop off because they find it hard to set up a project’ variables are poorly defined. Phrases like ‘drop off’ and ‘hard to set up’ are too vague. A much better way of saying it would be: If project automation rules are pre-defined (email sequence to responsible, scheduled tickets creation), we’ll see a decrease in churn. In this example, it’s clear which dependent variable has been chosen and why.

And remember, when product managers focus on delighting users and building something of value, it’s easier to market and monetize it. That’s why at Railsware, our product hypotheses often focus on how to increase the usage of a feature or product. If users love our product(s) and know how to leverage its benefits, we can spend less time worrying about how to improve conversion rates or actively grow our revenue, and more time enhancing the user experience and nurturing our audience.

2. Make the connection

The relationship between variables should be clear and logical. If it’s not, then it doesn’t matter how well-chosen your variables are – your test results won’t be reliable.

To demonstrate this point, let’s explore a previous example again: page load speed and signups.

Through prior research, you might already know that conversion rates are 3x higher for sites that load in 1 second compared to sites that take 5 seconds to load. Since there appears to be a strong connection between load speed and signups in general, you might want to see if this is also true for your product.

Here are some common pitfalls to avoid when defining the relationship between two or more variables:

Relationship is weak. Let’s say you hypothesize that an increase in website traffic will lead to an increase in sign-ups. This is a weak connection since website visitors aren’t necessarily motivated to use your product; there are more steps involved. A better example is ‘If we change the CTA on the pricing page, then the number of signups will increase.’ This connection is much stronger and more direct.

Relationship is far-fetched. This often happens when one of the variables is founded on a vanity metric. For example, increasing the number of social media subscribers will lead to an increase in sign-ups. However, there’s no particular reason why a social media follower would be interested in using your product. Oftentimes, it’s simply your social media content that appeals to them (and your audience isn’t interested in a product).

Variables are co-dependent. Variables should always be isolated from one another. Let’s say we removed the option “Register with Google” from our app. In this case, we can expect fewer users with Google workspace accounts to register. Obviously, it’s because there’s a direct dependency between variables (no registration with Google→no users with Google workspace accounts).

3. Set validation criteria

First, build some confirmation criteria into your statement . Think in terms of percentages (e.g. increase/decrease by 5%) and choose a relevant product metric to track e.g. activation rate if your hypothesis relates to onboarding. Consider that you don’t always have to hit the bullseye for your hypothesis to be considered valid. Perhaps a 3% increase is just as acceptable as a 5% one. And it still proves that a connection between your variables exists.

Secondly, you should also make sure that your hypothesis statement is realistic . Let’s say you have a hypothesis that ‘If we show users a banner with our new feature, then feature usage will increase by 10%.’ A few questions to ask yourself are: Is 10% a reasonable increase, based on your current feature usage data? Do you have the resources to create the tests (experimenting with multiple variations, distributing on different channels: in-app, emails, blog posts)?

Null hypothesis and alternative hypothesis

In statistical research, there are two ways of stating a hypothesis: null or alternative. But this scientific method has its place in hypothesis-driven development too…

Alternative hypothesis: A statement that you intend to prove as being true by running an experiment and analyzing the results. Hint: it’s the same as the other hypothesis examples we’ve described so far.

Example: If we change the landing page copy, then the number of signups will increase.

Null hypothesis: A statement you want to disprove by running an experiment and analyzing the results. It predicts that your new feature or change to the user experience will not have the desired effect.

Example: The number of signups will not increase if we make a change to the landing page copy.

What’s the point? Well, let’s consider the phrase ‘innocent until proven guilty’ as a version of a null hypothesis. We don’t assume that there is any relationship between the ‘defendant’ and the ‘crime’ until we have proof. So, we run a test, gather data, and analyze our findings — which gives us enough proof to reject the null hypothesis and validate the alternative. All of this helps us to have more confidence in our results.

Now that you have generated your hypotheses, and created statements, it’s time to prepare your list for testing.

Prioritizing hypotheses for testing

Not all hypotheses are created equal. Some will be essential to your immediate goal of growing the product e.g. adding a new data destination for Coupler.io. Others will be based on nice-to-haves or small fixes e.g. updating graphics on the website homepage.

Prioritization helps us focus on the most impactful solutions as we are building a product roadmap or narrowing down the backlog . To determine which hypotheses are the most critical, we use the MoSCoW framework. It allows us to assign a level of urgency and importance to each product hypothesis so we can filter the best 3-5 for testing.

MoSCoW is an acronym for Must-have, Should-have, Could-have, and Won’t-have. Here’s a breakdown:

Must-have – hypotheses that must be tested, because they are strongly linked to our immediate project goals.
Should-have – hypotheses that are closely related to our immediate project goals, but aren’t the top priority.
Could-have – hypotheses of nice-to-haves that can wait until later for testing.
Won’t-have – low-priority hypotheses that we may or may not test later on when we have more time.

How to test product hypotheses

Once you have selected a hypothesis, it’s time to test it. This will involve running one or more product experiments in order to check the validity of your claim.

The tricky part is deciding what type of experiment to run, and how many. Ultimately, this all depends on the subject of your hypothesis – whether it’s a simple copy change or a whole new feature. For instance, it’s not necessary to create a clickable prototype for a landing page redesign. In that case, a user-wide update would do.

On that note, here are some of the approaches we take to hypothesis testing at Railsware:

A/B testing

A/B or split testing involves creating two or more different versions of a webpage/feature/functionality and collecting information about how users respond to them.

Let’s say you wanted to validate a hypothesis about the placement of a search bar on your application homepage. You could design an A/B test that shows two different versions of that search bar’s placement to your users (who have been split equally into two camps: a control group and a variant group). Then, you would choose the best option based on user data. A/B tests are suitable for testing responses to user experience changes, especially if you have more than one solution to test.

Prototyping

When it comes to testing a new product design, prototyping is the method of choice for many Lean startups and organizations. It’s a cost-effective way of collecting feedback from users, fast, and it’s possible to create prototypes of individual features too. You may take this approach to hypothesis testing if you are working on rolling out a significant new change e.g adding a brand-new feature, redesigning some aspect of the user flow, etc. To control costs at this point in the new product development process , choose the right tools — think Figma for clickable walkthroughs or no-code platforms like Bubble.

Deliveroo feature prototype example

Let’s look at how feature prototyping worked for the food delivery app, Deliveroo, when their product team wanted to ‘explore personalized recommendations, better filtering and improved search’ in 2018. To begin, they created a prototype of the customer discovery feature using web design application, Framer.

One of the most important aspects of this feature prototype was that it contained live data — real restaurants, real locations. For test users, this made the hypothetical feature feel more authentic. They were seeing listings and recommendations for real restaurants in their area, which helped immerse them in the user experience, and generate more honest and specific feedback. Deliveroo was then able to implement this feedback in subsequent iterations.

Asking your users

Interviewing customers is an excellent way to validate product hypotheses. It’s a form of qualitative testing that, in our experience, produces better insights than user surveys or general user research. Sessions are typically run by product managers and involve asking in-depth interview questions to one customer at a time. They can be conducted in person or online (through a virtual call center , for instance) and last anywhere between 30 minutes to 1 hour.

Although CustDev interviews may require more effort to execute than other tests (the process of finding participants, devising questions, organizing interviews, and honing interview skills can be time-consuming), it’s still a highly rewarding approach. You can quickly validate assumptions by asking customers about their pain points, concerns, habits, processes they follow, and analyzing how your solution fits into all of that.

Wizard of Oz

The Wizard of Oz approach is suitable for gauging user interest in new features or functionalities. It’s done by creating a prototype of a fake or future feature and monitoring how your customers or test users interact with it.

For example, you might have a hypothesis that your number of active users will increase by 15% if you introduce a new feature. So, you design a new bare-bones page or simple button that invites users to access it. But when they click on the button, a pop-up appears with a message such as ‘coming soon.’

By measuring the frequency of those clicks, you could learn a lot about the demand for this new feature/functionality. However, while these tests can deliver fast results, they carry the risk of backfiring. Some customers may find fake features misleading, making them less likely to engage with your product in the future.

User-wide updates

One of the speediest ways to test your hypothesis is by rolling out an update for all users. It can take less time and effort to set up than other tests (depending on how big of an update it is). But due to the risk involved, you should stick to only performing these kinds of tests on small-scale hypotheses. Our teams only take this approach when we are almost certain that our hypothesis is valid.

For example, we once had an assumption that the name of one of Mailtrap ’s entities was the root cause of a low activation rate. Being an active Mailtrap customer meant that you were regularly sending test emails to a place called ‘Demo Inbox.’ We hypothesized that the name was confusing (the word ‘demo’ implied it was not the main inbox) and this was preventing new users from engaging with their accounts. So, we updated the page, changed the name to ‘My Inbox’ and added some ‘to-do’ steps for new users. We saw an increase in our activation rate almost immediately, validating our hypothesis.

Feature flags

Creating feature flags involves only releasing a new feature to a particular subset or small percentage of users. These features come with a built-in kill switch; a piece of code that can be executed or skipped, depending on who’s interacting with your product.

Since you are only showing this new feature to a selected group, feature flags are an especially low-risk method of testing your product hypothesis (compared to Wizard of Oz, for example, where you have much less control). However, they are also a little bit more complex to execute than the others — you will need to have an actual coded product for starters, as well as some technical knowledge, in order to add the modifiers ( only when… ) to your new coded feature.

Let’s revisit the landing page copy example again, this time in the context of testing.

So, for the hypothesis ‘If we change the landing page copy, then the number of signups will increase,’ there are several options for experimentation. We could share the copy with a small sample of our users, or even release a user-wide update. But A/B testing is probably the best fit for this task. Depending on our budget and goal, we could test several different pieces of copy, such as:

The current landing page copy
Copy that we paid a marketing agency 10 grand for
Generic copy we wrote ourselves, or removing most of the original copy – just to see how making even a small change might affect our numbers.

Remember, every hypothesis test must have a reasonable endpoint. The exact length of the test will depend on the type of feature/functionality you are testing, the size of your user base, and how much data you need to gather. Just make sure that the experiment running time matches the hypothesis scope. For instance, there is no need to spend 8 weeks experimenting with a piece of landing page copy. That timeline is more appropriate for say, a Wizard of Oz feature.

Recording hypotheses statements and test results

Finally, it’s time to talk about where you will write down and keep track of your hypotheses. Creating a single source of truth will enable you to track all aspects of hypothesis generation and testing with ease.

At Railsware, our product managers create a document for each individual hypothesis, using tools such as Coda or Google Sheets. In that document, we record the hypothesis statement, as well as our plans, process, results, screenshots, product metrics, and assumptions.

We share this document with our team and stakeholders, to ensure transparency and invite feedback. It’s also a resource we can refer back to when we are discussing a new hypothesis — a place where we can quickly access information relating to a previous test.

Understanding test results and taking action

The other half of validating product hypotheses involves evaluating data and drawing reasonable conclusions based on what you find. We do so by analyzing our chosen product metric(s) and deciding whether there is enough data available to make a solid decision. If not, we may extend the test’s duration or run another one. Otherwise, we move forward. An experimental feature becomes a real feature, a chatbot gets implemented on the customer support page, and so on.

Something to keep in mind: the integrity of your data is tied to how well the test was executed, so here are a few points to consider when you are testing and analyzing results:

Gather and analyze data carefully. Ensure that your data is clean and up-to-date when running quantitative tests and tracking responses via analytics dashboards. If you are doing customer interviews, make sure to record the meetings (with consent) so that your notes will be as accurate as possible.

Conduct the right amount of product experiments. It can take more than one test to determine whether your hypothesis is valid or invalid. However, don’t waste too much time experimenting in the hopes of getting the result you want. Know when to accept the evidence and move on.

Choose the right audience segment. Don’t cast your net too wide. Be specific about who you want to collect data from prior to running the test. Otherwise, your test results will be misleading and you won’t learn anything new.

Watch out for bias. Avoid confirmation bias at all costs. Don’t make the mistake of including irrelevant data just because it bolsters your results. For example, if you are gathering data about how users are interacting with your product Monday-Friday, don’t include weekend data just because doing so would alter the data and ‘validate’ your hypothesis.

Not all failed hypotheses should be treated as losses. Even if you didn’t get the outcome you were hoping for, you may still have improved your product. Let’s say you implemented SSO authentication for premium users, but unfortunately, your free users didn’t end up switching to premium plans. In this case, you still added value to the product by streamlining the login process for paying users.
Yes, taking a hypothesis-driven approach to product development is important. But remember, you don’t have to test everything . Use common sense first. For example, if your website copy is confusing and doesn’t portray the value of the product, then you should still strive to replace it with better copy – regardless of how this affects your numbers in the short term.

Wrapping Up

The process of generating and validating product hypotheses is actually pretty straightforward once you’ve got the hang of it. All you need is a valid question or problem, a testable statement, and a method of validation. Sure, hypothesis-driven development requires more of a time commitment than just ‘giving it a go.’ But ultimately, it will help you tune the product to the wants and needs of your customers.

If you share our data-driven approach to product development and engineering, check out our services page to learn more about how we work with our clients!

Hypothesis Driven Product Management

Post author By admin
Post date September 23, 2020
No Comments on Hypothesis Driven Product Management

hypothesis testing in product management

What is Lean Hypothesis Testing?

“The first principle is that you must not fool yourself and you are the easiest person to fool.” – Richard P. Feynman

Lean hypothesis testing is an approach to agile product development that’s designed to minimize risk, increase the speed of development, and hone business outcomes by building and iterating on a minimum viable product (MVP).

The minimum viable product is a concept famously championed by Eric Ries as part of the lean startup methodology. At its core, the concept of the MVP is about creating a cycle of learning. Rather than devoting long development timelines to building a fully polished end product, teams working through lean product development build, in short, iterative cycles. Each cycle is devoted to shipping an MVP, defined as a product that’s built with the least amount of work possible for the purpose of testing and validating that product with users.

In lean hypothesis testing, the MVP itself can be framed as a hypothesis. A well-designed hypothesis breaks down an issue into a problem, solution, and result.

When defining a good hypothesis, start with a meaningful problem: an issue or pain-point that you’d like to solve for your users. Teams often use multiple qualitative and quantitative sources to the scope and describe this problem.

How do you get started?

Two core practices underlie lean:

Use of the scientific method and
Use of small batches. Science has brought us many wonderful things.

I personally prefer to expand the Build-Measure-Learn loop into the classic view of the scientific method because I find it’s more robust. You can see that process to the right, and we’ll step through the components in the balance of this section.

The use of small batches is critical. It gives you more shots at a successful outcome, particularly valuable when you’re in a high risk, high uncertainty environment.

A great example from Eric Ries’ book is the envelope folding experiment: If you had to stuff 100 envelopes with letters, how would you do it? Would you fold all the sheets of paper and then stuff the envelopes? Or would you fold one sheet of paper, stuff one envelope? It turns out that doing them one by one is vastly more efficient, and that’s just on an operational basis. If you don’t actually know if the envelopes will fit or whether anyone wants them (more analogous to a startup), you’re obviously much better off with the one-by-one approach.

So, how do you do it? In 6 simple (in principle) steps :

Start with a strong idea , one where you’ve gone out a done customer strong discovery which is packaged into testable personas and problem scenarios. If you’re familiar with design thinking, it’s very much about doing good work in this area.
Structure your idea(s) in a testable format (as hypotheses).
Figure out how you’ll prove or disprove these hypotheses with a minimum of time and effort.
Get focused on testing your hypotheses and collecting whatever metrics you’ll use to make a conclusion.
Conclude and decide ; did you prove out this idea and is it time to throw more resources at it? Or do you need to reformulate and re-test?
Pivot or persevere ; If you’re pivoting and revising, the key is to make sure you have a strong foundation in customer discovery so you can pivot in a smart way based on your understanding of the customer/user.

By using a hypothesis-driven development process you:

Articulate your thinking
Provide others with an understanding of your thinking
Create a framework to test your designs against
Develop a standard way of documenting your work
Make better stuff

Free Template: Lean Hypothesis template

Eric Ries: Test & experiment, turn your feeling into a hypothesis

5 case studies on experimentation :.

Adobe takes a customer-centric to innovating Photoshop
Test Paper prototypes to save time and money: the Mozilla case study
Walmart.ca increases on-site conversions by 13%
Icons8 web app. Redesign based on usability testing.
Experiments at Airbnb
Tags Hypothesis Driven

Value Hypothesis 101: A Product Manager's Guide

Talk to Sales

Humans make assumptions every day—it’s our brain’s way of making sense of the world around us, but assumptions are only valuable if they're verifiable . That’s where a value hypothesis comes in as your starting point.

A good hypothesis goes a step beyond an assumption. It’s a verifiable and validated guess based on the value your product brings to your real-life customers. When you verify your hypothesis, you confirm that the product has real-world value, thus you have a higher chance of product success.

What Is a Verifiable Value Hypothesis?

A value hypothesis is an educated guess about the value proposition of your product. When you verify your hypothesis , you're using evidence to prove that your assumption is correct. A hypothesis is verifiable if it does not prove false through experimentation or is shown to have rational justification through data, experiments, observation, or tests.

The most significant benefit of verifying a hypothesis is that it helps you avoid product failure and helps you build your product to your customers’ (and potential customers’) needs.

Verifying your assumptions is all about collecting data. Without data obtained through experiments, observations, or tests, your hypothesis is unverifiable, and you can’t be sure there will be a market need for your product.

A Verifiable Value Hypothesis Minimizes Risk and Saves Money

When you verify your hypothesis, you’re less likely to release a product that doesn’t meet customer expectations—a waste of your company’s resources. Harvard Business School explains that verifying a business hypothesis “...allows an organization to verify its analysis is correct before committing resources to implement a broader strategy.”

If you verify your hypothesis upfront, you’ll lower risk and have time to work out product issues.

UserVoice Validation makes product validation accessible to everyone. Consider using its research feature to speed up your hypothesis verification process.

Value Hypotheses vs. Growth Hypotheses

Your value hypothesis focuses on the value of your product to customers. This type of hypothesis can apply to a product or company and is a building block of product-market fit .

A growth hypothesis is a guess at how your business idea may develop in the long term based on how potential customers may find your product. It’s meant for estimating business model growth rather than individual products.

Because your value hypothesis is really the foundation for your growth hypothesis, you should focus on value hypothesis tests first and complete growth hypothesis tests to estimate business growth as a whole once you have a viable product.

4 Tips to Create and Test a Verifiable Value Hypothesis

A verifiable hypothesis needs to be based on a logical structure, customer feedback data , and objective safeguards like creating a minimum viable product. Validating your value significantly reduces risk . You can prevent wasting money, time, and resources by verifying your hypothesis in early-stage development.

A good value hypothesis utilizes a framework (like the template below), data, and checks/balances to avoid bias.

1. Use a Template to Structure Your Value Hypothesis

By using a template structure, you can create an educated guess that includes the most important elements of a hypothesis—the who, what, where, when, and why. If you don’t structure your hypothesis correctly, you may only end up with a flimsy or leap-of-faith assumption that you can’t verify.

A true hypothesis uses a few guesses about your product and organizes them so that you can verify or falsify your assumptions. Using a template to structure your hypothesis can ensure that you’re not missing the specifics.

You can’t just throw a hypothesis together and think it will answer the question of whether your product is valuable or not. If you do, you could end up with faulty data informed by bias , a skewed significance level from polling the wrong people, or only a vague idea of what your customer would actually pay for your product.

A template will help keep your hypothesis on track by standardizing the structure of the hypothesis so that each new hypothesis always includes the specifics of your client personas, the cost of your product, and client or customer pain points.

A value hypothesis template might look like:

[Client] will spend [cost] to purchase and use our [title of product/service] to solve their [specific problem] OR help them overcome [specific obstacle].

An example of your hypothesis might look like:

B2B startups will spend $500/mo to purchase our resource planning software to solve resource over-allocation and employee burnout.

By organizing your ideas and the important elements (who, what, where, when, and why), you can come up with a hypothesis that actually answers the question of whether your product is useful and valuable to your ideal customer.

2. Turn Customer Feedback into Data to Support Your Hypothesis

Once you have your hypothesis, it’s time to figure out whether it’s true—or, more accurately, prove that it’s valid. Since a hypothesis is never considered “100% proven,” it’s referred to as either valid or invalid based on the information you discover in your experiments or tests. Additionally, your results could lead to an alternative hypothesis, which is helpful in refining your core idea.

To support value hypothesis testing, you need data. To do that, you'll want to collect customer feedback . A customer feedback management tool can also make it easier for your team to access the feedback and create strategies to implement or improve customer concerns.

If you find that potential clients are not expressing pain points that could be solved with your product or you’re not seeing an interest in the features you hope to add, you can adjust your hypothesis and absorb a lower risk. Because you didn’t invest a lot of time and money into creating the product yet, you should have more resources to put toward the product once you work out the kinks.

On the other hand, if you find that customers are requesting features your product offers or pain points your product could solve, then you can move forward with product development, confident that your future customers will value (and spend money on) the product you’re creating.

A customer feedback management tool like UserVoice can empower you to challenge assumptions from your colleagues (often based on anecdotal information) which find their way into team decision making . Having data to reevaluate an assumption helps with prioritization, and it confirms that you’re focusing on the right things as an organization.

3. Validate Your Product

Since you have a clear idea of who your ideal customer is at this point and have verified their need for your product, it’s time to validate your product and decide if it’s better than your competitors’.

At this point, simply asking your customers if they would buy your product (or spend more on your product) instead of a competitor’s isn’t enough confirmation that you should move forward, and customers may be biased or reluctant to provide critical feedback.

Instead, create a minimum viable product (MVP). An MVP is a working, bare-bones version of the product that you can test out without risking your whole budget. Hypothesis testing with an MVP simulates the product experience for customers and, based on their actions and usage, validates that the full product will generate revenue and be successful.

If you take the steps to first verify and then validate your hypothesis using data, your product is more likely to do well. Your focus will be on the aspect that matters most—whether your customer actually wants and would invest money in purchasing the product.

4. Use Safeguards to Remain Objective

One of the pitfalls of believing in your product and attempting to validate it is that you’re subject to confirmation bias . Because you want your product to succeed, you may pay more attention to the answers in the collected data that affirm the value of your product and gloss over the information that may lead you to conclude that your hypothesis is actually false. Confirmation bias could easily cloud your vision or skew your metrics without you even realizing it.

Since it’s hard to know when you’re engaging in confirmation bias, it’s good to have safeguards in place to keep you in check and aligned with the purpose of objectively evaluating your value hypothesis.

Safeguards include sharing your findings with third-party experts or simply putting yourself in the customer’s shoes.

Third-party experts are the business version of seeking a peer review. External parties don’t stand to benefit from the outcome of your verification and validation process, so your work is verified and validated objectively. You gain the benefit of knowing whether your hypothesis is valid in the eyes of the people who aren’t stakeholders without the risk of confirmation bias.

In addition to seeking out objective minds, look into potential counter-arguments , such as customer objections (explicit or imagined). What might your customer think about investing the time to learn how to use your product? Will they think the value is commensurate with the monetary cost of the product?

When running an experiment on validating your hypothesis, it’s important not to elevate the importance of your beliefs over the objective data you collect. While it can be exciting to push for the validity of your idea, it can lead to false assumptions and the permission of weak evidence.

Validation Is the Key to Product Success

With your new value hypothesis in hand, you can confidently move forward, knowing that there’s a true need, desire, and market for your product.

Because you’ve verified and validated your guesses, there’s less of a chance that you’re wrong about the value of your product, and there are fewer financial and resource risks for your company. With this strong foundation and the new information you’ve uncovered about your customers, you can add even more value to your product or use it to make more products that fit the market and user needs.

If you think customer feedback management software would be useful in your hypothesis validation process, consider opting into our free trial to see how UserVoice can help.

Heather Tipton

Start your free trial.

The Product Management Dictionary: hypothesis testing

Learn about hypothesis testing in product management with our comprehensive dictionary.

Hypothesis testing is a vital part of product management, providing a framework to test assumptions and make data-driven decisions. In this article, we'll explore the concept of hypothesis testing, how it works, and some common metrics and KPIs that you can use to measure success.

Understanding Hypothesis Testing in Product Management

At its core, hypothesis testing is a way to validate assumptions by gathering data through experimentation. In product management, it's used to test the effectiveness of new features or product changes, allowing teams to make informed decisions about how to move forward.

The Importance of Hypothesis Testing

Hypothesis testing is critical in product management because it allows teams to determine whether a proposed change or feature will have an impact on key metrics, such as conversion rates or user engagement. By conducting experiments and collecting data, teams can make more informed decisions and minimize risk.

Key Terminology and Concepts

Before we dive into the process of hypothesis testing, it's important to understand some key terminology and concepts.

Hypothesis: A testable statement about the expected outcome of an experiment.
Null hypothesis: A hypothesis that proposes there is no significant difference between groups or variables.
Alternative hypothesis: A hypothesis that proposes there is a significant difference between groups or variables.
Statistical significance: The likelihood of a result occurring by chance.

One important thing to note is that hypothesis testing is not just about proving or disproving a hypothesis. It's also about gathering data and insights that can inform future product decisions. For example, if a hypothesis is not supported by the data, it's important to understand why and use that information to refine the hypothesis or adjust the product strategy.

Another key concept in hypothesis testing is sample size. The larger the sample size, the more reliable the results will be. However, it's also important to ensure that the sample is representative of the target population. For example, if a product is targeted towards a specific demographic, it's important to ensure that the sample includes a similar demographic.

It's also important to consider the type of experiment being conducted. A randomized controlled trial is often considered the gold standard for hypothesis testing, as it allows for the most control over variables. However, there may be situations where other types of experiments, such as A/B testing or cohort analysis, are more appropriate.

Ultimately, hypothesis testing is a powerful tool for product managers, allowing them to make data-driven decisions and minimize risk. By understanding key terminology and concepts, and carefully designing experiments, product teams can gather valuable insights that can inform future product decisions and drive success.

The Hypothesis Testing Process

The hypothesis testing process is a fundamental aspect of scientific research and experimentation. It is a systematic approach to testing a hypothesis, which involves several key steps. These steps help to ensure that the testing is rigorous, reliable, and can be replicated by others.

Formulating a Hypothesis

The first step in the hypothesis testing process is to formulate a hypothesis. This is a statement that describes the expected outcome of an experiment. A hypothesis should be specific, measurable, and testable. This means that you should be able to track results and draw meaningful conclusions from your data.

For example, if you are testing a new website design, your hypothesis might be: "Changing the website design will lead to a 20% increase in user engagement."

Designing an Experiment

Once you have formulated your hypothesis, the next step is to design an experiment to test it. This involves developing a plan for how you will collect data and what variables you will measure. Your experiment should be designed to test your hypothesis in a controlled and systematic way.

For example, if you are testing the impact of a new website design, you may choose to divide users into two groups: one group that sees the new design, and one group that sees the old design. From there, you can track user engagement for each group to determine the impact of the new design on overall engagement.

Collecting and Analyzing Data

Once you have designed your experiment, the next step is to collect and analyze data. This involves tracking and recording data in a clear and organized way, so that you can draw meaningful conclusions from your results.

For example, you may track user engagement for both the group that sees the new design and the group that sees the old design. You can then compare the results to determine whether the new design has a significant impact on user engagement.

Drawing Conclusions and Making Decisions

The final step in the hypothesis testing process is to draw conclusions from your data and make informed decisions about how to move forward. If your data supports your hypothesis, you may choose to implement the change or feature that you tested. If your data does not support your hypothesis, you may choose to pivot and try a new approach.

Overall, the hypothesis testing process is a powerful tool for scientific research and experimentation. By following these key steps, you can ensure that your testing is rigorous, reliable, and can be replicated by others. This helps to build a strong foundation of knowledge and understanding in your field of study.

Types of Hypothesis Tests

As a product manager, it's important to use hypothesis testing to make data-driven decisions. There are several different types of hypothesis tests that you can use, each with its own strengths and weaknesses. Let's take a closer look at some of the most popular options.

A/B Testing

A/B testing is perhaps the most well-known type of hypothesis test, and involves randomly dividing users into two groups: one group that sees the original version of a feature, and one group that sees a variation of that feature. From there, you can track the performance of each group to determine which version of the feature outperforms the other.

For example, let's say you're testing a new button color on your website. You could randomly show half of your users the original blue button, and the other half a green button. By tracking the click-through rates of each group, you can determine which color is more effective at getting users to take the desired action.

Multivariate Testing

Multivariate testing is similar to A/B testing, but involves testing multiple variations of a feature simultaneously. This can be helpful when testing multiple features at once, or when trying to optimize a particular feature.

For example, let's say you're testing a new landing page for your website. You could test multiple variations of the page, each with different headlines, images, and calls-to-action. By tracking the performance of each variation, you can determine which combination of elements is most effective at converting visitors into customers.

Bayesian Hypothesis Testing

Bayesian hypothesis testing is a more complex type of hypothesis test that allows you to update your beliefs based on new data. This can be particularly useful when testing complex features or changes, where there may be several variables at play.

For example, let's say you're testing a new pricing model for your product. You could use Bayesian hypothesis testing to update your beliefs about the effectiveness of the new model as you collect more data. This would allow you to make more informed decisions about whether to stick with the new model or revert back to the old one.

Overall, there are many different types of hypothesis tests that you can use as a product manager. By understanding the strengths and weaknesses of each approach, you can make more informed decisions and drive better results for your business.

Common Metrics and KPIs in Hypothesis Testing

When it comes to hypothesis testing, there are a variety of metrics and KPIs that can help you measure the success of your experiments. In addition to tracking the specific outcomes you're interested in, it's important to keep an eye on some common metrics and KPIs to get a more complete picture of how your changes are impacting your product or service.

Conversion Rates

One of the most important metrics to track during hypothesis testing is conversion rates. Conversion rates are a measure of how many users take a desired action, such as making a purchase or signing up for a newsletter. By tracking conversion rates, you can determine whether a particular change or feature is leading to more conversions. For example, if you're testing a new checkout process, you might track conversion rates to see if the new process is leading to more completed purchases.

It's important to keep in mind that conversion rates can be impacted by a variety of factors, such as traffic sources, seasonality, and user behavior. To get a more accurate picture of how your changes are impacting conversion rates, it's often helpful to segment your data by different user groups or traffic sources.

User Engagement Metrics

In addition to conversion rates, it's important to track user engagement metrics during hypothesis testing. User engagement metrics, such as time on site or number of pages viewed, can provide insights into how engaged users are with your product. By tracking these metrics, you can determine whether a particular change or feature is leading to more engaged users.

For example, if you're testing a new homepage design, you might track metrics like bounce rate, time on site, and pages per session to see if the new design is leading to more engaged users. Keep in mind that user engagement metrics can also be impacted by a variety of factors, such as the quality of your content and the ease of navigation on your site.

Retention and Churn Rates

Retention and churn rates are also important metrics to track during hypothesis testing. Retention rates measure how many users continue to use your product over time, while churn rates measure how many users stop using it. By tracking these metrics, you can determine whether a particular change or feature is leading to improved retention and reduced churn.

For example, if you're testing a new onboarding process, you might track retention and churn rates to see if the new process is leading to more users sticking around for the long term. Keep in mind that retention and churn rates can be impacted by a variety of factors, such as the quality of your product and the level of competition in your market.

Overall, tracking these common metrics and KPIs can help you get a more complete picture of how your hypothesis testing experiments are impacting your product or service. By analyzing these metrics alongside your specific outcomes of interest, you can make more informed decisions about how to optimize your product or service for success.

Hypothesis testing is a critical part of product management, allowing teams to validate assumptions and make data-driven decisions. By understanding the process of hypothesis testing, as well as common metrics and KPIs, you can make more informed decisions and drive the success of your product.

Join hundreds of teams launching with Ignition

The first platform built explicitly to align product, product marketing, and GTM teams around product updates.

vs. Productboard

vs. Clickup

vs. Monday.com

Battlecard generator

Persona generator

Google ad copy generator

State of GTM Report

Customer stories

Help center

Product Talk

Make better product decisions.

Hypothesis Testing

As you get started with hypothesis testing, be sure to use these resources to make sure you get the most out of your experiments.

Start here to understand the big picture:

Why You Aren’t Learning as Much as You Could from Your Experiments

And then dive into these to master the tactics:

The 14 Most Common Hypothesis Testing Mistakes People Make (And How to Avoid Them)
Not Knowing What You Want To Learn
How to Improve Your Experiment Design (And Build Trust in Your Product Experiments)
How to Estimate the Expected Impact of a Product Change
Putting the 4 Levels of Product Analysis Into Practice: A Halloween-Themed Example
What to Do When You Don’t Have Enough Traffic to A/B Test

More than just knowing the mechanics of how to run a good experiment, you also need to know what to test and when.

Don’t Rely on Confidence Alone
Run Experiments Before You Write Code
Why You Can (And Should) Experiment When Building Enterprise Products

Hypothesis Testing Video Series

Hypothesis Testing: Levels of Product Analysis

Hypothesis Testing: The 5 Components of a Good Hypothesis

Do you want to learn more about hypothesis testing? Subscribe to the Product Talk mailing list to get new articles and videos delivered to your inbox.

Popular Resources

Product Discovery Basics: Everything You Need to Know
Visualize Your Thinking with Opportunity Solution Trees
Customer Interviews: How to Recruit, What to Ask, and How to Synthesize What You Learn
Assumption Testing: Everything You Need to Know to Get Started

Advisory boards aren’t only for executives. Join the LogRocket Content Advisory Board today →

Product Management
Solve User-Reported Issues
Find Issues Faster
Optimize Conversion and Adoption

How to write an effective hypothesis

Hypothesis validation is the bread and butter of product discovery. Understanding what should be prioritized and why is the most important task of a product manager. It doesn’t matter how well you validate your findings if you’re trying to answer the wrong question.

A question is as good as the answer it can provide. If your hypothesis is well written, but you can’t read its conclusion, it’s a bad hypothesis. Alternatively, if your hypothesis has embedded bias and answers itself, it’s also not going to help you.

There are several different tools available to build hypotheses, and it would be exhaustive to list them all. Apart from being superficial, focusing on the frameworks alone shifts the attention away from the hypothesis itself.

In this article, you will learn what a hypothesis is, the fundamental aspects of a good hypothesis, and what you should expect to get out of one.

The 4 product risks

Mitigating the four product risks is the reason why product managers exist in the first place and it’s where good hypothesis crafting starts.

The four product risks are assessments of everything that could go wrong with your delivery. Our natural thought process is to focus on the happy path at the expense of unknown traps. The risks are a constant reminder that knowing why something won’t work is probably more important than knowing why something might work.

These are the fundamental questions that should fuel your hypothesis creation:

Is it viable for the business?

Is it relevant for the user, can we build it, is it ethical to deliver.

Is this hypothesis the best one to validate now? Is this the most cost-effective initiative we can take? Will this answer help us achieve our goals? How much money can we make from it?

Has the user manifested interest in this solution? Will they be able to use it? Does it solve our users’ challenges? Is it aesthetically pleasing? Is it vital for the user, or just a luxury?

Do we have the resources and know-how to deliver it? Can we scale this solution? How much will it cost? Will it depreciate fast? Is it the best cost-effective solution? Will it deliver on what the user needs?

Is this solution safe both for the user and for the business? Is it inclusive enough? Is there a risk of public opinion whiplash? Is our solution enabling wrongdoers? Are we jeopardizing some to privilege others?

Over 200k developers and product managers use LogRocket to create better digital experiences

There is an infinite amount of questions that can surface from these risks, and most of those will be context dependent. Your industry, company, marketplace, team composition, and even the type of product you handle will impose different questions, but the risks remain the same.

How to decide whether your hypothesis is worthy of validation

Assuming you came up with a hefty batch of risks to validate, you must now address them. To address a risk, you could do one of three things: collect concrete evidence that you can mitigate that risk, infer possible ways you can mitigate a risk and, finally, deep dive into that risk because you’re not sure about its repercussions.

This three way road can be illustrated by a CSD matrix :

Certainties

Suppositions.

Everything you’re sure can help you to mitigate whatever risk. An example would be, on the risk “how to build it,” assessing if your engineering team is capable of integrating with a certain API. If your team has made it a thousand times in the past, it’s not something worth validating. You can assume it is true and mark this particular risk as solved.

To put it simply, a supposition is something that you think you know, but you’re not sure. This is the most fertile ground to explore hypotheses, since this is the precise type of answer that needs validation. The most common usage of supposition is addressing the “is it relevant for the user” risk. You presume that clients will enjoy a new feature, but before you talk to them, you can’t say you are sure.

Doubts are different from suppositions because they have no answer whatsoever. A doubt is an open question about a risk which you have no clue on how to solve. A product manager that tries to mitigate the “is it ethical to deliver” risk from an industry that they have absolute no familiarity with is poised to generate a lot of doubts, but no suppositions or certainties. Doubts are not good hypothesis sources, since you have no idea on how to validate it.

A hypothesis worth validating comes from a place of uncertainty, not confidence or doubt. If you are sure about a risk mitigation, coming up with a hypothesis to validate it is just a waste of time and resources. Alternatively, trying to come up with a risk assessment for a problem you are clueless about will probably generate hypotheses disconnected with the problem itself.

That said, it’s important to make it clear that suppositions are different from hypotheses. A supposition is merely a mental exercise, creativity executed. A hypothesis is a measurable, cartesian instrument to transform suppositions into certainties, therefore making sure you can mitigate a risk.

How to craft a hypothesis

A good hypothesis comes from a supposed solution to a specific product risk. That alone is good enough to build half of a good hypothesis, but you also need to have measurable confidence.

What comes after a good hypothesis?

The final piece of this puzzle comes after the hypothesis crafting. A hypothesis is only as good as the validation it provides, and that means you have to test it.

If we were to test the first hypothesis we crafted, “we believe that if we implement a one-click checkout on our ecommerce, we can grow our sales conversion by 30 percent,” you could come up with a testing roadmap to build up evidence that would eventually confirm or deny your hypothesis. Some examples of tests are:

A/B testing — Launch a quick and dirty one-click checkout MVP for a controlled group of users and compare their sales conversion rates against a control group. This will provide direct evidence on the effect of the feature on sales conversions

Customer support feedback — Track any inquiries or complaints related to the checkout process. You can use organic user complaints as an indirect measure of latent demand for one-click checkout feature

User survey — Ask why carts were abandoned for a cohort of shoppers that left the checkout step close to completion. Their reasons might indicate the possible success of your hypothesis

Effective hypothesis crafting is at the center of product management. It’s the link between dealing with risks and coming up with solutions that are both viable and valuable. However, it’s important to recognize that the formulation of a hypothesis is just the first step.

The real value of a hypothesis is made possible by rigorous testing. It’s through systematic validation that product managers can transform suppositions into certainties, ensuring the right product decisions are made. Without validation, even the most well-thought-out hypothesis remains unverified.

Featured image source: IconScout

LogRocket generates product insights that lead to meaningful action

Get your teams on the same page — try LogRocket today.

Stop guessing about your digital experience with LogRocket

Leader Spotlight: Closing the gap in nutrition literacy, with Peter Chau

Peter Chau discusses MyFitnessPal’s focus to close the gap in nutrition literacy by educating users on adequate nutrition intake and health.

Mastering customer surveys: Design, execution, and analysis

A customer survey is a structured research tool that product people use to gather insights about their customers.

Leader Spotlight: Growing the omnichannel market, with Christine Kuei

Christine Kuei, Director of Product Management at Forever 21, shares her experience growing and optimizing omnichannel experiences.

Decoding marketing jargon: A glossary of terms

The world of product marketing is always evolving. Even for experts, it can be hard to keep up with the latest concepts, terms, and jargon.

Hypothesis Testing: How to do it the right way

Home » Hypothesis Testing: How to do it the right way

“I believe that if we change the design of the landing page, it will lead to an improvement in signups”. In the regular, “normal” vocabulary of the natural world, the opening quote sentence is a passable hypothesis. However, in the world of product discovery, it is a terrible one. And the product discovery process will see a Product Manager formulate and make decisions based on hypothesis on an iterative basis. This is why we must conduct hypothesis testing the right way.

But what is Hypothesis Testing?

Simply put, Hypothesis Testing is a technique in product management that allows a product manager to validate their ideas about a product in the Product Discovery process.

A breakdown of the major “parts” of a hypothesis.

In hypothesis testing, after formulating a hypothesis, data gathering is done to test it. There are two types of hypotheses: null and alternative. The null hypothesis states that there is no difference or relationship between the two variables, while the alternative hypothesis states that there is a relationship or difference between the variables.

At the beginning of this article, we introduced a hypothesis that we said was terrible. A correct hypothesis concerning the same scenario would be:

A breakdown of a correct hypothesis for Hypothesis Testing during Product Discovery.

So, let us break down the most important things to note when conducting Hypothesis Testing.

Be Specific

This is probably the most essential thing to note about hypothesis testing. For instance, the first thing to note in the first “bad” hypothesis we introduced was that the landing page redesign was loosely defined. What aspect of the landing page is being changed? The colors? The button placement?

Also note that in the good hypothesis, the “impact” question of the hypothesis was practical and specific. According to Product expert Teresa Torres , saying a design change will “increase usability” is not specific enough. Why? Because it is not measurable. The same goes for hypothesizing an increase in engagement. Engagement, though measurable, is still not specific enough. Will it increase the time spent on the site? The number of button interactions? The email signups?

Product Managers should also note that targeting your hypothesis to a specific group of people is the only way to truly narrow it down to a measurable metric. Like the example in the diagram above, simply saying “design change x should…increase conversion of users” is not enough. What type of users are you targeting with this design change? Are you targeting seasoned experts? Or power users? Or first-time users? Is a user already utilizing a competitor’s product?

Being specific in hypothesis testing also involves measuring the best-guess degree of improvement the design change could provide for your product. This is often not more than guesswork, but if done right, it could make a world of difference between what design changes are thrown out and which ones are kept. For instance, if the degree of improvement expected from the hypothesis being tested is a 10 percent increase in conversion rate, then a 9 percent increase should denote a failure. This might seem extreme, but it helps protect your product from biases and mediocrity and might even inform your future estimates of what an acceptable expectation of improvement should be.

Finally, we should define the duration of the hypothesis being tested. This protects the product team from losing track of the data or identifying false positives where there are none. The hypothesis should have a finite timeline that lets the product team come back to the drawing board and compare ideas again.

Determine the Appropriate Sample Size

Sample size is another essential factor in hypothesis testing. A sample size that is too small can lead to inaccurate results, while a sample size that is too large can lead to a waste of resources. It is essential to determine the appropriate sample size when conducting hypothesis testing to ensure accurate results. A larger sample size increases the chances of obtaining accurate data and decreases the chances of making mistakes when analyzing the data.

Conduct Continuous Testing

Continuous testing is crucial in hypothesis testing. It enables product managers to keep testing their hypotheses throughout the product development process to ensure they are on the right track. Continuous testing helps product managers to identify and address any issues early before they become significant problems. It also enables product managers to adjust their strategies in response to changing circumstances.

Use the Right Statistical Tools

Product managers should use the right statistical tools when conducting hypothesis testing. Statistical tools enable product managers to analyze data and draw conclusions from it. The choice of statistical tools depends on the type of hypothesis being tested and the sample size. Product managers should seek the guidance of statistical experts when choosing the right tools.

Collaborate with Other Teams

Hypothesis testing is a collaborative process that involves different teams in an organization. Product managers should work closely with teams such as marketing, engineering, and design to conduct successful hypothesis testing. Collaboration helps to ensure that all teams are aligned in terms of goals, objectives, and timelines. It also helps to ensure that all teams have a stake in the product’s success.

Love the article? Read more about Product Discovery Basics For Building Better Products

Product-Led Growth: From Product to Profit

Related posts, focus group analysis: best ai analysis tools market researchers.

Best Practices for a Successful B2B Product Development Process with Chris Long

How To Prioritize Features In Product Research As A Product Manager

Product Mindset's Newsletter

Hypothesis Driven Product Management

"the true method of knowledge is an experiment.".

What is Lean Hypothesis Testing?

“The first principle is that you must not fool yourself and you are the easiest person to fool.” - Richard P. Feynman

In lean hypothesis testing, the MVP itself can be framed as a hypothesis. A well-designed hypothesis breaks down an issue into a problem, solution, and result.

How do you get started?

Two core practices underlie lean:

Use of the scientific method and

Use of small batches. Science has brought us many wonderful things.

The use of small batches is critical. It gives you more shots at a successful outcome, particularly valuable when you’re in a high risk, high uncertainty environment.

So, how do you do it? In 6 simple (in principle) steps : Start with a strong idea , one where you’ve gone out a done customer strong discovery which is packaged into testable personas and problem scenarios. If you’re familiar with design thinking, it’s very much about doing good work in this area. Structure your idea(s) in a testable format (as hypotheses). Figure out how you’ll prove or disprove these hypotheses with a minimum of time and effort. Get focused on testing your hypotheses and collecting whatever metrics you’ll use to make a conclusion. Conclude and decide ; did you prove out this idea and is it time to throw more resources at it? Or do you need to reformulate and re-test? Pivot or persevere ; If you’re pivoting and revising, the key is to make sure you have a strong foundation in customer discovery so you can pivot in a smart way based on your understanding of the customer/user.

By using a hypothesis-driven development process you:

Articulate your thinking

Provide others with an understanding of your thinking

Create a framework to test your designs against

Develop a standard way of documenting your work

Make better stuff

Free Template: Lean Hypothesis template

Eric Ries: Test & experiment, turn your feeling into a hypothesis

5 case studies on experimentation :.

Adobe takes a customer-centric to innovating Photoshop

Test Paper prototypes to save time and money: the Mozilla case study

Walmart.ca increases on-site conversions by 13%

Icons8 web app. Redesign based on usability testing.

Experiments at Airbnb

Enjoyed this issue? Please forward it to your friends.

Share ThinkProduct's Newsletter

Feel free to share on Instagram , LinkedIn and Twitter too. Your support is greatly appreciated!

Thanks for reading! Have a great week and go make something amazing.

Ready for more?

we know the way - show the way - go the way

Feb 28, 2023

Russia: Magnit launches its second discount concept B1

Updated: May 5, 2023

Russian largest retail chain Magnit (owned by Marathon Group and list on MCX : MGNT, LSE : MGNT) launches its second discount format B1. The first three pilot stores were opened in Stupino, Elektrostal and Tuchkovo in Moscow region.

For this new discount project, Magnit appointed Anzhela Ryabova who has a wide spread retail experience where off discount retail at the Siberian-based store chain Baton (part of Krasny Yar).

The new format B1 will strengthen the presence of Magnit in the segment of discounters, where Magnit is represented from 2020 by the soft-discount proximity small size concept "My Price", and will allow to cover even more consumer missions. B1 are stores with a focus on a limited quality assortment at attractive prices, covering the basic needs of the consumer.

The new discount format is close to the classic discount concept and is aimed at consumers who do not accept compromises in quality, but strive for rational purchases. The range of B1 with 1,000 SKU is narrower than the soft discounters "My Price" (average of 2,500 SKUs), and own brands (private labels) in the future will occupy about 50% of the B1`s product offer.

B1 will adhere the EDLP (every day low prices) pricing strategy. But the basis of the assortment of the discount format will be products of the middle price segment with a small share of goods of the "first price".

Due to a deeper optimization of operational processes in stores and logistics, prices in B1 will be lower than in "My Price" and in "Magnit" stores. In the B1 assortment emphasis is placed on offering high-quality fresh categories, as well as fruits and vegetables. Another key category of the new format is fresh pastries: in B1 stores there will be bakeries for the production of fresh bread, buns and pies.

Another distinctive feature of B1 will be a wide offer of weight goods such as cereals, confectionery, frozen foods, seafood, etc. At the same time, a significant proportion of goods in B1 will be presented in large packages, which will also help customers optimize their spending.

The store area of B1 stores will be 330 - 400 sqm. Only pallet and boxed goods are presented, which requires additional merchandising space. In addition, the B1 layout provides wide aisles for the convenience of buyers, fast shopping and store productivity.

Another element of B1 will be the use of modern technologies in stores, such as self-service cash registers. It will test the so-called "cold rooms", separate rooms with a special temperature regime for chilled products (meat, fish, salads, dairy products). Cold rooms will allow B1 to reduce operating costs, as well as the time of personnel for laying out goods.

B1 is managed by Angela Ryabova, who joined Magnit in November 2022. Before that Ryabova headed the Krasny Yar group of companies, developing the Baton discounter successfully.

In the future, B1 will set-up a separate logistics, maximally adapted to the needs of the format in terms of speed and cost of operations. It is assumed that the first own B1 distribution center will be opened in the fourth quarter of 2023. In the near future, Magnit will open up to 15 pilot B1 stores in different configurations and types of locations in different cities of the Moscow region to test several hypotheses, operating models and business processes. After that, the company will form the optimal model of the hard discounter and will decide on scaling the format, supported by DRC .

"Discounters remain one of the fastest growing segments of the market and retain high potential for further development. We want to create a standard of "high-quality" hard discounters on the Russian market, a format that is very poorly represented in our country. Our hard discounter is a store for modern consumers who rationally approach their budget and time, but are not ready to sacrifice the quality of life," says Angela Ryabova, Director of B1.

Magnit is presented in all grocery retail formats covering basic consumer missions. The company develops a network of convenience stores for frequent purchases of basic basket goods, large-format stores (supermarkets and superstores) with a wider range for future purchases, drug stores for purchases of non-food products. The new B1 format is focused on purchasing a basic food basket for a few days and daily purchases of fresh goods, while "My Price" covers the wider needs for daily purchases of basic goods on a smaller space.

In 2022, Magnit expanded its network of soft discounters to 700 stores and in 2023 it will continue to expand this format.

Source: retail.ru

#b1 #myprice #magnit #newdiscountformat # smartdiscount #discountfoodretail #discounter #discount #foodretail #retail #drc #discountretailconsulting #retailconsulting #consulting #consultancy

Russia establishes special site to fabricate fuel for China’s CFR-600

!{Model.Description}

A special production site to fabricate fuel for China’s CFR-600 fast reactor under construction has been established at Russia’s Mashinostroitelny Zavod (MSZ - Machine-Building Plant) in Elektrostal (Moscow region), part of Rosatom’s TVEL Fuel Company.

As part of the project, MSZ had upgraded existing facilities fo the production of fuel for fast reactors, TVEL said on 3 March. Unique equipment has been created and installed, and dummy CFR-600 fuel assemblies have already been manufactured for testing.

The new production site was set up to service an export contract between TVEL and the Chinese company CNLY (part of China National Nuclear Corporation - CNNC) for the supply of uranium fuel for CFR-600 reactors. Construction of the first CFR-600 unit started in Xiapu County, in China's Fujian province in late 2017 followed by the second unit in December 2020. The contract is for the start-up fuel load, as well as refuelling for the first seven years. The start of deliveries is scheduled for 2023.

“The Russian nuclear industry has a unique 40 years of experience in operating fast reactors, as well as in the production of fuel for such facilities,” said TVEL President Natalya Nikipelova. “The Fuel Division of Rosatom is fulfilling its obligations within the framework of Russian-Chinese cooperation in the development of fast reactor technologies. These are unique projects when foreign design fuel is produced in Russia. Since 2010, the first Chinese fast neutron reactor CEFR has been operating on fuel manufactured at the Machine-Building Plant, and for the supply of CFR-600 fuel, a team of specialists from MSZ and TVEL has successfully completed a complex high-tech project to modernise production,” she explained.

A special feature of the new section is its versatility: this equipment will be used to produce fuel intended for both the Chinese CFR-600 and CEFR reactors and the Russian BN-600 reactor of the Beloyarsk NPP. In the near future, the production of standard products for the BN-600 will begin.

The contract for the supply of fuel for the CFR-600 was signed in December 2018 as part of a governmental agreement between Russia and China on cooperation in the construction and operation of a demonstration fast neutron reactor in China. This is part of a wider comprehensive programme of cooperation in the nuclear energy sector over the coming decades. This includes serial construction of the latest Russian NPP power units with generation 3+ VVER-1200 reactors at two sites in China (Tianwan and Xudabao NPPs). A package of intergovernmental documents and framework contracts for these projects was signed in 2018 during a meeting between Russian President Vladimir Putin and Chinese President Xi Jinping.

Terms and conditions
Privacy Policy
Newsletter sign up
Digital Edition
Editorial Standards

History of cooperation
Areas of cooperation
Procurement policy
Useful links
Becoming a supplier
Procurement
Rosatom newsletter

Rosatom Global presence
Rosatom in region
For suppliers
Preventing corruption
Press centre

Rosatom Starts Life Tests of Third-Generation VVER-440 Nuclear Fuel

16 June, 2020 / 13:00

This site uses cookies. By continuing your navigation, you accept the use of cookies. For more information, or to manage or to change the cookies parameters on your computer, read our Cookies Policy. Learn more

IMAGES

Hypothesis Testing Definition
Hypothesis Testing- Meaning, Types & Steps
Hypothesis Testing: Levels of Product Analysis
Forming Experimental Product Hypotheses
Data-driven Product Management in 2023
What is Hypothesis Testing?

VIDEO

Hypothesis Testing
8.1: Basics of Hypothesis Testing
8 1 Basics of Hypothesis Testing 102623
Product Hypothesis Testing P2: Easy to Get Started
Statistics
Hypothesis testing #study bs 7 semester statics

COMMENTS

Hypothesis-driven product management
Yes, hypothesis-driven practices are ideal for building new features. Since the goal is to test the validity of each hypothesis, the uncertainty around the product development process is significantly reduced. In a way, hypothesis testing helps you make better decisions about your product lifecycle management.
A Guide to Product Hypothesis Testing
A/B Testing. One of the most common use cases to achieve hypothesis validation is randomized A/B testing, in which a change or feature is released at random to one-half of users (A) and withheld from the other half (B). Returning to the hypothesis of bigger product images improving conversion on Amazon, one-half of users will be shown the ...
How to test hypotheses as a product manager
Simple product development hypothesis testing using a Z-test. There are a few statistical hypothesis tests we could implement. A common one is a Z-Test. It allows us to take and test data samples and check if the observed differences deviate from what we would expect given the hypothesis. Let's look at an example:
Good Product Hypotheses: How to Write and Test
3. Set validation criteria. First, build some confirmation criteria into your statement. Think in terms of percentages (e.g. increase/decrease by 5%) and choose a relevant product metric to track e.g. activation rate if your hypothesis relates to onboarding.
Product Hypothesis Testing: Generating The Hypothesis
The key to do this is by product hypothesis testing, which is actually a two part process: Part 1: Product Hypothesis Generation - Figuring out what we should be testing for. Part 2: Hypothesis validation - How Do Product Managers Validate A Product Hypothesis. So let's dive in a bit and learn what exactly a hypothesis is and how it ...
How Do Product Managers Validate A Product Hypothesis?
As a product manager, you need to get comfortable saying "NO"! As previously described in Part 1 of this series, Product Hypothesis Testing: Generating The Hypothesis, t he first step in hypothesis testing involves setting up two competing hypotheses, the null hypothesis and the alternative hypothesis. Null hypothesis: states the "status quo".
Hypothesis Driven Product Management
Lean hypothesis testing is an approach to agile product development that's designed to minimize risk, increase the speed of development, and hone business outcomes by building and iterating on a minimum viable product (MVP). The minimum viable product is a concept famously championed by Eric Ries as part of the lean startup methodology.
Value Hypothesis 101: A Product Manager's Guide
Validating your value significantly reduces risk. You can prevent wasting money, time, and resources by verifying your hypothesis in early-stage development. A good value hypothesis utilizes a framework (like the template below), data, and checks/balances to avoid bias. 1. Use a Template to Structure Your Value Hypothesis.
What Is Product Management Hypothesis?
Product management hypothesis is a scientific process that guides teams to test different product ideas and evaluate their merit. It helps them prioritize their finite energy, time, development resources, and budget. To create hypotheses, product teams can be inspired by multiple sources, including: Observations and events happening around them.
A/B Testing: Hypothesis Testing for Product Management (Part I)
Hypothesis: Framing the right hypothesis is thus the core of A/B testing. Let us go back to stats 101 where we learnt hypothesis testing consisting of Null and Alternate hypothesis. H0 (The null hypothesis): μ − μ0 = 0. H1 (The alternate hypothesis): μ − μ0 ≠ 0. However, in product management, we use a slightly different version.
The Product Management Dictionary: hypothesis testing
Hypothesis testing is a critical part of product management, allowing teams to validate assumptions and make data-driven decisions. By understanding the process of hypothesis testing, as well as common metrics and KPIs, you can make more informed decisions and drive the success of your product. Learn about hypothesis testing in product ...
Hypothesis Testing
Putting the 4 Levels of Product Analysis Into Practice: A Halloween-Themed Example. What to Do When You Don't Have Enough Traffic to A/B Test. More than just knowing the mechanics of how to run a good experiment, you also need to know what to test and when. Don't Rely on Confidence Alone. Run Experiments Before You Write Code.
How to write an effective hypothesis
Hypothesis validation is the bread and butter of product discovery. Understanding what should be prioritized and why is the most important task of a product manager. It doesn't matter how well you validate your findings if you're trying to answer the wrong question. A question is as good as the answer it can provide.
Hypothesis Testing: How to do it the right way
Hypothesis testing is a collaborative process that involves different teams in an organization. Product managers should work closely with teams such as marketing, engineering, and design to conduct successful hypothesis testing. Collaboration helps to ensure that all teams are aligned in terms of goals, objectives, and timelines.
Hypothesis Driven Product Management
Lean hypothesis testing is an approach to agile product development that's designed to minimize risk, increase the speed of development, and hone business outcomes by building and iterating on a minimum viable product (MVP). The minimum viable product is a concept famously championed by Eric Ries as part of the lean startup methodology.
Forming Experimental Product Hypotheses
A hypothesis is a statement made with limited knowledge about a given situation that requires validation to be confirmed as true or false to such a degree where the team can continue their ...
How to create product design hypotheses: a step-by-step guide
Which brings us to the next step, writing hypotheses. Take all your ideas and turn them into testable hypotheses. Do this by rewriting each idea as a prediction that claims the causes proposed in Step 2 will be overcome, and furthermore that a change will occur to the metrics you outlined in Step 1 (your outcome).
Hypothesis testing methods. Product management.
Testing product hypotheses using HADI. HADI is a cycle of four blocks: Hypothesis — the formation of a hypothesis according to the principle "if, then …". Action — testing the product hypothesis, that is, we are working to launch the experiment. Data — we collect data and get a measurable result. Insights — we draw conclusions ...
How to Pick a Product Hypothesis
Key Takeaways: You need a hypothesis because it clearly defines a change you want to make and the impact you expect to have on your product. A good hypothesis can be proven false, validated with ...
Russia: Magnit launches its second discount concept B1
Russian largest retail chain Magnit (owned by Marathon Group and list on MCX: MGNT, LSE: MGNT) launches its second discount format B1. The first three pilot stores were opened in Stupino, Elektrostal and Tuchkovo in Moscow region.For this new discount project, Magnit appointed Anzhela Ryabova who has a wide spread retail experience where off discount retail at the Siberian-based store chain ...
BETA GIDA, OOO Company Profile
See other industries within the Manufacturing sector: Aerospace Product and Parts Manufacturing , Agriculture, Construction, and Mining Machinery Manufacturing , Alumina and Aluminum Production and Processing , Animal Food Manufacturing , Animal Slaughtering and Processing , Apparel Accessories and Other Apparel Manufacturing , Apparel Knitting Mills , Architectural and Structural Metals ...
Russia establishes special site to fabricate fuel for China's CFR-600
Unique equipment has been created and installed, and dummy CFR-600 fuel assemblies have already been manufactured for testing. The new production site was set up to service an export contract between TVEL and the Chinese company CNLY (part of China National Nuclear Corporation - CNNC) for the supply of uranium fuel for CFR-600 reactors.
Rosatom Starts Life Tests of Third-Generation VVER-440 Nuclear Fuel
The life tests started after successful completion of hydraulic tests (hydraulic filling) of the mock-up with the aim to determine RK3+ hydraulic resistance. Life tests are carried out on a full-scale research hot run-in test bench V-440 and will last for full 1500 hours. The aim of tests is to study mechanical stability of RK3+ components ...

Shipping Your Product in Iterations: A Guide to Hypothesis Testing

By Kumara Raghavendra

What Is Hypothesis Testing in Product Management?

Different Types of Hypothesis Testing

Multivariate Testing

Before/After Testing

Time-based On/Off Testing

Test Design

Significance and Power

Using Hypothesis Testing for Successful Product Management

Further Reading on the Toptal Blog:

Understanding the basics

What are the three required parts of a hypothesis?

Why do we do A/B testing?

What is A/B testing used for?

What is A/B testing and multivariate testing?

Kumara Raghavendra

About the author

Toptal Product Managers

How to Generate and Validate Product Hypotheses

Idea vs hypothesis

Start with a problem statement

How to generate product hypotheses

BRIDGES framework for ideation

How to write a product hypothesis statement

1. Identify variables

2. Make the connection

3. Set validation criteria

Null hypothesis and alternative hypothesis

Prioritizing hypotheses for testing

How to test product hypotheses

A/B testing

Prototyping

Deliveroo feature prototype example

Asking your users

Wizard of Oz

User-wide updates

Feature flags

Recording hypotheses statements and test results

Understanding test results and taking action

Wrapping Up

Hypothesis Driven Product Management

What is Lean Hypothesis Testing?

How do you get started?

So, how do you do it? In 6 simple (in principle) steps :

By using a hypothesis-driven development process you:

Free Template: Lean Hypothesis template

Eric Ries: Test & experiment, turn your feeling into a hypothesis

Leave a Reply Cancel reply

Value Hypothesis 101: A Product Manager's Guide

Talk to Sales

What Is a Verifiable Value Hypothesis?

A Verifiable Value Hypothesis Minimizes Risk and Saves Money

Value Hypotheses vs. Growth Hypotheses

4 Tips to Create and Test a Verifiable Value Hypothesis

1. Use a Template to Structure Your Value Hypothesis

2. Turn Customer Feedback into Data to Support Your Hypothesis

3. Validate Your Product

4. Use Safeguards to Remain Objective

Validation Is the Key to Product Success

Heather Tipton

The Product Management Dictionary: hypothesis testing

Understanding Hypothesis Testing in Product Management

The Importance of Hypothesis Testing

Key Terminology and Concepts

The Hypothesis Testing Process

Formulating a Hypothesis

Designing an Experiment

Collecting and Analyzing Data

Drawing Conclusions and Making Decisions

Types of Hypothesis Tests

A/B Testing

Multivariate Testing

Bayesian Hypothesis Testing

Common Metrics and KPIs in Hypothesis Testing

Conversion Rates

User Engagement Metrics

Retention and Churn Rates

Join hundreds of teams launching with Ignition

Hypothesis Testing