How Much Data Do You Really Need for a Successful UX Design Project?

In a perfect world, you would first conduct extensive user research, systematically derive requirements, present initial ideas and designs, and test them until the results say, “Now it’s perfect!”

Unfortunately, we experience this perfect case relatively rarely in the real world.

In reality, we have deadlines and limited resources, which create the need to keep the design process as large as necessary but as small as possible. If you’re like many of us, you probably have difficulty deciding how much is “enough.” To help you with this question, we have developed the flexibility and risk matrix shown below.

The idea behind this matrix is to evaluate your system, product, or service concerning the two dimensions of flexibility and risk and then draw conclusions about the effort you need to invest in data-driven UX design. This tool allows you to estimate the data required for the rest of the process. First, let’s introduce you to the flexibility dimension.

The Flexibility of Your Solution

By flexibility, we don’t mean how far you can stretch or bend your product but how adaptable it is after release.

If you manufacture a washing machine, for example, large parts of the product are fixed after development and delivery: The dimensions are defined, the control knobs are fixed, and even the electronics of your device can no longer be changed without great effort. Of course, you could send out a service technician and have all washing machines modified by hand—but the effort involved would most likely be disproportionate to the benefit.

This product therefore has very little flexibility in our perception. In addition, the lifecycle of a washing machine is usually relatively long—about 10 years—which ensures that customers who have just bought a new one will not buy another one for a while. So again, very little flexibility exists in this context. Your washing machine can only be less flexible if you also use the washing machine in a regulated market (let’s say, for example, the medical market). Then, in addition to the technical and organizational hurdles to change, regulatory hurdles may even prohibit you from changing the washing machine—at least, without retesting the new design with regulatory agencies. As a result, your product would not be flexible enough to be changed without incurring significant costs and effort for quite some time.

But if your washing machines were “smart,” for example, and you, as the manufacturer, could remotely change and adjust various cleaning programs via software updates, your device would gain significantly in flexibility. You still could not change the hardware

but you could change its functionality, at least within certain limits.

Hardware flexibility is also possible in principle. For example, a manufacturer of hot tubs offers the option of exchanging the backrests of the individual seats with a click system. The new backrest creates a unique experience with its different massage jets, which the customer can thus flexibly control himself.

Some complete systems are based entirely on hardware flexibility and have standardized interfaces. Examples include system cameras with exchangeable lenses or computers with exchangeable RAM and slots for add-on cards.

On the other side of the flexibility scale are products that can be easily changed even after market launch. For example, unregulated software products can usually be adapted quickly and flexibly. Manufacturers can make small or far-reaching changes via patches or updates, integrate or remove functions, and even completely replace the UI.

Therefore, the higher the flexibility, the more uncertainty you can accept in development. If it’s easy to fix bugs with one click and add features even though the product is already in the field and used, securing the design with data is less relevant. Unless, of course, you’re operating in a risky environment.

To determine whether flexibility is more likely to be high or low in your case, ask yourself the following questions:

• How long do your customers and users expect to work with an unchanged version of your solution?
• Can and may your solution be used without prior training of the users?
• Is it possible to update your solution without regulatory or organizational constraints that require (re)testing?
• Does your solution also have internet access during use, which you, as the manufacturer, can access from the outside?
• If no internet access is available, does your product have the ability to have new product versions installed on-site by users?

The more often you answer “yes,” the shorter the period customers and users expect to work with an unchanged product version, and the more flexible your product is. However, if you said “no” frequently, you should spend more time defining user requirements early on and testing them, as your solution will need to be classified as less flexible.

The Risk of Your Solution

The risk dimension refers to how dangerous it is if errors occur while using your product, system, or service. Risk in this context is defined according to ISO 14971 in the following formula:

Risk = Probability of occurrence × Severity

Therefore, the question is how often this error is likely to occur and how bad its possible consequences result. The risk can relate to different areas:

• The health or life of persons is endangered by operating errors caused by poor design.
• The economic basis of a company is jeopardized by poor design. This could happen, for example, if production losses occur due to faulty operation or if a lawsuit is sought because of these production losses.
• The reputation of a company is jeopardized by poor design. How bad the loss of reputation is for the company must be examined individually based on the brand value and promise.

The easiest way to understand the risk is probably if you’re developing a system, product, or solution whose use is directly related to human health or life.

Perhaps you now have the feeling that this can’t happen to you. However, as UXers, we have worked directly on such projects in many places! Especially in the medical sector, high-risk products are commonplace: developing a heart-lung machine, a defibrillator, or a patient monitor. But even outside the medical industry, we frequently encounter high-risk projects, such as an aircraft emergency landing system or an airport baggage scanner. In all these cases, incorrect operation and misjudgment can result in glaring damage. Therefore, solutions that endanger the health or human life are automatically (high) risk for us.

Also risky, but of course in a different way, are applications that, if misused and misinterpreted by users, cause damage to a company’s economic foundation. This can be your own, but more importantly, it can be your customers’ businesses. This category of products includes, for example, process controls in production or software systems that map a company’s primary work processes in the productive area. For manufacturing companies, this could be production planning or machine control; for service companies, it could be the project management tool or central data storage.

So, whenever the purpose of the business cannot be fulfilled or can only be partially fulfilled if your system, product, or service is operated incorrectly, we would also rate the application as risky. Compared to the risk of direct harm to people, however, it is more of a medium risk.

The company that runs into economic difficulties due to a poorly designed system, product, or service can also be your own. For example, if you only have the capital for one product development, this last throw should fit and not flop. You could not “afford” a poor design in this case.

The last point—admittedly a relatively soft one, but no less important—is when poor design threatens your company’s reputation to such an extent that short-term or lasting damage is to be expected, affecting the perception of the brand and its quality. How severe the damage is, of course, depends heavily on several factors. In our view, these include:

• How established is the brand already?
• What does the brand stand for?
• How closely is the brand linked to the product?

In our view, the potential damage from a poorly designed solution is significantly higher for established brands with outstanding quality than for unknown players.

For example, buying a premium sedan and then struggling with the super-expensive but stubborn-to-operate UI of the infotainment system hurts the brand. It’s not uncommon in this case to tell friends or acquaintances that the vehicle itself is excellent, but the operation is horrible. Suddenly, it’s not so “premium” anymore.

If you don’t have a well-known brand yet, you can’t lose brand equity through poor design. But of course, you won’t build one either. However, if you work for Apple, for example, and thus even stand for usability, UX, and innovation, the whole thing looks different: The damage caused by a problematic product would be immense.

The same applies if the poorly designed product is closely linked to the customer’s brand. Then bad design radiates strongly and clearly onto the core brand—just like good design. An example of products closely related to the core brand are Apple products, such as the Apple Watch. Here, it is clear from the name alone that there is a close connection between the product and the core brand. So, if the Apple Watch is poorly designed, it reflects directly on Apple.

However, the potential damage is lower if your company operates more in the background and presents many different brands to customers. An example of this could be Proctor & Gamble, whose in-house brands include the diaper manufacturer Pampers and the cleaning product Swiffer or the chest rubs from Vicks. If a batch of Pampers diapers does not work as intended, it is unlikely that customers will stop buying and using Swiffer.

Of all the factors mentioned, damage to the brand is the hardest to measure. Nevertheless, we would also rate an application as risky here if it is published by a strong brand closely linked to the product and the brand itself stands for high product quality in the premium price segment.

If you want to assess how risky your product, system, or service is, you can use the following questions to make an assessment:

• Can people’s lives be endangered by operating your product incorrectly?
• How many lives will be put at risk if there is a mishandling?
• Can incorrect operation of your product threaten the health of people?
• How many people’s health is at risk if something goes wrong?
• Can incorrect operation endanger the economic basis of another company or your company?
• Can poor design (not only in terms of usability but also UX) cause lasting damage to the value of your brand?

The more often your answer to these questions is “yes,” and the higher the number of people potentially affected, the higher the risk associated with your product, system, or solution. If you could answer “no” to all questions, it is probably a solution with relatively low risk.

Working with the Flexibility and Risk Matrix

With the definition of flexibility and risk from the previous sections, working with the flexibility and risk matrix is now relatively easy to explain. To work with the matrix, you should ask yourself the following two questions:

1. How flexible is my solution once I have finished developing it and brought it to market?
2. What would be the risk if my solution leads to mistaken operation and misinterpretation?

The point of using the matrix is not to justify your position in the matrix down to a tenth of an inch. Don’t fret about whether your solution scores a 58 or a 59 on the flexibility scale. We’re just trying to get a sense of the flexibility and risk of your solution.

In general, the further you move your solution to the top right of the matrix, the more data you should obtain during the development of your solution to ensure the quality of your designs, as shown in the figure below. The further you’re at the zero point of the matrix at the bottom left, the less effort you need to invest in testing your solution in advance.

The position of your solution on the matrix has several consequences:

• If you have a low-flex but high-risk application (top right), you must ensure everything works in advance. In this area, you must invest the most effort, and in the best case, you even perform a final test before release (of course, for a medical device, you must do this anyway, regardless of the matrix). This approach gives you multiple safeguards and also allows you to show that you have invested time and effort to ensure that your product not only looks good in the end but also works safely. An example of a product in this category would be a heart-lung machine: a regulated, approved hardware product that can’t be changed once shipped. What’s more, people die if this machine is operated incorrectly.
• If you have a very flexible but potentially risky application (bottom right), you should also ensure that everything works so that no unnecessary damage occurs. In this area, liability issues and due diligence can suddenly come into play. Again, a final test before release is a good idea. An example product in this area would be Tesla, whose hardware is fixed but can be widely updated via “over the air” software updates. Nevertheless, people can still be harmed in the event of operating errors. low risk high risk flexible not flexible
• If you have an inflexible but low-risk solution (top left), the worst that happens to you is that you must live permanently with a bad solution. Ultimately, only you (and your customers) can decide whether you want and accept that. In most cases, a final test is unnecessary, and the number of iterations can be reduced compared to risky applications. An example product in this category would be the washing machine from a non-premium manufacturer mentioned earlier. This machine is not flexible due to fixed hardware and software. However, no great (image) damage can be expected as long as the product works.
• If you have a flexible and low-risk application (bottom left), you’re very comfortable. You can get to market quickly and then address potential issues downstream. From our perspective, you don’t need a final test, can test less, or maybe not test at all, and in the best case, you can continue to work with usage data from live operations. An example product would be a to-do app without a well-known brand. The app can be changed flexibly anytime and does no critical damage if errors occur.

Editor’s note: This post has been adapted from a section of the book Usability and User Experience Design: The Comprehensive Guide to Data-Driven UX Design by Benjamin Franz and Michaela Kauer-Franz