There are several qualitative methods for understanding user behaviour, for assessing and researching user needs and unspoken needs. The appropriate choice between methods (methodology, level of detail, depth) is a matter of available financial resources and also depends on professional needs and standards. There are many ways of doing research, but the important thing is to do something, as even the most basic, small-scale research or survey can yield useful results.

But before we get to the individual methods, let us have a glance at a brief overview of the basic methods with the help of some summary diagrams.

Research methods in several dimensions

Perhaps the best known, understandable and concise way of presenting the available UX research methods is Christian Rohrer’s 2014 work, When to Use Which User-Experience Research Methods

In his work, he divides the methods into two parts. In one dimension, he depicts whether a given method investigates the user’s behaviour (behavioural) or attitude (attitudinal) towards the interface under investigation, prior expectations, conscious and unconscious needs (e.g. mental model research, user needs).

In the other dimension, he indicates whether the result of the method is qualitative (using quality data) or quantitative (using numerical data) data.

The third determining factor in the classification of methods is the context in which the data are collected. It is not the same whether the use of a product is studied in its natural context (natural use of product, natural context) or in a lab-based context. The aim is always to collect data in the most natural context possible, as close to the user as possible. Rohrer suggests that the method best suited to the research objectives should always be chosen.

In UX research, we usually start with a ‘user needs analysis’ and later, when there is something to look at, we focus on usage. Then we prefer to use behavioural and context-based methods, getting as close as possible to the real context.

These methods can also be mixed, for example by using the triangulation analysis methodology to draw conclusions. Since triangulation has been mentioned, we will briefly describe it after the methods.

Research from the past century in UX research today

In a particularly striking illustration, McCrindle Research presents research methods that can be applied in the UX field, looking back over almost half a century. It is worth bearing in mind that some of the methods, while nearly seventy years old, are still highly applicable in UX research today. The message is that, while the topics and interfaces under study change, the underlying motivations and behaviour of users tend to change less often.

The possible methods that can be used in development are also illustrated in the following graphic (see Figure 3), which shows how research can and should be conducted at each stage from the very beginning to the very end of development, and even on the existing interface. It is not necessary and not required to use all the methods (at the same time or one after the other), it is enough to pick and choose according to the objectives and the resources available.

Research journey – research methods in the development process

It is worth observing in the diagram how many different research methods can be applied from the very beginning of the development process to the finished interface. We need to use different methods when we are researching the problem itself, in response to which the interface is being created, and different methods when there is something to look at, however elaborate or undeveloped it may be. So the research repertoire is very rich: the research methods are varied and so are the results, from quantitative data to heat maps and detailed, qualitative research results.

It can also be seen that all research has its place in the development process: at the beginning of the design-development phase, we look at the problem (what needs to be solved?),  and during the design phase, we look at the quality of the solution (was it solved well?).

What is less often discussed: triangulation

It is a fantastic meta-solution (meta-solution or meta-method? 🙂 ). It means that if you use several data collection and interpretation methods at the same time, your research will be more accurate, reliable, valid and your results more thorough. This is also, of course, a social science research design and analysis solution (not research!) that the UX research field has discovered for itself.

What is it for?

Often, in UX research, the researcher runs into obstacles, e.g. being asked whether the results of qualitative research on a small sample of people in the research being conducted are representative or not (note: these qualitative results from very small samples, even pilot studies, are not usually representative, even in UX research, but they do not have to be!)

What can be done? The best approach is to use multiple research methods at the same time, so that the limitations of one method can be mitigated by data from another source. This approach, using multiple research methods, is called triangulation.

As defined in the Encyclopedia of Research Design (https://www.nngroup.com/articles/triangulation-better-research-results-using-multiple-ux-methods/): triangulation is the practice of using multiple sources of data or multiple approaches to analysing data, to enhance the credibility of a research study.

For example, it is possible to use multiple methods to investigate the same activity, such as quantitative and qualitative research, plus some form of peer review (e.g. heuristic analysis).

We draw conclusions by comparing the results of these. It is also a solution to compare and analyse several different metrics related to the same activity, such as satisfaction ratings, time spent and revenue data.

Solving simple questions does not require such complex analysis, but examining complex decisions or decisions with major consequences and drawing conclusions does. Experienced teams are aware that it is worth a few hours to see if the additional data available supports the chosen solution.

The most commonly used qualitative methods

In UX research, as mentioned above, we can work with both quantitative and qualitative data. Before and at the beginning of development, it is advisable to perform tests that help to shape the surface to be developed, typically qualitative tests. If you want to know the results of a large sample, the quickest way to work with quantitative data is to ask questions or study analytical data. However, when researching a small sample size, qualitative data is both more tangible and more valid. If you want qualitative information, you should also choose one of the qualitative research methods. You will not have difficulty: the vast majority of UX research methods are like this.

So then let us see which are the most popular and most commonly used qualitative research methods in the field of UX!

The best known method: usability research

One of the most popular testing methods is usability testing, in which tasks are given to users who try to perform them while sitting in front of a screen. For example, they may be asked to search our website for specific information, data or documents. During the execution of the task, we can observe them, for example, when, where and how they navigate the site, what and where they search, what feedback they give about the navigation and the content of the site. This method is so well known that for many, it represents the entire palette of UX research.

At which stage of the design and development process do we apply it?

Usability testing can be used to test designs at any stage of development: from less or better developed prototypes (hi/low fidelity) to live systems. Moreover, it is possible to test the system that does not yet exist, using the Wizard of Oz testing method, but more on that later.

Types: formative and summative testing

Usability tests are usually grouped in several ways, depending on:

• at which stage of development they can be used,
• whether or not they are tool-intensive, and
• their other characteristics, such as the role they play in the design process.

The latter can be referred to as formative or summative, where the focus is on how the data generated will be used in the development of the product, be it a website, a tool, etc.

Formative studies are an integral part of the development process. Their purpose is to establish the current state, provide feedback and identify defects for the next design phase. The goal of testing is to improve the design, and this is achieved through iterative retesting, including evaluating changes and making new recommendations. It is therefore better to start formative testing at an earlier stage of the design and complete it when the final design is completed.

By performing formative tests, the following questions can be answered:

• What are the main usability flaws in the design?
• What can users use well and not so well?
• Have improvements been made compared to the previous design version?

In contrast, the purpose of summative tests is to evaluate the finished design. Summative testing can also be used when there are several solutions to choose from and existing designs need to be compared. While formative testing is more about improving the design, in the case of summative studies, the question is whether the completed design meets certain criteria.

By conducting summative tests, the following questions are answered:

–  Have the usability objectives of the development been met?

–  How does our product perform compared to the competition?

–  Have there been any improvements compared to the previous version of the product?  (Péter Polgár: Formative and summative tests)

Data collection methods

There are several methods that can be used when conducting testing.

We can ask the test subjects to think aloud, and while they are doing so, we record them on audio or video. This is the Think Aloud protocol.

We can also play back the recording of the test and ask the interviewee to tell us what happened when, what they were thinking about when they did this or that. This is the Retrospective Think Aloud protocol.

We can interview them so they can share their experiences and opinions about the website in question.

In a focus group discussion, we can get the opinions and attitudes of several users representing the target group at the same time. The focus group testing can be performed in a traditional, i.e. face-to-face, or online format. It is important to note that the latter are not usability tests, but a way of collecting qualitative feedback on the interface and exploring related areas and themes. In the case of research on websites, it seems more appropriate to use the online version, as this allows the discussion and the viewing and use of the sites to take place in parallel.

Documenting what you see and hear

What users say can be recorded in various ways (audio, images, text) and analysed later. We can also record the navigation process on the website using various computer programs, including mouse movements and clicks, and eye-tracking can be used to understand how users view the interface of our website. In this case, we learn not what visitors think, but exactly what they did on the site and in what order. This data can be the basis for a future interview.

Data analysis

Depending on the method used, the data set containing the findings of the research and the analysis that is produced from it can be of different types. They may include the time spent on the tasks, the number of pages visited and/or the number of incorrect searches, satisfaction rates, the problems encountered by the participants in solving the tasks, and their opinions and ratings. Taking into account the results and lessons learnt from the testing, it is worthwhile to adapt, improve and finalise the structure and visual appearance of the site.

Other, more frequently used methods

Card sorting

One of the primary considerations in website development is that the information on the finished interface should be clearly and sensibly grouped for both the interface owner and the site users. Sometimes the structure of a website reflects the thinking and ideas of the owner and/or the graphic designer or the programmer, the internal organisational hierarchy of the company or firm that owns it, and thus does not meet the expectations of the users, much less their thinking. The task is simple: organise the content of the interface into logical groups and, where appropriate, give it a name.

Individual or group?

In the case of individual grouping, the pages are grouped individually by the test participants.

In the case of group work, participants discuss, agree and do the same according to a common understanding. The advantage is that the participants agree among themselves, thus, in our experience, many more and more complex aspects can arise when performing the testing.

Open or closed grouping?

In the case of open grouping, users themselves group content, create and name the groups.

In closed grouping, group members organise content items (pages, documents) into predefined categories and headings.

In open grouping, users can start to identify the categories, and in closed grouping, we can find out how they think about the naming and content of each category.

The advantage of this method is that it helps you to design the structure of your website, to organise the content and to name the categories. Since the content is designed by the users themselves, and if the results of the test are applied in the future, it is expected that the layout of the future site will be logical and clear to them.

Ethnographic observation or ethnographic research

Participant observation is the only way to find out how the people being observed actually behave, rather than what they say about themselves and how they think they behave. In practice, field research provides a direct and complete observation of phenomena in their natural setting.

Web analytics

We can run several types of analytics on our existing or emerging platforms. Which one we choose depends on what we are interested in (Mouse operations? For that we have Hotjar. Full web analytics? The correct choice is Google Analytics! For attention to workflows and the interfaces? Also looking for a heat map? The solution is Mouseflow or Eye tracking!)

Note that the results will not be qualitative, but quantitative data, presented in a graphical, visual way where appropriate.

A less frequently used but very cool method: 5-second test

This method helps us to know and understand the users’ first impressions of the designs we are preparing.  By finding out what a person remembers about a design after viewing it for just 5 seconds. It can be used to check whether the main message has got through.

Wizard of Oz (Woz) [Wizard of Oz (Woz)]

This method is named after the protagonist of the fairy tale, Oz, who deludes himself into believing he is powerful by hiding behind a curtain. In this test, too, we use a prototype to fool the user into believing that he is dealing with a working system. In reality, however, the answers are not generated by a computer, but by a human being behind the interface (e.g. a coffee or cash machine), pulling levers and turning switches. So a researcher simulates the operation of the system. At the same time, the most important functions are revealed, and the potential for error can be detected. Read more about the method in this article.