Mobile App Usability Testing in China – Case Study With iOS App Bazaart

If you localize your mobile app for China, how can you know if Chinese users are really going to like it?

We helped test the iOS app Bazaart with Chinese users and wrote up this post about it.

This post will show you:

1. How to do a simple usability test with five Chinese users.
2. What the tests showed for Bazaart in-particular (some tips will be relevant to other readers too).

Bazaart’s Situation

Here’s how Gili Golander, co-founder of Bazaart, described the situation:

“What is Bazaart

Bazaart is an award-winning photo editing and design app. People from all walks of life use it: teachers, writers, musicians, small business owners, kids and seniors. They use Bazaart for personal stuff, like making family collages or hilarious memes as well as for business, like making their social media posts, and even book and music album covers. It is also used in art and science classes in schools around the world. It helps people discover their creativity, accomplish projects, and be their own designers.

Bazaart is quite different from the photo editing tools Chinese users are accustomed to which focus heavily on beautification. Its strength lies in the ability to remove a photo’s background, combine multiple photo layers, and use design templates to easily create beautiful photos.

Our objective

Bazaart is currently serving people worldwide, with the United States being its main market. We are interested in penetrating the Chinese market and have made several efforts in this direction such as creating a custom Chinese App Store presence, translating the app, hiring a Chinese speaking employee to help with content and support, and preparing content specifically for the Chinese user.

Bazaart recently went viral in China due to a video posted by one of our users on Douyin. The user made the last frame in his video with Bazaart and posted a video to show how he accomplished it. That led to a surge in downloads from China and showed us a use case we had not thought of before.

What we want is to understand how Chinese users perceive our App Store presence, how they interact with the app, does it serve their needs well enough, what is missing for them, and if there is any cultural gap we need to overcome. When we see recorded user tests alongside users’ answers to a few questions, we can get a sense of what is working and what isn’t and fix it.”

So, for this case we were interested in user feedback from real live human beings. Bazaart is already able to test for most technical bugs on various devices.

How To Do Mobile App Usability Testing With Chinese Users

We wanted to find the most important problems quickly and without much expense, so we used a simple test with just five users.

We followed this five-step process:

1. Planned who to test with and what they should test
2. Chose which tools to use
3. Recruited Chinese testers
4. Ran the tests while recording
5. Summarized feedback

PLANNING – USERS, TASKS, QUESTIONS

Remember that Gili identified these types of people: “teachers, writers, musicians, small business owners, kids and seniors.” That is potentially quite a few use cases though, so we narrowed it down to focus on women aged 18-34 with an iPhone on iOS 12.

In order to get useful feedback quickly, we had just five people test the app. That is enough, based on advice from usability expert Normal Nielsen.

The app seems a bit like Photoshop, but not as advanced and hopefully easier to use, so we wanted to get the opinion of people that have used Photoshop and some that haven’t.

Testers were given a short background on the app and asked to perform three tasks: create an image using a template, create a social media post and create a sticker.

As they went through the tasks, an app recorded their screen and voice.

They were also given four post-test questions:

1. What was the most frustrating thing about your experience?

2. Which creation method did you prefer (template or starting from scratch) and why?

3. What did you like about the app? What would you improve?

4. What would you use this app for in your daily life?

TOOL SELECTION

Bazaart used usertesting.com to help with testing in the USA and other markets. This has a few drawbacks, the main one being that they aren’t yet able to recruit users in China.

For our purposes, we needed tools to do these things:

1. Let the testers use the app.
2. Take video of their screen.
3. Get an audio recording of their voice so they can speak as they go.
4. Let them answer the post-test questions.

We used the app 录丸 (Screen Recorder +) to capture their screen recording and audio. For the survey questions, we simply chatted with them on WeChat.

RECRUITING

We didn’t need to use an external partner in this case, because the respondent requirements were fairly simple.

Instead, we mostly reached out via our network of friends-of-friends.

We found five testers. All of them have used other photo retouching software. Some have also used Photoshop and other related software.

1. Copywriter – hasn’t used Photoshop.
2. Illustrator – familiar with Photoshop and other illustration software.
3. Youth Hostel Owner – often uses other photo retouching software.
4. Photographer – familiar with Photoshop.
5. Designer, familiar with Photoshop and other design software.

Our Learnings

Summary: The App Isn’t Localized Well Enough Yet

To make it easier for readers to understand, we summarized the responses and our own feedback, rather than providing the full translated responses. We then kept videos of each users’ experience so that we can go back and analyze as necessary.

The testers’ overall impression was not positive enough. They found many little things that can be improved and at least one major problem.

The most significant area for improvement is the templates, where were hard to download and may not be the most attractive for Chinese users anyways.

Besides that, the buttons and Chinese content could be better localized.

1. What was the most frustrating thing about your experience?

1. It was very awkward to download templates. Only 2/5 testers were successful in downloading the template, but it took them a long time. The other three kept seeing a loading symbol, regardless if they were using their mobile data network or Wi-Fi. It sometimes showed a loading error.
2. The toolbar/navigation was unclear. Testers had a hard time finding what they wanted. For example, the “改造” (transform) button looks like it is a button for editing the image size, but it’s actually for turning/copying.
3. Many of the functions in the basic edition are locked.

2. Which creation method did you prefer (template or starting from scratch) and why? (Since some couldn’t use the templates, we also asked if they would have preferred to use templates or not).

4/5 liked to use the templates because it was easy-to-use and convenient. The designs were well-suited for sharing on social media.

Tester E preferred templates for making promotional materials but not for editing personal photos.

3. What did you like about the app? What would you improve?

What they liked:

1. It’s like a mobile Photoshop
2. Image layers
3. Brushes
4. Supports touch sensitivity

What they wanted to be improved:

1. Not easy enough to download materials.
2. The layout of the features didn’t seem right.
3. Translation not well-localized.
4. Would like more styles for filters and Chinese character fonts.
5. Lack of retouching features.
6. The templates could be more stylish or original. For example, in China, the “save” icon is usually like this: . But Bazaart’s is like:  , which two testers thought looked like a “send” symbol.

4. What would you use this app for in your daily life?

4/5 said they could use it to edit images to post on social media.

Tester E said Bazaart could also be useful for creating simple promotional materials at work.

ALL SUGGESTIONS

Time to sum up all the user suggestions. We’ve also added some of our own advice.

The biggest issue is that the templates couldn’t be used.

For people that could use templates, they’d like to see new templates – “better looking” ones.

List more of the commonly used features higher up in the navigation. For example, see how the image editing tool Snapseed’s interface is organized.

The ‘magic wand’ feature that lets users select the object of the photo was only successfully used by one tester. They did click it, but it usually didn’t work for them.

The wording could also be changed to better localize it. The translation for the app is not incorrect, but it isn’t the most common terminology used in other apps.

As expected, users mentioned that they expect to see beautification features.

When users type in Chinese characters, they do so with an English keyword. Bazaart has a bug where it would add the Chinese character and the English character in front of it.

All users mentioned that the name “把杂艺” sounded awkward. We didn’t even ask them about it, but they each mentioned it when we were chatting on WeChat.

OUR TEAM’S SUGGESTIONS (NOT TESTER OPINIONS)

We could definitely come up with a better brand name. (Update: After the publication of this post, our team was commissioned with a brand name creation service, and we came up with the name which the brand uses at the moment: 百色特)

For Chinese apps, it’s common to use a free trial period and then ask for payment. Bazaart shows a notification right after downloading, which advertises the advanced features for 257 Yuan/year. We recommend asking them to upgrade after a week.

The App Store copy should be rewritten. There are some errors, the content could surely be made more concise and persuasive.