IBM Watson Business Coach

IBM needed an app to help Watson sales teams educate potential clients about the reality of using Watson offerings while generating excitement about implementing Watson in their organizations.

Problem

Watson sellers were losing deals because clients had an unrealistic expectation of what Watson could do. Clients often thought Watson was a magic box that immediately solved any problem presented, without any training or significant investment in time or money.

Goal: Create an app that would bridge the gap between expectations and reality.

Team

The core team consisted of: 

• a product owner

• a scrum master

• 2 engineers

• a junior developer

• a content strategist

• a user researcher and designer (me)

This team was augmented, as needed, by a Watson architect, a group of case study writers, and several part-time visual designers. We also contracted with a company that coded the iOS app and helped implement some of the most complicated Watson APIs.

Timeline

<70 days

This type of app usually takes at least 6 months!

Role

User research, user experience design, initial visual design, front-end development, Watson Conversation training

Define scope / requirements

We had clear requirements from stakeholders, including:

1. Include and showcase Watson APIs.

2. Show clients detailed case studies of how organizations similar to theirs had successfully implemented AI.

3. Educate clients about Watson capabilities.

4. App must work on the web and in iOS.

5. Present the app during the World of Watson conference in just over 2 months.

When I came on board, the team had already decided to meet the first three requirements by creating a “Write Your Story” section. This was, essentially, a Mad Libs-like interface in which clients would be asked to fill in the blank to create their own business story. The story would determine where the clients were in their personal “cognitive journey” and would offer case studies indicating how companies in similar industries or with similar challenges had successfully used Watson.

Due to the time frame, we quickly brainstormed ideas for other features—games, videos, chatbots, stories—that would be both interesting and fun for users as well as meeting stakeholder requirements. Initially, we aimed to have a chatbot, an interactive game, and a general education section with articles and videos about Watson and AI, as well as education sections specific to each Watson vertical. However, it was very hard to get accurate and a similar volume of information from the different offerings and industries. (This proved to be overly ambitious for the time frame!)

However, since the story was one of the focal points of the app — and the most complicated technically — we focused on this first.

Research

I knew good user research would be critical to the app’s success. I also knew that I could reach out to our internal users — the sales teams — but would not be able to recruit actual customers until we demonstrated it at World of Watson. In addition, because Watson sales teams are located throughout the world, I needed to recruit participants from different geographical territories, rather than focus only on US teams. Luckily, I was able to search our internal directory for potential participants and kept track of emails sent, responses, and schedules.

Testing across geographies

One of the biggest challenges in recruiting and scheduling sessions with participants was dealing with various time zones and being respectful of a seller’s time. Our sellers were very generous with their time, but the time differences sometimes required scheduling sessions at 2 or 3 a.m. my time — and lots of coffee late at night!

Initial research session 1: Understanding the sales process and client concerns

For the first round of testing, I wanted to understand how teams sold Watson offerings, why clients were interested in Watson, and what difficulties the teams encountered in closing sales. Because the team had already written the initial set of questions, my primary goals for the first session were to:

1. Understand how sales teams approach selling Watson.

2. Determine if geography affects this process.

3. Understand the challenges and obstacles sales teams encountered.

4. Ask participants to talk through each of the 12 questions.

It turned out that sellers globally had similar processes and challenges. Notably, they said that Watson marketing and brand awareness were great — clients knew about Watson and were excited about the ability to derive better insights from their data. Until, that is, the client discovered that Watson is not like a computer or server that you turn on and, voilà, you are now using AI. Although some APIs were relatively simple and fast to set up and train, most organizations needed to use multiple services, all of which required training, knowledge experts, data, time, and money. Sellers were frustrated and disheartened not only about lost sales, but also about the Watson line in general.

To succeed, the app needed not only to showcase Watson to help sales teams excite and engage prospects with AI, but also give clients a realistic sense of the commitment needed.

Watson is not like a computer or server that you turn on and, voilà, you are now using cutting-edge AI and will immediately derive meaningful insights from your data.

The questions for Write Your Story also needed improvement. These were marketing questions, most of which no sales team would ever ask a client in early meetings. Clearly, we had work to do!

Initial research session 2: Refining questions and understanding technical concerns

After presenting my results to the team, we revised the questions and the accompanying story. Then I went back to the sales teams. The goals were similar to the previous round, but I also spoke with technical sellers to understand the technical side of the sales process and clients’ early technical questions and concerns. I wanted to learn whether the wording of some questions was American vernacular or universally understandable by English-speakers.

Although sellers still thought the questions were too marketing focused and would turn off prospects, they said we were getting closer. We also needed to be more cognizant of the global nature of our users. Despite initial push-back, I now had data to show that, for example, many people do not understand the slangy meaning of questions, such as “What keeps you up at night?”

I finally had a good sense of our users and, perhaps more importantly, had established a collaborative relationship with sales teams.

Test early and often

I continued to conduct user research sessions throughout the process. This helped inform the design and usability, and ensured the content and focus stayed on the actual users — sales teams and their clients.

Users

We already knew we needed to focus on two primary users: sales teams and their clients. The app also needed to be presented at a major IBM Watson event and, although we would know exactly who was attending, we were not sure how attendees would interact with the app. Due to time constraints, we researched, but did not create, explicit personas.

My initial research clarified the main users:

1. Clients on the business side

2. Clients on the technical side

3. Sales teams

We had been considering all clients as business-focused, meaning they were interested in the benefits of AI, possibly wanted to be early adopters, needed help to use their data, and were concerned with finances. We erroneously assumed that these users were also the technical decision-makers, but this was definitely not the case when implementing AI. Business-focused and tech-focused clients had different concerns and interests, and sales teams approached them differently. We needed to have two distinct paths — business and technical — for users to take while writing their stories, as well as more detailed technical information for users who wanted it, or we could focus on the business side only. We ultimately opted for the second approach. We did not have time to develop additional content and it turned out that effectively targeting technical users would have required additional resources.

Now, we needed to determine use cases and focused on the following:

1. Clients by themselves: A potential client could find the Watson Business Coach on IBM.com or in the App Store while researching Watson, so the app needed to be accessible and intuitive to clients. We focused on legitimate prospects, rather than on anyone who just stumbled on the tool and tried it for fun, but the tool needed to educate and be interesting to anyone who interacted with it.

2. Sales teams or individual sellers at events: Sales teams interacted with prospects in several scenarios. First, sellers often met with possible clients during events. In this case, the seller and client would meet in a private room at an event, where the seller could network and walk a prospect through the app.

3. Sales teams or individual sellers in client meetings: Similar to the previous scenario, sales teams scheduled meetings with interested prospects. Here, however, teams rather than individual sellers met with clients, and sales teams often included technical sellers to better engage with a client’s implementation team.

In addition, clients could interact with the app during events, either at a kiosk or with IBM employees. This was less likely than the other scenarios, but needed to be considered since we would be launching the app at the Watson conference.

Refining The Story

Our sales teams were gracious enough to participate in user research and feedback sessions throughout the process. Their help was invaluable and contributed to the ultimate project success.

We continued to iterate on the questions and I tested every iteration on at least five sellers from different geographies.

Several key points emerged:

1. Questions needed to reflect actual sales team–client conversations.

2. Language needed to be clear and straightforward to avoid confusing users and giving them a negative view of Watson.

3. Even with clearer language, the questions were not always easy to answer. Users needed the ability to skip questions when they did not know the answer or did not want to answer. We needed to add contextual help to the most troublesome questions. While people usually know their company’s industry, for instance, they might not know what unstructured data they would use to train Watson. Contextual help, in the form of hints and possible answers, allowed users to write their story easily.

4. Users wanted an indication of how many questions they needed to answer and where they were in the process.

5. We needed to allow users to read their finished story and make necessary edits.

The final story was cleaner, clearer, understandable, and considerably shorter than the original version — 8 questions instead of 12. The questions made a client feel more like they were having an early sales conversation and were natural for a seller to talk through with a client. Finally, we changed the title from Write Your Story to Write My Story to be more personal and motivating.

Exploring functionality and design

Collaborating on content

Although the story feature was not complete, we needed to work on content. Originally, the content strategist and I had rather grandiose plans to have an interactive and instructive game, as well as a chatbot and sections for all Watson offerings. After many hours whiteboarding and talking with various sellers, we decided to shift our focus to highlight how Watson was used in different industries. Sales teams sold by industry — automotive, banking, retail, manufacturing, etc. — not by product line, which makes perfect sense since anyone in any industry can use whatever APIs they need to solve a problem, and companies in different industries can face similar challenges. For example, a company can stave off a PR nightmare recall scenario if they can spot potential failures early, which is useful in automotive, manufacturing, and healthcare.

Challenges and pivoting

The two biggest challenges we faced were time and the inability to get equal amounts of valid information about the industries. We had eliminated health early, due to its complexity and numerous regulations. It was surprisingly hard to get information about the use of Watson in the auto industry, whereas information about Watson in retail, say, was abundant. It is likely that some of this was due to certain industries being earlier adopters than others. After all, it’s reasonably simple, safe, and potentially lucrative to use predictive analytics to encourage shoppers to purchase pants but much more complicated to have a car schedule its own repair appointment and direct its driver to a dealer for the work.

In the end, we realized that we would not be able to include all our planned features. We quickly pivoted and decided to eliminate the game, as well as to reduce the industries to a single Watson for Industry section with content that applied to various industries but was still educational, approachable, and applicable to a variety of business challenges.

Sketching

Time to sketch! Since I now had a general idea of the content and features, I needed to start sketching. Sketching allows me to quickly generate many ideas, and to throw them out just as quickly. I find it’s much easier to dismiss a sketch that clearly doesn’t work than it is to get rid of an art board I’ve carefully been working on for an hour.

This was challenging, however, since we still didn’t have actual content so I wasn’t sure if the content and features I anticipated and designed for would be included. Original sketches and designs included content that was ultimately removed, but my concepts were general enough that they worked anyway.

User flows

Sketching helped distill my ideas and give a narrow focus to the design. This was a bit uncertain, however, as we were still working out content and features. At this point, we knew we would have a chatbot, and sketching (and experience as a user) had made it clear that the chat design needed to differ on mobile. On a full screen, a user can chat while viewing additional content but on mobile, the chat needs to take over the entire screen. In addition, we wanted a user to be able to quickly switch between sections — as an exploratory and educational tool, users should have the freedom to learn in a way that suits them best. In addition, the flow needed to work for a seller guiding a client through the app.

While I don’t always create user flows — good design is contextual — I find them helpful for apps that have limited screens with dynamic content. This not only ensures the anticipated screens and interactions will work and make sense to users, but also helps determine the navigation. In this case, I created an overall user flow for the app and then created different schemas for desktop/tablet and mobile versions, which allowed me to visualize how the design would differ on devices to optimize usability without altering the content.

Chat

The Watson chatbot was another key feature. I’d never designed a bot before and found it both challenging and fascinating — I always enjoy projects that teach me something new and improve my skills! The content strategist and I did some quick research on creating chatbots, and realized that the key problem we faced was setting user expectations. Nearly everyone has interacted with Siri or Google, which means that the general expectation is that an AI bot can answer anything you ask it. This, however, is not the case. We needed to show off Watson’s capabilities, not have users interact with the bot and walk away shaking theirs heads, disappointed.

Success depended on correctly setting user expectations about what the chatbot could and, more importantly, could not, answer.

We broke the challenge into the following steps:

1. Determine scope

2. Set Expectations

3. Figure out edge cases and unknowns

4. Create flow

5. Design the bot

6. Test and iterate

Determine scope

At first, we wanted a Watson on Watson bot that could answer almost any question a user might ask about Watson and AI. We also wanted to be able to answer questions about the weather or places to eat, but we had to narrow the scope to a strategically focused and guided conversation about the app and offer users relevant content and case studies, as well as answer common questions clients ask about Watson and AI. Again, the participant pool of sellers I was testing with were incredibly helpful in giving us questions and answers and, later, in training the bot.

However, the chatbot turned out to be one of the most challenging aspects of the project. Stakeholders wanted the bot to offer content from the app, such as case studies, videos, or Write My Story, and had given the team a preliminary set of questions to train Watson. When I tested this on both coworkers and sellers, the questions were so jargon-filled that no one could figure out how to phrase a question to get an accurate response. Even worse, the sales teams literally laughed at how bad the bot was while testing it.

Unsurprisingly, most people do not think to ask for a ‘client reference’ when they want to read a case study.

Training the bot was challenging.

Set Expectations

The most critical step was appropriately setting user expectations. For example, I would not expect my insurance app’s chatbot to recommend restaurants. We wanted to be very certain that no one would expect Watson to be another Siri or Google and walk away unimpressed. We checked out numerous chatbots, read about bot design, and spoke with the Watson brand team to ensure we understood and adhered to Watson’s tone of voice. Stakeholders also wanted to incorporate text-to-speech, so a user could literally chat with Watson. The team was opposed to this, as it introduced unnecessary complexity, and the app would be used at a crowded event with a lot of background noise.

While testing different phrasing with sellers, I found that users preferred something short and to the point — something they could easily scan and understand. In the end, we decided on the simple, instructional statement: “Hello, Watson here. I’m here to answer your cognitive business questions. You can ask to see demos and case studies or hear statistics about how business leaders are making an impact with cognitive computing. Type ‘tell me more’ to go deeper into a specific topic. Type ‘help’ to see this guidance again. Click the microphone to turn the sound on and off.”

This introduction sets parameters and prevents the users from getting stuck and annoyed. No one wants the frustration of automated phone systems that never take you where you want to go.

Lastly, I felt it was important that users’ chat history be maintained during their interaction with the app. The chat window could be opened or closed, and research showed that users clearly expected to see their chat history if they closed and reopened the chat.

User Flow

We had to figure out what Watson should say in response to a questions the bot can’t answer. I focused on three main cases to deal with:

1. A user asks a question Watson can answer.
   This involved mapping out a conversation that logically flows from the question and, based on the conversation, gives the user useful information or content.

2. A user asks for a joke or asks a silly question.
   We wrote two jokes and responses to some silly questions, such as “Are you aiming for world domination?” We limited these to two such questions in a row. The third request generated the response “Let’s get serious” to get the user back on track.

3. A user asks a question that Watson is not trained on or that is out of scope.
   We came up with several variations on the theme of “I’m still learning.”

Designing a good chat experience is deceptively difficult. I worked with our Watson architect to write the code to train the bot. In fact, we were still refining the bot in the hotel the night before we had to demo it. I certainly did not expect to spend so much time on a chatbot, but that reinforced the challenge our sales teams faced — presenting accurate but enticing information about Watson!

Case studies

We worked closely with the client reference team responsible for interviewing customers and writing case studies. They reached out to Watson customers for additional, detailed information about these projects, as well as permission to use the updated, blinded case studies, which had details most companies would not make public.

Objectives

The case studies needed to accomplish several objectives:

1. Show users how other companies in similar industries or facing similar challenges had successfully implemented Watson.

2. Be realistic by discussing timelines, teams, and challenges.

3. Show the structured and instructed data, and the training used.

4. Get customers excited and inspire them to talk to sales about using Watson in their organizations.

Designing stories

User research showed that there was a general hierarchy of client concerns about undertaking Watson projects:

1. Is this company like mine? Is this the same problem?

2. Did the solution work?

3. How much time and money are involved?

4. What do I need to actually do this — do I have the right data and team in place?

5. How did they do it?

Customers needed to easily imagine themselves using Watson products to solve their real problems. Based on research, I decided to focus attention on the challenge, solution, benefits, and timeline up front. This allowed a user to quickly check if the case study was worth reading — to quickly grab a user’s attention and imagination. Then they could delve deeper into the details about how the successful solution was implemented.

This was an example of the benefits of close collaboration with a great visual designer. He was able to translate what, at this point, were ideas and sketches into a simple and effective design.

Creating a focal point

I worked on the home page last, largely because it depended on the final content and features in the app. The goal here was to allow users to navigate to any part of the app, while encouraging them to write their story. The story was the highlight as well as being fun and useful for clients. It also allowed users to email the results to themselves, generating leads for sales teams.

Visual design

Initially, I was doing visual design in addition to user research and experience design, but I ultimately served more as a creative director.

This project was challenging in many ways, one of which was the need to conduct research, execute designs, develop and create content, and code the entire app simultaneously, resulting in somewhat frantic changes and pivots. I lobbied hard for a particular visual designer (he’s brilliant, by the way) to work with our team, since we worked well together and I could trust him to create great designs. My close and good working relationship with him gave me the luxury of literally talking through concepts or sketches and having something amazing happen. Given the time line, this was a critical factor in the overall success of the project.

Designing for web and iOS

Because this was the first native iPad app I had designed, I quickly studied Apple’s human interface guidelines. The app needed to take advantage of native functionality while adhering to IBM’s design guidelines, as well as Watson brand guidelines. These are similar, but distinct. Working with the visual designer, we decided to use iOS components for some interactions, such as navigation, so the app would feel more intuitive, but stick with Watson/IBM brands and design for the rest. Again, the main challenge was having time to produce and iterate on pixel-perfect designs in time to hand them off to the Ukrainian contractors doing the coding. Luckily, we were able to recruit two IBMers who were interested in design and knew Photoshop to finalize every iteration of the iOS app designs.

We also decided to use masonry so it would be easy to add and remove content without wrecking the design.

Other decisions

I based design decisions primarily on three factors:

1. Was there enough time?

2. What did research indicate?

3. What did stakeholders want?

Successful design involves figuring out how to marry business and stakeholder needs with usability and user needs, and do it within time and budget constraints.

Masonry allowed us to easily add and remove content cards.

Solution

RESULTs and lessons learned

The Watson Business Coach app that we developed invites clients to tell the story of their business challenges so Watson can suggest cognitive computer solutions. Working with sellers, we trained Watson in the language sellers and customers use. And we helped sellers and their clients understand just what Watson actually is. The app features a voice-enabled conversation with Watson that guides the clients as they share their business stories. It then gives customers a personalized roadmap for implementing Watson / AI in their own company and directs them to inspirational, relatable, real-life, tactical client success stories.

We had a very good team that worked well together. We communicated well, were flexible, and no one was married to their job title.

Due to time, I ultimately moved to designing and prototyping in code. I had known that the junior designer was struggling a lot. Once I moved into coding, I realized — with horror— that although the design seemed easy, it was really very difficult and almost impossible to reproduce. Luckily, we were able to work together to create a similar design that was easy to code. The lesson learned was that good communication and a good working relationship with devs  is essential. We need to we know what they are dealing with. With the new, young dev, I learned to be proactive in eliciting problems because, at first, he was loathe to speak up.

Clients and sales teams were excited to try the app at World of Watson and some clients downloaded the app on the spot.