Summer 2017

Bridging the experience gap between AI agents, bots, and Bing search

I spent the summer of 2017 interning at Microsoft as a UX Design Intern under their Artificial Intelligent and Research group. One of the three projects I got to work on a concept project thinking about what the Bing experience will be like a few years down the road with AI infused. This was a really exciting project as I had very little constraints and was able to take the project wherever I thought the future would be.

Role
Designer
Type of work
UI/UX

Initial Problem

How do we bridge the experience gap between AI agents, bots, and Bing search?

Problem with Status Quo

  • Siloed – No communication or connection between traditional search and bots
  • Unaware – Bing is not very predictive and good at anticipating user needs
  • Non contextual – Search and bots aren’t aware of their users’ background, preferences, lifestyle, activities, etc.

Core Elements

  • Conversational
  • Predictive
  • Aware (location, user, history, etc.)
  • Inform

Process

As this was a concept project, my design process was a bit different from usual. The process was still iterative but focused a lot more on my vision and what Microsoft has learned from their many AI experiments. I started off by looking at the AI products and experiments we had out in the public and examining users’ feedback to those products. Then after I had a good idea of what users liked and didn’t like about our different AI project, whether it’s a chatbot or entertainment app using computer vision, I started to brainstorm what the future of Bing with AI infused would look like. With the users’ feedback on other projects in mind, I was able to design knowing what frustrated users and what users wanted when it came to our next generation experiences.

Solutions

Through a number of explorations, I was able to land on a temporary solution, shown below. As this was a visionary project, we really don’t know when an experience is “finished.” Our explorations could have gone in so many directions which could all be considered “solutions.” I am glad about what we ended up with given the time constraint. I believe the desktop and mobile experience is really pushing the envelope when it comes to the future of search, encompassing the core elements we set out to incorporate. The experience, shown below in user stories, is truly conversational, predictive, aware, and informed.

Desktop Experience - UN Research Project User Story

Alex, a high school student in NY, researches about the UN for a school project

The desktop experience features a home page that has what a user would need to access right away - chat bar to the left and collections and saved topics to the right.
Alex can instantly start chatting with Bing and find what he needs for his research project without leaving the page.
Alex can then ask to see more and Bing will present him with the traditional search result page. However, Bing will still be on his side ready to answer any of his questions.
Bing will be able to dynamically change the search result based on what Alex asks it. Showing him the information he needs withouth having to dig through different queries.
This experience does not stop when Alex leaves the Bing environment. Alex is now reading a news article, but he can still continue to ask Bing questions.

This desktop experience specifically highlights a conversational and aware Bing. The search engine is able to understand the context of the conversation and continue to provide relevant answers even when Alex has moved on to a different screen. By using this chat model across our search result pages and non-Bing pages, we can allow users to ask all kinds of questions and receive high quality answers.

For example, on the search result pages users can ask questions like:

  • “Please summarize these articles for me”
  • Show me primary sources only
  • “Highlight articles from the New York Times”

And on a non-Bing websites, users can ask questions like:

  • “Who is this author?”
  • “Show me the section about UN Peacekeeping controversies”
  • “Is this source bias?”

For the desktop experience, we also experimented with a number of chat/search models shown below.

This model, as used above, was the most favored one throughout design reviews

Mobile Experience - Sneaker Shopping User Story

Tiffany, a creative in Dallas, shops for sneakers

The mobile experience is super simple. Tiffany is presented with the search bar immediately upon opening the app.
If she scrolls down, Tiffany can see her relevant collections and saved searches.
Tiffany decides to start chatting with Bing about looking for a new pair of sneakers. Because Bing knows Tiffany, it knows the kind of style and brands she likes and recommends her sneakers based on her taste, not general search results.
After tapping on a pair she likes, Tiffany is presented with a clean overview of the sneakers.
Scrolling down, this screen will give her access to the three main features we will be focusing on: Outfit Check, Try it on, and Find my size.
Using advance computer vision technology, Outfit Check allows Tiffany to see if the pair of sneakers will go well with her current style and wardrobe. The feature initially ask Tiffany to send Bing a few photos of her favorite outfits then ask her some questions about her style.
With Try it on, Bing will use AR to show Tiffany what the pair of sneaker looks like on her.
Finally, Find my size uses the camera and computer vision technology to measure Tiffany's foot and tell her what her shoe size should be for this pair of sneaker.

This mobile experience specifically highlights an aware and informed Bing. By knowing what Tiffany likes and her taste through shopping and search habits, Bing is able to provide her with the items it knows she’ll like. Different advance AR and AI technology is also shown in this experience making shopping easier for Tiffany.

Final Thoughts

Getting to intern at Microsoft was a great learning experience. I got to work on a total of three projects over the summer. While I was not able to show some of my other projects due to privacy concerns, this specific visionary project was extremely interesting. I was able to take full control of the directions I wanted to take it with minimum constrains. The openness of the project also allowed me to explore many aspects on how a search engine can help make life easier for users. This was my first time working within a big design team and I definitely loved being able to collaborate with other amazing designers. I learned the importance of storytelling and being able to communicate your design decisions to other collaborators and stakeholders within a greater organization.