{"id":19383,"date":"2024-08-13T11:25:17","date_gmt":"2024-08-13T11:25:17","guid":{"rendered":"https:\/\/uxmag.com\/?p=19383"},"modified":"2024-12-26T11:46:44","modified_gmt":"2024-12-26T11:46:44","slug":"when-words-cannot-describe-designing-for-ai-beyond-conversational-interfaces","status":"publish","type":"post","link":"https:\/\/uxmag.com\/articles\/when-words-cannot-describe-designing-for-ai-beyond-conversational-interfaces","title":{"rendered":"When Words Cannot Describe: Designing For AI Beyond Conversational Interfaces"},"content":{"rendered":"\n

As Artificial Intelligence evolves the computing paradigm, designers have an opportunity to craft more intuitive user interfaces. Text-based Large Language Models unlock most of the new capabilities, leading many to suggest a shift from graphical interfaces to conversational ones like a chatbot is necessary. However, plenty of evidence suggests conversation is a poor interface for many interaction patterns. Maximillian Piras examines how the latest AI capabilities can reshape the future of human-computer interaction beyond conversation alone.<\/em><\/p>\n\n\n\n

Few technological innovations can completely change the way we interact with computers. Lucky for us, it seems we\u2019ve won front-row seats to the unfolding of the next paradigm shift.<\/p>\n\n\n\n

These shifts tend to unlock a new abstraction layer to hide the working details of a subsystem. Generalizing details allows our complex systems to appear simpler & more intuitive. This streamlines coding programs for computers as well as designing the interfaces to interact with them.<\/p>\n\n\n\n

The Command Line Interface<\/strong>, for instance, created an abstraction layer to enable interaction through a stored program. This hid the subsystem details once exposed in earlier computers that were only programmable by inputting 1s & 0s through switches.<\/p>\n\n\n\n

Graphical User Interfaces (GUI)<\/strong> further abstracted this notion by allowing us to manipulate computers through visual metaphors. These abstractions made computers accessible to a mainstream of non-technical users.<\/p>\n\n\n\n

Despite these advances, we still haven\u2019t found a perfectly<\/em> intuitive interface \u2014 the troves of support articles across the web make that evident. Yet recent advances in AI have convinced many technologists that the next evolutionary cycle of computing is upon us.<\/p>\n\n\n\n

\"\"
Layers of interface abstraction, bottom to top: Command Line Interfaces, Graphical User Interfaces, & AI-powered Conversational Interfaces. (Image source: Maximillian Piras<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

The Next Layer Of Interface Abstraction<\/h2>\n\n\n\n

A branch of machine learning called generative AI<\/strong> drives the bulk of recent innovation. It leverages pattern recognition in datasets to establish probabilistic distributions that enable novel constructions of text, media, & code. Bill Gates believes<\/a> it\u2019s \u201cthe most important advance in technology since the graphical user interface\u201d because it can make controlling computers even easier. A newfound ability to interpret unstructured data, such as natural language, unlocks new inputs & outputs to enable novel<\/a> form<\/a> factors<\/a>.<\/p>\n\n\n\n

Now our universe of information can be instantly invoked through an interface as intuitive as talking to another human. These are the computers we\u2019ve dreamed of in science fiction, akin to systems like Data from Star Trek. Perhaps computers up to this point were only prototypes & we\u2019re now getting to the actual product launch. Imagine if building the internet was laying down the tracks, AIs could be the trains to transport all of our information at breakneck speed<\/strong> & we\u2019re about to see what happens when they barrel into town.<\/p>\n\n\n\n

\n

<\/p>\n\u201cSoon the pre-AI period will seem as distant as the days when using a computer meant typing at a C:> prompt rather than tapping on a screen.\u201d<\/em>

\u2014 Bill Gates in \u201c<\/em>The Age of AI Has Begun<\/a>\u201d<\/em><\/cite><\/blockquote>\n\n\n\n

If everything is about to change, so must the mental models of software designers. As Luke Wroblewski<\/a> once popularized mobile-first design, the next zeitgeist is likely AI-first. Only through understanding AI\u2019s constraints & capabilities can we craft delight. Its influence on the discourse of interface evolution has already begun.<\/p>\n\n\n\n

Large Language Models (LLMs), for instance, are a type of AI utilized in many new applications & their text-based nature leads many to believe a conversational interface, such as a chatbot, is a fitting form for the future. The notion that AI is something you talk to has been permeating across the industry for years<\/a>. Robb Wilson, the co-owner of UX Magazine, calls conversation \u201cthe infinitely scalable interface\u201d in his book The Age of Invisible Machines<\/em> (2022). Noah Levin, Figma\u2019s VP of Product Design, contends<\/a> that \u201cit\u2019s a very intuitive thing to learn how to talk to something.\u201d Even a herald of GUIs such as Bill Gates posits<\/a> that \u201cour main way of controlling a computer will no longer be pointing and clicking.\u201d<\/p>\n\n\n\n

\"\"
Microsoft Copilot is a new conversational AI feature being integrated across their office suite (Image source: Microsoft<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

The hope is that conversational computers will flatten learning curves. Jesse Lyu, the founder of Rabbit, asserts<\/a> that a natural language approach will be \u201cso intuitive that you don\u2019t even need to learn how to use it.\u201d<\/p>\n\n\n\n

After all, it\u2019s not as if Data from Stark Trek came with an instruction manual or onboarding tutorial. From this perspective, the evolutionary tale<\/a> of conversational interfaces superseding GUIs seems logical & echoes the earlier shift away from command lines. But others have opposing opinions<\/a>, some going as far as Maggie Appleton<\/a> to call conversational interfaces like chatbots \u201cthe lazy solution.\u201d<\/p>\n\n\n\n

This might seem like a schism at first, but it\u2019s more so a symptom of a simplistic framing of interface evolution. Command lines are far from extinct; technical users still prefer them for their greater flexibility & efficiency. For use cases like software development or automation scripting, the added abstraction layer in graphical no-code tools can act as a barrier rather than a bridge.<\/p>\n\n\n\n

\n

<\/p>\nGUIs were revolutionary but not a panacea. Yet there is ample research to suggest conversational interfaces won\u2019t be one, either. For certain interactions, they can decrease usability, increase cost, & introduce security risk relative to GUIs.<\/a><\/cite><\/blockquote>\n\n\n\n

So, what is the right interface for artificially intelligent applications?<\/strong> This article aims to inform that design decision by contrasting the capabilities & constraints of conversation as an interface.<\/p>\n\n\n\n

Connecting The Pixels<\/h2>\n\n\n\n

We\u2019ll begin with some historical context, as the key to knowing the future often starts with looking at the past. Conversational interfaces feel new, but we\u2019ve been able to chat with computers for decades.<\/p>\n\n\n\n

Joseph Weizenbaum invented the first chatbot, ELIZA<\/a>, during an MIT experiment in 1966. This laid the foundation for the following generations of language models to come, from voice assistants like Alexa to those annoying phone tree menus. Yet the majority of chatbots were seldom put to use beyond basic tasks like setting timers<\/a>.<\/p>\n\n\n\n

It seemed most consumers weren\u2019t that excited to converse with computers after all. But something changed last year. Somehow we went from CNET reporting<\/a> that \u201c72% of people found chatbots to be a waste of time\u201d to ChatGPT gaining 100 million weekly active users<\/a>.<\/p>\n\n\n\n

\"\"
A conversation with the first chatbot, ELIZA, invented in 1966. (Image source: Wikipedia<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

What took chatbots from arid to astonishing? Most assign credit to OpenAI\u2019s 2018 invention<\/a> (PDF) of the Generative Pre-trained Transformer (GPT)<\/strong>. These are a new type of LLM with significant improvements in natural language understanding. Yet, at the core of a GPT is the earlier innovation of the transformer architecture introduced in 2017<\/a> (PDF). This architecture enabled the parallel processing required to capture long-term context around natural language inputs. Diving deeper, this architecture is only possible thanks to the attention mechanism introduced in 2014<\/a> (PDF). This enabled the selective weighing of an input\u2019s different parts.<\/p>\n\n\n\n

Through this assemblage of complementary innovations, conversational interfaces now seem to be capable of competing with GUIs on a wider range of tasks. It took a surprisingly similar path to unlock GUIs as a viable alternative to command lines. Of course, it required hardware like a mouse to capture user signals beyond keystrokes & screens of adequate resolution. However, researchers found the missing software ingredient years later with the invention of bitmaps.<\/p>\n\n\n\n

Bitmaps allowed for complex pixel patterns that earlier vector displays struggled with. Ivan Sutherland\u2019s Sketchpad, for instance, was the inaugural GUI but couldn\u2019t support concepts like overlapping windows. IEEE Spectrum\u2019s Of Mice and Menus<\/a> (1989) details the progress that led to the bitmap\u2019s invention by Alan Kay\u2019s group at Xerox Parc. This new technology enabled the revolutionary WIMP (windows, icons menus, and pointers)<\/a> paradigm that helped onboard an entire generation to personal computers through intuitive visual metaphors.<\/p>\n\n\n\n

Computing no longer required a preconceived set of steps at the outset. It may seem trivial in hindsight, but the presenters were already alluding to an artificially intelligent system during Sketchpad\u2019s MIT demo<\/a> in 1963. This was an inflection point transforming an elaborate calculating machine into an exploratory tool<\/strong>. Designers could now craft interfaces for experiences where a need to discover eclipsed the need for flexibility & efficiency offered by command lines.<\/p>\n\n\n\n

\"\"
Susan Kare\u2019s early sketch for the pointer icon in Apple\u2019s GUI. (Image source: Susan Kare<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

Parallel Paradigms<\/h2>\n\n\n\n

Novel adjustments to existing technology made each new interface viable for mainstream usage \u2014 the cherry on top of a sundae, if you will. In both cases, the foundational systems were already available, but a different data processing decision made the output meaningful enough to attract a mainstream audience beyond technologists.<\/p>\n\n\n\n

With bitmaps, GUIs can organize pixels into a grid sequence to create complex skeuomorphic structures. With GPTs, conversational interfaces can organize unstructured datasets to create responses with human-like (or greater) intelligence.<\/p>\n\n\n\n

The prototypical interfaces of both paradigms were invented in the 1960s, then saw a massive delta in their development timelines \u2014 a case study unto itself. Now we find ourselves at another inflection point: in addition to calculating machines & exploratory tools, computers can act as life-like entities<\/strong>.<\/p>\n\n\n\n

But which of our needs call for conversational interfaces over graphical ones? We see a theoretical solution to our need for companionship in the movie Her<\/em>, where the protagonist falls in love with his digital assistant. But what is the benefit to those of us who are content with our organic relationships? We can look forward to validating the assumption that conversation is a more intuitive interface<\/strong>. It seems plausible because a few core components of the WIMP paradigm have well-documented usability issues.<\/p>\n\n\n\n

Nielsen Norman Group<\/a> reports that cultural differences make universal recognition of icons rare \u2014 menus trend towards an unusable mess with the inevitable addition of complexity over time. Conversational interfaces appear<\/em> more usable because you can just tell the system when you\u2019re confused! But as we\u2019ll see in the next sections, they have their fair share of usability issues as well.<\/p>\n\n\n\n

\n

<\/p>\nBy replacing menus with input fields, we must wonder if we\u2019re trading one set of usability problems for another.<\/a><\/cite><\/blockquote>\n\n\n\n

The Cost of Conversation<\/h2>\n\n\n\n

Why are conversational interfaces so popular in science fiction movies? In a Rhizome essay<\/a>, Martine Syms theorizes that they make \u201cfor more cinematic interaction and a leaner production.\u201d This same cost\/benefit applies to app development as well. Text completion delivered via written or spoken word is the core capability of an LLM. This makes conversation the simplest package for this capability from a design & engineering perspective.<\/p>\n\n\n\n

Linus Lee, a prominent AI Research Engineer, characterizes<\/a> it as \u201cexposing the algorithm\u2019s raw interface.\u201d Since the interaction pattern & components are already largely defined, there isn\u2019t much more to invent \u2014 everything can get thrown into a chat window.<\/p>\n\n\n\n

\n

<\/p>\n\u201cIf you\u2019re an engineer or designer tasked with harnessing the power of these models into a software interface, the easiest and most natural way to \u201cwrap\u201d this capability into a UI would be a conversational interface\u201d<\/em>

\u2014 Linus Lee in <\/em>Imagining Better Interfaces to Language Models<\/a><\/cite><\/blockquote>\n\n\n\n

This is further validated by The Atlantic\u2019s reporting on ChatGPT\u2019s launch<\/a> as a \u201clow-key research preview.\u201d OpenAI\u2019s hesitance to frame it as a product suggests a lack of confidence in the user experience. The internal expectation was so low that employees\u2019 highest guess on first-week adoption was 100,000 users (90% shy of the actual number).<\/p>\n\n\n\n

\n

<\/p>\nConversational interfaces are cheap to build, so they\u2019re a logical starting point, but you get what you pay for. If the interface doesn\u2019t fit the use case, downstream UX debt can outweigh any upfront savings.<\/a><\/cite><\/blockquote>\n\n\n\n

\"\"
A visualization of how easy it is to wrap an LLM\u2019s raw output into a conversational interface. (Image source: Maximillian Piras<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

Forgotten Usability Principles<\/h2>\n\n\n\n

Steve Jobs once said<\/a>, \u201cPeople don\u2019t know what they want until you show it to them.\u201d Applying this thinking to interfaces echoes a usability evaluation called discoverability<\/em>. Nielsen Norman Group<\/a> defines it as a user\u2019s ability to \u201cencounter new content or functionality that they were not aware of.\u201d<\/p>\n\n\n\n

A well-designed interface should help users discover what features exist. The interfaces of many popular generative AI applications today revolve around an input field in which a user can type in anything to prompt the system. The problem is that it\u2019s often unclear what a user should<\/em> type in to get ideal output. Ironically, a theoretical solution to writer\u2019s block<\/a> may have a blank page problem itself.<\/p>\n\n\n\n

\n

<\/p>\n\u201cI think AI has a problem with these missing user interfaces, where, for the most part, they just give you a blank box to type in, and then it\u2019s up to you to figure out what it might be able to do.\u201d<\/em>

\u2014 Casey Newton on <\/em>Hard Fork Podcast<\/a><\/cite><\/blockquote>\n\n\n\n

Conversational interfaces excel at mimicking human-to-human interaction but can fall short elsewhere. A popular image generator named Midjourney, for instance, only supported text input at first but is now moving towards a GUI<\/a> for \u201cgreater ease of use.\u201d<\/p>\n\n\n\n

This is a good reminder that as we venture into this new frontier, we cannot forget classic human-centered principles like those in Don Norman\u2019s seminal book The Design of Everyday Things<\/em> (1988). Graphical components still seem better aligned with his advice of providing explicit affordances & signifiers to increase discoverability.<\/p>\n\n\n\n

There is also Jakob Nielsen\u2019s list of 10 usability heuristics<\/a>; many of today\u2019s conversational interfaces seem to ignore every one of them. Consider the first usability heuristic<\/strong> explaining how visibility of system status educates users about the consequences of their actions. It uses a metaphorical map\u2019s \u201cYou Are Here\u201d pin to explain how proper orientation informs our next steps.<\/p>\n\n\n\n

Navigation is more relevant to conversational interfaces like chatbots than it might seem<\/strong>, even though all interactions take place in the same chat window. The backend of products like ChatGPT will navigate across a neural network to craft each response by focusing attention on a different part of their training datasets.<\/p>\n\n\n\n

\"\"
A visualization of how role-playing in prompt engineering loosely guides an AI model to craft different output. (Image source: Maximillian Piras<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

Putting a pin on the proverbial map of their parametric knowledge isn\u2019t trivial. LLMs are so opaque that even OpenAI admits<\/a> they \u201cdo not understand how they work.\u201d Yet, it is possible to tailor inputs in a way that loosely guides a model to craft a response from different areas of its knowledge.<\/p>\n\n\n\n

One popular technique for guiding attention is role-playing<\/strong>. You can ask an LLM to assume a role, such as by inputting \u201cimagine you\u2019re a historian,\u201d to effectively switch its mode. The Prompt Engineering Institute explains<\/a> that when \u201ctraining on a large corpus of text data from diverse domains, the model forms a complex understanding of various roles and the language associated with them.\u201d Assuming a role invokes associated aspects in an AI\u2019s training data, such as tone, skills, & rationality.<\/p>\n\n\n\n

For instance, a historian role responds with factual details whereas a storyteller role responds with narrative descriptions. Roles can also improve task efficiency through tooling, such as by assigning a data scientist role to generate responses with Python code.<\/p>\n\n\n\n

Roles also reinforce social norms, as Jason Yuan remarks<\/a> on how \u201cyour banking AI agent probably shouldn\u2019t be able to have a deep philosophical chat with you.\u201d Yet conversational interfaces will bury this type of system status in their message history, forcing us to keep it in our working memory<\/a>.<\/p>\n\n\n\n

\"\"
A theoretical AI chatbot that uses a segmented controller to let users specify a role in one click \u2014 each button automatically adjusts the LLM\u2019s system prompt. (Source: Maximillian Piras<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

The lack of persistent signifiers for context, like roleplay, can lead to usability issues. For clarity, we must constantly ask the AI\u2019s status, similar to typing ls<\/code> & cd<\/code> commands into a terminal. Experts can manage it, but the added cognitive load is likely to weigh on novices. The problem goes beyond human memory, systems suffer from a similar cognitive overload. Due to data limits in their context windows, a user must eventually reinstate any roleplay below the system level. If this type of information persisted in the interface, it would be clear to users & could be automatically reiterated to the AI in each prompt.<\/p>\n\n\n\n

Character.ai<\/a> achieves this by using historical figures as familiar focal points. Cultural cues lead us to ask different types of questions to \u201cAl Pacino\u201d than we would \u201cSocrates.\u201d A \u201ccharacter\u201d becomes a heuristic to set user expectations & automatically adjust system settings. It\u2019s like posting up a restaurant menu; visitors no longer need to ask what there is to eat & they can just order instead.<\/p>\n\n\n\n

\n

<\/p>\n\u201cHumans have limited short-term memories. Interfaces that promote recognition reduce the amount of cognitive effort required from users.\u201d<\/em>

\u2014 Jakob Nielsen in \u201c<\/em>10 Usability Heuristics for User Interface Design<\/a>\u201d<\/em><\/cite><\/blockquote>\n\n\n\n

Another forgotten usability lesson is that some tasks are easier to do than to explain, especially through the direct manipulation<\/a> style of interaction popularized in GUIs.<\/p>\n\n\n\n

Photoshop\u2019s new generative AI features reinforce this notion by integrating with their graphical interface. While Generative Fill<\/a> includes an input field, it also relies on skeuomorphic controls like their classic lasso tool. Describing which part of an image to manipulate is much more cumbersome than clicking it.<\/p>\n\n\n\n

Interactions should remain outside of an input field when words are less efficient.<\/strong> Sliders seem like a better fit for sizing, as saying \u201cmake it bigger\u201d leaves too much room for subjectivity. Settings like colors & aspect ratios are easier to select than describe. Standardized controls can also let systems better organize prompts behind the scenes. If a model accepts specific values for a parameter, for instance, the interface can provide a natural mapping for how it should be input.<\/p>\n\n\n\n

\"\"
A diagram of Visual Electric\u2019s input field showcasing how graphical controls can help a system organize a prompt behind the scenes. (Image source: Maximilian Piras<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

Most of these usability principles are over three decades old now, which may lead some to wonder if they\u2019re still relevant. Jakob Nielsen recently remarked on the longevity of their relevance<\/a>, suggesting that \u201cwhen something has remained true for 26 years, it will likely apply to future generations of user interfaces as well.\u201d However, honoring these usability principles doesn\u2019t require adhering to classic components. Apps like Krea are already exploring new GUI<\/a> to manipulate generative AI.<\/p>\n\n\n\n

Prompt Engineering Is Engineering<\/h2>\n\n\n\n

The biggest usability problem with today\u2019s conversational interfaces is that they offload technical work to non-technical users. In addition to low discoverability, another similarity they share with command lines is that ideal output is only attainable through learned commands<\/strong>. We refer to the practice of tailoring inputs to best communicate with generative AI systems as \u201cprompt engineering\u201d. The name itself suggests it\u2019s an expert activity, along with the fact that becoming proficient in it can lead to a $200k salary<\/a>.<\/p>\n\n\n\n

Programming with natural language is a fascinating advancement but seems misplaced as a requirement in consumer applications. Just because anyone can now speak the same language as a computer doesn\u2019t mean they know what to say or the best way to say it \u2014 we need to guide them. While all new technologies have learning curves, this one feels steep enough to hinder further adoption & long-term retention.<\/p>\n\n\n\n

\"\"
Canva markets its AI features as \u201cMagic Studio.\u201d (Image source: Canva<\/a>) (Large preview<\/a>)<\/figcaption><\/figure>\n\n\n\n

Prompt engineering as a prerequisite for high-quality output seems to have taken on the mystique of a dark art. Many marketing materials for AI features reinforce this through terms like \u201cmagic.\u201d If we assume there is a positive feedback loop at play, this opaqueness must be an inspiring consumer intrigue.<\/p>\n\n\n\n

But positioning products in the realm of spellbooks & shamans also suggests an indecipherable experience \u2014 is this a good long-term strategy? If we assume Steve Krug\u2019s influential lessons from Don\u2019t Make Me Think<\/em><\/a> (2000) still apply, then most people won\u2019t bother to study proper prompting & instead will muddle through<\/em>.<\/p>\n\n\n\n

But the problem with trial & error in generative AI is that there aren\u2019t any error states; you\u2019ll always get a response. For instance, if you ask an LLM to do the math, it will provide you with confident answers that may be completely wrong<\/a>. So it becomes harder to learn from errors when we are unaware if a response is a hallucination. As OpenAI\u2019s Andrej Karpathy suggests<\/a>, hallucinations are not necessarily a bug because LLMs are \u201cdream machines,\u201d so it all depends on how interfaces set user expectations<\/strong>.<\/p>\n\n\n\n

\n

<\/p>\n\u201cBut as with people, finding the most meaningful answer from AI involves asking the right questions. AI is neither psychic nor telepathic.\u201d<\/em>

\u2014 Stephen J. Bigelow in <\/em>5 Skills Needed to Become a Prompt Engineer<\/a><\/cite><\/blockquote>\n\n\n\n

Using magical language risks leading novices to the magical thinking that AI is omniscient. It may not be obvious that its knowledge<\/em> is limited to the training data.<\/p>\n\n\n\n