Stop Building Chat. Your Users Don't Want Another Box.

Every product I touch right now has someone on the team pushing to add an AI chat. Sales-led companies, dev tools, consumer apps, internal portals. Same pattern: a little sparkle icon in the corner, a modal that opens to a blank text box, a "How can I help?" placeholder. Ship it, call the AI feature shipped, move on.

I think this is the single biggest product engineering mistake of 2024 so far. Not because chat is bad. Because chat is what we reach for when we don't want to do the hard work of figuring out what the AI should actually do for the user.

The First-Try Bounce

Here's what I keep watching happen.

A user who has never typed anything into ChatGPT opens the new AI chat in some product. They don't know what to ask. They type something tentative and weird, sometimes literally "what can you do here?" The model gives them a generic, unhelpful answer. Or they ask the thing they actually want, phrased the way a normal human asks, and the model says "I can't help with that." Or, worst, the model confidently produces a wrong answer that the user has no way to evaluate.

In that user's head, the entire AI feature is now broken. They don't think "let me try a different phrasing." They don't think "let me read the documentation on how to prompt." They close the modal. They never open it again. They tell a coworker the AI thing in our product doesn't work.

The mental model from search does not transfer. If a search query returns nothing useful, people try a different query. The cost of iteration is low and they know how to iterate. AI chat does not feel that way to non-power users. A bad response feels like a verdict. Not "I asked the wrong question," but "this product cannot do this."

This is roughly the same shape Nielsen Norman Group documented in their accordion editing and apple picking study (published September 2023). Real users iterate constantly when given a chat surface, but the chat surface itself is hostile to iteration. They scroll, lose their place, paste prior responses back into the next prompt as crude context-passing. The interface is fighting the workflow.

We Are Not Our User

The teams shipping these chat boxes are full of people fluent in LLMs. They use Claude all day. They have ChatGPT muscle memory. They know how to phrase a prompt to get what they want, and when the model refuses they know which retry will work.

Their users do not. Jakob Nielsen, framing AI as the third major UI paradigm last summer, called out that current models advantage the verbally fluent half of the population and leave the rest behind. The product engineers building these features keep designing for themselves. The result is a feature that demos great to leadership (because leadership uses ChatGPT too) and dies on first contact with real users.

I keep thinking about how hard Google felt in 2002. A blank text box that did what, exactly? You had to learn to type keywords, not sentences. You had to learn that putting things in quotes meant something. You had to learn that there were "operators." It took a decade for "google it" to become a verb. We forgot how hard the empty box was because we grew up with it. Chat is going to take a decade too. And in the meantime, your product cannot rely on every user already being fluent.

What the Discourse Has Been Telling Us

I am not the first person making this argument. Amelia Wattenberger's Why Chatbots Are Not the Future puts it cleanly: good tools make it clear how they should be used, and more importantly, how they should not be used. The chat box affords nothing. It is a black hole that swallows whatever you type and returns text.

Linus Lee, who leads AI at Notion, made the same point on Latent Space last summer: most knowledge work is not a text-generation task. The same LLM call wrapped in an inline AI block, a slash command, or a smart autofill produces better outputs than the same call in a chat window, because the surface constrains the user toward what the model is good at.

This consensus exists. Every senior person designing AI products knows about it. And yet teams keep shipping chat boxes, because chat is the path of least resistance and "we shipped an AI feature" reads the same on a status doc whether the feature works or not.

Three Shapes That Beat Chat

What I have seen actually work on the AI product team I am on, and on a few teams I have advised:

Pre-suggested prompts that show what the model can do. Not as placeholder text in the chat box, which nobody reads. As actual buttons or cards that fire the prompt with one click. If you must ship chat, this is the cheapest improvement: a row of three to five suggested actions that match real, common intents. Users who click learn the surface area. Users who do not click at least see the surface area existed.

Inline AI inside the existing workflow. The user is already editing a document, a row in a spreadsheet, a row in a CRM, a draft email. You add an action that uses an LLM to do the next obvious thing. Summarize this. Translate this. Find duplicates. Generate three variants. The model call is hidden behind a familiar button. The user does not have to know they are prompting anything. This is how Notion AI's inline AI blocks, Linear's title suggestions, and the better versions of Gmail's Smart Compose work.

Hidden LLM calls behind a normal feature. This is the most underused shape. You wrap an LLM in a deterministic-feeling feature: a search bar that does semantic retrieval, a duplicate detector that uses embeddings, a categorizer that auto-tags. The user thinks "the search got better" or "the app figured out what this is." They never know an LLM was involved. The model is doing the work. The UI is doing the trust-building.

The unifying principle: meet the user where they already are. Do not open a new modal. Do not ask them to learn a new vocabulary. Use the AI to make the existing surface smarter.

The Power User Escape Hatch

There is a class of user who does want chat. They are fluent. They know what they want. They want to bypass the buttons and just type. Build for them, but build it as an escape hatch, not the default.

The pattern I like: surface the prompt the AI is actually running underneath your inline feature, and let power users edit it. If your "summarize this document" button works, expose the prompt behind it as an advanced view. The power user sees "oh, that is how the sausage is made," and now they can write their own. The fluent users get their text box. The other 95% of your users never see it and are not subjected to it.

This is the same instinct as keyboard shortcuts. Default to discoverable UI, expose power-user controls for the small minority who want them.

When Chat Is Actually the Right Answer

It is narrower than you think.

Chat is the right answer when the user knows exactly what they want, the task is open-ended, and the user expects to converse, not click. ChatGPT itself is a chat product because it is the AI. The chat is the value proposition. Same with Claude. Same with a customer support transcript replacement, where users are already used to talking to a human agent in a chat thread.

Chat is the wrong answer everywhere else, which is almost everywhere else. If you are putting AI into an existing product surface, your default should not be chat. Your default should be: which existing button gets smarter, which new button do users already wish existed, which background process can now run.

Where I Land

If your AI feature has a sparkle icon and opens a modal, it is probably going to die on the metrics. Not because the model is bad. Because the surface is wrong, and the surface is the product.

The teams that win this year are going to be the teams who do the boring product work: figure out the three or five intents users actually have, ship those as buttons, hide the model behind them, and save the chat box for the power users who genuinely want it. As I wrote last year, prompt engineering was never going to be a real discipline. The corollary I missed then is that prompt engineering should not be a user-facing surface either.

Ship the intelligence. Hide the prompt. Your users will thank you, even if they never know you were doing it.