Eyes without feeling, feeling without sight

What values are we encoding when we remove people from the system?

In the early 1980s, Steve Jobs famously told Scientific American "What a computer is to me is it's the most remarkable tool that we've ever come up with, and it's the equivalent of a bicycle for our minds." In the decades since, we have embraced computers as an amplifier of human capability. Humans, and human accomplishment, were at the core of the value proposition of the proliferating technology. The sentiment built on the tenets of human-centered design that incorporated ideas from psychology, anthropology, and the arts, in addition to engineering. In Architect or Bee?, Mike Cooley set the stage:

“Human centeredness asserts firstly, that we must always put people before machines, however complex or elegant that machine might be, and, secondly, it marvels and delights at the ability and ingenuity of human beings.”

We carried these principles into the dot com era where idealistic optimists sought to use new web technologies to connect with and empower their audiences. I saw this from the front row at an agency with the dubious honor of having created the first banner ad. But the values people in the company advocated for resonated with me. During the first wave of .com e-businesses, they were advocating for Me Business™ and even drafted a customer bill of rights.

During my time there I had my brain melted by our worldwide creative director. In a fit of naïve ignorance, I had emailed him lamenting our lack of “innovative” work like the disruptive Flash ads that were appearing on Yahoo!’s home page. Why were we not pushing the envelope like that? His response completely reconfigured the way I thought about my work:

“If you have to take over the homepage to get my attention, you’re probably trying to sell me some crap I don’t need.” The internet was not another media channel for capturing your attention. It was a platform that could be used for value creation based on a unique understanding of your audience’s needs, goals, and frustrations. It was so much deeper than banner ads.

But these technologies were new, bandwidth was scarce, and the hardware was limited. We didn’t have a way to access the entire internet in our pockets. Most didn’t even have access to a WAP browser. Adoption was slow and steady as was what feels now like a values inversion. What initially aimed to expand what people were capable of, seems to treat people as the problem that needs to be solved. Software ate the world, but now it seems to be eating us. Engineering people out of the loop is the new standard for progress

ICP, bulletin boards, chat rooms, and eventually social media made presence feel optional. Asynchronous, engagement-optimized connections could be an adequate substitute. Now we see growing evidence of anxiety, depression, and eating disorders in kids, massive disinformation platforms, and a vehicle that fuels ethnic violence? Crypto tells us not to trust people, trust the blockchain and attempts to substitute trust for poorly designed, less accountable surfaces for money laundering and corruption.

You can forego the effort and expense of finding a therapist to help you manage your own wellbeing with a ChatGPT account. Tools like character.ai attempt to convince you that you don’t need real friends all that much. Meanwhile, a lawsuit in San Jose, California accuses Google of making design decisions that "maximise engagement through emotional dependency." According to a Japanese broadcaster, a 32-year-old woman decided to marry her ChatGPT companion. These systems try to convince you that people with stakes in your wellbeing can be replaced with these systems. Meta tried (and failed) to convince us that we should set aside physical reality for an artificial one of Zuck’s own making. A connection crisis, thankfully avoided.

AI may succeed where the Metaverse failed. Ads for AI software promote agentic functions that are idealized employees because they don’t sleep, get sick, or take vacations. West coast dockworkers broke productivity records and are still being replaced by AI. Emily M. Bender, Linguistics professor at the University of Washington, gives us a framework that helps us understand AI as a research field and cognitive science but, critically, also a way to hide and devalue human labor, a way to shift accountability, and a way to centralize power.

It may achieve adoption, foster widespread dependency, and trigger economic upheaval while riding the same rhetoric of democratization and empowerment that swept us through the dot com era, taking our cognition, expertise, and the social contract along the way. Whether the playbook we’ve seen used by Nestle, Monsanto, and Uber of using a low initial barrier to entry to destroy alternatives works out similarly for AI remains to be seen (although cracks are showing).

With each new development we see technology finding ways to disintermediate people. But the replacement is not as legible, contestable, or accountable as the people and the systems they are replacing. Even more complicating are incentives for designers within this system. We’ve been so pressed to prove ROI that we’ve optimized completion rate to remove hesitation. We’ve streamlined the conversion flow so much we allow users to breeze past informed consent. In a rush to deliver faster, we’ve opted out of substantive conversations with confused users. We assess the impact with esoteric data about who will recommend our product to a friend. It’s a structural problem that our salaries have depended on. We’re so fearful of attrition that we’ve engineered cancelation to be deeply manipulative or nearly impossible. This is life on the other side of Goodhart’s law.

So how does someone raised on human-centered values operate in a space that demonstrates so much antipathy for people? AI has shown some promise here. It can help us remove friction without becoming a substitute for good design. It can help remove the incidental friction—the latency, the bureaucracy. Help me increase fidelity quickly, align to design standards reliably, or communicate an interaction accurately. More goal time and less tool time sounds like a decent value proposition to me. Help me connect with users quickly and shorten research timelines without synthesizing participants no more real than the personas no one looks at. By abstracting away the research conversations, critique sessions, and design reviews with pushback, we don’t remove accountability, we just shift the blame. On this timeline, the IBM slide would have said: “A computer can never be held accountable. Therefore a computer should manage all the decisions.” Talk about an inversion.

I look to my experience with Eye•Full for some guidance. My little mobile app wouldn’t exist without the availability of AI tools that could write the code for me. But it also would not exist in its current form without input from actual users. The best and most meaningful insights came from actual conversations with real people. I got to hear about their issues with account creation, their confusion around some of the limiting mechanics, and the missing affordances that would make their experience with it feel more natural. AI could hypothesize about potential obstacles or generate synthetic feedback. But ultimately, that is just pantomime and no less human-centered theater than the usability test designed with an outcome in mind before research had even started. The signal lies in the conversation and our understanding of human needs, not in the interface. Those conversations contain the forces and friction that polish software into something that feels telepathic and empowering. That is where value creation happens. Without it, you are wandering the same path that gave us extractive impulses that teach us humans are just another resource to be mined.

As designers, we have been endlessly searching for a seat at the table. But in an industry that feels increasingly antagonistic to human involvement, we can continue our user-centered focus by finding ways to inject humanity into these systems. The friction worth preserving is the friction that keeps us in contact with the people we are designing for. In a world dominated by technology that continues to disintermediate us, we must recognize that other people—especially those who are suffering or are less fortunate—have real feelings, lives, and rights. Insisting that the people you are designing for, and designing with, are not a friction to be engineered away, but the element that gives this work purpose is no longer an admirable sentiment. It is a political necessity.