Category Archives: trust

Toyota Gets It: Self-driving cars depend more on people than on engineering

I recommend reading this interview with Toyota’s Dr. Gill Pratt in its entirety. He discusses pont-by-point the challenges of a self-driving car that we consider in human factors, but don’t hear much about in the media. For example:

  • Definitions of autonomy vary. True autonomy is far away. He gives the example of a car performing well on an interstate or in light traffic compared to driving through the center of Rome during rush hour.
  • Automation will fail. And the less it fails, the less prepared the driver is to assume control.
  • Emotionally we cannot accept autonomous cars that kill people, even if it reduces overall crash rates and saves lives in the long run.
  • It is difficult to run simulations with the autonomous cars that capture the extreme variability of the human drivers in other cars.

I’ll leave you with the last paragraph in the interview as a summary:

So to sum this thing up, I think there’s a general desire from the technical people in this field to have both the press and particularly the public better educated about what’s really going on. It’s very easy to get misunderstandings based on words like or phrases like “full autonomy.” What does full actually mean? This actually matters a lot: The idea that only the chauffeur mode of autonomy, where the car drives for you, that that’s the only way to make the car safer and to save lives, that’s just false. And it’s important to not say, “We want to save lives therefore we have to have driverless cars.” In particular, there are tremendous numbers of ways to support a human driver and to give them a kind of blunder prevention device which sits there, inactive most of the time, and every once in a while, will first warn and then, if necessary, intervene and take control. The system doesn’t need to be competent at everything all of the time. It needs to only handle the worst cases.

Wiener’s Laws

The article “The Human Factor” in Vanity Fair is two years old, but since I can’t believe I missed posting it — here it is! It’s a riveting read with details of the Air France Flight 447 accident and intelligent discussion of the impact automation has on human performance. Dr. Nadine Sarter is interviewed and I learned of a list of flight-specific “laws” developed by Dr. Earl Wiener, a past-president of HFES.

“Wiener’s Laws,” from the article and from Aviation Week:

  • Every device creates its own opportunity for human error.
  • Exotic devices create exotic problems.
  • Digital devices tune out small errors while creating opportunities for large errors.
  • Invention is the mother of necessity.
  • Some problems have no solution.
  • It takes an airplane to bring out the worst in a pilot.
  • Whenever you solve a problem, you usually create one. You can only hope that the one you created is less critical than the one you eliminated.
  • You can never be too rich or too thin (Duchess of Windsor) or too careful about what you put into a digital flight-guidance system (Wiener).
  • Complacency? Don’t worry about it.
  • In aviation, there is no problem so great or so complex that it cannot be blamed on the pilot.
  • There is no simple solution out there waiting to be discovered, so don’t waste your time searching for it.
  • If at first you don’t succeed… try a new system or a different approach.
  • In God we trust. Everything else must be brought into your scan.
  • It takes an airplane to bring out the worst in a pilot.
  • Any pilot who can be replaced by a computer should be.
  • Today’s nifty, voluntary system is tomorrow’s F.A.R.

Kudos to the author, William Langewiesche, for a well researched and well written piece.

Discussion of Human Factors on “Big Picture Science” podcast

You all know I love podcasts. One of my favorites, Big Picture Science, held an interview with Nicholas Carr (a journalist) on over-reliance in automation. The entire podcast, What the Hack, also covers computer security. To skip to the HF portion, click here.

  • +points for mentioning human factors by name
  • +points for clearly having read much of the trust in automation literature
  • -points for falling back on the “we automate because we’re lazy” claim, rather than acknowledging that the complexity of many modern systems requires automation for a human to be able to succeed. Do you want to have that flight to NY on the day you want it? Then we have to have automation to help that happen – the task has moved beyond human ability to accomplish it alone.
  • -points for the tired argument that things are different now. Google is making us dumber. Essentially the same argument that happens with every introduction of technology, including the printing press. We aren’t any different than the humans that painted caves 17,300 years ago.

For more podcasts on humans and automation, check out this recent Planet Money: The Big Red Button. You’ll never look at an elevator the same way.

*While looking up support for the claim that people have always thought their era was worse than the previous, I found this blog post. Looks like I’m not the first to have this exact thought.

App Usability Evaluations for the Mental Health Field

We’ve posted before on usability evaluations of iPads and apps for academics (e.g.,here, and here), but today I’d like to point to a blog dedicated to evaluating apps for mental health professionals.

In the newest post, Dr. Jeff Lawley discusses the usability of a DSM Reference app from Kitty CAT Psych. For those who didn’t take intro psych in college, the DSM is the Diagnostic and Statistical Manual, which classifies symptoms into disorders. It’s interesting to read an expert take on this app – he considers attributes I would not have thought of, such as whether the app retains information (privacy issues).

As Dr. Lawley notes on his “about” page, there are few apps designed for mental health professionals and even fewer evaluations of these apps. Hopefully his blog can fill that niche and inspire designers to create more mobile tools for these professionals.

Prescription Smartphone Apps

I recently published a study (conducted last year) on automation trust and dependence. In that study, we pseudo-wizard-of-oz’ed a smartphone app that would help diabetics manage their condition.

We had to fake it because there was no such app and it would be to onerous to program it (and we weren’t necessarily interested in the app, just a form of advanced, non-existent automation).

Now, that app is real.  I had nothing to do with it but there are now apps that can help diabetics manage their condition.  This NYT article discusses the complex area of healthcare apps:

Smartphone apps already fill the roles of television remotes, bike speedometers and flashlights. Soon they may also act as medical devices, helping patients monitor their heart rate or manage their diabetes, and be paid for by insurance.

The idea of medically prescribed apps excites some people in the health care industry, who see them as a starting point for even more sophisticated applications that might otherwise never be built. But first, a range of issues — around vetting, paying for and monitoring the proper use of such apps — needs to be worked out.

The focus of the article is on regulatory hurdles while our focus (in the paper) was how potential patients might accept and react to advice given by a smartphone app.

(photo: Ozier Muhammad/The New York Times)

Everyday Automation: Auto-correct

This humorous NYT article discusses the foibles of auto-correct on computers and phones. Auto-correct, a more advanced type of the old spell checker, is a type of automation. We’ve discussed automation many times on this blog.

But auto-correct is unique in that it’s probably one of the most frequent touchpoints between humans and automation.

The article nicely covers, in lay language, many of the concepts of automation:

Out of the loop syndrome:

Who’s the boss of our fingers? Cyberspace is awash with outrage. Even if hardly anyone knows exactly how it works or where it is, Autocorrect is felt to be haunting our cellphones or watching from the cloud.


We are collectively peeved. People blast Autocorrect for mangling their intentions. And they blast Autocorrect for failing to un-mangle them.

I try to type “geocentric” and discover that I have typed “egocentric”; is Autocorrect making a sort of cosmic joke? I want to address my tweeps (a made-up word, admittedly, but that’s what people do). No: I get “twerps.” Some pairings seem far apart in the lexicographical space. “Cuticles” becomes “citified.” “Catalogues” turns to “fatalities” and “Iditarod” to “radiator.” What is the logic?


One more thing to worry about: the better Autocorrect gets, the more we will come to rely on it. It’s happening already. People who yesterday unlearned arithmetic will soon forget how to spell. One by one we are outsourcing our mental functions to the global prosthetic brain.

Humorously, even anthropomorphism of automation (attributing human-like characteristics to it, even unintentially)! (my research area):

Peter Sagal, the host of NPR’s “Wait Wait … Don’t Tell Me!” complains via Twitter: “Autocorrect changed ‘Fritos’ to ‘frites.’ Autocorrect is effete. Pass it on.”

(photo credit el frijole @flickr)

Virtual Assistants (automation) and Etiquette

This NYT article discusses the “new” scourge of rude people interacting with their phones in public via voice thanks in large part to Siri, Apple’s new virtual assistant.

This article reminded me of something slightly different about human interaction with virtual assistants or automation. In a 2004 paper, researchers Parasuraman and Miller wondered if automation that possessed human-like qualities would cause people to alter their behavior.

They compared automation that made suggestions in a polite way or a rude way (always interrupting you). As you might expect, automation that was polite elicited higher ratings of trust and dependence.

This might be one reason why Siri has a playful, almost human-like personality instead of a robot servant that merely carries out your commands. The danger is that with assistants that are perceived as human-like, people will raise their expectations to unreasonable levels. Like mistakenly ascribing political motivations to it.

Lastly, the graph shown below was in the latest issue of Wired magazine.  I think it’s a nice compliment to the perceived reliability graph we showed in a previous post:

Calibrating User’s Perception of Automation

Last week I had the pleasure of presenting in a symposium on automation in safety critical domains arranged by Dr. Arathi Sethumadhavan at the American Psychological Association annual meeting.  My fellow participants were:

  • Arathi Sethumadhavan, PhD (Medtronic)
  • Poornima Madhavan, PhD (Old Dominion University)
  • Julian Sanchez, PhD (Medtronic)
  • Ericka Rovira, PhD (United States Military Academy)

Everyone presented on issues related to human-automation interaction.  I do not have their permission to show their slides so this post is more generally a lay-person’s description of one aspect of automation research:  consequences of perceptions of automation reliability.

One of the most popular types of news items we post is stories of when people rely too much on unreliable automation with sometimes funny or tragic consequences.  For example, when people use in-car navigation/GPS systems and slavishly follow its directives without judging conditions for themselves.

This is a classic example of a mis-match between the user’s perception of how reliable the system is and how it actually is.  See the figure below:

from Gempler & Wickens, 1998

The Y-axis is how the user perceives the system’s reliability while the X-axis is the actual reliability of the system.  Let’s focus on the two zones in the upper left and lower right represent.  When the user perceives that the automation is more reliable than it actually is (RED CLOUD) they will over-trust the automation and perhaps rely too much on its occasionally faulty advice (this is where much of the GPS horror stories lie).  People may get their mis-judgements about the reliability from many sources (marketing literature, limited use, or recommendations).

For example, my digital camera has an auto mode that claims to be able to detect many types of settings (macro, landscape, night) and automatically adjust settings to suit.  However, in practice it seems less reliable than the marketing literature suggests.  The company exhorts me to TRUST iA (their name for automation)!

So in a few situations where I over-rely on iA, I end up with images that are too dim/bright, etc.  The system doesn’t tell me how it came to its decision leaving me out of the loop.  Now, I just don’t use iA mode.

The other zone (YELLOW CLOUD) is less studied but it represents situations where the automation is actually very reliable but people perceive it as not very reliable and so will depend on it less–even when their performance degrades as a result.  Examples are more difficult to come up with but one might be the example of health aids that doctors might use to assist in diagnosis of patients.

Finally, the line in the middle is proper calibration: perceived reliability is perfectly correlated with the actual reliability of the automation.  This is where we want to be most of the time.  When our calibration is perfect, we will rely on the automation when we should and NOT when we shouldn’t.

Getting people to properly calibrate their trust and dependence on automation is a complex human factors psychological problem.

Coming to APA 2011: A Conversation Hour on Use of Electronic Health Records in Clinical Practice

Drs. Kelly Caine (of guest post fame)  and Dennis Morrison will be presenting on human factors considerations for the design and use of electronic health records.  Audience participation is welcome as they discuss this important topic. See abstract below.

In this conversation hour we will discuss the use of electronic health records in clinical practice. Specifically, we will focus on how, when designed using human factors methods, electronic health records may be used to support evidence based practice in clinical settings. We will begin by giving a brief overview of the current state of electronic health records in use in behavioral health settings, as well as outline the potential future uses of such records. Next, we will provide an opportunity for the audience members to ask questions, thus allowing members to guide the discussion to the issues most relevant to them. At the conclusion of the session, participants will have a broader understanding of the role of electronic health records in clinical practice as well as a deeper understanding of the specific issues they face in their practice. In addition, we hope to use this conversation hour as a starting point to generate additional discussions and collaborations on the use of electronic health records in clinical practice, potentially resulting in an agenda for future research in the area of electronic health records in clinical behavioral health practice.

Kelly Caine is the Principal Reserach Scientist in the Center for Law, Ethics, and Applied Research (CLEAR) Health Information.

Dennis Morrison is the CEO of the non-profit Centerstone Research Institute.

Check out the full Division 21 program.

Are we too trusting of GPS automation?

A GPS certainly makes life easier — and although I think many of us might consider what would happen if we were without it or it was unable to identify where we were, it is less often we consider how it may lead us astray.

One of our early postings on the Human Factors Blog was about a bus driver following GPS directions that led under a too-short bridge. His case was augmented by the fact that he had chosen the “bus” setting on the GPS and assumed any route produced was therefore safe for buses. The actual model of the GPS under the bus setting was only to add routes that only buses could take, such as HOV exits, rather than to limit any route.

NPR just posted stories of people in Death Valley who got lost from following GPS directions down roads that no longer existed. In one of the cases, their car got stuck for 5 days and resulted in the death of a child. After hearing numerous stories about inaccurate GPS directions from lost drivers, a ranger investigated the maps used by the GPS systems and found roads included in them that had been closed for years. How accurate and updated do GPS systems need to be to be considered safe? How can they address over-trust in potentially dangerous situations (e.g., death valley)?