Category Archives: Division 21

Calibrating User’s Perception of Automation

Last week I had the pleasure of presenting in a symposium on automation in safety critical domains arranged by Dr. Arathi Sethumadhavan at the American Psychological Association annual meeting.  My fellow participants were:

  • Arathi Sethumadhavan, PhD (Medtronic)
  • Poornima Madhavan, PhD (Old Dominion University)
  • Julian Sanchez, PhD (Medtronic)
  • Ericka Rovira, PhD (United States Military Academy)

Everyone presented on issues related to human-automation interaction.  I do not have their permission to show their slides so this post is more generally a lay-person’s description of one aspect of automation research:  consequences of perceptions of automation reliability.

One of the most popular types of news items we post is stories of when people rely too much on unreliable automation with sometimes funny or tragic consequences.  For example, when people use in-car navigation/GPS systems and slavishly follow its directives without judging conditions for themselves.

This is a classic example of a mis-match between the user’s perception of how reliable the system is and how it actually is.  See the figure below:

from Gempler & Wickens, 1998

The Y-axis is how the user perceives the system’s reliability while the X-axis is the actual reliability of the system.  Let’s focus on the two zones in the upper left and lower right represent.  When the user perceives that the automation is more reliable than it actually is (RED CLOUD) they will over-trust the automation and perhaps rely too much on its occasionally faulty advice (this is where much of the GPS horror stories lie).  People may get their mis-judgements about the reliability from many sources (marketing literature, limited use, or recommendations).

For example, my digital camera has an auto mode that claims to be able to detect many types of settings (macro, landscape, night) and automatically adjust settings to suit.  However, in practice it seems less reliable than the marketing literature suggests.  The company exhorts me to TRUST iA (their name for automation)!

So in a few situations where I over-rely on iA, I end up with images that are too dim/bright, etc.  The system doesn’t tell me how it came to its decision leaving me out of the loop.  Now, I just don’t use iA mode.

The other zone (YELLOW CLOUD) is less studied but it represents situations where the automation is actually very reliable but people perceive it as not very reliable and so will depend on it less–even when their performance degrades as a result.  Examples are more difficult to come up with but one might be the example of health aids that doctors might use to assist in diagnosis of patients.

Finally, the line in the middle is proper calibration: perceived reliability is perfectly correlated with the actual reliability of the automation.  This is where we want to be most of the time.  When our calibration is perfect, we will rely on the automation when we should and NOT when we shouldn’t.

Getting people to properly calibrate their trust and dependence on automation is a complex human factors psychological problem.

Coming to APA 2011: A Conversation Hour on Use of Electronic Health Records in Clinical Practice

Drs. Kelly Caine (of guest post fame)  and Dennis Morrison will be presenting on human factors considerations for the design and use of electronic health records.  Audience participation is welcome as they discuss this important topic. See abstract below.

In this conversation hour we will discuss the use of electronic health records in clinical practice. Specifically, we will focus on how, when designed using human factors methods, electronic health records may be used to support evidence based practice in clinical settings. We will begin by giving a brief overview of the current state of electronic health records in use in behavioral health settings, as well as outline the potential future uses of such records. Next, we will provide an opportunity for the audience members to ask questions, thus allowing members to guide the discussion to the issues most relevant to them. At the conclusion of the session, participants will have a broader understanding of the role of electronic health records in clinical practice as well as a deeper understanding of the specific issues they face in their practice. In addition, we hope to use this conversation hour as a starting point to generate additional discussions and collaborations on the use of electronic health records in clinical practice, potentially resulting in an agenda for future research in the area of electronic health records in clinical behavioral health practice.

Kelly Caine is the Principal Reserach Scientist in the Center for Law, Ethics, and Applied Research (CLEAR) Health Information.

Dennis Morrison is the CEO of the non-profit Centerstone Research Institute.

Check out the full Division 21 program.

Coming to APA in August: Information Foraging in the Social Web

Peter Pirolli (currently a Research Fellow at Xerox/PARC) will be presenting on Information Foraging Theory. See below for an abstract of his upcoming talk.

Information Foraging Theory is a theory of human-information interaction that aims to explain and predict how people will best shape themselves to their information environments, and how information environments can best be shaped to people.  The approach involves a kind of reverse engineering in which the analyst asks (a) what is the nature of the task and information environments, (b) why is a given system a good solution to the problem, and (c) how is that “ideal” solution realized (approximated) by mechanism.

Typically, the key steps in developing a model of information foraging involve: (a) a rational analysis of the task and information environment (often drawing on optimal foraging theory from biology) and (b) a computational production system model of the cognitive structure of task. I will briefly review work on individual information seeking,  and then focus on how this work is being expanded to studies of information production and sense-making in technology-mediated social systems such as wikis, social tagging, social network sites, and twitter.

In recent years, we have been extending our studies to deal with social interactions on the Web (e.g., wikis, tagging systems, twitter). This has lead to studies of how people assess source credibility (expertise, trustworthiness, bias ) and how user interfaces might affect such judgments.

Check out the full Division 21 program.

Hope to see you at APA 2011!

There will be an extensive program for the Applied Experimental Division of the American Psychological Association at their conference in D.C. from August 4-7, and I invite all to come! We’ve collected a who’s-who from established to up-and-coming researchers to bring you cutting edge work in human factors, ergonomics, automation, human machine systems, aviation, video gaming, and much more!

Below is a listing of the program. You can find the dates and times on the APA website by selecting Division 21 from the combo box.

Special Talks

1. Presidential Address by Pat DeLucia, Texas Tech University – “Engineering Psychology for Technology”

2. Invited Address by John W. Senders, University of Toronto
Winner of the Franklin V. Taylor Award for outstanding contribution to the field of Applied Experimental and Engineering Psychology – You can read about the talk Dr. Senders gave at APA 2010.

3. Invited Address by new APA Fellow Peter Pirolli, Palo Alto Research Center, CA
“Information Foraging in the Social Web”

4. Address by the winner of the Alluisi Early Career award – Bruce Walker, Georgia Institute of Technology
“Sonification and Auditory Displays for Assistive Technology: Science and Service”

5. Address by the winner of the Briggs Dissertation award – Michael A. Nees, Georgia Institute of Technology
“Flexibility of Representation in Working Memory for Nonspeech Sounds: Theoretical and Practical Implications”

Paper session on the Psychology of Aviation

1. Durso, F. T., Pop, V., Stearman, E. J., & Kazi, S. – “How to Create a Vigilance Decrement in NextGen Air Traffic Control”

2. King, R., Carretta, T. R., & Chappelle, W. – “Ab Initio Selection of Remotely Piloted Aircraft Operators”

3. Krajewski, J., Schnieder, S., Sommer, D., & Golz, M. – “Estimating Fatigue From Simulated Air Traffic Controller Communication”

4. Bleckley, K., & Broach, D. – “Selecting Air Navigation Service Providers for the Next Generation of Air Traffic Control”

5. Gorman, J. C. – “Team Coordination Dynamics in Uninhabited Air Vehicle Operations”

6. Nelson, E., Baker, K., Gee, S., Boehm-Davis, D. A. – “The Party Line: Who’s Listening?”

Paper session focusing on Behavioral Research at George Mason University

1. Baldwin, C. L. – “Design and Examination of In-Vehicle Auditory Collision Warning Messages”

2. Peterson, M. S. – “Guiding Attention”

3. Parasuraman, R. – “Neurogenetics of Individual Differences in Working Memory and Decision Making: Implications for Selection and Training”

4. Thompson, J. – “Recognizing Human Movement”

5. Shaw, T. – “Group-Level and Individual Differences Approaches to the Study of Sustained Attention”

6. Boehm-Davis, D. A. – “Do Pilots Really Need the Party Line?”

Conversation Hour on the Symbiosis of Basic and Applied Research


Doug Gillan, North Carolina State University – “In Psychology, Applied Research = Basic Research = Applied Research = ….”

Tim Nichols, Microsoft Game Studios, Redmond, WA –  “New Technologies and New Modes of Interaction: Collaboration Opportunities for Industry and Academic Researchers”

Invited Address on Current NHTSA Human Factors Research Priorities

Tim Johnson, Department of Transportation, Washington, DC

Symposium session with Games to Explain Human Factors – Come, Participate, Learn, and Have Fun

Chair – Ron G. Shapiro, Providence, RI

Conversation Hour on Electronic Health Records in Clinical Practice


Kelly E. Caine, Indiana University Bloomington

Dennis Morrison, Centerstone Research Institute, Bloomington, IN

Symposium on Human-Automation Interaction in Safety Critical Domains

1. Arathi Sethumadhavan, Medtronic, Inc., Mounds View, MN
2. Julian Sanchez, Medtronic, Inc., Mounds View, MN
3. Poornima Madhavan, Old Dominion University
4. Ericka Rovira, United States Military Academy
5. Richard Pak, Clemson University

Paper Session – Human Factors Potpourri

1. Brown, C. M., Kobus, D. A. – “Changes in Cognitive Performance During Extended Periods of Heavy-Load Carriage”

2. Smarr, C., Serrano-Baquero, D., Cullen, R. H., McBride, S. E., Beer, J. M., & Rogers, W. A. – “Using Knowledge Engineering to Understand Communication Processes in the Mowing of Citrus Groves”

3. Beer, J. M., McBride, S. E., Mitzner, T. L., Springman, J. M., & Rogers, W. A. – “Challenges in Patient Education: Teaching Older Adults Home Health Care Tasks”

4. McBride, S. E., Tsai, W., Knott, C. C., & Rogers, W. A. – “Understanding User Needs for an Osteoarthritis Management Tool”

5. Lane, S., Peterson, J. V., Taylor, E. J., Jackson, J. A. – “Understanding Factors Leading to Computer Loyalty”

Conversation Hour on the Activities of the Board on Human-Systems Integration in the National Academy of Sciences

Barbara Wanchisen, National Academies, Washington, DC
“Human-Systems Integration at The National Academies: Health, Military, and Aviation Applications”


1. Billings, D. R., Oleson, K. E., Chen, J. Y. C., & Hancock, P. A. – “Mitigating Inappropriate Trust in Human-Robot Interactions: A Review of Trust Calibration Strategies in the Literature”

2. Morrow, J., Adler, M. C., Schleicher-Dilks, S., Oliver, T., Andrews, A. P., & Golden, C. J. – “Neuropsychological and Demographic Correlates to Impairment in Simulated Driving”

3. Dan, C. S. – “Effects of Time of Automation Failures and Expectation on Trust in Automation”

Automation Issues Hit the Big Time on NPR

NPR brings home the safety issues of too much cockpit automation.

From the NPR story:

“It was a fairly busy time of the day. A lot of other airliners were arriving at the same time, which means that air traffic control needed each of us on very specific routes at specific altitudes, very specific air speeds, in order to maintain this smooth flow of traffic,” he says.

So, air traffic control told him to fly a particular course. He and the other pilot flying the jet set the flight automation system to do it.

“What I anticipated the aircraft to do was to continue this descent,” he says. “Well instead, the aircraft immediately pitched up, very abruptly and much to my surprise. And both of us reached for the yoke, going, ‘What’s it doing?’ and there’s that shock of, ‘Why did it do that, what’s it going to do next?’ “

We’ve posted on this topic before, when we discussed Dr. Sethmudnaven’s work and Dr. Sanchez’s work. For more cutting-edge automation failure research, watch these labs:

If your lab should be listed and isn’t, send me an email!

Profiles in Human Factors: Dr. Ron Shapiro

This post is from our series of human factors career profiles. Check them all out if you’re curious about what kinds of careers you can have in this field!

Dr. Ron Shapiro received his Ph.D. from Ohio State University. He has had a long career in human factors, including being a visiting Assistant Professor at Denison University, consulting for three years with Dunlap and Associates, and then spending 23 years at IBM in their Large Systems Group, Software Group, and in Corporate Learning and Human Resources. He has taught as an Adjunct Assistant Professor at the University of Conneticut, Dutchess Community College, and at Marist College. He is currently an independent consultant in Human Factors and Human Resources.

Anne: Hi Ron, would you briefly describe your job and what you enjoy most about it?

Ron: Right now, my favorite activities are introducing Human Factors/Ergonomics to students, faculty and to organizations and helping to grow our profession. This includes consulting on career planning for students and offering recommendations on how to solve problems.  An advantage of being on my own is that I get to do work that I want to do.

Anne: That’s a nice advantage. So, how did you get interested in Human Factors as a career?

Ron: As an undergraduate I was interested in people (psychology) and computers/information processing.  A graduate associate recommended that I look into Cognitive Psychology, which I did.

While I wanted an academic appointment when I graduated, they were few and far between for Cognitive Psychology.  A number of the graduate students at Ohio State were taking applied jobs in Human Factors, so I decided to learn more about HF. One very valuable discussion which I had, that actually became a turning point in my career, was with Tom Eggemeier at the University of Dayton.  As I learned more about HF from Tom and others, I found that I was very much in demand in the applied world (after a year of getting mostly academic rejections I received numerous job offers without even filling out applications!!!) Indeed, I was not prepared for this level of success, and as I think back about it I probably could have managed the success better.

Anne: It sounds like you have gotten to do a number of different things in your career.  What skills do you need the most for your current job?

Ron: Listening to people and drawing on the human factors literature, experiences which colleagues, many of whom I have met through HFES and APA Division 21, have shared with me as well as my own personal experiences to propose solutions to problems.

Anne: Could you share an example of how you’ve seen HF make a difference in the world?

Ron: Actually, the example I’ll give is of something I have not seen.   Neither has anyone else, but I can certainly imagine it: The number of accidents/injuries/deaths which have been prevented through HF Design.  I think about it whenever I’m going to do something significant like ride in an airplane.

Anne: If you could tell an undergraduate psychology major about opportunities in human factors, what would you say?

Ron: Actually, I do this very frequently both formally and informally. My next formal address on this will be at the Eastern Psychological Association (EPA) Conference in Boston in March.

First, I highly recommend the career for individuals interested in both people and technology. The advantages are that it is interesting work, there are very good opportunities for internships and jobs, and they are high paying. Now, the disadvantages are that you will need to live where the jobs are and you won’t necessarily know whose life you saved.

Anne: Good summary! Undergraduates frequently tell me that they want to help people, but they imagine that as a 1 on 1 job, rather than creating systems and products that help a large number of people, as you describe. I saw from the HFES site that you have been on a number of panels on that very topic.

Ron: Yes. I agree with you. I have also found that students do not think of careers in prevention as a means of helping people — probably more people than they could ever help in a treatment role, but anonymously. I began to work in the career development area for HF students and professionals years ago when I observed that students needed guidance from professionals with experience working in government and industry. In 1996 Tony Andre, who was also doing significant work in this area, and I decided to work together on HFES Career panels. We traditionally offered these on Tuesday afternoon at the HFES meeting immediately before the student reception, but last year we decided to team with Sandra Garrett and move these to Student Career Monday. By coincidence, this year Tony is the President of HFES and I’m the Secretary-Treasurer, so Tony and I have the opportunity to work together in a new capacity. Tony is currently on a two-year leave from the Career panel organization to serve as HFES President.

I also participate in the Student-Professional lunches at HFES which are organized by Haydee Cuevas. I would encourage blog readers to attend the career panels and to participate in the lunches. I would also like to acknowledge Bill Moroney’s work in providing analysis and interpretation of data on “where the jobs are” to help shape educational programs and to help students prepare for their careers. (Since you are in North Carolina I might add parenthetically that my career development work expanded beyond the HF profession. One of my management jobs at IBM was managing Career Services for IBM North Carolina Employees.)

Anne: I know you are heavily involved in organizations like APA, especially Division 21, and HFES. Can you tell me why that is a priority for you?

Ron: First, our success as individuals and as a profession is in part highly dependent upon our developing a market for our services and developing future as well as current members of the profession is critical to our growth and survival. I believe that in order to be a profession we need to communicate with each other both personally and technically… transcending corporate boundaries for our entire career… not just until we graduate from school. Professional societies are critical to doing all of the above.  While the internet is useful, without an organized structure its utility is limited.

Anne: Many of us have heard about your Games to Explain Human Factors: Come, Participate, Learn & Have Fun!!! Outreach Program. How can we learn more about it?

Ron: You might check out the Games website. The 168 page program is available free to HFES and APA Division 21 members and teachers upon request.

Anne: And finally, how can one arrange to have you speak or consult?

Ron: Just send me an email: DrRonShapiro1981 at SigmaXi.Net or call. I’d be pleased to work with you.

Reminder: Proposals due for APA 2011 on Dec. 1st!

APA Division 21 (Applied Experimental and Engineering Psychology) invites submissions for the 2011 Convention of the American Psychological Association, to be held in Washington, D.C. August 4-7, 2011. Proposals for papers, posters, or symposia in areas related to applied experimental/engineering psychology, or human factors/ergonomics, are encouraged.

Broad topics of interest include, but are not limited to:

  • Issues in automation across domains
  • The contribution of applied research to fundamental knowledge and theory
  • Advanced student work
  • International perspectives
  • Use and effectiveness of technology in clinical/healthcare settings

Individuals must submit proposals via the submission portal provided below:

***Submissions are due DECEMBER 1, 2010.***

For more information about Division 21, please see the Division 21 website:

If you are not already a member, join or take advantage of the free trial membership in the division!

Full instructions concerning submissions, submission length, etc. may be found on APA’s site:

Blogging APA Division 21: “One Thing at a Time” (but over a really long time)

I held off for a while writing this post because I wanted to make sure I could include media Dr. John Senders included in his talk. I think you’ll agree it was worth the wait!

At the 2010 APA convention, John W. Senders, Ph.D. presented “One Thing at a Time: From Eye Fixations (1951), to Sampling (1954), to Information Theory (1955), to Workload (1959), to Queuing Theory (1964), to Attentional Demand (1966), Followed by a Lapse of 40 Years.” Video of the talk will eventually be posted on the Division 21 website.

Dr. Senders mentioned an eye tracking experiment from the 1950s, before any “eye trackers” existed. The method was to film the eyes of pilots as they scanned each instrument in a cockpit according to instruction. The position of the pilots eyes were coded in the close-up video. For example, a pilot might be told to “look at the altimeter,” and then the exact position of the pilot’s eyes was coded as “looking at altimeter.” Then, when the pilots were using their instruments naturally, a video of this use could be coded by eye position to know exactly when and where they were looking at any moment.

In another ingenious methodological development, Senders created a vision sampling device. The video below illustrates how it worked – a mechanical visor rose and fell in front of the driver’s eyes.

Dr. Senders came up with this idea of sampling while driving through a heavy rainstorm at different speeds, while the speed of the windshield wipers stayed the same. The visor in the video does the same — the rate of viewing can be controlled and the attentional demands of the driving task measured.

For more information, see the CogWorks website.

I’ve already covered these techniques in my classes as they are great demonstrations of creativity in research methods. If one has a well defined purpose and goal, a tool can often be created from surprising materials. Another example I often cite is the actual picture of Thorndike’s puzzle box — a splintered and rickety contraption that bears little resemblance to the finely drawn illustrations in intro psychology textbooks.

Call for Papers! APA 2011 in Washington D.C.

APA Division 21 (Applied Experimental and Engineering Psychology) invites submissions for the 2011 Convention of the American Psychological Association, to be held in Washington, D.C. August 4-7, 2011. Proposals for papers, posters, or symposia in areas related to applied experimental/engineering psychology, or human factors/ergonomics, are encouraged.

Broad topics of interest include, but are not limited to:

  • Issues in automation across domains
  • The contribution of applied research to fundamental knowledge and theory
  • Advanced student work
  • International perspectives
  • Use and effectiveness of technology in clinical/healthcare settings

Individuals must submit proposals via the submission portal provided below:

***Submissions are due DECEMBER 1, 2010.***

For more information about Division 21, please see the Division 21 website:

If you are not already a member, join or take advantage of the free trial membership in the division!

Full instructions concerning submissions, submission length, etc. may be found on APA’s site:

Photo credit ehpien

Blogging APA Division 21: The Cost of Automation Failure

Arathi Sethumadhavan, currently of Medtronic and recently of Texas Tech, was this year’s winner of the George E. Briggs dissertation award, for the best dissertation this year in the field of applied experimental psychology. Her advisor was Frank Durso.

Her work was inspired by our need to increase automation in aviation, due to increases in air traffic. However, automation does not come without costs — what happens the performance of air traffic controllers and pilots when the automation someday fails? At what point is the operator so “out of the loop” that recovery is impossible?

Sethumadhavan addressed this question by giving people different levels of automation and observing their performance after failures of the automated system. The more automated the system, the more errors occurred when that system failed.

She also measured the situation awareness of her participants in the different levels of automation — results were similar. Those who had more of their task automated had less situation awareness, and even after a system failure their awareness continued to be lower. In other words, they weren’t shocked out of complacency, as one might predict.

Sethumadhaven’s work directly contributes to understanding the human in the loop of the automated system, so that we can predict their behavior and explore design options to prevent errors due to putting the controller out of the loop.

You can read more on Dr. Sethumadhavan’s work here. Congratulations to her on this award!

Photo credit isafmedia under a Creative Commons license.