Artificial intelligence underpins many areas of everyday life, from chatbots that answer tax inquiries to algorithms that operate autonomous cars and provide medical diagnostics. According to experts at the University of California, Irvine, developing smarter, more accurate systems necessitates a mixed human-machine approach. They describe a novel mathematical model that may increase performance by integrating human and computational predictions and confidence ratings in a paper published this month in Proceedings of the National Academy of Sciences.
“The strengths and shortcomings of humans and computer algorithms are complimentary. Each makes predictions and judgments using distinct sources of information and tactics “Mark Steyvers, a cognitive sciences professor at UCI, is one of the study’s co-authors. “We demonstrate, via empirical demonstrations and theoretical studies, that humans may enhance AI predictions even when human accuracy is somewhat worse than the AI’s – and vice versa. And this accuracy outperforms combining estimates from two people or two AI systems.”
To put the framework to the test, researchers ran an image classification experiment in which human volunteers and computer algorithms competed to accurately identify altered images of animals and ordinary objects such as chairs, bottles, bicycles, and trucks. Human participants assessed their confidence in the correctness of each picture identification as low, medium, or high, but the machine classifier produced a continuous score. The findings revealed significant disparities in confidence between people and AI systems across photos.
“In several situations, human participants were pretty certain that a certain image had a chair, for example, whereas the AI system was perplexed by the image,” said co-author Padhraic Smyth, Chancellor’s Professor of computer science at UCI. “In other cases, the AI program was able to reliably offer a name for the thing depicted, although human participants were dubious whether the deformed picture included any recognized object.”
When the researchers’ novel Bayesian framework was used to integrate predictions and confidence ratings from both, the hybrid model outperformed either human or machine predictions alone.
“While previous research has demonstrated the benefits of combining machine predictions or combining human predictions — the so-called ‘wisdom of the crowds,'” Smyth said, “this work forges a new direction in demonstrating the potential of combining human and machine predictions, pointing to new and improved approaches to human-AI collaboration.”
The Irvine Initiative in AI, Law, and Society enabled this multidisciplinary effort. The researchers believe that the convergence of cognitive sciences, which are concerned with understanding how humans think and behave, and computer science, which is concerned with the production of technologies, will provide more insight into how humans and machines can collaborate to build more accurate artificially intelligent systems.
Heliodoro Tejada, a UCI graduate student in cognitive sciences, and Gavin Kerrigan, a UCI Ph.D. student in computer science, are also co-authors.
The National Science Foundation provided funding for this project under grant numbers 1927245 and 1900644, as well as the HPI Research Center in Machine Learning and Data Science at UCI.
