From minimally invasive robotic surgery to training computers to detect breast cancer, the potential of artificial intelligence (AI) to transform healthcare is staggering. But what if the data used to develop some AI-based tools, such as those that a clinician can use to predict whether a patient continues to have a disease, is incomplete or inadequate?
Researchers at the University of Toronto’s Institute of Health Policy, Management and Assessment (IHPME) at the Dalla Lana School of Public Health asked exactly this question.
A paper from Quynh Pham and Joseph Cafazzo shows that the vast majority of research on AI-based diabetes interventions does not include or report on ethnic or racial training data – a key finding since diabetic patients from certain ethnic and racial groups are more likely to have poor outcomes.
The paper, entitled “The Need for Ethnoracial Justice in Artificial Intelligence for Diabetes Management” was recently published in the Journal of Medical Internet Research.
“Much artificial intelligence is generated by retrospective data training models, and historically these data sets represent poorly Canadians,” said Cafazzo, professor at IHPME and executive director of the Center for Global Health Care Innovation at the University Health Network (UHN).
“People associated with academic centers usually take part in research studies. We tend to collect data on them, but people who are more rural and our indigenous peoples are not asked to participate in research so their data is never collected and therefore never included in these models. “
Pham, assistant professor at IHPME and researcher at UHN, notes that there are cultural and biological factors that make it difficult for certain racially motivated communities to cope with diabetes.
About one in three Canadians has prediabetes or diabetes, which increases the risk of heart disease, stroke and kidney failure.
Pham, Cafazzo and colleagues Anissa Gamble and Jason Hearn conducted a secondary analysis of a frequently cited review article published in 2018 entitled “Artificial Intelligence for Diabetes Management and Decision Support”. The 2018 review looked at research articles on diabetes interventions with AI that include applications for: clinical decision support; Identification of adverse events; Self-management, which for example asks a person to change their lifestyle; and tools that predict the risk of developing diabetes based on genetic or lifestyle factors.
The team found that of the 141 articles included in the 2018 review, 90 percent failed to mention the ethnic and racial makeup of the datasets used to inform about AI algorithms. Only 10 of the articles in the original review reported ethnic or racial data, with the average distribution being 70 percent white, 17 percent black, and four percent Asian.
Skewed training information is problematic because of a concept called distribution drift. This means, for example, that a diabetes prediction tool developed using data from a group different from the people on whom it is used could be completely wrong.
“If you train your model on a dataset that doesn’t match the population you’re trying to apply it to, there is a massive mismatch,” says Pham, who is the first author of the paper.
“Some of these studies were 99 percent white populations – if you apply that to Markham, Mississauga or Scarborough, it obviously won’t work because of the demographics of those communities.”
The researchers recommend using representative training data sets for digital health interventions to improve accuracy and generalizability.
“I think the most important thing for research, especially in Canada, is to have more extensive prospective data sets on which to train these models,” says Cafazzo. “It brings it back to: How comprehensive do we want to be in research and be honest about our differences?”
The team has also developed a tool – a series of five questions – that researchers can use to assess how they are collecting data and the ethnic and racial relevance of the AI algorithm. The goal of this work, said Pham, is to ensure that equity is built into the design of health innovations so that all communities can benefit.
“In five years time, AI-based interventions will become the standard of care,” says Pham. “They eventually come together to the point where everyone has some degree of access to care that has been modeled to assist a clinician with diagnosis or evaluation. We want to make sure that everyone is looked after equally. “
