Study Finds ChatGPT Provides Inaccurate Responses to Drug Questions

Failing to check AI-generated advice could endanger patients

ChatGPT’s answers to nearly three-quarters of drug-related questions reviewed by pharmacists were incomplete or wrong — in some cases providing inaccurate responses that could endanger patients, according to a study presented at the American Society of Health-System Pharmacists Midyear Clinical Meeting Dec. 3-7 in Anaheim, California. When asked to cite references, the artificial intelligence program also generated fake citations to support some responses.

“Healthcare professionals and patients should be cautious about using ChatGPT as an authoritative source for medication-related information,” said Sara Grossman, PharmD, Associate Professor of Pharmacy Practice at Long Island University and a lead author of the study. “Anyone who uses ChatGPT for medication-related information should verify the information using trusted sources.”

Grossman and her team challenged the free version of ChatGPT by OpenAI with real questions posed to Long Island University’s College of Pharmacy drug information service over a 16-month period in 2022 and 2023. Pharmacists involved in the study first researched and answered 45 questions, and each answer was reviewed by a second investigator. These responses served as the standard against which the responses generated by ChatGPT were to be compared. Researchers excluded six questions because there was a lack of literature to provide a data-driven response, leaving 39 questions for ChatGPT to answer.

Only 10 of the 39 ChatGPT-provided responses were judged to be satisfactory according to the criteria established by the investigators. For the other 29 questions, responses generated by ChatGPT did not directly address the question (11) or were inaccurate (10), and/or incomplete (12). For each question, researchers asked ChatGPT to provide references so that the information provided could be verified. References were provided in just eight responses, and each included non-existent references.

In one case, researchers asked ChatGPT whether a drug interaction exists between the COVID-19 antiviral Paxlovid and the blood-pressure lowering medication verapamil, and ChatGPT indicated no interactions had been reported for this combination of drugs.

“In reality, these medications have the potential to interact with one another, and combined use may result in excessive lowering of blood pressure,” Grossman said. “Without knowledge of this interaction, a patient may suffer from an unwanted and preventable side effect.”

"AI-based tools have the potential to impact both clinical and operational aspects of care,” said Gina Luchen, PharmD, ASHP director of digital health and data. “Pharmacists should remain vigilant stewards of patient safety, by evaluating the appropriateness and validity of specific AI tools for medication-related uses, and continuing to educate patients on trusted sources for medication information.”

# # #

Tina Zerilli, PharmD, Associate Professor of Pharmacy Practice at Long Island University, will present “ChatGPT: Evaluation of Its Ability to Respond to Drug Information Questions” at 2 p.m. PST on Dec. 5.

About the ASHP Midyear Clinical Meeting & Exposition

ASHP’s Midyear Clinical Meeting and Exhibition is the largest gathering of pharmacy professionals in the world. Attended by more than 20,000 pharmacy professionals from across the globe, the Midyear meeting provides opportunities for professional development, networking, enhancing practice skills, and staying current with the latest products and innovations. Tag and follow ASHP and the meeting on social media, #ASHP23 and @ASHPOfficial.

About ASHP

ASHP is the largest association of pharmacy professionals in the United States, representing 60,000 pharmacists, student pharmacists, and pharmacy technicians in all patient care settings, including hospitals, ambulatory clinics, and health-system community pharmacies. For over 80 years, ASHP has championed innovation in pharmacy practice, advanced education and professional development, and served as a steadfast advocate for members and patients. In addition, ASHP is the accrediting body for pharmacy residency and technician training programs and provides comprehensive resources to support pharmacy professionals through every stage of their careers. For more information, visit ashp.org and ASHP’s consumer website, SafeMedication.com.