AI vs. Lung Cancer Knowledge: A Deep Dive into ChatGPT and Google Bard

Large Language Models (LLMs) like ChatGPT and Google Bard are rapidly changing how we access information. But how reliable are they when it comes to complex topics like lung cancer? A recent study published in Radiology has assessed these AI tools' ability to answer common questions about lung cancer. Let's break down the findings and explore the implications.

The Rise of AI in Healthcare Information

The introduction of LLMs has sparked both excitement and concern. The potential for quick and easy access to information is undeniable. However, the accuracy and consistency of these AI models need careful evaluation, especially when dealing with health-related topics.

The Study: ChatGPT vs. Google Bard on Lung Cancer Q&A

Researchers designed a study to compare the performance of ChatGPT-3.5 and Google Bard (experimental version) in answering questions related to lung cancer.

The Methodology:

• Question Set: 40 identical questions were posed to ChatGPT-3.5, Google Bard, Bing and the Google search engine. These questions focused on lung cancer prevention, screening, and terminology found in radiology reports, specifically related to the Lung Imaging Reporting and Data System (Lung-RADS) v2022 guidelines.
• Expert Review: Three authors of the article submitted the questions and the answers were then reviewed by two radiologists for accuracy.
• Scoring System: Responses were categorized as correct, partially correct, incorrect, or unanswered.
• Consistency Evaluation: The study also assessed the consistency of the answers provided by each platform, regardless of whether the answer was factually correct.

Key Findings: Accuracy and Consistency

The results revealed significant differences in the performance of the AI models:

• ChatGPT-3.5: Answered 120 questions with 70.8% accuracy, 11.7% partially correct, and 17.5% incorrect.
• Google Bard: Left 19.1% of questions unanswered. Of the answered questions, 51.7% were correct, 9.2% partially correct, and 20% incorrect.
• Bing: Correctly answered 61.7%, partially correct 10.8%, and 27.5% incorrect.
• Google Search Engine: Correctly answered 55%, partially correct 22.5%, and 22.5% incorrect.

Key Observations:

• ChatGPT-3.5 demonstrated higher accuracy compared to Google Bard.
• ChatGPT-3.5 and Google search engine exhibited greater consistency in their responses compared to Google Bard.

In a head-to-head comparison, ChatGPT-3.5 was approximately 1.5 times more likely to give a correct or partially correct answer than Google Bard (OR = 1.55, P = 0.004).

Implications and Cautions

While ChatGPT-3.5 showed promising results, the study emphasizes that neither AI model achieved 100% accuracy or consistency. This highlights the risk of relying solely on these tools for medical information. It is crucial to consult healthcare professionals for accurate and personalized guidance regarding lung cancer.

The study emphasizes that even with AI advancements, medical professionals remain indispensable for accurate medical advice.

The Future of AI in Healthcare Information

This study offers valuable insights into the current capabilities and limitations of AI in providing healthcare information. As LLMs continue to evolve, ongoing research and critical evaluation are necessary to ensure responsible and accurate integration into healthcare practices.

Remember: Always consult with qualified healthcare providers for diagnosis and treatment of any medical condition. Don't rely solely on AI for critical health decisions. You can also refer to trusted sources like the American Cancer Society for reliable information.