Abstract

Aim: To compare the quality, readability, and emergency referral performance of four major language models in answering post-septorhinoplasty patient questions in Turkish and English.

Materials and Methods: A cross-sectional evaluation study comprised 31 questions: 20 routine postoperative and 11 emergency/red-flag scenarios. Each question was submitted once to ChatGPT, Google Gemini, DeepSeek, and Claude in Turkish and English within independent conversation windows. Two otolaryngologists independently evaluated the responses using the Quality Analysis of Medical Artificial Intelligence tool. English responses were additionally assessed using Flesch Reading Ease, Flesch-Kincaid Grade Level, and Simple Measure of Gobbledygook indices. Emergency referral accuracy was evaluated separately.

Results: Total quality scores differed significantly among models for Turkish routine postoperative questions (p<0.001, Kendall’s W=0.775) and for Turkish emergency/red-flag questions (p=0.009, W=0.350), whereas the corresponding overall differences for English questions were not statistically significant. Within-model language comparisons showed significant Turkish-English differences for routine questions across all four models, whereas no language-related differences were observed for emergency/red-flag questions. ChatGPT, DeepSeek, and Claude provided appropriate emergency referrals in all 22 responses, whereas Gemini did so in 16 of 22 responses (72.7%). All models exceeded recommended readability thresholds for patient education materials.

Conclusion: The performance of large language models in post-septorhinoplasty patient education varied by language context and question category. Although emergency referral performance was generally high, model-specific omissions occurred, and all models exceeded recommended readability thresholds. These findings support language-specific evaluation, simplification to improve readability, and expert review before clinical use.

Keywords: artificial intelligence, large language models, patient education, rhinoplasty, septorhinoplasty, postoperative care

Copyright and license

Copyright © 2026 The Author(s). This is an open-access article published by Bolu İzzet Baysal Training and Research Hospital under the terms of the Creative Commons Attribution License (CC BY) which permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited.

How to cite

1.
Küçüktağ Z, Özdoğan A, Uysal GS, Uluhan Y. Comparative evaluation of four large language models for postoperative patient education after septorhinoplasty: A bilingual cross-sectional study. Northwestern Med J. 2026;6(4):317-29. https://doi.org/10.54307/NWMJ.2026.262

References

  1. Alsudays AM, Almanea KA, Alhajlah AA, Alroqi A. Artificial intelligence in postoperative sinus care: a comparative study of ChatGPT-4, Google Gemini, and DeepSeek in patient education and support. J Craniofac Surg. 2026; 37(3-4): 490-3. https://doi.org/10.1097/SCS.0000000000011922
  2. Aliyeva A, Alaskarov E, Sari E. Postoperative management of tympanoplasty with ChatGPT-4.0. J Int Adv Otol. 2025; 21(1): 1-6. https://doi.org/10.5152/iao.2025.241797
  3. Ayoub NF, Lee YJ, Grimm D, Balakrishnan K. Comparison between ChatGPT and Google Search as sources of postoperative patient instructions. JAMA Otolaryngol Head Neck Surg. 2023; 149(6): 556-8. https://doi.org/10.1001/jamaoto.2023.0704
  4. Crowson MG, Nwosu OI. The integration and impact of artificial intelligence in otolaryngology-head and neck surgery: navigating the last mile. Otolaryngol Clin North Am. 2024; 57(5): 887-95. https://doi.org/10.1016/j.otc.2024.04.001
  5. Alghamdi AM, Alzahrani RS, Alghamdi FS, Albilasi TM, Doubi AO. The role of artificial intelligence in assisting patient education post-rhinoplasty. International Journal of Medicine in Developing Countries. 2024; 8(10): 2794-8. https://doi.org/10.24911/IJMDC.51-1730291500
  6. Ibas M, Dursun S, Paksoy M, Ocal R, Karatas E. Accuracy and safety of ChatGPT-4o responses in rhinoplasty postoperative counseling: a panel-based study. Acta Otolaryngol. 2025; 145(9): 851-6. https://doi.org/10.1080/00016489.2025.2541612
  7. Capelleras M, Soto-Galindo GA, Cruellas M, Apaydin F. ChatGPT and rhinoplasty recovery: an exploration of AI’s role in postoperative guidance. Facial Plast Surg. 2024; 40(5): 628-31. https://doi.org/10.1055/a-2219-4901
  8. Xie Y, Seth I, Hunter-Smith DJ, Rozen WM, Ross R, Lee M. Aesthetic surgery advice and counseling from artificial intelligence: a rhinoplasty consultation with ChatGPT. Aesthetic Plast Surg. 2023; 47(5): 1985-93. https://doi.org/10.1007/s00266-023-03338-7
  9. Sallam M, Alasfoor IM, Khalid SW, et al. Chinese generative AI models (DeepSeek and Qwen) rival ChatGPT-4 in ophthalmology queries with excellent performance in Arabic and English. Narra J. 2025; 5(1): e2371. https://doi.org/10.52225/narra.v5i1.2371
  10. Sari F, Çelik Z, Mirza Y. ChatGPT-4 vs. DeepSeek-V3: a comparative study of response quality, reliability, usefulness, and readability for exercise and rehabilitation strategies in patients with ankylosing spondylitis. Clin Rheumatol. 2026; 45(1): 187-95. https://doi.org/10.1007/s10067-025-07789-y
  11. Mayo-Yáñez M, Rameau A, Vaira LA, et al. Patient perceptions of artificial intelligence in otolaryngology-head and neck surgery: an international study. Ear Nose Throat J. 2025; undefined: 1455613251351774. https://doi.org/10.1177/01455613251351774
  12. Meyer MKR, Kandathil CK, Davis SJ, et al. Evaluation of rhinoplasty information from ChatGPT, Gemini, and Claude for readability and accuracy. Aesthetic Plast Surg. 2025; 49(7): 1868-73. https://doi.org/10.1007/s00266-024-04343-0
  13. Palmer WJ, Michlin D, Estephan L, et al. Utility and safety of artificial intelligence for patient-initiated contact after functional rhinoplasty. OTO Open. 2025; 9(4): e70170. https://doi.org/10.1002/oto2.70170
  14. CHART Collaborative. Reporting guidelines for chatbot health advice studies: explanation and elaboration for the Chatbot Assessment Reporting Tool (CHART). BMJ. 2025; 390: e083305. https://doi.org/10.1136/bmj-2024-083305
  15. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007; 370(9596): 1453-7. https://doi.org/10.1016/S0140-6736(07)61602-X
  16. Vaira LA, Lechien JR, Abbate V, et al. Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms. Eur Arch Otorhinolaryngol. 2024; 281(11): 6123-31. https://doi.org/10.1007/s00405-024-08710-0
  17. McLaughlin GH. SMOG grading - a new readability formula. The Journal of Reading. 1969; 12(8): 639-46.
  18. Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006; 33(3): 352-73. https://doi.org/10.1177/1090198105277329
  19. Dhar S, Kothari D, Vasquez M, et al. The utility and accuracy of ChatGPT in providing post-operative instructions following tonsillectomy: a pilot study. Int J Pediatr Otorhinolaryngol. 2024; 179: 111901. https://doi.org/10.1016/j.ijporl.2024.111901
  20. Chen FJ, Nightingale J, You WS, Anderson D, Morrissey D. Assessment of ChatGPT vs. Bard vs. guidelines in the artificial intelligence (AI) preclinical management of otorhinolaryngological (ENT) emergencies. Aust J Otolaryngol. 2024; 7: 19. https://doi.org/10.21037/ajo-24-1