What are the pros and cons of using AI chatbots in musculoskeletal rehabilitation? What are chatbot applications in education, clinical practice, and research?
How does the chatbot advice compare to the recommendations of trusted clinical practice guidelines? Dr Giacomo Rossettini shares the results of 2 new papers from his research team to answer these questions.
This episode is about the best ways clinicians can think about using AI chatbots to advance their clinical practice, while avoiding the pitfalls.
------------------------------
RESOURCES
Pros and cons of AI chatbots in musculoskeletal rehabilitation practice: https://www.jospt.org/doi/10.2519/jospt.2023.12000
ChatGPT vs. CPGs for managing lumbosacral radicular pain: https://www.jospt.org/doi/10.2519/jospt.2024.12151
[00:00:00] Hello and welcome to JOSPT Insights, the podcast that aims to help you translate quality research to quality practice. I'm Claire Ardern, the editor-in-chief of the Journal of Orthopaedic and Sports Physical Therapy. It's great to have you listening today.
[00:00:23] So AI is a constantly changing, exciting at times, disconcerting technological development that seems to bring with it massive potential as well as profound pitfalls in the world of healthcare. Today we're sitting down with Dr Giacomo Rossettini to talk about the pros,
[00:00:37] the cons of using AI within physical therapy, how well it matches up with our CPGs and how we shouldn't should not be using it within our practice. Dr Giacomo Rossettini is a physiotherapist and researcher from Italy. He completed his
[00:00:49] PhD in Neuroscience in 2018 and has been a treating physiotherapist since 2009. He currently works at the Department of Neuroscience, Rehabilitation, Ophthalmology, Genetics and Maternal Child Sciences within the University of Genoa, where he performs research related to orthopedic surgery, physiotherapy, sports medicine and allied health science.
[00:01:09] My name is Dan Chatman. I'm a U.S.-based physical therapist and owner of Chatman PT in Baltimore, Maryland. And I'm Chelsea Kuhnman, a physical therapist and athletic trainer at Stanford University Athletics. Dr Giacomo Rossettini, thank you so much for joining us today on JOS PT Insights.
[00:01:24] We cannot wait to get into AI chatbots with you. Thanks a lot for the invitation. I'm really happy to be here with you to talk about this very important topic. So you are part of two papers. We're going to talk about both of them today. But the
[00:01:41] big one is that we want to base everything on is the viewpoint that you and your colleagues wrote in the December 2023 issue of JOS PT, the pros and cons of using artificial intelligence chatbots for musculoskeletal rehabilitation management.
[00:01:58] So this is, I mean, at this point it's kind of not new, but it's still kind of fresh. And I'm sure there are rehab clinicians trying to figure out the best way to use this in their clinical practice and in their research and in their education.
[00:02:11] I think that it is important to start before to say that artificial intelligence chatbots are the specific formal, large language models capable to understand, interpret and generate the text on the basis of specific prompts that we offer.
[00:02:31] It is really nice because through this type of tools, we can create a text that is very similar to the human. Another important element is that they are very, very simple to be used. So very user-friendly.
[00:02:49] And for that reason, I think that they have produced a very important hype towards the scientific and clinician community. So we can use this type of chatbot in different fields. When we think about the education, they can help us to help the students to provide
[00:03:10] some tutorials for a specific topic or act as a virtual assistance in order to help the students to achieve specific knowledge in a specific field. And also within the educational fields, there are some problems because there is the risk to miss some meta-competence.
[00:03:30] The students cannot achieve some specific knowledge that will help them in clinical practice when they will be future clinicians of course. So I think that there are some pros, but also we need to be very aware about the possible limitation and issues that can produce.
[00:03:51] So then what about in clinical practice? I think also in clinical practice, we have some pros and some elements of limitation. Of course, chatbot can help the clinicians to think about the specific patient's presentation.
[00:04:07] For example, inform about which kind of symptoms there is a need to be investigated, such as pain, disability or numbness or whatever we want. Or they can help us to understand and inform our clinical reasoning about which kind of test we need to perform.
[00:04:27] So for example, provocation tests, mobility tests or neurotransmitters tests. So they can help us. They can guide us. But also I think that they can accelerate some administrative elements, some administrative work load activities that we need to do every day in our clinical sector.
[00:04:48] They can help also to tailor specific communication with our patients. We know every day how difficult it is to create and use a specific language to engage the patients within the clinical pathway and the physical therapy plan.
[00:05:06] I think that they can also help us to do a step forward in this direction. There are some problems up to now. There are no specific guidelines about the use of this chatbot in clinical practice. We don't know nothing about the strategy that this chatbot adopted
[00:05:25] to store and to manage the specific information of the patient. So there could be some ethical and also legal problems that we need to know. Another important concern is that up to now the clinicians are not trained about the right use.
[00:05:44] From my personal perspective, I think that they are only tools that we can use but we need to be aware that they are a strong limitation up to now. That's a great segue into the actual research that you and your colleagues did
[00:06:00] called the performance of chatbot compared to clinical practice guidelines in making informed decisions for lumbosacral, ridiculous pain. So you actually dug into this. If maybe one of the advantages for clinicians is that it can remind us of, oh, we need to look at these things.
[00:06:18] It can kind of prompt us to perform a more thorough exam. But now you looked at that and you compared what chat GPT came up with versus what the clinical practice guidelines recommend. How about you tell us what you found there?
[00:06:32] We decided to know and to compare the accuracy of different artificial intelligence at bot to clinical practice guidelines. And in the JOSPT papers, we focused in particular on chat GPT version 3.5. What we had done was a cross-sectional study and we decided to extract the recommendation
[00:06:56] from a recent clinical practice guidelines that talk about the noses and treatment of lumbosacral, ridiculous pain. And what we done was to compare what the chatbot inform us about the treatment and the management of this specific clinical condition compared to the clinical guidelines recommendation.
[00:07:19] We extracted nine clinical questions. And what we observed is that the accuracy between the chat GPT answers and the clinical practice recommendation was really slight, demonstrating an agreement only in 33% of recommendations. So these elements is really, really important for clinicians and also for the patients
[00:07:45] because up to now in this moment, we cannot use this type of tools to inform our clinical practice. And also the patient, if they decide to understand a specific clinical condition or ask about the best treatment or the best management for specific problems,
[00:08:06] I think that they can find a lot of not accurate information with a risk to guide them towards an not-invited based practice. I think also part of the danger of this is that these chatbots oftentimes will give you answers with a lot of confidence.
[00:08:25] And so it's hard sometimes on the receiving end, you know, you get an answer. It's confidently telling you the answer, but as you said, it's only actually matching the CPG recommendations 33% of the time, especially if you're a new clinician and you're looking for guidance
[00:08:41] that can lead to a lot of issues. Can we talk about how this factors into the research realm? Yes, I think that also research could be influenced by the use of the artificial intelligence chatbots. We have also here cross and cons.
[00:08:57] For example, the use of this chatbot can help to facilitate the preparation of the achievement of specific information about a topic that we want to study, for example, or can stimulate the researcher to nurture their creativity about a specific topic.
[00:09:19] Or they can help to structure and to help them to prepare a paper. But it is important to remember that they can help us to create a general structure of the paper, not the content. They can help us to have a linguistic review to adjust the grammar.
[00:09:39] And in that way, they can help to overcome some disparities regarding the use of English. Anyway, we have some different limitations because we don't know how is the effect of the contents produced by this type of artificial intelligence chatbot and their impact within their research field.
[00:10:02] Sounds really important to treat these as assistance, as aides, but not reproducing or for generating content, right? Because of the issue that we have with inaccuracy. Is that fair? Yeah, I think that up to now, we can use only the artificial
[00:10:21] intelligence chatbot as supportive tools that can help us to improve the speed or some procedural task. But we need to remember that there is a need to keep in the center the human abilities, the human skills in the education, in the clinical practice and also in research.
[00:10:44] The machine and the artificial intelligence chatbot, the algorithms can help us to facilitate the process, but they cannot replace the you. I feel like you summed it up really well. Like that's kind of the big takeaway. It's useful, but you have to be aware of the cons
[00:11:01] to be able to use it appropriately. If it's citing things for you, it might not be accurate and it comes off very confident. It's like a very successful social media influencer. These poor patients are getting information from everywhere.
[00:11:15] It's robots and humans who don't know what they're talking about. So we have to make sure that we're double checking and making sure that what it's giving us is accurate. As you know, what we say often is, in Italy, is that all the leaders are not gold.
[00:11:32] There is a strong need to continue the study of these chatbots and try to investigate the knowledge of clinicians, the knowledge of students, the knowledge of researchers about this topic in order to guide the creation, for example, of a specific guidelines about their use and management
[00:11:54] or to prepare specific learning materials or educational content for the clinician and the researcher and the scientific community. And just to make the listeners aware, so this is the viewpoint is awesome. There's not only these pros and cons laid out very clearly,
[00:12:13] but there's also some tables and examples that are really helpful to reference and there's a great SWAT, the strengths, weaknesses, opportunities and threats chart that's laid out to kind of summarize all of this just for like a quick reference, especially if you're chatting with students or colleagues
[00:12:31] about this topic that can be a good one to point them to. Where do you see like the future of this in the next couple of years? My suggestions and my idea is that there is a strong need to continue the study,
[00:12:45] the study of the performance of this topic in different fields. For example, in clinical practice, I would like to know the performance of this chatbots compared to the human physical therapist. For example, in identifying a specific red flex or in developing specific clinical reason
[00:13:06] and clinical decision making process. I would like to have a comparison between the chatbots and the human. But also I would like to know if the adoption of this type of tools can help us to obtain better therapeutic results, better therapeutic outcomes with our patients.
[00:13:28] But I'm also interested to understand if the use of these tools can improve or decrease the meta competence of the clinicians, of the students and of the researchers because we don't have that up to now. There is a great hype toward the topic
[00:13:46] and if you have the possibility to check the numbers of paper published on chatGPT and also on others, artificial intelligence chatbots presented within PubMed, it is really incredible because in two years there are more than 3,000 of paper published on this topic. So the hype is really hard.
[00:14:07] There are a lot of editorials, a lot of viewpoint and review, but what we need now is to have primary study to inform us about the benefit on the limitation of these tools. Yeah, I think with how fast these chatbots are changing and improving and learning,
[00:14:25] this is gonna be a topic that we kind of keep coming back to. You just covered a lot of different things that you're looking at in the future for people that are interested in continuing to read about this. Are there any research trials
[00:14:38] that you have either in the works or on the horizon that you're excited about in terms of this topic? Up to now if we have a check on PubMed, there are not a lot of paper published within the physical therapy fields. If we want to have more information,
[00:14:54] we can read some different papers presented, for example, within Nature or within Lancet or within the New England Journal of Medicine in order to continue the investigation and the knowledge on this field because every day some paper go out so we need to be updated about these elements.
[00:15:16] It sounds to me like we're gonna have to have you on every year, maybe even every six months because this change is so fast, but we're just gonna have to have you continue to come on the show and update our knowledge about these chatbots and AI
[00:15:30] and what we can trust, what we cannot trust and how we should be using them because this is something that is changing quite quickly. The most important message is that artificial intelligence output are tools that can help us but they cannot substitute the enormous
[00:15:48] that it is always within and in the center of the therapeutic process in physical therapy and via rehabilitation. I love it, that's a great place, I think, to leave it. That's a fantastic message to go out on. Dr. Rosettini, thank you so much for joining us
[00:16:04] on JSPT Insights. It's been a pleasure to have you on. Thanks for the invitation. Always a pleasure to work and to discuss with you. And so one last thank you to Dr. Rosettini for coming on the show and sharing his time and knowledge with us.
[00:16:16] And as always, we wanna thank you for listening to JSPT Insights. Thanks for listening to this episode of JOSPT Insights. For more discussion of the issues in musculoskeletal rehabilitation that are relevant to your practice, subscribe to JOSPT Insights on Apple Podcasts, Spotify, TuneIn, Stitcher, Google
[00:16:39] or your favourite podcast app. If you like JOSPT Insights, help others find us. Tell your friends and colleagues and raid on reviewers. To keep up to date with all the latest JOSPT content, be sure to follow us on Twitter. We're at JOSPT and Facebook, we're JOSPT official.
[00:16:56] Talk with you next time.