But....
Dont we run into the exact same problem when we get rules interpretations from people?
The difference is that people have the goal of giving you the correct answer, though of course they may make mistakes or misinterpretations. ChatGPT itself quite honestly does not have the goal of giving you a correct answer at all. The
designers of ChatGPT have the goal of creating something that probably gives you a correct answer, but the architecture of ChatGPT only does that indirectly. This probably seems like a subtle distinction, but it is pretty fundamental to the limits of things like ChatGPT.
When people make mistakes or misinterpretations they can engage in a rational discussion to support their interpretations and correct them. Other people chime in with correcting information or interpretations. Within the context of ASL rules this means the discussion follows a logical path to a correct conclusion or an agreement that the interpretation is ambiguous and needs errata or a Q&A.
When ChapGPT makes a mistake and you tell it that its first answer was wrong its response is typically a completely new answer seemingly from left field that may be correct or may just be differently incorrect. ChatGPT does not understand logic directly at all whereas people do. It is a language model and so it only understands logic as an inference from how articles on the web it was trained on happened to use logic in their content (correctly or incorrectly).
This is the key point of
@Jazz's reply. If you take the time to interact with ChatGPT on a topic you are in fact an expert on already you will pretty soon find it really gives some odd but "sounds correct" answers. Because you are already an expert in the field you'll know this. If you aren't you'll be led further astray.
ChatGPT is different from doing say a Google search and parsing the responses yourself. The top Google results are typically written by experts and if you take just a little time you can understand the context of the referenced page and confirm that the answer is actually contextually relevant to your question. Sure, there is always a small chance the referenced page is just plain wrong but if this was the case ChatGPT would do no better (same training data).
When you ask ChatGPT there is an additional (pretty frequent) potential for error in that ChatGPT will base its answer on entirely the wrong context. So even if ChatGPT based its answer on text from an "correct" webpage it is very possible the webpage didn't actually match the context of your question at all.
Again, the very best distillation of the immense complexity behind ChatGPT that I've read from AI experts is that what things like ChatGPT do are produce output that "sounds like the correct answer". That is fundamentally what they do, they have no idea or mechanism to evaluate whether the answer is correct. Indirectly this often results in a correct answer, but with distressingly high frequency it does not. And as has been pointed out since having references is part of "sounding correct" ChatGPT will happily create non-existent references or point you at actual references that in fact do not support the answer ChatGPT gave you.