See Watching ChatGPT explain a reasoning problem is like watching someone who learned the language phonetically take an improv class on observationalepidemiology.blogspot.com

We need to start thinking of these systems not as generative AI but as regurgative. In some cases that’s good enough, niche applications like generating unimportant boilerplate text or writing code snippets, but even in those limited area, it can only function where humans have not only explored a question, but have done so in such exhaustive detail that the algorithm can autocomplete its way to a useful response.

This is pretty much what I discovered the other day.