Searching for Songs in a Sea of Hallucination.
"But there are also unknown unknowns — the ones we don't know we don't know"
- Donald Rumsfeld
Sometimes a song or a lyric becomes stuck on the tip of your tongue. Sometimes a fleeting reference drives you on a search into madness. And when you add AI to the mix, then everything is built on lies.
I recently read a book, "On the Shores of the Mediterranean", by Eric Newby. In it, he writes the following throw-away line:
It is a pity in a way that one could not make a litany or a song about them, as someone once did about hangovers: 'the cachet faivres we bought in Sèvres, the Enos in Buenos Aires ...'
I was curious, and fired up my search engine. Google had no results, but that wasn't surprising - the quality of search has been declining for years ago, and while Google spent decades "organizing the world’s information", they have given up on letting people simply search through it.
DuckDuckGo also failed to find anything. I abandoned my quest.
Nowadays it's almost de rigueur for any conversation online to include the musings of ChatGPT or another it's ilk. Even though it's well known that the responses from these engines are not factual, but simply sound realistic, people quote AI like an oracle. An entire generation has got confused on the difference between truth, and plausability. We are spending billions on creating tools to make statements that are indistinguishable from the truth.
But I have been accused many times of being curmudgeonly, so what the hell, I thought, let's give AI a chance to shine.
I opened ChatGPT (using the 4o model), and asked:
"What song contains the lyrics 'the cachet faivres we bought in sevres'"
A stream of wrong answers followed:
The conversation continued for quite some time, obviously wrong answer after wrong answer, I instructed the model to stop hallucinating. But no joy:
At this point I ran out of tokens, and gave up on ChatGPT. But my quest wasn't over. I also have access to some other models - I thought I would try Mistral Large, another cutting edge model.
This was more promising - after giving me an incorrect answer, the model accepted defeat. I persisted and gave some more clues:
Although the book was incorrect, the lyrics sounded plausible, and to be honest, I had forgotten that I had the next line from the book that would prove that this was a hallucination. I persisted, and asked the LLM for more information, including a link to the lyrics.
It seemed very good at generating broken links. On the one hand generating plausible responses is the r'aison d'etre of an LLM. But Mistral claimed to be able to check the links:
We went down a long rabbit hole, of Mistral giving me the name of an album and a tracklist, and me doing research and finding that it didn't exist. The fact that so much old copyrighted music has been removed from the internet made this a lot harder.
At some point it became clear that the entire charade was a hallucination.
At this point I gave up on Mistral, and, amused by the amount of time I had wasted, started writing this blog. But there are more models, surely I should try some more.
Meta-Llama-3-70B-Instruct gave a good answer - it didn't know. Even with some prompting and some further information, it admitted it's ignorance.
Gemini asked to add an extension "Youtube music", which had a privacy warning dialogue. Not a great start. It then gave a list of songs that were unrelated. It was a worse result than simply searching on youtube.
I gave up on testing Anthropic Claude as they wanted my telephone number to sign up.
At this point I gave up on finding an answer.
I had wasted a few hours, but got a sense of how the cutting edge models deal with uncertainty. ChatGPT blatantly made stuff up, but gave up as soon as it was accused of hallucination. Mistral made stuff up, then lied persistantly and convincingly about it's hallucinations. It even fooled me with some of the lies. Llama admitted ignorance, which I consider the best result of all the models. And Google failed with all of it's products.
If anyone knows where the line in the song comes from, I would love to hear from you.