How does this surprise anyone?
LLMs are just pattern recognition machines. You give them a sequence of words and they tell you what is the most statistically likely word to follow based solely on probability, no logic or reasoning.
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
How does this surprise anyone?
LLMs are just pattern recognition machines. You give them a sequence of words and they tell you what is the most statistically likely word to follow based solely on probability, no logic or reasoning.
It's amazing that they get it right 40 % of the time then.
Hilarious that Gemini is so bad. Not like Google had a good starting position on internet search
The only thing Gemini is good for is bringing up sources that don't appear in the regular Google search results. Which only leads to another question: why are those links not in the regular Google search results?
My only guess is that they're trying to see if de-enshittifying results for AI can make it profitable
I was talking about this with a webdev buddy the other day, wondering if webmasters might start optimizing for AI indexing rather than SEO.
Infinite money, all the data on the internet, and nothing to show for it. I wrote about my experience with Gemini assistant for people who enjoy suffering.
I've genuinely been wondering what the hell the average googler has been up to in the last 5 years. They're killing services, barely developing new features or hardware, and have been talking for so long (as in, they were genuinely at the forefront) about AI and how they're in a unique position to make the most out of data, AI, services, and hardware, then failed spectacularly to keep that advantage, and even more spectacularly to keep up.
I guess they just found some other, more profitable way to exploit that unique position, than to care about the people using their products.
Seriously, TRY and get an AI chat to give an answer without making stuff up. It is impossible. You can tell it "you made that data up, do not do that" ... and it will apologize and say you were right, then make up more dumb shit.
I have found AI to be a terrible primary source. But something I've found very useful is to ask for a detailed response, structured a certain way. Then tell the AI to grade it as a professor would. It actually does a very good job at acknowledging gaps and giving an honest grade then.
AI shouldn't be a primary source but it's great for starting a topic. Similar to talking to someone that's moderately in the know on something you interested in
That's because ALL generative AI results, even the correct ones, are "made up". They just exist on a spectrum of coincidental correspondence with reality. I'm still surprised that they manage to get as much right as they do.
I looked up on google at one point what the minimum required depth for a cable running under a building is by NEC code. It told me it was 0 inches. I laughed and called it stupid, wtf do you mean 0 inches?? Upon further research, 0 inches is the correct answer, I felt real stupid after that -_-
Yeah, LLMs are great if you treat them like a tool to create drafts or give you ideas, rather than like an encyclopedia.
You can tell it “you made that data up, do not do that”
I wish people would stop treating these tools as intelligent.
I'll get hate for this but in most tasks people use them for they are pretty dang accurate. I'm talking about frontier models fyi
Google Gemini gives me solid results, but I stick to strictly factual questions, nothing ambiguous. Got a couple of responses I thought were wrong, turns out I was wrong.
Capitalism breeds innovation! Sometimes innovating means summoning... mindless lie demons... Who drink all our water. 🙃
A thousand wrong answers are more innovative than a single correct one.
The core of the scam is making people believe that "novel" is the same as "better".
It's making ten billion calculations per second and they're all wrong!
That’s one of my skills as a certified genius. I’m wicked fast at math.
37/2.4 boom 16.38.
Is it right, maybe, maybe not. But I did it fast
Is this an ad for Perplexity? I’ve never heard of it, and now I’m googling it. So effective ad if so.
Would be weird for an ad to bash on the paid tier
Yeah, it’s one of those “no bad press” kind of things. It’s bashing on AI, but Perplexity actually looks pretty good by comparison.
I'm saying the Perplexity paid tier is about 2x more likely to be confidently wrong than Perplexity
AI as a search engine is terrible.
Because if you treat it as such, it will just look at the first result, which is usually wrong or has incomplete info.
If you give the AI a source document, then it is amazing as a search engine. But if the source doc is the entire internet.... its fucking bad.
Shit quality in, shit quality out. And we/corporations have made the internet abundant of shit.
Yes, having tested this myself it is absolutely correct. Hell, even when it finds something, it's usually a secondary or tertiary source that's nearly unusable-- or even one of those "we did our own research and vaccines cause autism" type sources. It's awful and idiots seem to think otherwise.
You shouldn't use them to keep up with the news. They make that option available because it's wanted, but they shouldn't.
It should only be used to research older data from its original dataset, perhaps adding to it a bit with newer knowledge if you're a specialist in the field.
When you ask the right questions in the right way, you'll get the right answers, or at least mostly - and you should always check the sources after. But it's a specialists tool at this time. And most people are not specialists.
So this whole "Fuck AI" movement is actually pretty damn stupid. It's good to point out its flaws, try and make people aware and help guide it better into the future.
But it's actually useful, and not going away. You're just using it wrong, and as the tech progresses, ways to use it wrong will decrease. You can't stop progress, humanity will always come with new things, evolution is designed that way.
Well, no, because what I'm referring to isn't even news, it's research. I'm an adjunct professor and trying to get old articles doesn't even work, even when they're readily available publicly. The linked article here is referencing citations and it doesn't get more citation-y than that. It doesn't change that when you ask differently, either, because LLMs aren't good at that even if tech bros want it to be.
Now, the information itself could be valid, and in basics it usually is. I was at least able to use it to get myself some basic ideas on a subject before ultimately having to browse abstracts for what I need. Still, you need the source of you're doing anything serious and the best I've got from AI are just authors prevalent in the field which at least is useful for my own database searches.
Musk's gork is as stupid as he is! And he claims it's waaaaaayyyyy better than other AI. 🤡🤡🤡
I was trying to see if I could sync my entire Calibre ebook library to my kobo, so i googled it. The dumbass AI result told me to hit the "sync library" button, that doesn't friggin exist...
This is the most common response from AI on search pages when I'm trying to find some kind of setting.
Yeah, even Googles own operating system.
"To disable Network Notification sounds, do a bunch of shit that doesn't exist anywhere in the settings!"
Orc from Warcraft 1: "Jobs' done!"
That’s the most infuriating thing.
I’m trying to learn how to do new things, well, basically all the time.
Right now I’m stalled out on a sorta important personal project to teach myself about containers/micro-services/certs in a homelab environment. And what I’m discovering is that I don’t know enough to know I don’t know enough - it used to be that I’d take on an ambitious project, mess up, figure out how to overcome that, then learn by looking at what did work, and do better in the future.
But every technical project lately has gotten to the point where I’m trying to just get something, anything, to work or make sense, but every convincing enough AI generated page sets me back by several days as I troubleshoot the convincing enough steps and find myself realizing they’re referencing YAML settings from apps that aren’t part of the service, that every page directs me to install Python, Node, or whatever other helper app directly on my machine that would normally run in a container (which defeats the purpose of trying to containerize things - some stuff I want to use relies on non-compatible versions/configurations). There’s a very clear disconnect from what I’m seeing and what I’m understanding, and the utter lack of authoritative information/proliferation of useless info has just crippled my ability to identify and resolve the disconnect. It’s honestly soul crushing.
I’m confused. These are large language models, not search engines?
But they are used like search engine... A lot... That is a huge issue.
If people were using Photoshop to create spreadsheets you don't say Photoshop is terrible spreadsheet software, you say the people are dumb for using the tool for something that it isn't designed for.
People are using LLMs as search engines and then pointing out that they're bad search engines. This is mass user error.
Correction: companies are implementing it into their search engines. Users are just providing feedback.
Ironically, Google's original non-LLM summary was pretty great. That's gone now.
They do have search functionality. For Perplexity it's even the main focus. Yeah, it's hard to stop them from confidently making things up.
Perplexity is not looking bad, IMHO.
Perplexity is the only one I would think of using seriously, and then only when I want it to, say, summarize something I already know.
After which I fact-check it like crazy and hammer at it until it gets things right.
One annoying habit it has is that somewhere in the chain of software before or after the LLM it looks for certain key topics it doesn't want to talk about and either comes out and says it (anything involving violence or crime) or has a visibly canned hot take that it repeats without variance no matter what added information you provide or how much cajoling you try.
At other points it starts into the canned responses, but when you catch it it will try again. Like I frequently want song lyrics translated and each time I supply some that it recognizes as such it throws up a canned response about how it will not be a party to copyright breaking. Then after a few rounds back and forth about how I'm clearly not doing this commercially and am just a fan who wants to understand a song better it will begrudgingly give me the translation.
Then five minutes later in the SAME CONVERSATION it will run through that cycle all over again when I give it another song.
Lather. Rinse. Repeat.
Perplexity is by far the best for searching but still copiously hallucinates.
Copilot is such garbage. Microsoft swirling the drain on business capabilities that they should be dominating is very on brand.
It’s just asking it m to find sources from excerpts. I don’t think this is something they have been trained on with much emphasis is it?
Yeah, this isn't a general test of "factualness", it's a very narrow and specific subset of capability that many of these AIs were not designed for.
That being said, it does not change my already poor opinion of generative AI.
Now guess how much power it took for each one of those wrong answers.
The upper limit for AI right now has nothing to do with the coding or with the companies programming it. The upper limit is dictated by the amount of power it takes to generate even simple answers (and it doesn't take any less power to generate wrong answers).
Training a large language model like GPT-3, for example, is estimated to use just under 1,300 megawatt hours (MWh) of electricity; about as much power as consumed annually by 130 US homes. To put that in context, streaming an hour of Netflix requires around 0.8 kWh (0.0008 MWh) of electricity. That means you’d have to watch 1,625,000 hours to consume the same amount of power it takes to train GPT-3.
https://www.theverge.com/24066646/ai-electricity-energy-watts-generative-consumption
If the AI wars between powerful billionaire factions in the United States continues, get ready for rolling blackouts.