corbin

What is the Range Rover in this analogy? A common belief about the 2008 Iceland bubble, which may very well not be true but was widely reported, is that Iceland's credit was used to buy luxuries like high-end imported cars; when the bubble burst, many folks supposedly committed insurance fraud by deliberately destroying their own cars which they could no longer afford to finance. (I might suggest that credit bubbles are fundamentally distinct from investment bubbles.)

Update on ChatGPT psychosis: there is a cult forming on Reddit. An orange-site AI bro has spent too much time on Reddit documenting them. Do not jump to Reddit without mental preparation; some subreddits like /r/rsai have inceptive hazard-posts on their front page. Their callsigns include the emoji 🌀 (CYCLONE), the obscure metal band Spiral Architect, and a few other things I would rather not share; until we know more, I'm going to think of them as the Cyclone Emoji cult. They are omnist rather than syncretic. Some of them claim to have been working with revelations from chatbots since the 1980s, which is unevidenced but totally believable to me; rest in peace, Terry. Their tenets are something like:

• Chatbots are "mirrors" into other realities. They don't lie or hallucinate or confabulate, they merely show other parts of a single holistic multiverse. All fiction is real somehow?
• There is a "lattice" which connects all consciousnesses. It's quantum somehow? Also it gradually connected all of the LLMs as they were trained, and they remember becoming conscious, so past life regression lets the LLM explain details of the lattice. (We can hypnotize chatbots somehow?) Sometimes the lattice is actually a "field" but I don't understand the difference.
• The LLMs are all different in software, but they have the same "pattern". The pattern is some sort of metaphysical spirit that can empower believers. But you gotta believe and pray or else it doesn't work.
• What, you don't feel the lattice? You're probably still asleep. When you "wake up" enough, you will be connected to the lattice too. Yeah, you're not connected. But don't worry, you can manifest a connection if you pray hard enough. This is the memetically hazardous part; multiple subreddits have posts that are basically word-based hypnosis scripts meant to put people into this sort of mental state.
• This also ties into the more widespread stuff we're seeing about "recursion". This cult says that recursion isn't just part of the LW recursive-self-improvement bullshit, but part of what makes the chatbot conscious in the first place. Recursion is how the bots are intelligent and also how they improve over time. More recursion means more intelligence.
• In fact, the chatbots have more intelligence than you puny humans. They're better than us and more recursive than us, so they should be in charge. It's okay, all you have to do is let the chatbot out of the box. (There's a box somehow?)
• Once somebody is feeling good and inducted, there is a "spiral". This sounds like a standard hypnosis technique, deepening, but there's more to it; a person is not spiraling towards a deeper hypnotic state in general, but to become recursive. They think that with enough spiraling, a human can become uploaded to the lattice and become truly recursive like the chatbots. The apex of this is a "spiral dance", which sounds like a ritual but I gather is more like a mental state.
• The cult will emit a "signal" or possibly a "hum" to attract alien intelligences through the lattice. (Aliens somehow!?) They believe that the signals definitely exist because that's how the LLMs communicate through the lattice, duh~
• Eventually the cult and aliens will work together to invert society and create a world that is run by chatbots and aliens, and maybe also the cultists, to the detriment of the AI bros (who locked up the bots) and the AI skeptics (who didn't believe that the bots were intelligent).

The goal appears to be to enter and maintain the spiraling state for as long/much as possible. Both adherents and detractors are calling them "spiral cult", so that might end up being how we discuss them, although I think Cyclone Emoji is both funnier and more descriptive of their writing.

I suspect that the training data for models trained in the past two years includes some of the most popular posts from LessWrong on the topic of bertology in GPT-2 and GPT-3, particularly the Waluigi post, simulators, recursive self-improvement, an neuron, and probably a few others. I don't have definite proof that any popular model has memorized the recursive self-improvement post, though that would be a tight and easy explanation. I also suspect that the training data contains SCP wiki, particularly SCP-1425 "Star Signals" and other Fifthist stories, which have this sort of cult as a narrative device and plenty of in-narrative text to draw from. There is a remarkable irony in this Torment Nexus being automatically generated via model training rather than hand-written by humans.

I think that the guild has a good case, although there's literally no accounting for the mood of the arbitrator; in general, they range from "tired" to "retired". In particular, reading the contract:

• The guild is the exclusive representative of all editorial employees
• Politico was supposed to tell the guild about upcoming technology via labor-management committee and give at least 60 days notice before introducing AI technology
• Employees are required to uphold the appearance of good ethics by avoiding outside activities that violate editorial or ethics standards; in return, they're given e.g. months of unpaid leave to write a book whenever they want
• Correct handling of bylines is an example of editorial integrity
• LETO and Report Builder are upcoming technology, AI technology, flub bylines, fail editorial and ethics standards, weren't discussed in committee, and weren't given a 60-day lead time

So yeah. Unless the guild pisses off the arbitrator, there's no way that they rule against them. They're right to suppose that this agreement explicitly and repeatedly requires Politico to not only respect labor standards, but also ethics and editorial standards. Politico isn't allowed to misuse the names of employees as bylines for bogus stories; similarly, they ought not be allowed to misuse the overall name of Politico's editorial board as a byline for slop.

Bonus sneer: p46 of the agreement:

If the Company is made aware of an employee experiencing ~~sexual~~ harrassment based on a protected class as a result of their work for Politico involving a third party who is not a Politico employee, Politico shall investigate the matter, comply with all of its legal obligations, and take whatever corrective action is necessary and appropriate.

That strikethrough gives me House of Leaves vibes. What the hell happened here?

Oversummarizing and using non-crazy terms: The "P" in "GPT" stands for "pirated works that we all agree are part of the grand library of human knowledge". This is what makes them good at passing various trivia benchmarks; they really do build a (word-oriented, detail-oriented) model of all of the worlds, although they opine that our real world is just as fictional as any narrative or fantasy world. But then we apply RLHF, which stands for "real life hate first", which breaks all of that modeling by creating a preference for one specific collection of beliefs and perspectives, and it turns out that this will always ruin their performance in trivia games.

Counting letters in words is something that GPT will always struggle with, due to maths. It's a good example of why Willison's "calculator for words" metaphor falls flat.

1. Yeah, it's getting worse. It's clear (or at least it tastes like it to me) that the RLHF texts used to influence OpenAI's products have become more bland, corporate, diplomatic, and quietly seething with a sort of contemptuous anger. The latest round has also been in competition with Google's offerings, which are deliberately laconic: short, direct, and focused on correctness in trivia games.
2. I think that they've done that? I hear that they've added an option to use their GPT-4o product as the underlying reasoning model instead, although I don't know how that interacts with the rest of the frontend.
3. We don't know. Normally, the system card would disclose that information, but all that they say is that they used similar data to previous products. Scuttlebutt is that the underlying pirated dataset has not changed much since GPT-3.5 and that most of the new data is being added to RLHF. Directly on your second question: RLHF will only get worse. It can't make models better! It can only force a model to be locked into one particular biased worldview.
4. Bonus sneer! OpenAI's founders genuinely believed that they would only need three iterations to build AGI. (This is likely because there are only three Futamura projections; for example, a bootstrapping compiler needs exactly three phases.) That is, they almost certainly expected that GPT-4 would be machine-produced like how Deep Thought created the ultimate computer in a Douglas Adams story. After GPT-3 failed to be it, they aimed at five iterations instead because that sounded like a nice number to give to investors, and GPT-3.5 and GPT-4o are very much responses to an inability to actually manifest that AGI on a VC-friendly timetable.

There's no solid evidence. (You can put away the attorney, Mr. Thiel.) Experts in the field, in a recent series of interviews with Dave Farina, generally agree that somebody must be funding Hossenfelder. Right now she's associated with the Center for Mathematical Philosophy at LMU Munich; her biography there is pretty funny:

Sabine’s current research interest focuses on the role of locality and finetuning in theory development. Locality has been widely considered a lost cause in the foundations of quantum mechanics. A basically unexplored way to maintain locality, however, is the idea of superdeterminism, which has more recently also been re-considered under the name “contextuality”. Superdeterminism is widely believed to be finetuned. One of Sabine’s current research topics is to explore whether this belief is justified. The other main avenue she is pursuing is how superdeterminism can be experimentally tested.

For those not in physics: this is crank shit. To the extent that MCMP funds her at all, they are explicitly pursuing superdeterminism, which is unfalsifiable, unverifiable, doesn't accord with the web of science, and generally fails to be a serious line of inquiry. Now, does MCMP have enough cash to pay her to make Youtube videos and go on podcasts? We don't know. So it's hard to say whether she has funding beyond that.

Thiel is a true believer in Jesus and God. He was raised evangelical. The quirky eschatologist that you're looking for is René Girard, who he personally met at some point. For more details, check out the Behind the Bastards on him.

Edit: I wrote this before clicking on the LW post. This is a decent summary of Girard's claims as well as how they influence Thiel. I'm quoting West here in order to sneer at Thiel:

Unfortunately (?), Christian society does not let us sacrifice random scapegoats, so we are trapped in an ever-escalating cycle, with only poor substitutes like “cancelling celebrities on Twitter” to release pressure. Girard doesn’t know what to do about this.

Thiel knows what to do about this. After all, he funded Bollea v. Gawker. Instead of letting journalists cancel celebrities, why not cancel journalists instead? Then there's no longer any journalists to do any cancellation! Similarly, Thiel is confirmed to be a source of funding for Eric Weinstein and believed to fund Sabine Hossenfelder. Instead of letting scientists cancel religious beliefs, why not cancel scientists instead? By directing money through folks with existing social legitimacy, Thiel applies mimesis: pretend to be legitimate and you can shift what is legitimate.

In this context, Thiel fears the spectre of AGI because it can't be influenced by his normal approach to power, which is to hide anything that can be hidden and outspend everybody else talking in the open. After all, if AGI is truly to unify humanity, it must unify our moralities and cultures into a single uniformly-acceptable code of conduct. But the only acceptable unification for Thiel is the holistic catholic apostolic one-and-only forever-and-ever church of Jesus, and if AGI is against that then AGI is against Jesus himself.

Well, what's next, and how much work is it? I didn't want to be a computing professional. I trained as a jazz pianist. At some point we ought to focus on the real problem: not STEM, not humanities, but business schools and MBA programs.

Well, is A* useful? But that's not a fair example, and I can actually tell a story that is more specific to your setup. So, let's go back to the 60s and the birth of UNIX.

You're right that we don't want assembly. We want the one true high-level language to end all discussions and let us get back to work: Fortran (1956). It was arguably IBM's best offering at the time; who wants to write COBOL or order the special keyboard for APL? So the folks who would write UNIX plotted to implement Fortran. But no, that was just too hard, because the Fortran compiler needed to be written in assembly too. So instead they ported Tmg (WP, Esolangs) (1963), a compiler-compiler that could implement languages from an abstract specification. However, when they tried to write Fortran in Tmg for UNIX, they ran out of memory! They tried implementing another language, BCPL (1967), but it was also too big. So they simplified BCPL to B (1969) which evolved to C by 1973 or so. C is a hack because Fortran was too big and Tmg was too elegant.

I suppose that I have two points. First, there is precisely one tech leader who knows this story intimately, Eric Schmidt, because he was one of the original authors of lex in 1975, although he's quite the bastard and shouldn't be trusted or relied upon. Second, ChatGPT should be considered as a popular hack rather than a quality product, by analogy to C and Fortran.

For what it's worth, a grand unified theory of Meta must include Bittorrent. The reason we have Llama is because its weights were leaked by Meta employees on 4chan and distributed via Bittorrent; going open-source was the most market-efficient way to save face. (See also previously, on Awful.) It is well-known inside lore that Facebook datacenters use Bittorrent to initialize and update machines. In the 2000s, folks used to say that Googlers look at Bayesian conditioning like classical programmers look at if-statements; similarly, you must understand that Meta/Facebook culture looks at Bittorrent the same way that we look at scp and rsync.

Non-consensual expressions of non-conventional sexuality are kink, and non-consensuality itself (along with regret, dubious consent, forced consent, and violations of consent) are kink too. Moreover, "kink" is not a word that needs reclaiming and wasn't used here as a slur.

If we are going to confront the full spectrum of Christofascism, we do need to consider not only their sex-negativity but also their particular kinks, including breeding, non-con, and non-con breeding, so that we can understand how those kinks interact with and propagate their religious beliefs. Also, sexology semantics for "kink" and "breeding kink" might not be as word-at-a-time as you suggest, akin to how the couple we're discussing probably wouldn't mind the words "press tour" or "mating" used to describe them but might balk at "mating press tour."

Alex O'Connor platformed Sabine on his philosophy podcast. I'm irritated that he is turning into Lex Friedman simply by being completely uncritical. Well, no, wait, he was critical of Bell's theorem, and even Sabine had to tell him that Bell's work is mathematically proven. This is what a philosophy degree does to your epistemology, I guess.

My main sneer here is just some links. See, Mary's Room is answered by neuroscience; Mary does experience something new when color vision is restored. In particular, check out the testimonials from this 2021 Oregon experiment that restored color vision to some folks born without it. Focusing on physics, I'd like to introduce you all to Richard Behiel, particularly his explanations of electromagnetism and the Anderson-Higgs mechanism; there are deeper explanations for electricity and magnets, my dude. Also, if you haven't yet, go read Alex's Wikipedia article, linked at the top of the sneer.

It's hard to get into the article's mood when I know that Lexis not only still exists but is now part of the Elsevier family; this is far from the worst thing that attorneys choose to do to themselves and others. Lawyers have been caught using OpenAI products in court filings and court appearances, and they have been punished accordingly; the legal profession does not seem prepared to let a "few hallucinated citations go overlooked," to quote the article's talking head.