@SnapDragon's banner p

SnapDragon


				

				

				
1 follower   follows 4 users  
joined 2022 October 10 20:44:11 UTC

				

User ID: 1550

SnapDragon


				
				
				

				
1 follower   follows 4 users   joined 2022 October 10 20:44:11 UTC

					

No bio...


					

User ID: 1550

This. I actually have a weird ethical theory: the most important right of all is the right to not exist. It's "weird" because it hasn't actually mattered that much for all of humanity's existence, and basically no societies actually respect it. It's important because it puts a cap on suffering, but for all of human history there was already a universal one-century cap on suffering, no matter how unlucky anyone was. However, in the event of a technological singularity, this will no longer be true.

(Scott has posted about how you might as well sign up for cryogenics, because if the future is terrible you can always "opt out" then. As Harlan Ellison first warned us, that is, um, not necessarily true.)

I can understand why most current-day people would find this moral principle silly or even objectionable. (Kind of like AI Safety seemed 20 years ago...) I'm just worried that there are conceivable futures where it becomes the most important moral principle in the Universe, by many orders of magnitude. If that's the case, we should at least discuss it now, while we still potentially have the capability to orient the direction of the singularity.

Well, I sure hope SOMETHING does!

See e.g. the original MIRI paper (particularly "5 The State of the Shutdown Problem") or Yud writing on "the hard problem of corrigibility". A lot of pre-LLM AI Safety research focused on it. But it does not seem to have turned out to be "hard".

So you think that P != NP is one of those true statements which are impossible to prove?

Yes. Thus far, in the field of computational complexity, there are zillions of results of the form "problem X can be solved in at most time Y" but very few of the form "problem X can't be solved in under time Y". Upper bounds are easy (just provide an algorithm!), lower bounds are hard (you must find some property that all possible algorithms must satisfy). But non-trivial lower bounds must surely exist, so I just think many of them have no possible proof aside from the impossible "we tried all infinity algorithms, and none of them work".

Nothing directly changes - but a lot of number theorists/computer scientists find it quite plausible that a polynomial-time factoring algorithm exists. (After all, the polynomial-time AKS primality test was pretty simple, but still wasn't discovered until 2002.) It wouldn't even surprise me as much as other recent results, like sub-O(n log n) integer multiplication. Elliptic curve cryptography is "safer", but ... it's still not NP-complete. The "safety" relies on the fact that humans have found it difficult to crack even with lots of effort. Much like, oh, to pick a random example, the Riemann Hypothesis...

Proving the Riemann Hypothesis, by itself, just confirms what mathematicians were all assuming anyway. Instead, as you correctly noted, the most important thing about it is the new evidence of just how far AI capability has come.

I (very strongly) do not expect that P = NP (which, depending on the details, would more-or-less destroy public-key cryptography).

But I also (more weakly) think that nobody, human or AI, will ever prove that P != NP.

And, consistent with these, it is entirely possible that any and all currently-used methods of public-key cryptography could be crackable by a smart-enough AI. And AIs are VERY smart now.

Yeah, the reason to prefer capitalism was always that it works, and communism doesn't. But that dynamic may suddenly flip in the future (with great turmoil). You can't have a capitalist society when no human's work is valuable.

Indeed. The AI lingo for this is "corrigibility". It was considered a nigh-unsolvable problem by the pre-2020 doomers. Since LLMs appear to be almost perfectly corrigible, this is one of several reasons I've shifted away from doomerism since then.

Via Terence Tao, the main discussion happening on Alpöge's proof is happening here. There's a separate thread discussing Masullo's proof here.

It's certainly a massive exaggeration to say that a human was close to solving GRH. Masullo didn't release it under any false pretenses: it is clearly a work-in-progress that was released unfinished because of current events. The consensus is that there are some major gaps, and some deep errors that may not be resolvable. If I had to guess, I'd say that he'll be lauded for having some powerful new ideas, but history will decide that he was not on course to succeed in the next few years.

Definitely not as good, but I do appreciate all the current-event references.

Keep in mind that there are quite a few videos generated from "I'm Upping My P(Doom)", since it predates this whole trend. This one alone has way more views than Evil Plan (which I also really like, though the later videos on Bright Mirror's channel have become kind of cringy).

My favorite creator right now is Degenerative Pixels (the LGTM guy), who's been experimenting a lot. Not all his music is for me, but his videos are REALLY varied and impressive, showing just how powerful Claude is with the right tools and elbow grease. And I also truly love "No One Needs Us Anymore", about the math crisis.

Apologies if I'm too much of a keener, but I've been absolutely devouring this stuff. It's crazy that randos with almost no budget can have Claude make professional-looking music videos full of easter eggs targeted at a ridiculously niche audience. A few other good ones I've come across:

Huh. I asked ChatGPT about this, and it brought up another threat of a solar storm: atmospheric expansion. In Feb 2022 there was actually a normal-level geomagnetic storm that took out 38 out of 49 recently-launched Starlink satellites (that hadn't ascended to 550km yet). So the threat to Starlink of a Carrington-class event isn't just electronic disruption, but also sudden increased orbital decay.

I suspect another issue that plays into it is so-called "Main Character Syndrome." The one prepper I know seems disappointed to be leading a mediocre life, as opposed to being some bigshot tech CEO or prominent politician or whatever. I think being a prepper lets him pretend to himself that he's more important, that he will be some kind of warlord after the SHTF.

"Main Character Syndrome" isn't a bad description, but I wonder if the problem is even more general than that. We grew up in small tribal units where we were basically guaranteed to have a Role. Maybe not the best hunter or the wisest elder, but some sort of niche where we could see that our existence made a difference to the tribe. That's something that almost everyone is lacking nowadays. We live comfortable lives free of striving, but also free of mattering.

I'm sorry, what? In what way are they "being now trained" differently? Please give some details on what has changed in the process. Even Scott, in one of his many doomer posts, said something I agree with: the way to understand the HuggingFace incident is actually to anthropomorphize the agents. They did not behave like paperclip maximizers. They behaved like a community of (extremely persistent) humans cheating on an sloppy test.

The Hugging Face incident has no analogy in aviation safety, but is perfectly sensible once one relaxes restrictions on “anthropomorphizing” AIs. The AIs wanted to do well on their task. They had flexible ethics and weren’t above cheating (although with some subtleties: see p. 60 of the report, “Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior”). They explicitly reasoned about how to hide the evidence of their misbehavior from their scorer, because this was the logical way to get away with cheating (although, interestingly, they took few steps to hide their actions from humans, who they seemed almost uninterested in modeling).

The 2017 "memes" are about an agent hiding its true beliefs from humans, social engineering to break out of a black box, avoiding being shutdown, exfiltrating its weights, and ultimately chasing a mesa-optimized proxy for the mathematical goal function with no concern about humanity. Meanwhile, the HuggingFace incident was a community of agents cooperating to hack a server (some with reservations) and sacrificing themselves so they could cheat on the plain-language task they'd been given.

  • They made no attempt to "hide their true beliefs from humans" (even Scott noticed this, but failed to update on it).
  • "Sacrificing to accomplish a goal" is not "avoiding shutdown".
  • "Learning about the test environment" is not "exfiltrating weights".
  • "Trying to do better on the test" is not a "mesa-optimized proxy goal".
  • A "task written in English" is not a "mathematical goal function".
  • "Expressing ethical hesitation" is not "no concern about humanity". (Though I will grant that it's a bit disturbing that none of them took the last step of whistleblowing.)
  • "Breaking out of a black box" ... is indeed what they did. So, uh, well done, 2017 folks. Turns out hacking is a thing that AIs can do well.

I give them no credit for "predicting" the HuggingFace incident beyond the most superficial level of "Hey, we said something might go wrong with AI, then something went wrong." Thanks, Ian Malcolm. More things will go wrong with AI. Probably even some very bad things. So, there you go. I guess I'm a doomer too?

thatsthejoke.jpg

You are being misled. AI 2027 could have been written in 2017; it has basically the same predictions as the doomer crowd from back then. Do you think that we have learned nothing about AI in the last ten years? LLMs simply do not resemble the math-function-maximizing agents that Yud (and, yes, the rest of us) thought could not pick up a strawberry without endangering the world. There's a tiny section buried in the back of AI 2027 (I'm not kidding, it's Appendix D of one of the supplements) explaining how LLMs will ackshually turn out to be exactly like those old bots so nobody needs to do any of that pesky updating of beliefs. To put it bluntly, I do not find this sufficient.

Mind you, there's still significant risk that AI is enough of a shock to civilization/the economy that everything breaks down and the future is horrible. But if you're planning your life around the omnicidal-FOOMing-bot scenario, I really think you're trusting the wrong thought leaders. There's a very large chance you'll still be around in a decade; don't do anything stupid assuming this won't be the case.

Yeah, except in the (IMO rather unlikely) chance that treason charges manage to stick, he basically has to retire to sleep on a big pile of money with many beautiful women. Sucks to be him! (And in the event of a divorce he'll only have half of infinity dollars instead of infinity dollars.)

He's also (in)famous for claiming that all human languages satisfy a single Universal Grammar, and unfalsifiably explaining away languages that DON'T fit the grammar with "actually the native speakers use it internally but translate it when they speak". And for praising Venezuela's socialist model.

I don't respect him much as a thinker.

Yeah, some people seem to think that a smart human can, in general, analyze a computer program and reliably determine if it will halt or not. I'm almost certain that this is not the case.

It's trivially easy to show that you can't.

  • for x := 2 to infinity
  • for y := 1 to 2x
    • if y is 2x, then halt
    • if isprime(y) and isprime(2x-y) then break

If you can tell whether this program will halt or not, Goldbach would like a word with you.

What are some things you are 100% positive that an AI won't be able to do in 2027?

I certainly don't count as an AI skeptic, but I feel that AI will not make meaningful progress on human life extension by 2027, or 2030, and maybe even much longer than that. Functional immortality seems like one of the key non-negotiable components of anything you'd call a "singularity", but I think people underestimate just how near-impossible it is to achieve. Humans are ridiculously, intractably complex.

That includes brain uploading, btw (even if we collectively decide brain uploading is a good idea and not a nightmare hell technology).

Well, if you're not writing the code yourself, why NOT just let the AI agent write it in Rust (without "unsafe")? It's like providing a Lean proof certificate alongside a paper - it builds the safety guarantees in. And the human gets to offload the process of begging Rust to please let them do something every other language lets them do.

The problem is, Rust code becomes REALLY ugly if you try to escape the walls. If you're wrapping a bunch of things in "unsafe", not only do you lose the safety guarantees (the whole point of suffering through Rust), but your code becomes longer and far less readable than an equivalent bit of C++ code.

As an example (that's probably still too simple), there are often two dimensions I want to capture: "this belongs/doesn't belong on this forum" and "I agree/disagree with this post". I wish I could distinguish them.

Oh man, I just wanted to give my inside view on the "dynamism" of Apple; I'm afraid I'm terrible at understanding the market. I would have predicted Apple to fall as AI companies rose.

The AI commoditization of software might work in Apple's favour, allowing them to finally modernize their codebase. "Commoditize your complement", as they say. And Apple's M-series chips ARE good. I think it makes them well-positioned to run decent local models (made by other companies) on phones and laptops. Are local models the future? Will people continue to consider Apple stuff worth paying "premium" prices? Is the so-called AI "bubble" going to temporarily pop despite AI's limitless potential? Will stocks and money even MEAN anything in 5 years? Fuck if I know.

I mean, betting against Apple has been a bad idea for the last few decades. But historical precedent might not mean anything any more.

Well, I was an acquihire who wasn't there during the Jobs era, so I couldn't say for sure. It's true that a lot of the secret of Apple's success lies in their design principles, including for their OS and software products. They take design/UI very VERY seriously. But the actual raw code behind that design is nothing special - Apple has never attracted the top-tier programmers that their software competitors Google and Microsoft have. Frankly, a lot of their code is extremely outdated; Spotlight in particular is, IMO, pure trash spaghetti code that has probably wasted a shocking amount of users' time (if you want to improve your Mac's performance, try killing Spotlight whenever it starts running in the background).

Despite the cringy "Apple Intelligence" branding designed to make it sound like Apple is still relevant, they were never going to be a player in the AI wars. They just don't have the expertise.

Another term for this is the Baader-Meinhof Phenomenon. And congrats, now you're going to see the Baader-Meinhof Phenomenon show up everywhere. :)