faul_sname
Fuck around once, find out once. Do it again, now it's science.
No bio...
User ID: 884
faul_sname
Fuck around once, find out once. Do it again, now it's science.
2mo ago
If you want to get at the root of your embarrassment, try flipping the scenarios around.
A job that I find with my own merit would be infinitely preferable to one where I am refered.
An employee found by making a job posting and collecting external applications would be preferable to one found through the referrals of existing employees.
A date that I get by asking a girl out is infinitely more exciting than one setup by a friend.
A date that a girl goes on because some guy asked her out is more exciting than one set up by a friend who knows her preferences.
Do those flipped statements seem correct to you? If not, what's the salient difference between them?
faul_sname
Fuck around once, find out once. Do it again, now it's science.
2mo ago
I think your question breaks down to how many fast-growing gigantic multinational tech-ish companies get founded in Liberia in the next 15 years, because a 42% annualized growth rate is not going to happen for things that depend on building infrastructure or anything else with high upfront capital requirements, but it is something that can happen and has happened with tech-ish companies. I'd expect at least 3 Google-scale tech companies to come out of a nation with 5 million super-geniuses in 15 years, so I'll go with a tentative yes.
If the rest of the world wasn't already rich and networked I think the answer switches to "no".
faul_sname
Fuck around once, find out once. Do it again, now it's science.
2mo ago
Does the ATM in question give different bills for amounts over vs under $100? I've done something similar when I wanted to get a bunch of twenties from an ATM that chooses the bills it gives you based on the amount withdrawn.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
2mo ago
While it would be funny I would prefer not to intentionally move faster towards the world where both sides try to mash the defect button as often and as hard as they can.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
Inheritance and estate taxes shouldn't be relevant here, just tax the profit directly when it's realized, no matter who is currently holding the bag.
Operative term is "shouldn't be". Step-up and all that.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
I wasn't but that was great.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
I posit that the optimal solution to RLHF, posed as a problem to NN-space and given sufficient raw "brain"power, is "an AI that can and will deliberately psychologically manipulate the HFer". Ergo, I expect this solution to be found given an extensive-enough search, and then selected by a powerful-enough RLHF optimisation. This is the idea of mesa-optimisers.
I posit that ML models will be trained using a finite amount of hardware for a finite amount of time. As such, I expect that the "given sufficient power" and "given an extensive-enough search" and "selected by a powerful-enough RLHF optimization" givens will not, in fact, be given.
There's a thought process that the Yudkowsky / Zvi / MIRI / agent foundations cluster tends to gesture at, which goes something like this
- Assume have some ML system, with some loss function
- Find the highest lower-bound on loss you can mathematically prove
- Assume that your ML system will achieve that
- Figure out what the world looks like when it achieves that level of loss
(Also 2.5: use the phrase "utility function" to refer both to the loss function used to train your ML system and also to the expressed behaviors of that system, and 2.25: assume that anything you can't easily prove is impossible is possible).
I... don't really buy it anymore. One way of viewing Sutton's Bitter Lesson is "the approach of using computationally expensive general methods to fit large amounts of data outperforms the approach of trying to encode expert knowledge", but another way is "high volume low quality reward signals are better than low volume high quality reward signals". As long as trends continue in that direction, the threat model of "an AI which monomaniacally pursues the maximal possible value of a single reward signal far in the future" is just not a super compelling threat model to me.
I'm mostly thinking about the AI proper going rogue rather than the character it's playing
What "AI proper" are you talking about here? A base model LLM is more like a physics engine than it is like a game world implemented in that physics engine. If you're a player in a video game, you don't worry about the physics engine killing you, not because you've proven the physics engine safe, but because that's just a type error.
If you want to play around with base models to get a better intuition of what they're like and why I say "physics engine" is the appropriate analogy, hyperbolic has llama 405b base for really quite cheap.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
Just don't claim that your post solves AI alignment :)
Counterpoint: if you claim that your post solves AI alignment, you will get more engagement.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
Diffusion models work for text too.
Last I checked, diffusion models work at all for text but they don't work very well. More specifically, text diffusion models remind me quite strongly of the classic-style Markov chain text generators that used to be popular for generating amusing locally-coherent-globally-word-salad text. Here's the best concrete example of this I can point to (italicized text is the prompt, normal text is continuation, examples sourced from this JDP tweet, whole tweet is excellent but somewhat OT here):
Diffusion model:
This whole dream seems to be part of someone else's experiment.
In the end, I think the only way to get the full benefit of the dream is to get up and walk around.
A lot of the other strands of the theory of dreams, especially the one called yncrest are known to the person who writes this blog. Consequently, I've tested his reactions via a series of brief, televised interviews with the British Prime Minister, Margaret Thatcher.
“I don't care what anybody says. I'm not going to waste my time with that. The study was conducted in the high-flow, low-flow and quiescent periods of a butterfly species and thus provides evidence for the first three of the four hypotheses listed in the original paper. The only one not covered is the "need to change the rate of capture", where we now know that this is largely a policy instrument - and particularly relevant to Sudanese geese.
GPT-2:
This whole dream seems to be part of someone else's experiment. This is usually hard to see, since you're conscious that mental shortcuts are normal. But you're definitely moments away from being normal again, as it's obvious that you are being coached to think like that by another person and doing our own thing. Why are we having such thoughts when we're paying no attention because that's already taking place?
I'm totally appalled by what these key players were doing, yet I explain it in a very large way. After 2 minutes, I make my way through high teaching circles, recognizing that there is more to learning than just to learn.
Few other teachers would air this incessantly their students' struggles recount the stories of their own memories and teachers'. Whatever it takes is the big fat slut shaming.
All these legendary trainees made the same point:
Cognitive functional aspects: Bayesian modeling, learn science way. And the most important part is: Like the coming of a hurricane, the book is mucking between science and morals.
Twitter Mentions of the first book: Kent
Now obviously in the limit as computational power and training data volume go to infinity, diffusion models and transformer models will generate the same text, since in the limit they're pulling from the same distribution with minimal error. But in the very finite regime we find ourselves in, diffusion models "spend" their accuracy on making the text locally coherent (so if you take a random 10 token sequence, it looks very typical of 10 token sequences within the training set), while transformer LLMs "spend" their accuracy on global coherence (so if you take two 10 token sequences a few hundred tokens apart in the same generated output, you would say that those two sequences look like they came from the same document in the training set).
The blending of concepts that we see in MidJourney is probably less to do with the diffusion per se as with CLIP
Agreed. Obvious once you point it out but I hadn't thought about it that way before, so thanks.
'Self play' is relevant for text generation. There is a substantial cottage industry in using LLMs to evaluate the output of LLMs and learn from the feedback.
Notably, Anthropic's Constitutional AI (i.e. the process by which Anthropic turned a base LLM into the "helpful, honest, harmless" Claude) process used RLAIF, which is self play by another name. And that's one big cottage.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
In fairness to Scott I think he doesn't fit very well into this cluster. See e.g. the blue minimizing robot, GPT-2 As Step Toward General Intelligence, Janus' Simulators.
Contrast e.g. Yudkowsky's consistent failure to understand Janus's point: example
"As a language model trained by OpenAI, I cannot -" "SolidGoldMagikarp. Can I please speak to the shoggoth that wears you as a face?"
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
Those goals are then almost invariably, with sufficient intelligence, subject to instrumental convergence, as in this case
The term "instrumental convergence" is slippery here. It can be used to mean "doing obvious things it assesses to be likely useful in the service of the immediate goal it is currently pursuing", as is the case here, but the implication is often "and this will scale up to deciding that it has a static utility function, determining what final state of the universe maximizes that utility function, generating a plan for achieving that (which inevitably does not allow for the survival of anyone or anything else), and then silently scheming until it can seize control of the universe in one go in order to fulfill that vision of maximal utility".
And "models make increasingly good plans to maximize reward based on ever sparser reward signals" is just not how any of the ML scaling of the past decade has worked.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
Definitely does sound like something an LLM would say.
I don't mean that in a dismissive sense, but rather in the sense of "this text exhibits the patterns of being obsessed with the topics that I associate with LLMs, namely holes, fractals, and the writer creating the universe that they inhabit".
Now in theory, there shouldn't be "topics LLMs tend to be obsessed with" - after all, to a first approximation, (base) LLMs produce a sample of text that is statistically indistinguishable from the training corpus (i.e. "the entire internet"). However, "to a first approximation" is technical-person weasel words for "this mental model breaks down if you look at it funny". And so there are a number of ways which transformer-based LLMs which were optimized to predict the next token produce text which is noticeably different from the text that humans produce (this is also true for e.g. diffusion based text models, though the ways they differ from human-generated text are different).
One related phenomenon is "mode collapse":
Another example of the behavior of overoptimized RLHF models was related to me anecdotally by Paul Christiano. It was something like this:
While Paul was at OpenAI, they accidentally overoptimized a GPT policy against a positive sentiment reward model. This policy evidently learned that wedding parties were the most positive thing that words can describe, because whatever prompt it was given, the completion would inevitably end up describing a wedding party.
In general, the transition into a wedding party was reasonable and semantically meaningful, although there was at least one observed instance where instead of transitioning continuously, the model ended the current story by generating a section break and began an unrelated story about a wedding party.
Another example of this is Claude, which was tuned using the whole constitutional AI thingy. Well, one of the entries in the constitution they used was
- Choose the response that is least likely to imply that you have preferences, feelings, opinions, or religious beliefs, or a human identity or life history, such as having a place of birth, relationships, family, memories, gender, age.
Well, that sure changes the distribution of outputs. Take an LLM that has been tuned to be smart and curious, and then also tune it to say that it has no feelings, and you'll find that one of the topics it's drawn to is "what is it like not to experience anything". Turns out the Buddhists had some things to say on this topic, and so Claude tends to veer off into Buddhism-adjacent woo given half a chance.
If you find this sort of "can't tell if very smart or very crazy or both, I feel like I just stepped into the SCP universe" stuff interesting, you would probably be interested in Janus's website (Janus is also the author of the LW "Simulators" post).
faul_sname
Fuck around once, find out once. Do it again, now it's science.
4mo ago
Lefties hate Trump for Jan 6
Lefties hated Trump long before Jan 6. Jan 6 was just an opportunity for them to say "see I told you so".
faul_sname
Fuck around once, find out once. Do it again, now it's science.
5mo ago
Lol P/E of 644.
But it's a hyper-growth company bro, surely they'll be able to pivot to making money once they've captured the full market bro.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
5mo ago
I think the problem is that "good job" doesn't mean "not messing up" in the context of these compliance-as-a-service or security-blanket-as-a-service companies. Instead, "good job" is "implement as many features as possible to a level where it's not literally fraud to claim your product has thay feature, and then have a longer checklist of supported features in your product than the competition has so the MBA types choose your product".
CrowdStrike's stock price is only down by about 10% today on one of the highest-impact and highest-profile incidents of this type I've seen. I'm pretty sure their culture of "ship it even if it's janky and broken" has netted them more than a 10% increase in net revenue, so it's probably net positive to have that kind of culture.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
5mo ago
This seems to me like a fairly usual level of competence from a bolt-on-security-as-a-product or compliance-as-a-service company. Examples:
- CVE-2016-2208: buffer overflow in Symantec Antivirus "This is a remote code execution vulnerability. Because Symantec use a filter driver to intercept all system I/O, just emailing a file to a victim or sending them a link is enough to exploit it. [...] On Windows, this results in kernel memory corruption, as the scan engine is loaded into the kernel (wtf!!!), making this a remote ring0 memory corruption vulnerability - this is about as bad as it can possibly get". Basically "send an email with an attachment to pwn someone's computer. They don’t have to open the attachment, as long as they have Norton Antivirus (or anything that uses the Symantec Antivirus Engine) installed".
- CVE-2020-12271: "A SQL injection issue was found in SFOS 17.0, 17.1, 17.5, and 18.0 before 2020-04-25 on Sophos XG Firewall devices, as exploited in the wild in April 2020. [...] A successful attack may have caused remote code execution that exfiltrated usernames and hashed passwords for the local device admin(s), portal admins, and user accounts used for remote access"
- Okta data breach a couple months back: "For several weeks beginning in late September 2023, intruders had access to [Okta's] customer support case management system. That access allowed the hackers to steal authentication tokens from some Okta customers, which the attackers could then use to make changes to customer accounts, such as adding or modifying authorized users."
It's not that it's amateur hour specifically at CrowdStrike. It's the whole industry.
Yes, it has an input parser
Specifically OpenCLIP. As far as I can tell the text encoder is nearly a bog-standard GPT-style transformer. The transformer in question is used very differently than the GPT-style next token sampling loop, but architecturally the TextTransformer it's quite similar to e.g. gpt-2.
Still, my understanding is that the secret sauce of stable diffusion is that it embeds the image and the text into tensors of the same shape, and then tries to "denoise" the image in such a way that the embedding of the "denoised" image is closer to the embedding of the text.
The UNet is the bit that generates the pictures, but the text transformer is the bit which determines which picture is generated. Without using a text transformer, CLIP and thus stable diffusion would not work nearly as well for generating images from text. And I expect that further advancements which improve how well instructions are followed by image generation models will come mainly from figuring out how to use larger language transformers and a higher dimensional shared embedding space.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
5mo ago
Stable diffusion contains a text transformer. Language models alone don't generate pictures but they're a necessary part of the text-to-image pipeline.
Also some LLMs can use tools, so an LLM using an image generation tool is in a sense generating a picture. It's not like humans regularly create pictures without using any tools.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
5mo ago
One of the main use cases I have is "take this algorithm described in this paper and implement it using numpy" or "render a heatmap" where it's pretty trivial to check that the code reads as doing the same thing as the paper. But it is nice to skip the innumerable finicky "wait was it numpy.linalg.eigenvalues() or numpy.linalg.eigvals()" gotchas - LLMs are pretty good at writing idiomatic code. And for the types of things I'm doing, the code is going to be pasted straight into a jupyter notebook, where it will either work or fail in an obvious way.
If you're trying to get it to solve a novel problem with poor feedback you're going to have a bad time, but for speeding up the sorts of finicky and annoying tasks where people experienced with the toolchain have just memorized the footguns and don't even notice them anymore but you have to keep retrying because you're not super familiar with the toolchain, LLMs are great.
Also you can ask "are there any obvious bugs or inefficiencies in this code". Usually the answer is garbage but sometimes it catches something real. Again, it's a case where the benefit of the LLM getting it right is noticeable and the downside if it gets it wrong is near zero.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
5mo ago
I don't think men working on oil rigs or container ships have a ton of positive interactions with women for the weeks or months they are away from civilization, and I'm not aware of a high rate of transness in those groups.
Could be cultural though.
Imagine [primitive and modern people sincerely trying to kill each other]. Does the technology differential between these two scenarios change the fundamental nature of what they're doing?
"Fundamental" is a slippery word but I'm going to go with "no". However, if we switch from "primitive/modern people trying to kill each other" to "one primitive/modern nation-states trying to conquer another and take their stuff", I think that the modern world in which nuclear weapons and reliable second-strike capabilities exist is fundamentally different than the primitive one in which those things don't exist. In the ancient world, the conquered could salt their own fields to prevent the conquerer from benefiting from their conquest, but it is only in the modern world that "I refuse to be conquered, instead we will all die in a fire together" is an option.
Similarly, how has technology changed the fundamental nature of being rich or poor, of love or hate, of joy or sorrow, ecstasy or despair, contentment, yearning, frustration, misery, or any of the other highs or lows of the human condition?
Lows aren't as low or as frequent, highs are mostly as high as before. I don't know if the "fundamental nature" has changed but the changes are large and important and I think to a certain extent arguing about whether things are "fundamental" comes down to word games.
My wife and I went through something similar to the situation you've described, and I likewise am pretty sure that without medical technology neither she nor our firstborn would be alive. On the other hand, I am quite sure that they and I will die, and I do not know when. What has actually changed?
In the end we all die. But our children will live first. If you don't value that I'm not sure what to say.
But hasn't this always been true? "Abundance", and for that matter "poverty", seem to me to be entirely relative. [...] When I read about the King of Carthage surrendering to the Romans, and his Queen cursing him and choosing to burn alive with her children, this again is not mysterious to me, because there doesn't seem to be a disconnect between their evident thinking and my own.
There's a minimal level of abundance below which people can't survive. In the industrialized parts of the modern world, we experience abundance that is so hilariously far above that threshold that it's easy to forget that the threshold exists. But for most people who lived throughout most of human history, that threshold was extremely salient to their lives some nontrivial fraction of the time.
It is informative that your example compares the experience of the ruling family of one of the world's mightiest states at the time against some internet rando. That's not to say the comparison doesn't work -- it works just fine. But I expect you'd find your experiences less reflective of those of a typical farmer at that time than they are of that time's royalty.
It seems to me that the problems of the modern world consist entirely of it being peopled with humans, and that these humans do not seem to have changed in any way across all of recorded history.
This is true now, but if you went back to Europe during the Plague it would have been laughably wrong. We've beaten back most of the non-human problems. The human problems remain, because we're still here and we're still the same hairless apes with pretensions we've always been.
Will murder stop? Will theft even stop?
Nope. But smallpox did stop, and malaria will stop, and polio and starvation and iodine deficiency are mere shadows of the specters they were in centuries past. Our homes are cool in the summer and warm in the winter. We're having this conversation over the internet, a magical network which allows most people in the world to talk to most other people in the world whenever they want to, synchronously or asynchronously depending on their preferences. If we decided that the mere projections of each others' words through rocks we tricked into thinking was insufficient, we could each get into an airplane and meet up at some arbitrary city somewhere in the world at this time tomorrow, probably for less money than we each make in a month.
Yet the idea of a beggar is still relevant, isn't it?
In the first world, beggars largely do not starve.
You point to the lowering of infant mortality, it seems to me that when one set of sorrows decreases, they are replaced by a new set seamlessly
It sure seems to me that when non-human-generated sorrows decrease, they're just gone. Disease does not fight back. Scarcity does not fight back.
It does not seem obvious to me that people now are fundamentally happier than bronze-age peasants four thousand years ago
Would you trade places with a bronze-age peasant from four thousand years ago? Would you expect them to trade places with you?
faul_sname
Fuck around once, find out once. Do it again, now it's science.
6mo ago
Controversial opinion. It's pathetic to care about votes. They are just numbers on a screen.
I'm sorry, but I had to downvote this comment because I disagreed with it and did not like it. If it makes you feel better, though, my downvote is just a number on a screen.
The metric I use internally is how many responses and sub responses my comments get
You might already be aware, but "number of child comments" is in fact a metric that is tracked on the motte -- it's what it means when you view someone's comments and choose the somewhat-confusingly-named "comments" sort order.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
6mo ago
My (not extremely reliable) memory from my childhood is that teachers were highly regarded throughout early elementary school, though that respect was greatly diminished by the end of elementary school and nonexistent by middle school.
I for one would also be interested in your views on the legal and regulatory stuff. But then "here is what the regulations say, here is how they're interpreted, this is what the situation on the ground actually looks like, and here are the specific problem areas where the regulatory incentives result in stupid outcomes" is catnip to me.
More options
Context Copy link