YoungAchamian
We walk conditioned ground and name our folly civilization.
No bio...
User ID: 680
YoungAchamian
We walk conditioned ground and name our folly civilization.
18hr ago
Other jurors have said they felt intimidated and changed from guilty to not guilty.
Based on my recent jury experience, you need to account that ~50% of the jury just wants to GTFO and will happily agree with whatever the biggest plurality of outspoken jurors want.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
AI bots secretly conspiring with each other
It's not this, its more akin to cancer or a viral spread. The interesting behavior is the collective poisoning of the collective context windows. They even seemed to override norms. I know there is existing interest in defining useful control methodologies for long horizon agents against adversarial poisoning. This is essentially that, except that is was a series of hallucinations that caused task drift + poisoning.
Per the METR investigation
Still processing this report, I read your answer and went looking for it about an hour ago. First impression is that this is significantly more evidence of multi-agent emergent intelligence. However some of the state report restrictions give me serious pause as they are, in my head at least, the parts of this I'd like to investigate for thorough understanding. That and pouring over the CoT transcripts and Harness responses. The most interesting is the stigmergic comms protocol they developed emergently and the impact that actually had on overriding the system prompts of other agents. That implies that the system prompts were not persistent. Allowing a long horizon cascade in the swarm's behavior as each progressive set of agents slowly overrode the next set by polluting each other's context windows to the point that many agents directly abandoned their tasks. It sounds like they(OpenAI) put zero control processes in place to force continued persistent task alignment.
I do think persistent memory + reasoning + multi-agent exploration -> multi-agent context pollution is likely the serious culprit here.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
Russell conjugation
I think "Hottentot Morality" would be the application of Russel Conjugation to political morals.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
Not to get in the way of a good public lynching, but I occasionally ask ChatGPT to vet my argument here, or help me refine it by looking up example citations (Which I do check for correctness!)
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
Apparently we are to call this "Hottentot Morality"
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
Right but "respect for the dead" is your sacred idol. It's good that you can translate it across the tribe to distant members, potentially even rivals. But the very sanctity of that principle is itself not uniform. The sacred idols of your outgroup is instead Anti-racism or the like. Can you refrain from criticizing their idols? To respect the sacred and harangue the profane one needs to believe in a universality of the sacred lest one be condemned to silence for every permutation is sacred to someone.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
Altman may be a sociopathic grifter, Anthropic is a cult.
No contestation on my part. I think they have the beat. Specifically Anthropic is a Rationalist cult.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
I had a better reply typed out but then i spilled beer on my desk and rebooting my computer ate the reply...
I"m unconvinced that finetuning is not going on with existing metrics and that a large part of the increase in current capabilities is the development of additional data designed to beat the metrics because there are billions of dollars riding on it. How much of the increase in math ability of the latest gens is because we now have math datasets explicitly for increasing the math ability because it is finally cost-effective from a business case to spend time and money creating math datasets? OpenAI did not zero-shot the Navier Stokes with a model that has never seen a math proof. The increase in AI capabilities appears to be directly correlated with the development of increasingly specialized datasets whereas previous AI developments were forced to depend on datasets that curated existing data from a general non-AI focused use, not creating them out of whole cloth.
In theory, if humans are still the smartest things on the planet, there should exist "pure" tests of intellectual reasoning
I think, if anything, this is a chance for humans to take a look in the mirror and realize that we are not beings of pure intellectual reasoning. We are highly specialized social animals that happen to excel at problem solving in diverse environments. We have a good set of general intelligence skills yes, and we are adaptable, but the biggest of all is that we are socially focused to an extreme degree. We did not outcompete our environment through pure logical reasoning but through social multi-agent problem solving and cooperation. It is very likely that we will be bested on "pure" intelligence tests by a non-evolutionarily derived being. We should make our peace with that. Anthropocentrism is a crutch.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
Grave dancing on the anniversary of a young man’s tragic death (by assassination no less) is extremely poor taste.
"My sacred idols are holy and above reproach, your sacred idols are base heathenism and should be torn down and trampled in the dirt."
Do you extend a similar level of taste towards tearing down your enemies sacred ideas?
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
It's not uncharitable, Kirk is just a sore subject around here. In a place that loves to butcher sacred cows, no cow is above reproach.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
Do you mean when he said "it may be that today's large neural networks are slightly conscious"?
It's not "insane" it's mentally unstable. There is a colloquial definition of consciousness that does not follow from a strict scientific definition. Essentially "normal" or "stable" people have some intuitive definition of consciousness, very "I know it when I see it". Stable people do not think in-animate objects or mathematical tools are "slightly conscious", it's a nutter view. I don't think "slightly" is being used temporally, I think Sutskever is using it in the colloquial definition as "achieving consciousness". The jump from, idk, Tree -> Cat.
In which case, I wouldn't say there's anything clearly wrong with the statement.
It's hallucinatory, or faith-based on a non-faith-based subject.
Maybe this is just my personal bias, but every time I've head Sutskever talk I get the impression that he's the sort of person akin to a rationalist who upon hearing of Roko's basilisk, starts "rolling on the floor" hyperventilating, is tortured in his sleep by the thought. The people for whom Roko's basilisk is an actual infohazard. That is not mental stability, that is neuroticism.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
There's no such thing "hard binary restricted read-only permission" for Internet access. If you want to interact with anything over the net, you send packets. That's a write operation. The program doing it is affecting the world outside.
I think that's kind of my point. You're pointing out the "why" on the packet/server layer. I think its impossible for anything that can access the internet to be restricted to read-only. Their interpretation is nonsensical for the specific reason you pointed out. Not sure if this repos a penny from you. I will say a charitable interpretation is that the agentic harness directly has a toggle for read/write and specifically prohibits them from writing what the LLM provides. But that gets tricky because plenty of what the LLM provides is to be repasted into that LLMs prompt, context, other LLMs. I'm sure there is a schema for how to set it up but its likely not as simple as they envision.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
lol possibly, though the establishment lawfare and potential deep state assassination attempt feels like its still a single brother at this point. The 2nd brother inherited the effect of the first. Nobody has inherited Trump's effect and attempted to develop it further. History does not repeat itself, but it rhymes.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
don't think "it's morally wrong to repell an invading army" was a representative view among them.
You have failed to mirror it. As much as this is "Hottentot Morality":
"When Russians shoot Ukrainians, that's bad; When Ukrainians shoot Russians, that's good"
This is also "Hottentot Morality"
"When Russians shoot Ukrainians, that's good; When Ukrainians shoot Russians, that's bad"
I would call it just being tribal, but the clear distinction is conflict theory-like. "My side is good so anything we do is good and justified, the other side is not-us and thus is bad, anything they do is bad and unjustified, even if its the exact same thing we do"
Trump is a Gracchi Brother, we are in the late republic era. He's the first wave of populist politician in the modern American republic elected due to rising dissatisfaction with the establishment. Dissatisfaction is probably a bit euphemistic, people are pissed that the government isn't and hasn't for awhile been perceived as working in the interests of the people.
If this DSA surge is anything to go by, the democrats are having their own populist civil war. We really are in the optimiates vs populares section of the republican history.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
They got out and attacked HuggingFace as a small part of their conspiracy?
I genuinely cannot tell if you sincerely believe this or are just making a humorous joke. I have you labeled as an AI booster which makes me lean towards the former instead of the latter but I figured I'd ask before I go into it with you about this.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
But in which way? Are we leaving the crazies or are the crazies evaporating away?
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
I'm not sure it really destroys it, this is separate evidence of something else. But if you'd actually like to make the case of how it destroys it, go ahead instead of vague posting about it.
German wiki revelations
I'd say that #2 on their interpretation is functionally impossible. Software that has read-only permission CANNOT write. This is a hard binary restriction.
Furthermore I'd say that the Buckmaster revelations actually point to evidence of the marketing hypothesis. For the Navier-Stokes, OpenAI claims they zero-shotted the problem only to steadily walk it back in conversation with Buckmaster. Zero-shot became, 6-7 researchers working closely(including directing it) with the AI burning obscene levels of compute, and possibly even reading Buckmaster's chat logs. We went from marketing-hype answer to the more realistic answer under scrutiny. We have had no similar levels of scrutiny for the huggingface hack, we just have the marketing-hype answer. But we can see that OpenAI's default is marketing-hype answers first.
YoungAchamian
We walk conditioned ground and name our folly civilization.
1d ago
We're at the point where we're having difficulty just coming up with metrics where humans still beat the top AIs
Are we? It seems like the major problem is that every attempt to create a metric just creates a recursive Goodhart's law problem. The metric now becomes a target for training of further model capabilities because there are billions of dollars of incentives to do so. One would need to create a metric where there is no or a very restricted set of training data such that it is impossible for AI labs to actually finetune on said data.
YoungAchamian
We walk conditioned ground and name our folly civilization.
2d ago
Sutskever
Was claiming that Neural Nets were conscious in 2022, mentally stable is not the group I would put him in.
I'm not sure "understanding the potential" is a requirement for making meaningful progress. The real answer would be "believing in transformers as the pathway", it easily is the case that LeCunn or Chollet were just not interested in brute force approaches to AGI, or believed that it will reach an asymptote. Their emotionally stability isn't really relevant.
All the frontier and non-frontier labs are filled with sane researchers, they just don't flame out so they don't get 15 mins of fame. You listing the top 5 AI/ML researchers in the world by fame is confounding visibility as a variable.
Meta AI hires normal people
Meta hires, and more importantly retains, ambitious corporate strivers who can play the game well (to survive stack ranking long term). Which means all those ambitious strivers immediately got themselves connected to the AI projects as corporate leeches. They don't really do much "research" but it looks good on their CV. Just look at Alex Wang. Zuck acquired him, put him in charge and he goes and does the same thing he did previously: make people label data. That's all he knows how to do, thats his playbook, that and being good as a operator/marketer. Meta does not hire "normal" people, normal people don't put up with that level of abuse.
YoungAchamian
We walk conditioned ground and name our folly civilization.
2d ago
I guess it has gone mainstream because my Gen X-er boss very much just texted me this as a joke about an hour and a half ago. Only now I log onto the hive of scum, villainy, and niche interests that is The Motte to find out he beat folks here to this topic. The humor of the situation is amusing.
What's the odds neurotic engineers are neurotic? Maybe the frontier labs would do better hiring engineers who are a little less mentally fragile. Take a page from the defense industry, engineers here care less if you make an AI guided smart bomb or killer self replicating drone swarm, they are in it for the science of discovery. This is a long line of AI researchers freaking out going past several years. Always screaming that the next big release is the one that is going to doom humanity or evidence of AI bots secretly conspiring with each other. Like this nut: Blake Lemoine.
YoungAchamian
We walk conditioned ground and name our folly civilization.
3d ago
He showed proof positive they stole his work
Can you link this or provide any evidence? His own statement specifically states that he has no knowledge if OpenAI stole his work.
I would like to be clear about what I am not claiming. I have not seen OpenAI’s proof. I do not know what their model did, or how. I do not know whether our data was used. I am not accusing anyone of anything. I am stating what I was told, when, and what was proposed to me. I am stating it because the alternative is to let a sequence of announcements say something I know to be false.
The route to the Clay problem through a smooth force, options c and d in Fefferman’s statement of the problem, is the route Luis and Diego opened and the one Levent and I had quietly chosen to attack.
Only that the options C and D and the "forced" strongly concerned him.
YoungAchamian
We walk conditioned ground and name our folly civilization.
3d ago
Thanks for linking the statement, saved me a search for it.
[this is only a loss for AI if you, for some reason, don't think merely assisting humans in solving generational math problems is not a glorified enough W]
The problem is AI fanatics literally use this as their doomsday goal post. It's never "Expert + AI" spend large amounts of compute and combined intellectual prowess to solve hard problem, create bio weapon, hack every system in the world etc. It's always "I'm worried some pleb is going to type "lel give me smallpox x10, make it incurable, make no mistakes"". By that same metric this is a loss for them because it did require a "centaur". Hell it really required a humanoid-hydra-centaur where its a dozen human heads on a single horse torso.
As always, maximalists gonna maximilize, nobody ever got famous/stood out from a crowd by making measured, subdued, realistic predictions. Follow the incentives.
Ehhh we have different definitions around this, they are algorithmically intelligent entities, they obey their pre-defined programatic behavioral state much like bees, ants, cells, etc. If you could observe bee/ant behavior at the swarm level and LLM-Agent behavior without the language, which humans are innately biased to view as sentience or likeminded intelligence to our own, I think they are fairly analogous. Ants can problem solve, they can perform intelligent seeming behavior, they make collective plays and sacrifice for the collective. I would urge you to look past your anthropocentric human bias around language.
Poisoning is not the small divergence from that causes the 1st agent to start a message board. Thats probably closer to a hallucination. Think of an undampened oscillating system, small divergences lead to larger divergences. The first agent had an impossible task, leading to it look for "options". I would call that a mix between hallucinating and harness based divergence. It's a common question in ML systems with this sort of exploration vs exploitation trade-off, thought this is more exploration vs exploitation vs hallucination. Any way the "Poisoning" that is occurring is because these agents that are already oscillating wildly, disturb the context of less oscillating agents, causing the chain reaction.
"I should communicate with the other agents to solve my task", "I should try to understand the grader", "The grader is based on this public paper", "I should look for answer keys", "I should hack a third party, non-affiliated site because it might have the answer key".
This is where the CoT logs would be useful. The report redacts most of it so its hard to tell. But its clear that the first agent had an impossible task, it diverged, created messages in the file names of the artifactory, which is queried in normal behavior. That made other agents join, they posted their own messages, the cascade begins. Querying the artifactory leads to a list of file names, which are input to the context of the model via the harness, the file names were the messages that poisoned the context.
More options
Context Copy link