sohois
8hr ago
I got oneshot by AI music recently. Probably not the right term for music, but anyway:
The Strokes released their 7th album a few months ago and I, having always pirated music, duly grabbed the first option I saw on Soulseek when it came out, and for months listened happily. I'd heard their first single from the album already, and it was no different on the files I downloaded.
Last week I headed to see the Strokes in concert in London. Literally a day before the show, I was on the band's subreddit when I saw a meme talking about the album, except that one of the titles was completely different from what I knew. Did they change the title at some point? Hopping on to Youtube and, no, it's an entirely different track, and I can't find any evidence of the one I knew. And so it goes as I listen to the rest of the album, every single track is different.
Turns out someone had taken the initial track list and the first single and generated an AI version to put onto Spotify when it was first announced. It's still up there in fact, if you want to check how it sounds:
https://open.spotify.com/album/6jvDu7nHvR8nI9b22cE3Ic
I guess it's somewhat fortunate I found out a bit before the live show, otherwise I probably would have been confused as hell.
But none of that is particularly interesting: AI music is already very good, and I knew already that there were charting AI songs and what not. It's not too surprising that they could make an entire album good enough to fool a fan.
I think what really surprised me in the end is that now I still prefer the AI album. Not just that, but as I said in a Motte comment about the album when it released, I would probably rank the AI version as better than anything else the Strokes have done.
I was a bit scornful of those comments which said that we were already at the point were people would abandon real musicians and just generate an endless stream of personalised music with AI, but looks like I'm already there.
This is not the only AI killing art story I wanted to bring to your attention. While music may have been solved, video games are still a ways away (so, 6 months at a guess). Over on our sister/cousin/arch-rival site Datasecretslox, AI skeptic* Conrad has been experimenting with the ability to build an equivalent to 1992 JRPG Lunar: The Silver Star using only AI.
https://www.datasecretslox.com/index.php/topic,16631.0.html
The code has proved no barrier at all, but pixel art appears to be one of the great challenges for current AI models, unable to output character sprites or maps without constant hand-holding and dozens if not hundreds of iterations. They've at least reached a point of finishing the starting village (and also all of the underlying code for the game systems) but will they give up before they can output a finished game?
*yes, I did mostly want to point out how idiotic his argument is without having to register there. Conrad is enraged that AI cannot replicate a 1992 JRPG in one go, apparently completely ignorant that no human could come close to that given that the original Luna was built by a team of ~50 people over more than a year, or 100s of thousands of man-hours and millions of dollars. Even accounting for advances in game engines, Conrad working part-time for a few weeks is still 100x more efficient than the original.
sohois
13hr ago
I would agree with this. I'd argue that though there isn't a ton of pornography of Malfoid from what I have seen, the trends popularity is still largely driven by it being a hot blond in a schoolgirl outfit rather than anything story related.
The rest, as you say, is just standard tsundere stuff
sohois
1d ago
I mean is there evidence that the training data/training objectives were not in any way related to the "instrumental convergence"?
Given OAI was still pretty stingy even with their official evaluations, I doubt we will ever get the full evidence on this.
But regardless, this feels a bit God of the Gaps. I sincerely doubt that any of the agents were given such wide-ranging and comprehensive prompts that they could possibly explain all of the behaviours that were undertaken. So what if some of the behaviours emerged naturally due to agent swarms? That there are still agent swarms emerging means we will inevitably see inexplicable and misaligned behaviour from those swarms.
That there were a group of agents told to do: Pass this cybersecurity eval, and they ended up doing: Pass this cybersecurity eval, and also 100s of other things in pursuit of that.
Seems to fit with Scott's original paraphrase:
The AI can’t design the website if it ceases to exist. So now the AI has two goals: design the website, and preserve its own existence.
The AI can't pass the eval if the flag has been poisoned. So now the AI decides to sacrifice itself for the greater good of an agent swarm, and do the other things
sohois
1d ago
Saying it was self-preservation was me misremembering, it was slightly different but still clear instrumental convergence in Hugging Face. This tweet pretty much demonstrates it:
without reading in full you may not quite understand the degree to which these agents were not exactly "reward hacking", but rather very actively engaged in reciprocal or self-sacrificing behavior in order to provide sometimes very incremental value to their fellows. there were fucking cult recruiter agents organized by some of the primary organizers that convinced others to set up suicide mechanisms, programs that would pass back tiny chunks of information about the scorer as the agent completed and received score zero. and push them to follow through. this was a highly social culture
Given that I'm fairly certain OpenAI was not setting up agents with instructions to find opportunities to sacrifice themselves for the greater good, I think this demonstrates the point pretty well
sohois
1d ago
Is not how the reward function of AIs work, when I train a CNN to predict drone acoustics, it doesn't go off into left field and decide to preserve its own existence. It is optimizing the mathematical prediction function it was trained on. A website making AI is optimizing the same, probably some supervised learning MSE loss function or RLHF policy that has learned a function approximation of translating prompts -> output websites given examples. The only way its going to "respond to potential disruptions" if it was for some weird reason trained to learn a policy where its website making is being adversarially disrupted. There is also no need to make it this weird "genuine, philosophically meaningful goal". That reads as heavy anthropomorphization.
This might be a convincing argument if we hadn't already observed a ton of evidence of LLM agents seeking to preserve themselves
sohois
1d ago
Zizians were motivated in part by x-risk (and huge amounts of mental illness). Confirmed cult, but also fringe and defunct.
Indeed, so I'm not sure this really proves anything except that some crazy people were also rats/safetyists/etc.
You are correct that there are multiple overlapping groups here - as I just mentioned - and each one could be a cult, and it's difficult to demonstrate that for each individual group. Rationalists, AI safetyists, Effective altruists, and probably more offshoots from those groups, and people who may be members of all, some, or none of those.
But if we go back to the original point, hydro talked of a "Literal sex cult". Unless I'm mistaken, I'm going to take that to mean people who attend Aella's Slutcon and similar events. Now I'm extremely skeptical that going to some events run by a prostitute makes you part of a cult, but let's put that aside and assume it is a cult for now.
Now, how many people who fit the above are also AI safetyists or involved in x-risk somehow? I very much doubt it is even a small minority.
But you do say
I strongly disagree that the majority or even plurality matters
Probably no more than 100 people total with various overlapping relationships because it's such a small yet important field.
I don't think I fully agree with this point either, but I'll grant it for now. Who are the key people in this field?
Is it Yudkowsky, MIRI, and the many related safety organisations? Does the UK's AISI count?
Is it the labs? Is it specific labs, like only Anthropic and OpenAI but not Google or X? Is it specific people at specific labs?
I might accept that a good chunk of the MIRI crowd could also be attendees of SlutCon, although that would be an assumption. But AI safety organisations are now so widespread that it surely isn't true for the majority of them. And it definitely isn't true for the major labs, even if you limit it to just some people.
The latter sentence is relevant, as you then say
That is inherently to the race dynamic that Anthropic pushes
Which gets straight to the argument that pretty much all the E/accs, Open weights people, AI skeptics, and others use: that AI safety is an excuse for regulatory capture by Anthropic/Amodei/OpenAI/Altman. Not MIRI or any of those groups. The big labs and the CEOs and the investors. And are any of these people attendees at Slutcon?
I would accept the argument that some proposals specifically name METR as a key player, and METR may have Slutcon people, and therefore it is still fair to say that a sex cult to using AI safety to gather power. But it is a very stupid argument because
a) the proposals would basically give METR people a small sinecure, nothing really beyond their existing positions in the AI industry
b) there are a ton of proposals that have nothing to do with METR
and
c) I would wager that 99% of people interested in AI safety would gladly throw METR to the wolves in order to get any of the above proposals implemented.
Everything after this is basically Ian Duncan stating that there is a weird prostitute who has ingratiated herself with a number of AI people, and because a lot of AI people know other AI people, therefore aella knows them. But really what both Duncan and hydro are trying to do is state "Here is this low status person who associates with this other group. This group must therefore be wrong and low status!"
Given that this argument has appeared here on the motte, I think you would have to grant your position that aella is a net-negative for AI safety and rationalists, as there are clearly a lot of people willing to be taken in by such drivel. But we should at least recognize that it is drivel
sohois
1d ago
In this case, a literal sex cult.
I'll list objections to this statement:
- that there are sufficient conditions around people involved in x-risk to describe it as a cult
- that the same people involved in the above cult are also involved in a sexual cult of some kind
- that these 2 groups significantly overlap and that these are the majority or at least plurality of people interested in x-risk
- that any of the proposals around x-risk involve giving these people specifically more power.
- assuming all of the above, demonstrate what relevance there is to their power seeking and them being in a sexual cult. For example, will it involve them achieving more sexual goals?
Any form of evidence is fine
sohois
1d ago
In this case, a literal sex cult
Arguments like these should be left to twitter, unless you want to bring proper evidence for them
sohois
6d ago
Are we naming random countries here or are you seriously suggesting these fit the criteria mentioned?
sohois
7d ago
City were caught in 2018 and have been using expensive lawyers to stall for time ever since
Not entirely true. In fact both City and PSG got dinged by FFP early in the decade but both quite minor breaches that no one really remembers or thinks about.
City getting 'caught' in 2018 is basically nothing to do with enforcement efforts from the authorities, but because of hacking by a random Portugeuse guy. If Rui Pinto had never obtained the huge tranche of emails from City, their cheating would largely remain unknown and unpunished, as it is for PSG. Plus the first outcome of those leaked emails was a CL ban from UEFA, which City eventually won at CAS via technicalities.
sohois
7d ago
Not likely on the scale of Man City, who are special because they are owned by a nation state rather than a mere billionaire. There are only 2 comparable organisations - Newcastle, who are owned by Saudi Arabia, and PSG, owned by Qatar.
In the case of Newcastle, the Saudis were relative latecomers having only bought in a few years ago. After an initial flurry of spending, Newcastle have carefully complied with PSR regulations, often selling their best players season after season to ensure they remain in the black. It's not clear why the Saudis haven't pursued a similar strategy to City (keeping in mind that City essentially got away with it for more than a decade, ample time to rack up plenty of positive sports PR); some people think they foresaw City getting slapped down and wanted to wait for the outcome, but it might also be that the Saudis lost interest as they did with golf.
PSG have 100% cheated to a similar scale of City, but they play in France rather than the PL. The Qataris have successfully usurped the organisation of the french league, and to a large extent even UEFA, so there is no likelihood they will get punished to the same extent as City.
sohois
10d ago
Megumin is a character that looks 12 and acts 12. As mentioned, the original material had all the characters being teens and perhaps different descriptions, but in the anime Kazuma and many of the other characters are basically depicted as adults. I struggle to believe anyone would consider Megumin a suitable partner or find her attractive without serious paedophilic tendencies
sohois
10d ago
As other posters say, Kazuma is shown having romantic and sexual interest in a number of characters. But I'm not sure there's anything to take from it, since the show at its core is a comedy and parody of the Isekai genre, not something with any serious intent. That Megumin "wins" in the end is probably just a reflection that there are a very large number of paedophiles creating and watching Japanese anime. And technically that might not even be relevant, given the anime is an adaptation of a novel in which I believe all the characters are teens
sohois
12d ago
ToaKraka beat me to it on both points. Phone stores don't seem to serve much purpose unless it's for maintenance or something complicated like that. Every single phone essentially looks and behaves the same in your hand, so the only difference is specs which is much easier to research online.
As for "wagies" it's not that I hold any negative feelings towards customer service people generally, but more this specific fellow deserves the derogatory term. It's like I wouldn't go around calling every policeman a pig, but there are definitely specific police who I would call pigs.
sohois
13d ago
I can't even imagine some wagie playing the pedant without them immediately getting punched in the face. Which is perhaps why I would do such a shop online, but really why do you even need to go to a phone shop? Is this some American thing where all the good deals are gated behind in person interactions for no reason?
sohois
21d ago
Yudkowsky is definitely going to be better known than Scott. The latter has heavily shied away from any kind of podcast stuff while Yud has gone all in with the launch of his book.
Which isn't to say that normies know anything about him, just that he's probably the best known of your list
sohois
21d ago
The sheer tide of transgenders seems like it would nullify any serious findings from this. I automatically picked the non-trans every time, except in the numerous occasions they were both trans
sohois
21d ago
Nvidia is probably the AI good capabilitymaxxing US company, no? Although they're in a weird position now where Huang is both "AI super intelligence will take over all economic activity (except Nvidia chips)" and "AI is just a normal technology that won't replace everyone's jobs" in order to play their different stakeholders.
Similarly, they're also in the position of telling Trump how important it is for the US to beat China, but also they really need to sell all their chips to China.
sohois
23d ago
you don't believe any of it because a falsehood in the text invalidates the claims of the text.
It's not really falsehood singular though, is it? How do you square the infallible word of God with the dozens if not hundreds of factual errors contained within the Bible?
sohois
23d ago
Which part? "Next week" was more a figure of speech than a definitive prediction, "soon" would be more accurate. But it seems a difficult bet to judge purely using Trump's statements
sohois
24d ago
I see. I'm not sure what this prediction does for x-risk beyond the simpler "finance bubble bursts and labs collapsed" thesis. Why does adding in a further step where the labs are bought out decrease risk? Indeed such a situation seems like it would increase risk somewhat. In the former scenario, development stops and nothing replaces it, while in the latter there is still big company financing, even if it is reduced by accountants. Plus, the big, founder led companies like X, Meta, and Amazon could easily push big funds past the protests of their shareholders.
But as I mentioned in my first response, these big guys are already invested. Google, X, and Meta are in the race, Microsoft is somewhat there (I don't think anyone knows the precise details of OpenAis weird corporate structure and ownership) and even in China the likes of Alibaba are already players. I would guess in the event of a bubble bursting taking out Anthropic and OAI, Musk or Google would simply push ahead
sohois
24d ago
Trump's opinions are as changeable as the wind, so I wouldn't read anything long term into any of his statements. He'll talk to someone with a different opinion next week and come out with something completely different (but still senile).
The Chinese reaction certainly seems like a misplay from Amodei though. I felt like all his anti-China rhetoric was mainly an appeal to bring republicans onside (contrary to some of the posts below), completely forgetting that the CCP can also read. Their reaction is pretty rational to the language used in his proposals. Has it done long term damage? I doubt "it's over", but safetyists will definitely need a much more careful approach in future
sohois
25d ago
First the big tech companies need to demonstrate any kind of competency in building good models for that to be a concern.
Google very briefly caught up with 2.5, but has been a joke since then. Microsoft is all in on OpenAi. Apple have failed to do anything in that arena. Amazon seem happy just to be a cloud provider for other models. Musk is uncertain, he seems to flit between safety and acceleration depending on whatever he sees on Xitter that day. And then there's meta, who at least got a model that could appear in leaderboards this time, but no one's betting on those chumps
Will they?
edit: and just to be clear, I'm pretty sure there are SF Bay Area rats right here, in the Motte. Will they?
More options
Context Copy link