Shrike
34m ago
It is still a problem with whatever models people are using. Just had to deal with a pretty bad incident at work a couple of weeks ago (thankfully not one due to any failures on our part).
Shrike
36m ago
elevator operator.
Hilariously I've been in elevators with an operator.
there is a silver lining for accountants: knowing which rules to bend or break
I'm pretty sure another silver lining for accountants is that "my freakin' AI did it wrong" is probably not an excuse that the IRS will accept. They will accept "my accountant did it wrong."
Shrike
37m ago
I can only assume that you haven't of the Vogons.
Surely that sort of thing is in the domain of the CIA!
So they do have incentive to cram as big models as possible in as limited hardware.
You're sort of missing my point - the goal here is to cram the most optimized models - that is, the models that are going to best allow them to do meshed swarm coordination, or whatever. Larger is not always better; better is better.
I have no particular reason to think that the direction LLMs research is currently developing is good at all for that sort of thing. Cramming Astra into a drone is going to result in something that's much slower and less decisive than a simple deterministic script, because LLMs are compute-hungry. That's not to say that neural networks won't be helpful - computer vision is great, for instance.
But LLMs are not the only form of artificial intelligence, and for most military tasks a strong general-purpose LLM is both overengineered and poorly suited for the mission, especially the mission of terminal target determination. For intelligence synthesis, they likely have some promise.
Shrike
4hr ago
All the world militaries and terrorist groups have great interest in making full fledged AI work in the limited hardware a drone possesses.
Eh, in the sense that the AI can do independent target discrimination, sure, but that's already a solved problem (LRASM can do this, for instance) although it can be iterated out further.
World militaries and terrorist groups have no interest in making sure their drone processor can write poetry, or whatever.
Shrike
18hr ago
A conventional war in an environment saturated with large underwater drones, perhaps.
Perhaps (particularly since the US is looking into a low-cost heavyweight) but...over what?
I maintain that there are no plans to have a nuclear war or any war over Taiwan.
This is a sort of ambiguous statement. Are you saying that the US does not intend to start a war over Taiwan, or that it has no contingencies in place for waging a war over Taiwan?
Shrike
19hr ago
Nobody is getting ready to stop the Chinese invasion.
You can tell what war a country is planning to fight by following the money to their munitions procurement. In the case of the United States, if you follow the money, you will notice (among other things) that they intend to increase production of torpedoes.
Are they likely to be low on torpedo stockpiles? No; a quick look at Wikipedia tells you that they had 1,000 heavyweight torpedoes stockpiled in 2001, they have continued production at a low rate since then (50/year as of 2017, Wikipedia thinks, so you can estimate total inventories somewhere between 1,000 and 2,000).
Background knowledge tells you that training torpedoes are reusable, so their burn rate from training is low. The entire US submarine fleet can only hold 1,500 torpedoes or so, even if they were all deployed at once (they won't be; a quick Google tells you that 20ish attack submarines will be out of service at any one time) and were only carrying torpedoes (they won't be).
The entire Russian Navy is about 70 submarines and 100 major fleet combatants. The entire Chinese Navy is about 75 submarines and fewer than 200 major surface vessels. (Not including minesweepers or amphibious assault/transport ships in the count here, or, say, patrol boats, but including some low-capability combatants such as corvettes.) In any conflict involving one or both of those opponents, the US would not rely on its submarines alone to sink their ships, and it would be fighting alongside allies.
So one can conclude that the US is preparing to fight a war against ocean-going opponent(s) at a scale so grand that they anticipate their attack submarines will expend their entire onboard munitions stockpile, return to port, reload, and sally forth with more torpedoes.
Based on that evidence, what exactly would you say they are "getting ready" for?
Shrike
1d ago
Yeah, as a religious and relatively conservative person, I would say that the natural role of a parent is to take care of their child and a child to be instructed by their parent. Thus, while the parent (or rarely the child) can do something bad enough to warrant severing that bond, severing the parental bond is itself an act of violence and should only be done in extreme circumstances.
Thus I wouldn't want to take kids from "woke communist parents" or whatever - even if I was absolutely convinced that they were wrong, taking kids away from their parents is a good way to convince them you are wrong. Whereas I think that a good parent-child relationship and familial relationship serves as a model and a stepping-off point for other relations in society. Turning someone towards religious truth (or social or political truth, to the extent that there is such a thing) is, I think, more difficult if their first relationship (with their parents) is disordered. Thus, while a sufficiently disordered relationship may be cause for separation, disordering a sufficiently ordered relationship may itself make a person's relationship with society (and God) more difficult.
Hopefully that makes some sense.
Shrike
2d ago
Also he left OpenAI well before they expected to be making any money at all, all the way back in 2021.
So he left his job as an employee at a nonprofit to start a for-profit organization and I am not supposed to believe this is an obvious financial trade-up?
I don't think humans are entirely motivated by money. But (at least in America) quitting your job to make more money is quite routine.
Which narrative is that?
I'm not sure this means chooky is wrong – training and inference are different tasks, and so don't (necessarily) run on the same datacenters/GPUs. So chooky could be correct that they are slowing down on unsustainable capex (training) but that doesn't necessarily mean that pausing unavoidably means less capex spend total, because you could pivot to building inference datacenters to serve customers with existing models, shifting from expensive R&D to increasing revenue via selling inference. Right?
Now, caveat – I don't know of the top of my head what the current datacenter build out looks like – how much of it is inference versus training, what's being financed directly by the primes versus third parties, etc. etc. I gather that you're not familiar with it either (and I'm not sure it's all public knowledge, anyway). So I am just thinking about the logical levers at play here.
Shrike
2d ago
pausing unavoidably causes less capex spend
Why would this be the case, in your telling? Why wouldn't AI companies pivot towards customer compute build-out?
Shrike
2d ago
Is Amodei making more or less money now than when he resigned from OpenAI complaining about safety?
Shrike
2d ago
Can you be a bit more clear about this? Who intentionally works hard never to associate with who?
Shrike
2d ago
I will just add that a hugely underrated aspect of belief formation is your friend group and social circle. (Thankfully a professor of mine mentioned this to me relatively early in my life, and I have been blessed in some other respects as regards my career and geographic placement).
Who you are around shapes who you are, including your beliefs and your moral values. Your friends and whoever you spend time interacting with control your own personal Overton Window. Certainly many people are capable of withstanding the compulsive power of Overton Window but it's not generally a bet you want to take or a position you want to place yourself in. Bad company corrupts good morals. And you make a choice as to whether or not you spend time with bad company.
Shrike
2d ago
Imagining being an Anthropic employee but putting "do not give me AI psychosis" into my personalized prompt so I never have to worry about it.
Shrike
2d ago
what exactly is the benefit of blowing up tankers when Iran is already under blockade?
Tankers are used to store and transport oil, which is a major Iranian cash export. By destroying them, Iran's adversaries reduce Iran's post-blockade economic potential.
It also threatens existing oil fields in a somewhat indirect manner. Shutting down an oil well is a potentially irreversible process, so best practices are to maintain an existing oil rich area is to keep pumping oil. But Iran only has so much oil storage, and the blockade was forcing them to store excess oil in tankers. Every time the US strikes an Iranian oil tanker, it constrains Iran's future economic prospects, and my guess is that it squeezes Iran's already tight storage capacity.
Shrike
3d ago
Surely the people at AI companies don't spend a lot of time conversing with AI, right?
Shrike
3d ago
Supposing that we 100% accept this as true, the result here is not that we should assume AI companies are being sincere and honest; the result that we should assume is that AI companies are intentionally making the AI capabilities that they have seem more frightening (and capable) than they really are, because the precise concern of AI alarmists is that the alarm is raised too late.
Thus there is an ideological incentive for AI companies run by "AI concerners" (let's call them, instead of doomers) to
- Overstate the capabilities and dangers of their models, and
- Kneecap competitive models
- Monopolize the market and institute a nanny surveillance state to ensure that nobody anywhere uses products besides theirs
The fact that this dovetails with their economic interests, of course, just means that there are multiple forces pushing them to be deceptive and/or self-deluded.
Shrike
3d ago
I also think the primes have a vested interest in putting a stick in the eye of open-source models, regulating them out of existence or at least sidelining them. There's been a recent trend of adopting bespoke models based on open-source offerings: Harvey, Thomson Reuters, and of course Palantir's data sovereignty activism, and a common thread here is saving costs.
For Anthropic and OpenAI, 80% of revenue reportedly comes from 1% of their customers. Thus, well-funded organizations with big spending training their own models is a potentially existential threat to the primes' business model.
Shrike
3d ago
If the Dems hold both houses, or even get enough Republicans to flip on the votes in question, they could defund the war. Literally attach riders to budget legislation saying no funds can be used in X region after Y date.
That probably would work, albeit slowly, and I wonder if by the time the new Congress seats if they won't be distracted by other outrages of the day (AI?) and/or cowed into submission by scary intelligence briefings.
There's a bunch of other things they could do like actually invoking the War Powers Act
I think there is hesitancy about doing this because they don't want SCOTUS to rule the WPA unconstitutional, granting POTUS more power.
Shrike
3d ago
Do you actually think that the Democrats will give Iran nuclear weapons? Like, pass a law gifting them weapons from the US stockpiles, or something?
If not, what concretely do you think the Democrats will do? Winning the midterms does not make them President.
Shrike
3d ago
Oh, I'm not saying anachronisms based on "100kya doesn't match up with the dates for Cain and Abel", but rather "100kya doesn't match up with the dates for farmed fields existing".
Oh, I gotcha. Yeah, I'm never super sure how seriously to take even our best date estimates for [thing], in part because they keep getting shifted around. I have a friend who works in a geology-related field, and he says dating is heavily reliant on assumptions, too. I certainly don't think we have enough geological evidence to rule out "a farm" 100kya, but it does look like you're correct that farming is believed to be much much more recent than, say, stone tools.
But I brought it up more as an example of the sort of thing I find interesting to toy around with, not (obviously) a systematic and well-thought-out theory I have.
The initial "agriculturally developed Canaan looks like a land flowing with milk and honey to pastoral Israelite nomads" vs "but you have to learn some of their practices if you want to keep cultivating and harvesting it all" conflict alone
Wouldn't the Egyptians also be agriculturalists by the standards of the time? My understanding is that the Creation myth in Genesis is thought by some to be essentially a refutation of the pagan creation myths the Israelites were likely familiar with, and it makes me wonder if the Israelites saw the Egyptians in Cain.
(Although interestingly one could argue that Egypt is portrayed fairly positively in Genesis.)
Shrike
4d ago
There's this "historical evidence on Sparta was largely written down by Sparta's political opponents" factoid that I recall of note.
Sure, I think careful scholars are cognizant of that sort of thing.
Early writings on Christ are interesting because we both have early writings from Christians (who mostly didn't profit much by believing in Christ as per their own writings) and writings from neutral or hostile third parties.
I haven't read as much of the early writings from and on Christians as I would like, but from what I can tell the pro- and anti- Christian writings often read exactly as you would expect, e.g. the Christians are writing about church discipline, doctrine, and defending their beliefs as intellectually legitimate and their opponents are calling them gullible retards from first principles.
I get the impression that the main difference between second century theological debates about Christianity and Reddit today is that a lot of Christian arguments have remained the same (sans the ones aimed at the pagan gods), whereas "Christ was not the Son of God, merely a magician" or "Christ was not the Son of God since God would never come to Earth as a peasant" have fallen out of favor. "Christianity will destroy our way of life because they are opposed to infanticide and licentiousness" still has some traction though.
Shrike
4d ago
The debate at that link is mostly about what may have happened around 100kya, way too early for the Cain/Abel story to work without being full of anachronisms.
Setting aside the question of whether the archeological timeline is correct, isn't it true that in Genesis you only get to the more recent dates if you interpret ben as "son" and never "descendant"?
Cain vs Abel is a pretty good parable for the conflicts of around 10kya, though, between early agriculturalists and pastoralists.
A conflict that might be relevant to the Jews at the time Genesis was written.
Telling a story whose details aren't literally true in order to aid understanding of a deeper truth was like God's entire M.O.! What else would we expect? "Okay, before primordial nucleosyntheis, when the cosmic microwave background hadn't even red-shifted yet, the photons were ... are you following any of this? ... okay, forget it, just write down: 'Let there be light'."
Yes, I 100% agree with this.
I just think God is sneaky and capable of pulling off multiple levels of truth. In the afterlife we might all be like "oh wow that was all true in ways we didn't even think of evaluating."
I just don't think sticking Astra or Fable (as opposed to a lightweight image processor) is the solution here.
The problem with a lightweight tool is that it can be fooled by changing small details (e.g. using Ukrainian spelling on your Russian tank). But those are exactly the same sorts of things that are going to trip up Astra, too, because it is going to be instructed to use a similar decision matrix, which, again - why not just use a lightweight image processor if you are going to give them both the same targeting library.
In many ways, I think militaries would prefer a tool that isn't doing independent reasoning to strike a target. The reason for this is pretty obvious: independent reasoning historically leads to friendly fire. At least if your independent reasoner is on the bridge of a ship (and this goes for a human or a larger AI model) and it screws up, you can figure out what happened by conducting interviews and data-dumps afterwards.
If you design a missile with an independently reasoning seeker-head and it starts friendly-firing, you may have a hard time figuring out what is going on because there may be nothing left of the
malfunctioning"misaligned" system to evaluate and its reasoning may be too complex to review in a timely manner. So if something happens, you might have to yoink an entire production line of weapons and/or re-train your model in the middle of a war. If a deterministic program is friendly-firing, it means:All of which are probably easier to figure out and patch during an active conflict than "what is going on with Astra" - witness the difficulties the AI companies are having following chain-of-thought now, when they have total access and control over the models.
I am not saying that we will never get the fabled "self-aware weapons system" from AI, and I wouldn't be surprised if testing is done relatively soon. But in the current threat environment, it strikes me as buying a sledgehammer to kill a fly.
That's not to say that LLMs have no military value at all. I just don't think they are optimized for weapons guidance.
More options
Context Copy link