@DirtyWaterHotDog's banner p

DirtyWaterHotDog


				

				

				
4 followers   follows 0 users  
joined 2022 September 05 16:31:20 UTC

				

User ID: 625

DirtyWaterHotDog


				
				
				

				
4 followers   follows 0 users   joined 2022 September 05 16:31:20 UTC

					

No bio...


					

User ID: 625

Haha, yep, tables and rich extraction is pretty bad out of the box.

In this case though, I can confidently say I'm an expert on PDF extraction for llm use.

Starting a greenfield project to build a stable software environment for ai coding. I can't think freely when operating within legacy codebases. So starting from scratch to build my ideal ai-native scaffolding.

I'm starting with a tool to auto-redact pdfs. Simple, useful, well constrained. Would appreciate suggestions on software paradigms that have worked well for ai development.

Stack:

  • Python FastAPI backend + Streamlit Frontend + Sqlachemy ORM over rel-db
  • uv for packaging + environment
  • firebase for cloud provider + github actions for ci/cd
  • prefect (or something similar) for orchestration
  • Openai codex + github copilot as my LLM coding friends
  • Dockerized deployments

Some ideas:

  • Monolithic codebase to make it easy for agents to operate on it
  • Minimize implicit everything (state, side effects)
  • Maximize explicit everything (types for everything, explicit validators)

I have a basic demo ready. Codex is already raising PRs. The redacted bounding boxes are off. And the LLM redaction logic is wonky. But, so far I am impressed at the LLM's ability to build a greenfield project by itself.


I'm a serviceable software engineer. Cracked engineers of the motte, what are some software systems paradigms that you think I should play with ? I would especially like to know paradigms that make it easier for agents to understand, write & verify auto-generated code.

For Hall of Mosses, remember to get there early. Parking can be tricky if you get there past 8-9 am.

  • Hoh Rain Forest and the short hall-of-mosses hike is unmissable. If you had to pick one. It has to be this. There is no place in the US like Hoh-rain-forest. Olympic national park is massive.
  • My 2nd recommendation would be Hurricane Ridge and the Hurricane hill hike. Great drive, great views.
  • Rialto Beach-> Hole in the wall is the perfect long stroll. The day I was there, the Pacific was roaring. Gives you newfound respect for this behemoth of an ocean. No other Ocean compares to the Pacific. The hike doesn't end at any point. You can keep going forward, but be wary of the tides.
  • Cape Flattery is a nice short walk. The drive is amazing, if a bit out of the way. The views at Cape flattery are beautiful. And for cookie points, it's the lower-48's north western most point.
  • Mt. Storm king is a short but demanding hike. The views are unbeatable. The last section has ropes and is a bit vertigo inducing. Still recommend it. I generally love this area. There is kayaking near lake crescent. And Granny's cafe has good food, as does the lake crescent lodge.

Some tips:

There are nice waterfalls in Olympic, but Oregon has the best waterfalls in the country. So nothing I can recommend would impress (Solduc, Marymere). Definitely spend one day up the Columbia river in Oregon.

On the way down from the Olympic, stop by the world's largest Sikta Spurce and the Tree of life. They should be 5 minute stops each.

Don't stop by Forks, Washington. It was popularized by Twilight. I passed through it, and the town made me feel uneasy. It's probably safe, but has the air of a sad place that's been left behind.

Acc. to Yougov, the older powerful female politicians are - Hillary, Kamala, Pelosi, Warren. The powerful female democrats are all angry. I'm surprised that no other female archetype has succeeded in US politics.

In India, I've seen other archetypes work out. Indira Gandhi represented the stoic & steeled strongwoman. Jayalalitha went from film star to holy-mother of sorts. Even the shriekers demonstrated excellence in verbal combativeness. Sushma Swaraj & Mamata Banerjee had a sharpness about them that I have not seen from female American leaders. (AOC is growing into it, but she isn't a traditional democrat).

Now that I think about it, the 2 party system seems to have lot to do with it. Indira Gandhi had to win an internal civil-war to rise to power. Jayalalitha & Mamata Banerjee built their own cult-like 3rd parties to ensure internal loyalty. Unconventional candidates need to find the space to build an army of unconventional loyalists to usurp power. In that sense, AOC seems to be doing all the right things. Alas, she supports some of the most braindead policies and politicians.

Missing Petes - Where are the 30-something liberals?

This write-up was prompted by Zohran Mamdani’s rising popularity in the NYC mayoral race.

Pre-2016, American politics was run by boomers. As the youngest boomer, Obama was expected to pass the baton to the next generation of Democrats. Alas, geriatrics returned with a vengeance, and Gen-X tapped out for good.

Of the dominant American political groups, I'm most sympathetic to neo-libs with a YIMBY flavor. Therefore, I’ve kept an eye out for Millennial newcomers who fit into this mold. 'Left of center with accommodations for changing times' is a tried and tested formula for fresh Democrats. It started off great. Tulsi and Pete had respectable presidential runs for their age.

Then began the woke revolution and the COVID crisis. During this period, I expected radicals to be ascendant, and they were. Progressive Millennial faces were introduced through 'The Squad,' prison abolitionists, and protest movement leaders. All positioned in opposition to the neo-lib incumbents, all terrible policymakers. Thankfully, the progressives haven’t won anything at the national level just yet.

Their mortal enemies, the Boomer neo-libs (Kamala, Biden, Blinken, Pelosi), ran the nation for four years. Most of it was in a post-woke era where the nation was shifting to the right. Yet, we saw no new neo-lib faces during that time. At both the national and local levels, less-progressive democrats like Tulsi and Ann Davidson were pushed out despite their popularity, as proven by their rise in the Republican camp.

Train-man Pete is the obvious exception. But where are the other Petes? If boomer Democrats dislike AOC’s allies, why haven’t they groomed any young leaders of their own? Have boomers reinforced the stereotype by once more pulling up the ladder behind them?

I ask rhetorically, of course. The answer is yes. Boomers crushed the political prospects of an entire generation behind them. Millennials weren't going to have it any easier. The sheer greed of 80-year-old geriatrics is embarrassing. No policy goals left to pursue, just a legacy of corruption and unmet promises.

I dislike Zohran. Among my fellow Indians, he is what we call a 'chutiya' (hard to translate; the closest synonym would be wanker). Yet, I feel dirty saying anything positive about Cuomo. Do the two options have to be a corrupt neo-lib boomer versus a Millennial wanker? As the boomers die off, who will take their place in Democratic power structures? Because from my perspective, all the young leaders are socialist wankers.

So I ask again: Where are the other Petes?

US facilitates illegals in their country

Couldn't agree more.

Moreover, politicians on both sides phrase their rhetoric around immigration without regard for whether it is legal or illegal. This creates a system that is annoyingly hostile to legal immigrants, while being uniquely welcoming of illegal immigrants.

Require some proof of being legally in the US for opening a bank account

This is where the issue comes in. Who enforces it ? There is bipartisan opposition to federal overextension. Each state operates as its own pseudo nation. Consensus is impossible. Don't even get me started on the absurd amount of power dispersed through city govts and unelected courts.

The US political system is built with paranoid protections against authoritarian powers at the center. It's gridlock taken to an extreme. Nothing gets done in the US, because everyone and their mother holds veto power. I'm not an US citizen. But, power is so dispersed, that politicians get away with pointing fingers and doing nothing. All sorts of local veto roles are elected. These roles are decided by low turnout and susceptible to political capture (Soros playbook). The US is unique in having elected state judges, sheriffs, superintendents and more for some confusing reason. If a mayor or governor has such limited control over local law, policing and education then what use is their election ?

California has been unable to build a highspeed rail envisioned 50 years ago despite throwing $100 billion at it. States can't agree on a legal age for sex. What makes you think this system is capable of executing on a nation-wide ID proof platform ?

This isn't a war between Israel and Iran. It's Netanyahu vs Khamenei. 2 unpopular leaders living on borrowed time. (1 literally, 1 figuratively).

Netanyahu wants to leave behind a legacy. 'Securing Israel's safety from nukes for a generation' seems to be it. Khamenei cares less about regime change per se, and more continued existence of Iran as a clergy driven autocracy. Khamenei has consolidated power for 35 years. His succession struggle was bound to be full of conflict. This is before 2025, when his civilian and IGRC right hand men both passed away. His priority is for that successor to be one of the clergy and not the armed forces. (Armed forces != IRGC) (I don't believe an arab-spring-like color revolution is on the cards).

Iran and Israel will survive with limited damage to civilian infrastructure. Regime change is likely to be a good outcome for both nations. Over their long reigns, neither leaders have acted in their nation's best interest. What's at stake is which leader's legacy will be remembered as a positive one.

At this moment, Netanyahu looks to be succeeding. Sure, his actions have fanned the flames of antisemitism, but he won't be blamed for it. On the other hand, if Khamenei gets a non-clergy successor, then this new govt won't remember him fondly. Today, non-clergy succession is the top choice for prediction markets.

To your point, if Khamenei was in good health, then Israel would not have been able to get much done. Instead, they're adding a straws to a weary camel's back. Khamenei is old. His chosen successor Raisi is dead. Khamenei's son is weak and beholden to IRGC, who've endured the biggest losses over the last few days. (intentional by Israel). The youth only knows this regime, and doesn't share the previous generation's fervent hatred towards Shah or the US. Lower religiousness means lower allegiance to the clergy. That's a lot of dry straw. The spark is all that's missing. (I like my malaphors)

Trump looks to be in a good mood to make 'deals' and doesn't have the same obsession with democracy that the liberals have. If Iran agrees to audits of its nuclear sites, then I can see Trump backing a less hostile (ie. non clergy) regime. Israel would agree to any non-IRGC leader as long as Iran is under nuclear surveillance. In contrast to liberals, Trump doesn't need this regime to look like a liberal democracy. This gives Iran a lot of leeway for what the successor regime can look like. IMO, the obsession with democracy was the downfall of liberal/neo-lib orchestrated revolutions of the last 30 years.

I meant October 2023.

"Don't paint the devil on the wall"

For a while now, the Left has made a past time out of calling Israel every bad name under the sun. In contrast to those accusations, Israel has behaved honorably in victory. Over the last century, Israeli moderates have proposed many 2 state solutions despite overwhelming victories in wars that were started against it. They've withdrawn from territories they've won and prisoners of war were treated in line with the western standard. Despite every war being started by the Arabs, the left labels Israel as the evil ones.

More recently, (Sharon) acted with generosity by withdrawing from Gaza in 05. In return they got rewarded with Hamas. Through the Arab spring, Muslim nations performed the worst acts of violence on each other, as the western left cheered on the revolutionaries. During this period, Israel remained a peaceful place for its resident Arabs. Yet, 2 newly empowered enemies emerged with self-professed genocidal intent (Houthis and Hezbollah). They're armed by Iran, who through proxy, attests to the same genocidal intent. Once Iran starts developing nukes, the west once more, tried to extend an olive branch. JCPoA (Iran Nuclear deal) was signed. And once again, this generosity was rewarded by resumed development of nukes. Yet, in the eyes of the western left, Israel remained the evil one.

This is where the the first domino fell. Netanyahu solidified his power because the Israeli left was left with no political space to maneuver in. Israelis hadn't changed, but the clearly rising antisemitism among the western left and its Islamic neighborhood pushed Israelis to vote for the one cynical hawk in town : Bibi. While politics shifted right, the average Israeli remained a normal person. 2012-2023, Israel greatly expanded labor permits so Gazans could work on the Israeli side. (~200k daily cross border workers). At home, things were stable.

Then you got, Oct 23. Frankly, the reaction to the tragedy was despicable. I was shocked by the complete lack of empathy from elite western institutions and a "they had it coming" undertone. I think this broke the average Israeli for good. Imagine if your daughter got raped and murdered. Then your friend says "she had it coming". I know I'd see red. A century of accusations being called the devil. If you're going to be called evil either way, might as well go scorched earth and solve the problem once and for all.

Think about it:

  • If Iran's nuclearization is inevitable, then why stop at precise assassinations? Makes more sense to cripple their nuclear infrastructure for good.
  • If a ceasefire with Gaza means another Oct 23 in a decade, then why not raze Hamas for good ?
  • If the next generation of politicians are going to be antisemitic, then why not conduct major military actions while the boomers are still alive ?

Trump's truth social posts suggest otherwise.

Wouldn't be surprised if it's the other way around. Trump encourages Netanyahu to go for the attack. The hawks in Israel have been aching to go at it for the last decade. Not only would the US have to complicit, it would need to have given an explicit go ahead.

I'm surprised that the Islamic Republic of Iran has stood for as long as it has. The urban areas don't want the conservatism. Khamenei is at death's door. Succession is unclear. Economy has been doing worse YOY and elite human capital leaves the country on first opportunity.

I know the Persians are a civilized people, so they may not resort to brute force violence. But, 30 years of stability under a continuously deteriorating economy is unheard of.

There's definitely an averseness towards the median Indian. I mean the demeanor of the average Indian immigrant: Kumail Nanjiani in Silicon Valley, but additionally unkempt, ponchy and flaunting a chicken neck. In my experience, Indians immigrants are the least fit and worst dressed of any ethnic group. OKCupid was primarily rating this subset. No wonder they were rated terribly.

How Indian do you look ? Often, Indians can blend into other ethnicities with demeanor, accent & fashion changes. Gets you past a person's initial mental block.

As long as Indians have their basics out of wack, it's pointless to discuss their attractiveness. Kumail's transformation is a good example, if slightly exaggerated. I can give other examples. Women are obsessed with Dev Patel and Sendhil Ramamurthy[hot]/[not]. Both look like average Indian dudes in their less-handsome roles. Many Indians are blessed with thick hair, beards and eye brows. Play to those advantages and you'll get +2 boost.

The woke are right about one thing : representation. Women want to date the man of their dreams, but the dreams are manufactured via media. With Indian men getting fresh representation in sexy-man roles, there now are Indian men who women pine for. It's on you to fit into those molds. Additionally, it helps that Brown has become a generic identity. If you don't want to be Indian, you can be brown.

4chan/twitter hate for Indians can be safely ignored. A woman who goes swimming in those sewers is probably too nuclear for a simple man any way. There are exceptions ofc, but as football fans like saying, "[too much ball knowledge means too much ball knowledge].

suspect is undiagnosed BPD

Looks, intelligence, wit and mentally healthy. Pick 3 ?

No more than there is a genocide going on against minorities in Muslim nations. There is strong pressure for displacement. But, I won't call it genocide. We need to reserve that word for the real deal. Can't be diluting definitions for war crimes. (Might be a lost cause)

America has low standards for granting refuge. Indian Sikhs have a 50%+ refugee approval rate despite facing no violence since 1990 and being quite rich by Indian standards. Hell, I'd argue Indian Sikhs are treated a lot better than Hindus in India. (legally and otherwise). White people have a reason to feel unsafe in South Africa. They should leave. They should likely receive refuge by the current standards for refugees in the US.

When has policy ever been about the numbers ? It's about sex appeal.

Utilitarians can be surprisingly blind at times. Covid wasn't sexy. No spectacle, no myth building, no clear narrative. Deaths were slow, honorable & blameless. Ofc people don't care in proportion to the numbers.

No movie or even harrowing video to speak of. Hell, there wasn't even an iconic photo. Statistically, I know the Bengal famine killed a lot of people. But viscerally, my emotions are tied to the photos of piled up bodies, literally (not figuratively) looming vultures and 1st hand stories of families prostituting themselves for food. There is a villain (Churchill). There is intrigue ( what if they hadn't diverted reserves to Australia). There is a story.

Plane crashes are sexier than car accidents, which causes disproportional worry. Tuberculosis, a 'CURABLE DISEASE' !! kills 1.25 million every year. No one cares. Malaria kills 600k every year. Yet, the most visceral image of it is Bill Gates releasing non-viral mosquitoes to a room of white people. Covid is no exception.

The Ukraine war went from being sexy urban warfare 'Hordes of migrants, tanks built up in front of Kyiv, hot women crying and destroyed cities' to more conventional unsexy warfare in the woods. No one cares anymore. Israel and Palestine keep producing visceral imagery at an unheard-of rate, and it stays sexy.

Tragedy has pretty privilege. It's all that matters.

Bay Area != East Bay.

He's paying a lot less in Oakland than he would be in SF / South Bay / Berkeley. There is a large undocumented/recently-documented population there, who works at or lower than minimum wage. You can get it down to ~$150/month (4 visits). That's not too bad.

Love how starvation and disease suddenly become acceptable excuses.

Yeah, how do you think most people died in concentration camps ? (Extermination camps != concentration camps)

That isnt my experience. Colonialism is frequently presented as one/all of.

  1. Yes it was extractive, but still more competent than the natives.

  2. Yeah, but we civilized them.

  3. They killed themselves in civil wars after we left anyway.

Worst of all, even in 2025, Colonial powers have little remorse for their actions.

France still lays claim to the 150 million in Haitian ransom. The British refuse to accept blame imposed on Churchill for the Bengal famine. The portugese inquistion was famous for grotesque torture in Goa. The Spanish straight up genocided the entire now-world despite knowing it was their germs causing it. Not many apologies to go around.

Yes, they werent as effective as communists or nazis at killing. And they werent as comically cruel as imperial japan. But, these were still fairly fucked up periods for the colonized nations. IMO, Pretty close to slavery.

No radiation leak from any nuclear facility in Pakistan, says IAEA amid buzz after Indian claims


Addressing a press conference, Air Marshal AK Bharti said that Indian forces did not know about the site. He said, "We did not hit Kirana hills, whatever is there."

During a press briefing on Tuesday, MEA spokesperson Randhir Jaiswal was asked about social media buzz on ‘nuclear leakage’ in Pakistan after the Indian strikes.

“…Those are questions for them (Pakistan) to answer, not for us. Our position was made very clear during the defence briefing. As for your question, the Pakistani minister has already made some remarks on that,” he said.

India explicitly denied hitting nuclear facilities. The buzz was manufactured by media/social-media accounts.

The obvious hypotheses are valid, but boring. Yes, men are worried about being cancelled. Yes, online alternatives (dating apps) disincentivize in-person courtship. Yes, if women have better outcomes than men, they don't need men. Yes, by forcing men to be same as women, men aren't doing the things men were supposed to anymore.

I want to go in another direction : 'Revenge of the Nerds '

Culture reflects the traits of economically ascendent groups. So far, the 21st century belongs to introverted tech-nerds. Therefore, the next generation has traits of introverted nerds.

Our American culture has venerated mediocrity over excellence for way too long (at least since the 90s and likely longer). That doesn’t start in college, it starts YOUNG. A culture that celebrates the prom queen over the math olympiad champ, or the jock over the valedictorian, will not produce the best engineers.

Vivek is correct, about his youth at least. 90s Cincinnati was a place that valorized the Jock. Aspirational Americans looked to become a partner at McKinsey, BigLaw litigator or to own a Auto showroom. IE. to be a charismatic man in a suit.

But Vivek appears to have missed the last 30 years. Right after his youth came Bill Gates and Steve Jobs. The nerd became cool. Over the late-90s/early-2000s, the nerd was an ascendent underdog. But, NY Finance clearly stood atop the American caste system. Then 2008 happened. The financial crisis destroyed finance's chokehold on the American psyche and nerds swooped in with the 1-2 punch of the Social network & Iron Man. It was done. Nerds won. The first generation that's grown up under nerd-supremacy is reaching high school, and families can now see the fallout.

Woke culture, dating apps, asexual movie leads can be traced back to tech & nerds running the show. Influencer-media allows basement dwellers to become role models overnight. Like it or not, that's nerd culture.


I don't dislike nerds. I like them and am one of them.

But I dislike 2 aspects of nerd culture.

  • Anxiety
    • Nerds are anxious. And nerd culture is built to work around these anxieties.
  • Repression
    • Nerds are horny and embarrassed about it. On sexuality, nerds are dishonest. Their dishonesty leads to a weird disconnect between their behaviors behind the scenes and media they endorse.
    • Nerds want to wield power and are embarrassed about it. On the surface, they endorse universality and equal treatment. Power corrupts, and now they too want to wield their newfound power. Once again, disconnect between stated ideologies and the irresistible temptations of power.

This worst aspects of nerd culture aren't more or less degenerate than what came before. But Nerd culture (and as a result our culture in general) has failure modes that are a result of this unique tendency towards anxiety and repression.

Personally, I'll take a jockish and fertile culture over a nerdy and barren one any day.

  • fake france
  • fake NY
  • fake midwest
  • fake seattle

The credible/non-credible forums can be alright. /r/noncrediblediplomacy , /r/crediblediplomacy, /r/noncredibledefense, /r/credibledefense

That's why I specifically compared it to other bicameral systems.

slow down the passing of legislation

Can't slow down a stationary object. The Senate can only limit the power of the house, a house that already moves at snails place. The Executive and Courts wield their power independently.

give a seniority track to successful politicians

Works better when people were dying at age 50. When the average age of the Senate is higher than the life-expectancy 100 years ago, you know something went wrong.

reserve of statesmen

All elections become popularity contests. Why make the senate elected, if the goal is to bring in experienced statesmen.


The American system was created for a different America. A white-protestant nation run by proven men who rose up the ranks through merit (college, military achievement). 75% of the Senate had a college degree in 1945, when less than 5% of the nation had gone to college. The need for fund-raising and media-access meant that running for office was exclusively limited to the elites. This meant a high degree of consensus on what America should be. Therefore, they worried about the excesses of democracy.

In 2025, America is a diverse nation with public-office having exceptionally low barriers to entry. Consensus is nonexistent and core values of various groups are at odds with each other. In such a place, the system should encourage compromise. This means giving power back to the house.

If an downstream institution can unilaterally torpedo a bill (Senate filibuster), then the house would never go through the painful process of reaching compromise. The congress can override the president, but not the senate.

On filibusters and the Senate..............

The US senate is an odd institution.

The house does the legislation. The executive executes. The courts maintain constitutional sanctity. The states already elect governors to represent them. What is the role of the Senator ? It made made some sense until the 1913 (17th amendment), when Senators were effectively subordinate (selected) to Governors. That way, state elections served as a useful way to remove both unpopular governors and senators.

An elected senate is just odd.

  • The Senate isn't representative. (Californians have the same representation as Wyoming)
  • The Senate can't do anything but block. (Net negative institution)
  • The Senate can filibuster, the House can't. (1 man anti-democratic weapon)

Most democratic nations don't have anywhere near as powerful of a Senate (or equivalent institution). The Indian Rajya-Sabha & House of Lords can only delay a bill by a short amount. A balancing counter-weight also makes sense in a parliamentary system where the executive (Prime-Minister) is selected by the house (making the house too powerful) unlike the US where the President is separately elected.

This means, in India, a person only thinks about 2 elections. Once for their state (governor, who selects senators) and once for the nation (house, which selects the executive). A British person only thinks about the Commons.

In comparison, An American must think of 4 elections. The governor, senators, house reps and the President. That's exhausting. Only takes 1 lapse, 1 midterm rando, to block legislation for the next 6 years. Doesn't the US already have enough checks-and-balances ? The house churns every 2 years. The last time someone held onto Senate+House in a midterm was in 1978.


I am just learning about the 17th amendment & the history of filibuster. so bear with me. Some wikipedia exerpts:

Those in favor of popular elections for senators believed two primary problems were caused by the original provisions: legislative corruption and electoral deadlocks

Appears that it made things worse than better. In an era where they were capable of pushing constitutional amendments, it's hilarious to think that they were complaining about deadlocks. Yeah buddy, try getting anything done in 2025.

Rule XXII of the Standing Rules of the United States Senate allows the Senate to vote to limit debate by invoking cloture on the pending question. In most cases this requires a majority of three-fifths of the senators duly chosen and sworn (60 votes if there is no more than one vacancy),[3]: 15–17  so a minority of senators can block a measure, even if it has the support of a simple majority.

Interestingly, the most important change on senate filibusters was also made in the same decade (1917). Clearly they knew filibusters were a bad idea. House filibusters were eliminated in 1842 ! Not sure why they left it half-complete in 1917.

China is fine with nation-wise oppression of minorities. They have no issues with oppression of Muslims in China, Hindus in Pakistan or the Chinese in Malaysia.

Second, the Muslim identity is primarily an Arab identity. East-Asian and great-lake-African muslims are oppressed all the time, and global Islam does not care. (note: North African islam is not the same as Subsaharan islam). These are the Muslims that matter to other muslims.

Do they just think they own Pakistan as a counterweight to India

Yes. It's less friends, and more that Pakistan is a client state of China. Pakistan is Turtle to China's Vince.

and Pakistan is happy to be owned?

No, but what option do they (Pakistani Army) have ? IMF isn't giving new loans anymore. Even the Saudis stopped giving freebees. Big daddy China is all that's left. Anything to be in opposition to India.