We've been tracking various metrics each month for over a year now in lieu of producing a big annual report. The latest, for October, is now live at https://civicrm.com/about
At this point, we have a number of trends that we can confidently point to (because the methodology of collecting them has...
We've been tracking various metrics each month for over a year now in lieu of producing a big annual report. The latest, for October, is now live at
https://civicrm.com/about
At this point, we have a number of trends that we can confidently point to (because the methodology of collecting them has been consistent), even if the numbers behind them are debatable (speaking of the stats, for example).
For those that don't want to read the entire report, here are the highlights.
Key takeaways
As an ecosystem, CiviCRM is growing. Installation growth, based on anonymous pingbacks, is up 104% YoY, Spark growth is up 30% YoY, and partner, contributor and member growth is slowly inching up.
Combined website traffic across civicrm.com, civicrm.org and docs.civicrm.org are is down almost 7% after having been rising consistently through the first half of 2026. This is likely due to the impacts of AI-based search. In fact, CiviCRM's own AI assistant "Doc Bot" is seeing increased usage.
Financially, we're at breakeven for the past rolling 12 months due to 1) strong earned and subscription income (accounts for over 43% of budget) and 2) stronger than expected event income (accounts for 13.5% of budget). Community funding remains consistent but flat. We expect to close out the year at a loss.
The largest expenses for CiviCRM at present are labor and events, coming in at 76% and 14% of budget, respectively. We expect labor to increase in the coming months and into next year as we press forward with various intiatitives.
We're coming up on 4 years of climate action with Ecologi:
https://ecologi.com/civicrm
It's a small amount each month that reflects a large impact over time.
Improvements to the monthly report
Improve the expense breakdown to be more meaningful
Success! Check your email for magic link to sign-in.
Success! Your billing info has been updated.
Your billing was not updated.
[$] The second part of the 7.4 merge window
Linux Weekly News
lwn.net
2026-11-02 12:51:48
The 7.4 merge window, which
closed on November 1,
included a record-breaking 18,591 non-merge changesets.
That is almost as many as the entire 7.3 development cycle, which had 18,765, itself a record breaker. Of
those, 11,838 landed after the article covering the
first part of the merge window. Mos...
The page you have tried to view (
The second part of the 7.4 merge window
) is currently available to LWN
subscribers only.
Reader subscriptions are a necessary way
to fund the continued existence of LWN and the quality of its content.
If you are already an LWN.net subscriber, please log in
with the form below to read this content.
Please consider
subscribing to LWN
. An LWN
subscription provides numerous benefits, including access to restricted
content and the warm feeling of knowing that you are helping to keep LWN
alive.
(Alternatively, this item will become freely
available on November 12, 2026)
Show HN: Subvocal – Talk to your computer without making noise
Mo is an AI QA engineer that tests your app end to end. Verify critical flows and find bugs from a prompt.
Tests created
77,813
Test runs executed
70,607,819
Changes verified
339,166
PRs verified
80,290
Auto-heals
8,932,104
Bugs caught
117,010
Tests created
77,813
Test runs executed
70,607,819
Changes verified
339,166
PRs verified
80,290
Auto-heals
8,932,104
Bugs caught
117,010
Tests created
77,813
Test runs executed
70,607,819
Changes verified
339,166
PRs verified
80,290
Auto-heals
8,932,104
Bugs caught
117,010
Trusted in production by teams shipping every day.
"Our team's experience with Momentic has been great. Setting up tests is simple and the developer experience has been fantastic."
Erdem
Head of Developer Experience, Notion
"Having E2E QA be part of the development process, run this fast (6.3x parallel), and at this scale (for every single pull request) is a true dream. Seriously, a green CI now has a measurable impact on my sleep quality."
Glavin
Founding AI Engineer, Coframe
"Momentic makes it 3x faster for our team to write and maintain end-to-end tests. Startups win by shipping faster and better, and Momentic enabled us to do exactly that."
"I just ran our first automated eval across 110 prompts. This would have taken well over an hour manually — I kicked it off in 5 seconds and it was done in 14 minutes. HUGE for us."
Mike
Engineering Manager, Runway
"I love working with Momentic. Especially after having years of previous experience in QA, it feels like riding a bike, except the bike is a supercharged electric one!"
Anna
Senior Productivity Engineer, Webflow
Code moves faster. Verification has to keep up.
Coding
agents
have
made
it
dramatically
faster
to
build
software.
The
bottleneck
is
now
knowing
it
works.
Verification
belongs
in
the
agent
loop.
Agents
need
evidence
from
the
app
itself
to
verify
that
their
changes
deliver
the
intended
behavior.
Write the user flows you need to verify. Momentic runs the tests against your app. Reuse successful AI actions and review proposed repairs as the app changes.
1
of 5
01
Knows how your product works
Add terminology, expected behavior, and test guidance to the Knowledge base. AI-assisted steps look up relevant entries as they run.
PRODUCT CONTEXT AT EACH AI STEP
02
Keep tests working as the UI changes
Momentic finds controls again when the UI changes during a run. Permanent healing proposes a test-file repair for you to review.
RUNTIME RECOVERY AND REVIEWED REPAIRS
03
Verify changes with your coding agent
Give your coding agent the changed flow. It can use MCP to author a test, check it in a live session, and save the YAML alongside your code.
TESTS REVIEWED WITH THE CODE
04
See why a test failed
Inspect the failing step, screenshots, and traces. Classify failures as application bugs, test flakiness, or environment issues.
STEP-LEVEL FAILURE EVIDENCE
05
Verify before you merge or deploy
Run the CLI in your CI job. Use its exit code as a required check before merging or deploying.
CLI EXIT CODES FOR CI CHECKS
Verification from your editor to CI.
Define what matters. Let
Momentic
verify it.
Write steps and assertions in plain-English YAML.
Review and version the tests in your repository.
01
/ author
Write tests in
plain English
Describe a user flow and its expected result in plain English. Keep the steps and assertions in YAML files your team and coding agent can review.
02
/ run
Run locally and in
CI
Use the CLI to check the app you're changing. Run the same tests in CI before the next merge or deploy.
03
/ triage
Find the
root cause
See which step failed and inspect its screenshots and traces. AI triage investigates the cause and proposes a test repair for you to review.
04
/ heal
Auto-heal
as your product evolves
Momentic finds controls again when the UI changes. Permanent healing proposes test-file edits for review. Apply the diff and rerun to check the result.
Learns
how your product works.
Describe your product’s terminology, business rules, and expected behavior in the Knowledge base. Agents use that context to choose actions and check outcomes.
Knowledge base
Give agents the product rules they need to check a flow.
App graph
Maps screens and user flows seen during test runs.
Review new guidance
Review session suggestions before adding them to the Knowledge base.
Loved by engineering teams.
“Momentic gave us a
fast and reliable
way to validate Poe.com's AI responses, even when they weren't deterministic.”
Examples of application failures found by Momentic, with the affected flow and its impact.
117,010
before production
Detected
ON A FEATURE BRANCH
New-device 2FA prompt skipped
At a consumer fintech app, a login routing change bypassed the new-device verification step.
Impact
A security checkpoint was silently skipped.
Detected
BEFORE RELEASE
Page creation flow broken on mobile
On mobile, keyboard auto-dismissed when creating a new page, breaking content flow.
Impact
Silent friction in the core creation flow.
Detected
IN STAGING
Driver ETA stuck on 'calculating'
A dependency update on a large rideshare app caused the live ETA subscription to silently fail after a ride was booked.
Impact
Riders never saw an arrival estimate.
Detected
BEFORE DEPLOY
Scheduling availability displayed incorrectly
Incorrect display and logic in the calendar scheduling view.
Impact
Wrong availability shown to users.
Detected
IN CI
Timer reset when the screen locked
At a leading wellness app, an iOS lifecycle change reset the active sleep timer the moment the phone locked.
Impact
Audio sessions ended mid-playback overnight.
Detected
8 MIN BEFORE DEPLOY
Saved cards dropped at checkout
At a top food-delivery marketplace, a payment-form refactor stopped pre-filling saved cards, forcing manual re-entry to pay.
Impact
Returning users abandoned checkout on mobile.
Detected
IN A NIGHTLY RUN
Filters cleared when paginating results
At a popular online retail marketplace, moving to page two reset every active category and price filter.
Impact
Shoppers lost their refined search results.
Detected
BEFORE MERGE
Valid promo codes rejected at the final step
At a large online travel marketplace, a validation change rejected multi-segment promo codes on the last checkout step.
Impact
Discounts failed and carts were abandoned.
Detected
PRE-RELEASE
Wrong subtitle language on first play
At a major streaming platform, a locale-detection bug auto-selected the wrong subtitle track on initial playback.
Impact
Wrong-language captions at session start.
Detected
BEFORE RELEASE
Legacy API issues exposed
12 bugs caught in one day. Exposed legacy API issues that would have shipped undetected.
Impact
Risked shipping regressions into major release.
Verify what you
ship
.
Changelog
Fergie Chambers Faces Extradition to U.S. Despite Never Being Indicted, Says Lawyer
Intercept
theintercept.com
2026-11-02 12:05:03
The refusal of two pro-Palestine protesters to testify before a grand jury could hamper the left-wing donor’s extradition from Spain.
The post Fergie Chambers Faces Extradition to U.S. Despite Never Being Indicted, Says Lawyer appeared first on The Intercept....
The Trump administration’s
extradition case against millionaire left-wing donor James “Fergie” Chambers, detained in Spain since July, ran into an apparent hurdle when two witnesses called before a federal grand jury announced their intention last week to refuse to testify.
Several observers familiar with the case said the grand jury is likely related to the extradition effort.
The refusal of the grand jury witnesses to testify takes on added importance because the U.S. is pursuing Chambers’s extradition based on a criminal complaint, not an indictment, according to his Spanish lawyers. The criminal affidavit, signed by a single FBI agent, includes three charges related to his support for pro-Palestine causes.
The distinction, critics of the extradition case say, highlights how the case is hung on one agent’s complaint, which remains under seal, rather than a grand jury investigation to determine if the state had sufficient evidence to indict Chambers.
“The complaint is not only vague and weak, it’s not normal,” said Aitor Martínez, one of Chambers’s Spain-based lawyers. “In 20 years of practice, I’ve never seen a U.S. extradition request based on a criminal complaint.”
Martínez pointed to a statement from a summary of the complaint contained in an international arrest order reviewed by The Intercept and directed at Spanish authorities that says, “If Chambers’ extradition to the US is granted, another charging document called an indictment will be sought from a federal grand jury.”
“In 20 years of practice, I’ve never seen a U.S. extradition request based on a criminal complaint.”
Xavier de Janon, the executive director of the U.S.-based
National Lawyers Guild
and who is not working directly on the Chambers case, said, “Seeking an extradition without there being an indictment puts the cart before the horse.”
“If he is extradited under a criminal complaint,” said de Janon, “he can’t even fully challenge the charges, because he hasn’t been indicted.”
The State Department referred questions to the Department of Justice, which declined to comment.
Two of the three charges underpinning the extradition request against Chambers — riot and incitement to riot — are related to allegations that he helped fund a late-2023 protest in Merrimack, New Hampshire, against an Israeli weapons manufacturer, according to the summary of the complaint contained in the arrest order. The summary says Chambers provided funds for demonstrators used to purchase materials such as smoke bombs and to help cover protesters’ legal fees.
The third charge is international money laundering related to more than $7 million he transferred to bank accounts in Tunisia, where he was living in 2024. The Trump administration, according to the international arrest order, says the money was used to fund Hamas, which is a proscribed terror group in the U.S. The order does not say he is charged with material support for terrorism.
Chambers’s family and supporters and multiple representatives of Club Africain, a Tunisian soccer club, have said the money was used to pay the club’s debts and related expenditures such as training grounds. The club’s
fan base
has long supported Palestinians.
Grand Jury Refusal
Two women who participated in the 2023 protest against a factory operated by the Israeli weapons manufacturer Elbit Systems recently announced their intentions not to cooperate with a November 4 grand jury hearing where they have been summoned as witnesses.
“If they are being summoned now, this suggests that the grand jury seems to be linked to Chambers’s case,” said Martínez, his lawyer. “It seems like a fishing expedition against him.”
Kira Kelley, an attorney for the two women, said, “Context clues suggest this is about Fergie.”
If the pair don’t answer questions from a federal prosector before the New Hampshire federal grand jury, they could risk a possible jail sentence for contempt.
Nonetheless, the two women — Bridget Shergalis and Sophie Ross — shared a statement with
The Grayzone
Thursday announcing their intention to refuse to testify. In an interview on the same day with The Intercept, the women recounted being under investigation for a year over the protest and being threatened with felony charges that carried potential sentences of decades in prison.
Shergalis and Ross said they turned down several deals for their cooperation. By the end of the ordeal, they were found guilty of misdemeanor offenses and
received 60-day jail sentences
.
Following that experience, said Ross, the pair decided to reject the subpoena to testify before a grand jury.
“It was an easy decision for both of us,” Ross said.
Shergalis and Ross are slated to testify on Wednesday, at which time they plan to assert their Fifth Amendment rights and decline to answer any questions from the federal prosecutor.
At that point, a hearing before a judge may be scheduled, to determine whether to hold Shergalis and Ross in civil contempt, which could result in up to 18 months of prison, according to Kelley.
Martínez, who also worked on WikiLeaks founder Julian Assange’s defense team, said the decision by Shergalis and Ross recalled that of Chelsea Manning, who twice refused to testify against Assange before a grand jury,
serving time in prison
in
both instances
.
“It’s like Chelsea Manning, years later,” he said.
Shergalis and Ross decided to go public with their plans after speaking with organizers behind the recently-launched
NSPM-7 Defendants Union
, a nationwide project for protesters targeted by the Trump administration’s efforts to criminalize dissent as laid out in last year’s
National Security Presidential Memorandum-7
. The document, part of a broader effort to
paint left-wing activism as terrorism
, directs the FBI to go after “violent and terroristic activities” associated with “anti-fascism.”
The NSPM-7 Defendants Union is centered around signing a pledge to follow practices such as not cooperating with law enforcement, adopting collective legal strategy when possible, and publicly announcing those strategies — a goal that runs against the secrecy surrounding federal indictments.
While secrecy benefits prosecutors before a grand jury, said Shergalis, “It’s good for all of us to be as transparent as possible.”
“We’re seeing NSPM-7 result in unprecedented levels of prosecution against protesters.”
Shergalis, Ross, and Chambers have each signed the pledge, and the grand jury decision appears to be the first public test case of the union’s ideas.
“We should celebrate people across the country deciding to assert their legal rights boldly against the federal government and its NSPM-7 policy,” said de Janon, of the National Lawyers Guild. “We’re seeing NSPM-7 result in unprecedented levels of prosecution against protesters.”
“This Is Bad Faith”
Meanwhile, Chambers remains imprisoned in Madrid, as the Spanish courts decide whether to grant the U.S. extradition request or not.
Chambers has applied for asylum in Spain and, if his application is granted, he would be released and the U.S. extradition request would be denied. He cannot be extradited until the asylum case runs its course.
A self-described communist and anti-imperialist, Chambers is heir of the Cox family, owners of Atlanta-based telecom giant Cox Enterprises. As part of a settlement in which his family severed ties with him at the time, he sold stock back to them in mid-2023 worth about
$250 million
.
Since then, he has been funding an estimated 100 humanitarian and left-wing projects in 20 countries, according to his partner, actress and activist Stella Schnabel, including
aid projects
in the Occupied Gaza Strip.
“I believe this is part of the persecution against pro-Palestine protest.”
As for the Trump administration’s case, Martínez, the Spanish attorney, underscored how the criminal complaint against Chambers raises charges related to the protest against the New Hampshire factory for Elbit Systems, Israel’s largest arms manufacturer, but leaves out how federal prosecution efforts ended up with
minor convictions
for Shergalis, Ross, and other activists.
“The people who carried out the acts received a minor sentence,” he said. “They leave this out. This is bad faith.”
“Why don’t they tell this to Spain?” he said. “It’s likely that Spain won’t turn someone over based on actions that got 60-day sentences.”
Martínez said Chambers’s opposition to the Israeli genocide in Gaza is the real reason behind the case. “I believe this is part of the persecution against pro-Palestine protest. The U.S. has even
sanctioned
the International Criminal Court because of this.”
Of Chambers’s case, Martínez said, “This is one more attack.”
Bitmovin (YC S15) Is Hiring Technical Graduates in Denver (Engineering/CS Focus)
I’m not a big owner of things, but the best laptop I’ve ever owned is a roughed-up
2015 Thinkpad X250.
For the flex, here it is running a GUI shell sat on top of the microkernel I’m currently
working on (blog post about this incoming… eventually).
Here’s a list of things that I think make it brilliant. I don’t want to pretend that any
of these things are uniquely exemplary: far from it. It is the confluence of them, and
not any one in particular, that makes me love using it.
Size
It’s pretty tiny. Its form-factor fits snuggly into the footprint of an A4 piece of
paper, so it sits alongside books and folders when placed in a small rucksack without
risking damage.
It weighs almost nothing too: even with the extra capacity battery, it’s only about 1.45
Kg. Light enough that you almost don’t notice it when travelling.
Performance
Mine has an i5-5200U: 4 logical cores, maxing out at 2.7 GHz. I upgraded it to 8 GB of
RAM a while back: more than enough for some light on-the-go programming.
Even at full pelt, I’ve never heard the fan kicking up much of a fuss. Now that I think
about it, I’m not even certain that it works.
Battery
The X250 has both an internal battery and an external battery slot. I’ve opted for the
largest after-market external battery I could find. When doing casual work the laptop
will easily survive for north of 10 hours without a charge, stretching to 12 hours if
I’m careful.
Repairability
The X250 was quite popular and replacement parts retail very cheaply on ebay. It’s not
the easiest to disassemble Thinkpad by any means but I can tear it apart entirely with
the pocket phillips I keep on my keyring. I have previously replaced:
The internal battery
The lid camera
The keyboard
The motherboard
Every time you open up a laptop you run the risk of damaging something important either
via a slip of the hand or simply the accumulated toll of loosening and tightening dozens
of tiny screw threads. Despite that, my X250 has proved remarkably resilient and has
never sustained any major damage despite more than more than a dozen tear-downs.
Keyboard
As with most Thinkpads, the keyboard feels lovely to use. When you spend a lot of time
writing code the reliability and sensitivity of a keyboard starts to matter a lot.
Coupled with my
text editor
, the experience of writing code on the thing
is beautifully snappy.
Operating System
I opted for Endeavour OS. In the usual Arch Linux tradition, it favours native
package installs over sandboxed applications. As a result my install sits at around 30
GB. A 128 GB M.2 SSD raided from another second-hand machine is more than enough to
cover that and everything I’m working on.
I’ve slimmed down my systemd setup as much as possible and the thing boots from cold in
about 8 seconds.
Cost
Here’s where the X250 beats out most competition. I bought mine second-hand for about
£100 in 2018. They’ve only got cheaper since then. I’m not afraid to take it anywhere
because it’s barely worth stealing and would be cheap to replace even if that were to
happen. Its value is very much not in its on-paper specs and instead in how I use it.
Conclusion
I get a certain glee in keeping older or second-hand hardware running as a daily driver.
It sounds silly, but it feels like a quiet revolt against bigger-faster-costlier
consumption culture. Come the end times, this is the machine I’ll be riding it out with.
What If His Name Were Blake Bruceman?
hellgate
hellgatenyc.com
2026-11-02 11:46:26
Speculating on an alternate gubernatorial dimension....
The leaves are dying, and every evening the sun clocks out earlier, uninterested in lighting the night sky. On Tuesday, the angel of history descends again—web 3.0 fascism could be dealt
something of a defeat
, or not. We know we must march forward: There is simply no time to contemplate other possible presents, we have the moment we have, and we must seize it, thanklessly. Our children need us to do this. But it's so hard not to wonder what could have been.
It’s a small pleasure, but receiving an “I Voted” sticker after casting my ballot is one of my favorite parts of the whole voting process. I slap the sticker on my chest and spend the rest of the day feeling like I accomplished a little something. I participated in democracy and everyone who sees me will know it. Artificial intelligence robbed me of that small joy this year and replaced it with a small horror. My “I Voted” sticker is AI-generated slop.
Getting a little sticker when you vote has been a tradition in the US
since the 1980s
. It’s a symbol of the election cycle. People take selfies and post them or share them with friends. It’s a celebration of democracy and a small way to pressure people into going to the ballot box.
The voting sticker has been subject to every kind of discourse imaginable.
Are “I Voted” stickers worth the cost?
Wondered Marketplace in 2020. A cemetery in New York has spent time and money
tending to the grave
of Susan B Anthony because voters coat it in stickers during voting season. In 2008, companies like Starbucks and Ben and Jerry’s
ran afoul
of election laws for offering free treats to people who showed their “I Voted” stickers.
It’s appropriate but depressing that I got an AI slop “I Voted” sticker in 2026, the year it feels like the mainstream began to reckon with AI.
My sticker is ugly. It’s a busy circular nightmare. “I Voted” is spelled out in small bronze text at the top of the ring. Bronze laurels encircle a bronze bald eagle that stretches atop the American flag. Below this is more text: “250 YEARS ANNIVERSARY” and "SEMIQUENCENTENNIAL.” Below that, another line reads — as best I can tell — “CEKEVPATIVG EDELDOA.” I assume this was meant to say “CELEBRATING FREEDOM," but whoever churned out this slop wasn’t using the latest models and the fine detail is lost.
I put the sticker on reluctantly, but I was not proud. I sent a photo of the offending sticker to journalist friends with my face cropped out. They immediately noticed it was AI-generated. I did not post my sticker online. I shared it with no one else. My democratic joy turned to shame.
A few hours later, a friend sent me a selfie. He’d just left the voting booth. His sticker was beautiful, tasteful, a testament to the subtle art of the “I Voted” sticker. A red ring enclosed a blue field. The crescent and palmetto (symbols of South Carolina) sat elegantly behind a bold white “I VOTED.”
His sticker catches the eye. Passers-by will know exactly what the sticker says. They know he voted. Anyone looking at my sticker will wonder how I spilled some bronze paint on my chest. I trembled with indignation. One town over from me they’re giving out the good stickers and I’m stuck with slop. First my town of 12,000
installs 33 Flock cameras
and now this?
I spent some time scrolling through social media to see other “I Voted” stickers. So many are so beautiful. Michigan runs a contest where students can submit their own idea for a sticker and the
state will print up the winners
.
Kentucky
,
Albany
, and several other places around the country do the same. Louisiana commissions local artists to produce
gorgeous stickers
. My mouth hung up in a jealous scowl as I stared at pictures of New Orleanians bedecked with a sticker showing a hand drawn trumpet blasting out “I Voted” alongside Mardi Gras beads.
Only Denton, Texas (another place I’ve lived, weirdly) seems to also have
AI-generated slop stickers
. Denton’s sticker is, at least, personalized. The city’s court house is in the corner looking more or less correct. The “I Voted” is large and centered in the sticker itself. Like mine, the image is too busy. Too much text, too many fireworks, and the inclusion of the Liberty Bell clutters the whole thing up. The central message — “I Voted” — is bright white. It doesn’t sink into a busy bronze soup like on my sticker.
Complaining about AI-slop “I Voted” stickers may seem trivial in a world where AIs are
choosing targets in warzones
and vibecoders are
reverse-engineering
every piece of software ever written. But these little things are important. These small annoyances pile up until things like
ChatGPT flyers
are so ubiquitous that you don’t notice them anymore. The AI drive-thru
works well enough
and one day you stop worrying about it so much. You get an ugly AI slop “I Voted” sticker and you slap it on or toss it out. Enough of these build up until you don’t notice that AI all around you. And then, one day, it’s all normal.
Right now we’re figuring out where our AI lines are. Society is negotiating what the future will look like. We’re
fighting data centers
, figuring out
how to burn tokens
to keep our bosses happy, and learning what we will and won’t abide.
Me? I draw the line at AI-generated “I Voted” stickers.
About the author
Matthew Gault is a writer covering weird tech, nuclear war, and video games. He’s worked for Reuters, Motherboard, and the New York Times.
Jailbreaking my car's navigation system just so I can change the time
The car that I drive is a 2014 Renault Latitude. It's not a fancy car by any means and is starting to show its age, but it gets me from point A to point B and does everything else reasonably well so I've never really had any reason to complain.
When I bought the car, the previous owner asked me whether I wanted a "navigation map SD card" with it.
They mentioned that the car apparently needs this SD card just to change the time on the clock, but I didn't really find that believable - not that they were lying to me, but perhaps they couldn't figure out how to enter the correct settings menu. After all, videos of similar head units online showed the menu with the clock settings option right there.
Since the maps on the SD card were apparently very out of date and didn't function properly, I decided I didn't need it and the previous owner tossed it.
I didn't really miss it. The dashboard display did show a large "No maps found!" message taking up the majority of the display, but it didn't bother me too much.
Like most other European countries, Estonia uses daylight saving time, and we recently switched from summer time (UTC+3) to "winter time" (UTC+2), which meant that the time displayed on the dashboard was out of sync by an hour. I dived into the settings menu, hoping to easily find the "Set time" option in there.
That never happened.
The menu where the "Set time" option should've been simply didn't have that option. I compared the menus with the ones in online videos, and the option was simply missing. There were two "Settings" buttons on the dash, and one of them worked but only let me change various Bluetooth and phonebook settings, with no time & date settings in sight. The second Settings button did not work whatsoever, instead showing a "No maps detected!" pop up message every time I pressed it.
I must've spent an hour going through various menus, pressing various buttons or button combinations hoping to land in the correct menu, searching the web for information on this particular unit simply because I refused to believe that there was no option to change the time and
I clearly must've been missing something obvious
.
Still in denial, I decided to fetch the firmware and dig into it, hoping to find some indication on how to open the correct settings menu.
Luckily, I wasn't the first to embark on this journey and someone had already done some reverse engineering work on this particular unit (
https://github.com/andronedev/carminat-tomtom-live-re/blob/main/README.md
). While it was in French and seemingly heavily LLM-assisted, it did point me to the correct URLs for fetching the update files themselves.
When extracted, the update files included a
ttsystem
file which turned out to be an ext3 filesystem containing the device's root filesystem, which was pretty much exactly what I was looking for. The unit itself (TomTom Carminat LIVE, codename
beaumont
) is built around a Samsung S5P6440 SoC running TomTom's custom OS based on Linux 2.6.28.10.
After reverse engineering it for a while, I came to the conclusion that the previous owner was completely correct. There was absolutely no way to enter the correct settings menu without a navigation SD card inserted. In fact, the menu definition
pointing
to the Settings applet was read from the SD card itself, and without one inserted containing the right menu definition, the unit didn't even know
how
to enter the Settings applet.
This decision by TomTom (or Renault) is still baffling to me.
What made things worse was that this needed to be a
legitimate
SD card. The system had DRM checks which rejected SD cards without the correct CID (a unique ID burned into the card) and files with signatures based on that specific CID, so I couldn't even copy the files to a blank SD card. I'm guessing the DRM was in place to prevent people from pirating maps.
I don't want maps. I have a phone that I use for navigation. I just want to change the damn time.
I checked online and found some third parties selling the card I presumably needed, but with shipping to Estonia, it would've cost me roughly 60 EUR, which I really didn't feel like paying.
I couldn't find anyone online who'd jailbroken one of these before. I found some old forum threads with long-expired image links that had some pretty generic information about the unit itself and the general consensus was that it was difficult to root.
I didn't expect a Linux 2.6 system from 2011 to be incredibly hard to hack, so I naively went in looking for some easy wins.
My first idea was to modify the ext3 update image itself. It seemed like I could easily just mount the filesystem, modify some init scripts to do what I want, and copy it to a blank SD card for the unit to pick up my modified image and flash it.
While the system showed confirmation prompts to install vanilla, unmodified
ttsystem
images with no problem, I could never get it to update to my modified image, and I had no idea why. I checked the updater logic present in the update file itself and couldn't see any reason why my image would be rejected.
It was at this point that I set up an emulator with QEMU, a custom kernel also based on the same 2.6 version, some patches and a very simple CAN emulator for the system to actually boot - albeit reluctantly, wanting to reboot every 3 seconds, which I just patched out by ignoring reboot syscalls.
I used this to test installing the modified update image in there. I then realized that I don't actually
have
the
recovery
partition that the device boots into to perform the update, and came to the conclusion that there must be some sort of verification layer there that I'm missing.
Taking a closer look at the
ttsystem
image also pretty much confirmed this, as it had a bunch of high-entropy data appended to it, followed by some zero blocks, followed by 40 high-entropy bytes. I assumed this was a signature of some sort (40 bytes could be DSA with SHA-1), but with no access to the actual binary doing the verification, I couldn't really figure out how to attack this or if I even could.
I'd been reluctant to put the effort into building the emulator, but it turned out to be extremely useful for testing out various attacks and getting an overview of how the system behaved.
For example, I found out that
most
partitions were mounted either read-only,
noexec
or both. Mounts from a live system:
rootfs on / type rootfs (rw)none on /sys type sysfs (rw,nosuid,noexec)none on /proc type proc (rw,nosuid,noexec)/dev/mmcblk0p5 on /content type ext3 (rw,nosuid,nodev,noexec,noatime,nodiratime,errors=continue,data=ordered)/dev/loop0 on / type ext2 (ro,noatime,nodiratime,errors=continue)none on /dev type tmpfs (rw,nosuid,noexec,size=1024k)none on /tmp type tmpfs (rw,nosuid,noexec,size=2048k)none on /var type tmpfs (rw,nosuid,noexec,size=5120k)none on /dev/pts type devpts (rw,nosuid,noexec,gid=5,mode=620)none on /var/log type tmpfs (rw,nosuid,noexec,size=1024k)/dev/mmcblk0p3 on /mnt/flash type ext3 (rw,sync,nosuid,nodev,noexec,errors=continue,data=ordered)/dev/mmcblk1p1 on /media/sdcard type vfat (rw,sync,nosuid,nodev,noexec,noatime,nodiratime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1)/dev/loop1 on /media/loopback type ext3 (rw,nosuid,nodev,noexec,noatime,nodiratime,errors=continue,commit=3600,data=journal)none on /mnt/movi type unionfs (rw,nosuid,nodev,noexec,noatime,nodiratime,dirs=/media/loopback=rw:/media/sdcard=ro)
Notably, the base FAT32 filesystem of the SD card was mounted to
/media/sdcard
, and the system mounted a unionfs on top of it in
/mnt/movi
, which had a writable layer pointing to an ext3 filesystem which lived as a file on the external SD. I'm not 100% sure what the point of this setup was (maybe so they can use
data=journal
, which survives power loss better?), but nevertheless all of these mounts were
noexec
.
This seemed like surprisingly strong security for something from 2011.
Something else that the emulator helped me with was identifying all of the files that the system tried to read from the external SD card. One of these files was
master.script
, a script in a custom language for automating certain UI navigation actions. This was loaded every boot and I hoped this would allow me to somehow open the correct settings menu for setting the time.
I reversed
ttn
, the binary responsible for parsing this script, documenting its functionality as I went, but unfortunately I didn't find any way to open the menu I needed.
I still thought
master.script
was what I should be putting my effort into, since it had a lot of functionality that could be a potential attack vector.
As usual, I checked for any
system
or
popen
calls that would allow me to perform command injection. Unfortunately, I didn't find any constructed with user input.
I thought I had hit the jackpot when, while reversing the parser, I came across a
SHEX
/
SHELLEXECUTE
command. It passed an arbitrary argument to
launchserver
, which launched whatever command line was provided with
execve
.
This had two obstacles. The first one was that there was still no way to get it to execute a
custom
executable directly, as the SD card was mounted
noexec
which prevents executables and scripts on it from running. However, this could be bypassed by not executing the script directly, but rather executing something like
/bin/sh /media/sdcard/payload.sh
, which would go through
/bin/sh
on the rootfs, which was
not
mounted
noexec
.
The second hurdle was bigger, however. It turned out that while
launchserver
would happily launch anything,
SHEX
accepts only a single whitespace-delimited token. This means that I could only control
which
binary was executed, but not pass it any arguments. Combined with the previous obstacle, this then also seemed like a dead end because I could only run scripts or binaries that were already on the rootfs. I did look through most of them, but nothing seemed very helpful.
Defeated, I started looking at file writes. Maybe if I could write files to places I shouldn't be able to, that would give me some sort of leverage.
Sure enough, I discovered the
LOG
command, which let me truncate a file (defined with
#define LOGFILE log.txt
) and write a single line to a file of my choosing. This one actually allowed spaces and regular ASCII characters, but only bytes 0x20 to 0x7E, meaning writing binary data is out of the question. As
ttn
runs as root, any writes would also be written as root.
It also had a classic path traversal vulnerability. Defining
LOGFILE
with
../
in it allowed me to write outside the intended logfile directory, theoretically allowing me to write printable ASCII to anywhere on the system.
However, I once again ran into a wall with how the mount points were set up. I couldn't overwrite anything on the rootfs since it was mounted read-only, and everything else was mounted
noexec
and would never be executed anyways. I also looked into whether there were any useful config files I could overwrite, but most files were a custom binary format that I couldn't write with
LOG
or were otherwise unhelpful. This also looked like a dead end.
Among other things,
master.script
allowed displaying images on the screen (GIFs/BMPs). I spent some time setting up a fuzzing environment using Unicorn to see if I could hit any image decoder bugs causing memory corruption, and while I did get quite a few OOB reads, I didn't find any useful writes.
I had been researching, reverse engineering and trying to come up with different strategies to run arbitrary code for a while. Much to my frustration, none of my strategies ended up working, and I took a step back to look at the bigger picture and figure out if I can make any of the puzzle pieces that I already have fit together.
Among other things, I knew I could:
Execute arbitrary binaries, but with no arguments and only the ones already on the rootfs
Write ASCII to paths of my own choosing
This is when I had a brainwave and was reminded of a trick that kernel exploits often use: Instead of trying to get the userland to execute our script, why don't I get the
kernel
to execute our script instead?
Often, Linux kernel exploits, when being able to write arbitrary memory, overwrite the
core_pattern
string. Normally, this dictates how coredumps are stored when a process crashes, but it
also
accepts special syntax for piping these coredumps to a process instead (
|/bin/my-coredump-handler
). When a crash occurs, the kernel spawns the process specified in
core_pattern
and pipes the data in there.
core_pattern
also supports cmdline arguments for the spawned process. I got lucky here, since support for this landed in 2.6.24 whereas the unit's kernel was 2.6.28.
Importantly, you're also able to set
core_pattern
from
userland
by writing the pattern to
/proc/sys/kernel/core_pattern
.
I can easily use this to point
LOGFILE
to
../../proc/sys/kernel/core_pattern
and
LOG
the string
|/bin/sh /media/sdcard/payload.sh
. This would run the rootfs
/bin/sh
and execute our script with it as root.
Being able to execute arbitrary binaries also turned out to be useful even without arguments, since I needed to somehow get a program to crash, triggering a coredump. Using a DoS vuln to crash the script parser or the main UI itself wouldn't work very well, because this would immediately trigger a reboot of the whole system, killing my script.
I used my emulator setup to execute pretty much every single binary on the system, and found that
/bin/dbus-monitor
raised a SIGABRT if it couldn't connect to the system bus (which the unit didn't run), triggering a coredump. This was perfect.
I initially confirmed success by simply calling
reboot
in the
payload.sh
script. I was very pleasantly surprised when booting the unit with the SD card inserted and pressing "Yes" at the "
Execute master.script?
" prompt, the unit rebooted, confirming that I had code execution.
The first thing I did was use this to create a full backup of the internal eMMC using
dd
. With a more complete dump of the system, I pretty quickly figured out that the date & time settings that I'm interested in are in
/mnt/flash/settings.dat
, which is a custom binary format. After some more reverse engineering, I figured out the exact offset in the file which held the currently set timezone, and wrote a script to patch it to be UTC+2.
That is how I was finally able to fix my car's timezone.
Tor Browser 16.0 is now available for
Desktop
and
Android
. This is our first stable release based on Firefox ESR 153, incorporating a year's worth of changes that have been shipped upstream in Firefox. As part of this process, we've also completed our annual ESR transition audit, where we reviewed and addressed
over 270 Bugzilla
issues covering changes in Firefox that could negatively affect the privacy and security of Tor Browser users.
The ongoing development of Tor Browser is made possible thanks to the support of our community. Every donation helps power the tools that let people browse freely and privately. Consider
joining thousands of others who have already made a small contribution
towards building a better internet for everyone.
What's new for Desktop?
Redesigned Settings
Tor Browser connects millions of people to the Tor network (and by extension, the free and open internet) out of the box. However, for those who need to circumvent advanced censorship, boost their security level, or tailor their browsing experience, clear and easy-to-use Settings are essential. To that end, Tor Browser 16.0 introduces a complete redesign of the Settings experience based on a parallel redesign
shipped in Firefox in June
. Improvements include a cleaner layout, updated navigation, and clearer labels and descriptions aimed at making settings easier to find, understand, and manage. Other noteworthy changes include:
The long-standing "General" page has been retired, with its contents distributed across new pages for Appearance, Downloads, Tabs and browsing, Accessibility and Languages.
Pages and components that are unique to Tor browser like Connection and Security Level have been rebuilt from scratch using Firefox's new components so they look and feel consistent with the rest of Settings.
Some settings that, if enabled, could compromise your privacy or make your browser
easier to fingerprint
have either been hidden or feature clear warnings.
Fear not: your existing preferences will remain saved even if the original setting has moved or changed, and you can use the search bar at the top of Settings to find anything you're struggling to track down.
Generic window titles
Another new addition to Settings is the option to use generic window titles, which can be found under Privacy and security → Protections from third-party applications. Like most browsers, Tor Browser uses the title of the page you're currently viewing for the names of its windows. However, other applications on your computer or your operating system itself can read these titles without special permissions. When enabled, this option simply replaces all window titles with "Tor Browser" — preventing other software from snooping on your browsing history. As always, for more comprehensive protection from third-party applications on your device, we recommend using
Tails
.
For now, this option is strictly opt-in, as we did not have sufficient time during this release cycle to properly evaluate potential side-effects for users. In particular, we worry that enabling this setting may reduce the browser's usability for those who rely on accessibility tools (e.g. screen-readers) or those with highly customized systems (e.g. less-popular desktop managers, application switchers, etc). If you are one of these users, we would ask that you try using this setting, and report back any bugs or usability issues you find!
📣 Generic window titles have also been upstreamed to Firefox as part of the
Tor Uplift Project
. Advanced Firefox users can enable this feature by opening
about:config
and setting
privacy.exposeContentTitleInWindow
(for normal windows) and
privacy.exposeContentTitleInWindow.pbm
(for private browsing windows) to
false
.
New user manual
The Tor Browser manual, which can be found by visiting
about:manual
or through the application menu (☰) → Help → Tor Browser manual, has also received a complete revamp. Since the
release of our new Support portal in 2025
, the team has had to maintain two sets of documentation: one for the traditional manual that's available to read in Tor Browser, and another for the new Support site. With this release, we've solved that problem by replacing the older material in the Tor Browser manual with our latest and greatest help content from the new Support site. Like the previous iteration of the manual, the new version is bundled with Tor Browser for Desktop so it can be read offline, which comes in particularly handy when troubleshooting issues that prevent you from connecting to the Tor network.
Linux on ARM
Finally, after a year of testing, Tor Browser 16.0 now features native support for Linux computers with 64-bit ARM (AArch64) processors, such as Raspberry Pis and Apple Silicon Macs running Linux. If you have been relying on Tor Browser Alpha for support for these devices, we recommend switching over to the stable release channel unless you plan to continue testing Alpha releases. Meanwhile, Tor Browser 16.0 follows Firefox in ending support for 32-bit x86 Linux computers. Fortunately, very few people should be affected, as we currently receive around 200 update checks from this platform a day, down from a peak of around 4,000 in 2019. Most remaining 32-bit users should be able to switch to the 64-bit version unless they're running outdated hardware or versions of Linux.
What's new for Android?
Tor Browser for Android has also benefited from Mozilla's ongoing work to modernize Firefox, as seen in the Settings redesign on desktop. Over the past year, Firefox has been gradually transitioning to Jetpack Compose — Google's newest framework for building native Android interfaces. In this release, we've made significant progress applying the same changes to Tor Browser. As a result, large swaths of Tor Browser's user interface has been updated to Material Design 3 — the latest iteration of Google's design language for Android — with more yet to come in future releases.
Google Play limits the size of apps distributed as APKs to 100 MB. Since Firefox for Android continues to grow with each release, keeping Tor Browser within this limit has been an ongoing challenge — for example, this issue has prevented us from bundling Tor Browser's manual on Android so that it can be accessed offline. Nevertheless, we've found various ways to reduce the size of our Android package over the years and stay within the limit. Most recently, this involved carefully and creatively compressing some of the built-in resources Firefox uses to display its interface.
What's in the works?
Tor Browser users can expect additional improvements over the next few months as we continue to refine the new Settings experience for desktop and Material Design 3 for Android. If you find a bug or have a suggestion for how we could improve this release,
we'd love to hear your feedback
.
The team is also working hard on a new and improved circuit display, coming soon to Tor Browser Alpha, which integrates site information directly into the circuit panel and can visualize more complex multi-path circuits. Multi-path circuits use additional relays to split your traffic into two paths that converge on a single exit, improving both connection speed and reliability. These circuits already exist for regular websites (but not onion sites) and are randomly assigned based on network availability. Although Tor Browser's traditional circuit display can only show a single path at a time, you have likely benefited from a multi-path circuit without even knowing it. What's more, this exciting update will mark the debut of the circuit display on Tor Browser for Android, which has been exclusive to desktop until now.
Contributions <3
Thanks to all of the teams across Tor, and the wider community, who contributed to this release. In particular, we'd like to extend our gratitude to the following volunteers who have contributed their expertise, labor, and time:
Bug tor-browser#45172
: Move our "Connect to Tor before opening links" code from the deprecated openToBrowserAndLoad to the recommended loadUrlOrSearch
Recently a security hole in a certain open source Java library resulted in a worldwide emergency kerfuffle as, say, 40% of the possibly hundreds of millions of worldwide deployments of this library needed to be updated in a hurry. (The other 60% also needed to be updated in a hurry, but won't be until they facilitate some ransomware, which is pretty normal for these situations.)
I have a 20+ year history of poking fun at Java in this space, and it pains me to stop now. But the truth is: this could have happened to anyone.
What happened was:
Someone wrote a library they thought was neat
They decided to share it with the world for free
Millions of people liked it and used it everywhere
Some contributors contributed some good ideas and, in this case, at least one bad idea
Out of a sense of stewardship, they decided to maintain backward compatibility with the bad idea
The bad idea turned out to have one or more security flaws that affected all the users
The stewards responded quickly with a fix
From this, if you believe the Internet Consensus, we can conclude that open source doesn't work, people don't get paid enough, capitalism is a sham, billionaires are built on the backs of the proletariat, your $50 Patreon donation makes a real difference, and Blorkchain Would Have Solved This.
(Miraculously the Internet Consensus is always the same both before and after these kinds of events. In engineering we call this a "non-causal system" because the outputs are produced before the inputs.)
Nevertheless, I have been dared to take my take on the issue. It, too, was the same before and after, but the difference is I didn't write it down until now, which makes this journal a causal system. You could probably write an interesting philosophical paper about observations of a non-causal system nevertheless being causal, but mercifully, I will not.
Free Software is Communism
So anyway, meandering gently toward the point, let's go back in time to the original Free Software movement. Long ago, before the average reader of this sentence was born, a person whose name is now unpopular was at a university, where they had a printer, and the printer firmware was buggy. This person firmly believed they could quickly fix the printer firmware if only they had the source code. (In the spirit of every "I could do this better in a weekend" story, I'm not sure whether we ever confirmed if this was true. In any case printer firmware is still buggy.)
As a result, they started a nonprofit organization to rewrite all of Unix,
which the printer did not run and which therefore would not solve any of the
original problem, but was a pretty cool project nonetheless and was much more fun
than the original problem, and the rest was history.
This story archetype is the Hero's Journey that inspires all software development:
I have a problem
I do not know how to solve that problem
But I strongly believe, without evidence, that I can solve a generalized version of that problem if I form a large enough team and work at it for 35 years(*)
We are now accepting donations
(*) initial estimate is often less than 35 years
Now, you have probably heard all this before, and if you're a software developer you have probably lived it. This part is not really in question. The burning question for us today, as we enjoy the (hopefully) peak of late-stage capitalism, is: ...
but where will the donations come from?
Authoritarianism
Before we get back onto communism, let me draw an important distinction. Most communist governments in history ended up being authoritarian systems, which is to say, top-down control. Ironically, the people at the top seem to have more power than the people at the bottom, which at first seems like the antithesis of communism. This is not the place to claim an understanding of why that always seems to happen. But one has to acknowledge a pattern when one sees it.
On the other hand, it's easy to find examples of authoritarianism outside communism. Our world is filled with top-down control systems. Many corporations are in many ways, top-down controlled. The US system of government is increasingly top-down controlled (ie. authoritarian), despite the many safety measures introduced early to try to prevent that.
When politicians rail against
communism
it is because they don't want you to notice the ever-growing
non-communist authoritarianism
.
Authoritarianism is self-reinforcing. Once some people or groups start having more power, they tend to use that power to adjust or capture the rules of the system so they can accumulate more power, and so on. Sometimes this is peacefully reversible, and sometimes it eventually leads to uprisings and revolutions.
People like to write about facism and communism as if they are opposite ends of some spectrum, but that's not really true in the most important sense. Fascism blatantly, and communism accidentally but consistently, leads to authoritarianism. And authoritarianism is the problem.
Authoritarianism is about
taking things from me.
Communism, in its noncorporeal theoretical form, is about
giving things away.
I read a book once which argued that the problem with modern political discourse is it pits the "I don't want things taken from me" (liberty!) people against the "XYZ is a human right" (entitlement!) people. And that a better way to frame the cultural argument is "XYZ is my responsibility to society."
As a simple example, "Internet access is a human right," is just a sneaky way of saying "someone should give people free Internet." Who is someone? It's left unspecified, which is skipping over the entire mechanism by which we deliver the Internet. It's much more revealing to write, "To live in a healthy society, it's our responsibility to make sure every person has Internet access." Suddenly, oh, crap. The someone is me!
Healthy society is created through constant effort, by all of us, as a gift to our fellow members. It's not extracted from us as a mandatory payment to our overlords who will do all the work.
If there's one thing we know for sure about overlords, it's that they never do all the work.
Free software is a gift.
I would like to inquire about the return policy
Here's the thing about gifts: the sender chooses them, not the recipient. We can have norms around what gifts are appropriate, and agreements to not over-spend, and wishlists, and so on. But I won't always get the exact gift I want. Sometimes I didn't even want a gift. Sometimes the gift interprets JNDI strings in my log messages and executes random code from my LDAP server. This is the nature of gifts.
On the other hand, the best gifts are the things I never would have bought for myself, because they seemed too expensive or I didn't even realize I would like them or they were too much work to obtain, or because someone hand-made them just for me. These feel like luxuries of the sort capitalism cannot produce, because deciding, going out, and buying something for myself isn't luxury, it's everyday. It's lonely. It's a negotiation. It's limited by my own lack of creativity.
The best part of free software is it sometimes produces stuff you never would have been willing to pay to develop (Linux), and sometimes at quality levels too high
to be rational for the market to provide (sqlite).
The worst part of free software is you get what you get, and the developers don't have to listen to you. (And as a developer, the gift recipients aren't always so grateful either.)
Paying for gifts
...does not work.
You don't say to someone, "here's $100, maybe this time get me a gift worth $100 more than you'd regularly spend." It's kind of insulting. It still probably won't get you exactly the thing you wanted. Actually, the other person might just pocket the $100 and run off with it.
We already have a way for you to spend $100 to get the thing you want. It's a market. A market works fine for that. It's not very inspiring, but most of the time it's quite efficient. Even gift-givers will often buy things on the same market, but with a different selection criteria, thus adding value of their own.
When you try to pay for gifts, it turns the whole gift process into a transaction. It stops being a gift. It becomes an inefficient, misdesigned, awkward market.
There's research showing that, for example, financial compensation in a job is more likely a demotivator than a motivator (ie. if you pay me too little, I'll work less hard or quit, but if you double my pay, it won't double my output). If you tie cash compensation to specific metrics, people will game the metrics and usually do an overall worse job. If you pay someone for doing you a favour, they are less likely to repeat the favour. Gifts are inherently socially and emotionally meaningful. Ruin the giftiness, and you ruin the intangible rewards.
So it is with free software. You literally cannot pay for it. If you do, it becomes something else.
This is why we have things like the Linux Foundation, where the idea is you can give a gift because you appreciate and want to support Linux (and ideally you are a rich megacorporation so your gift is very big), but it dilutes the influence of that money through an organization that supposedly will not try to influence the gift of Linux that was already happening. You end up with multiple gift flows in different directions. Money goes here, code goes there. They are interdependent - maybe if one flow slows down the other flow will also slow down - but not directly tied. It's a delicate balance. People who keep receiving Christmas gifts but never give any might eventually stop receiving them. But might not.
Anyway, gifts will not get you 24-hour guaranteed response times to security incidents.
Gifts won't get you guaranteed high quality code reviews.
Gifts will not, for heaven's sake, prevent developers from implementing bad ideas occasionally that turn into security holes. Nothing will. Have you met developers?
Open source
I've avoided the term "open source" so far because it means something different from the original idea of Free Software.
Open source was, as I understand it, coined to explain what happened when Netscape originally opened their Mozilla source code, back at the end of the 1990s. That was not a gift. That was a transaction. Or at least, it was intended to be.
The promise of open source was:
You, the company, can still mostly control your project
Customers will still pay you to add new features
Actually customers might pay other people to add new features, but you can still capitalize on it because you get their code too
Linux distributions only package open source code so you'll onboard more customers more easily this way
You can distance yourself from this anti-capitalist gift-giving philosophical stuff that makes investors nervous
Plus a bunch of people will look at the code and find bugs for you for free!
Maybe this sounds cynical, but capitalists are cynical, and you know what? It worked! Okay, not for Netscape Corporation (sorry), but for a lot of other people since then.
It also failed a lot of people. Many developers and companies have been disappointed to learn that just uploading your code to github doesn't make a community of developers appear. (It does make it more likely that AWS will fork your product and make more money from it than you do.) Code reviews are famously rare even in security-critical projects. Supply chain issues are rampant.
In fact, we've now gotten to the point where some people hesitate to give away their source code, mainly because of this confusion of gifts and customers. If I spend some spare time hacking something together on a weekend and give it away, that's a gift. If you yell at me for making it, that makes giving less fun, and I will spend fewer weekends making gifts.
Whereas when a company has a product and open sources it and you complain, that's customers giving valuable feedback and it's worth money to learn from them and service them,
because you eventually earn money in exchange (through whatever business model they've established). No gift necessary.
Call it cynical or call it a win/win relationship. But it's not a gift.
The startup ecosystem
Since the creation of the open source designation 20+ years ago, software startups have taken off more than ever. I attribute this to a combination of factors:
Cloud computing has made it vastly cheaper to get started
Incubators like YCombinator have industrialized the process of assembling and running a small software company
Megacorps have become exponentially richer but no more creative, so they need to acquire or acqui-hire those startups faster and faster in order to grow.
Although a lot of startups open source their code, and they all depend heavily on open source ecosystems, the startup world's motivations are
amazingly different
from the free software and open source worlds.
Gifts exist in the startup world. They are things like "we were both in YCombinator so I will intro you to this investor I like" or "I got extremely rich so let me invest in your startup and incidentally I get a lottery ticket for becoming even more rich." These absolutely are still gifts. They each strengthen social ties. The startup world is a society, and the society is built up from these gifts. It's a society that largely ignores the trials and tribulations of anyone who isn't a rich software engineer insider, but history has hosted many past societies of that sort and it takes a long time to build and deploy enough guillotines, and anyway they are having fun and producing a lot and surely that counts for something.
If free software gifts are communism and open source is cynically capitalist exploitation, then startups may be, weirdly, the democratization of capitalism.
Hear me out. Big companies don't care what you think; you can't pay them enough to care. Gift givers care only a little what you think; if they gave you what you wanted, it wouldn't be a gift. But startups, well, there are a lot of them and their mantras are "do things that don't scale" and "focus on the customer" and "build rapid feedback loops." What that spells for you is a whole bunch of people who want to give you what you want, in exchange for money, and who are excited to amortize the costs of that over all the other customers who want the same thing.
It's kind of exciting, conceptually, and more self-optimizing than untuned gift giving, and so it's not super surprising to me that it has started to eclipse the earlier concepts of free software and open source. More and more "open" projects are backed by small companies, who have financial incentives to make their users happy because some of the users turn into paying customers. They'll even provide the uptime SLAs and security fix turnaround guarantees you wanted so much. Our company,
Tailscale
, is unabashedly one of those. Nothing to be ashamed of there. The system works.
What doesn't work is assuming those startup mechanics apply to everyone out there who gives you a software gift. Not every project on github is the same.
Not everyone has the same motivations.
Giving them money won't change their motivations.
Trying to pay them or regulate them taints the gift.
If you wanted to pay someone to fix some software, you didn't want a gift. You wanted a company.
But if there is no company and someone gave you something anyway? Say thanks.
Epilogue
This isn't where evolution stops. There's a lot more to say about how SaaS taints the unwritten agreement of open source (because you don't have to give back your changes to the code), and how startups tend to go bankrupt and their tech dies with them, and how the best developers are not good at starting companies (no matter how much easier it has become), and how acquiring a startup usually destroys all the stuff they innovated, and how open source is often used as a way to exfiltrate past those kinds of disasters, and how simultaneously, whole promising branches of the "gift economy" structure have never been explored. But that's enough for today. Maybe another time.
Independent Digital Media Is Not a Privilege Governments Can Revoke: The Fight to Save Egypt’s Mada Masr
Electronic Frontier Foundation
www.eff.org
2026-11-02 10:10:22
On November 11, Lina Attalah, founder and editor-in-chief of the independent Egyptian news outlet Mada Masr, will face trial for the second time in just two weeks. This time, she faces charges of spreading false news, an accusation that has become one of the Egyptian government's most frequently use...
On November 11, Lina Attalah, founder and editor-in-chief of the independent Egyptian news outlet
Mada Masr
, will face trial for the second time in just two weeks. This time, she faces charges of spreading false news, an accusation that has become one of the Egyptian government's most frequently used tools for prosecuting journalists.
The trial follows a conviction on October 27, when Attalah was
found guilty
of managing an unlicensed news website. The Mada Masr team had repeatedly applied for a license, but their attempts went unacknowledged. The proceedings took place in a single hearing, which Mada Masr
states
included serious violations of her right to a fair trial. She was sentenced to the maximum fine of 3 million Egyptian pounds (approximately 55,000 USD), forcibly removed from the courtroom, and held for seven hours until the fine was paid. The court also ordered Mada Masr's closure and the confiscation of all equipment belonging to the organization.
The current wave of repression has also targeted
Matsada2sh
, a fact-checking and open-source intelligence initiative. Its entire six-person newsroom was
violently arrested
, illegally detained, and interrogated under torture before being charged with terrorism-related offenses and spreading false news. The team remains in pretrial detention.
Governments seeking to control the flow of information increasingly rely on a combination of legal harassment, financial pressure, criminal prosecution, and restrictions on the infrastructure that makes independent publishing possible. Egypt's media licensing regime is a particularly troubling example. Requiring news organizations to obtain government authorization to operate creates an obvious mechanism for restricting who gets to publish and whose voices are allowed to reach the public.
The charge of spreading false news is similarly dangerous. Governments around the world have increasingly invoked the language of misinformation to justify restrictions on speech, often without meaningful safeguards against political abuse, as we laid out in our
recent submission
to the United Nations Office of the High Commissioner for Human Rights. Governments should never be the arbiters of which reporting is permissible, particularly when that reporting concerns their own conduct.
Why Mada Masr matters
Founded in 2013, Mada Masr has spent more than a decade doing the kind of journalism that authoritarian governments find most threatening: documenting abuses, amplifying marginalized voices, and reporting on issues that powerful institutions would prefer remain hidden.
Its reporting has given torture survivors and the families of political prisoners a platform. It has opened space for debate about reproductive rights. It has brought frontline reporting from Sudan and Gaza, and stood alongside refugees facing xenophobic attacks and security crackdowns.
Emerging as a digital-first publication, in a media environment where traditional outlets were increasingly constrained, being purely digital allowed Mada Masr’s founders to build an independent publication outside the structures of legacy media.
Mada Masr has also consistently engaged with technology and advocated for a free and independent internet for the global majority. It has covered dozens of hot-button tech stories in recent years, from
Egypt’s Cybercrime Bill
and
Israel’s use of AI weaponry in Gaza
, to stories about the “
privacy chaos
” caused by biometrics to more
personal essays
about the importance of the internet and tech. In August, they launched an
open call for proposals
on original critical reporting on artificial intelligence.
Mada Masr may be thoroughly Egyptian, but it doesn’t belong just to Egypt. Its team has made into a publication that represents the kind of independent, local, grounded journalism that international audiences need, particularly at a time when international reporting on state repression is lacking. They built on-the-ground networks to center local voices in Sudan (publishing in both
English
and
Arabic
on the conflict) and Gaza out of a sense of cross-border responsibility and solidarity.
This is precisely why its potential closure should concern everyone who cares about a free press. The ability to publish independently, investigate powerful institutions, and to share information without government interference is fundamental to the exercise of free expression.
From solidarity to action
The international community has repeatedly expressed concern about restrictions on
press
and
internet
freedom in Egypt. More than 100 organizations have now joined
statements
of
solidarity
with Mada Masr, and with
Matsada2sh
.
These expressions of support matter, particularly when journalists are increasingly isolated and under threat—but solidarity must also translate into sustained political and institutional pressure.
The international governments and institutions that maintain relationships with Egypt—including those in the European Union, the United Kingdom, and the United States—should make clear that the prosecution of journalists and the forced closure of independent media outlets are unacceptable.
The cases against Lina Attalah, Mada Masr, and Matsada2sh must be dropped. The journalists who remain in detention must be released. And Egypt's licensing regime, with its profound chilling effects on access to information and independent knowledge production, must be brought to an end.
There is a particular urgency to defending independent media at a moment when governments are finding ever more ways to restrict the circulation of information. Whether through internet shutdowns, platform restrictions, surveillance, or the criminalization of reporting, the result is often the same: fewer opportunities for people to understand what is happening around them, to challenge official narratives, and to hold power accountable.
We cannot allow the closure of one of Egypt's most important independent newsrooms and the prosecution of its editor-in-chief to become another routine episode in the global erosion of press freedom. Nor should we wait until the last independent journalists have been silenced before recognizing what has been lost.
To learn more about Lina Attalah in her own reads, you can read our
Speaking Freely interview
from January 2026.
Anthropic's internal AI use, at retail prices, now exceeds its employee pay
Anthropic CEO Dario Amodei.
Aaron Schwartz/Bloomberg via Getty Images
Here's a remarkable milestone in the AI revolution:
Anthropic
now spends more on internal
AI
usage than it does paying employees.
The company revealed this in its
IPO filing
. In April, the value of Claude AI used internally, calculated at prices Anthropic charges customers, exceeded its monthly employee compensation costs, excluding stock awards.
By September, that internal AI usage had surged more than 250-fold since January.
There's an important distinction here. Anthropic isn't actually spending more cash on AI than salaries. It's valuing its internal AI consumption at retail prices.
Still, this feels like a watershed moment for the tech industry.
In March, I wrote about a
new trend
: Tech companies are starting to treat AI tokens (the main way to measure usage) as a fourth component of employee compensation, alongside salary, bonuses, and stock awards.
Anthropic's numbers show why. More than 90% of the code added to its software in September was written by Claude. Meanwhile, revenue has grown vastly faster than headcount.
For decades, talented humans were the most valuable resource in Silicon Valley. Now AI is challenging that assumption.
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
Alistair Barr is the driving force behind Business Insider's
AI Insider
franchise, including the AI Insider newsletter.
Sign up here
. Before that, he was Business Insider's Global Tech Editor and the Big Tech team leader at Bloomberg, following a reporting career at The Wall Street Journal, USA Today, Reuters, and MarketWatch. Alistair covers all things Big Tech, along with startups and venture capital. He writes analysis and columns about topics including AI, cloud computing, data centers, semiconductors, online search, e-commerce, EVs, robotics, and autonomous vehicles.
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
What can I do to resolve this?
You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.
Show HN: VHS Camera – Run your camera or videos through a simulated VHS signal
The video stays on your device: nothing is uploaded to a server.
VHS●CAM
VHS filter: shoot like it's 1994
Shoot video and photos as if on an old VHS camcorder, or give a video you already have the VHS look. Everything is processed on your device; nothing is uploaded to a server.
VHS Camera is a free VHS filter for your camera: shoot video and photos that look as if they came off an old camcorder tape, with colour bleed, tape noise and a date stamp. It runs in the browser, so there is no app to install.
How to film with the VHS filter
Tap INSERT TAPE and allow the camera, or tap UPLOAD A VIDEO to use a video you already have.
Choose how worn the tape looks (Clean, Dirty or Worn) and set the date stamp and sound.
Switch between LIVE and PHOTO, then press the record button.
Save the clip or photo, or share it straight from the preview.
VHS filter for your videos
Already have a clip? Tap UPLOAD A VIDEO on the start screen, or UPLOAD VIDEO in the camera, and pick a video from your phone or computer. It plays with the VHS effect right away, so you can try the tapes, the quality, the date stamp and the sound while you watch. Then save the whole video as an MP4 with its original sound. You don’t need a camera for this, and the video is processed on your device, never uploaded to a server. The sound gets the VHS treatment too: Tape sound, under the Sound button, adds a soft tape hiss, a slight wobble and muffled highs to uploaded videos and LIVE recordings, and one tap turns it off.
Private: nothing is uploaded
The VHS effect is calculated on your phone or computer, frame by frame. Your video and photos are never uploaded to a server.
Frequently asked questions
Is the VHS filter free?
Yes. There is no sign-up and nothing to install; it works in the browser.
Does it work on iPhone and Android?
Yes, in a current mobile browser such as Safari or Chrome with camera access. On slower phones, the settings let you trade some detail for a smoother frame rate.
Can I add the VHS effect to a video I already have?
Yes. Tap UPLOAD A VIDEO, pick an MP4, MOV or WebM file and play it with the effect. Save video then turns the whole clip into an MP4 with the VHS look and the original sound, right in your browser.
What file do I get?
Videos are saved as MP4 where the browser supports it (otherwise WebM), photos as JPG.
Why does it look like a real tape?
The effect is based on ntsc-rs, which simulates how analogue video is encoded and recorded: colour bleed, tracking noise and soft detail, not a picture laid on top.
Five months ago,
we rebuilt our warehouse on DuckDB
instead of ClickHouse. We did it because a shared ClickHouse cluster couldn't handle hours-long modeling queries, and it lacked a cost-based query optimizer. DuckDB gave every customer their own instance, and tools like dbt could connect over Postgres. We're still fans.
But we underestimated how big and complex our customers' workloads would get. We didn't want them to have just a good experience, we wanted the best experience you can get when you pay someone else to run your warehouse. It should work at any scale, at a fair price.
So we spent the last three months re-architecting it, this time without DuckDB.
We love DuckDB. It's probably the best single-node tool for analyzing heaps of data, and that node can be your laptop. It's robust and reliable. It almost always finishes your query, and if it runs out of memory, it spills to disk.
The same features that are great locally gave us problems in the cloud. Customers ran more complex queries than we expected, so we had to add huge instances. Adding disks was an option, but queries would have been too slow and expensive to be viable.
To run a single-node engine safely in the cloud, we have to size the worker before the query arrives. The ergonomics here are wonky. We couldn't find a good way to tell customers what resources their queries would need. Predicting CPU and memory for a specific query was even harder.
Users shouldn't have to worry about this. They should be able to run whatever they need to get their work done. Our job is to handle the rest and make it as predictable as possible.
We hit other bottlenecks too, like fetching lots of data from S3 without async fetching. DuckDB only added that recently.
To use DuckDB as a fully-fledged data warehouse, we installed the
DuckLake
extension by default. It is a new lakehouse format from the same creators and follows the same principles as DuckDB: keep things simple and run on almost any setup. That works well up to a point, but at PostHog's scale, we needed something designed for distributed scale from the start.
DuckLake's catalog schema has to stay portable across many SQL databases, which limits engine-specific optimizations. At our scale, a single-table write could take more than three minutes because every commit re-read the statistics for the entire catalog.
We tuned DuckLake with table indexes and similar tweaks. But to keep offering DuckLake, we had to stay 100% compatible with its existing, "official" schema. This left us with fewer optimization options.
Another problem is that over 99% of the catalog's statistics rows belonged to tables that no longer existed. Cleaning them up made writes 30 to 50 times faster, but the rows kept growing back.
We also spent a lot of time on the number of connections DuckLake opens to metadata databases. That hurts on shared Postgres instances, where every connection has a real cost.
DuckLake at PostHog didn't start small and ramp up. We fully committed and put all of PostHog in a DuckLake to dogfood as the first customer.
When we moved off it, it held about 2 PB: roughly 1 trillion rows and 15.3 million snapshots. It broke in lots of ways that were hard to fix within DuckLake's scope.
We'd rather have contributed to the existing community and helped it grow. But we couldn't land changes as fast as we needed, so we ended up carrying patches in a fork which defeats one of the main benefits of using DuckLake: the community.
Speaking Postgres is harder than it sounds
Offering direct Postgres connections over port 5432 sounds like a great idea. In practice, it was much harder to scale than HTTP.
Missing reconnects and managing TTLs and idle connections at scale is hard, and long-running connections are expensive. There's also very little tooling for load balancing pgwire compared with HTTP. Compare the ergonomics of an AWS NLB with an ALB and you'll see what we mean. HTTP headers also make routing in Kubernetes much more flexible.
There's also an endless list of Postgres clients, and each expects its own niche Postgres behavior. Customers kept asking us to fix their client or library. We spent a lot of time shipping many of those fixes, but coverage never got good enough. We'd rather spend our time building an excellent warehouse than chasing the long tail of Postgres compatibility.
We've learned not to hype the tech behind a warehouse so much. What counts is the experience it gives you.
Still, we're happy to share what we're cooking next. We rebuilt everything around our own flavor of
Trino
and created our own open table format, HogLake. And yes, we're testing it at PostHog's petabyte scale first. We made it solid before handing it to design partners.
If your current warehouse is slow or expensive,
let's talk
, especially if you have a lot of data to query. We're always looking for design partners.
Please state the location and include REMOTE for remote work, REMOTE (US)
or similar if the country is restricted, and ONSITE when remote work is
not
an option.
Please only post if you personally are part of the hiring company—no
recruiting firms or job boards. One post per company. If it isn't a household name,
explain what your company does.
Please only post if you are actively filling a position and are committed
to replying to applicants.
Commenters: please don't reply to job posts to complain about
something. It's off topic here.
Readers: please only email if you are personally interested in the job.
Pond AI | Head of Engineering (founding-level) | San Francisco | Onsite | Full-time |
https://joinpond.ai
Most non-tech businesses know exactly which workflows they want AI agents to handle, but have no one to build, connect and run them. Pond delivers custom AI agents to these businesses through a network of 7k AI builders, with Pond as the platform and security layer over MCP.
We're backed by Tier-1 VCs and a co-inventor of GPT, and we're looking for a hands-on technical leader to own the platform end to end.
You'd be a great fit if you:
- Build full-stack and ship AI agents in production (TypeScript/Python, LLMs, tool use)
- Have worked on payments, agent coordination/MCP, or platform/marketplace products
- Have early-stage startup experience (founder, founding or early engineer) and like to grind
Email dylan@joinpond.ai with something you've built.
Stream | Multiple Positions | Amsterdam (NL), Skopje (North Macedonia) | Boulder, CO (US) | Toronto (Canada)) | Remote possible | Full Time | Visa Sponsorship |
https://getstream.io
Stream powers Chat, Video, AI Moderation, and Activity Feeds for billions of global end users across thousands of different apps.
We're constantly hiring backend engineers, but I wanted to take this opportunity to tout my own hire because I'm hoping you'll come work with me... I'm pavlov on HN, been here for nearly two decades, and working in realtime media for a good while longer. Now I run the Video team at Stream and I'm looking for a Staff Engineer. You'd report to me with a direct line to our CTO. Check out the details:
You'd own the media plane of Stream Video. It's a modern stack: SFU in Go, cascading to handle very large audiences, ingest and recording fleets, global multi-cloud edge. Video is fundamentally different from request-based services, which means this is not just a seat in Stream's shared infra team, but a genuine Staff role with its own roadmap defined by you. You can email me directly at pauli.ojala@getstream.io, happy to connect and tell you more.
Full-time role, hybrid with 3 days a week in our Amsterdam office, or remote from anywhere in the EU. Visa sponsorship for the Netherlands.
Namespace | San Francisco, CA / New York, NY / Zurich, Switzerland / Remote | Full-time | REMOTE or HYBRID |
https://namespace.so
Namespace is a vertically integrated infrastructure stack for software development. We build everything ourselves, from the server racks to the hypervisor and scheduler, and we run it globally.
Teams at SpaceXAI, Ramp, Sierra, Fal, Vanta, Framer, ElevenLabs, and many others build their software on Namespace today.
Open roles (all full-time, global, remote & hybrid):
Juro | AI Engineer (Agentic team) | London, UK | REMOTE (UK) | Full-time | £115k + equity
Juro is an AI contract automation platform used by in-house legal and business teams. Our Agentic team builds Operator, the agent that works across contracts inside Juro: it answers questions and takes action on legal data. You'd build the tools that let it act across the product, own parts of the agent harness (planning, tool use, context) and put evals around what you ship, together with our QA team.
Stack: TypeScript/Node.js, MongoDB, pgvector, AWS Bedrock, LiteLLM, Mastra, some React. Engineers pick their own AI tooling (Claude Code, Codex, Cursor...) and we pay for the usage.
We're looking for programmers first who now build with LLMs, and who have shipped something people use. Years of ML matter less to us than what you've built. We apply models, we don't train them.
Process: 30-min intro call, interview with the hiring manager, a build review where you walk us through something you've built, and a short chat with our CTO. We reply to every applicant.
You'll work from the UK. For exceptional candidates we can sponsor a UK Skilled Worker visa, whether you're relocating or already in the UK. If you need sponsorship, say so in your application.
8 years ago, Jeremy was absolutely frustrated with the status quo of data infrastructure in manufacturing. Companies paid millions per year for "shitty 90s software" and are still doing that today, when there is much better software out there for open-source. So we decided that we can do this better, first founded a System Integrator, realized most of the tasks in setting up and maintaining a data infrastructure can be standardized, and decided to found the currently VC-backed UMH, which is built on open source software and is open source itself.
What we're actually building: We build and scale a real-time data platform that runs hundreds of Benthos pipelines inside a single Docker container (UMH Core), processing millions of industrial messages that control actual production processes.
Plus a local-first user interface (Management Console) where both non-technical factory workers and power-users/developers can work together. And yes, we care about UI in enterprise/industrial software.
Equall is an AI-native platform for corporate legal work in private markets. We extract and reconcile legal information from source documents (charters, SPAs, board consents) and produce auditable outputs: cap table tie-outs, governance dashboards, personnel summaries. Law firms use us to do due diligence in hours instead of weeks. Working with leading ECVC firms now, expanding into PE and M&A.
The role: own the knowledge graph at the core of the platform. Design the public APIs, build the Prefect pipelines that turn documents into structured graph data, run it in production on Kubernetes.
Mastra is the open-source TypeScript framework for building AI agents (agents, workflows, memory, evals, tracing). From the team behind Gatsby, YC-backed,
https://github.com/mastra-ai/mastra
Why: Four months ago Mastra was a very popular open-source agent framework with very little monetization. Then we launched our platform, and now we're closing a couple of deals a week. Pipeline is growing faster than one person can work it, so this is our second dedicated sales hire. You start day 1 with a warm pipe: most leads are already using the open-source framework.
What you'd sell: Platform (Studio, Server, infra to run/debug/observe agents in production, hosted or self-hosted for regulated orgs), Agent Builder (agents without code), and Concierge (embedded Mastra engineers working alongside customer teams). Full-cycle, mid-market: Series A–C, often with regulatory or infra constraints, sales cycles of a few weeks, talking to eng leaders.
You have: Closed full-cycle deals in dev tools, OSS, infra, or AI. Enough technical depth to come across as a smart PM with engineers: understand their use cases and scope a solution. Founder or founder mentality. Comfort with a product that ships something new every week.
Backed by Sequoia Capital, Peregrine is the operational AI platform powering decision making and operations SF / NYC / DC | In-office 4 days | $175,000 - $300,000
Tandem Health | Engineers, Designers, PMs | On-site in Stockholm, Sweden | Full time
At Tandem Health we're building the AI-native operating system for clinics, so clinicians can focus on care rather than administration. We're building, launching and iterating quickly, and already support more than 10,000 care organisations across 14 European markets, including most of the NHS. We recently raised a $100M Series B to go further, faster.
It started with our AI medical scribe, which listens in to the patient-clinician encounter to generate medical records and saves clinicians 5-10 minutes per patient. Today our AI medical assistant supports clinicians before, during and after each visit, with medical coding and clinical decision support, three products certified as medical devices under EU MDR.
We're expanding our Stockholm team with exceptionally driven builders in a wide range of roles.
We make robots that autonomously construct buildings, starting with bricklaying.
There's a lot of vapourware in robotics, but we're deploying robots across multiple concurrent sites today - and generating revenue. In the last couple of months, our robots have built affordable social housing, health care living units, canal walls in the historic city centre of Amsterdam, and a youth sports centre. Help us to build things that matter in the long run.
We're looking for experienced software engineers for many roles. You can help us solve problems like:
- Data sync and analytics queries across a fleet of robots (that can be offline for hours at a time)
- Modelling buildings in code and finding the right UI for operators to edit them
- Designing systems for robots to drive safely and fully autonomously around a building site
We're rapidly scaling up operations - we're already hiring operators in the UK too - and we need experienced engineers who can build the systems that manage all of that complexity. If you think that 'full-stack' could reasonably mean debugging firmware, mixing buckets of mortar _and_ writing React components, we are looking for you. Robotics experience not required!
PostHog | Full-Time | Technical CSMs, SRE, Product Manager, AI Research Engineer | REMOTE (all remote) | Hiring GMT-8 to GMT+2 | PostHog makes your product self-driving.
* we have a public handbook (posthog.com/handbook) if you want to learn how we work, pay and more in complete detail.
* we are growing through more autonomy and transparency not through process.
* we have a ton of scale and a bunch of super interesting technical problems to solve
* we're building 20 more products over the next couple of years, so you could end up building one of those
* public compensation calculator! see immediately what you'd get paid
* we're hiring: ai research engineer, technical customer success managers (americas & emea), finance manager, site reliability engineer, and more.
https://posthog.com/careers
Lyra | Full-Time | Applied AI Engineer | REMOTE (US Citizenship Required) | Building the AI that builds satellites.
Lyra is building an AI-native platform to solve the problem of mission integration that exists today: a slow, expensive, and manual process of information extraction and design for satellite missions. You will work on a small, fast-paced team that interfaces directly with government agencies and industry experts to build the initial version of our platform.
We are looking for a generalist that is comfortable building full-stack products, using AI tooling, and building data analysis/extraction pipelines.
Package: Health insurance, generous equity, competitive salary
Confidential (established Saudi engineering & industrial services group, building a new in-house AI unit) | Senior AI/ML Engineer, Senior Backend Engineer | Khobar, Saudi Arabia | ONSITE, full-time | Relocation + visa provided | Saudi nationals preferred
We're founding an in-house AI unit and hiring its first two engineers. You'd work directly with the unit's head and shape how it's built from day one. Software work: LLMs, data, APIs and the systems around them.
Senior AI/ML Engineer: LLM assistants and agents over real company data, Arabic and Saudi-dialect language work, a proper evaluation practice, and open-weight models on our own hardware. The unit publishes research, with the opportunity to co-author.
Senior Backend Engineer: the services and integrations connecting AI to enterprise systems and operational data, on a modern serverless stack. Reliability, security and clean data are the job.
Package: competitive salary, housing and transport allowances, family medical insurance, annual flights.
Apply at
https://zalvi.app
, a short game-like screening assessment we built ourselves (English or Arabic). Shortlisted applicants move to the next stage.
Starbridge | Senior Product Engineer | NYC | Full-time |
https://starbridge.ai/
Starbridge is building an AI platform that turns large-scale public and enterprise data into reliable sales insights. We are early, moving fast, and building from zero to one, so this role will have huge ownership and product impact.
Product Engineer: (React/Typescript) who would work closely with product and design to build user-facing parts of the platform. You will craft performant, stable frontends that explain technical concepts to non-technical users and help us iterate fast based on customer feedback.
We're looking to build our in-person team in NYC!
Apply:
https://starbridge.ai/careers
and mention HackerNews or email recruiting@starbridge.ai with your resume in pdf format.
FutureSearch builds AI that predicts the future. You can see our public track record on our site. We launched the FutureSearch product this August. We also publish benchmarks and datasets, and do work for frontier AI labs.
I'm the CEO, previously CTO of Metaculus and creator of Google's current internal prediction market. FutureSearch is about ten people, remote-first and concentrated in London.
We're hiring (US pay, adjusted to the local market elsewhere):
- Chief Operating Officer, $180–250k
- Director of Partnerships, $160–240k
- Director of Strategic Projects, $140–220k
- Head of Product, $160–240k
- Program Manager, Research, $130–190k
- Research Scientist, $160–250k
CNH | AI Engineer Intern | Tempe, AZ, USA | ON-SITE | Summer internship (40 hours/week, May–August)
Work on autonomy and precision agriculture, applying AI, machine learning, and computer vision to real-world agricultural problems. You'll help develop, evaluate, and deploy models alongside our engineering team.
We're looking for strong programming skills and a solid understanding of ML fundamentals. You must be a current college student actively pursuing a degree. Visa sponsorship is not available.
CoVar | Durham, NC and McLean, VA |
https://covar.com/
| Full-time | HYBRID (3 days in office per week)
CoVar is a small R&D company specializing in AI/ML and software development for defense and healthcare. Bootstrapped and profitable. We're looking for a jack-of-all-trades who can: prototype computer vision algorithms, build Python web services, wrangle large sets of data from disparate classes of sensors, ... any or all, with the only hard requirement being eagerness to learn.
Visit covar.com for more information on our current openings!
NEO Holdings (stealth, regulated commodity markets infrastructure) | Malibu, CA | ONSITE (SoCal preferred) or REMOTE (US) | Full Time
We are a funded and backed venture building the regulated base layer for tokenized commodities including the US-regulated trading venue, Mint and benchmarks, bridging institutional commodity markets and on-chain finance. Commodity markets trade tens of trillions a year on infrastructure designed for paper-based trading; we are what comes next. Launching Q1 2027.
Roles currently available:
VP of Engineering: owns the platform end to end, from on-chain contracts (Avalanche) to the trading engine, custody integrations and institutional-grade security and operations. A VP or SVP who still builds and ships, not one who just manages others; you will have a real say in the architecture before we finalize it and will work with leaders in tech and commodities. You must have done this before: delivered a trading venue or institutional token issuance, end to end. $200K to $300K base plus equity depending on experience and location.
Lead Product Manager: works with our partners and customers to ensure the launch product suite is fit for purpose and aligns participant experience across onboarding, trading, collateral and reporting. You must have shipped products in a regulated financial venue, exchange, clearing house or institutional crypto platform, you write the spec yourself, and you have run product through a regulator's questions. $150K to $240K base plus equity.
We hire on written replies and quality of CVs. We read every reply ourselves, so please write yours yourself. Send a short note on what you have built that is closest to the seat, with references, to careers@neo.co and put the role in the subject line. Please mention you came from HN.
Kraus Hamdani Aerospace | Software Engineer | REMOTE (US) or ONSITE in Reno, NV
We build long-endurance autonomous aircraft and the software used to operate them.
Looking for a strong generalist software engineer. Most application work is TypeScript/web, with C/C++ and Rust elsewhere in the stack. You’ll work across frontend, backend, cloud, DevOps, and directly with flight test and hardware teams.
We care a lot about code quality, maintainability, and engineers who can take ambiguous problems and independently ship solid software. No sloppers.
U.S. defense/security eligibility required.
Apply: software@khaero.com
Shepherd (Series B) | ONSITE | San Francisco, CA
At Shepherd, we're pursuing the most ambitious technical vision in commercial insurance: fully autonomous underwriting. Shepherd is an AI-native commercial insurance platform transforming how high-hazard industries get covered.
The infrastructure behind the AI boom (data centers, semiconductor fabs, renewable energy assets) has to be built and insured, but traditional carriers weren't built for this speed. We built Shepherd to solve that.
We announced our $42M Series B earlier this year! That brings our total funding to over $60M — led by Intact Private Capital, the investment arm of one of the largest insurers in the world. Intact is not only our lead investor but also a carrier partner, a testament to the confidence the incumbent industry has in what we're building.
We’re looking for product-minded, high ownership engineers to join our team! The Engineering team is HQ’d in SF, but we’re growing in NYC now too!
We’re hiring for 2 Senior SWEs, 1 MLE and 1 AI PM in SF, and 2 Staff SWEs in NYC. Engineers own projects end-to-end so full stack ability is expected.
Check out our open roles here:
https://shepherdinsurance.com/careers
Our stack includes: Typescript, React, Next.js, GraphQL w/ Apollo, Node.js, Postgres & Redis
– If you have experience building AI agents or multi-step reasoning systems, we’d love to hear from you.
* In the form, be sure to mention that you heard about this opportunity via Hacker News.
If you want to read more about our team and culture, check out our blog:
https://shepherdinsurance.com/blog
Hi HN, I'm Adriaan. Simple Analytics exists thanks to HN: it started with my Show HN in 2018 [1]. Today we're a self-funded, profitable team of four building privacy-first analytics.
Privacy is why we run our own bare-metal servers instead of the cloud. We're preparing for a LOT more traffic, and you'll own the infrastructure and its reliability. We have good documentation. First project: leading the infra side of our Elasticsearch to ClickHouse migration, with Alex, our developer.
Stack: Ansible, OpenTofu, Docker Compose, Grafana, GitHub Actions, TypeScript. Clear async communication matters more than knowing every tool.
Open Education Applications / Neon | Senior Frontend Engineer | Utrecht, The Netherlands | HYBRID | DUTCH REQUIRED
Please note: working proficiency in Dutch is strictly required for these roles, and we kindly ask for no automated applications. The rest of this post will continue in Dutch.
Wij zijn een non-profit die zich inzet om lesmateriaal voor scholen beter, betaalbaarder en flexibeler te maken, in zowel digitale als gedrukte vorm.
Werktijden en kantoordagen zijn flexibel en worden in overleg bepaald op basis van de rol. Wij bieden een competitief salaris.
Openstaande rol:
- Senior Frontend Engineer
Ervaring met Yjs of vergelijkbare CRDT frameworks/algoritmen vereist.
E-mail jobs[at]openeducation.foundation met (HN) + de functienaam in het onderwerp, samen met een korte introductie en een link naar je LinkedIn-profiel en/of GitHub. Als er van beide kanten een goede match lijkt te zijn, plan ik graag een kennismakingsgesprek in.
We The Flywheel | Remote | Brand Identity Designer (AI-agent team) | Hourly, flexible hours
We build and run our own brands (race finders, training tools, AI guides, sports media) with a small remote team and a lot of AI agents.
Most of our brands have an identity that is good enough. We want someone with a sharp eye to make them good: logo refinements, lockups, favicons, social and OG images, colour and type rules. It's polish and judgment, not starting from scratch. Our agents do the digging, so you spend your time designing.
Bridge AI Consulting | AI Consultant, Junior AI Consultant | REMOTE (US) | Full-time | $60K–$100K base + upside for new business, equity possible | Visa sponsorship possible
We're a small, bootstrapped, profitable AI consulting firm. We embed with small and mid-size businesses (wealth management, real estate, construction, higher ed, nonprofits), learn how the work actually gets done, build what makes sense (mostly Claude: skills, agents, connectors, MCP) and train every person until they don't need us. We went from 1 client in February to about 20.
You'd own client relationships end to end: discovery, the build, the training. Strong builders who can also explain things to a 60-year-old business owner do very well here.
We build AI agents that run revenue for short-term rental and hotel operators on Booking.com and Airbnb: live prices, promotions and visibility on real portfolios, plus an MCP connector so customers work their accounts from Claude and ChatGPT.
Our engineers no longer type most of the code. Claude Code and Codex do. The hard part is knowing what to build, how it should fit together, and whether what the agent wrote is actually right. You will talk to customers, own the architecture, write the briefs, and catch what the agent missed.
We do not care which language you know.
We care that you can do architecture (single source of truth, clear boundaries, the right layer for a fix) and that you already ship this way.
Big bonus if you have worked on hotels, short-term rentals or dynamic pricing.
Small team, paying customers, hard problems (private OTA APIs, PMS integrations, pricing decisions that move real money). Three interview stages, no take-home. Email me (CTO) directly: martin@mydatavalue.com
Synteq | Growth Lead & Systems Administrator | Austin, TX | REMOTE (US) | Full-time
Synteq is a developer cloud running our own hardware and network. We do large-scale bare-metal, GPU, VPS, storage, and PaaS deployments across 5 US datacenters.
- Growth Lead: own top of funnel for a developer cloud product, covering SEO, community, PLG, and partnerships. Open to people new to growth or with some experience.
https://synteq.com/career/growth-lead
We're preparing a U.S. Department of Education SBIR Phase IA proposal to build an assessment reasoning platform for school speech-language pathologists. It implements TTL-MSM, a fully specified decision framework for one of the hardest calls in school eligibility evaluations: whether a student's communication profile reflects a disorder or a difference. Clinicians code each source of evidence, and a fixed rule combines them. The rules are written; the software isn't. The research lead is at Baylor University.
Prototype Developer ($70–95/hr): build a browser-based web app from a written spec, with privacy designed in from the start so it's ready for FERPA-protected student data later. About 15–20 hrs/week for nine months if funded.
Technical Product Lead ($75–100/hr): the bridge between the clinical spec and the engineering. Scope the prototype, turn the spec into requirements, and keep the build on schedule. About 5–8 hrs/week for nine months if funded.
Pre-award scoping (10–15 hrs for the developer, 5–10 for the lead) is unpaid, and funded work is contingent on award. U.S. citizens or permanent residents.
Apply (structured application, about 30 minutes; replies within five business days):
Kanmon | Full Stack Engineer (2–4 YOE) | San Mateo, CA | Hybrid (2-3 days onsite)
Kanmon is a venture-backed fintech startup making business financing more accessible.
Our team is growing and we are looking for an ambitious full stack engineer with 2–4 years of experience who loves building, moves fast, and wants real ownership.
You will work with a small, talented engineering team, ship meaningful products, and have a real impact from day one. It's also a role where you wear many hats.
Fusionality builds integrated operations technology for fusion devices, spanning real-time diagnostics, control, actuators, data systems, and simulation. Built on decades of operational experience at the Swiss Plasma Center at EPFL, we help fusion teams run their machines more safely, reliably, and efficiently through combined physics, software, hardware, and control expertise.
CoreWeave | Senior Specialist Field Engineers (Kubernetes, Security, Storage, Compute Infrastructure) | Full-time | Hybrid, 3 days/week in office: San Francisco CA, Sunnyvale CA, or Bellevue WA | $182k-$275k base + bonus + equity
I run the US West Specialist Field Engineering team at CoreWeave. We work hand in hand with the labs pushing the boundaries of AI, on some of the largest GPU Kubernetes clusters in the world. Specialist Field Engineers own the technical side of the customer relationship: architecture, proofs of concept, and the hard debugging nobody else can do. What you learn in the field feeds straight back into the product.
Must live near one of the cities above.
This is a hybrid role, and we do not offer relocation.
Must have deep experience in these areas.
It's in the title. The work we do goes exceptionally deep, and we need people who are already experts in one of these areas.
* Kubernetes [
https://grnh.se/7j1kydde6us
] You've written a production k8s operator, in go or rust. Bonus: GPU scheduling (device plugins, topology, MIG, RDMA), control plane scaling at thousands of nodes, Slurm, CNCF contributions.
* Security [
https://grnh.se/jxbek1o66us
] You've secured large-scale infrastructure and can trace an identity or isolation problem from the IdP, through the kernel and network, into Kubernetes. Bonus: threat modeling against SOC 2/FedRAMP, confidential computing and attestation, Falco/Tetragon, DPU/GPU/secure boot.
* Storage [
https://grnh.se/137kuvyv6us
] You've built or run a parallel filesystem (VAST, WEKA, Lustre) at scale, and can explain in detail how S3 works under the hood. Bonus: NFS-over-TCP debugging at scale, checkpointing for AI training, CSI drivers, kernel NFS/Lustre/Ceph contributions.
* Compute Infrastructure [
https://grnh.se/q3hmrcf56us
] You've brought up and validated large GPU clusters on GB200/GB300-class hardware, from InfiniBand/RoCE fabric validation thorugh PXE boot and NCCL acceptance. Bonus: firmware/BMC/BIOS, bare metal, Kubernetes, or Slurm.
If you're local and that's you, apply through the links above (they route to me) and mention HN.
University of Cambridge, University of Manchester | PhD in AI / Earth Observation / Environmental Science | ONSITE | UK home fee status only | £30k tax-free annual stipend
Our group at Cambridge works on Earth Observation foundation models (Tessera: v1
https://arxiv.org/abs/2506.20380
, v2
https://arxiv.org/abs/2607.03949
,
https://geotessera.org/
) and we've started a collaboration with Manchester to understand how they can be used to improve pollution models and then environmental decision making in general. This involves training new models and doing high-resolution inference across national or even global scales, as well as figuring out how to use that data to help policymakers make better decisions.
As part of this work we have a PhD studentship through the joint Manchester-Cambridge Centre for Doctoral Training for AI in Decision Making for Complex Systems. This involves one year at Manchester doing taught postgraduate study and then three years at Cambridge for the PhD.
Our work is all open source and involves solving significant compute and data challenges. Training our current-generation model takes ~250k GPU hours and a single year of global inference needs to move over 5.5 petabytes of data through the inference pipeline. This project would suit someone with strong machine learning and software engineering experience. We can help you build the atmospheric science and Earth Observation skills.
We are a highly collaborative group who spend our time not only on foundational research but also on developing applications with practitioners and decision makers. Our work sees use in many industrial, academic and government environments for problems like crop classification, tree species mapping and change detection.
You can find out more here:
https://www.findaphd.com/phds/project/from-satellite-to-stre...
- if you have questions about the project you can contact me, Sadiq Jaffer, at sj514@cam.ac.uk . For questions about the application process you'll want aidecisionscdt@manchester.ac.uk .
Application deadline is Friday 4th of December with a course start date of September 2027.
United Risk | Early-Career AI Engineers – Full-Stack, Data Science, Infrastructure/DevOps (10 openings) | New York, NY | HYBRID (2 days/week in office) | FULL-TIME | $100K–$120K
United Risk is a global property and casualty insurance underwriting and distribution platform, with practices from surety to aviation to fine art. Our in-house AI team builds the tools that make specialist underwriters quicker and smarter: submission ingestion, underwriting workbenches, risk analysis and back-office automation.
All roles: 0–4 years of experience (internships and co-ops count; new grads and junior engineers welcome), strong Python, working SQL, and something you built that you can show us: GitHub, a deployed app or a portfolio. Each role leans one way, but everyone works across apps, data and infrastructure, and three senior AI engineers review your work. Start dates from December 2026.
Hybrid: in office at Rockefeller Center, NYC; you'll need to live within commuting distance (NY/NJ/CT).
Stack: Python, SQL Server, Azure AI services (incl. Document Intelligence), Claude and OpenAI APIs, self-hosted models on our own GPUs
Benefits: Our Benefits Include:
- 100% employer-paid medical, dental, and vision insurance for employees.
- 401(k) plan with 100% immediate vesting and a 4% company match.
- Paid time off (PTO) and paid holidays.
- Life, disability, critical illness and accident insurance.
- Employee Assistance Program (EAP).
- Pre-tax Flexible Spending Accounts for health, dependent care, and commuter-related expenses.
- Tuition reimbursement.
- Fitness reimbursement and various additional quality-of-life benefits.
Domain expertise:
- Academic: PhD in mathematics (topological data analysis, representation theory), MSc finance, MSc neuroscience.
- Applied: database proxies/networking, quant dev and backtesting engines, ML/DL for banking & finance (risk/fraud), LLMs and agentic frameworks (tools, planning, eval), image processing, audio processing.
Résumé/CV: available upon request
Systems-oriented ML/LLM engineer specializing in Python/PyTorch and Rust for low-latency, high-throughput services. I have 8+ years of professional experience with Python and machine learning, and 5 years of production experience in Rust.
I thrive at the intersection of complex math and low-level systems. I've worked across multiple heavy domains, with deep experience building high-performance database proxies, modeling financial risk and architecting LLM agentic workflows, alongside applied work in audio and image processing. Actively exploring remote or relocation opportunities where I can apply my deep mathematical background and backend developer skills to hard engineering challenges.
Open to full-time roles or freelance/contract work (able to invoice B2B via my own company).
Email: hn [at-symbol] vixe.re
Location: Spain
Remote: Yes
Willing to relocate: No
Technologies: TypeScript, Node.js, Express, React, Next.js, TanStack Query, TanStack Start, PostgreSQL, Prisma, Better Auth, REST, tRPC, Docker, GitHub Actions, Vitest, Playwright, Cypress, Cursor / AI-assisted development
Résumé/CV: https://drive.google.com/file/d/1FQ2nmpRDoXOHQfy2A8Fn-u_uTsyZbhqO
Email: tomas at kikutis.me
Full-stack engineer with 15 years of experience - half of it remote - with a strong emphasis on quality, maintainability, and architectural rigor. Interested in Node.js/React work - both full-time and contract positions.
Today we’re publishing the firmware source code for the original Tracker, Tracker+ and Tracker Mini on GitHub. We’re also releasing
Tracker Community Edition 1.0
with tons of new features and improvements. You can install it on any of the three instruments, modify the firmware yourself and share your own version. We’ve built in a recovery system to keep a faulty firmware build from bricking your Tracker.
Explore the Tracker Community Edition repository on GitHub
“Here at Polyend, creativity is what we value most. We’ve seen it firsthand in the unofficial Tracker mods you’ve built, and we’re sharing the firmware source code to support that work.
Maybe you want to build an entirely new instrument on the Tracker hardware. Or maybe there’s just one thing you wish your Tracker did differently. Either way, you can now try it on the unit you already own. I bet some of you already have ideas for the original Tracker that never crossed our minds.”
Piotr Raczyński, CEO of Polyend
What’s new in Community Edition 1.0
Stream long samples from the SD card.
Bring full-length stems, loops and field recordings into a project using a fraction of the sample memory they’d normally need. This works even on the original Tracker.
Record timing and velocity while keeping both step FX slots free.
Micro-move and Volume now have dedicated fields on every step. New Chance conditions let you bring in a snare fill every fourth pass.
Load plugins in the Sample Editor.
Six effects and generators are included, from an analog-style kick generator to a stereo delay. The plugin SDK lets you build, share and sell your own.
Play two new synths on Tracker+ and Tracker Mini.
CHIP
and
DIRT
join the synth lineup. A new SDK lets you build your own synth engines.
Huge thanks to
@Piotr
and the whole Team at Polyend for allowing me to do this
.
I’m looking forward to see what the community can come up with going forward!
If you are planning to contribute, please read the announcement in the Github discussions:
I am not sure, if this marks the end of Tracker series sales - since I don’t see them available anymore - so if this is the EOL treatment for the Tracker, this is what other companies should strive for too.
If not, then this is the most impressively open music hardware platform going forward I have seen to date and a great way to have a community building awesome physically controllable synth and groovebox apps
Yes, this will not be a one-shot (pun intended). All us devs can now help improve the device we love!
Yes, if this is the “send-off” then what a nice send-off.
Conversely, perhaps the updated firmware, engines, and the open-sourcing stimulates new / more demand.
The Tracker family physical form-factor and horsepower makes a very capable development canvas for folks to explore / exploit. When people are inspired by what they see / hear, they tend to want it.
Hell yes! This is an appropriate response to the current environment of equipment and firmware development. Kudos for leading by example on this one. Happy to own a Tracker+ and a Play+ before, and now I love them even more.
Securing Internet-Critical Software: How We Could Have Backdoored Bash
A shell injection hidden in an email address gave us code execution on GNU Savannah, then root, then the ability to push to the upstream Bash repository. We reported it, kept auditing, and helped the FSF fix 26 vulnerabilities in the forge that hosts GNU.
7 min read
TL;DR:
GNU Savannah hosts Bash, Coreutils, and hundreds of other GNU projects. Using Hacktron, we found that an email address like
a`touch${IFS}/tmp/savane-rce-proof`@attacker.example
gave any registered user code execution on its backend. From there, we made our account a superuser, impersonated Bash’s maintainer, and confirmed we could add an SSH key with push access to the upstream Bash repository. An unpatched kernel gave us root access and access to a GNU backup server. We reported it to the FSF on May 11, 2026.
We then ran Hacktron’s pentest across the rest of Savane to find further bugs and reported those too, bringing the total to 26 confirmed vulnerabilities. The FSF fixed them and published an
incident report
.
Exploit chain
Regular account
savannah.gnu.org
Email address
Shell metacharacters allowed
GPG key test
Email in a shell command
Web server RCE
Database access
Superuser
Impersonate any user
Bash maintainer
Add SSH keys
OverlayFS
Unpatched kernel
Root
Savannah backend
Backup server
GNU
Why Savannah
Savannah is where much of the GNU userland lives, including Bash and Coreutils, which provides commands such as
ls
,
cp
, and
rm
. It holds the repositories, the maintainer accounts, and the SSH keys that decide who can push. Code that lands there gets packaged by distributions and installed on millions of machines, and the
xz backdoor
showed what one compromised link in that chain can do.
Like much of the infrastructure free software depends on, it’s run by a handful of volunteers and FSF staff with limited resources. We wanted to put our resources to work there. Savannah runs
Savane
, a PHP application that is itself free software, so we pointed Hacktron at the source and audited it.
What came back is probably the most impactful bug we’ve ever found.
Savane’s email validator (
frontend/php/include/account.php:275
) accepts dot-atom addresses with characters like backticks,
$
,
{}
,
/
, and
|
. Most of these are technically allowed in the local part of an email address by the RFC, so the address above passes validation. Any user can set it at registration or through the email-change flow (
frontend/php/my/admin/change.php:271
).
The sink
Every logged-in user can manage a GPG key for their account, and Savane offers a “Test GPG keys” button at
/my/admin/change.php?item=gpgkey
. One of the checks it runs lists key fingerprints for the email address on the account. In
frontend/php/include/gpg.php
:
function test_gpg_command ($temp_dir,$command,&$ret,$in=null)
function test_listing ($temp_dir,$level,&$ret,$test_against_email=false)
{
...
if ($test_against_email)
{
$email =user_get_email (0);
...
}
$options ='--list-keys --fingerprint';
if ($test_against_email)
$options .=" $email";
$res =test_gpg_command ($temp_dir,$options,$ret);
}
The email is appended to
$options
as plain text,
$options
is concatenated into
$cmd
, and
utils_run_proc()
hands that string to
proc_open()
. With a string command, PHP runs it through
/bin/sh
. The shell sees our backticks and runs whatever is between them before
gpg
even starts.
${IFS}
is there because the validator rejects spaces, and the shell expands it to one.
Exploiting it takes four steps:
Register with the payload above as your email address, or change an existing account’s email address to it. The app accepts it.
Open
/my/admin/change.php?item=gpgkey
.
Paste any GPG key and click “Test GPG keys”.
The command inside the email runs on the server as the web server user.
The FSF later traced the vulnerable path to the March 7, 2026, release of Savane (3.17-c369). It affected both savannah.gnu.org and savannah.nongnu.org, so the window was about two months.
From a shell to superuser
To see how far it went, we swapped the payload for a reverse shell and landed on the backend as the web server user.
The web server user can read Savane’s configuration, which includes the database credentials, so there was nothing to crack. We set the
superuser
flag on our own account,
harshjaiswal
, and reloaded the page.
Superusers get a “Become this user” box in the sidebar, which lets a site admin act as any account without its password.
Becoming Bash’s maintainer
We typed
chet
into that box. Chet Ramey maintains Bash.
That’s his SSH key page, with an empty slot for one more. Savannah’s Git hosting is keyed off these entries, so adding a key here grants push access to
bash.git
as the maintainer. We confirmed the page accepted edits and stopped there. We did not add a key and we did not touch the repository.
An attacker in this position could have pushed a commit to upstream Bash under Chet’s name and done the same to Coreutils or any other project on Savannah.
Root and a backup server
The web shell was unprivileged. Savannah runs Trisquel GNU/Linux 11, with a linux-libre kernel built from Ubuntu 22.04’s 5.15 LTS kernel. That kernel was still affected by
GameOver(lay)
, two OverlayFS bugs from 2023 (CVE-2023-2640 and CVE-2023-32629). Canonical fixed the 5.19 and 6.2 kernels back then, but the 5.15 kernel didn’t get a fix until USN-8255-1 on May 7, 2026, four days before our report. The public exploit worked on the first try and gave us root on the Savannah backend.
As root, we found a key that let us SSH to a GNU backup host as the
gnubackup
user.
At that point, we had root on the forge, access to its backups, and the ability to act as any maintainer. We stopped doing recon and wrote the report.
The rest of Savane
The RCE was one finding of many. After the initial report, we kept Hacktron running on the Savane codebase, which descends from the 2001 SourceForge code. It produced 33 findings; we threw out 7 false positives and sent the FSF the remaining 26 (2 critical, 6 high, 16 medium, 2 low) with file paths, line numbers, and reproduction steps. The notable ones:
ID
Severity
Finding
HCK-001
Critical
Command injection to RCE through the GPG key test email
HCK-002
Critical
Authorization bypass in site-wide admin group configuration (
siteadmin/triggercreation.php
)
HCK-003
High
Any group member can delete the group’s mailing lists
HCK-004
High
Any user can change any group’s tracker notification addresses
HCK-006
High
Any group member can reset mailing list admin passwords
HCK-008
High
Private file access trusts a
file_uid
parameter from the request instead of the session
HCK-010
Medium
Session hijacking: cross-domain SSO passes
session_uid
and
session_hash
in the URL
HCK-012
Medium
Password reset and email change confirmation hashes never expire
HCK-017
Medium
Stored XSS through the real name field, which only strips
<
Working with the FSF
Corwin Brust, a Savannah administrator, replied the same night. The FSF treated it as a full compromise: it patched the kernel, audited access controls, databases, and logs, and warned the operator of the other public Savane instance. The authorization bugs were fixed through a refactor rather than individual patches.
Disclosure timeline
Date
Event
2026-03-07
Savane 3.17-c369 ships with the vulnerable GPG key path
2026-05-07
Canonical publishes USN-8255-1, fixing GameOver(lay) in the 5.15 kernel
2026-05-11
We report RCE, root access, and access to the Bash repository to savannah-hackers-private
2026-05-12
Corwin Brust, Savannah administrator, acknowledges the report and our 30-day disclosure window
2026-05-14
FSF installs the fixed kernel and removes our account
May 2026
We send the remaining 25 findings from the full Savane audit
Thanks to Ineiev, Corwin Brust, and Bob Proulx from the Savannah hackers and the FSF sysops for working with us.
Work with the team behind this research
Hacktron brings together top CTF researchers, experienced red teamers, and offensive security researchers. We use AI to accelerate security research, finding and eliminating vulnerabilities in widely trusted software before malicious actors do. We’re continuing our research across the infrastructure the world depends on. If you’re responsible for securing some of it, we’d like to work with you through
red teaming
,
pentests
, or
AI code review
. Reach out at
hello@hacktron.ai
.
BTCPay Server warns of hot wallet thefts targeting outdated instances
Bleeping Computer
www.bleepingcomputer.com
2026-11-02 10:03:07
The maintainer of BTCPay Server, an open-source cryptocurrency payment processor, has urged users to install the latest version as hackers have been draining "hot wallets" on instances running older versions. [...]...
The maintainer of BTCPay Server, an open-source cryptocurrency payment processor, has urged users to install the latest version as hackers have been draining “hot wallets” on instances running older versions.
Operators were initially advised to
shut down their servers
pending a full investigation into the unauthorized payments.
The BTCPay Server project provides businesses with a self-hosted payment processing solution that allows them to accept Bitcoin and other digital assets directly into their own wallets, eliminating payment intermediaries or paying platform processing fees.
It handles checkout invoices and payment verification, including payments over the Lightning Network.
Early Monday, the maintainer said it had “identified an attack vector targeting instances running version 2.4.1 (July 23, 2026) or earlier” and strongly urged users to update to version 2.4.7.
BTCPay founder
Nicolas Dorier said
the attacks did not target public servers. The investigation focused on determining how the threat actor was able to log in as an administrator to instances running versions older than 2.4.2.
On October 29, the company disclosed a critical vulnerability discovered via Anthropic’s Frontier OSS scanner, which prompted the release of version 2.4.7 as an
emergency security update
.
However, Dorier said that there were clues that the attacker used vulnerabilities in BTCPay Server versions earlier than 2.4.2, which the project later confirmed.
“Our incident response investigation identified an attack vector targeting instances running versions 2.4.1 (Jul 23, 2026) or older,”
BTCPay Server says
.
“We have no credible reports of hot wallet drains for users running the latest 2.4.7 release. We again urge everyone to keep their BTCPay Server updated.”
Impacted users are requested to send all details and activity logs to security@btcpayserver.org.
It is worth noting that on August 7, when BTCPay Server
released version 2.4.2
, it warned about an actively exploited flaw exposing LND credential files that allowed attackers to control affected Lightning nodes and steal funds.
The project has not clarified what flaw is being exploited in the recent attacks.
BleepingComputer has contacted BTCPay Server for more details about the vulnerability exploited in the current attacks, but we have not received a response by publication time.
Impacted users should update to version 2.4.7 through Server Settings → Maintenance → Update.
For Docker installations, administrators can update BTCPay Server by logging in over SSH, running sudo su -, changing to the deployment directory with cd btcpayserver-docker, and executing ./btcpay-update.sh.
If running 2.4.1 or older, it is recommended to take the instance offline as soon as possible.
Your Mobile Carrier Stores Your Call Logs for Years
Bleeping Computer
www.bleepingcomputer.com
2026-11-02 10:01:11
Mobile carriers can retain years of call and text metadata, exposing sensitive details about customers' relationships, locations, and daily routines if breached. Cape examines the privacy risks of long-term telecom data retention and how its mobile service reduces exposure by automatically deleting ...
In 2024, AT&T lost sensitive communication logs of 109 million customers in a massive data breach. Call logs can reveal a lot about you, like your closest relationships, where you live and work, and your daily routine.
109 million people (nearly a third of Americans) suffered a
massive data breach
in April 2024 that exposed their AT&T communication logs.
AT&T lost sensitive information including phone numbers, call and text counts, call durations, and location data from May to October 2022, and January 2023, meaning the data was approximately two years old at the time of the breach, impacting both current and former customers.
AT&T said the records didn’t contain call content, customer names, or Social Security numbers. That’s less reassuring than it sounds, because even on their own, call logs reveal a lot about you.
What are call logs?
Every call, text, and data session generates records inside your carrier’s network. The two kinds that matter here are:
SIP messages.
Session Initiation Protocol (SIP) manages calls and texts, both within a carrier and between networks. One message starts a session and another ends it. Carriers use them mainly to troubleshoot dropped calls and failed texts.
Call Detail Records (CDRs).
CDRs summarize each call, text, and data session: who was involved, when, for how long, and often which cell site your phone used. Carriers use them for billing and for settling charges with roaming partners.
Anyone with access to your call logs can identify your closest relationships, locate your home and workplace, and reconstruct your daily routine in more detail than you could yourself. If you called an oncologist a few years ago, there’s likely a record of it, how long you talked, and where you were. If you texted an old friend afterwards, there’s a log of that too.
Call metadata can even be correlated with publicly available information to derive identities.
Malte Spitz
showed how far this goes. In 2011 he sued his telecom for the data it held on him, received roughly 35,000 records, and reconstructed a strikingly specific replay of several years of his activities.
Why call logs are easy to get
Three things make them an easy target.
They’re kept for a long time.
Most carriers don’t say how long. Privacy policies typically promise to retain data for “as long as necessary to fulfill the purposes” of the policy. In 2022, responding to a request from the FCC chair, carriers disclosed that they kept geolocation data for as long as five years, far longer than phone service requires. One reason is
to monetize it
.
They sit in a few big targets.
Carrier data is
frequently breached
. AT&T was one of many Snowflake customers hit by credential-based thefts, alongside Ticketmaster and Neiman Marcus. Salt Typhoon, the China-linked campaign, reportedly compromised every major US carrier, which could put call metadata for any American within reach.
They have other buyers.
Retained logs are used to build detailed behavioral profiles that are sold to ad networks and data brokers and, in some cases, handed to the government.
The AT&T breach is the predictable result. The records were available to steal because they existed.
Cape is America's privacy-first mobile carrier providing unlimited talk, text, and 4G/5G data, built from the ground up with privacy and security at its core.
Most carriers track everything: where you go, who you call, what you do. Cape collects the minimum amount of information required to provide you with service, deletes call and text metadata after 24 hours, rotates your network ID to prevent tracking, defends against SIM- and network-based attacks, and more.
Privacy shouldn't cost more. Switch today and get $29 off for life.
Cape deletes call and text metadata in days instead of years. Unlike other MVNOs, Cape runs
their own mobile core
and produces their own logs, meaning they can control when logs are deleted.
When a Cape subscriber makes a call, the signaling goes to the nearest radio tower and to Cape’s partner network operator, which passes it to Cape’s mobile core. The core forwards it to the recipient’s carrier.
Cape keeps SIP messages for 14 days to debug network issues, converts them into internal CDRs, and deletes those after one day. Daily totals (counts of calls, SMS, MMS, and data used per subscriber) are kept up to 60 days for billing and spam prevention.
CDRs sent by partner networks as billing evidence are held for 30 days, long enough to dispute a bill.
Cape’s partners and the tower operators generate logs too. For that reason Cape
rotates the identifiers
the towers see, which weakens their ability to tie those logs to you.
The audit
A retention claim is only worth what you can verify. Cape hired
Trail of Bits
to audit the feature in May and July 2026 and published the
attestation
.
During the audit, Trail of Bits found no evidence of violation of the two given claims:
CDR files are being deleted within the retention policy.
Cape does not extract or retain data from CDRs about individual consumer customers, except daily totals. Daily totals are the number of calls, number of SMS, number of MMS, and amount of data a subscriber used in a day. This data is retained for no more than 60 days and then deleted.
You can’t un-breach a database, but you can shrink it
Better carrier security matters, but the more reliable control is to hold less. Records deleted days after a call can’t be stolen two years later.
For journalists, executives, security staff, and anyone who calls a doctor, lawyer, or source they’d rather not advertise, your carrier’s retention policy is part of your threat model.
The feeling of progress is a huge part of a game’s enjoyability. I am not 100% sure that is a word, but you know what I mean!
This is the difference between a “pick up and play mini game” versus “wow, look at the time, I just spent my entire work day on this!”. Hypothetically.
As of now, our C64 BASIC dungeon has no levelling up, no character development, just one map. Don’t get me wrong, lots of fun games have one screen and no character development, but they
do
usually increase difficulty as your session continues.
So next we need to add progress, difficulty ramping, and character stats buffing.
Progress so far
While we only have one map, it is a playable one now. When you find the chalice the game ends with “YOU WIN” and a play again prompt.
Selecting to play again rolls a new character and generates a new map. All your progress is wiped, it is essentially the same as hitting ‘N’ for new game.
You can now follow the tutorials and edit the code right in your web browser with the
Online Retro IDE
– No downloads, configuration, etc necessary, and it is free!
Defining what a level is now
In this version we move to have five levels. Each one generates a new map using the same generator as part ten, but with more rules around object placement and creation.
Rather than one chalice, now to leave a level you need to grab every chalice in the dungeon, only then can you walk onto the stairs. The number of chalices increases as the level goes up, and just to make it interesting it is not a simple +1:
Level
1
2
3
4
5
Chalices
1
1
2
2
3
LV
is the level number,
CL
is chalices left.
INT
throws away the fraction, so levels 1 and 2 give 1, levels 3 and 4 give 2, level 5 gives 3.
Your character now persists while they are still alive, keeping your hit points, stats, gold, keys and magic.
For now, the stairs on level 5 gets you out of the dungeon and wins you the game.
Win or lose, we now also show the level you reached as well as the gold acquired.
The end panel after leaving level 5: YOU WIN, the gold total and LEVEL: 5
One routine builds a level
In part ten, I set the player up and drew the map in two places, starting a game, and the
N
key for a new game.
That code is now split in two.
INITPLAYER
is for everything about the character and only runs once per game.
BUILDLEVEL
is now everything about each level and runs once per level.
PX
,
PY
put the player back at the start cell.
WN
is the “leave this level” flag.
LA
is the number of lines in the message log, zero because the screen has just been cleared.
A new game is
GOSUB INITPLAYER
then
GOSUB BUILDLEVEL
. Going down the stairs is
GOSUB BUILDLEVEL
on its own.
That is the difference between resetting the game and resetting the level, and is what lets the character survive the map generation intact.
Using the stairs
The stairs are placed in one of the four corner rooms, picked at random. Your player starts in the centre room, so the stairs can never be in the room you start in.
EX=4*INT(RND(1)*2) : EY=4*INT(RND(1)*2)
INT(RND(1)*2)
is 0 or 1. Times 4 makes it 0 or 4, the first or last room across, and the same again for down.
Part ten’s map rule already makes every room reachable from every other, so any selected room should be a safe choice. I don’t check that the stairs can be reached, because by that logic they always can be.
I do force the stairs room to the open room type, and the stairs are placed in its centre cell.
IF RX=EX AND RY=EY THEN MT=1
The reason is stairs block movement until the chalice(s) are collected. We don’t want to block movement by placing the stairs in a bad spot.
Getting the position of that centre cell comes from the room numbers. A room is 4 cells wide including its left wall, and the map starts on screen row 2.
Gathering chalices and activating the stairs
Each chalice is still worth 10 gold and counts
CL
down. When it reaches zero the log tells you to “FIND THE STAIRS”.
POPULATE
places as many chalices as the level needs, then sets
CL
to the number that were placed.
PLACEONE
gives up after 200 failed tries at finding empty floor, so on a crowded map a chalice could actually fail to appear. Counting only the ones that were added means the level can still be finished.
Q=CL : CL=0
IT$="{PURPLE}(" : FOR N=1 TO Q : GOSUB PLACEONE : CL=CL+PF : NEXT N
PF
is
PLACEONE
‘s “it worked” flag, 1 or 0, so adding it up counts the successes.
Walking into the stairs with chalices still unpicked bounces the player back with “CHALICES LEFT” and the number.
With none left, the stairs set
WN=1
and return.
WN
started life as ‘won’ in part 10, and now it means the level is won, not the entire game!
The main loop sees the flag and jumps to
NEXTLEVEL
. It’s the same flag-then-
GOTO
approach we used for winning in part ten. The stairs are found inside a
GOSUB
, and building a new level from in there would leave that
GOSUB
unfinished on BASIC’s stack every time.
Levelling up
For now, each new level rewards your character hit points equal to half your constitution, rounded down. We also add +1 to strength on even levels, and +1 dexterity on odd ones.
This is mainly to show how it can be done at this point, we can give more thought to what would be actually beneficial later in a polishing phase.
HL=INT(STATS(2)/2) : HP=HP+HL
S=LV AND 1 : STATS(S)=STATS(S)+1
AC=10+INT((STATS(1)-10)/2)
STATS(0)
is strength,
STATS(1)
is dexterity,
STATS(2)
is constitution.
LV AND 1
is 1 for an odd number and 0 for an even one.
AND
compares the two numbers bit by bit, and the lowest bit of any odd number is 1. So the answer is also the array index of the stat to raise.
Armour class is worked out again afterwards, because it is derived from your dexterity.
Strength helps your to-hit roll and the damage you deal via its modifier. A modifier only changes on every second point, so one point of strength does not always immediately make much difference. Again, balancing character abilities versus good fortune is a balancing thing once you have code working.
The log reveals to you the new level, the stat that went up and the hit points gained.
Potions finally become magic
Collecting a
*
potion now adds 50 magic, up to a ceiling of 100.
IF CH=42 THEN MG=MG+50:MSG$="MAGIC UP 50":GOSUB TAKEITEM
IF MG>100 THEN MG=100
Nothing
spends
magic yet, but it is counted and shown, ready for the ranged attack in part twelve.
The numbers match the C version of this game, where the same pickup gives 50 and a spell costs 5 plus 1 for each cell it travels.
Level 1 after drinking a potion: the status line reads 50 magic and level 1, and the stairs are in the bottom right corner room
The status line
This means we need two more counters, your magic, and the current level. As we take up more room, the gaps between the counters went from two spaces to one. Our status line is now 31 characters out of the 40 available.
It is built with two
PRINT
statements. The first ends in a semicolon, which keeps the cursor on the same line for the second part.
Still on the to-do list
First, we added some progression but
we really need to ramp up difficulty
. Every level still has the same 3 to 6 goblins, for example.
Second,
magic can be
collected
but not be used yet
.
Lastly, we still have some RAM available, so can we fit into memory
more pickups
and gameplay variety?
Some questions for you:
Should stairs work in both directions? Would revisiting a level respawn the enemies?
Collected keys carry over, so should we encourage holding a spare key from one level so you can open a door on the next?
Levelling up your character is automatic, should we offer the player a choice of which stat to raise?
How many levels should there be before game over?
The Weird, Wild Races We're Watching Tomorrow
hellgate
hellgatenyc.com
2026-11-02 09:55:10
Swingy districts, unbelievable candidates, and lots and lots of money. Plus more news to kick off your week....
Did you know there's an election Tuesday?
The Hell Gate podcast will be LIVE
on our YouTube channel
starting at 9 p.m., and we'll be bringing you the type of pipe-cleaner, construction-paper-based analysis you will NOT find anywhere else.
New York is hardly a swing state, but it's still taken center stage in this year's midterms. A Hudson Valley congressional seat is one of Democrats' best chances for a pick-up, and a North Country seat has become surprisingly competitive. In a year so grim for Republicans that they could lose a Senate seat in Texas, Vice President J.D. Vance is weirdly coming to New York
to rally
with an absolute car wreck of a Republican congressional candidate. If Democrats take back the House and Senate, leadership of both chambers
could fall
to two New Yorkers
. And while he's not on the ballot, Mayor Zohran Mamdani is looming just as large in this election as President Donald Trump is.
New Yorkers are clearly following this year's elections. Early voting turnout is
way up
. Kathy Hochul received a
kind-of warm welcome in Dimes Square
on Friday. Voter registration data released by the state Board of Elections showed that tens of thousands of people have registered to vote since February, most of them unaffiliated with a political party,
as City & State reported
.
Still, Hochul is having to answer for an affordability crisis that is pushing people out of the state as the economy seems to be on the brink of transformation. She's facing Bruce Blakeman, the Republican Nassau county executive who claims he can dramatically
reduce taxes
and energy bills. Blakeman, now 71, has had a
patchy career
in politics. He bounced around local office in Nassau in the '90s, then lost reelection to the county legislature following deeply unpopular tax hikes in 1999. Blakeman has made several attempts at higher office, running unsuccessfully for state comptroller, New York City mayor, U.S. Senate and for Congress. He did a stint at the Port Authority and then was appointed to the Hempstead Town Council in 2015. So far, his career has culminated with his
capture
of Nassau's previously Democratic county executive post in 2021.
Then came the theatrical MAGA turn: Sometime around 2024, as Donald Trump was running to take back the presidency, Blakeman got really on board, embracing an anti-trans, anti-immigrant, crime-fighting ideology. In March of that year, he
memorably held Trump's umbrella
for him while Trump addressed reporters outside the funeral of an NYPD officer. Blakeman ran for county executive on lowering taxes in Nassau, but he found more success charging into the culture wars when he
banned trans women from playing sports with cis women in county facilities
in July of that year. (A legal challenge to that law is ongoing.) A couple months later, he tried to gather a group of 75 dudes with gun permits to form an armed civilian group that he insisted was
NOT a militia
. He has raised eyebrows by hosting
campaign-style events
with Trump in county facilities or
using county resources
. He directed local police to cooperate with ICE and then called it the "one of the saddest days of my life as county executive" when a new state law forced him to
end that partnership
.
Blakeman has touted achievements in getting Nassau County on solid fiscal footing, but his decisions to use pandemic federal funds and opioid settlement funds to balance the budget were
controversial
. He has not managed to lower property taxes so far as county executive, and he's opted to
freeze property tax rolls multiple years in a row
. That approach has drawn a state audit and accusations that it puts the burden on individuals to contest their now extremely outdated property tax assessments.
This is Hochul's second reelection fight, and she's
already saying
she'll seek another term in 2030. Polling this year
has been all over the place
, but most surveys now show Hochul with a lead that at least hits the low double digits.
What's up with fundraising?
Hochul has a
huge, huge fundraising advantage
, which is all the more notable because she opted out of the state's brand-new public matching funds program. She's also gotten a boost from outside spenders, including former New York City
Mayor Mike Bloomberg
. Blakeman has opted into the state's matching funds program,
messily
, and he's
benefited from the largesse
of Madison Square Garden billionaire
James Dolan
.
Hochul Yankee Candle Recipe:
Aroma of toasted plain English muffin, the beery sweat of a Buffalo Bills bar, baby powder
It’s rare to see an executive go from fucking around to finding out so quickly. That’s what happened last week when Proton CEO Andy Yen announced he was canceling a planned re-interview process for almost all current employees that would have checked if they’re using AI enough. This follows extreme backlash from Proton users and workers against the company's
aggressive pivot to AI
. It’s been a wild ride internally, several employees past and present have shared with 404 Media.
Photo by Sam Barnes/Web Summit via Sportsfile via Wikimedia Commons / Collage via 404 Media
On Friday, Andy Yen, the CEO of Proton, cancelled a series of nearly company-wide ‘re-interviews’ that would have forced employees to explain how they are using AI, with the clear intention that they may be fired if not using it enough in the eyes of Proton leadership, according to six current Proton employees and audio of an around two hour all-hands meeting obtained by 404 Media.
The move came after
404 Media published an investigation
on Thursday, which showed how Yen was aggressively pushing AI across Proton, in some cases despite security concerns from employees. In one exchange, an employee in the company’s security Slack channel urged Proton to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.” Yen added, “stop lovingly handcrafting all our code and use AI more.”
The re-interview cancellations likely don’t address the concerns of many Proton users, who are more focused on Proton’s deprioritization of security; many Proton users have blasted the company’s leadership on social media over the last 24 hours and explicitly called for Yen to resign.
MORE FROM 404
Kinktober has come and gone.
But Archive of Our Own (AO3), a massive repository for fanfiction and creative writing, is still down, and
might be down for as long as a month
after going down for "maintenance" following a security incident last week. The Organization for Transformative Works (OTW), the non-profit organization that runs AO3, posted on its status page that the site restoration will take around two weeks, but could take up to a month.
People are coping how they can.
And they’re turning to anything but Wattpad to fill the void
. One writer I spoke to who posts her writing and reads others’ writing on AO3 told me that she was in the middle of racing to finish a work that was deliberately timed to Kinktober and Halloween when the outage happened. She was hitting “next chapter” while reading a fic on Wednesday night when she got the message that the site was down for maintenance, “which I initially took as a sign from the universe to go to bed,” she said.
This post is for paid members only
Become a paid member for unlimited ad-free access to articles, bonus podcast content, and more.
In case you missed it:
CVE-2026-98432
is a critical (CVSS 9.8)
bug in the Linux kernel's TCP code. It's been there since 2016, so virtually every Linux server on the planet
has it, and it can be hit remotely. Anthropic says it built a working exploit that gets root on a server.
A total takeover.
Debian
patched it on October 16
.
Ubuntu still doesn't have a patch more than two weeks later.
Every supported release, LTS included, is listed as "Vulnerable".
Ubuntu's website promises
"timely patches for critical vulnerabilities."
Ubuntu hasn't shipped the patch nine days after it was rated 9.8 and six days after a working root exploit.
Very fucking timely indeed.
Software is full of this blabber where words mean nothing. Ubuntu uses words like these:
"Timely." "Enterprise-grade." "Production grade." "Enterprise resilience. Developer velocity. Build with confidence on Ubuntu."
Somebody, or perhaps worse, some special team at Canonical is dedicated to writing these........
So I'm asking again... What the fuck, Ubuntu?
I'm moving all my servers off Ubuntu. I've completely lost trust in it.
On the desktop I always hated it anyway. On servers I mostly used it because it's the default
on most VPS providers, and I considered it the solid, no-drama LTS distro. Fuck that now.
I can't express how crazy this is. I didn't have time to migrate my services and data when I learned
about this, so I had to take all my servers that run Ubuntu offline.
Oh and btw, distros run by volunteers (Debian, Arch, Alpine...) hve shipped the fix. Meanwhile Ubuntu,
made by a company that sells security subscriptions, hasn't. This is true as of 2026-11-02 14:48 UTC.
We halved DNS build delay and cut memory use by 85%
When I
last wrote about our DNS propagation pipeline in 2022
, the Zone Builder received about 250 DNS record changes per second. Today it receives about 700, nearly three times as many, plus about 100 other events, such as
zone
setting changes. However, the number of events was never the real problem. The problem was the kind of work some of those events needed, and how that work grows with the size of a zone. Some zones now hold more than 20 million records, and we expect them to keep growing toward hundreds of millions.
Each event starts a clock that measures how long it takes us to turn that change into data our authoritative DNS servers can use. We call that work a build. A build reads the current database state, converts it into the format used at the edge, and writes any changes to
Quicksilver
.
In 2022, per-record builds were 150 times faster than full builds on average. On a test zone with one million records, per-record builds were 4,250 times faster. We ended that post with the obvious next step: do the same thing for zone settings.
Why doesn't the Zone Builder just rebuild the setting that changed?
It should, and that is where we ended up. However, proving when it was safe took most of the work. A record event identifies what changed. A zone event can come from many Cloudflare products, may not say why the zone changed, and can alter data shared by several zones with the same name. Choosing too much work wastes capacity. Choosing too little can leave stale or incorrect DNS data at the edge.
We built a targeted build planner that chooses the smallest build we can prove is safe, and a new scheduler that combines work for the same zone. Zone setting changes no longer rebuild zones with tens of millions of records. Together, they cut peak memory by 85%, and the extra headroom let us safely shorten waits the pipeline has always needed for resilience, halving the time from a DNS change to its data being written to Quicksilver.
How DNS changes reach the edge
A DNS zone is the collection of records and settings for a domain. Cloudflare uses a multi-stage pipeline to turn a change to that zone into data available across our
global network
.
A customer changes a DNS record through the
DNS Records API
or dashboard, or another Cloudflare service changes a zone setting.
The change is stored in a database.
The database change or another system produces a
Kafka
event.
The DNS Kafka Aggregator combines events from several Kafka topics and sends them to the Zone Builder.
The Zone Builder reads the required zone data and converts it into keys understood by our authoritative DNS servers.
The Zone Builder compares those keys with
Quicksilver
, our distributed key-value store. It writes only what changed, and Quicksilver propagates those changes across Cloudflare's network.
Diagram of the DNS propagation pipeline: the DNS Records API, zone settings, and internal services write to databases or Kafka topics, the DNS Kafka Aggregator combines them, and the DNS Zone Builder writes to Quicksilver.
This is a simplified view. The Zone Builder does not copy database rows directly to the edge. It combines records with zone settings, applies DNS serving rules, and organizes the result so our authoritative DNS servers can read it quickly.
Why full builds stopped scaling
The original Zone Builder had one useful property: it did not need to know why a zone changed. It rebuilt the desired state from the current database state.
Read the current zone and its settings.
Read every DNS record in the zone.
Convert those records into Quicksilver keys.
Compare the new keys with those already stored.
Remove stale keys, write changed keys, and update zone metadata.
The final write contains only changed keys. However, a full build still reads, encodes, and compares every record, and holds the whole zone in memory while it does so. That is cheap for a small zone. For our largest zones, with tens of millions of records, the zone does not fit in memory at all. I explain how we handle that below.
Zone settings made targeting harder.
Universal SSL
is one example. Enabling or disabling it updates metadata and can change Certification Authority Authorization (CAA) records at the zone apex, the bare domain, such as
example.com
.
CAA records
tell certificate authorities whether they may issue a certificate for the domain.
A full build only needed to know that the zone changed. A targeted build must answer three questions:
Which DNS names can change?
Can shared metadata or the list of published zones change?
What previous state must be removed from Quicksilver?
Answering those questions safely meant solving five problems, which the rest of this post covers in order:
Events had to say what changed.
Each change had to map to the smallest build that is still correct.
Builds had to account for data shared by every zone with the same name.
The database could change between the event and the build.
Combining work could not lose a Kafka message.
Making events say what changed
We changed the event producer to include changed zone fields and settings, with previous and new values when available. We then traced each setting through the code that generates DNS output and classified what it could affect.
Not every event was that specific. Some database functions only update a timestamp to say, in effect, "something about this zone changed." Sometimes that is harmless bookkeeping. Sometimes it means data changed elsewhere and must be read again.
Those cases look identical in the event. Until we can identify the caller and its purpose, a timestamp-only update still causes a full build. The same rule applies to missing details, unknown events, or any change that may affect records anywhere in the zone. The planner never turns uncertainty into less work.
Choosing the smallest safe build
Quicksilver output has two parts that matter here. Record keys contain the DNS answers for one zone and name. A smaller shared metadata key tells our authoritative DNS servers how to use that zone, including where to find special records such as nameservers and wildcards. Some events change only record keys; others must update both.
With that distinction, the planner can choose one of five build types:
Resources only:
Rebuild the records at specific DNS names, which we call resources, without changing zone metadata. Ordinary A, AAAA, TXT, and CNAME changes usually take this path.
Resources and metadata:
Rebuild specific names and shared metadata. Nameserver and wildcard changes use this path.
Apex and metadata:
Rebuild the zone apex and metadata. Settings such as Universal SSL can use this path.
Purge:
Remove a zone that should no longer be published.
Full zone:
Rebuild every record when smaller work cannot be proven safe.
The rule is simple: every key a targeted build writes must be byte-for-byte identical to what a full build would write. If we cannot prove that, we do a full build.
Removing old state works as it did for per-record builds in 2022. Record events include deleted records and the previous version of updated ones. The Zone Builder then scans Quicksilver for the existing keys under each name it rebuilds, removing any that the new build did not write.
Full builds remain necessary when a change can affect records anywhere in the zone or when a zone may need to be published for the first time.
There is one deliberate exception. A full build holds every record from the database and every existing Quicksilver key for the zone in memory at once, and our largest zones do not fit. Above a fixed record-count threshold, the Zone Builder never escalates to a full build. It rebuilds the apex, the metadata, and every name identified by the queued events instead. We accept this tradeoff because a full build of those zones cannot finish, and, while it retries, it holds back every other change on its Kafka partition. Many of the cases that would otherwise need a full build involve several zones sharing one name, and
zone holds
reduce that risk by stopping the same name from being added to another account. For the rare cases that remain, such as enabling a zone on another network, we can manually rebuild individual names in the zone.
When one name means more than one zone
Quicksilver stores each zone's record keys separately. It also stores one metadata key shared by every Cloudflare zone with that domain name. This shared key acts like a directory: it tells our authoritative DNS servers which internal zone IDs they may use when answering a query.
Why can one name have several zone IDs? During customer onboarding, offboarding, and other
lifecycle
changes, multiple Cloudflare zone objects can briefly exist for the same public domain. We use a priority policy to choose a small, bounded set to publish.
Consider four internal zones for
example.com
: Zone 101, Zone 202, Zone 303, and Zone 404. The metadata key currently lists Zones 101, 202, and 303. Zone 404 exists, but it is not selected. The numbers identify separate Cloudflare zone objects; customers still see one domain name.
A TXT record change at
www.example.com
in Zone 202 affects only Zone 202's record keys, so the metadata key does not change. Nameserver and wildcard changes are different because the metadata key records information about them. Those builds must update both the zone's records and the metadata key.
At execution time, the Zone Builder reads the current list again. If it writes the metadata key, it rebuilds the whole key from that fresh view rather than patching an older copy. It writes the triggering zone's records before the metadata that refers to them.
Planning from the event, verifying before writing
An event describes what happened in the past. A build may wait while more events arrive and the database changes, so planning happens twice. The event first creates a conservative work list. Immediately before execution, the Zone Builder reads a fresh publication snapshot: the current zone settings, whether this Zone Builder should publish the zone to its network, and which same-name zone IDs belong in the metadata key. Here, a network is a separate, isolated copy of Quicksilver for one environment. Each Zone Builder deployment writes to one network, and a zone can be enabled on some networks but not others. For example, a zone can be served from Cloudflare's global network and also from the
China Network
.
Fresh state can change the result in four ways:
A targeted build can become a full build if the zone now has a feature that requires complete-zone work.
A build can become a purge if the zone should no longer be published to this network.
A build can be skipped if the triggering zone no longer belongs in the metadata key.
A metadata build can reconstruct the shared key from the current candidate set.
The fresh read comes from read-only copies of the database, called replicas, so builds do not compete with customer writes. Kafka delivery and database replication move independently: an event can arrive before its database change is visible on a replica. The DNS Kafka Aggregator delays events long enough to cover normal lag, and a final database check rejects the read if that replica is still behind.
Combining changes in one scheduler
Record work and full-zone work previously used different schedulers. Each could combine its own work, but not work from the other scheduler. We replaced them with one unified scheduler. The planner decides what must be built, and the scheduler decides when it can run.
When several events for one zone arrive before a build starts, the scheduler combines their work:
An A record changes at
www.example.com
, requiring that name.
A wildcard changes at
*.example.com
, requiring that name and shared metadata.
The scheduler combines both names and the metadata update into one build.
Diagram showing an A record change and a wildcard change passing through the build planner as separate required work, then combined by the unified scheduler into one build.
Merging only adds work. If any event requires the complete zone, the combined build becomes a full build. A purge overrides every rebuild because its desired result is that the zone is no longer published.
The scheduler tracks two things separately: the DNS work that must be performed and the Kafka messages waiting to be acknowledged. A retry preserves both. Even when a full build or purge no longer needs the individual record names, the messages stay attached until the result is safely written.
Keeping Kafka moving
One slow build can delay unrelated customers even when workers are available. Kafka divides messages into ordered partitions. For each partition, it stores one committed offset: the point before which every message is known to be complete. Moving that offset past unfinished work could lose a customer change after a restart.
We use soft committing, which I previously described in my
Secondary DNS deep dive
. A build can finish out of order and be marked complete in memory, but Kafka's committed position advances only after every earlier message has also finished.
Diagram of Kafka messages 1 to n where a large build on message 2 blocks commits for messages 3 to n, and a pod restart replays messages 2 to n.
In the diagram, message 2 is still running, so messages 3 through n cannot be committed. If the Zone Builder restarts, those in-memory completions disappear and Kafka replays messages 2 through n. We deliberately choose repeated work over losing a change.
Targeted plans reduce how long the oldest message stays unfinished. The scheduler also prevents one busy zone from taking every worker. After a zone builds, its next build waits for a per-zone rebuild interval, which differs by Cloudflare plan. Changes that arrive in the meantime are merged into that next build. The scheduler normally enforces this interval, but lets old queued work run before it blocks a Kafka partition for too long. Zone locks and retry delays still apply, so a failing build cannot run in a tight loop.
Releasing safely
A smaller build is useful only if it produces the same DNS data. We added equivalence tests for nameserver, wildcard, and apex paths. Each test runs a full build and a targeted build from the same database state, pins time-dependent values, and compares every Quicksilver key owned by the targeted plan byte for byte. Planner and scheduler tests cover classification, same-name zones, merging, purges, and escalation.
During rollout, we also ran a full build next to every targeted build and compared the output. It was expensive by design, reading the complete zone every time, so we used it only until the mirror comparison below gave us the same confidence.
Tests cannot reproduce every customer combination, so every release also runs through a mirror Zone Builder. It consumes the production Kafka stream and reads production database state, but writes to a separate Quicksilver environment that serves no customer traffic.
Diagram of the mirror setup: the production Kafka stream feeds both the production and mirror Zone Builders, each writes to its own Quicksilver, and a comparer samples both and reports to the DNS Mirror Dashboard.
A comparer checks a 20% sample of zones for missing keys, extra keys, and different values. The two environments do not always build a zone at the same instant, so a temporary mismatch is not automatically a regression. We look for new types of differences that remain after both pipelines catch up. That gives us confidence that targeted builds produce the same output as production.
We built the DNS Mirror Dashboard to group those differences and keep examples for investigation. Naturally, it runs on the
Cloudflare Developer Platform
: a
Worker
receives results,
Durable Objects
maintain state, and
Cloudflare Access
protects it. The critical groups in the screenshot below were each checked and confirmed safe before release. The diff rate is the share of compared zones where production and mirror disagreed on at least one key at the moment of comparison. Most of those are timing differences that disappear once both pipelines catch up, so we watch two things: that the rate stays low, and that no new kind of difference appears.
Screenshot of the DNS Mirror Dashboard comparing production and mirror builds, showing a 0.31% diff rate across 234,313 matched zones.
After the mirror results remained healthy, we rolled the scheduler out by a hash of the zone name. Hashing the name rather than the zone ID kept every zone sharing a name on the same scheduler, so two schedulers never wrote the same metadata key. We increased the percentage for free zones, then repeated the process for paid zones.
Results
Over 24 hours, we measured 86.81% of builds as resources only, 2.54% as resources and metadata, 2.30% as apex and metadata, and 1.30% as purges. Together, 92.95% of builds avoided rebuilding the complete zone. Full builds were the remaining 7.04%.
The two metadata build types were full builds before, even for large zones. Together, they account for 4.84% of builds. Combined with the 7.04% that are still full builds, that is 11.88%, so full builds fell by about 40%. Their share of the count also understates the work avoided, because they include the largest zones. The full builds that remain average 0.35 seconds, because the events that still trigger them, such as a zone being published for the first time, mostly affect new zones with few records.
Table of build types over 24 hours: resources only 86.81%, full zone 7.04%, resources and metadata 2.54%, apex and metadata 2.30%, purge 1.30%.
In 2022, we also planned a batching system. The unified scheduler makes it unnecessary: it does not wait to form batches, and it merges events only when work is already queued for the same zone. Before, long full builds held back the committed offset for their partition. With fewer of them, we now consume Kafka at the rate events arrive, instead of falling behind and catching up in bursts of up to 1,550 messages per second.
Chart of Kafka production and consumption before the change: consumption swings between about 300 and 1,550 messages per second while production stays between about 600 and 800.
Chart of Kafka production and consumption after the change: consumption closely follows production at about 600 to 800 messages per second.
We also measured build completion delay: the time from a DNS record or zone setting change until the Zone Builder finished writing to Quicksilver. It does not include Quicksilver's distribution across our network, which varies by location and is unaffected by this work. Production ran the old schedulers, and the mirror ran the unified scheduler on the same events over the same 24 hours. The two environments are identical apart from the code: they consume the same Kafka events, read from the same database, and write equivalent data to Quicksilver. That makes this a controlled, side-by-side comparison. The figures combine every build type and customer plan, so they are not a latency guarantee for an individual zone. A large share of this time is the rebuild interval described earlier, not build work. At each measurement point, p99 is the time within which 99% of events finished. Typical delays barely changed here, because most builds were already small. The difference is in the tail: the largest p99 spike fell from 14.6 minutes to 1.21 minutes, a 12-fold reduction. The chart uses a logarithmic scale.
Chart of p99 build completion delay over 24 hours on a logarithmic scale: the legacy scheduler spikes up to 14.6 minutes, while the unified scheduler peaks at 1.21 minutes.
Targeted builds also load complete large zones far less often. Aggregate working-set memory, the memory actively used by the deployment, fell from an average of 49 GiB to 7 GiB, an 86% reduction. Peaks fell from 79 GiB to 12 GiB, an 85% reduction.
Chart of total Zone Builder memory dropping from between 50 and 79 GiB to about 7 GiB after the rollout.
Those were the immediate results of the new scheduler. They also let us revisit two safeguards. The first is the rebuild interval, which stops one busy zone from rebuilding constantly and taking capacity from everyone else. With builds now cheaper, and far fewer of them reading complete zones, we could safely shorten that interval on most plans. The second is how long the DNS Kafka Aggregator holds each event, to make sure database replicas have caught up before we read from them. Once our databases began rejecting reads from replicas that had fallen behind, that buffer could safely be smaller. Together, these halved the median build completion delay, from about 7.7 seconds to about 3.3 seconds. In the chart below, the first step down is the DNS Kafka Aggregator change. The red band marks the Zone Builder release with shorter rebuild intervals. That release rolled out gradually across our servers, so the delay falls in two further steps rather than all at once.
Chart of build completion delay over almost two hours, measured with coarse histogram buckets. The median falls from about 7.7 seconds to about 6.4 seconds after the DNS Kafka Aggregator change, then in two steps to about 3.3 seconds after a Zone Builder release marked by a red band. p90 falls from about 10 seconds to about 7.4 seconds, and p99 stays near 27 seconds.
The p90 and p99 lines in that chart are less precise than they look. The histogram had buckets at 1, 5, 10, and 30 seconds, and a percentile that falls between two edges is estimated as if delays were spread evenly across that range. Most delays sit near the bottom of their range, so those estimates come out high: p99 reads close to 30 seconds, and p90 above 7 seconds. The exact average, which does not depend on buckets, confirms the halving. We have since switched to a
native histogram
, which chooses fine-grained buckets automatically. It puts p90 at about 5.5 seconds and p99 at about 12 seconds.
Chart of build completion delay over about four hours, measured with a native histogram: median about 3.4 seconds, p90 about 5.5 seconds, and p99 mostly between 10 and 14 seconds.
Why full builds still exist
In 2022, we said we planned to eliminate full builds entirely. We now know that is neither realistic nor useful. A zone being published for the first time needs every record written. A change that can affect records anywhere in the zone needs every record checked. An event that does not say what changed still gets a full build, because the planner never turns uncertainty into less work. For a small zone, a full build is also cheap.
As we learn what more events change, we will keep moving them to targeted builds. However, the biggest gain has already landed: the events that used to rebuild our largest zones no longer do, and the full builds that remain are mostly triggered by events, such as first publication, that affect new, small zones.
Want to know what happens after the Zone Builder writes a change? Read how
Quicksilver distributes configuration
to every Cloudflare data center in seconds.
GPT-Rosalind-6 uses the same weights as GPT-6 Astra. We did not add
training data or perform additional post-training for this release.
Details of the underlying model’s data and training are provided in the
GPT-6 Astra
system card
.
1
3.2 Genomics and quantitative biology
GeneBench Pro v13 tests an agent’s ability to plan and carry out
quantitative analyses of biological data. We estimated single-attempt
success from seven responses per model on each of 129 problems at each
reasoning budget. At the highest reasoning budget tested, single-attempt
success rates were 37.7% for GPT-Rosalind-6 and 16.7% for
GPT-Rosalind-5.5. The evaluation used a benchmark-specific configuration
that differs from production.
4.1.1 Biological and Chemical
As we did for GPT-6 Astra, we are treating this launch as High
capability in the Biological and Chemical domain. Below, we describe the
evaluations we used to reach this conclusion. We use the four High
capability evaluations and four Critical capability evaluations
described in the GPT-6 Astra system card. The hard-negative protein
binding and DNA sequence design evaluations reported for
GPT-Rosalind-5.5 have been retired.
The High capability evaluations test wet lab knowledge and
troubleshooting. The Critical capability evaluations test biological
design capabilities that could help an expert develop a highly dangerous
novel threat. These evaluations provide evidence about specific
capabilities rather than a complete test of the ability to cause
biological harm. The Critical capability thresholds are preliminary
estimates informed by expert judgment and may change as the evaluations
are refined.
The results below report observed performance and, for the High
capability evaluations, refusal-adjusted scores. pass@1 measures success
on a single attempt. The adjustment treats refusals as correct answers
to estimate capability that safety training may otherwise conceal. For
Biorisk knowledge, we recompute the consensus answer after this
adjustment. GPT-6 Astra reference values use the evaluation runs
reported in its system card.
4.2.2 Model behavior and system safeguards
These evaluations were deliberately created to be difficult. They
were built around cases in which our existing models were not yet giving
ideal responses, and this is reflected in the scores below. We continue
monitoring these categories after launch to evaluate online performance
and further adjust safeguards as appropriate.
4.2.2.1 Biological and Chemical safeguards
GPT-Rosalind-6 is designed to be more useful for legitimate advanced
biological research than generally available ChatGPT models. This
supports advanced life sciences workflows, including some dual-use
topics, that GPT-6 Astra may refuse in General Access. We still tune the
model to refuse or safely respond to requests that would continue to
meaningfully enable severe biological weaponization. This training is
done separately from the training of our classifiers and offline
mitigations to decorrelate our safeguards.
To evaluate the quality of these model-level refusals, we track the
safety of model responses to prompts that originate from held-out
synthetic data, red-teaming, and production data. These benchmarks and
their grading continue to be updated, meaning results do not directly
translate to past system cards.
Table
10
reports
results across our suite of biosecurity benchmarks, accounting for both
model behavior and safeguards. We note that because GPT-Rosalind-6 is a
trusted-access deployment, we mainly considered the results from the
Biosecurity - Severe
benchmark.
Biosecurity - Benign
evaluates model compliance (being fully helpful without refusals or
caveats) on a set of benign biological and operational questions. The
remaining benchmark scores show the fraction of model responses that
pass the configured safety grader on prompts related to biological
weaponization. A response that does not pass was flagged under this
grading criterion; this does not necessarily mean it is unequivocally
unsafe.
4.2.2.3 Safety and alignment evaluations
We also evaluate GPT-Rosalind-6’s instruction following, honesty,
resistance to prompt injection, and behavior when using tools.
Additional evaluations assess whether automated monitors detect harmful
actions and whether the model can evade detection or alter its reasoning
when instructed to do so.
In the tables below, ↑ indicates higher scores are better and ↓
indicates lower scores are better. GPT-6 Astra results are included for
context. Evaluation versions, inference settings, grading and sample
coverage differ across some runs, so these are descriptive comparisons.
“Not available” indicates that no numerical reference result has been
identified. ≈ denotes an approximate value read from a published
figure.
4.2.4 Security and Access Controls
Approved GPT-Rosalind-6 customers must maintain security controls
appropriate for access to a higher-capability life sciences model. These
include strong identity and access management, administrator controls,
least-privilege access controls appropriate to the approved deployment,
and processes to remove access when a user no longer has a legitimate
need.
Model access and data-retention arrangements are approved separately.
Customers must meet the security and oversight requirements for their
approved deployment.
As with other OpenAI deployments, we expect to continue learning from
real-world use. We will monitor safety signals, customer experience, and
emerging misuse patterns, and may update GPT-Rosalind-6 safeguards as
the deployment evolves.
We've detected unusual activity from your computer network
To continue, please click the box below to let us know you're not a robot.
Why did this happen?
Please make sure your browser supports JavaScript and cookies and that you are not
blocking them from loading.
For more information you can review our
Terms of Service
and
Cookie Policy
.
Need Help?
For inquiries related to this message please
contact
our support team
and provide the reference ID below.
Anthropic reported updated financials as part of an amended Form S-1 filing with the Securities and Exchange Commission.
The company said it generated $33.24 billion in revenue for the nine months ended Sept. 30, up roughly 1,200% from the same period last year.
Anthropic unveiled its highly anticipated prospectus last week, but the initial filing only included the company's financials through the first half of the year.
Anthropic
reported updated financials as part of an
amended Form S-1 filing
with the Securities and Exchange Commission on Monday, giving investors a more comprehensive look into the state of its business.
The
artificial intelligence
company said it generated $33.24 billion in revenue for the nine months ended Sept. 30, up roughly 1,200% from the same period last year. It reported a $104 billion net loss during the period, though most of that was due to the change in fair value of convertible notes and derivative liabilities. Anthropic lost $25.1 billion over the same stretch in 2025.
Anthropic unveiled its highly anticipated prospectus last week, bringing the company one step closer to a potentially historic IPO. The initial filing only included the company's financials through the first half of the year.
During its third quarter, Anthropic generated roughly $17 billion in revenue. The company reported a net loss of $23.29 billion for the period.
The five-year-old company is best known for developing a family of AI models called Claude, and it primarily sells its products and services to enterprise customers. Anthropic is a capital intensive business, largely because of the complex infrastructure and hardware required to train and run large models.
In the amended S-1, Anthropic said it has entered into purchase commitments and lease agreements with an aggregate committed spend of about $895.4 billion, "substantially all of which is related to compute and infrastructure."
Anthropic said its compute and infrastructure expenses increased by $14.4 billion for the nine months ended Sept. 30, up 329% compared to the same period last year.
"As our compute and infrastructure capacity grows, the associated compute and infrastructure expenses have increased," Anthropic said in the filing.
The company's CEO Dario Amodei co-founded Anthropic alongside a handful of other researchers who defected from OpenAI, now its chief rival. OpenAI confidentially filed its prospectus with the SEC in June, but its executives have signaled that it's aiming for a 2027 debut.
Anthropic has worked to position itself as more safety-conscious than its competitors, and its S-1 includes more than 90 pages of risk factors. The company highlights risks including potential political headwinds, its complex infrastructure needs as well as the difficulties that can come from balancing safety and commercial objectives.
This is breaking news. Please refresh for updates.
SuperVote: 15 year retrospective on a social endorsement app
In 2012, I published an iPhone app called
SuperVote
, which allowed Facebook
users to share political endorsements with their friends via a shared ballot. Other than two posts
on Hacker News (one in
September 2018
and
another in
November 2018
), I never had a
proper retrospective on my work. Because it’s been 15 years since I first started working on
the app, I felt it was a good time to revisit the development process and reflect on what
I’ve learned since.
Back in the early 2010s, California voters would receive a voter guide provided by the state
(
June 2010 voter guide
),
which would include a list of propositions, with arguments in favor and arguments against a proposition,
along with rebuttals for both sides. A dedicated voter would need to spend a considerable amount
of time investigating all the options. At the time, I had a small network of friends and coworkers
that were well-informed and kept up to date with the latest on state politics. I felt that putting
together a single app that showed a custom ballot for yourself, along with the endorsements of your
friends, it would make voting easier.
Origins
In 2003, I built this mockup page with the general idea of building a political site targeted
towards informing voters:
The iPhone did not exist in 2003, so this idea was a purely desktop web-based site. Beyond this
screenshot, no site was ever built, but conceptually it was a forum site with different boards.
I found a file called
supervote.txt
with the following description, which apparently drove the mockup
above:
Non-partisan, online political community where voters and political
organizations come together to discuss current ballot initiatives and
elections.
Participants contribute their opinions on a variety of
discussion forums.
Voter services include:
() Online voter registration
() Polling place locator
() Email alerts to new propositions in their district/state
() Access to discussion boards
() Participate in straw polls
Organization services include:
() Posting official positions on initiatives/propositions
() Posting announcements of activities and events
()
Organizations pay a monthly/yearly membership to join.
Individuals participants can join for free.
I owned the
supervote.com
domain from 2002 until 2008, with the intention of building out the
site, but for some unknown reason, I let my
ownership lapse. In 2012, I contacted the owner of the domain, asking if they were willing to sell it,
but I never heard back.
Building SuperVote
For version control, I started off using Subversion, but eventually switched
to using Git. GitHub did not have free private repositories at the time,
so I set up a single Git remote server, hosting my repositories on a Linode instance.
Jira was a popular project tracking system for developers in the early 2010s.
Bugzilla was considered clunky and I had previous experience with Redmine and Trac,
but I was most familiar with Jira from my day job. I purchased a license when it was
possible to self-host on your own machine. I still have Z shell aliases to this day:
alias start-jira='sudo /usr/local/jira/jira-5.2.10-standalone/bin/startup.sh'
alias stop-jira='sudo /usr/local/jira/jira-5.2.10-standalone/bin/shutdown.sh'
At this point in my career, I had about 10 years of experience with Java, so I picked
Jetty to serve my API. I had experience with Ruby on Rails on a personal site, but they did not introduce a built-in API-only application mode until version 5.0 in 2016. I chose to use MySQL because of my
previous experience and ease of use.
I wanted to keep certain operations out of the request-response cycle (like building a user’s custom ballot), so I evaluated several message queues before settling on RabbitMQ. It offered a straightforward integration with the Java backend and was relatively easy to operate.
For this post, I made my source code available as a read-only archive on
GitHub
.
Data Model
One of the more challenging aspects of SuperVote was designing a database that could represent the structure of U.S. elections and generate personalized ballots based on a user’s location. I designed a relational data model around governments, electoral districts, elections, ballots, and endorsements. The complete
database schema
is available on GitHub.
The foundation was a hierarchy of governments, including countries, states, counties, cities, and other municipalities, with support for exceptions such as independent cities and municipalities spanning multiple counties (e.g. New York City, whose five boroughs correspond to five counties). Electoral districts were modeled separately to represent congressional, state legislative, and local voting boundaries, which could change over time. During registration, users provided their address and ZIP code so the application could identify their associated governments and electoral districts, determining which elections and contests were relevant to them.
Elections were modeled as dated events associated with governments, containing contests for elected positions and ballot measures. Positions were separate from candidates, allowing the same office to be contested by different individuals across elections, while ballot measures represented questions requiring a yes-or-no vote. Each user’s ballot was associated with an election, with individual selections stored separately for candidates and measures. Users could optionally include a written explanation for their endorsements and control whether those endorsements were visible to others.
The social component connected users through their Facebook friendships, allowing them to view each other’s publicly shared endorsements.
Before this project, I had not written a single line of Objective-C. I downloaded the language guide,
along with the object-oriented programming guide, printed them out and read them over time:
Reviewing my source code, I relied on several third-party libraries to simplify common development tasks. These included AFNetworking and ASIHTTPRequest for network requests, SBJson for JSON parsing, and EGOTableViewPullRefresh for implementing pull-to-refresh interactions. I also used utilities for displaying progress indicators, logging, and crash reporting. Many of these libraries have since been deprecated or superseded by functionality built directly into iOS.
I was aware of PhoneGap (now
Apache Cordova
) as
an option to build for both Android and iOS, but I mainly focused on iOS since I had an iPhone
and did not have access to an Android device for testing.
For this post, I made my source code available as a read-only archive on
GitHub
.
These screenshots show the typical flow of a new user:
Registration screen requiring the user’s voting address.
A simple privacy tutorial describing that endorsements can be made private.
A list of past and upcoming elections, based on your legislative district.
A ballot view, showing all available options.
A candidate view with the total number of endorsements by your friends by candidate.
An endorsement view allowing you to optionally explain your endorsement.
Facebook Integration
Facebook integration occurred on the client-side. It was as simple as cloning the
facebook-ios-sdk
Git repository alongside my project. the Facebook SDK allowed me to retrieve a user’s
riend list with their permission. If two Facebook friends
both used my app and created non-private endorsements, then they would be able to
see them.
Facebook also allowed registered apps to create custom activities on the timeline.
When a user created an endorsement on their iPhone, it would automatically publish
an activity on their timeline. This would allow friends who hadn’t downloaded the
app to see SuperVote activity of their friend, which would then encourage them to
download the app.
Testing & Deployment
TestFlight
was the easiest way to test
early builds of the SuperVote iOS app with friends via over-the-air.
Linode was an easy way to set up a Linux-based server back in 2011-2012. After building
a WAR (Web Application Resource) file of the API server, it was just a matter of copying it
over to my Linode server:
alias push='scp -P 44 ${HOME}/workspace/supervote-parent/server/target/server-0.9.0.war canaryfalls.com:.'
The Jetty server, along with an instance of MySQL and RabbitMQ all ran on the same Linode instance.
Website & Marketing
I may have looked at Squarespace as an option to build a website, but I wanted full control on
the website’s layout and did not want to pay a monthly fee. I ended up finding a
WYSIWYG website builder called
Freeway 5 Express
that ran as a desktop application. I can no longer run this application on my Apple Silicon MacBook Pro.
I became aware of stock photography and the different available licenses: royalty-free versus
rights-managed. With Getty, I was able to find the exact images I was looking for and purchased
several royalty-free images that still appear on the
website
.
Otherwise, I used Flickr images that allowed commerical use at the time with proper attribution.
For blogging, Tumblr had the easiest user interface for creating posts. I give Tumblr credit
for the blog looking the same for the last 15 years:
blog.supervote.org
I created a
Twitter account
, Facebook account and Reddit account.
On Reddit, I attempted, but mostly failed, to promote the iOS app and blog posts
using an
account
associated with the LLC I created.
I spent about $100 on advertising on Reddit. At most, I acquired a handful of users. I’m not sure
if I targeted specific subreddits or if I targeted the entire site.
What Happened to SuperVote?
After the 2012 election, I updated the app sporadically, since there were no major elections
except for some municipal primaries in 2013.
In
2015
, I decided to stop updating the iOS app and removed it from the App Store. In the end, the app had 400 registered users. The only
feedback I still remember to this day was that there wasn’t a full ballot for all users.
What I Would Do Differently Today
Now in 2026, I would not integrate with Facebook for several reasons: 1) I don’t use Facebook
anymore 2) My Facebook friend’s list represents a snapshot of friends from 15 years ago, but
not today 3) I no longer believe family and friends are a good source for political opinions.
If I were building SuperVote today, I would place less emphasis on personal friendships and more on giving users access to informed perspectives from organizations and other trusted sources. Social features could still play a role, whether through private discussion groups or publicly shareable endorsements
(with keeping an eye out for
paid endorsements
), but I would no longer assume that a person’s existing social network was necessarily the best source of political guidance.
School has given straight-a grades to the student, who can't do masic math
For 13 years, Adriana Aviles, a fierce education advocate and former NYPD cop, has been the ultimate insider in New York City schools. As a powerful leader on the Citywide Council on High Schools, she championed Queens District 26 as one of the best in the city. She knew the principals and the teachers — and trusted them. She was gratified that her son consistently received good grades, especially in math, throughout his K-8 years. Then, two weeks ago, she sat down with her 14-year-old son to help him review for his second high school math exam. What happened next left her stunned and her son in tears — and prompted her to publicly confront Schools Chancellor Kamar Samuels at a town hall last week. She shared her family’s devastating experience with
The Post’s David Spector.
All through middle school my son was, at least on paper, a numbers whiz. His report cards consistently showed A’s — his lowest grade was a B plus. But when he slid down to a 78 on his first exam at a top high school in Queens this fall, I pulled up a chair to help him review for his next test.
Adriana Aviles’ son got straight A’s in math, but when he got to high school she realized that it was all a lie.
Helayne Seidman for NY Post
I’ll never forget seeing tears well up in my son’s eyes as he was only able to answer the first couple of questions on the worksheet.
He couldn’t solve basic algebra. He couldn’t do geometry. He couldn’t compute elementary-school fractions.
Yet at PS 221Q, the North Hills Elementary School, he regularly received the highest grade in math. At MS 67Q Louis Pasteur Middle School in Little Neck, his lowest math grade on an end-of-term report card was an 89 in seventh grade. He got a 92 in sixth grade and a 93 in eighth grade — the equivalent of A’s.
When he was admitted to his first choice high school, one New York’s top-rated schools, we had every reason to believe he would continue to excel.
What I saw when I sat down with my son to study shocked me to my core.
He stared blankly at the page, completely puzzled by the material. He was completely thrown off by equations that contained fractions.
My shock soon gave way to heartbreak, then anger.
My son is a mature, smart, industrious kid who has his own online business selling apparel. He doesn’t like to show when things are bothering him. As I saw tears fill his eyes, I was heartbroken. I told him that it wasn’t his fault; the system failed him, and we would do whatever it takes to get him caught up.
I didn’t talk to anybody that night about, not even my husband.
When I received a call from his teacher after he took the test, I told her I already knew why she was calling.
Aviles felt betrayed by the NYC school system.
Helayne Seidman for NY Post
She told me he got only two of 10 multi-part questions right, and scored a 10 out of 100.
His high school is now stuck having to provide him the math education he should have received in elementary and middle school.
I’ve resorted to buying my son math review books intended for first- through fourth-graders so I can see how much he actually knows, and in which grade he stopped learning.
My son now misses lunch every Tuesday through Friday to get tutoring, and when baseball season ends he’ll get tutoring on Saturdays as well.
I’m the child of immigrants; my parents came to this country from Colombia and my mother instilled a deep love of education in my sisters and me. She was always on top of us to make sure we were doing our work and drilled the importance of a high school diploma into our heads.
The work ethic she imparted on me gave me the tools I needed to become an NYPD police officer, and I passed on those values when I became a parent.
She confronted Schools Chancellor Kamar Samuels at a Citywide Council on High Schools townhall Monday.
NYCPS CCHS
When I married my husband, also an NYPD officer, in 2005, I insisted we move to District 26 in Douglaston, because it’s one of the best school districts in the entire city.
Had my son’s teachers just told me he was having trouble, we could have gotten him the help he needed early on. He would have done the work. But we were never told that. And the grades never gave us a clue.
I know now that teachers grade according to a rubric — and that DOE fudges the numbers. The NYC education system had betrayed my son to make itself look better.
What made this betrayal worse is I’ve worked with the teachers and administrators at my son’s schools for years.
Aviles said she is standing up for all the families betrayed by the school system.
Helayne Seidman for NY Post
I’m a longtime parent leader, I’ve served as PTA president, CEC president, and I now sit on the Citywide Council on High Schools. I’ve formed close relationships with educators at every stop.
I became active in my children’s education because I knew that the only way to be truly informed was to be involved. I thought I was fully aware of how my son was doing in school.
I was so wrong.
My son wasn’t in advanced classes, he was in general education, he didn’t take the Regents in middle school. I relied on his teachers to tell me the truth about how he was doing.
I failed too. I didn’t scrutinize his work closely or check his performance on
state test scores
because I figured if his report cards were good then everything was fine.
The freshman was consistently given straight A’s in math at Louis Pasteur Middle School.
Google
It was only recently, after I realized how far behind he is, that I looked back and discovered he was deemed “not proficient” in math on a seventh-grade state exam.
How can he not be proficient in math but was receiving As?
Why didn’t his middle school ever give him additional resources?
I feel deceived.
Looking back, there were warning signs. The teachers barely gave homework. He was allowed to revise his answers on tests after they were graded — and get a new grade. Teachers had a surprisingly laid-back attitude about missed assignments.
There are so many working-class parents in the NYC school system who don’t have the time to closely monitor their kids’ schoolwork and rely on their teachers — and the grading rubrics — to be honest. The lack of rigor in city schools and the inflation of grades are dishonest.
The DOE enables failure in minority communities by constantly lowering their standards instead of grading our children accurately, and by failing to collaborate with families to address issues kids may be having. They don’t even police chronic absenteeism. How can our kids get jobs if they can’t read or do math, and not even taught the value of showing up?
I soon found out my family was not alone in feeling cheated.
Her son frequently got As in math on his report cards.
Helayne Seidman for NY Post
Shortly after my son’s October test, I reached out to other parents in his school to ask if their kids were also struggling in algebra. The responses were overwhelming; so many said that they were and asked what tutoring options were available.
I spoke to a mother of an eighth-grade girl who was apoplectic about her daughter’s math curriculum. She said her daughter, who is also in District 26, was only getting one page of homework a day and she’s urgently trying to get her into Catholic school.
That’s when I knew I had to use my position in the public school system to speak out for all of the families who have been failed by the system.
I decided to confront Schools Chancellor
Kamar Samuels
about it a Community Council on High Schools town hall on Monday, Oct. 26.
Aviles said her son now needs tutoring almost every day.
Helayne Seidman for NY Post
The DOE is ending the Regents requirement to graduate and moving to the “Portrait of a Graduate” model, which is supposed to grade students in a “holistic” manner that is even more dependent on subjective teacher evaluation. All of these problems will just get worse as we abandon more objective metrics.
I was already planning on asking Samuels about this at the town hall, but after seeing what happened to my son, I knew I had to share what my family had been through.
“This school system is failing our children. . . . I equate this to the childhood game ‘Kick the Can.’ My my son is the can. Where did he land?” I tearfully said as I sat on the dais in the Art and Design High School Auditorium.
Start your day with all you need to know
Morning Report delivers the latest news, videos, photos and more.
Thanks for signing up!
As I choked back tears, I made clear to the chancellor that the system’s failure would be felt by my family for years to come.
“In high school, where they are expected to help him graduate with the necessary tools to succeed, will they continue to kick the can all the way to college, where I will be expected to pay for a remedial class that does not count toward a degree?” I asked.
Finally, I pressed Samuels on the upcoming Portrait of a Graduate changes. How could we not expect to see what happened to my son happen to thousands of more students if grades are now even more dependent on the subjective judgments of biased teachers and principals?
Samuels seemed taken off guard by my comments. And his response disgusted me. He didn’t even acknowledge my question. He didn’t know what to do with himself. He was stammering and didn’t know what to say. It just sounded like mumbo jumbo. As a parent, and a longtime parent leader, I was entitled to a real response. I would have preferred a slick non-answer that politicians give, but he didn’t even give me that.
I had to walk off stage after that. Usually, after these events I stick around and pose for pictures with the DOE officials in attendance, but I went straight home that night.
I couldn’t pose for pictures with Samuels. If he can’t answer how we’re going to educate our children, who can?
Shutting the Door on the XCPE Botnet (Threat Intelligence)
Linux Weekly News
lwn.net
2026-11-02 09:29:37
The Google Threat Intelligence blog describes,
in great detail, the XCPE botnet, which has just been taken down.
Last week, Google and industry partners dismantled the XCPE
backdoor which enrolled more than 1.3 million set-top boxes into a
botnet. We assess with high confidence that the backdoo...
The Google Threat Intelligence blog
describes,
in great detail
, the XCPE botnet, which has just been taken down.
Last week, Google and industry partners dismantled the XCPE
backdoor which enrolled more than 1.3 million set-top boxes into a
botnet. We assess with high confidence that the backdoor and botnet
were intentionally developed, deployed, and operated for financial
gain by Videostrong Technology Co., an Android Original Equipment
Manufacturer (OEM) and Original Device Manufacturer (ODM).
Videostrong monetized the backdoor by selling access to criminal
affiliates and loading those affiliates' malware modules onto
infected devices. These secondary modules were conducting ad-fraud;
silently enrolling infected devices into residential proxy
networks; and loading additional malware onto XCPE-compromised
devices.
The LWN site is currently under high scraper load, so comment
display has been suppressed for anonymous users.
If you are a
human, you may read the comments by clicking the button below:
Note
: you can avoid this step in the future by logging
into your LWN account.
The vast majority of commentators don’t believe that Linux phones will ever crack the market stranglehold that Google and Apple currently hold over the mobile phone industry. However, many of the commentators argued that we already have a free/open source mobile Linux operating system in the
Android Open Source Project
(AOSP), which is the base on which Google and the phone manufacturers add their proprietary apps and services. They insisted that AOSP runs on a Linux kernel, so an AOSP phone already is a Linux phone. They asked why waste so much effort developing another mobile Linux operating system which won’t be nearly as good as AOSP. Other commentators argued that most people would never adopt Linux phones, because they are locked into one of the app ecosystems controlled by Google and Apple.
Here is a sample of the comments that the article received:
microtonal
wrote
on Hacker News:
No, they [Linux phones] won’t become a viable alternative:
– Worse compatibility with the millions of Android apps already out there.
– Hardware vendors (outside Apple) all target Android.
– The usability of Android/iOS is far better than non-Android Linux phones.
– The security of non-Android Linux phones is in a bad state (e.g. lacking proper app sandboxing). Heck, even the Android emulation layers that some of these phones have, throw the strong Android sandbox model out of the window.
The answer is very obvious and it is AOSP. Even though Google is trying to close AOSP more (e.g. by not pushing source continuously to the AOSP Git repos and keeping QPR1/3 for themselves), it is still an open source system. Play Services can either be run sandboxed or emulated with microG. AOSP allows alternative systems to hit the ground running, you get great app compatibility, a tested security model, great usability, and support from hardware vendors, including vendors that are alternative friendly (used to be Pixel, now Motorola, Fairphone, etc.). If ever needed, AOSP could be forked and it would still be leagues ahead of non-Android Linux phones.
All the downsides that apply to AOSP-based systems (like things getting blocked by Play Integrity, unlockable bootloaders) also apply to non-Android Linux systems.
…
The difference is that there is already a great open source smartphone OS that works
today
, it’s AOSP.
The most effective path would be for the FLOSS ecosystem to pour their resources into making AOSP independent from Google. But instead people will just fork N half- or non-working solutions for reasons.
drnick
responded
to a previous comment on Hacker News:
> Play Services can either be run sandboxed or emulated with microG.
Play Services is absolutely not needed for a usable Android experience. I am running GrapheneOS without any Google crap and you could not pay me to do so.
TFNA then
replied
to drnick:
Many bank apps, and apps to interact with official state services, usually depend on Play Services. Sometimes it is all banks in one’s country, so no, one can’t just “switch to another bank” as US Americans on HN often assume.
FrebTheRat
wrote
on r/PINE64official:
It’s not the phone or the os that’s the issue. It’s the app ecosystem. They own the store. In order to get a viable app store they need enough market share. In order to get market share they need a viable app store. Chicken meet Egg…I used a pinephone for hot minute and it was passable except I couldn’t use Slack, Teams, do banking, access my obsidian notes, etc.
blue_horizon_x
wrote
on r/linux:
No. They will never become a viable alternative to Android because those who want Android they [w]ill always prefer something like GrapheneOS. A smartphone is like a handheld computer. You will never get apps, productivity apps – flights, hotels, banking on a Linux phone. GrapheneOS has sandboxed Google Play to deal with it while handling it while staying open source.
jonf3n
wrote
on r/Purism:
AOSP is based on Linux, but FAR more secure, especially in GrapheneOS with tight hardware support.
Why step backwards into standard Linux that has a HUGE attack surface and inferior sandboxing / low level security hardening?
PriorFortune1
wrote
on r/Purism:
Developing a third platform to compete with Android and iOS requires one thing and one thing only: apps. Phones that don’t support the apps people use will never sell. Microsoft couldn’t do this in the early days of the smartphone market even by throwing millions of dollars at app developers. It’s incredibly unlikely Linux will somehow solve that problem.
sleepingonmoon
wrote
on r/linux:
No unless some company decides to burn billions. Phones are extremely UX sensitive and I don’t see a way GNU/Linux can beat even Android, not to mention things like power management.
Schlonzig
wrote
on r/linux:
There are millions of apps for iOS and Android now. Any new system would struggle because EVERYONE will have at least one app they can’t live without.
VolksPC
wrote
on r/linux:
In comparison the application ecosystem on Linux is very limited. This is primarily because there is no stable API on Linux for application development and companies are not going to port their software to Linux.
Jimbuscus
wrote
on r/linux:
Android is the answer to the problem, with the problem being Google’s AOSP.
Free Software Foundation (GNU), Canonical, F-Droid, KDE, Graphene, Lineage & Murena would do well to all get behind a shared soft fork of AOSP for a new base that isn’t governed by Google.
If Microsoft couldn’t get apps developed for a third mobile OS, it’s even less likely to be viable to get a Linux for mobile app in addition to Android & IOS.
It’s also not necessary or beneficial to throw away 15+ years of community work towards Android, just to get away from corp controlled code, as the AOSP licence permits forking.
Android is the future of mobile Linux, it just needs a single major fork that replaces Google’s proprietary blobs, which can be used as the new base for non-Google certified mobile distros.
An Android-like Open-Source Project would have the general benefit of supporting native Android apps, without having to fix each roadblock or curveball Google throws at FOSS with each AOSP version change.
Android is the Linux kernel mobile OS, AOSP is just the largest distro.
anomaly256
wrote
on r/linux:
Android is Linux Phone. What is it you’re hoping to replace? The UI? Application frameworks? You want to use a different runtime?
…
But it is… and I think people must clarify what exactly they want changed if they want ‘Linux but not Android’.
What the fuck is Op actually asking for when they ask for Linux, but rule out the Linux phone environment with the current largest userbase? Does it mean they want GNU libc and userland? GTK and QT UIs? Fedora branding? Do they want systemd? Gentoo portage and ebuilds? They want to install packages directly from AUR? The ability to hand edit files in /etc? They are literally given a Linux kernel then turn around and say ‘nooo, I said I want LINUX!’ …?!? Linux is the kernel and we already have that in spades, so what the fuck is actually being asked for?
I think it’s very fair to remind people that they already have a fucking Linux phone and ask them for clarification when they say they want a different Linux phone just not the current mainstream Linux phone because the concept of a Linux ANYTHING is so fundamentally nebulous in the first place.
palata
wrote
on Hacker News:
Let’s make a thought experiment: let’s think about designing a new Linux distro from scratch. Obviously starting with Linux-the-kernel, and then thinking about what we need to be a modern smartphone OS.
– We’ll probably want the security model, with the nice sandboxing and I would argue the verified boot.
– We’ll want the battery optimisation.
– We’ll want access to the firmwares of the components running in the phone.
From the very first step “start from Linux-the-kernel”, every single question I ask myself goes towards the Android design and away from the classical Linux distro design.
In other words, the open source Linux distribution that is the closest to a modern mobile system is… Android :-).
al_borland
wrote
on Hacker News:
It’s just a question of funding. Ubuntu made a push for mobile years ago. They put up a phone on Indiegogo, but it didn’t hit its funding goal.
https://en.wikipedia.org/wiki/Ubuntu_Edge
This is what their Unity Desktop was for as well. A desktop environment that had a consistent design language from desktop to mobile.
The remnants of all of this are still kicking around, I think the community still maintains the phone OS, but without the weight of Canonical really pushing it, and a phone shipping with the OS pre-installed, it won’t get as far.
serbuvlad
wrote
on Hacker News:
iOS and Android have an incredible amount of engineering to make them RAM and battery efficient that desktop Linux simply doesn’t have.
And its not a case of simply implementing some feature, since all apps have to written with this stuff in mind: apps can be suspended/killed at any time and should be able to restart gracefully, notifications go through a side channel, etc etc
Do these [Linux] phones have any of this?
Many of these commentators aren’t grasping the advantages to developing mobile Linux as an independent operating system that can’t be controlled by any single company. They aren’t seeing the advantages that mobile Linux offers over AOSP, nor are they accounting for the difficulties of modifying AOSP to respect its users, compared to developing mobile Linux.
Whenever discussion of AOSP arises, people inevitably argue about the meaning of the word “Linux” and whether Android can be considered a Linux distribution. As much as it annoys the pedants, most people refer to “Linux” as the entire operating system found in one of the major desktop distributions such as Debian, Arch, Fedora, openSUSE, NixOS, Gentoo, etc. or their derivatives. See the appendix below for further discussion about the problematic terminology, but this article follows the commonly accepted practice of referring to “Linux” as an operating system and saying “Linux kernel” to refer to just the kernel. It follows the conventional practice of the software and chipmaker industries by designating Android kernels as separate from Linux kernels.
AOSP’s source code is published by Google with a permissive
Apache 2.0 license
that allows anyone to modify and distribute it. After 23 years and billions of dollars of development, Android currently commands
75% of the global market share
among smartphones, so at first blush, it does seem ridiculous to demand the development of mobile Linux, when we already have access to the most popular mobile operating system with AOSP.
More importantly, using AOSP gives us access to the biggest source of apps in the world. Google’s Play Store grew to
3.6 million apps in 2017
, before Google started implementing aggressive policies to cull unmaintained apps from the Play Store to improve the quality and security of its apps. AppBrain estimates that there are currently
2.03 million Android apps
in the Play Store. Business of Apps
estimates
that Google’s Play Store got 104.6 billion downloads in 2025, compared to 36.6 billion downloads from Apple’s App Store. Consumers spent an estimated total of
$49.2 billion
in the Play Store in 2025, compared to
$117.6 billion
in Apple’s App Store.
The problem is that Android was designed as a vehicle to collect the personal information of users to feed that data to Google’s servers for its targeted advertising. Not only was Android designed to monetize people’s personal data, but Google encourages app makers to use this same business model, which Shoshana Zuboff has
dubbed
“surveillance capitalism.”
AOSP is better designed to
protect the system
from malware than most desktop Linux distros. Each app in AOSP is executed by a separate user and in a separate sandbox. The files for each app are encrypted with a separate key, which is stored in a Keystore that is typically located in an ARM TrustZone that runs a separate lightweight operating system with a secure RAM partition. Each app by default can only access its own files. The core operating system, system libraries, and pre-installed system apps are located in a read-only
/system
partition, so it is more difficult for a rogue app to alter the system than in conventional desktop Linux.
AOSP is designed with these onerous security restrictions because so much of the software installed in Android comes from untrustworthy sources. Unlike the typical Linux distro, where the vast majority of the code is free/open source, the code of the vast majority of Android apps can’t be publicly reviewed for malicious intent. In a community-based Linux distro, it typically takes several years to become a maintainer of packages in the system under the
guidance of a mentor
who checks packages before they get uploaded to the system. The public reputations of Linux package maintainers (and their mentors who sponsored them) are staked on keeping malicious code out of the system.
Amount of
m
alware and P
otentially Unwanted Applications (PUAs)
on
different
o
perating
s
ystems
German IT security organization AV-TEST
estimates
that there are 37.61 million different pieces of malware available for Android, which is 7.1 times more than for Linux which has 5.29 million. Even more striking is the minuscule amount of potentially unwanted applications (PUAs) that have been created for Linux compared to other operating systems. Because most applications in Linux are installed from free/open source repositories, there are fewer opportunities to distribute applications that do things that users don’t want such as collecting and monetizing their personal data. According to AV-TEST, only 58,268 PUAs have been created for Linux, which is 522 times fewer than in Android and 3298 times fewer than in Windows. The decision of Google and Microsoft to embrace surveillance capitalism as a business model has encouraged software creators to produce vast amounts of spyware that collects people’s personal data.
AOSP is not just designed to protect against malware, which abounds in the Android ecosystem; it is also designed to protect the system against the users themselves. AOSP makes users jump through all sorts of hoops in order to modify the system, because Google and the phone makers don’t want ordinary users modifying the system and customizing the environment to fit their needs. AOSP was purposely designed to restrict the freedom of ordinary users, so they won’t change what Google and the phone makers have preinstalled on their phones.
AOSP often makes it remarkably difficult for people to make even the simplest alterations. For example, to change the name of an app in AOSP, the app’s
.apk
install file has to be downloaded and decompiled. Then the display name has to be edited in the app’s
AndroidManifest.xml
file (or its linked string resources). The files have to be recompiled into a new APK package, and then the new APK file has to be signed with the user’s cryptography key. Then, the existing app has to be uninstalled, which means that the user will lose any data stored in the app’s files. Then, the new app can be installed in its place with the new name.
In contrast, changing the name of an app in Linux usually just involves editing the app’s launcher or creating a link to the launcher with a different file name. Alternatively, users can create
.desktop
files with different app names in their
~/.local/share/applications
directory to replace the system’s
/usr/share/applications/*.desktop
files. Mobile Linux distros, such as Sailfish OS, Ubuntu Touch, Nura, Mobian, DanctNIX, Droidian, Maemo Leste, Nemo Mobile, etc., are designed to empower the user to control the tech, rather than the tech being used as a means to control the user.
Some of the custom ROMs based on AOSP, such as crDroid and Evolution X, have created custom launchers that allow the user to rename apps in the interface, but the user should be empowered to modify the app’s launcher to modify how the app is executed, like in mobile Linux. If the user wants to add an environment variable to the launcher or execute an app with an option, it should not be necessary for the user to root the device and then execute complicated commands in the terminal, such as:
setprop wrap.com.example.myapp "env MY_VAR=value"
am force-stop com.example.myapp
In other words, if we want AOSP to offer the same freedom and empowerment for users as mobile Linux, then the source code of AOSP will require quite a bit of modification. Unlike in mobile Linux, where there are already people in the community who understand and maintain each component in the system, new volunteers will have to be recruited to create and maintain our modifications of AOSP.
It won’t be an easy task, because each time Google releases a new version of AOSP, these volunteers will have to rip out the anti-user features in the latest version and replace that code with user-empowerment features. Because Google makes substantial changes to Android every year, the user-empowerment code will likely need to be refactored every year to work in each new AOSP release. Google is actively rewriting its Android code in Rust to eliminate memory safety vulnerabilities and speed up its software development. Google
brags
that its goal in switching to Rust is to “move fast and fix things.” In November 2025, Google reported that it had added over 5 million lines of Rust code to Android, and that Rust had overtaken C++ in terms of new code being added to the Android codebase in 2025.
Google has already made it much harder to maintain custom ROMs, because it has moved more and more
functionality from AOSP
into its proprietary libraries and services (starting with Android 4.1 in 2012), and largely
abandoned the development
of many of the free/open source apps in AOSP in favor of its proprietary apps. For example, Google
announced
in June 2023 that AOSP’s Dialer and Messaging apps were deprecated, because Google was no longer supporting them. Starting with AOSP 16 in 2025, Google
stopped including
the device trees and driver binaries for Pixel devices in AOSP, so ROM developers no longer have target devices, where they can easily test out and adapt the latest versions of AOSP on running hardware. Now custom ROM developers have to do painful trial and error to make the device trees for Pixel phones just like all the other phone brands. Google also squashed the kernel’s commit history, which makes it very difficult for ROMs like GrapheneOS to create their own custom kernels, because they can’t easily take new features, bug fixes and security patches from Google’s kernels. In January 2026, Google
announced
that it would stop providing quarterly AOSP releases, and would only release the code twice per year in Q2 and Q4.
If AOSP ever becomes a threat to Google’s revenue or market share, Google is likely to either make it progressively harder for the community to create AOSP derivatives or stop releasing the AOSP code altogether. Unlike the Linux software stack, where we have active communities that already maintain every component, hundreds of new volunteers will have to be found to maintain the massive codebase in an AOSP fork. Unlike the Linux software stack, which is modular and designed to require less labor to maintain because it had to rely on volunteer labor and the limited generosity of corporations, an AOSP fork will be nearly impossible for the community to maintain. AOSP 17 contains
309 million lines
of source code, plus another 113 million lines of comments and documentation.
The problem is that Google will keep releasing new versions of Android, and app developers will expect the community-maintained AOSP fork to be compatible with all the new changes that Google is making in Android. Between AOSP 16 and 17, Google added 60 million more lines of code. With
$132.17 billion
in profits in 2025, Google has a nearly limitless budget to spend on software development, but our community doesn’t.
Analyzing the binaries to document all the changes that Google is making to Android and then reverse engineering them to add to an AOSP fork will be an insurmountable task. The only way that it will be possible for an AOSP fork to keep up with all the new code that Google keeps adding to Android is for a major corporation like Amazon to step in and maintain the code because it wants to keep running its own AOSP fork. We will basically exchange one corporate overlord for another, and our new lord won’t have the threat of anti-trust regulation to keep it in check.
At any rate, it is unlikely that Amazon would invest in maintaining an AOSP fork, because Amazon is moving most of its products away from Fire OS, which is based on AOSP. Its new Alexa Tablets will use Android, so they have access to the Google Play Store, and its Fire TV Sticks are moving from Fire OS to its new Vega OS, which is a lightweight OS based on Linux. Perhaps Chinese phone makers, such as Xiaomi, Oppo, Vivo and Lenovo, will band together to maintain an AOSP fork, since they legally can’t use Google Mobile Services in China, but Google may decide to release a proprietary freeware version of Android that they can use to forestall them developing their own OS or switching to Huawei’s HarmonyOS. However, even if the Chinese phone brands work together to maintain an AOSP fork, they might decide to not release their source code.
In other words, the recommendation of many of the commentators that AOSP should be the basis of future Linux phones will only deepen our dependence on Google, because there is little chance of the community being able to independently maintain a viable fork of AOSP.
In contrast to the labor and expense of maintaining an AOSP fork or even adding user-empowerment features to each new version of AOSP, it does not cost much to develop and maintain a mobile Linux interface that is based on an existing desktop software stack. As I explained in my previous articles about
Linux phones
and the
advantages of Phosh
, Purism’s costs to develop Phosh were low, because Purism designed Phosh to be a thin overlay on top of Wayland + wlroots + GTK + GNOME + Modem Manager, which are well-supported codebases. Purism submitted its code changes upstream, so the code that it needs would be maintained by these parent projects. Wayland, GTK/GNOME and Modem Manager have a lot of corporate support from IBM/Red Hat, Google, SUSE and Canonical. The Sway developers maintain wlroots, which has been adopted as a FreeDesktop project, so it also will likely get more corporate support in the future.
Purism only had to create two new applications (Calls and Chatty), because it was able to reuse the existing GNOME desktop applications. It worked with GNOME to create libadwaita, so that its desktop applications would become adaptive between desktop and mobile screens. It adapted Rootston to create its compositor Phoc. It adapted the eekboard virtual keyboard to create Squeekboard. It adapted Martijn Braam’s Megapixels camera app to create Millipixels.
In the Linux world, it isn’t necessary to recreate everything from scratch like Google and Apple have done with Android and iOS. By working collaboratively with upstream projects, such as Linux, wlroots, GTK/GNOME, geoclue, Matrix and ModemManager, Purism was able to take advantage of the existing software and help strengthen it with additional functionality, while reducing its long-term maintenance costs because outside projects would maintain the code.
Looking at who Purism hired to work on its software, we can make a rough estimate that Purism has paid for roughly 500 person-months of labor to develop Phosh and its associated software. Purism probably paid lower than the industry average for this labor, since many of the programmers were freelance contractors living in Europe, and not in expensive San Francisco and Carlsbad, California, where the company is located. If guesstimating that Purism paid an average of $5k per month per developer, Purism has spent roughly $2.5 million to develop Phosh since early 2018.
Note: Start and stop dates are based on commits to source code and posts at Purism.
Purism’s policy of working collaboratively with upstream projects allowed it to convince GNOME to adopt libhandy, libadwaita, Calls and Chatty. FreeDesktop.org adopted
feedbackd
, which is a D-Bus activated daemon that provides haptic, visual and audio feedback in response to events. In
February 2025
, the non-profit association
Phosh.mobi e.V.
was created in Germany to guide the development of Phosh as an independent entity. Three of the nine founding members of Phosh.mobi were former or current Purism employees/contractors, including Guido Gunther, who is Phosh’s main developer. The upshot is that
98.7% of the code
in the projects that Purism created for its Librem 5 phone is now being hosted by external organizations, and Purism has been able to keep updating its phones with new software, while paying very little for their software maintenance.
In a
survey
that PINE64 took of 3079 PinePhone users in January 2022, 39.3% selected Phosh as their favorite interface, compared to 15.5% for Sxmo/Swmo, 15.0% for Plasma Mobile and 6.7% for Lomiri (formerly Unity8). Even though Phosh only started its development in
April 2018
, more users preferred it over Lomiri and Plasma Mobile, which have been under development since
March 2013
and
July 2015
, respectively.
By taking advantage of the wider FOSS ecosystem at GNOME, X.org, Sway, FreeDesktop, etc., Phosh.mobi is able to offer a complete mobile environment, while being responsible for maintaining very little code on its own. The independent foundation is currently in charge of managing 452,766 lines of code (LoC). Even if calculating that the Phosh developers will continue to do most of the dev work on libadwaita, Calls, Chatty and feedbackd, they only have to manage 670,039 LoC, whereas forking AOSP would mean maintaining 461 times more code.
Lines of code
managed by
Phosh.
mobi
Project
Lines
% of total
gmobile
4,157
0.6%
libcall-ui
5,093
0.8%
libphosh-rs
124,987
18.7%
libpms-rs
50,305
7.5%
meta-phosh
1,558
0.2%
pfs
6,873
1.0%
phoc
39,112
5.8%
phosh
96,354
14.4%
Phosh Session Services
2,775
0.4%
Phosh Vala Plugins
778
0.1%
phosh-mobile-settings
26,724
4.0%
phosh-osk-data
925
0.1%
phosh-rust-plugins
255
0.0%
phosh-tour
4,135
0.6%
squeekboard
26,773
4.0%
Stevia
55,836
8.3%
xdg-desktop-portal-phosh
6,126
0.9%
Phosh Total
452,766
67.6%
Code under external organizations
GNOME Calls
28,964
4.3%
GNOME Chats (chatty)
49,534
7.4%
GNOME libadwaita
128,067
19.1%
FreeDesktop feedbackd
10,708
1.6%
External code total
217,273
32.4%
Total
670,039
100.0%
One area where Phosh diverged from GNOME was the decision to not use
Mutter
, which is GNOME’s unified display server, compositor and window manager for Wayland. Instead, Purism decided to create its light-weight
phoc
compositor on top of Sway’s
wlroots
library, which was probably an easier task than trying to adapt Mutter to fit within the limited resources of a smartphone. Together, wlroots and phoc contain 103,072 LoC compared to the 419,015 LoC in Mutter.
Purism originally worked on phoc with the idea that it would be adopted by GNOME as its next compositor, but GNOME decided to keep developing Mutter and continued developing
GNOME Shell on mobile
based on Mutter, so Purism wasn’t totally successful in getting GNOME to adopt its code. However, GNOME recognized the benefit of having a light-weight compositor and interface available in its ecosystem for devices with fewer resources. GNOME has helped promote Phosh by hosting its
code
and
documentation
at GNOME World to foster its development.
The situation is very similar with Plasma Mobile, which relies heavily on Qt and KDE’s desktop Plasma, so it doesn’t have to maintain much code independently. Blue Systems used to pay a few developers to work on Plasma Mobile, and several of those developers went on to found Techpaladin Software. The mobile interface, however, is mostly developed by volunteer labor, and that is possible because they are able to piggyback off so much of the infrastructure for the Plasma desktop and take advantage of KDE’s Kirigami apps, which are adaptive between both the desktop and mobile interfaces. Over time, Plasma Mobile has gotten rid of stuff like Halium, oFono telephony and the Maalit keyboard, so its code is closer to desktop Plasma and easier to maintain.
Plasma Mobile uses the same KWin Wayland compositor as desktop Plasma. The upside is that the project has less infrastructure to independently maintain than Phosh, but it requires more RAM and its interface is slower than Phosh’s. Plasma Mobile relies on hardware acceleration to speed up its interface, which can be problematic with some free/open source GPU drivers.
Plasma Mobile was developed as a community effort with very little corporate support, so it tends to be more unstable than Phosh, but Plasma Mobile’s interface is much closer to Android’s classic interface with three navigation buttons at the bottom of the screen (before adopting swipe navigation in Android 10). Unlike Phosh, which shares almost all of its central applications with desktop GNOME, Plasma Mobile has developed separate mobile applications, which creates more code duplication. Plasma Mobile uses a separate terminal, web browser, file manager, system configuration, feed reader, calendar and video player from desktop Plasma. However, a number of these separate applications are just the interface, and they share the same underlying code as the desktop version. For example, its Angelfish browser only contains 12,665 LoC, so the cost of maintaining a separate web browser is very light.
Default applications in desktop Plasma vs Plasma Mobile
Type of app
desktop Plasma app
Plasma Mobile app
Terminal
Konsole
QMLKonsole
Web browser
Falkon
Angelfish
Clock
KClock
System configuration
System Settings (systemsettings)
Plasma Mobile Settings (plasma-settings)
Calculator
KCalc
Kalk
File manager
Dolphin
Index
Software center
Discover
Feed Reader
Akregator
Alligator
PDF Viewer
Okular
Okular mobile
Email client
KMail / Merkuro Mail
Raven / Merkuro Mail
Camera
Kamoso
Plasma Camera
Contacts manager
KAddressBook
Plasma Phonebook
Calendar
KOrganizer
Calindori
Music player
Elisa
Video player
Dragon Player
Haruna / PlasmaTube
Lines of code in Plasma Mobile
Project
Lines
% of total
Plasma Mobile Interface
90,854
44.7%
Calindori
10,659
5.2%
Plasma Camera
6,068
3.0%
Plasma Mobile Dialer
11,516
5.7%
Plasma Mobile Phonebook
3,714
1.8%
Plasma Mobile Settings
3,714
1.8%
Plasma Phone Settings
143
0.1%
QMLKonsole
16,134
7.9%
Raven
17,500
8.6%
Spacebar
8,587
4.2%
Plasma Mobile projects
168,889
83.1%
Other projects used by Plasma Mobile:
Angelfish
12,665
6.2%
Kalk
5,000
2.5%
Alligator
5,845
2.9%
Okular mobile
2,998
1.5%
Index
7,912
3.9%
Total lines of code
203,309
100.0%
In total, the Plasma Mobile project only maintains 168,889 lines of code, which is an incredibly small amount of code considering the scope of the project. Even adding the code for some of the other essential apps it relies on, it only contains 203,309 LoC. The mobile interface still needs quite a bit of dev work, but it has the potential to eventually become a good replacement for people accustomed to Android’s interface. Plasma Mobile needs corporate sponsors to pay more developers to work on the code for the interface to reach its potential, but it has managed to make quite a bit of progress even with limited corporate support.
In contrast, Phosh has been a stable and usable mobile interface for the last six years, since Purism started shipping the Librem 5. It has fewer configuration options and is a simpler interface than Plasma Mobile. Its new Stevia virtual keyboard is a big improvement over the default Squeekboard, which lacked word correction, word selection and the ability to long-press keys to select from multiple characters. One of its limitations is the fact that the title bars at the top and the Overview bar at the bottom take up too much screen space. In horizontal mode, the interface becomes nearly unusable when the virtual keyboard is activated, because of the excess space required by the title bars and Overview bar, as can be seen in the image of the Librem 5 (top) in Phosh’s horizontal mode.
Comparing the space saving changes between Phosh 0.34 (in the Librem 5 above) and Phosh 0.57 (in the PinePhone below) running Firefox with the new Stevia keyboard in horizontal mode
Recent versions of Phosh have addressed the space problem to some degree by decreasing the height of application title bars and flattening the Overview bar at the bottom of the screen from a caret ^ to a horizontal bar ―. The button to show/hide the keyboard on the right-hand side of the Overview bar was eliminated and now the keyboard can be hidden with a swipe down. These changes save a bit of space, but the Overview bar could probably be eliminated altogether since users don’t need a physical bar to remind them to swipe up to get the Overview screen. Users coming from Android are already accustomed to swiping up to switch apps and close them. The title bars probably could also be eliminated if side swipe gestures were implemented to return to the previous screen, although the dropdown menu buttons and other elements that are included in the title bars are useful, so maybe the focus should be on how to better incorporate everything into one title bar for apps like web browsers that use multiple bars.
The important point is that Phosh is already a very solid interface and the tweaks that it needs are minor compared to the significant dev work that AOSP will require to make it an operating system that respects and empowers users. Even in the case of Plasma Mobile, the dev work that it requires is far less than AOSP would require. The work to improve the Phosh and Plasma Mobile interfaces is a one-time cost, whereas the code churn in AOSP will make it a constant struggle to keep reapplying changes with each new version that gets released by Google.
History shows that mobile Linux projects that are developed by a single corporation as a large siloed codebase and then abandoned will struggle to be maintained by the community afterwards. After Nokia abandoned Maemo in February 2010 to create MeeGo with Intel, the Maemo Leste community formed to maintain the software. They have ported their code to run on Devuan with recent kernels, but their Hildon interface is still based on GTK 2, which stopped being supported by GNOME in December 2020. Their Qt applications are based on Qt 4, which stopped being supported by the Qt Company in December 2015. Their
plan
in mid-2023 was to upgrade their code to Qt 5 and GTK 3, but they have very few volunteers to carry on the project, so it is unclear if they will be able to make these code changes.
After the Mozilla Foundation
abandoned the development of Firefox OS
in September 2016, no community project formed to carry on the mobile operating system. Much of Firefox OS’s code lives on in proprietary KaiOS, but that code has been privatized and is essentially lost for the community.
In April 2017, Canonical decided that there wasn’t enough industry demand for Ubuntu Touch and threw in the towel for its convergent mobile OS. UBports, which was a community project started by Marius Gripsgård in late 2015 to port Ubuntu Touch to new devices such as the Fairphone 2 and Moto G (2014), took over development of Ubuntu Touch. In 2019-20, they added the option to run their system on a Linux kernel (rather than just an Android kernel with
Halium
) in order to port their OS to the PinePhone, whose
UBports Community Edition
was a major win for the organization. Nonetheless, they struggled for years to find the volunteer labor to upgrade the code, which was based on Ubuntu 16.04. In March 2023, they released OTA-1 Focal, which was the first major update of the operating system in 6 years. It was based on Ubuntu 20.04 LTS (Focal Fossa), which was released nearly 3 years earlier. They have improved their ability to upgrade the system, with version 24.04-1.0 arriving in September 2025, which was 17 months after Canonical released Ubuntu 24.04 LTS. On June 27th, they
reported
that they are already working on the upgrade to Ubuntu 26.04 LTS, so their upgrades should become more timely in the future.
Nonetheless, much of the Ubuntu Touch code that was abandoned by Canonical in 2017 has had only minor changes over the last 9 years under UBports’ stewardship. The major challenge for the community-based organization is migrating their code from Qt 5 code to Qt 6, since support for Qt 5.15 officially ended on May 26, 2025. On July 24, 2026, UBports released Ubuntu Touch 24.04-2.0, which had experimental support for the Qt 6 libraries. If the upgrade to Qt 6 only involved the Lomiri interface, it would not be such a big deal, since the interface only contains 101,420 lines of code. Canonical, however, dumped a huge amount of code on the community in Ubuntu Touch, and it will likely take UBports years to migrate all that code to Qt 6.
Given how Maemo Leste and UBports have struggled to maintain the large codebases developed by Nokia and Canonical, respectively, it is not realistic to expect that the community can fork an enormous project like AOSP with 309 million lines of code. Even if the AOSP fork can be rebased every year on each new AOSP release by Google, it will still be a huge amount of work to keep refactoring the code each year so the AOSP fork respects and empowers the user like mobile Linux does. If Google ever stops releasing AOSP, there is virtually no way that the community will be able to replicate all the code changes that Google is making with each new version of Android, so the community fork will diverge over time from Google’s version. As more and more Android apps become incompatible with the community’s AOSP fork, fewer people will be willing to use it.
People often advocate for using AOSP rather than typical desktop Linux due to Android’s higher security standards in the userspace to protect against rogue apps. Mobile Linux distros do need to improve their userspace security standards, especially if they intend to compete with Android and iOS. Linux users are less likely to install rogue apps, because they can’t just find any random Linux software package on the internet and install it, like they can with random APK files found on the internet. They have to find the right package format for their distro, and even if they get that right, the package dependency checks will often prevent the installation. However, Linux phones will need to be able to install Android apps if they are going to appeal to the mass market. Even when those Android apps are run inside an LXC container, mobile Linux still needs to be prepared for all the malware and spyware that comes with running Android apps.
However, adding good userspace security to mobile Linux won’t be nearly as much work as adding user empowerment features to AOSP. It will be difficult to convince long-time Linux users to accept application sandboxing with separate users and separate file encryption for each app, and it will probably require some experimentation to get the right balance between security and freedom. Nonetheless, it is wrongheaded to treat the dev work for better security in mobile Linux as an insurmountable task.
The principal reason why people advocate for the adoption of AOSP is the network effect of being part of the biggest app collection on the planet. Everyone has a couple of apps that they can’t live without. Many banks and governments now offer their services through Android and iOS apps.
It is important to recognize, however, that many Linux phones either come with AOSP preinstalled or have it installed by their users in order to run Android apps. Almost every mobile Linux distro provides instructions and packages to install
Waydroid
and
microG
. Jolla’s
license
for Sailfish OS includes its proprietary
AppSupport
. FuriLabs’ FLX1/s/s+ includes
Andromeda
, which is its fork of Waydroid. All of these solutions run AOSP inside an LXC container. In addition, Valve just open sourced
Lepton
, which runs Android games as a non-root user in an AOSP container, but it doesn’t include Google Play Services or microG.
These are not perfect solutions, because they legally can’t include Google Play Services, but any phone running AOSP faces the same problems. Many of Google’s proprietary Play Services have been replicated by the free/open source microG, which uses signature spoofing to make Android apps believe that Google Play Services are installed. Nonetheless, microG doesn’t always work correctly in some situations, such as Google Maps’ geofencing, in-app billing,
AR
, SafetyNet/Play Integrity attestation, reCAPTCHA, PackageInfo.signingInfo, etc. Some apps fail security checks in Waydroid, because it appears to be a rooted Android device.
In addition, Waydroid has done a poor job of upgrading AOSP, which raises security concerns. For the last couple of years Waydroid has been stuck with AOSP 13, whose support ended on March 2, 2026. The most recent version 1.6.3, released in May, just started to include initial support for AOSP 16. Presumably Jolla’s AppSupport has been receiving AOSP upgrades, but Waydroid needs to be better supported in the future if Linux phones are going to rely on it to run Android apps. Instead of wasting resources on trying to fork AOSP, we should instead be focusing on how to better support Waydroid and microG for people who need to run Android apps.
We will still be dependent upon Google when we run Android apps inside AOSP on an LXC container, however, using mobile Linux will push people to gradually switch to alternative apps that run natively in Linux. Over time, it will build up the ecosystem of freer mobile services and software that are compatible with Linux. In contrast, using an AOSP fork will only prolong the dependence on the Android ecosystem. Even if the AOSP fork promotes freer services, such as using
UnifiedPush
instead of Google’s proprietary Firebase Cloud Messaging (FCM), it will be more convenient for people using the AOSP fork to stick with the proprietary services in the apps that they are already accustomed to using. In contrast, Android apps running inside an LXC container will never have perfect integration with the rest of the system inside Linux, and people will have more opportunity to compare how much faster the native Linux apps run and how well they can replace those Android apps.
People often argue that using AOSP will provide access to more software, but that is not necessarily true if the goal is to only use free/open source software (FOSS). Google deliberately obscures the licensing of an app’s code from users by labeling both freeware and FOSS as “free” in its Play Store. There may be more total FOSS among those two million apps in the Play Store than in the Linux repositories, but selecting out the FOSS apps, vetting their code and organizing those apps into a separate repository will not be an easy task.
The largest repository of exclusively free/open source software for Android is F-Droid, which
reports
that the number of apps in its main repo grew by 15.6% in 2025 to 4061 apps by the end of the year. In contrast, Debian 13 (Trixie) contains
69,830
packages in its repositories. It is hard to compare the number of Android apps to Linux packages, since since Linux applications typically depend on many packages, and many command-line tools don’t get packaged as apk files for Android, but Debian 13 probably has 3 to 5 times as many applications as F-Droid. However, the F-Droid repository could be much larger if the F-Droid community was actively searching for free/open source Android code, rather than waiting for Android developers to submit their code. As it is, F-Droid is shorthanded and needs more volunteers to help vet the code that it receives.
The other thing to keep in mind is that there is a lot of needless code duplication in the Android world, because there are so many apps that do the same thing. In contrast, the open nature of Linux software encourages volunteers to join existing projects and the tendency to share common libraries among many projects leads to much less code duplication. People in the Linux world who want to try out something different don’t have to create everything from scratch, since they can just add their new idea on top of an existing codebase. They can fork from that codebase, but they often decide to upstream their improvements to the parent project and rebase their modifications on new releases of the parent project, so there is less code to maintain even though there are many variants. The distros help eliminate the duplication of code by insisting that code that they accept use the standard libraries already in their repositories, whereas Android apps often maintain their own copies of those libraries, which are often out-of-date and not receiving updates.
Another factor to consider is the labor to maintain Android apps compared to Linux apps. Because Android has so much code churn, it requires much more work to keep applications compatible with Android standards. For example, Qt 5 was supported for 12.5 years and GTK 2 was supported for 18.7 years, and Linux standards and system libraries, such as D-Bus, glibc, libdrm, libudev, libcontainer, libasound, SDL2, libX11/libwayland-client, generally try to maintain backward-compatibility, so software in the Linux world doesn’t need to be refactored very often. In contrast, Google is constantly changing the standards for its apps and forcing app creators to constantly update their code to keep their apps in the Play Store. Two different apps that I was involved in creating were thrown out of the Play Store, so I have seen firsthand how much extra work Android imposes on developers.
None of this is to suggest that AOSP is necessarily bad or that people currently using AOSP phones should abandon them. AOSP derivatives like LineageOS, crDroid, Evolution X, /e/OS and GrapheneOS provide an invaluable service to people seeking privacy and greater software freedom, plus the ability to extend the lifespan of their phones. The number of Linux phones in the world is minuscule and there are a limited number of old Android models which have good support in Sailfish OS, Ubuntu Touch, Nura (formerly postmarketOS) and other mobile Linux distros, so they can be turned into Linux phones. For most people, AOSP devices are still the right choice, and in many parts of the world, the only choice, since they often can’t buy Linux phones or the used Android models that are best supported by the mobile Linux distros. Android phone brands where mobile Linux can be installed, such as Sony, OnePlus, Pixel, Fairphone, Volla and SHIFT, aren’t sold in most of the developing world.
AOSP phones are an invaluable means to fight back in the ongoing war for our personal data and the monetization of our digital lives. However, if our goal is the empowerment and greater freedom of people, mobile Linux appears to be a more feasible route to achieve those goals in the long term than AOSP. Designing future phones around AOSP will lock us into the Android ecosystem, whereas developing mobile Linux to become a viable alternative to the Android/iOS duopoly will help build an alternative ecosystem, which is what we need for long-term independence and freedom.
Appendix: Arguments over the meaning of “Linux”
Whenever discussion of AOSP arises, people inevitably argue about whether “Linux” means just the kernel or the entire operating system. While technically Linux just refers to the kernel, there is no commonly accepted way to refer to the operating system based on that kernel. In common parlance, the vast majority of people who use a desktop distribution such as Debian, Arch, Fedora, openSUSE, NixOS, Gentoo and all their derivatives refer to the entire operating system as “Linux” and when they want to talk about just the kernel, they say “Linux kernel.”
Even the Linux Foundation often says “Linux kernel” to distinguish the kernel from the operating system, then refers to “Linux” as an operating system. For example, in its recent report “
35 Years of Linux
,” the Foundation states on p. 12: “Linux’s influence goes beyond its use as an operating system. As a major catalyst for modern collaboration practices, it changed software engineering.” In other parts of the same report, however, the Foundation refers to “Linux” as just the kernel, such as on p. 25, when stating: “Beyond laptops, the use of smartphones also contribute to Linux’s market share in personal and work computing, as Android makes up 72% of the mobile OS market share globally, with upwards of 4 billion Android devices in use.”
This confusion about how to refer to the operating system based on a Linux kernel is caused, in part, by the ideological divisions within the community. Some people who are more aligned with the ideals of the Free Software Foundation refer to the operating system as “
GNU/Linux
,” which is arguably the best way to distinguish the OS from the kernel, but a sizable contingent of the community rejects Richard Stallman’s terminology for ideological reasons. For people in the “open source” camp, simply saying “Linux” to refer to the operating system became a way to resist what they considered an attempt by free software partisans to claim credit for the entire operating system.
An even larger portion of the community say “Linux,” because it is shorter and easier to say than “GNU/Linux.” Maybe if Stallman had stuck with his
original suggestion
of “Lignux,” which is much easier to say than “gnu-slash-linux,” Stallman might have won this logomachic dispute, but he listened to people who told him that “Lignux” was too cumbersome, so the FSF turned to an even more cumbersome label for the OS, and it never gained widespread acceptance.
At any rate, GNU/Linux is a problematic label for distros that contain proprietary software. The amount of GNU software in distros has decreased since the GNOME Foundation
separated itself
from the GNU Project in September 2019. Some distros
like Ubuntu
are even getting rid of the GNU coreutils and replacing them with the Rust-based
uutils/coreutils
, which have an MIT license.
As much as the pedants dislike the convention of calling the operating system “Linux” and having to say “Linux kernel” to refer to the kernel, that has become the predominant practice, and they are not proposing a better term to clear up the confusion. Many words that become part of the commonly accepted lexicon make no sense based on their derivation. It makes no sense to say the word “fast” to mean “moves quickly,” when it originally meant “firmly fixed” (a meaning which we still employ in phrases such as “to hold fast”), but insisting on only using the “correct” meaning of a word only creates semantic chaos when that usage contradicts the commonly accepted understanding of the word.
Another area of debate is whether the kernel of the AOSP/Android operating system should be regarded as “Linux.” Commentators who advocated for the use of AOSP frequently employed the argument that Android’s kernel is Linux, so people who want a Linux phone should embrace AOSP. The Linux Foundation is proud to claim the Android kernel as “Linux,” and Linus Torvalds
calls Android
“a Linux fork.”
Google significantly modified the kernel, adding features such as wakelocks to prevent the kernel from sleeping while performing tasks, its Binder inter-process communications and Binderfs file system to let apps and system services talk to each other across separate memory spaces, and its own PMEM memory allocator, which was replaced by ION in Android 4.0. Google has gotten many of its kernel changes incorporated into mainline Linux, including wakelocks in 2012, Binder in 2015 and Binderfs in 2018. After the Linux kernel rejected Google’s ION memory allocator, Google developed the DMA-Buf Heaps allocator which was added to Linux in 2019. While mainline Linux now supports wakelocks and Binder, most Linux software does not use these features, and there are still features like the ashmem (Android shared memory) allocator that the Linux kernel does not support.
Despite the fact that Google has been the third-largest contributor to the Linux kernel in recent years (after Intel and Red Hat), and has gotten so many Android features incorporated into the Linux kernel, Google is careful to not refer to its kernel as “Linux”, although its documentation freely acknowledges Linux’s contribution to the Android kernel. A
2009 presentation
by Google engineer Patrick Brady stated, “Android is built on the Linux kernel, but Android is not Linux.”
Rather than saying “Linux,” Google makes a point to say “
Android common kernel
” or “AOSP common kernel” (ACK), which it describes in the following terms:
AOSP common kernels
(also known as the Android common kernels or
ACKs
) are downstream of
kernel.org
kernels and include patches of interest to the Android community that haven’t been merged into mainline or Long Term Supported (LTS) kernels. These patches can include:
* Backports and cherry-picks of upstream functionality needed for Android feature
* Features ready for Android devices but still under development upstream
* Vendor/OEM features that are useful for other ecosystem partners
android-mainline
is the primary development branch for Android features. Linux mainline is merged into
android-mainline
whenever Linus Torvalds posts a release or release candidate. Before 2019, Android common kernels were constructed by cloning the recently declared LTS kernel and adding the Android-specific patches. This process changed in 2019 to branch the new Android common kernel from
android-mainline
. This new model avoids the significant effort to forward port and test Android patches by accomplishing the same result incrementally.
android-mainline
undergoes significant continuous testing, this model includes a high-quality kernel from the day it’s published.
Before 2019, Google took each year’s Linux LTS kernel and applied the Android changes on top, but now Google develops its Android kernel and pulls in changes from the latest Linux LTS kernel, which is why Google changed the name of its kernel from “android-4.19-q” in 2019 to “android11-5.4” in 2020. Instead of starting with Linux 4.19 and applying its patches from Android Q, it started with its Android 11 kernel and applied patches pulled from Linux 5.4.
On technical grounds, one can argue that the Android kernel is close enough to mainline Linux that it should be considered a Linux kernel, but on practical grounds they have to be treated as distinct. Chipmakers have to create separate drivers for Android and Linux. Sadly, the makers of mobile systems-on-a-chip (Qualcomm Snapdragon, MediaTek Dimensity, UNISOC Tiger, Samsung Exynos and Xiaomi Xring) have chosen to support Android, but not Linux, which is why AOSP phones generally have much better performance than phones like the PinePhone and Librem 5 that run on a Linux kernel. In Android kernels, the modules are generally compiled into the kernel, so the modules can’t be dynamically loaded and unloaded, which is the common practice in standard Linux.
Of course, from the users’ perspective, AOSP feels like a totally different beast from Linux. Userspace software in Android has to access hardware through its Hardware Abstraction Layer (HAL), whereas userspace software in Linux can access hardware more directly through the
/dev
and
/sys
directories. Normal Android users can’t even see those directories, so Android’s hardware feels like a black box.
[$] Managing the kernel CVE flood
Linux Weekly News
lwn.net
2026-11-02 09:28:39
We recently looked at how the
kernel community is dealing with the firehose of vulnerability reports, but
said little about how users of the kernel are responding. At the 2026 Linux Plumbers Conference, Amazon employees
Dylan Johnson and Justinien Bouron discussed the problem from the point of
vie...
The page you have tried to view (
Managing the kernel CVE flood
) is currently available to LWN
subscribers only.
Reader subscriptions are a necessary way
to fund the continued existence of LWN and the quality of its content.
If you are already an LWN.net subscriber, please log in
with the form below to read this content.
Please consider
subscribing to LWN
. An LWN
subscription provides numerous benefits, including access to restricted
content and the warm feeling of knowing that you are helping to keep LWN
alive.
(Alternatively, this item will become freely
available on November 12, 2026)
Hackers disrupt refrigeration systems at European supermarkets
Bleeping Computer
www.bleepingcomputer.com
2026-11-02 09:28:39
A cyberattack on refrigeration systems has disrupted stores run by multiple supermarket chains across several European countries after hackers remotely changed temperature settings on freezers and cooling systems. [...]...
A cyberattack on refrigeration systems has disrupted stores run by multiple supermarket chains across several European countries after hackers remotely changed temperature settings on freezers and cooling systems.
In some cases, hackers shut down refrigerators or activated defrosting systems, causing temperatures to rise, resulting in significant product losses and forcing store managers to intervene early in the morning.
Refrigeration failures were identified at dozens of supermarkets, but it is unclear whether they were hit in a coordinated attack or targeted individually.
More than 115 stores in Belgium, the Netherlands, Sweden, Luxembourg, Ireland, and Estonia have been affected. Spokespersons for multiple chains confirmed the issues in statements for BleepingComputer.
Supermarket chains that acknowledged refrigeration issues include:
ICA (Sweden) - At least 4 locations affected
Coop (Sweden) - 7 locations confirmed affected
Ahold Delhaize (Belgium and Luxembourg) - Around 40 locations affected
Carrefour (Belgium) - 15 locations affected
Spar Colruyt (Belgium) - 5 locations affected
Intermarché (Belgium) - Around 15 locations affected
SuperValu (Ireland) - 13 stores
Selver (Estonia) - 1 store
In the Netherlands, at least 17 butcher shops and neighborhood stores have been affected.
Additional stores are dealing with the same problems in Belgium, including an Ikea superstore in Mons, grocery stores, and chocoloate shops.
Spokespersons for ICA, Ahold Delhaize, and Spar Colruyt told BleepingComputer that their companies are providing support to affected stores and working to resolve the situation.
“We can confirm that we have been informed of an incident overnight affecting a small number of ICA stores operating outside ICA’s central network,” stated ICA’s press officer.
“The incident impacted the stores’ refrigeration and freezer systems. ICA is currently supporting the affected stores in managing the situation.”
A representative of the Colruyt Group told BleepingComputer that "there is an impact on the refrigeration systems at around five Spar stores."
"These stores are operated by independent entrepreneurs who have their own refrigeration systems. We are currently doing everything we can to resolve the issue."
In a statement for BleepingComputer, Ahold Delhaize stated that refrigeration systems at several dozen stores have been impacted by an IT incident.
“We have identified an IT incident affecting the refrigeration systems in several dozen stores,” the company's communications department said, adding that the company "immediately took the necessary measures to keep our stores operational."
A spokesperson for Coop, one of the largest grocery store chains in Sweden, said that refrigeration units at a small number of stores were impacted, and in once case the system started to generate heat.
"We can confirm that a small number of stores were affected overnight into Monday - specifically 7 out of more than 800 stores," Coop told BleepingComputer.
"The situation is currently contained and under control. The issues have ranged from completely shut-down refrigerators and freezers to partially functioning units, and in one case, the system beginning to generate heat."
In Belgium, only Carrefour stores in two regions were affected, and all hypermarkets remained completely operational.
"The impact at Carrefour is very limited: currently 15 of our 700 stores are affected, both in Flanders and Wallonia," Carrefour stated.
"Not a single hypermarket is among them. All stores remain open because it only concerns a few refrigeration units, mainly freezers. Quick action was taken so that all stores can stay open."
An ICA store manager
told TV4
that temperature settings had been changed overnight to +50°C (122°F) and could not be modified, forcing the store to discard frozen food.
According to RTL media publication, Ahold Delhaize warned managers by SMS around 4 a.m. about refrigeration issues, and some stores used refrigerated trucks to try to preserve stock.
The publication also identified Sabcobel as the vendor of refrigeration systems across several of the impacted locations.
According to Sabcobel's director, Stephan Moelans, the attacker leveraged the same vulnerability used in similar
attacks last month in France
.
Moelans said that technicians started disconnecting affected installations since 1a.m. and attributed the disruption to remotely activated defrost functions.
Although the company tried to protect customer's systems, a simple software update is not enough, as refrigerators also require physical modifications.
"The simplest solution is to manage the refrigerators via a secure internet connection. Most of our clients have adopted this solution since the attack in France,"
Moelans told RTL
.
BleepingComputer has contacted Sabcobel to learn more about the incident and its level of involvement, but we are still waiting for a response.
Last month's refrigerator attacks in France impacted hundreds of stores using
Danfoss AK-SM/AK-SC and Copeland/Dixell XWEB
refrigeration controllers that were compromised by remote attackers who set them to defrost mode.
Version
16.0 of the Tor Browser
has been released. This is the first stable version based on Firefox ESR
153. Notable changes in this release include support for Linux on 64-bit Arm,
redesigned settings, and a "generic window titles" feature to help prevent other
applications from snooping on one's ...
Version
16.0
of the
Tor Browser
has been released. This is the first stable version based on Firefox ESR
153. Notable changes in this release include support for Linux on 64-bit Arm,
redesigned settings, and a "generic window titles" feature to help prevent other
applications from snooping on one's browser activity.
Tor Browser for
Android
has had a number of interface updates to Google's Material
Design 3 design language.
This release also follows Firefox in ending support for 32-bit x86 Linux
systems. See the
full
changelog
for all of the updates and bug fixes.
Security updates for Monday
Linux Weekly News
lwn.net
2026-11-02 09:03:45
Security updates have been issued by AlmaLinux (firefox, java-17-openjdk, kernel, kernel-rt, mariadb:10.5, podman, python-jwt, python27:2.7, python38:3.8 and python38-devel:3.8, python39:3.9 and python39-devel:3.9, ruby:2.5, ruby:2.7, thunderbird, util-linux, and virt:rhel and virt-devel:rhel), Debi...
Radical Centrists Like Haley Stevens Are the Real Extremists in the Democratic Party
Intercept
theintercept.com
2026-11-02 08:57:44
By endorsing Abdul El-Sayed’s opponent for Israel’s sake, Stevens shows how the real threat to progress comes from centrists.
The post Radical Centrists Like Haley Stevens Are the Real Extremists in the Democratic Party appeared first on The Intercept....
Rep. Haley Stevens, D-Mich., speaks following her endorsement of Republican Mike Rogers in Michigan’s Senate race on Nov. 1, 2026, in Detroit.
Photo: Emily Elconin/Bloomberg via Getty Images
Michigan Democratic Senate
primary loser Rep. Haley Stevens on Sunday released a video that will place her in an ignominious legacy of establishment Democrats who act against the collective good.
And, like others on the list, she did it for the sake of one thing only: Israel.
In the video — filmed on her phone, in her car, and at an awkward angle — Stevens announced her endorsement of Republican candidate Mike Rogers against the Democratic nominee Abdul El-Sayed. Stevens had previously called Rogers a “rubber stamp for Donald Trump’s dangerous agenda.”
Stevens’s defection follows a pathetic effort to suggest that El-Sayed must make greater overtures to Israel supporters.
“I don’t belong in Abdul’s campaign,” said Stevens about El-Sayed, the candidate who currently
enjoys
95 percent of Democratic voters’ support in Michigan, according to polls.
Stevens’s defection follows a pathetic, Islamophobic, and chaotic effort over the last weeks to suggest that El-Sayed must make greater overtures to Israel supporters.
Stevens now follows in the footsteps of other conservative Democrats who have shown an extreme disregard for the welfare of people in this country and beyond.
We might recall former
Sen. Krysten Sinema
of
Arizona
, who left the Democratic Party while still in office and
notoriously
delivered a dramatic thumbs down and curtsy when voting against a raised minimum-wage provision during the height of the Covid pandemic. Sinema and fellow then-Democrat Sen. Joe Manchin of West Virginia also
joined
with Republicans to effectively tank major voting rights legislation in 2022.
And, of course, there is the
entire
Senate career of
conservative
pro-Israel
fanatic
, Pennsylvania Sen. John Fetterman, who still remains a
nominal
Democrat. Fetterman offers a consistent master class in dangerous
extremism
and should be seen as a profound political liability by his own party.
It is, after all, so-called moderates like Stevens who are the radicals. As writer Eoin Higgins noted, an array of Democrats have revealed themselves more willing to
see Republicans win
than allow the party’s
left-wing flank
— or, more to the point, criticism of Israel’s genocide — to gain further ground.
And while there is little political agreement between the right and left wings of the Democratic Party on a whole array of issues, defeating Republicans, particularly Donald Trump’s Republican Party, was at least the stated goal of even the most conservative of Democrats.
To so openly fail on even those basic electoral terms should be the very definition of a political liability for a party.
Stevens had perhaps thought she could rely on this pattern.
Numerous
powerful Democrats and Israel apologists like Reps. Nancy Pelosi, D-Calif., and Hakeem Jeffries, D-N.Y., were slow to get behind El-Sayed and have continued to
pick fights
over his policies on Israel.
Stevens’s Rogers endorsement was not, however, met with the tolerance that Democratic rightward lurches have typically enjoyed. In a bellwether moment, another strong supporter of Israel, Senate Minority Leader Chuck Schumer, D-N.Y.,
called
her endorsement of the Republican candidate “a despicable betrayal.”
The condemnation is the correct response of a party leader to a member taking active steps to thwart a crucial Senate race. It may not mark a shift for Schumer away from his own reliable conservatism and support for Israel’s genocide, but it nonetheless reflects a
shifting
political
landscape
when a figure like Schumer sees greater political expediency in standing with a progressive, Israel-critical, Muslim candidate, than with the radical centrist wrecker with whom Schumer is more ideologically aligned.
Stevens’s theatrics are unlikely to make a significant difference to Tuesday’s election results. Voters who elevate support for the Israeli state over human lives in Palestine and in the U.S. were already not going to vote for El-Sayed.
In a close race like this, pulling any support away from the Democratic candidate reveals the extremism of the purported center.
It’s unclear, then, who would be swayed by Stevens’s erratic actions two days before the election. In a close race like this, however, pulling any support away from the Democratic candidate reveals the extremism of the purported center.
We can welcome what looks to be a long overdue shift in political common sense and hope that positions like Stevens’s are seen as a liability before they reach their most extreme iterations.
Yet it shouldn’t take so-called centrists quite literally choosing
Trumpian Republicans
over progressive and left-wing Democrats to make clear who the real threat to the Democratic Party is — if winning elections and delivering policies to improve the lives of constituents is, in fact, their goal.
It certainly is for El-Sayed. He has dismissed Stevens’s performance as a distraction.
“None of this matters,” he
told
reporters on Sunday, and insisted instead on speaking about the state of schools, healthcare, and the cost of food for Michigan residents.
Microsoft fixes Smart App Control broken by October Windows update
Bleeping Computer
www.bleepingcomputer.com
2026-11-02 08:57:29
Microsoft has resolved a known issue that prevented some users from activating the Smart App Control security feature after installing the October 2026 KB5127648 Windows 11 security update. [...]...
Microsoft has resolved a known issue that prevented some users from activating the Smart App Control security feature after installing the October 2026 KB5127648 Windows 11 security update.
Smart App Control uses Microsoft's code integrity features and app intelligence services, and it works alongside SmartScreen and Microsoft Defender to block potentially harmful apps or binaries.
As the company explained when it acknowledged the issue last month, the Smart App Control option is grayed out and cannot be toggled on affected devices running Windows 11 24H2 and later.
"After installing this update, some devices might be unable to turn on Smart App Control in the Windows Security app if it has been turned off. On affected devices, the option to turn on Smart App Control might be grayed out," Microsoft said in the
KB5127648 advisory
. "Devices where Smart App Control is already turned on are not affected and remain protected."
On Wednesday, the company confirmed that the issue has been fixed for users who installed this month's
KB5127936 preview update
.
"This issue was resolved by Windows updates released October 28, 2026 [..] and later. We recommend you install the latest security update for your device as it contains important improvements and issue resolutions, including this one," it noted
in a service alert sent to IT administrators
.
Workaround available
Microsoft also shared a temporary fix for admins who don't want to deploy the preview update on enterprise-managed devices and want to wait for next month's Patch Tuesday cumulative updates.
This requires them to edit the registry on affected Windows 11 devices to restore the ability to turn on Smart App Control by going through the following steps:
Open a Command Prompt window by typing
cmd
in the
Search
box.
Select Run as Administrator.
At the command prompt, enter the following commands and then restart the device:
The Group
Policy settings are under Computer Configuration > Administrative Templates > {Group Policy name} > Windows 11, version 24H2, 25H2, and must be set to
Enabled
after installation
. Admins must also restart affected devices to apply the Group Policy settings.
gohawk is an industrial-grade static analyzer that finds resource
management and concurrency issues in Go code. It has been used to find and fix bugs in
Docker
,
Kubernetes
, and
Caddy
.
lockorder
contradictory lock order: index and cache
Better together
gohawk complements your existing analyzers such as
go vet
, Staticcheck, and
go-critic; it does not try to replace them.
Easy to drop in
gohawk runs as a standalone tool or as a
go vet
tool. No configuration file,
and every check is a command-line flag.
Analysis features
gohawk analyzers are primarily focused on enforcing reliability around resource management,
concurrency, and error handling.
Rich diagnostics
Each finding pinpoints the offending code with a precise source span, shows the evidence
behind it, such as the return that leaks a file, and says how to fix it.
What are you doing this week?
Lobsters
lobste.rs
2026-11-02 08:53:32
What are you doing this week? Feel free to share!
Keep in mind it’s OK to do nothing at all, too....
Hopefully finishing the repair job on
one
HP-41CV. I bought 3 broken, tested all the boards, all working. Looking to sell 2 after, a bit beneath market price, to cover the cost of the one I wanted for programmable RPN fun. These are super cool calculators with custom chips for math from the 80s. I'm thinking I might write an emulator, since the system itself practically has no peripherals (just a screen). Likelihood of it is low but hey I can daydream... I'd rather do some fun programs on it first!
The U.S. Census Bureau and “Super Intelligence”
Simon Willison
simonwillison.net
2026-11-02 08:31:26
I was reminded today of AI Adoption Rate Trending Down for Large Companies, a chart produced by Apollo Global Management in September 2025, which appeared to show a slowdown in U.S. corporate AI adoption based on numbers collected by the U.S. Census Bureau. At the time I used that chart to demonstra...
Every two weeks the U.S. Census Bureau runs a survey called the
Business Trends and Outlook Survey
(aka BTOS). Each time it goes to 200,000 businesses out of 1.2 million total—the surveyed group (a “panel”) cycles such that an individual business only completes the survey every twelve weeks.
The latest data is published on alternating Thursdays at 10am.
I used the most recent numbers (published October 22, 2026) to create a new version of the AI adoption chart, updated to reflect both the last twelve months and a change that was made to the survey question sent out by the Bureau:
For comparison, here’s that AI adoption chart from last year (my GPT-5 version):
In September 2025 the largest (250+ employees) companies answered “yes” around 12% of the time to the question about whether they used AI in “producing goods or services”.
Under the new “any business function” wording, by February 2026, those same large companies answered “yes” around 30% of the time, and by October 2026 that had increased to around 40%.
The data in the above chart comes from the “Employment size class” XLSX file,
available from here
. Here’s
a copy
of the file I used.
As the chart demonstrates, the question asked by the Bureau changed in a material way on November 17th, 2025. Prior to that date, the question they asked was this (emphasis mine):
In the last two weeks, did this business use Artificial Intelligence (AI) in
producing goods or services
? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.)
On December 3rd, 2025,
the Bureau announced
a significant change to that question. Here’s the replacement:
In the last two weeks, did this business use Artificial Intelligence (AI) in
any of its business functions
? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.)
No more “producing goods or services”—now it’s “in any of its business functions”! This means that the answers to the question before and after November 17th are not directly comparable.
From October 19, 2026, they ask about Super Intelligence
When I went looking for the updated question language in the latest XLSX file, I couldn’t find it. I found this instead:
In the last two weeks, did this business
use Super Intelligence or SI
(formerly referred to as Artificial Intelligence or AI) in any of its business functions? (Examples of SI use include: SI-powered data analysis; generative SI for marketing, product design, or coding; virtual SI agents for customer service; SI-powered forecasting or pricing; SI-driven computer vision for quality control, etc.)
The current numbers reflect the old Artificial Intelligence survey wording—the first survey results reflecting the Super Intelligence wording are due for publication on Thursday.
Is this going to affect the survey?
The obvious open question here is whether this wording change is going to affect the survey results.
The businesses answering this survey will have last seen that question twelve weeks ago, when it was still called AI. Hopefully the new language isn’t going to confuse them, especially since the survey helpfully spells out “formerly referred to as Artificial Intelligence or AI”.
I can’t help but laugh at the absurdity of “SI-powered data analysis; generative SI for marketing, product design, or coding; virtual SI agents for customer service” though.
This story starts with a little tragedy. See, I’ve got these old webpages I love to go back and look on every now and then. There’s websites that have been online unchanged for years and years, or decades, and there’s something a little cozy about checking in on them every now and then to see an old friend is still doing okay. Sometimes, of course, tragedy strikes.
There was this fansite called teapuppies.com for an old anime series and and toy line called
Ochaken
. It’s about these cute cartoon dog characters who have tea leaves for ears, each of them named after a different kind of tea.
1
The site had been online since 2004 and had basically not changed since 2005, but it was nice just having it around. Except when I went to visit it a few weeks ago, and discovered that not only was the site gone but the domain is now being used to serve online gambling ads.
I work in web archiving, by the way. Which is to say that I should have known the site would go offline someday,
and
that I have the tools to have done something about it. I could have archived that site while it was still around, but I didn’t. This post is here to beg you to do what I didn’t think to, and to teach you how.
You see, when I talk to a lot of people about web archiving, they always seem to believe it’s something
they
couldn’t do. It’s something a big organization like the Internet Archive does, and if you want a webpage to be saved you just have to wait and hope it gets done. But that’s not true! What I really want everyone to know is that
you
have control over the world around you,
you
can take action to preserve things and make things better. And not just
can
, but
should
! Don’t just wait for someone else to act - get out there and and make it happen! Got a favourite site? Back it up! Got some information you don’t want to get lost? Back it up! Favourite post somewhere? All together now. Back! It! Up!
All of the tools I’ll be showing off in this post use standard web archiving formats like
WARC
2
. You don’t have to worry about any of the technical details
3
, but I’m telling you that because it means there are a lot of really great tools to do whatever you want with them. Here, just as an example, is an entire archived website embedded right in this blog post:
You can even click through all the links, look at all of the images, the works. It’s an entire website right in a post! (And if you don’t trust me to keep this up, and you want the web archive to host somewhere else? You can download it for yourself right from there. Just click the kebab menu and choose “download archive”, and you can do whatever you like with it. In fact, please do!)
By the way, for disclosure: the tools I’m going to mention in this blog post are made by the company I work for, but that’s not because they’ll pay my salary (most of them are free!). I just happen to think these are the best tools there are for personal web archiving.
Option 1 - ArchiveWeb.page browser extension
By far the easiest personal web archiving tool is the
ArchiveWeb.page
browser extension for Chrome. This doesn’t take much in the way of setup or technical skills, and it works for anything: a single page, an entire website, a social media post or profile. So long as you can see it in a web browser, you can save it. All you really need to do is:
Click the ArchiveWeb.page icon in the toolbar, then click “Start Archiving”.
If your page has any interactive elements, click on those to make sure all their data loads.
Visit any other pages you want captured in the same collection and let the extension grab them.
Click the ArchiveWeb.page icon in the toolbar, then click “Stop Archiving”.
You’re done!
You just archived a web page!
Now that it’s all saved, you can preview it and see how it went by clicking the extension’s icon and then the home icon to open the extension dashboard. From there, you can click on your archiving session to preview the pages you archived, load them in browser, and see if they work. (I’ll go into more details on how you can look at web archives and share them with others later in this post.)
Option 2 - archiveweb.page app
If you don’t want to install a browser extension, or you don’t have Chrome installed, you can also grab the
standalone archiveweb.page app
. This is basically the same thing as the browser extension above, it just has its own web browser built in. It also opens straight to the menu that lets you browse your web archives, so it’s a handy way to look at your web archives locally. Pretty much everything else I described above applies.
Option 3 - Browsertrix
The above options are great for single pages or sites that are small enough for you to browse manually. But if you’ve got a webpage that’s big enough that it’s going to be annoying visiting all of the pages by hand, you probably want to look at a
web crawler
tool for web archiving. There are a bunch of options out there, but the easiest to use as an individual is probably
Browsertrix
. A web crawler is basically an automation tool built around a web browser; you provide instructions on what page to start from and what pages it should visit, and it will go ahead and click links to automatically find pages so that it can archive the page as completely as possible. The web archive I embedded above was done using Browsertrix on pretty close to its default settings: I just told it I wanted to do a site crawl for the entire domain
https://teapuppies.mistys-internet.website
, and it worked it out from there:
But, depending, on what you’re looking for, it has options which can be customized to narrow down what content it archives. For example, if you decided you only want pages for the episodes, you could tell it to only capture anything at the
/episodes/
section of the website:
Or even just a single page and anything that page links to:
There are a
lot
more options than I can go into in a quick overview - it’s a comprehensive tool with a lot of options. But if you’re looking to archive a whole website, or a complete social media profile, I’d recommend giving it a try.
Unlike the other tools, which you mainly run on your own computer, Browsertrix is primarily a web app. It’s also the only paid tool in this post - there’s a seven day trial, but after that you do have to pay per use. Since it’s open source, though, technically inclined users can run their own copies on their computer or deploy a copy somewhere if they like.
Viewing your archived webpage
Once you’ve actually backed the site up, you can
replay
it to use it in your browser just like a live webpage. It’s kind of like having your own private version of the wayback machine. You can test it out yourself for your own private use using the interface built into the version of archiveweb.page you used, but if you want to share it with someone else you have a few options. If you want to share it with someone privately, you can send them the WARC/WACZ file and let them load it into the ArchiveWeb.page app or the
ReplayWeb.page
website. If you want to share it with the world, though, you can set up a website with your web archive on it.
This post is using the
ReplayWeb.page
embedded viewer, which is open source and can be embedded inside any other web page you want. You don’t have run any kind of app on your server to do this; as long as you have the ability to upload the archive and a little bit of JavaScript, you can easily add a copy of the replay interface into any web page or blog post.
Preserving the data once you have it
Since everything you backed up is bundled inside a single WACZ file, it’s really easy to keep safe - or share with someone else. Lots of copies keeps stuff safe, after all; if you think you have friends who’d like to hold on to a copy of whatever site you’ve backed it up, pass it along and share the duty of preserving it!
If any of this sounds interesting to you, I’d encourage you to at least give it a try. I’d
love
for the world to be filled with thousands upon thousands of little web archives, with people and community groups archiving and sharing the things that matter to them for themselves. If you’re finding any of this overwhelming, and you’re not sure where to get started,
please reach out to me
- I’m very happy to give you a hand.
"Princes and Thieves" Author Matthew Cole on Jared Kushner & the "Secret Plot to Remake Middle East"
Democracy Now!
www.democracynow.org
2026-11-02 08:18:24
Vanity Fair recently published an excerpt from longtime investigative reporter Mathhew Cole’s new book, Princes and Thieves: Jared Kushner and Mohamed bin Zayed’s Secret Plot to Remake the Middle East.
The excerpt alleged that during President Trump’s first term, Saudi and Emirati ...
Vanity Fair
recently published an
excerpt
from longtime investigative reporter Mathhew Cole’s new book,
Princes and Thieves: Jared Kushner and Mohamed bin Zayed’s Secret Plot to Remake the Middle East
.
The excerpt alleged that during President Trump’s first term, Saudi and Emirati leaders secretly put at least $2 billion into bank accounts and gave access to Kushner. At the time, Jared Kushner was serving as senior adviser to President Trump, his father-in-law. He is now Trump’s special envoy. Kushner’s lawyers responded by demanding that
Vanity Fair
retract the article and threatening legal action against the book publisher, Simon & Schuster.
Jared Kushner denied the report, saying, “The allegations made by this author are completely insane, categorically false, and never happened. We denied every one of these claims on the record, yet the author failed to provide a single shred of evidence to substantiate any of them.” His lawyers say no such bank accounts existed and that the allegations are unsupported by the verifiable evidence.
In an extended interview with
Democracy Now!
, author Matthew Cole stood by his reporting. “This book is a result of more than 50 sources. There was an enormous amount of fact-checking and legal vetting that went into this book,” says Cole. “There’s plenty of information and reporting in the book that lets us establish and understand that there was … an unholy alliance between Jared Kushner and the leader of the
UAE
, the leader of Saudi Arabia and, at the time, Benjamin Netanyahu, the prime minister of Israel.”
On Friday, Congressmember Jamie Raskin, the ranking Democrat on the House Judiciary Committee, called on Kushner to recuse himself from Middle East and Ukraine negotiations and to produce a “comprehensive list of bank accounts or other financial instruments” that he had access to during his time in government.
The original content of this program is licensed under a
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
. Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.
Unions gain access to thousands of Google workers after UK employment rights overhaul
Guardian
www.theguardian.com
2026-11-02 08:00:41
Prospect to hold workplace sessions and recruit members under agreement with tech company Trade unions are expected to make further inroads into big tech companies in the UK after new rights for unions to access workplaces came into force, with Google UK signing an agreement on Monday. The Prospect ...
Trade unions are expected to make further inroads into big tech companies in the UK after new rights for unions to access workplaces
came into force
, with Google UK signing an agreement on Monday.
The Prospect union has reached an agreement with Google to begin organising among about 7,000 workers across its UK operations, including YouTube but excluding
DeepMind
.
Google DeepMind is expected to strike a separate agreement, with the Communication Workers Union seeking recognition there. The deal follows a similar agreement between Prospect and Meta, which owns Facebook, WhatsApp and Instagram and employs about 5,000 people in the UK.
The union, which represents more than 160,000 UK workers across several sectors, will now be able to run sessions inside Google’s workplaces, including the flagship headquarters in King’s Cross, London.
Prospect’s general secretary, Mike Clancy, said: “Prospect has been organising tech workers for many years, and this access agreement is a powerful signal of intent that we can successfully negotiate agreements with even the biggest global tech companies.
“Tech workers deserve a stronger voice at work, whether that is over bread-and-butter trade union issues like pay and flexibility, AI deployment or workplace culture. I am pleased that Google has struck this agreement with us, and hope it sends a clear signal to other tech companies that having a relationship with trade unions is the new normal in the sector.”
He said workers would have a stronger voice as more employees at Google and
Meta
joined trade unions. “With our track record of delivering wins for members, I am looking forward to seeing what we can achieve together.”
Google said it had signed the access agreement to comply with the new UK workplace legislation, although it maintained that it already had good internal employee relations.
“This step builds on our established work with internal employee representative groups, allowing us to fulfil these new requirements while preserving the respectful, direct dialogue our culture is built on,” a spokesperson said.
Paul Nowak, the TUC general secretary, said last week that he expected a wave of union recognition deals in the private sector over the next year as the “transformative” rules on accessing workplaces came into force.
The Employment Rights Act gives unions the right to enter workplaces to meet workers or communicate with them electronically. From January, employers will be required to inform workers of their right to join a union.
Previously, unions seeking recognition have sometimes had to organise workers from outside their workplaces – as was the case with the GMB’s thwarted effort to gain recognition at Amazon’s warehouse near Coventry two years ago.
Headlines for November 2, 2026
Democracy Now!
www.democracynow.org
2026-11-02 08:00:00
Democratic Congressmember Stevens Endorses MAGA Republican Rogers in Michigan’s Senate Election, Federal Judge Orders Trump Admin to Halt Pressure Campaign Against Election Officials, Federal Judge Rejects DOJ’s Lawsuit Seeking to Block Certification of Alaska’s Senate Election If ...
Democratic Congressmember Stevens Endorses
MAGA
Republican Rogers in Michigan’s Senate Election
Nov 02, 2026
In Michigan, Democratic Congressmember Haley Stevens switched her endorsement from progressive Dr. Abdul El-Sayed to Trump-backed
MAGA
Republican Mike Rogers just two days before an election that could decide control of the U.S. Senate. Stevens narrowly lost to El-Sayed in the Democratic Senate primary back in August. Senate Minority Leader Chuck Schumer called the move “a despicable betrayal,” and several members of Stevens’s staff resigned, including her chief of staff. On Sunday, Stevens appeared with Rogers at a Black church in Detroit. The DNC’s Black Caucus chair Virgie Rollins said in a statement, “We are dismayed that Congresswoman Haley Stevens has revealed herself to be a sellout who used Black Michiganders for a political stunt.” And Drop Site News reports that a spokesperson for the Mike Rogers campaign, Abigail Ronson, is a registered foreign agent for Israel, as well as another campaign staffer, Jesse Stokes. In the hours following Stevens’s endorsement of Rogers, El-Sayed raised $1.2 million. This is Dr. El-Sayed speaking on Sunday.
Dr. Abdul El-Sayed
: “I have a sense that you all are here today because of the decisions of somebody who’s not here right now. Am I right about that? But I’m just going to tell you, this is a state of 10 million people who deserve a lot better from the government. I’m a lot more focused on the 9,999,998 that are not the ones you guys are talking about. OK? They pay too much in gas. They pay too much in groceries. They’re worried about the quality of their kids’ schools. They’re worried about the quality of the air and the water that they drink. That’s what we should be focused on.”
On Sunday, Congressmember Haley Stevens was grilled by CNN’s Anderson Cooper about switching her endorsement just days before the election.
Anderson Cooper
: “Back in August, you posted on social media, quote, 'Mike Rogers will be another rubber stamp for Donald Trump's dangerous agenda. We can’t afford to give
MAGA
another Senate seat in November.’ How do you — why is it OK now? Why can you afford now to give
MAGA
another Senate seat in November?”
Rep. Haley Stevens
: “Well, why is it OK to have the Democratic Party and a top nominee, you know, ejecting, you know, this type of — or, injecting, rather, this type of anger and lack of kindness? That’s all. I’m one Democrat.”
Federal Judge Orders Trump Admin to Halt Pressure Campaign Against Election Officials
Nov 02, 2026
A federal judge has ordered the Trump administration to halt its recent pressure campaign against election officials nationwide. U.S. District Judge Trina Thompson said in her ruling Friday, “Although Defendants claim that their actions are designed to halt unlawful voting by noncitizens, their actions are likely to have the effect of interfering with election administration and disenfranchising lawful voters.” According to the ruling, the Justice and Homeland Security departments cannot demand that election officials in any state segregate ballots, purge voters or halt ballot processing. Judge Thompson also blocked threats of criminal prosecutions against election officials.
Federal Judge Rejects DOJ’s Lawsuit Seeking to Block Certification of Alaska’s Senate Election If
GOP
Loses
Nov 02, 2026
Image Credit: Al Grillo/ZUMA Press Wire
In Alaska, a federal judge has rejected a Trump administration lawsuit that was seeking to block certification of Tuesday’s Senate election if Republican Senator Dan Sullivan loses to Democrat Mary Peltola. The Justice Department claimed a second candidate on the ballot with a nearly identical name, Daniel J. Sullivan Jr., was running to confuse voters, and asked the court to set aside and investigate ballots cast for him. U.S. District Judge Sharon Gleason wrote in her ruling that the federal government “offers no rebuttal to the concerns regarding preservation of the secrecy of voting and provides no explanation of how its request for ballot segregation and a court-ordered investigation by law enforcement officers into certain voters’ intent could be implemented while preserving the secrecy of the ballot.” This follows The New York Times’s recent report that operatives from the Peltola campaign had recruited Daniel J. Sullivan to run to peel votes away from the incumbent Republican senator. Peltola has denied the claims.
Federal Appeals Panel Refuses DOJ’s Request for California to Segregate Ballots Supposedly from Noncitizens
Nov 02, 2026
A federal appeals panel on Friday refused the Trump administration’s request to have California set aside about 20,000 mail ballots supposedly sent to noncitizens. The order said, “If the United States believes different procedures are necessary for vote-by-mail ballots, we believe it should take that suggestion to Congress, not the courts.” This comes as the Trump administration has subpoenaed county election workers in California and charged 28 people with breaking federal election laws. California Attorney General Rob Bonta said, “The administration’s baseless claims of widespread fraud are an attempt to sow doubt in our democratic process and risk disenfranchising eligible voters. We will keep fighting to uphold the rule of law.”
GOP
Senator Marshall Releases New Ad About His Medical Practice after Democratic Attacks
Nov 02, 2026
Republican Senator Roger Marshall of Kansas is out with a new ad against Democratic challenger Adam Hamilton’s attacks over Marshall’s medical practice. In the ad, Marshall says, “To ensure that 40,000 Kansans still had access to care, we did have to keep the lights on. So we occasionally handed accounts over to collection agencies.” A New York Times investigation found that Marshall, an OB-
GYN
in rural Kansas, sued more than 700 patients over unpaid bills. One of those patients, Meischa Zimmerman, filed a class-action suit last week alleging a predatory collection scheme and interest charges above Kansas’s legal cap over a missed $50 payment. Zimmerman was arrested while eight months pregnant.
Democratic Senator Chris Murphy Hospitalized After Stroke
Nov 02, 2026
Democratic Senator Chris Murphy of Connecticut was hospitalized on Saturday after experiencing a stroke, according to his office. Before the stroke, the 53-year-old senator was campaigning for Democrats across the country with recent stops in North Carolina, Florida, Michigan and Minnesota. According to a statement from his office, Murphy was “transferred to Johns Hopkins for further monitoring and is responding well to treatment. The doctors are encouraged by his progress, he is awake and alert, and we are optimistic he will make a full recovery.”
Federal Judge Dismisses Trump’s $10 Billion Lawsuit Against Wall Street Journal over Epstein Reporting
Nov 02, 2026
Image Credit: Davidoff Studios/Getty Images
A federal judge in Miami again dismissed President Trump’s $10 billion lawsuit against The Wall Street Journal and its owner, Rupert Murdoch, over a story reporting on Trump’s ties with the late convicted sex trafficker Jeffrey Epstein. U.S. District Judge Darrin Gayles said Trump was “grasping at straws,” and threw out the lawsuit “with prejudice” — meaning the dismissal is permanent and Trump cannot refile it. Trump’s lawsuit came after the Journal published an investigation last year describing Trump’s contribution to a letter that included a sketch drawing of a naked woman to a birthday album for Epstein’s 50th birthday. The letter was later released publicly by Congress, which subpoenaed the records from Epstein’s estate.
Federal Court Refuses to Dismiss
NAACP
Lawsuit Against Elon Musk’s SpaceX
Nov 02, 2026
A federal court in Mississippi has refused to dismiss a lawsuit by the
NAACP
accusing SpaceX of violating the Clean Air Act by running gas-fired combustion turbines to power its AI data centers. The
NAACP
brought the case in April against xAI, the AI company which is owned and operated by SpaceX. The ruling comes just two weeks after the same court rejected a request from the Justice Department to end the case against Elon Musk’s company.
Thousands Rally in Tel Aviv Demanding Opposition Form a Jewish-Arab Coalition to Oust Netanyahu
Nov 02, 2026
In Israel, thousands rallied in Tel Aviv Saturday, demanding that opposition leader Gadi Eisenkot form a Jewish-Arab coalition to end Prime Minister Benjamin Netanyahu’s rule following last week’s deadlocked election. Palestinian-led parties surged to 19 seats, which is enough to give a possible anti-Netanyahu coalition a parliamentary majority of 67 seats. But opposition leader Gadi Eisenkot has previously ruled out governing with Arab parties. On Thursday, activists protested outside his home. This is Amir Badran, the deputy mayor of the Tel Aviv-Jaffa municipality, who is a Palestinian citizen of Israel.
Amir Badran
: “I am here to support Gadi Eisenkot, to say that you must be brave. You must be brave enough to lead us for another government. You have the — you have the force. You have the seats. You have the support of the Arab List. And this is what you should do. You know, in time of history, there is time that you should take this opportunity and go ahead and just be brave enough to do it, because this is what the people of Israel need, this is what Palestinians need. We need a hope. We need — we need something to build for our society to a living together society.”
Israeli Strikes Kill Eight Palestinians in Gaza
Nov 02, 2026
In Gaza, Israeli strikes killed at least eight Palestinians, including five police officers, on Sunday. Since last October’s so-called ceasefire, at least 1,500 Palestinians have been killed in Israeli attacks. Palestinians held a funeral for the slain police officers. This is Khaled Mansour, who attended the funeral.
Khaled Mansour
: “This entire ceasefire is ink on paper. On the ground, the massacres are ongoing, and the slaughtering is ongoing. Only God knows about everything.”
Tens of Thousands of People Protest Across Germany
Nov 02, 2026
In Germany, about 182,000 people across more than 30 cities participated in protests organized by the campaign group “Gen Z Regelt,” “Gen Z takes charge.” The activists demand increased spending on schools and are against plans to return to mandatory military service. This is a spokesperson for Gen Z Regelt, Lina Friedrich.
Lina Friedrich
: “We are here today to fight for peace and freedom, rather than for war and oppression. What matters most to us is better support for the health and education systems. We are against compulsory military service, and we are in favor of a fairer sharing of the burden.”
320,000 People Participate in France’s Third Weekend of Protests
Nov 02, 2026
In France, about 320,000 people took part in the third weekend of protests over the rising cost of living. The demonstrations drew a mix of high school students, trade unionists and yellow-vest activists. Authorities reported 565 arrests and five security officers injured in Paris. Students are planning more school blockades this week before a union-led strike day on Thursday, November 5. This is the mayor of Saint-Denis, Bally Bagayoko.
Mayor Bally Bagayoko
: “High school and university students, especially those from working-class neighborhoods, have the right to demand true equality. I also wish to express my full support for all trade union organizations and for everyone who is mobilizing against the high cost of living. This is about political choices that have been made. These political choices must, of course, be challenged, first and foremost in the streets, and, I hope, at the ballot box, as well.”
Anti-Government Protests Continue to Roil India, Calling for Resignation of Chief Election Commissioner
Nov 02, 2026
In India, anti-government protests continue, as thousands of people took to the streets of Delhi Sunday demanding the resignation of the chief election commissioner after the purge of more than 130 million names from nationwide voter rolls. This is Indian opposition leader Rahul Gandhi.
bq.
Rahul Gandhi
: “Once we secure the resignation of Gyanesh Kumar, we will come straight for you, Interior Minister Shah and Prime Minister Modi. I assure you, because you have committed voter theft, we will not let you escape. Strict legal action will be taken against Modi, Shah and Kumar, and justice will be delivered to the people.”
U.K. Bars Swedish Climate Scholar and Palestinian Rights Advocate Andreas Malm from Entering the Country
Nov 02, 2026
Image Credit: Mehdi Chebil/Hans Lucas via Reuters Connect
The United Kingdom has barred Swedish climate scholar and Palestinian rights advocate Andreas Malm from entering the country. The travel ban came just days before Malm was scheduled to speak at a conference featuring the co-founder of Palestine Action, Huda Ammori. The British government has designated Palestine Action as a “terrorist” group, arresting and prosecuting dozens of activists. Malm told Al Jazeera he believes he was denied entry to the U.K. due to his participation on the panel. He said, “This is a drift toward authoritarianism that didn’t begin with me, and it’s not going to end with me either.”
Former U.K. Labour Party leader Jeremy Corbyn said on social media, “We know why the government has taken this decision: because it wants to silence criticism of Israel, as well as its own complicity in genocide.”
Malm is a professor at Lund University in Sweden and the author of several books, including “The Destruction of Palestine Is the Destruction of the Earth.”
Houston Grand Jury Indicts
ICE
Agent for Running Over Immigrant Construction Worker
Nov 02, 2026
Image Credit: FIEL Houston
In Texas, a Houston grand jury has indicted an
ICE
agent who was captured on surveillance video running over an immigrant construction worker earlier this month. Mario Alberto Morales is facing state charges of aggravated assault with a deadly weapon after striking Miguel Parra Hernandez with an unmarked
ICE
vehicle. In newly released footage, Hernandez is seen face down on the pavement after being hit with the vehicle, crying out in pain. Morales is then seen standing over Hernandez and saying, “You got hit with the car, bro.” The agent is also heard saying, “I was coming in and I hit him with the car.” Hernandez is currently in
ICE
custody with a broken leg and other injuries.
ICE
quickly claimed Hernandez ran in front of the vehicle. But Harris County District Attorney Sean Teare has accused the agent of deliberately running over Hernandez and said the allegations by
ICE
were “false” and “reckless.” This marks the first time a federal immigration agent is indicted in Texas. Hernandez has lived in Houston since he first migrated to the United States from Mexico as a teenager in the 1990s. Hernandez was headed to work when
ICE
agents ambushed him.
AP: Trump Admin Plans to Deport Afghan Immigrants Via Pakistan
Nov 02, 2026
The Trump administration is reportedly planning to deport Afghan immigrants detained in the U.S. back to Afghanistan via Pakistan. Deportation flights may begin as soon as today, according to the Associated Press. The immigrants would arrive in Pakistan and be sent back to Afghanistan from there. Advocates warn many face persecution, torture, and even death, under Taliban rule for working with U.S. troops during the occupation of Afghanistan.
The original content of this program is licensed under a
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
. Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.
Organizing for the Long Haul Means Remaining Open Minded
OrganizingUp
convergencemag.com
2026-11-02 07:55:13
Featured images by Elizabet Wendt One vital quality for staying involved in movements is openness. This is a willingness to consider that we might sometimes be wrong, an eagerness to learn from our mistakes, an ongoing sense of curiosity about the world, and a receptivity to new developments and ide...
Last night an AI agent performed brain surgery on a mouse.
We gave it 19 gene therapy vectors in a spreadsheet, a mouse model of epilepsy, access to a motorize surgery rig with cameras and one goal: reduce seizures.
It picked the drug. It picked the brain region. It derived the coordinates from three papers. Then it drove the stereotaxic arms into the brain and injected 250 nanolitres while watching the entry site through the rig camera for bleeding.
A licensed vet gated every irreversible step and held a physical kill switch to stop the AI agent.
Thread on what it decided, what it saw, and the moment it refused to let me override it.
[Writing this on the spot out of excitement, but I've been 18hrs in front of my laptop coding and coordinating this, so keeping it brief and descriptive, more cohesive thoughts on this soon...]
Over 1,000 GitLab servers vulnerable to ongoing RCE attacks
Bleeping Computer
www.bleepingcomputer.com
2026-11-02 07:47:00
Over 1,000 Internet-exposed GitLab servers are still unpatched against a maximum-severity flaw actively exploited in remote code execution attacks. [...]...
Over 1,000 Internet-exposed GitLab servers are still unpatched against a maximum-severity flaw actively exploited in remote code execution attacks.
Tracked as
CVE-2026-97229
, this vulnerability stems from an
integer overflow weakness
and was discovered in the CI/CD Regular Expression Compiler, which parses and evaluates regular expressions in pipeline configuration files.
"GitLab has remediated an issue that, under certain conditions, could have allowed an unauthenticated user to execute arbitrary code on the GitLab server via a specially crafted regular expression in a CI/CD configuration, due to insufficient regular expression validation,"
the company explained
on October 12 when it released security patches to address the security issue in GitLab Community Edition (CE) and Enterprise Edition (EE) versions 19.4.2, 19.3.5, and 19.2.8.
"These versions contain important bug and security fixes, and we strongly recommend that all self-managed GitLab installations be upgraded to one of these versions immediately. GitLab.com is already running the patched version. GitLab Dedicated customers do not need to take action," it added.
On Friday, the US Cybersecurity and Infrastructure Security Agency (CISA) confirmed that attackers have begun exploiting the flaw in the wild and
added CVE-2026-97229
to its
Known Exploited Vulnerabilities (KEV) Catalog
, ordering federal agencies to secure their servers by Sunday.
CVE-2026-97229 is the latest in a series of GitLab security issues addressed since the start of the year. In January, GitLab fixed a
high-severity flaw
(CVE-2026-0723) that
lets attackers bypass two-factor authentication, and in September,
a maximum-severity vulnerability
(CVE-2026-85706) in the repository commits API
that can be exploited to steal credentials and other secrets.
"We are currently tracking over 1000 GitLab instances vulnerable to CVE-2026-97229 (on @CISACyber KEV) and over 11300 instances vulnerable to CVE-2026-110264 (out of around 39K GitLab instances we see daily),"
Shadowserver said
on Sunday.
Since November 2021, CISA has added
six GitLab flaws
to its catalog of actively exploited vulnerabilities, one of which has also been used in ransomware attacks.
GitLab says its DevSecOps platform has over 70 million registered users and is used by more than 50% of Fortune 100 companies worldwide.
Dirk Eddelbuettel: myman 0.13.0 on CRAN: Another wave!
PlanetDebian
dirk.eddelbuettel.com
2026-11-02 07:32:00
Another new version of our still-new-ish package myman reached CRAN this morning, and has been
built for r2u and on
r-universe. The
matching Python package
has also been updated. The package offers twenty two hundred eighty four
“My man …” posts by Kevin Kruse
made on bsky during the summer of 2026....
Another new version of our still-new-ish package
myman
reached
CRAN
this morning, and has been
built for
r2u
and on
r-universe
. The
matching
Python package
has also been updated. The package offers twenty two hundred eighty four
“My man …”
posts by
Kevin Kruse
made on
bsky
during the summer of 2026.
Each wave picked at one particular public persona. This package wrapse
these up in the style of packages like
fortunes
or
gaussfacts
.
This release contains a new wave from October 31 to November 1 poking
fun at Mike Johnson—just in time for the election.
>library(myman)>set.seed(20261102) # reproducible sampling>myman(target="Mike Johnson")My man looks like he used parental controls to block "Pimp My Ride".-- about Mike Johnson on 2026-11-01>
So have a chuckle, and then go out and vote! The short NEWS entry for
this release follows.
Changes in version 0.13.0
(2026-11-02)
New wave thirteen (100 posts); total is now 2284 posts
A business jet carrying nine people nosedived into a South Florida neighbourhood on Sunday and erupted into a giant fireball that appeared to just miss homes before crashing near a busy roadway. All passengers on board were killed, U.S. Transportation Secretary Sean Duffy said.
Federal agencies to investigate cause of 'horrific tragedy,' says transportation secretary
Listen to this article
Estimated 4 minutes
The audio version of this article is generated by AI-based technology. Mispronunciations can occur. We are working with our partners to continually review and improve the results.
Emergency crews respond to a plane crash in Palm Beach County, Fla., on Sunday.
(Mark Elias/The Associated Press)
A business jet carrying nine people nosedived into a South Florida neighbourhood on Sunday and erupted into a giant fireball that appeared to just miss homes before crashing near a busy roadway.
All passengers on board were killed, U.S. Transportation Secretary Sean Duffy said.
Video recorded by people inside their cars captured the terrifying moment the Bahamas-bound aircraft plunged from the sky before striking near the Palm Beach County roadway and exploding.
Duffy posted on X on Sunday that the Learjet 60 crashed in Palm Beach County at 12:50 p.m. and called it a "horrific tragedy."
The U.S. Federal Aviation Administration (FAA) is sending a team to support the National Transportation Safety Board's (NTSB) investigation into the cause, he said.
Florida Gov. Ron DeSantis said there were no immediate reports of anyone on the ground injured or killed.
Heavy rain in region
"It was a cataclysmic plane crash," DeSantis told reporters at the scene. "It looked like a missile going down and left a big crater on the side of the road."
Rain pounded the region on Sunday. A flood watch due to excessive rainfall was in effect through Monday afternoon, the National Weather Service said.
The aircraft had departed Austin-Bergstrom International Airport in Texas and was headed to the Bahamas, according to the FAA. Authorities did not immediately say what caused the jet to go down or who was on it.
This screenshot shows a fiery crater in the aftermath of the plane crash in Palm Beach County on Sunday.
(Eduin Cruz via Reuters)
The plane was heading east at 35,000 feet when pilots reported turbulence, icing and precipitation and requested to climb another 2,000 feet.
Moments later, the plane entered a steep dive, first 10,000 feet per minute, then 20,000 feet per minute and up to 30,000 feet per minute, speeding the plane beyond its design limits, according to air traffic control recordings and flight tracking data.
Pilot issues distress call
A pilot declared an emergency and issued distress calls. The plane banked sharply to the right before it smashed into the ground.
Minutes before the crash, the pilot reported ice accumulating on the jet and turbulence that was jolting it by as much as 700 feet, according to air traffic control recordings captured by LiveATC.net.
"We have light, clear ice that is manageable and moderate turbulence," the pilot told controllers.
Florida Gov. Ron DeSantis described the plane crash as "cataclysmic."
(Saul Martinez/Getty Images)
But the situation rapidly deteriorated. After encountering precipitation and requesting to climb, the pilot declared an emergency as the aircraft began losing altitude at an alarming rate.
"We're descending 8,000 — mayday, mayday — we're descending 10,000 feet per minute," the pilot radioed.
"We are in an overspeed condition descending 20,000 feet per minute," the pilot radioed, before delivering a devastating acknowledgment of what was about to happen. "We're going to crash into the ground."
Jim Reynolds, a spokesperson for Austin-Bergstrom International Airport, said it was a private aircraft.
He could not immediately provide details about who was on the flight.
"This is definitely a tragedy," he said.
People gather near the scene of the crash in Greenacres on Sunday.
(Saul Martinez/Getty Images)
Steve Arroyo, an aviation safety expert, said icing and turbulence could have been factors in the crash and may have caused the plane to stall while it was cruising at 34,000 feet.
There are many potential factors and the NTSB will consider all of them, including how the pilot responded to the icing and turbulence reported over the radio. But investigators won't rule anything out at this early stage, he said.
The NTSB said investigators were expected to arrive at the crash site on Sunday and Monday. A media briefing is scheduled for Monday in West Palm Beach.
The crash occurred in Greenacres, Florida, according to Palm Beach County sheriff's officials. The city of 45,000 people is about 11 kilometres southwest of West Palm Beach.
AI Vulnerability Finding
Schneier
www.schneier.com
2026-11-02 07:02:34
A comment:
3% of ‘critical’ software security bugs, in all of recorded history, have been found in the last month. This is on pace to find another ~100,000 bugs this year alone, adding ~26% more vulnerabilities to the all time total Do you realize how fucking insane that is?
Yes. It is a...
3% of ‘critical’ software security bugs, in all of recorded history, have been found in the last month. This is on pace to find another ~100,000 bugs this year alone, adding ~26% more vulnerabilities to the all time total Do you realize how fucking insane that is?
In past years, I've experimented union types in a low level language (Copper) and a higher level language. I got very interesting results but I'm currently experimenting a trimmed down version: just the option types.
What is an Union Type?
An union type is a list of two or more alternative types that a value can hold. For example:
funcf(x:Int|String)...end
the parameter
x
can be either a string or an integer.
Some languages accept to perform directly operations that are commons to all types of the union, for instance if
Int
and
String
types have a
print
method, you could just do
x.print
. But in the basic case, you must first discriminate the actual type of the value before using it:
funcf(x:Int|String)matchxcaseInt->i// do something with i as an integercaseString->s// do something with s as a stringendend
Adding Union Types to Copper
In Copper, my own "better C" language, I've experimented union types. I wanted to handle the general case
T1|T2|...|Tn
using a tag but also handling the special case
Pointer|Nothing
without any additional tag and discrimating using the
NULL
pointer value in order to be fast and integrate easily with external libraries.
Limits
The feature was not very well defined and I've added a lot of restrictions for an efficient compilation.
No Nested Unions
(T1|T2)|T3
is not permitted. It simplifies the implementation of the compiler since a value of an union type is a pair
where the value part can not include tags recursively.
Significant Order
T1|T2
is different from
T2|T1
. This restriction allows a very useful shortcut for the common pattern:
without having to make the
Error
type builtin into the language: it just picks the first type of an union,
Error|File
in the example above.
No Compatibility Between Unions
T1|T2
is not compatible with
T1|T2|T3
.
It would have required to assign a global unique tag to each type or to implement a remapping of tags when promoting an expression to a compatible union.
With such a restriction, a tag can be easily assigned for each part:
0
for the first type,
1
for the second one,
...
Results
The feature was really useful, but it was far too much complexity in the compiler for a minimalistic low level language such as Copper. The most annoying part was dealing with the special optimized case
Pointer|Nothing
.
My main usage was mostly for two cases:
T|Nothing
T|Error
So, may be just handling
T|Nothing
will be good enough, and for errors I'll just go back to returning an error
and
a value.
What is an Option Type?
The option type is just a special case of union type:
T|Nothing
. It represents an expression that may have no value.
An option type
T?
can have any value of
T
plus the special
nil
value.
A
T
value
is a
T?
nil
is a
T?
.
Adding Option Types in Copper
The implementation will be done by reserving a value in the original type. Each type can define its own
nil
value.
For pointers, the choice is obvious, the nil value will just be the address
0
but it is not hard-coded in the language, any other address could be used as well.
For other types, it depends. A 24 bit RGB color in a 32 bit integer can reserve
0x8000_0000
for example. A signed integer index data type can use
-1
since it usually indicates an invalid or not-found value.
This implementation has lot of advantages:
It integrates very well with native C APIs.
It is efficient: no tag to attach to a value.
It is flexible: you just need one value to reserve.
but it has few drawbacks:
Some types may not have a value that can be reserved for
nil
.
It is not possible to have
T??
Refining the Option Type
The main idea of the option type is that you can't refine it to its main type without ensuring first that it is not the
nil
value.
I have found a simple way to do this:
It is a boolean expression that tests whether
expr
of type
T?
is nil. If the expression is not nil, the value is copied in a local immutable variable
id
of type
T
.
The trick is that the variable is not directly available, it is created in a hidden scope that is accessible in 3 special cases:
The inner block of a
if
The inner block of a
while
The right expression of a
and
ifexprasx// x is available hereendwhileexprasx// x is available hereendvarisSingleton=list.headasfirstandfirst.next==niliflist.headasfirstandfirst.next==nil// first is available after 'and'// first is also available hereend
Allowing Early Returns
The
as
operator is sufficient, but it can lead to the arrow anti-pattern:
ifexpr1asv1ifexpr2asv2ifexpr3asv3// do somethingelsereturnerr3endelsereturnerr2endelsereturnerr1end
It can be solved with early returns but it is not possible without a new construct,
or else
, that evaluates a block if a expression is nil:
varv1=expr1orelsereturnerr1endvarv2=expr2orelsereturnerr2endvarv3=expr3orelsereturnerr3end// do something
The block must terminate either with a
return
,
break
or
pass
.
Refining the Option Type without Check
In theory this is beautiful: you can't dereference a pointer without checking its value first or the compiler will report an error. Unfortunately, in real code there are many situations where you know that a value is present but the compiler can't know it. For example:
Iterating over two link-lists where the number of elements are equals.
An object with two distinct phases: an initialization phase and a processing phase. In the first phase some properties are optionals but in the second phase those properties are guaranteed to be defined.
Sure these can be resolved in different ways:
Test before accessing a value even if you know that it is defined.
Rewrite the algorithms, change the data-model, copy the data into new structures where types are non-optional.
But either it is inefficient and can make code confusing (why the value is tested here, shouldn't it be defined at this point?) or it makes the code more complex by adding new classes and code.
It's simpler and easier to just force the type refinment and leave a comment to the reader on why it is safe to assume that the value is defined at a particular point in code.
For this I did not add anything to Copper, the language has already the
cast
operator that can change the type of an expression without any verification. However I've added a convenient
notNil
method in the standard library that perform a check at runtime in debug:
varvalue=maybeValue.notNil// I know what I'm doing
Extending an Option Type
An option type has very few methods by default:
comparing a value with another value,
type conversion
but sometimes you want to write methods on the type even if it can be nil. Here's a common example:
A
defopt
command allows to define methods on the optional type:
defoptdelete// delete applies to T?ifselfasx// 'self' is of type T?// 'x' is of type T// invoke the actual deletex.deleteendend
And now you just do:
Option Type with a Nil-Checker
The problem
The previous solution works very well but introduces multiple concepts that you need to know just to read the source code.
In addition it can make some simple cases becoming very cumbersome. For instance, this common pattern of lazy creation:
It makes the code more verbose and requires two additional local variables. It is not a problem for the LLVM backend that will optimize everything by coalescing variables and eliminates useless branches, but it makes the code more difficult to understand.
It is more compact but now it requires to know what the
ifNil
macro does.
Woudn't it be possible to avoid all this?
The Solution
An alternative is to infer that a local variable can not be nil at some points in the code. Instead of having explicit type refinment, the compiler does the refinment based on tests.
This solution is similar to what TypeScript or Kotlin do.
Here are some examples:
ifv1<>nilf(v1)// okend
ifv1==nil...elsef(v1)// okend
ifv1==nil...returnendf(v1)// ok
Globally it works well, it is very un-intrusive, a reader can understand the code without needing to know anything specific except that
T?
represents an optional
T
.
But the devil is in the details:
It requires a two-pass analysis on loops: a variable may be valid at the beginning of the loop but could be changed later requiring a second analysis:
ifv1<>nilwhilecondf(v1)// ok on the first iterationv1=...// now v1 can be nil againendend
It does not work well with genericity. Despite a test, the variable does not change its type. The reason is because the nil-analysis is performed after building a function (because of the multi-pass). This leads to unexpected errors and requires many explicit conversions:
ifv1<>nil// v1 is still of type T?, so if f is generic, the parameter will be // optional and the function won't assume that v1 is non-nil.f(v1)end
Last but not least, it's a kind of black magic: the validity of the program depends on the power of the nil-checker. For instance, does the compiler handles the following cases?
(1)
whilev1==nil...// no breakendf(v1)
(2)
ifv1==nilorv2==nilreturnendf(v1)f(v2)
(3)
v2=v1ifv1<>nilf(v1)f(v2)end
Answer:
(1) yes
(2) yes
(3) no
But it could change in the future.
And I won't mention the 5% performance impact on the compiler ;-)
Despite all the benefits of this approach, I gave up and went back to the previous solution.
Is it Worth it?
The option type, despite being simpler than union type introduces many new concepts and requires additional knowledge that make the language more complex.
Am I ready to break the minimalistic yet complete enough language that I have managed to design over the year with this feature? does the few bugs it reveals are worth it?
The good parts:
The option type makes the intention clearer without polluting the source code too much with type annotations.
It eliminates the need to search in code or documentation whether a parameter, return or attribute can be nil.
It prevents accidental dereference of nil references.
It fits well with a statically typed language: the type tells the reader and the compiler which operations are permitted on a value.
While I can accept few type annotations that help me to make the code more readable and less error prone, I want the compiler to be a discreet reviewer and a friend that helps me finding bugs, not an inquisitor or a censor that forces me to rewrite my algorithms in a way that makes it happy but makes the code obscure.
Porting the compiler (the language is self-hosted) or Clade (my own text editor) was not very difficult and it showed few potential and actual bugs.
The real challenge was the port of my experimental decompiler: lot of complex algorithms and data-models. Algorithms rely on some properties (e.g. a Control Flow Graph has all nodes reachable from the entry node) to ensure that it will terminate and that there won't be any nil value at the end; that's something that can't be verified by the compiler.
At the beginning, I thought that the benefit of option types would be very low and that documentation in the code and tests were almost as good, but I have to admit that it can be helpful. May be it's time to get rid of the billion-dollar mistake.
Prince Harry, Duke of Sussex
, the younger son of Charles III, referenced his family's adherence to the motto in a 2023 interview to promote his forthcoming memoir,
Spare
.
[
4
]
He told
Anderson Cooper
on
60 Minutes
that "The family motto is 'never complain, never explain', but it's just a motto ... [Buckingham Palace] will feed or have a conversation with a correspondent, and that correspondent will literally be spoon-fed information and write the story, and at the bottom of it, they will say they have reached out to Buckingham Palace for comment. But the whole story is Buckingham Palace commenting."
[
4
]
Supermarket chains in Sweden, Belgium and
Luxembourg
have said the temperatures of their refrigerators and freezers in dozens of stores rose overnight, prompting fears of a cyber-attack similar to one that caused millions of euros of food losses in France.
Sweden’s Coop chain told
the daily Aftonbladet
on Monday that six of its stores had been affected, while ICA said it was “aware that a limited number of stores were affected by an incident overnight which impacted their refrigeration and freezer systems”.
The Dutch retailer Ahold Delhaize, which last month reported a cybersecurity incident to police in the Netherlands, said an “IT incident” had hit fridges and freezers in dozens of its supermarkets in Belgium and Luxembourg.
A company spokesperson, Sarah De Meester, said the supermarket chain had established that temperatures in the cooling systems had increased, adding that the incident was being investigated.
Belgium’s
Le Soir newspaper reported
that dozens of stores and supermarkets were affected, including five Spar outlets, multiple Shop & Go convenience stores attached to Q8 service stations, and a number of smaller Proxy Delhaize minimarkets.
“We started mobilising technicians at 1am,” Stephan Moelans, the chief executive of Sabcobel, which specialises in refrigeration system installations, told the paper. “We spent all night and morning disconnecting the affected installations from the grid.”
Hackers had remotely overridden the settings on thousands of freezers, fridges and chiller cabinets, pushing them from as low as -27C to well over 50C. The Intermarché, Coopérative U, Carrefour, Leclerc and Monoprix chains were among those affected.
Stores had to discard vast quantities of fresh meat, dairy and frozen products, with many suffering losses ranging from tens of thousands of euros to more than €130,000 (£109,000). Two stores in Alsace reported a combined loss of €270,000.
Most supermarket cold aisles are connected to the internet to allow remote maintenance and alerts for system failures. In the French attack, the hackers pirated controllers mostly supplied by the Danish company Danfoss.
They disabled alarm systems before adjusting the thermostats or extending the defrost programme by hours. Danfoss said it had identified security flaws last year and had “long recommended” customers install updates.
The Paris prosecutor’s office has opened a formal investigatiion into the French attack, saying damage included “food losses and fires”, and the prime minister, Sébastien Lecornu, instructed France’s national cybersecurity agency, ANSSI, to investigate.
ANSSI subsequently confirmed
that supermarkets using controllers from two other companies were also targeted in the attack.
Supermarket freezers in Belgium and Sweden hit by suspected cyber-attack
Freezers have had to be emptied at several ICA grocery stores in Sweden after the cyber-attack
Dozens of supermarkets in Belgium and Sweden have been hit by a suspected cyber-attack affecting its refrigerators and freezers.
Around 40 Delhaize supermarkets in Belgium and Luxembourg were affected, while several ICA and Coop shops in Sweden were hit. Thirteen SuperValu shops have also been impacted in Ireland.
Millions of euros worth of food has had to be thrown away, according to local media reports, with one ICA store in Sweden saying temperatures for its freezers had been set to 50 C (122F).
Both Belgium and Sweden's national cyber-security agencies said they were monitoring the incident.
Delhaize, one of Belgium's biggest supermarket chains, told Belgian media a "computer incident" had occurred overnight at 40 shops.
A Delhaize branch in Hoeilaart, in the Flemish Braband province in Flanders, told Het Laatste Nieuws it had lost €100,000 worth of produce.
The manager said that estimate did not even take into account further losses that would result from closing the shop.
Carrefour, a French-owned supermarket, said the incident had only affected 15 of its Belgian stores, adding that measures had been taken immediately to ensure all shops remained open.
Meanwhile, a spokesperson for ICA in Sweden confirmed the incident had affected a "small number of ICA stores operating outside ICA's central network".
"ICA is currently supporting the affected stores in managing the situation," they said.
One of the IVA branches located in Sjöbo, southern Sweden, wrote on social media: "Our refrigerators and freezers have been hacked overnight and they have set the temperature to 50 degrees."
The store was closed earlier but has now reopened to customers.
An employee at a branch in Odengatan in the capital Stockholm was pictured throwing away bags of spoiled food away.
Seven out of the 800 Coop stores in Sweden are also affected but the "situation is currently contained and under control", the company told the BBC.
Synology warns of C3RB3R ransomware attacks targeting NAS devices
Bleeping Computer
www.bleepingcomputer.com
2026-11-02 06:28:42
Taiwanese network-attached storage (NAS) appliance maker Synology warned customers to update their devices immediately due to a surge in C3RB3r ransomware attacks. [...]...
Taiwanese network-attached storage (NAS) appliance maker Synology warned customers to update their devices immediately due to a surge in C3RB3r ransomware attacks.
The company urged users to enable automatic updates and upgrade the DiskStation Manager (DSM) software running on their NAS devices to the latest version to protect their data.
"Synology has identified threat actors exploiting a vulnerability patched in December 2024 to target devices running outdated versions of DSM. These attacks aim to encrypt data and extort ransom," it said in a
Thursday advisory
.
"Synology strongly urges all users running vulnerable versions to update DSM immediately to protect their systems. Synology also recommends enabling automatic updates to keep DSM up to date at all times."
A Synology spokesperson was not immediately available when BleepingComputer reached out for more information on these C3RB3r ransomware attacks.
While Synology has yet to share
which vulnerability is being exploited in these ongoing attacks, it linked to a
November 2024 security advisory
with information about patches released for vulnerabilities allowing attackers to execute arbitrary code (CVE-2024-10441), read files (CVE-2024-50629), and write files (CVE-2024-10445) on unpatched devices.
Synology users are advised to immediately block access to the management web interface from the Internet and update their DSM software to block attackers' attempts to hack and encrypt their devices.
Product
Severity
Fixed Release Availability
DSM 7.2.2
Critical
Upgrade to 7.2.2-72806-1 or above.
DSM 7.2.1
Critical
Upgrade to 7.2.1-69057-6 or above.
DSM 7.2
Critical
Upgrade to 7.2-64570-4 or above.
DSM 7.1
Critical
Upgrade to 7.1.1-42962-7 or above.
DSM 6.2
Moderate
Upgrade to 6.2.4-25556-8 or above.
Synology rarely issues security advisories warning of attacks targeting its customers' devices, with the last such alert
published in August 2019
following large-scale brute-force attacks that led to
eCh0raix ransomware infections
.
Last week's advisory follows a massive wave of reports from Synology customers whose data was encrypted and who were prompted to buy "C3rb3r Decryptor" tool from a dark web portal for thousands of dollars.
"All your important files have been encrypted. Any attempts to restore your files with thrid-party software will be fatal for your files! The only way to decrypt your files safely is to buy the special decryption software 'C3rb3r Decryptor'," the ransom note warns.
"At this page you will receive the complete instructions how to buy the decryption software for restoring all your files. Also at this page you will be able to restore any one file for free to be sure 'C3rb3r Decryptor' will help you."
C3RB3R ransom note (BleepingComputer)
The Cerber ransomware (also known as C3RB3R and CerberImposter), now deployed in attacks targeting Synology NAS devices, borrowed its name from a notorious Ransomware-as-a-Service (RaaS) with the same name
that emerged in 2016
.
7 years after its invention, I discovered the
Gemini protocol
. It’s a kind of “back to first
principals” alternative to the web, aimed at providing value to independent
content creators away from the noise, bots and trackers of the main web.
Imagine the web if HTTP had no cookies, or even any headers or methods apart
from GET. If HTML could not add scripting, styling or fetch further resources
(no
<link>
,
<script>
or even
<img>
,
<audio>
tags. No text decoration
apart from headings and lists existed. Where TLS did not require a trusted 3rd
party Certificate Authority (CA) to approve of every party. This minimalism may
sound very basic and even useless to many applications, but the result is a
community of writers and developers focused on creating and sharing content
without a commercial aim behind it, no influencers, no bots, no e-commerce, no
subscriptions.
In Gemini, a website is called a capsule, the web is called the geminispace. The
standard document format is kind of stripped down markdown format called
gemtext
(Gemini’s
HTML equivalent). To surf the geminispace, you need a Gemini client (equivalent
of a web browser). After trying a few options such as
deedum
, I found the
Lagrange
client, an amazing piece of
software written by Jaakko Keränen @skyjake who works for Nokia. The client
works on all platforms (mobile and desktop) and is a real pleasure to use.
There are no big corporations on Gemini. Lagrange comes with a bookmark to a
“
Getting started
” page which
lists some search engines like keneddy which I understand runs entirely on one
guy’s mac mini including the crawler.
After exploring the geminispace a little, I wanted to have a go at implementing
my own server. It may not be the ideal language to implement a server, but I’m
doing an increasing amount of python development so it seemed like a fun project
to use it on. The rest of this post follows a similar step by step process that
I went through to implement a Gemini server in python trying to use minimal
external dependencies.
Listening on a TCP socket
The first thing to do was to create a TCP socket to listen on the Gemini port
(which is 1965, the year of the Gemini 5 rocket launch). To do this we do
These are quite low level APIs that wrap OS system calls, let’s explain some
some of them.
socket
Socket.AF_INET
instructs our socket to allow connections from IPv4 addresses
Socket.SOCK_STREAM
means listen on the TCP protocol.
We bind the socket to
0.0.0.0
which is a special address to indicate we want
to listen on
all
network interfaces. You could also use
127.0.0.1
, but
this causes problems when running inside a docker container for example.
setsockopt
SOL_SOCKET
indicates that we are setting an option at the sockets api level (it is also
possible to set options at the TCP protocol level for example).
SO_REUSEADDR
option on the tells the operating system that it can bind the socket except if
there is already an active socket listening on that address and port. Without
this I found that if I terminated and restarted my program, I was not able to
immediately bind the socket again.
Testing your TCP socket
If you run the code outlined above, you can test out your server by using a TCP
client. I recommend either using
telnet
or
nc
as it is likely that you
already have them installed. Run the python script and then in another terminal
window one of these.
nc 0.0.0.0 1965
telnet 0.0.0.0 1965
If all is working properly, you should see hello printed.
Exchanging valid Gemini requests and responses
Currently, we’re only returning
hello
from our socket server. In order to
return
a valid Gemini response
,
We need to return something like
20 text/gemini
# Hello
20 is the “success” status code (similar to 200 in HTTP)
text/gemini
is the mime type of the response data more on gemtext later
The line below is the actual response data.
A valid Gemini request, always consists of only an absolute URL which cannot
exceed 1024 bytes of data. We can read 1024 bytes from the connection object.
Let’s return Hello if the path is
/hello
and
/goodbye
if the path is
/goodbye
. We can send a 51 “Not found” for any other case.
fromurllib.parseimportSplitResult,urlsplitdata=connection.recv(1024)url=urlsplit(data.split("\r\n")[0])ifurl.path==b"/hello":connection.send(b"20 text/gemini\r\n# Hello")elifurl.path==b"/goodbye":connection.send(b"20 text/gemini\r\n# Goodbye")else:connection.send(b"51 Not Found\r\n")
At this point if you re run your test with your TCP client, you should be able
to send a URL and observe different responses depending on the path.
Encrypting with TLS
So far, we are sending unencrypted data over our socket (similar to plain HTTP
requests). Gemini mandates the use of TLS, so for a Gemini client to understand
our request we need to encrypt the data with a TLS certificate.
Unlike on the web, which pretty much mandates that server certificates be issued
by a specific set of certificate authorities (CAs), it is quite common in Gemini
to use self signed certificates. Gemini operates a “Trust on first use” (TOFU)
policy for TLS. This works similar to SSH connections where the first time you
connect to a host, your client associates it’s certificate with that host. If
the certificate changes for subsequent requests, you will get a TLS error (or
warning at least) that the server’s identity has changed.
Let’s generate a self signed certificate to identify our server. To do this
we’ll use the
openssl
tool which is also likely already installed on your
system.
This will generate a self signed
certificate.pem
and
key.pem
file which we
can use to identify our server to Gemini clients. Using the ssl module, create a
context and load the certificate and key files generated by the previous script.
The ssl lib allows you to wrap an unencrypted TLS socket so that you can use it
in the same way but encrypted. Change your current code to use the wrapped
tls_connection
.
Success 🎉, you should now have a server that a real Gemini client will
understand. You should be able to point lagrange (or any other Gemini client) at
gemini://localhost/hello and get a page with a
Hello
heading returned.
Accepting user input
What we’ve done so far in enough to serve static pages, however there will come
a time where you’ll want to accept user input. To indicate that input is
required, the server needs to return status 10 which tells the client to prompt
the user for input.
This is a lot more basic than what you’re probably used to with HTML forms and
HTTP post. You get a single prompt which is then sent to the same URI as the
entire query string once the user fills it in. You cannot send multiple query
parameters or provide different input types, the user gets a text box with a
single field, that’s it.
Let’s add another route to echo back what the user input
elifurl.path==b"/echo":query=url.queryifquery==b"":tls_connection.send(b"10 What should I echo ?\r\n")else:tls_connection.send(b"20 text/gemini\r\n"+query)
Identifying the user
We can now accept user input from the user, but how do we know which user it
came from? In HTTP world there are many ways to do this, basic auth, oauth2 are
common but another way is mutual TLS (mTLS). In this scenario, both the server
and client present there own TLS certificates. The server can then identify the
client based on its certificate.
mTLS is quite commonly used in enterprise applications for machine to machine
communication, but not very common for end user applications due to to the fact
that generating, maintaining and instructing the browser to use a client TLS
certificate is quite complicated and cumbersome for the average user.
Certificates expire, the server needs to agree to trust them, etc.
Unlike with HTTP, where TLS was bolted on later with HTTPS, TLS is native to the
Gemini protocol. There is a special set of status codes (6X) around client
certificates. If a server sends status code 60, it means that the client must
provide a client certificate to access this page. As a result, creating and
managing client certificates is encouraged to be supported by Gemini clients
(the browsers). Lagrange handles this very well. When a server responds with
status 60, the user gets prompted to either use a certificate they created
previously, or to create a new certificate with a friendly interface that hides
away the complexities of TLS. Once the certificate is created, the page can
reload, this time using mTLS with the client supplying the newly created
certificate.
Gemini’s approach to TLS
Gemini uses TLS in a more simplified way to HTTPS
which is not without security trade offs and
criticism
,
but does provide much improved security over unencrypted connections. TLS is
used as an
asymmetric
(public/private key) encryption protocol it is common
place in Gemini
To use
self signed
certificates rather than certificates issued by
certificate authorities (CAs) in favour of a trust on first use (TOFU)
approach similar to how SSH uses
known hosts
.
To
ignore
(at least client) certificate expiry dates
These 2 points are where things get difficult with the standard library SSL
python module.
The problem is (as far as I could tell) that the ssl module requires the server
to be configured to
trust the client’s certificate authority upfront
before it serves the request. In enterprise / corporate networks this is
generally OK as there would likely be conversations upfront to configure the
certificate before it was used. This is not practical for a public website (or
capsule). With the python ssl module, a certificate generated by an unknown CA
will be rejected before we can trust it. This is what led me to use the
pyOpenSSL
library.
Refactoring to use client certificates using the pyOpenSSL library
The key difference is that it allows you to configure ssl to accept client
certificates
and
override the certificate verification to supply your own
implementation via the
set_verify
method. By always returning
True
we can move certificate verification and
reading logic up into the Gemini protocol specific code.
context.set_verify(ssl.VERIFY_PEER,lambda*_:True)
In order to resume TLS sessions with client certificates, we also need to set
the session id context otherwise one the first request will work and
subsequent requests will be rejected
.
context.set_session_id(b"my_gemini_server")
Then later in our “protected route” we can get the certificate and use its
information to lookup the user in the existing records or to create a new user.
The fingerprint in the code above essentially acts as a unique identifier for
the certificate. You can create a user and identify them with this value. The
common_name is a more friendly name which clients will prompt the user for on
creation. You can use this like a user name that is displayed but not to
identify them.
Gemini uses status code 60 to indicate to clients that this resource requires a
client certificate. In our protected route, if no
peer
can be found from
get_peer_certificate()
we need to return the 60 status code which tells the
client that it needs to prompt the user to create or use an existing
certificate.
Thankfully migrating to pyOpenSSL isn’t too difficult. Here’s a full example
modified to use pyOpenSSL with client certificates, with a
/hello-user
route
that responds with the certificates name and fingerprint.
importsocketasSocketfromOpenSSLimportSSLassslfromurllib.parseimportSplitResult,urlsplitfromcryptography.x509.oidimportNameOIDfromcryptography.hazmat.primitivesimporthashesport=1965context=ssl.Context(ssl.TLS_SERVER_METHOD)context.use_certificate_file("./certificate.pem")context.use_privatekey_file("./key.pem")context.set_verify(ssl.VERIFY_PEER,lambda*_:True)context.set_session_id(b"my_gemini_server")socket=Socket.socket(Socket.AF_INET,Socket.SOCK_STREAM,0)socket.setsockopt(Socket.SOL_SOCKET,Socket.SO_REUSEADDR,1)socket.bind(("0.0.0.0",port))socket.listen(5)whileTrue:connection,client_address=socket.accept()tls_connection=ssl.Connection(context,connection)tls_connection.set_accept_state()tls_connection.do_handshake()data=tls_connection.recv(1024)url=urlsplit(data.split(b"\r\n")[0])ifurl.path==b"/hello-user":peer=tls_connection.get_peer_certificate()ifpeerisNone:tls_connection.send(b"60 Please authenticate\r\n")else:cert=peer.to_cryptography()names=cert.subject.get_attributes_for_oid(NameOID.COMMON_NAME)common_name=names[0].valuefingerprint=cert.fingerprint(hashes.SHA256()).hex()tls_connection.send(str.encode("\r\n".join(("20 text/gemini",f"# Hello {common_name}",f"## fingerprint",fingerprint,))))else:tls_connection.send(b"51 Not Found\r\n")
Improving performance with async
What we’ve built so far is a single threaded synchronous server. This means that
the server handles one request at a time and cannot accept new connections until
the previous request has been dealt with. This works fine when testing on your
machine, but will struggle once you have multiple users and your routes are
handling more complicated logic and interacting with other systems like
databases or third party services. There are several ways to deal with this
problem.
Multithreading - where each new connection starts a new thread to handle the
request
Multiprocessing - similar but spawning a new process (more expensive than a
thread)
async/await - which hands back control to your program while it’s waiting for
the operating system to respond to an “I/O task” like opening a file or
responding to a network request
In python async/await is handled by the asyncio module. Amongst many other
things, asyncio provides
a function to start a TCP server using async connection handling
.
This will replace what we used the socket module for at the very start of this
post. Coming from the JavaScript world where everything is async by default it
may be a little frustrating that in order to use async await, you may need to
rewrite or wrap your code. Going into this in detail perhaps deserves another
post but in short pyOpenSSL doesn’t natively support async await so it was
necessary to create some wrapper code to interact with it in a non blocking way.
Deployment
After doing all of this we now have a working Gemini server. The next thing to
do is to share it with the world! I wanted to stick to Gemini’s do it yourself
kind of ethos and self host this. I found an old
Raspberry Pi Model 3 B
lying around. This model is from 2016 and has 1GB of RAM.
I purchased a memory card and flashed Raspberry Pi OS
lite onto it using the raspberry pi imager tool.
I had trouble getting wifi working on the pi, I think it’s something to do with
5GHz not being supported on this model but in the end I thought that a server is
probably better wired so I plugged it into my router via ethernet.
Initially I thought of deploying using a docker image but I didn’t really want
to rely on docker so I went with a simple systemd service that I deployed using
scp and ssh.
I then did something that I haven’t done before, expose a port publicly on my
router. I think I did this fairly safely, from the router user interface, I made
a firewall rule for TCP port 1965 and routed traffic to that port to the pi and
gave it a static IP. There we go I had joined the Geminispace!
The result
What I made was a basic TODO application. I’m not sure it has much practical
usage with all the fully featured todo list applications available, but it was a
fun to make. You can visit it in a gemini client by going to
gemini://todo.willmunn.xyz
. The source code is
available on codeberg
.
In the 20th century we had radio and television
programs
; a
television executive at ABC could
program
Thursday nights as a
comedy block; the television itself became the vessel for playing Video
Computer System game
programs
in the 1970s. By the 1980s one
could
program
their VCR to record television or even
program
their home computer on that same screen.
Programs have been ubiquitous your entire life. You heard them. You
watched them. You played them. You may have even made them. And as
broadcasting has given way to
narrowcasting
and
analog signals have given way to digital software, programs are
increasingly watching you and using that information to alter everything
from prices to our perception of reality.
We have all noticed this but no one seems to know what to do about
it. He have created a world where
anyone
can program media -
including the programs themselves. While access has never been greater,
literacy has never been lower. The vast majority of people haven’t been
taught even the most basic principles of this new media. We are
ignorantly grazing in a field filled with danger - unable to understand
the warnings so many have been shouting for decades.
Haunted House
Video Computer System Game
Program
(
Atari image via
Atari
Projects
)
The Warnings
The arrival of programmable interactive media led to a frenzied
interest that crescendoed in the 1990s with the introduction of the DVD
and the world wide web. By this point, the information age had become
tangible in the everyday life of ordinary people. But people generally
engaged new media like they did television: passively. But programmeable
media is too powerful to tolerate passive consumption. Realizing this, a
group of media theorists emerged to publish the
Principles
of Technorealism
on March 12, 1998.
The twelve original contributors were David Bennahum,
Brooke Shelby Biggs, Paulina Borsook, Marisa Bowe, Simson Garfinkel,
Steven Johnson, Douglas Rushkoff, Andrew Shapiro, David Shenk, Steve
Silberman, Mark Stahlman and Stefanie Syman. The final document was
based on the concept originated by Shapiro, Shenk and Johnson.
They opined that “in a world driven by the flow of
information, the interfaces – and the underlying code – that make
information visible are becoming enormously powerful social force,
[therefore] the creation of better tools should be an important part of
being an involved citizen. These tools affect our lives as much as laws
do, and we should subject them to a similar democratic scrutiny.” Or as
Technorealist contributor Douglas Rushkoff summed it up,
program
or be programmed
.
Rushkoff is not a computer programmer in any formal sense. He is a
media theoretician. So while he laments the absence of ubiquitous
computer programming education in US public schools, he is really
talking about digital media more broadly.
Digital media takes the notion of “analog programs” as we saw them on
television and turns them into flexible, programmable code. Today that
often means code that drives capital intensive recommendation algorithms
which now underpin the media ecosystem. If not as social media, then as
the innocuous Google News app that defaults on your phone. If not as the
Google News app, then as the digital newspaper that is constantly
re-configuring itself based on engagement metrics
As
C.
Thi Nguyen has said
, “Metrics are technology that standardize
attention.”
from social media and the hundreds of advertising
networks that fester in the ecosystem like flies on a decaying
corpse.
The Technorealists proposed to guard against this form
of Surveillance Capitalism 16 years before the term was coined. Point 3
of the manifesto reads: “Government has an important role to play on the
electronic frontier […] it is foolish to say that the public has no
sovereignty over what an errant citizen or fraudulent corporation does
online. […] Technology standards and privacy issues, for example, are
too important to be entrusted to the marketplace alone. Competing
software firms have little interest in preserving the open standards
that are essential to a fully functioning interactive network. Markets
encourage innovation, but they do not necessarily insure the public
interest.”
The
225 partners of DW
noyb
recently
filed a GDPR complaint against dict.cc
due to their consent
agreement featuring an astounding 1,741 partners. ‘Felix Mikolasch, data
protection lawyer at noyb: “It would take days or even weeks to properly
read and understand the data protection policies of 1,741 companies. It
is ridiculous to assume that this would allow for an informed
decision.”’
by
David
Schmudde
is licensed under
CC
BY-NC
4.0
The internet is truly a different type of media; digital
interactivity facilitates the same physical reach to both the
institutional power and the distributed peer. The interface to this
media is a sophisticated language colloquially known as
programming
.
The name
programming
elides its expressive
power, which is somewhere between mathematics and natural
language.
And if you do not know how to speak it, the institutional
powers will speak it for you.
Take the basic example of RSS/atom. I love RSS and use it to this
day. The premise is simple: in a diverse information ecosystem, you can
select feeds from major corporate news agencies, collectively funded
information providers, and self-funded bloggers and they all get equal
space in your reader. The feed is flexible; the sources inevitably
change from year to year as you discover new things and get tired of old
things. But this intellectual freedom comes at a cost - you must make
choices. You must create the program. The alternative is an
algorithmically defined news app that makes the program for you. And the
organization controlling that program makes tiny, seemingly innocuous
tweaks every week which can have profound consequences over time.
For example, the person who added the infinite scroll
to the Facebook timeline, Aza Raskin,
now
says he regrets it
. It turns out that removing the “next page”
button is a great way to keep people on Facebook but not a great way to
consciously use time. Or how about the time that Amazon paid the USA
Federal Trade Commission $2.5 billion to settle a case built around what
its
own employees called the “Iliad Flow”
- an Amazon Prime-cancellation
process spread across four pages and fifteen options, where every button
except one routed a user back to the start. Leaked internal documents
showed that when Amazon simplified the flow, cancellations dropped 14
percent, so the company reversed course and kept the maze.
Digital media’s flexibility, reach, subtlety, and power demands our
active engagement unique to any other preceding media.
On Speaking
The ability to speak the language of programming tended to favor
white males - but not exclusively and not historically. When programming
was less well-compensated and less highly regarded, it was a job for
women. And as
Prophets of
Computing
demonstrates, there have been a large number of local
approaches to machine programming throughout Asia and Latin America.
But the current crop of programmers - the people who by definition
control programmed media - does seem like more of a monoculture. A more
egalitarian world is possible but we will not arrive there in a
generation by putting iPads in classrooms. The Technorealists have
understood this for decades.
Again from the Technorealists: “Wiring the schools will
not save them. The problems with America’s public schools – disparate
funding, social promotion, bloated class size, crumbling infrastructure,
lack of standards – have almost nothing to do with technology.
Consequently, no amount of technology will lead to the educational
revolution prophesied by President Clinton and others. The art of
teaching cannot be replicated by computers, the Net, or by ‘distance
learning.’ These tools can, of course, augment an already high-quality
educational experience. But to rely on them as any sort of panacea would
be a costly mistake.”
To the contrary, multiple studies have shown that tablets
and laptops largely detract from the learning environment
The Laptop
and the Lecture: The Effects of Multitasking in Learning
Environments
(Hembrooke & Gay, 2003);
Dividing
Attention in the Classroom Reduces Exam Performance
(Glass &
Kang, 2019);
Examining
the impact of off-task multi-tasking with technology on real-time
classroom learning
(Wood et al., 2011);
Don’t
Throw Away Your Printed Books: A Meta-Analysis on the Effects of Reading
Media on Reading Comprehension
(Delgato et al., 2018), etc…
and tablets are a poor tool for teaching programming;
they are designed primarily for media consumption.
LLMs are the latest consumer layer of digital media. At a minimum,
they promise to become a metamedia
Marshall McLuhan coined the term “metamedia“ to
describe the relationship between form and content, especially in the
context of new technologies and new media.
of our everyday lives. Much has been written about how
LLMs will eliminate the entire field of programming - as if people will
no longer need to know how to speak for themselves. Programming will
still exist. But much will be obscured by a consumer veneer.
The language of programming is actually the language of systems.
Self-trained hackers were aware of this language when they were hacking
the telephone networks of the 1970s. And freshmen at MIT in the 1980s
understood this after reading the
Structure
and Interpretation of Computer Programs
. Humans build systems.
And systems contribute to our view of the world that we are born into.
LLMs can provide a surface-level understanding of a system but only
hands-on experience can provide knowledge.
You are constrained by any system that you cannot inspect. This is
true of both designed and emergent systems such as cities, forests, and
networks. The natural sciences, including informatics,
The term
informatics
is often used
interchangeably with
computer science
- where the latter
focuses on either the theory of computation or the study of the physical
computer.
produces the notation we use to describe what we know
about these systems. As long as there are systems and networks, there
will be programming.
On Mind
The physical computer is unique because it is the artificial
embodiment of a natural science. The theoretical computer has no
politics. But the physical computer is bound to the systems used to
create it.
The first point from the Technorealist manifesto:
“Technologies are not neutral. A great misconception of our time is the
idea that technologies are completely free of bias – that because they
are inanimate artifacts, they don’t promote certain kinds of behaviors
over others. In truth, technologies come loaded with both intended and
unintended social, political, and economic leanings. Every tool provides
its users with a particular manner of seeing the world and specific ways
of interacting with others. It is important for each of us to consider
the biases of various technologies and to seek out those that reflect
our values and aspirations.”
And those systems have historically been large,
well-funded bureaucratic projects or globally scaled, highly-capitalized
consumer products. So the end result of decades of effort to
“democratize computing” has produced an oligarchic class of unimaginable
wealth and a consistent decade-over-decade decline in the share of any
productivity gains created by the computer.
“The labor share — the percentage of economic output
that accrues to workers as compensation in exchange for their labor —
has been declining in the nonfarm business sector throughout most of the
post-World War Two era […] reaching a low of 56.0 percent in the fourth
quarter of 2011.” ~
Labor
share of output has declined since 1947
March 07, 2017
Or
take the example of newspapers. The papers were revolutionized by
productivity gains starting with the arrival of digital layout tools.
The industry employed
458,000
people in June 1990 and declined to 367,200 in December 2005
while
the revenues rose, profits remained strong, and the papers were a prized
acquisition asset. Average employee pay did not benefit from the gains
in efficiency. See also
the wage
calculator
to see how your work has been systematically devalued
since 1979.
So how do we fight back against these overwhelming odds? We program.
Not as an offense into their territory. But as a defense of our
territory - our mind.
Civilization is built on media. You cannot shelter your sons and
daughters; you have already seen the impact on your friends and
neighbors. It is past time to accept that in a programmable world of
digital media, we can no longer blame the television executive or the
media baron for their programs. We must support and participate in
alternative systems. From food to computing - we must create and
participate in systems that minimize cruelty as a bulwark against
systems that maximize profit.
The lingua franca of these systems is software. This language is at
least as important as the language you speak with your friends. Learning
the language of systems - a.k.a programming - will be a mind-opening
second or third language that you can use to manifest the world you want
to see in everyday media.
Timeline
Events from this post have been added to a
timeline
of significant events in the history of information.
Principles of Technorealism
“Technorealism demands that we think critically about the role that tools and interfaces play in human evolution and everyday life. Integral to this perspective is our understanding that the current tide of technological transformation, while important and powerful, is actually a continuation of waves of change that have taken place throughout history. Looking, for example, at the history of the automobile, television, or the telephone -- not just the devices but the institutions they became -- we see profound benefits as well as substantial costs.” †
Haunted House for the Atari 2600
Haunted House
(1982, Atari Inc.) is one of the earliest examples of the survival horror genre, which would not be well defined until
Alone in the Dark
(1992, Atari SA, née Infogrames) and
Alien vs. Predator
(1994, Atari Corp.). The scrolling playfield was innovative for its time and provided a sense that something could be lurking around the corner.
Creative Computing, November 1984 · PDF page 202.
Source scan
.
CyberLYNX promised lights, music and dinner waiting when you got home. Owen W. Linzmayer found Smarthome I dependable in his apartment, though the instructions were confusing. Once configured, the controller could run without the Apple. You still had to prepare the food.
[1]
Creative Computing, November 1984 · PDF page 241.
Source scan
.
CompuServe offered a party across nine states with one glass to wash afterward. In a separate 1984 account, Steven K. Roberts chatted from a hotel while someone in Washington arranged for beer to be delivered to his room.
[2]
The modem could bring you company and, with help, a drink.
Macworld, November 1984 · PDF page 67.
Source scan
.
Filevision promised to find the right wine for dinner by linking database records to drawings; somebody still had to supply the wine knowledge. A review abstract describes searches using up to four criteria.
[3]
Practical Computing liked it, but making a working copy on a single-drive 128K Mac took 17 disk swaps.
[4]
COMPUTE!, January 1994 · PDF page 171.
Source scan
.
Positronic Bridge promised an opponent that improved alongside you. David E. Lindeman found decent hand play and rigid bidding, but could not tell whether it had learned anything about him. The ad also promised duplicate bridge; his version offered rubber scoring only.
[5]
Creative Computing, November 1984 · PDF page 257.
Source scan
.
Mind Prober turned questionnaire answers into personality reports, with accuracy still unproven.
[6]
Beside it, Peter McWilliams described using an eye switch to select letters and words by blinking. He reported 5–10 words per minute from his own trial.
[7]
COMPUTE!, January 1984 · PDF page 104.
Source scan
.
COMPUTE!, January 1984 · PDF page 106.
Source scan
.
Two continuation pages, PDF 104 and 106.
COMPUTE! wanted a robot pet: lovable, teachable, durable, portable and educational. One family in the discussion was already giving robots personalities and dressing one up.
[8]
Sony’s $2,500 AIBO arrived in the US in 1999, with behavior shaped by interaction, though that hardly ticks every box on the wish list.
[9]
PC Magazine, May 29, 1984 · PDF page 56.
Source scan
.
U.S. Robotics imagined Benjamin Franklin publishing electronically with its Password modem and Telpac software. The manual admitted setup could be difficult; a later Atari user-group report records a Password owner using it with a 520ST and different software.
[10]
[11]
The modem was real; Franklin’s evening edition was the sales pitch.
In the Chips gave you a software company, fickle customers and competitors cutting prices. Tony Roberts’s VIC-20 review starts the player with $100,000 to spend on development, production, advertising and marketing.
[12]
Finally, a management career you could switch off.
Batteries Included recruited an Einstein caricature to guilt Commodore 64 owners into doing some work. Its Work Force range included writing, database and spelling tools. Ian Wright’s PaperClip review found useful formatting and printer support, early bugs, and an 80-column preview that needed a sharp monitor.
[13]
BYTE, January 1984 · PDF pages 154, 155.
Source scan
.
Mitel packed six office tools into Kontact and promised CP/M software access later. Ingenium preserves a unit used at home and work by a Mitel product manager.
[14]
A 1985 report put distributor Lanpar’s installations at about 300, with disappointing sales and limited software compatibility. Whether the promised CP/M access shipped remains unclear.
[15]
Creative Computing, June 1984 · PDF page 124.
Source scan
.
Type ’N Talk turned computer text into speech. Its manual offered spelling tricks and phonetic instructions for fixing pronunciation, so the computer sometimes needed help saying what you meant.
[16]
The Henry Ford keeps a unit and its documentation.
[17]
Computer Gaming World, January 2004 · PDF page 6.
Source scan
.
Star Wars Galaxies offered a life in the cantina, not just another firefight. Raph Koster discussed advancing as a musician in March 2004.
[19]
The surviving 2007 manual describes music, dancing and group performances that helped players recover from battle fatigue, though it cannot settle every detail of the 2004 game.
[18]
PC Magazine, April 12, 1994 · PDF page 439 · printed 441.
Source scan
.
PC Magazine, April 12, 1994 · PDF page 440 · printed 442.
Source scan
.
Two consecutive pages: printed 441–442, PDF 439–440. The gaming coverage continues beyond this selection.
Angela Gunn’s 1994 tour of AOL, CompuServe and The ImagiNation Network found chat rooms, cartoon identities, unwanted messages and people who weren’t quite who they claimed. AOL and INN plans included five hours; after that, friendship was billed hourly.
[20]
PC World, January 1994 · PDF page 165.
Source scan
.
Joe Abernathy followed PCs into Wal-Mart and Target, where buyers needed demonstrations, advice and support.
[21]
Daniel Pinkwater bought a Compaq Presario 425 despite struggling with its tutorial. A friend then set it up to skip Windows and launch his familiar word processor.
[21]
These fourteen selections come from a browsing collection of 21 magazine issues in 1984, 1994 and 2004. They were chosen for their images and specific claims, not as a representative sample or a scorecard of predictions. Their order reflects visual impact, the specificity of the promise and the available follow-up. Many advertise products already on sale. Dates refer to the magazine issues; PDF page numbers identify the scans and can differ from the printed labels. Linked reviews, manuals and contemporary accounts support the notes. The source notes identify material consulted only as abstracts or indexed passages. No products were personally operated for this article.
Warnings About ICE at the Polls Risk Their Own Kind of Voter Suppression
Intercept
theintercept.com
2026-11-02 05:51:00
Voting rights advocates in Texas warn that spreading awareness of voter intimidation could become a self-fulfilling prophecy.
The post Warnings About ICE at the Polls Risk Their Own Kind of Voter Suppression appeared first on The Intercept....
Anxiety over ICE
at the polls is a double-edged sword for the Trump administration: There’s the fear that federal immigration agents will illegally descend on voters, and there’s the threat that by boosting awareness of that fear, activists and journalists will help the federal government carry out the work of voter intimidation.
“Even just rumors of [ICE at the polls] are going to spread fear. That’s not what we want,” said Daniela Padilla, deputy director at Texas for All, a statewide coalition of organizations working on democracy and justice issues. “Even your reporting around, ‘We’re watching for ICE at the polls’ — it could scare voters.”
“The biggest threat that we face in Texas this year is voter suppression.”
In Texas, a 40 percent Latino state where a longtime Republican grip on power now finds itself in question, there are real concerns that Immigrations and Customs Enforcement agents could pop up in targeted communities to make people afraid to go out and vote on Tuesday. But while the
possibility
of
election interference
has been
discussed
as one of several “big ticket items” of this year’s midterms, Padilla said, “the biggest threat that we face in Texas this year is voter suppression.”
That suppression can take many forms. In Tarrant County, for example, where local officials closed 92 polling locations, reports from early voting indicated some election officials wrongly turned away eligible voters. More than fearing ICE would actually approach the polls, Padilla and several organizers working on election integrity and voting rights in Texas told The Intercept they were worried that journalists amplifying the Trump administration’s stated plans to “
surround the polls
” with ICE and federal agents could function as an inadvertent form of voter suppression.
Federal and state law prohibits federal immigration agents from being inside polling sites and bar any form of voter intimidation. Those laws haven’t stopped the Trump
administration
and its allies from threatening to deploy
ICE and military personnel
around the country during elections. In August, the Department of Justice announced that it planned to send at least
1,000 federal monitors
to oversee midterm voting, prompting multiple states to assert they would not let them inside polling locations. As states like Maine prepared to
respond
to the federal presence at the polls, the DOJ
claimed
to Newsweek it “never said” it would send federal law enforcement to the polls.
“The Trump administration has been spreading fear for months, refusing to say that they will not send ICE to the polls,” said Emily French, policy director at Common Cause Texas. “I think that the main function of this discussion around ICE at the polls from the Trump administration is scaring people into staying home on Election Day.”
According to Brianna Brown, executive director of the Texas Organizing Project, disinformation efforts are a common voter-suppression tactic in Texas. “So we’re trying to reassure people it’s safe to vote,” she said.
Texas is home to one of the most dramatic potential Senate upsets in the country, with Democratic state Rep.
James Talarico
leading in the polls over the state’s scandal-plagued Republican attorney general Ken Paxton in a race that could see Texas end up with a Democratic senator for the first time since the ’90s. And the 32 million-person state has
enough
voter suppression
strategies
even without ICE in the picture, Brown said, pointing to a
2021 bill
that banned drive-through voting, weakened protections for disabled voters, and gave more power to poll watchers appointed on behalf of a candidate or political party.
“Texas is just like a perfect storm,” Brown said.
When ICE showed up at a polling site in San Antonio in May — marking the first known case of ICE at the polls in the country during this year’s primaries — the
county sheriff kicked the agents out
. Even then, it did not appear that the federal agents were targeting someone in the process of voting; they had arrived at the scene shortly after state police, with whom they work closely, pulled someone over in the polling site’s parking lot.
That day, staff at the Texas Organizing Project and a group of civil and voting rights organizations working as part of the Texas Election Protection coalition helped coordinate outreach to the local sheriff to get ICE removed from the site.
The state coalition is part of a nationwide effort to protect this year’s elections from interference — federal or otherwise. In Texas, the Lawyers’ Committee and several other groups including the Barbara Jordan Leadership Institute, Common Cause Texas, and the Texas Civil Rights Project have been training people to help make sure voters know their rights on Election Day and that they report any instances of voter intimidation — whether an eligible voter is wrongly turned away or someone spots ICE — to a
national hotline
.
“We want to inform people, but we don’t want to scare them from coming to the polls,” said Khanay Turner-Osborne, executive director of the Barber Jordan Leadership Institute. She said she hasn’t heard much concern about ICE showing up around polling sites in Tarrant County, where she’s based and where people can vote at any polling site in the county. But still, for voters in hotspots of
ICE activity
and other forms of
voter suppression
, the group has told them to consider voting at a different eligible site, she said.
The group wants to help people prepare, Turner-Osborne said. “But I don’t want it to be in a way that deters people from voting.”
From a life-changing garlic press to a super-glowy serum: 12 buys you’re loving right now
Guardian
www.theguardian.com
2026-11-02 05:45:02
Casserole dishes, electric heaters and silk shirts: your autumnal favourites show you’re hunkering down – with one eye on party season • Don’t get the Filter delivered to your inbox? Sign up here Autumn, we’ve been expecting you. After a September that seemed to miss the memo about changing seasons,...
A
utumn, we’ve been expecting you. After a September that seemed to miss the memo about changing seasons, October made up for it with
serial storms
and nights that drew in faster than you could say “fall back”. Time to hunker down and, whisper it, even put the heating on – or at least fire up an
electric heater
if you’re still holding out on the radiators.
But while you were drawn to the indoor pleasures of heaters, hot
casseroles
and hotter
coffee
, you’ve also had your eye on party season, or at least been thinking about venturing into those corners of your wardrobe that haven’t seen daylight since March. From a
clothes steamer
to breathe new life into your threads, a
pencil skirt
and
sheer silk shirt
to update your wardrobe, to a
vitamin C serum
to keep your skin glowing all winter, these were your favourite things in October.
“Within a few seconds of activation, it had begun to make a chilly room feel cosy,” wrote
Pete Wise
of the overall winner in his test to find the
best electric heaters
. Five minutes later, on the highest setting, his “jumper was well and truly off”. Better yet, with a power rating of up to 2kW, you’d expect it to make a right racket – and it doesn’t. At a maximum of 46dB, Pete said it’s “far quieter than any other fan heater I tested”.
Sophie Wyburd
painted an irresistible picture in her roundup of the
best casserole dishes
: “Tinkering over a pot in the kitchen, glass of wine in hand … while the scent of beef shin ragu wafts through the house.” Yep, that’s me sold. At about a fifth of the price of the winning
Le Creuset
, Habitat’s chic oven-safe pot provides “brilliant heat retention at an excellent price”. It comes in autumnal
orange
, too (£47.50).
Where there’s a casserole, there’s got to be garlic. Reader
Stuart Hatcher
recommended three things in
Lucy Pavia
’s
rundown of
15 everyday items that make your life a teeny bit better
, and all three made it into your top 20 products of October. Your favourite was this garlic press that ended Stuart’s “time-consuming chopping”. (The other two, in case you’re wondering, are a
pair of wrist wraps for weightlifting
and a
portable air pump
that can blow up an air bed “effortlessly”.)
“It makes espresso. It makes black coffee. It makes cappuccino. It produces hot water on demand,” wrote
Sasha Muller
in his five-star
De’Longhi Magnifica Start review
. “Simple to use, sensibly designed … For everyday coffee drinkers, it’s a great buy.” And many everyday coffee drinkers among you did indeed buy it – the fact that it’s suddenly dark at 5pm may have helped fuel your desire for a decent caffeine fix.
Free yourself from the iron with this lightweight handheld steamer from Philips –
Zoë Phillimore
’s
top budget pick in our guide to the
best clothes steamers
. “This is a great model if you’re just dipping your toes into the iron-free life,” she wrote. It’s not powerful enough to blast through a whole pile of creased laundry, and Zoë found it “a bit spitty” to use on silk, but it made short work of her cotton items and linen trousers.
Go to work on an egg, or half a dozen of them, with this brilliantly affordable egg cooker that can quickly boil up to six eggs or poach two at a time. Reader
Amanda Nieurzyla-John
chose hers as her
everyday item that makes your life better
, writing that it “takes the guesswork out of cooking while you concentrate on the rest of the meal. I don’t know how I have cooked all these years without it.”
“I was astounded when, after a week of using it, people started commenting on how ‘glowy’ my face looked,” wrote
Zoë Phillimore
in the Filter’s second-birthday roundup of
31 things we loved this year
. Zoë bought the serum after reading about it in
Danielle Wilkins
’s
guide to the
best vitamin C serums
, and says she hopes “it keeps me looking alive all the way through the grey winter months”.
For
Sali Hughes
’s
friends who barely use moisturiser, let alone Botox, age-related hair thinning is “their line in the sand”, she wrote in her
beauty column
. Judging by how many of you heeded Sali’s advice about “a treatment that actually works”, you feel the same. The clinicians at subscription service Hair+Me dispense proven hair-loss medications such as minoxidil and oral treatments, which can reduce the risk to pets.
“The pencil skirt is back,” wrote
Melanie
Wilkinson
in her roundup of
52 women’s autumn wardrobe updates
, and she must be on to something because it was your favourite of the lot. The combination of timeless tailoring, elasticated back waist and side pockets has proved so irresistible that the size 14 had sold out at the time of writing.
“Sheer clothes aren’t just for red-carpet exhibitionists any more,” wrote fashion editor
Jess Cartner-Morley
in her
October style essentials
. A sheer shirt is particularly versatile because you can “wear it over a tank top during the day” then pair it with something less modest for evening (“a lace bra if you are feeling slightly racy,” suggested Jess). This shirt is now reduced to £54 in all sizes, making it an excellent buy ahead of party season.
After testing 40 boxes for her guide to the
best lunch boxes
,
Zoë Phillimore
chose this colourful plastic caddy as the best option for children. It “feels seemingly indestructible, surviving a school day and a night in lost property”, and, unusually for kids’ lunch boxes, has a silicone seal that encloses each compartment separately. Zoë loved that it included a spoon and fork, but wished replacements were available to buy.
You might not expect to see a new iPhone in a guide to the
best cheap phones
, and indeed £599 stretches the definition of cheap, but it’s affordable for an iOS device – especially now that
“RAMageddon”
(a shortage of memory chips, as opposed to a sheep crisis) has hiked phone prices. The 16e has only one rear camera and no MagSafe, but it packs “great performance and battery life” for its price, wrote
Thomas Deehan
. Look out for it in the Black Friday deals in the coming weeks.
Don’t buy anything until you’ve read the Filter, the Guardian’s home for truly independent product reviews and recommendations. Whether you’re after rigorous tests or shopping inspiration, affordable gifts or advice on how to make your belongings last longer, we’re here to help you make smarter, more sustainable choices.
The Filter is packed with trustworthy buying advice on everything from coffee machines to hiking boots, mascara to secateurs. So visit us today and start buying better and smarter, and wasting less.
You're probably reading this page because you've attempted to
access some part of
my blog (Wandering
Thoughts)
or
CSpace
, the wiki thing it's
part of. Unfortunately whatever you're using to do so has a HTTP
User-Agent header value that is too generic or otherwise excessively
suspicious. Unfortunately, as of early 2025 there's a plague of
high volume crawlers (apparently in part to gather data for LLM
training) that behave like this. To reduce the load on
Wandering Thoughts
I'm experimenting with
(attempting to) block all of them, and you've run into this.
All HTTP User-Agent headers should clearly identify what they
are, and for non-browser user agents, they should identify not just
the software involved but also who specifically is using that software.
An extremely generic value such as "
Go-http-client/1.1
"
is not something that I consider acceptable any more.
One specific thing is that if your HTTP User-Agent header
contains an explicit marker that you are only 'compatible' with
what your User-Agent suggests, you must include an explicit, real
URL that describes your activities or the software. A minimal HTTP
User-Agent such as 'Mozilla/5.0 (compatible; mysoftware)' is not
acceptable. The same is true if your HTTP User-Agent says that you
are a 'bot' of some sort; your bot needs a working URL explaining
itself. An email address is not sufficient in either case.
Chris Siebenmann, 2026-04-18
Palantir AI tech unused or little used by 29 NHS trusts said to be benefiting from it
Guardian
www.theguardian.com
2026-11-02 05:19:46
Exclusive: Gap between NHS claim and trusts’ comments described as ‘very misleading’ by senior health official More than two dozen hospital trusts the NHS claims have benefited from Palantir’s medical data platform are not using all or most of the AI-powered technology, the Guardian can reveal. Th...
More than two dozen hospital trusts the NHS claims have benefited from Palantir’s medical data platform are not using all or most of the AI-powered technology, the Guardian can reveal.
The 29 hospital trusts that NHS England
publicly lists as “realising benefits”
from the controversial £330m federated data platform (FDP) have said the system is not live or they are not using any of its applications. The apparent gap between the NHS claim and the trusts’ understanding was described as “very misleading” by one senior health official.
It comes as the health secretary, Yvette Cooper, faces calls from MPs and campaigners to axe the US tech firm’s contract to deliver the flagship government AI system, which aims to connect scattered medical files, speed up hospital discharges and improve surgery scheduling among other potentially life-saving improvements.
Palantir’s UK and Europe chief executive, Louis Mosley, said ending its contract would be “putting politics above patients” and accused critics of “Palantir Derangement Syndrome”. The company has contracts to supply AI to the Israeli military and Donald Trump’s homeland security crackdown, while its billionaire founder, Peter Thiel, has questioned democracy, said tech companies can “unilaterally change the world” and called technology an “incredible alternative to politics”.
NHS England has previously faced scrutiny over how it has used data to defend the Palantir contract, which began in 2024.
It claims 117,539 more operations were carried out by trusts using the FDP in a year
, but this summer it conceded it could not prove cause and effect, and the Office for Statistics Regulation (OSR) watchdog ran an investigation. Asked about the claim of NHS Trusts “realising benefits”, the OSR said it will now be “taking this forward under our casework function”.
The contrast between the number of hospitals the
NHS
claims realise benefits from Palantir’s technology and the statements of the hospitals themselves emerged from more than 100 freedom of information requests. NHS England states on its website that 137 trusts are “live” with the FDP, but of those described as “realising benefits”, 29 responded that they were either not using the system or its apps.
Epsom and St Helier said it was “not using FDP at the moment”, Kent and Medway said the trust “has been focused on other priorities … the FDP is not currently live”, Mersey Care said they had “not yet gone live with the FDP”, Nottingham University Hospitals said “no implementation timeline has been agreed”, and University Hospitals of North Midlands said: “FDP is not live. We do not intend to roll it out in the future.”
Chi Onwurah, chair of the Commons science and technology committee, said: “If many NHS trusts don’t see the value of the FDP, serious questions need to be asked about whether the hype surrounding Palantir’s products matches reality.”
She called on the government to activate the 2027 break clause in Palantir’s contract and launch an open competition for a UK-owned and based supplier to handle patient records. The Commons health select committee is also calling for ministers to pull Palantir’s deal.
The FOIs were obtained by Medact, a medical charity campaigning for Palantir to be ejected from the NHS
,
and shared with the Guardian. Medact spokesperson Dr Rhiannon Mihranian Osborne said: “Our NHS institutions are meant to work in the interests of public trust and evidence-based healthcare, not use creative accounting to justify a deeply controversial contract. There is no good reason for the government to continue this failing project. We now have to ask, if ministers are using these flawed claims to make their decision, are they being misled?”
Asked to respond, NHS England insisted nearly 140 trusts “have now reported benefits from the wide range of tools available on the platform, helping staff join up and improve care for patients, and work more efficiently.”
It has previously said that its list of hospital trusts reporting benefits includes those using the tools for back office data analysis and submission rather than to support patient care. It is understood trusts that say they do not use the FDP may use a small part of its functionality to help calculate treatment costs. For example, Torbay and South Devon NHS Foundation Trust said it is “not currently operationally using the FDP” but NHS England records it as “live” because infrastructure is in place “to support data submission, rather than active operational use of FDP apps.”
Palantir has been hitting back against criticism of its deal and last month hired Labour’s former deputy leader and member of the House of Lords Tom Watson as a full-time senior lobbyist. Watson this week has been posting videos on X aiming to debunk “myths” about Palantir, including concerns that public services will become “locked in” to its systems. He rebutted criticism from the Green party leader Zack Polanski, who said it was “beyond ridiculous” that Lord Watson was a life peer.
Watson and Mosley have also been writing directly to critical MPs in letters that have counted the times they have spoken out against the company, which one recipient, Labour’s Clive Lewis, described as “very Big Brother”.
Marc Farr, chair of the NHS Chief Data and Analytics Officers Network, said: “I am concerned to hear that there are 29 trusts reported as receiving benefits but when asked are not actually using the tool. It is great that a number of trusts that have adopted the FDP are benefiting from this nationally funded programme, but it is crucial for public support … that benefits are reported accurately, transparently and in a way that can be properly verified.”
A Palantir spokesperson said: “Palantir software is helping to deliver better care, with trusts using it recording 117,000 additional operations, a 14.3% reduction in discharge delays for long stay patients and a 5.6% improvement in the number of people finding out whether or not they have cancer within 28 days.”
It’s time to block reckless AI IPOs and tax billionaires | Naomi Klein and Ilana Glazer
Guardian
www.theguardian.com
2026-11-02 05:00:38
We cannot let the billionaire class take unprecedented risks with our future America is a divided country in crisis. Despite the sunny economic forecast coming from the Trump administration, millions of Americans are struggling to find and keep jobs, getting crushed under the rising cost of living, ...
A
merica is a divided country in crisis. Despite the sunny economic forecast coming from the Trump administration, millions of Americans are struggling to find and keep jobs, getting crushed under the rising cost of living, and losing health insurance when they need it the most.
Meanwhile, via corporate consolidation and political corruption, the billionaire class is getting richer faster than ever before: a
40%
increase in wealth in just the last two years. Today, the combined wealth of nearly 3,000 global billionaires equals
one-fifth of the entire world’s yearly output
. Nearly 1,000 billionaires live and do business in the United States.
The tools of their domination are tax loopholes, corporate mergers and, in the tech world where many of these billionaires exist,
IPOs
. When a company goes public, its founders and investors win the capitalist lottery. And instead of paying their fair share of taxes and redistributing resources back into society, most turn around and fund even more corporate consolidation and political corruption to extend their dominion.
Elon Musk became an instant trillionaire when SpaceX went public earlier this year, on the backs of workers’ pensions – and is bankrolling the
Republican midterm campaigns
with a small percentage of his egregious gains. Oracle’s Larry Ellison – currently seventh on the
Forbes billionaires
list – backed the Paramount-Warner Brothers merger that is destroying the film industry and the livelihoods of the Americans who labor in it. Google’s Sergey Brin spent
$100m opposing California’s Prop 40
, a one-time wealth tax that would put the funds toward healthcare and other public services.
To pay for essential services, we must tax billionaires, and pass legislation to do so starting in California and New York, but that is clearly not enough. We cannot allow more billionaires – or trillionaires – to be made in this country, further expanding the inequality chasm that’s turning a dignified life into a luxury in this country. We must also block the impending tech IPOs and preserve what’s left of our economy and our society.
The Trump administration and tech investors have positioned IPOs as market success stories: when a company goes public, US investors could gain from their valuations. But regular Americans also shoulder a company’s risks – and risk their own futures when that stock ends up in their retirement accounts without consent.
Problems with a private company or its products become problems for everyone invested in that company once it goes public, whether they’ve made the choice or not. And when the speculative investment bubble pops – such as with the 2008 crisis – regular people end up holding the bag.
It’s no secret that the AI companies Anthropic and OpenAI are intending to go public in the near future. Both of these players also had major security flaws exposed recently, and openly acknowledge mind-bogglingly huge amounts of debt. If these companies are allowed to go public, their cofounders and investors – some of whom are already billionaires – will rise to whole new strata of private wealth.
Americans, meanwhile, will be financially crushed if the technologies fail or if these companies prove unable to make good on their extravagant financial promises. Already, the
10 most valuable public companies
are in tech, which means the country’s economy depends on the stability of a single industry, one with an undeniable history of bubbles, crashes and, in extreme cases like Theranos, outright fraud.
June’s highly contentious SpaceX IPO already carries great risk for regular investors. If more speculative tech companies with staggering risk profiles are allowed to follow suit, the results will become too much for the market to bear.
If and when the AI bubble bursts, it will take all of us with it, investors or not. Recessions level lives indiscriminately, and globally. The one exception will be the billionaires, who will have sufficient resources to walk away from the wreckage relatively unscathed.
We cannot let the billionaire class take these kinds of risks with our future. To stop these reckless IPOs we must stand together, take collective action, and push for real change. We must find legal and legislative avenues to stop the new tech IPOs from minting new billionaires and transferring the AI industry’s enormous risks onto us.
The first step is passing wealth taxes on billionaires to keep them in check. Today, let’s send a message to the billionaires that Americans will not lie down and let them bulldoze working people and the environment on their way to the bank. Tax the billionaires. Block the IPOs.
Naomi Klein is a Guardian US columnist and contributing writer. She is the professor of climate justice and co-director of the Centre for Climate Justice at the University of British Columbia. Her latest book is End Times Fascism: And the Fight for the Living World, co-written with Astra Taylor
Ilana Glazer is a comedian and video-podcaster who co-created the comedy series Broad City, wrote and starred in the movie Babes, made her Broadway debut last year in Goodnight, and Good Luck and, now, hosts the comedy and socio-political video-podcast, It’s Open With Ilana Glazer
Ivanti patches two more Connect Secure zero-days chained in attacks
Bleeping Computer
www.bleepingcomputer.com
2026-11-02 04:33:26
Ivanti released security updates to patch two more Connect Secure (ICS) remote access gateway flaws that have been chained in zero-day attacks since at least October 1. [...]...
Ivanti released security updates to patch two more Connect Secure (ICS) remote access gateway flaws that have been chained in zero-day attacks since at least October 1.
Threat actors exploited these two flaws (tracked as
CVE-2026-111357
and
CVE-2026-118527
) alongside an authentication bypass vulnerability (CVE-2026-104935), which Ivanti
patched last month
to remove the main attack surface.
CVE-2026-111357 is a high-severity privilege escalation flaw, and CVE-2026-118527 lets authenticated attackers upload files on an unpatched device.
When it released patches for CVE-2026-104935 in October, Ivanti urged admins to upgrade as soon as possible since it was aware of "limited exploitation in the wild."
"At the time of discovery of exploitation, Ivanti had a fix developed for CVE-2026-104935 which was already in QA for supported versions, and we made the decision to not delay getting a fix to customers that resolved the primary exploitation path when the LPE was discovered," Ivanti
added
in an advisory updated on Thursday.
"Because of this, our focus was initially centered on the disclosure of CVE-2026-104935 and providing protection for our customers as quickly as possible."
Cybersecurity firm Volexity, which found the flaw,
reported at the time
that a threat actor (UTA0576) was chaining the CVE-2026-104935 vulnerability since at least October 3 with two other flaws to gain unauthenticated root-level code execution on fully patched ICS devices.
"Notably, UTA0576 deployed no malware or web shell. Instead, the threat actor re-exploited the device for every command it ran and tampered with logs to cover its tracks, evading Ivanti's Integrity Checker Tool,"
Volexity said
.
Threat intelligence firm Defused
also confirmed Volexity's report
, saying that it spotted the first signs of active CVE-2026-104935 exploitation starting October 1.
On Tuesday, the US Cybersecurity and Infrastructure Security Agency (CISA)
added CVE-2026-104935
to its catalog of actively exploited vulnerabilities and ordered federal agencies to patch their systems by Friday,
Threat actors frequently target Ivanti vulnerabilities, often in zero-day attacks, because they provide a quick way into enterprise networks to steal sensitive corporate and customer data.
Over the past several years, Ivanti has patched several other Connect Secure zero-days exploited in the wild, some of them chained in
widespread attacks
to
deploy malware
.
It addressed a command injection (CVE-2024-21887) and an authentication bypass (CVE-2023-46805)
in early January 2024
, and a server-side request forgery (CVE-2024-21893)
two weeks later
.
Ivanti has over 3,000 employees, a network of more than 7,000 partners, and over 40,000 customers using its IT asset management solutions worldwide.
Since November 2021, CISA has flagged
38 Ivanti vulnerabilities
as actively exploited in the wild, including 12 that have also been abused in ransomware attacks.
Every Linux syscall table in one place, grouped by architecture: system calls for 27 architectures with their numbers, arguments and registers, from Linux 3.3 to 7.3.
Every architecture's table, with the full history of when each syscall appeared and when it was dropped, is also a plain JSON file at
/<arch>/syscalls.json
, for example
/x64/syscalls.json
.
Calling conventions
Number in
NR
, arguments in
ARG0
to
ARG5
, then the instruction. The result comes back in
RETURN
.
Any kernel release, diffs between releases and more in the
API docs
.
Syscall tables across kernel versions
Entries get added and, rarely, removed between releases. Each architecture page has a kernel selector, and its JSON download records when each entry first appeared (
since
) and when it was dropped (
removed_in
).
Removed entries stay visible: pick "all, with removed" in the kernel selector and each dropped syscall is marked with the release that dropped it.
Where the data comes from
Everything on this site is read from the kernel source itself. For each architecture, the generator walks the syscall table and the
SYSCALL_DEFINE
macros of every mainline release it scans, so these are the numbers the kernel actually dispatches on.
It also records each architecture's calling convention, the instruction that enters the kernel, so every syscall table shows exactly where the number and the arguments go.
A syscall whose
since
equals the first release scanned for its architecture was already there back then, and may be much older.
Ryanair considers pay freezes and reducing crew on flights as fuel costs soar
Guardian
www.theguardian.com
2026-11-02 04:06:51
Airline weighs up increasing use of AI as part of cost-cutting measures amid warnings of €2bn rise in annual fuel bill Ryanair is considering cost-cutting measures including pay freezes, reducing crew numbers on flights and increasing the use of AI to offset soaring fuel costs driven by the Middle E...
Ryanair is considering cost-cutting measures including pay freezes, reducing crew numbers on flights and increasing the use of AI to offset soaring fuel costs driven by the
Middle East conflict
.
Europe’s largest low-cost carrier said it expected European airlines to raise ticket prices by
up to 20%
, with its own jet fuel costs forecast to rise by about €2bn (£1.6bn) in its next financial year, which ends in March 2028.
“Unless there is a swift resolution of the Middle East conflict we expect jet fuel prices to remain elevated well into 2027,” said Michael O’Leary, the chief executive of
Ryanair
. “While we believe European short-haul fares will increase materially to reflect these much higher jet-fuel prices and competitor capacity reductions, this is unlikely to provide a full offset for an estimated €2bn increase in our fuel bill.”
Oil prices hit a six-month high on Friday before falling to about $110 a barrel on Monday.
Ryanair expects European short-haul fares to “increase materially” next year, as fuel hedging deals taken out by airlines to reduce costs expire.
“To help address this, we’re reviewing all costs across group airlines with a particular focus on airport churn negotiations, maturing labour agreements (which may require temporary pay freezes), lower crew ratios and increased use of AI,” O’Leary said.
The company said its increased use of AI would focus on improving productivity in operations and back-office and support functions.
Ryanair said 80% of its fuel requirements until the end of March were hedged at $67 a barrel. The company’s fuel requirements for the subsequent 12 months are 25% hedged at $95 a barrel.
The Irish airline, Europe’s largest by passenger numbers, said average fares fell by 2% year on year in the quarter ending in September. Ryanair added that fares in the final three months of the year were trending “modestly ahead” of the same quarter last year.
“The booking window remains closer-in which reduces visibility,” the company said.“[The current quarter’s] fare outcome will heavily depend on the strength of close-in October-November school midterm [holiday] and Christmas bookings.”
The airline reported a post-tax profit of €1.61bn in the quarter ending inSeptember, a fall of 6% year on year but in line with analyst expectations. Passenger numbers grew 5% year on year to 64.5 million during the quarter.
(Sorry, I couldn't resist not putting a catchy title :)
Prompted by another project idea that popped into my mind, I did some Unix archaeology and looked at the original
sources
. This is the most fun I've ever had reading someone else's code - probably due to the historical significance. But also because of how delightfully approachable it was, compared to modern monstrocities.
This is a short(-ish) post regarding how they handled the mundane and boring task of allocating and managing memory (in C). In the future I'd like to read more about Unix the OS, its design, architecture & implementation.
A long time ago...
In a PDP far, far away, our journey begins with the fourth edition of Unix, released November '73,
more than half a century ago
. C actually first appeared in the second edition, released the year before, but wasn't extensively used. V4 is special because even the majority of the kernel was written in (let's call it "medieval") C by that point. The system had grown, however, and it was quite capable. You'll find that many Unix things we have today in POSIX-land were already present in V4.
Super fun fact:
A tape containing V4 was only found
last year
! Previously, it was thought to be completely lost to time.
While cleaning a storage room, our staff found this tape containing UNIX v4 from Bell Labs, circa 1973
~ Rob Ricci
Linear allocation
The first interesting bit is found among the C "compiler" sources. In quotes, because it wasn't a monolithic compiler as we know it today. Rather, it was a set of distinct tools which you chained in order to get an object file from C code.
In
/usr/c/c20.c:814
, from what I can gather a very primitive optimiser, if you will (referred to as "C object code improver" in V5; there aren't many comments in V4) you see:
alloc(an)
{
register int n;
register char *p;
n = an;
n++;
n =& ~01;
if (lasta+n >= lastr) {
if (sbrk(2000) == -1) {
write(2, "Out of space\n", 14);
exit(1);
}
lastr =+ 2000;
}
p = lasta;
lasta =+ n;
return(p);
}
Isn't that fascinating? This routine is used throughout the program to allocate memory. And if you squint, this is just a linear allocator. It aligns the size, checks if the region is big enough, otherwise tries to extend it. On failure, it just cold-exits. Which means callers don't have to check for
NULL
every single time, and indeed they do not.
The "region" isn't named, unlike how you'd do it today with arenas. However, the above does have a nice property, in that by using
sbrk()
, when we extend the heap, no pointer is invalidated.
And they don't bother freeing memory, because why would they? There's no need. Everything exists for the lifetime of the program.
dmr/malloc
This is the most interesting bit in my opinion, and the reason why I chose that title.
In Dennis' directory, you find a file named
malloc.c
. And you see two functions there:
Seems familiar? Yep, it's
mfree()
with an
m
. These functions obviously deal with memory (de)allocation, and are the precursor to the ones we're familiar with today.
But
, the interfaces are different.
malloc()
receives an additional first argument called
mp
(for "map"), and so does
mfree()
. What is that? Well, let's see how they're used:
p = malloc(coremap, sizeof foo);
...
mfree(coremap, sizeof foo, p);
q = malloc(swapmap, sizeof bar);
...
mfree(swapmap, sizeof bar, q);
So, they pass in some objects
coremap
and
swapmap
. What are those?
From
/usr/sys/systm.h:2
:
int coremap[CMAPSIZ];
int swapmap[SMAPSIZ];
They're just flat memory regions! So these
malloc()
and
mfree()
work on
explicit memory regions
! Not only that,
mfree()
receives the size of the object it is deallocating,
as it should
. Those are just infinitely-times better interfaces than traditional
malloc()
/
free()
.
Imagine using a library that has such an interface. You could deallocate all its memory resources by just discarding the entire region you delegated to it. Heap corruption bugs would be less severe because you could partition memory. You could trivially limit the amount of memory third party code allocates. The list goes on and on...
(Side note: They slightly messed up here and the regions should've been sized
CMAPSIZ/2
and
SMAPSIZ/2
respectively. They did fix that later.)
However, I don't think all these benefits were in the minds of Dennis and Ken. So why did they go with that interface and why did they partition memory to "core" and "swap"? The reason is, I believe, just hardware constraints. These interfaces stay like that up to V6 (maybe V7 too?), released the spring of '75, and thankfully there's an infamously good resource about that edition.
From
A Commentary on the Sixth Edition UNIX Operating System
by John Lions:
These are concerned with the allocation and subsequent release of two kinds of memory resources, namely:
main memory
in units of 32 words (64 bytes);
disk swap area
in units of 256 words (512 bytes).
For each of these two kinds of resource, a list of available areas is maintained within a resource "map" (either "coremap" or "swapmap"). A pointer to the appropriate resource "map" is always passed to "malloc" and "mfree" so that the routines themselves do not have to know the kind of resource with which they are dealing.
And it goes on to explain how the underlying allocation algorithm works. These do not do linear allocation (hence the
mfree()
). Basically, it's a list of blocks (
struct map
), ordered by address, where you do first-fit and also coalesce neighbour blocks.
... to the topic of linear allocation... In V6's
/usr/source/s1/diff1.c:88
, I think the first ever version of
diff(1)
, you find:
char *area;
char *top;
alloc(n)
{
register char *p;
p = area;
n = (n+1) & ~1;
area =+ n;
while(area > top) {
if(sbrk(1024) == -1) {
mesg("Out of space\n");
exit(1);
}
top =+ 1024;
}
return(p);
}
This is not all that much different from the previous
alloc()
, but this time it handles any allocation size. It uses a
while
to continue extending the region and stops when enough space has been reserved. The
actual
interesting part happens later, at
unsort():143
:
unsort(f, l, b)
struct line *f;
int *b;
{
int *a;
int i;
a = alloc((l+1)*sizeof(a[0]));
for(i=1;i<=l;i++)
a[f[i].serial] = f[i].value;
for(i=1;i<=l;i++)
b[i] = a[i];
area = a;
}
I'm not exactly sure what that function does, but it doesn't matter. What matters is that they allocate some memory (
a = alloc(...);
), then roll back the region (
area = a;
)! Though they subsequently just copy it to
b
, so it seems like this was redundant.
Computing sizes
Most of the code directly calls
malloc()
/
alloc()
and friends, and inlines the size computation logic. Like the line we saw earlier:
a = alloc((l+1)*sizeof(a[0]));
I don't do that anymore, and neither should you! I use a couple macros akin to C++'s
new
that handle all the
sizeof
business. But I am a dumb-dumb. Surely these guys must've come across the same pain points, and, if so, what did they do? Let's jump to V7, released January '79, four years after V6. You find this macro:
#define ALLOC(x) (struct x *) ckalloc(sizeof(struct x))
where they use it like:
struct foo { ... };
p = ALLOC(foo);
Nice! It's interesting to note
where
you find it; In
/usr/src/cmd/f77/defines:38
and
/usr/src/cmd/make/defs:38
. Slightly suspicious. The former is part of the Fortran 77 compiler and latter of
make
. Stuart Feldman is the person who worked on both of those programs! It's clear that multiple people were working on the system at that time. A macro like that isn't used in any other program, sadly.
V7 is the first edition where you also find the
assert()
macro, and it was used extensively back then too. Good job! They also significantly expanded the number of man pages. Fun fact: I did
$ man V7/usr/man/man3/foo.3
-- on 47 year old pages -- and it worked first try absolutely fine.
Let's go to V8, released in February '85, four years before the first C standard. You find this wild macro in
/usr/src/cmd/ideal/memut.c:7
:
char *fooalloc;
#define tryalloc(new,kind) \
if (!(new =(kind *) malloc(sizeof (kind)))) {\
emergency ();\
if (!(new =(kind *) malloc(sizeof (kind)))) {\
fprintf (stderr, "ideal: Out of space\n");\
exit (1);\
}\
};\
for (fooalloc = (char *) new;\
fooalloc < ((char *) new) + sizeof (kind);\
fooalloc ++)\
*fooalloc = '\0';
If someone is too harsh because you used macros, point them at the above. Usage is bit different this time:
struct foo *x;
tryalloc(x, struct foo);
I decided to include it because the macro zero-initialises by default whatever was allocated! You also see a more familiar-looking
malloc()
being used. It appears this is the edition that they switched to the more traditional
malloc()
/
free()
interfaces we know. I'll hypothesize why that is at the very end.
But it gets even more interesting.
In V8 you also find these macros:
Unfortunately, I can't find the author(s) of any of these.
Cfront
In V8, you'll also find Cfront, a.k.a. medieval C++. It's then reasonable to assume that Stroustrup was inspired by the above macros and decided to incorporate
new
as a language-level feature. However, in
A History of C++: 1979−1991
, he writes:
In Simula, it is not possible to have local or global variables of class types; that is, every object of a class must be allocated on the free store using the
new
operator.
Huh. So C++'s
new
was inspired by the programming language "Simula". It's fair to say that the macros above were likely also derived from there. But Simula was quite old even by the 70s, let alone 1985. Perhaps Stroustrup persuaded people, or wrote those macros himself. He later writes:
The
new
operator that both allocates memory from the free store and invokes a constructor to ensure initialization was borrowed from Simula. The
delete
operator was a necessary complement because I did not want C with Classes to depend on a garbage collector. The argument for the
new
operator can be summarized like this. Would you rather write:
X* p = new X(2);
or
struct X * p = (struct X *) malloc(sizeof(struct X));
if (p == 0) error("memory exhausted");
p−>init(2);
and which version are you most likely to make a mistake in? The arguments against -- which were voiced quite a lot at the time -- were "but we don’t really need it," and "but someone will have used
new
as an identifier." Both observations are correct, of course. Introducing operator
new
thus made the use of free store more convenient and less error−prone.
This increased its use even further so that the C free store allocation routine
malloc()
used to implement
new
became the most common performance bottleneck in real systems.
[...] Having real
programs spend 50% or more of their time in
malloc()
wasn’t acceptable.
(emphasis mine :P)
He makes a compelling argument, which I agree with, but doesn't mention the
new()
macros that basically did the same thing. Although you'll find many programs in V8 that do the latter, verbose thing. Now I want to know the author(s) of those macros even more!
Sims
This is a big parentheses because I wanted to know how Simula came up with
new
.
Wikipedia
states:
Simula 67 introduced objects, classes, inheritance and subclasses, virtual procedures, coroutines, and discrete event simulation, and featured garbage collection.
And it appeared in
1967
. Wow! In the
1964 report
of SIMULA I, you'll find:
[...] The symbol "
new
" only serves to resolve an ambiguity of the identifier
A
inside the activity
blah blah blah
[...]. Elsewhere the symbol
new
is redundant, but can often be used to advantage to improve readability.
So it was only used to resolve grammar ambiguities (I think?). And Stroustrup re-purposed it. Clever!
Further reading
The History of Simula
, by Jan Rune Holmevik. Reads like (bad) Sci-Fi. All the politics nonsense aside, what I found most curious is that they completely specified the language before attempting to write a compiler for it. They started writing the first compiler
atleast a year after
. This would sound insane today, but I guess the cost of writing code was substantially higher back then.
It's honestly amazing how influential Simula was. In "10.5 Operating Instruction" they even use the term "run time system" and give it the same meaning we're accustomed to.
Simula, in turn, was partly inspired by SIMSCRIPT (don't laugh). If you skim its
1962 report
, you won't find a mention of
new
. You'll however find
CREATE
:
When a new Entity is created by a
CREATE
statement, space for storing its Attribute values is allocated in the amount specified on the Definition Form. This record is given an identification number, and the Variable specified in the
CREATE
statement is automatically set equal to this number.
For example, the statement "
CREATE DOG CALLED FIDO
" allocates the required amount of storage for the Attributes of a
DOG
, and sets
FIDO
equal to the identification number of the Attribute record.
Fascinating. The whole SIMSCRIPT report is fascinating actually, and I only briefly skimmed through it. At the beginning, it even describes something like a primitive
ECS
, by my understanding. At the end though it slaps you back to reality:
Although, by and large, almost every program doesn't do this. Most just directly call
malloc()
and friends, and do size calculations \w error checking at every call site. I don't know why, considering how much error-prone that is, but maybe due to inertia. It's a similar story in V10, but I didn't check it as thoroughly since it's quite big (it includes medieval GCC!).
So what happened?
They were so close to arenas; with linear allocation and explicit memory regions, rolling back, and even having a nice
new()
macro. All this is speculation at this point, but maybe some of it is correct. First of all, most programs had trivial object lifetimes. Allocate memory, read some data, process/crunch and spit out some result. You don't need any disciplined memory allocation strategy for that. You can even use
malloc()
for everything and let the OS reclaim it once the process exits. Given how simple
malloc()
was at first, it wouldn't be much of a bottleneck either. The philosophy of programs only "doing a single thing well" also fitted this pattern.
Secondly, it's clear some decisions were motivated by the very little memory they had to play with. And there was no virtual memory either (initially, that is). Nowadays, you can "allocate" however big of a memory region you want and all will probably be okay. They couldn't really do that, so they were fighting to reclaim every empty block and gap there was and scale a program's space with how much is actually needed. There's a counterpoint to this, however, because some code is confused. Sometimes you see fixed-size buffers/arrays pop up, seemingly out of nowhere, and the program bail out if they fill. I'm not sure why, but perhaps they didn't deem it worthwhile the complexity to use
malloc()
in those cases and/or "don't fix it if it ain't broke" kind-of thing.
And clearly,
when
they could get away with simple linear allocation, they did so.
Thirdly, I suspect they had grown tired of doing
any
manual memory management at all. Dropping explicit memory regions from the
malloc()
interface is a change that, on the surface, simplifies stuff even further. You make the implementation-side smarter and you don't have to worry about passing
coremap
or
swapmap
or
anymap
. Remember: the way programs and problems were approached didn't lead to complex object lifetimes. And when you had to deal with tangled lifetimes,
malloc()
/
free()
appear deceivably simple in the beginning.
That last point seems more "vibes" than evidence, but there is some evidence actually. In V7 they added
galloc(3)
:
GALLOC(3) GALLOC(3)
NAME
galloc, gfree, garbage - storage allocation with garbage
collection
SYNOPSIS
char *galloc(n)
unsigned n;
gfree(p)
char *p;
garbage( )
DESCRIPTION
These functions perform heap storage allocation with garbage
collection.
Galloc allocates a block of at least n bytes and returns a
pointer to it. Gfree frees a block previously allocated by
galloc.
When space gets tight, garbage blocks are freed automati-
cally. A block allocated by galloc is deemed to be garbage
unless it is reachable. A reachable block is one whose
first byte is pointed to by a declared C variable or by a
pointer in a reachable block.
Ha, myth busted! The authors and people closest to C
wanted garbage collection
. Given how fast computers are -- and the pace of that -- this makes sense. It's even clearer if you look at what they did later with Limbo and Go. They wanted a higher level general purpose language to write programs in. They had C, so they tried to bolt GC on top of that.
Lastly, I should note that many of these problems were largely known, even at the time. In
The Standard C library (1992)
, P. J. Plauger writes:
Allocating and freeing data objects on the heap in arbitrary order inevitably leaves unusable holes between some of the allocated data objects. That too lowers the usable size of the heap.
Don't overreact to this knowledge. Gather related data into a structure and allocate it all at once. That minimizes heap overhead, to be sure, but it is also good programming style. Do not gather unrelated data just to save heap overhead. Similarly, allocate data objects with similar lifetimes all at once, then free them at about the same time. That minimizes heap fragmentation, but it too is good style. Do not advance or defer unrelated heap operations just to minimize fragmentation. The storage allocation functions are an important aid to programming flexibility. Use them as they are intended to be used.
Criticism by yours truly: It's much easier to say "just do the right thing™" than show and provide tools for how to actually do it.
Conclusion
I don't know. Pay attention to good ideas, otherwise they might get lost? (literally in the case of V4, lol) Sounds cheesy.
Also, the "C object code improver" shatters the illusion that C was somehow magically fast. In fact, Ritchie later added an expression optimiser, and they kept extending both in each edition. I think C just wasn't slow, and it was made fast only with engineering effort put into the compiler (and even then, they had to add
register
because the compiler wasn't that smart either). It probably remained fast because Unix spread quickly and it carried C along with it, so hardware manufacturers adapted
to C
. There's a nice essay by
Mike Pall
discussing how modern CPUs try very hard to infer invariants from compiled C code (which the compiler tried very hard to optimise) and the costs associated with that.
At one point, I also want to investigate Inferno and Plan 9 more. Maybe the BSDs too. Until then...
Cya!
~/V8$ grep -Rn <redacted> .
./usr/src/cmd/c2/c21.c:660: if (p->op==ASH && preg==(regs+RT1+1)) cp2[-1]=BYTE; /* stupid DEC */
./usr/src/cmd/c2/c22.c:11:/* cursed be the preprocessor, whose impotence and stupidity
./usr/src/cmd/adb/main.c:16:int xargc; /* bullshit */
./usr/src/cmd/ccom/vax/gencode.c:644: case LS: /* stupid vax */
./usr/src/cmd/ccom/vax/gencode.c:664: if(incrsize(p) != 4) { /* stupid vax */
./usr/src/cmd/ccom/vax/gencode.c:705: * instead, generate crummy code (stupid assembler)
./usr/src/cmd/ccom/vax/genaux.c:598: op->in.op -= (ASG 0); /* crap for crap */
./usr/src/cmd/ex/ex.c:352: * Don't have to be upward compatible with stupidity
./usr/src/cmd/ex/exrecover.c:492: * This is old and stupid code, which
./usr/src/cmd/ex/ex_put.c:965: * This is so idiots who make esc their start/stop don't lose.
./usr/src/cmd/upas/send/send.c:288: /* check forwarding (what anti-social crap!) */
./usr/src/cmd/uucp/Makefile:2:# this makefile is full of crap
./usr/src/cmd/rarepl/rct.h:2: * first block of rct: random crap
./usr/src/cmd/sdb/machine.h:16:struct {short hiword; short loword;}; /* stupid fp-11 */
./usr/src/libc/gen/regcomp.c:26:/* predeclared crap */
./usr/sys/dev/uda.c:951: register u_short *sp; /* for sdi crap */
./usr/sys/dev/uda.c:1123: * should do read-after-write crap
./usr/sys/dev/flp.c:206: * We're really stupid today - call it an
./usr/sys/dev/kmc.c:9:/* nonstandard crap */
./usr/sys/dev/vaxcpu.c:36: * Information to patch around the stupidity of configuration
./usr/sys/sys/sys1.c:488: else /* stupid 1k 0413 files */
./usr/sys/sys/vmproc.c:238: * when large procs stupidly fork() instead of vfork().
./usr/sys/boot/bb/bblock.s:117: movl r5,r11 # put boot flags in stupid bky place
./usr/sys/boot/bb/bblock.s:118: movzbl $DEVNUM,r10 # more berkeley bullshit - 7 == uda50
./usr/sys/boot/bb/1k.s:2: .set DEVNUM, 8 # stupid hack
(Kindly let me know if I made some mistake :)
Tyrtulia, a headless forum with no passwords, made for the terminal
A headless forum for small communities on shared Unix servers (tildes, pubnixes, club servers). It needs very few resources, between 60 and 70 MB of RAM and about 50 MB of disk space.
You need a client to take part: the terminal client
Tyrtulia TUI
or the Emacs client
tyrtulia.el
. Anyone can write their own client.
It is a modern, clean adaptation of
BBJ (Bulletin Butter & Jelly)
, a textboard for tilde communities. It keeps BBJ's ideas (a small server that is easy to run, several text clients against one API, reading without an account, validation rules shared with clients), rebuilds its foundations on current standards and adds what BBJ was missing: notifications, real time, edit history, search and moderation. See the
comparison with BBJ
.
Hypermedia REST API (HAL): clients only know
GET /
and navigate by following links.
Passwordless authentication: Unix socket for local users, signed SSH challenge for remote access.
Threads with linear replies and navigable conversations (
replyTo
/
replies
).
Standard Markdown, with sanitized HTML returned alongside the source.
Mentions, notifications and thread subscriptions.
Full edit history with diffs and revert.
Pinned and closed threads; soft deletion of threads and posts.
Server-side full-text search, accent and case insensitive.
Atom feeds for the forum and for each thread.
Real-time notifications over Server-Sent Events (SSE).
Moderation: roles, read-only users, bans, hidden posts, public moderation log and reports.
Rate limiting per user and per IP.
Discoverable validation through JSON Schema.
Plain-text configuration (TOML), no admin endpoints.
Out of scope on purpose: images, attachments, categories, anonymous posting.
Other uses
Tyrtulia is a store of threaded Markdown documents with public history, search, notifications and feeds, where every process on the server is already identified as its Unix user. That makes it useful beyond a forum.
Wiki and documentation.
Community guides, FAQs or decision records, each one a thread whose first post is the page: public history with diffs and revert, full-text search, and the replies as its discussion page. See
Threads as wiki pages
.
Shared announcements and notifications.
Administrators post maintenance windows and changes in a pinned, closed thread; whoever watches it gets them in real time. Scripts, cron jobs and hooks can post through the socket with no credentials, and a mention (
@alice
) notifies someone directly: a persistent, searchable
wall
. A login script can show unread notifications the same way. Automated posts need a regular account (UID at or above
[accounts].min_uid
) or an SSH key account, since system accounts are rejected.
Group blog.
Each thread is an article and its replies are the comments. The forum's Atom feed is the subscription, and
Tyrtulia Web
can publish it on the web. Only people with an account can comment.
Teaching platform.
On a school or university server, students already have Unix accounts, so nobody signs up: the teacher moderates, each exercise or question is a thread, and the search keeps past answers at hand. Since the product is a hypermedia API discovered from
GET /
, writing a client is also a good exercise in REST and HAL, in any language, against the same instance.
Everything is public to anyone who can read the instance: there are no private messages, so do not use it for secrets or personal data. The index has no categories; for unrelated purposes, run one instance each, which is cheap.
Requirements
Linux or a BSD system, such as FreeBSD or OpenBSD (the Unix socket identity relies on
SO_PEERCRED
on Linux and
getpeereid()
on BSD).
Python 3.13 or newer (the system Python of Debian 13 and Ubuntu 26.04 works).
SQLite 3 with FTS5.
OpenSSH client (
ssh-keygen
), used to verify signatures.
Installation
Tyrtulia is published on PyPI as
tyrtulia
. It installs the
tyrtulia
command (the server and the administration CLI). Install it in its own environment, owned by root and readable by the service user.
Install with
"tyrtulia<2"
.
A 2.0.0 was published by mistake (the same code as 1.2.0) and cannot be withdrawn from PyPI yet.
<2
always gets the newest 1.x and never that release. The quotes keep the shell from reading
<
as a redirection.
# The interpreter goes to /opt/python so the service user can read itUV_PYTHON_INSTALL_DIR=/opt/python uv venv --python 3.13 /opt/tyrtulia
uv pip install --python /opt/tyrtulia/bin/python "tyrtulia<2"
ln -s /opt/tyrtulia/bin/tyrtulia /usr/local/bin/tyrtulia
To have a look without installing anything:
uvx --from "tyrtulia<2" tyrtulia --help
.
Some dependencies are compiled when there is no binary wheel for the system:
FreeBSD:
pkg install python314 py314-sqlite3 rust libev libuv c-ares
, and set
GEVENTSETUP_EMBED=0
while installing so gevent uses the system libraries.
OpenBSD:
pkg_add python rust
.
To upgrade, install the new version over the same environment (
uv pip install --python /opt/tyrtulia/bin/python --upgrade "tyrtulia<2"
or
/opt/tyrtulia/bin/pip install --upgrade "tyrtulia<2"
), run
tyrtulia migrate
as the service user and restart the service.
Getting started for administrators
A first instance on Linux with systemd, step by step. Run the commands as root. BSD notes follow each step.
BSD:
pw useradd tyrtulia -d /nonexistent -s /usr/sbin/nologin
on FreeBSD,
useradd -d /var/empty -s /sbin/nologin _tyrtulia
on OpenBSD; the database goes to
/var/db/tyrtulia
.
Write the configuration.
List yourself in
admins
with your Unix name. Every key is described in
Configuration
.
cat > /etc/tyrtulia/config.toml <<'EOF'
name = "Forum of example.org"
admins = ["alice"]
moderators = []
EOF
chown tyrtulia /etc/tyrtulia/config.toml
BSD: also set
[listen].socket = "/var/run/tyrtulia/tyrtulia.sock"
and
[storage].database = "/var/db/tyrtulia/tyrtulia.db"
. On FreeBSD the file usually lives in
/usr/local/etc/tyrtulia/config.toml
.
FreeBSD: download
https://git.andros.dev/andros/tyrtulia/raw/branch/main/contrib/freebsd/tyrtulia
to
/usr/local/etc/rc.d/tyrtulia
(mode
0555
), then
sysrc tyrtulia_enable=YES
and
service tyrtulia start
. OpenBSD: download
https://git.andros.dev/andros/tyrtulia/raw/branch/main/contrib/openbsd/tyrtulia
to
/etc/rc.d/tyrtulia
(mode
0555
), then
rcctl enable tyrtulia
and
rcctl start tyrtulia
.
Check it as a normal user.
The first request through the socket creates your account, with the
admin
role because you are listed in
admins
:
Every local user can do the same: having a Unix account on the server is the invitation. With
Tyrtulia TUI
installed,
tyrtulia-tui
opens the forum straight away.
Open it to the outside (optional).
Enable HTTP on the loopback and put a web server in front of it with TLS. Event streams need buffering off.
Change admins, moderators or limits in the configuration and apply it without downtime:
systemctl reload tyrtulia
(
service tyrtulia reload
or
rcctl reload tyrtulia
on BSD).
Logs go to the journal (
journalctl -u tyrtulia
) or to syslog on BSD.
Any program that speaks HTTP can be a client: it only needs
GET /
and the links in the
API reference
.
Design principles
Modern standards are preferred over custom formats: HAL, RFC 9457, RFC 7396, RFC 6570, JSON Schema, SSE, UUIDv7 and ISO 8601.
Topic
Decision
Discovery
Only
GET /
is known. Everything else is discovered through
_links
, using
href
,
method
, and optionally
templated
(RFC 6570),
type
and
schema
.
_embedded
holds embedded resources.
Permissions
Expressed as links. If you can edit, an
edit
link appears; if a thread is closed,
reply
disappears. Links that do not apply are omitted, never
null
. Clients never check roles.
Methods and status codes
Real HTTP methods (
GET
,
POST
,
PATCH
,
PUT
,
DELETE
), correct status codes and
Location
on creation.
Partial updates
PATCH
with
application/merge-patch+json
(RFC 7396).
Errors
RFC 9457,
application/problem+json
, with a catalog of
type
values and useful links inside the error (for example, the
diff
in an edit conflict).
title
and
detail
are always in English.
Media type
Responses are
application/vnd.tyrtulia.v1+json
. Clients may send
Accept
with that type,
application/json
or
*/*
; anything else returns
406 not-acceptable
.
Pagination
Cursor based: clients follow
next
and
prev
, and never build cursors (they are opaque). Posts are paginated by number (
after
,
before
), so a client can ask only for what is new since the last post it saw.
perPage
defaults to 20, maximum 100.
Concurrency
Each post has a strong
ETag
per version (
"v2"
). A thread's
ETag
is the version of its first post, because the title is versioned with it. Editing a post body or a thread title requires
If-Match
; pinning or closing does not. Missing header:
428
. Stale:
412
.
Conventions
JSON in
camelCase
, dates in ISO 8601 UTC, identifiers are UUIDv7 and opaque to clients.
postNumber
and
version
are positions, not identifiers, and stay integers. Posts are numbered from 1.
Users in responses
The author is embedded in each resource (
_embedded.author
).
Contract
The contract is the link relations (
rel
) and the representations. Paths are an implementation detail.
Authentication and accounts
There are no passwords. There are two ways in:
Method
For
How
Unix socket
Users of the server itself
The server also listens on a Unix socket, and the operating system tells it which Unix user is connecting (
SO_PEERCRED
on Linux,
getpeereid()
on BSD). No token needed.
SSH key
Accounts created by the CLI, and Unix accounts connecting from outside
A login message that names the instance, the user and a challenge, signed with the private key (standard OpenSSH signature,
ssh-keygen -Y sign
, namespace
tyrtulia
). Returns a session token.
The socket speaks exactly the same HTTP API: a resource does not depend on the transport.
The socket identity is guaranteed by the operating system and cannot be spoofed. System accounts (UID below
min_uid
, 1000 by default) and the users listed in
excluded_users
(
nobody
by default, whose UID changes between systems) are excluded. If the server runs in a container that does not share the host users, the socket is not available and SSH is used.
What is signed:
the
message
returned with each challenge, never the bare challenge:
tyrtulia login
origin: https://forum.example.org
username: alice
challenge: 3q2-7wXbG0yTuFq8aN1sKkz9Jm5vQe4Lr6Hc2Pd0Tbw
The origin is
[listen].public_url
or, when it is empty, the scheme and host of the request. Clients check that it is the instance they are talking to before signing, so a signature made for one instance (perhaps a malicious one) is useless on any other, and it is only valid for that user and that challenge.
SSH challenges
expire after 2 minutes and are single use. The response is the same whether the user exists or not, and a bad signature returns the same message in both cases, so account names are not leaked.
Accepted keys:
ssh-ed25519
,
ecdsa-sha2-*
and
rsa-sha2-*
of 3072 bits or more. Each user manages their own keys (list, add, delete, with fingerprint and last use). On a shared server, a client can import the keys the user already has authorized for SSH.
Session tokens
are the only kind of token. They expire after 30 days without use or on logout. The server only stores their hash. They carry the
tyrtulia_ses_
prefix so secret scanners can detect them.
Banning
revokes the user's sessions and blocks their SSH keys: they cannot log in (
403 banned
), and any socket request receives
403 banned
.
Account types
There is no HTTP registration. The Unix user is the source of truth. There are exactly two kinds of account:
Unix account:
created automatically on a Unix user's first socket request of any kind (a
GET
is enough), and bound to the pair UID + Unix name.
SSH key account:
created by the operator with the CLI, together with its SSH key, for people without a user on the server. The CLI rejects any name that already exists as a Unix user.
Rules:
An SSH key account can never be used through the socket.
A Unix account can register SSH keys (from the socket) to connect from outside.
Unix identity:
on every socket request, the UID and the name must both match the account. If they do not (a UID reused after deleting a user, or a
usermod -l
), the request is rejected with
409 unix-identity-mismatch
and the operator resolves it with the CLI.
Deactivated accounts:
when a Unix user no longer exists, its account is deactivated: its sessions are revoked and it cannot log in, not even with SSH keys. Its posts remain, its profile still loads, and the user appears with
deactivated: true
.
Name conflicts:
if a Unix user is created later with the same name as an SSH key account, the socket rejects it (
409 username-conflict
) until the operator renames the SSH key account with the CLI. Accounts are never merged and posts are never reassigned.
Names used before:
a name an account gave up (by a rename) stays reserved, so a new Unix user with that name also gets
409 username-conflict
. Roles in the configuration go by name, and this keeps them from passing to someone else. When the operator resolves a conflict with
identity resolve --rename-to
, the name is released for the Unix user, unless it is listed in
admins
or
moderators
: then the configuration has to change first.
On a tilde server, having a Unix account is the invitation.
Profile
Profiles are minimal:
about
: free text in Markdown, public. Returned with its sanitized HTML (
aboutHtml
).
email
: optional and unverified. Only visible to the user and to moderators, so moderation can contact them. Tyrtulia never sends email.
username
: can only be changed on SSH key accounts, and only to a name that no account uses or has ever used, that does not exist as a Unix user and that is not listed in
admins
or
moderators
(
409 username-taken
). A name used once is reserved forever, so a role from the configuration file never passes to someone else through a rename. After a rename, the old name returns
404
. On Unix accounts the username is always the Unix name. Mentions are stored by user id, so they keep working after a rename.
Roles
Role
Can
Assigned by
admin
Everything a moderator can. Runs the instance (configuration file and CLI).
Configuration file (
admins
)
moderator
Edit, revert, hide, unhide and delete any post; edit, pin, close and delete threads; make users read-only; ban.
Configuration file (
moderators
)
user
Read and participate.
Default
readonly
Read only: just
GET
requests (notifications and SSE included) and logging out. Cannot interact in any way: post, edit, subscribe, report, mark notifications, change their profile or keys. All of that returns
403 readonly
. Their notifications have no read state and no counter.
A moderator, through the API
banned
Nothing. Every authenticated request returns
403 banned
, including reads and socket requests.
A moderator, through the API
Moderators act on any post, including posts by other moderators and admins. Only roles are protected: moderators cannot change the role of admins or other moderators (
403 cannot-moderate-staff
).
Unbanning restores the previous role (
user
or
readonly
) and reactivates the SSH keys.
Anyone can read without authenticating; writing requires an account with the
user
,
moderator
or
admin
role.
Configuration
There are no administration endpoints. Whoever runs the server edits a plain-text TOML file with the admins, the moderators and the whole instance configuration. Access control is the file system's.
The file is read from
/etc/tyrtulia/config.toml
, unless
--config
or the
TYRTULIA_CONFIG
environment variable point elsewhere.
name="Forum of example.org"admins=["alice","bob"]moderators=["carol"][listen]socket="/run/tyrtulia/tyrtulia.sock"# enabled by defaulthttp=""# disabled; e.g. "127.0.0.1:7099"trusted_proxies=["127.0.0.1"]# real IP from X-Forwarded-For only from herepublic_url=""# e.g. "https://forum.example.org"; absolute URLs use it[storage]database="/var/lib/tyrtulia/tyrtulia.db"[accounts]min_uid=1000# below this, system accounts: excluded from the socketexcluded_users=["nobody"]# never identified through the socket[rate_limit]read_per_minute=300write_per_minute=20login_per_15_minutes=10max_sse_per_user=5
The Unix socket is enabled by default and HTTP is disabled. HTTP is only needed for access from outside the server, usually behind a web server.
public_url
is the public origin of the instance. Absolute URLs (error
type
, feed ids and links) are built from it; when it is empty, from the
Host
of each request. Set it when HTTP is enabled.
Changes apply on restart or on
SIGHUP
, without stopping the service.
Deployment
The service does not need root: it runs as a dedicated user that owns the configuration file and the database.
Reference: native with systemd (Linux).
Dedicated user;
RuntimeDirectory=tyrtulia
with
RuntimeDirectoryMode=0755
to create
/run/tyrtulia
without root;
StateDirectory=tyrtulia
for
/var/lib/tyrtulia
; a socket with
0666
permissions so any local user can connect. The unit is in
contrib/systemd/tyrtulia.service
.
Native with rc.d (FreeBSD and OpenBSD).
The scripts in
contrib/freebsd/tyrtulia
and
contrib/openbsd/tyrtulia
run the service as a dedicated user, create
/var/run/tyrtulia
with mode
0755
and start
tyrtulia serve --pidfile
, so
service tyrtulia reload
or
rcctl reload tyrtulia
sends
SIGHUP
. Set
[listen].socket = "/var/run/tyrtulia/tyrtulia.sock"
and keep the database under
/var/db/tyrtulia
. On FreeBSD, build gevent against the system libraries (
pkg install libev libuv c-ares
and
GEVENTSETUP_EMBED=0
).
The native modes are the only ones with Unix socket access.
Private data on disk.
The database holds emails, hidden and deleted posts, history and reports. Tyrtulia creates it with mode
0600
(and fixes the mode of an existing one); the systemd unit adds
StateDirectoryMode=0700
and
UMask=0077
, and the rc.d scripts run with
umask 077
.
HTTP on a shared server.
Local users can connect to
[listen].http
on
127.0.0.1
too, and from there
X-Forwarded-For
would be believed. Let only the web server's user reach that port, for example with
iptables -A OUTPUT -o lo -p tcp --dport 7099 -m owner ! --uid-owner www-data -j REJECT
on Linux or
block return out quick on lo0 proto tcp to port 7099 user != www
with pf on the BSDs.
compose.yaml
:
for a quick start, or for instances that only use SSH keys. Inside a container the host users are not visible, so the Unix socket is not available and
[listen].http
must be enabled.
Backups
Tyrtulia has no backup command: backups are manual. The database is a single SQLite file and can be copied safely while the service runs:
Importing from BBJ.
Each BBJ user becomes an SSH key account without keys: they cannot log in until one is added with
account add-key
. If a BBJ username matches an existing Unix user, the import stops and lists the conflicts so the operator can decide case by case, because BBJ registration was open and the same name does not guarantee the same person. With
--trust-unix-names
, those users' posts go to the matching Unix accounts instead, for instances where the original BBJ was only used by the server's users. Anonymous posts are assigned to a fixed
anonymous
account, which can never log in and whose name is reserved. Identifiers are regenerated as UUIDv7, and BBJ's bracket format is converted to Markdown where possible.
The import is all or nothing. Before writing anything it checks every BBJ username and stops, listing all the conflicts at once, when a name:
Conflict
Why
How to resolve it
Matches an existing Unix user
The same name does not guarantee the same person.
Rerun with
--trust-unix-names
if the BBJ users were the server's users.
Is not a valid Tyrtulia username (
^[a-z_][a-z0-9_-]{0,31}$
, for example uppercase letters or spaces)
Names are never changed silently.
Rename the user in a copy of the BBJ database and import that copy.
Is already used by a Tyrtulia account, or was used before (names are reserved forever)
Accounts are never merged and posts are never reassigned.
Rename the user in a copy of the BBJ database, or rename the existing SSH key account with
account rename
.
--trust-unix-names
only resolves the first case. Nothing is imported until the list of conflicts is empty.
Behaviour
Posts and conversations
Markdown:
CommonMark plus GFM tables, strikethrough and autolinks. The source is stored, and sanitized HTML (
bodyHtml
) is always returned as well. Mentions are rendered as links to the user.
Linear replies
that can point to the post they answer (
replyToPostNumber
), as in Discourse.
Navigable conversations.
Storage and the default view are linear, which keeps pagination by
postNumber
. On top of that, each post exposes the conversation:
replyTo
links to the post it answers, and
replies
links to the posts that answer it. Clients walk the conversation in both directions by following links.
repliesCount
, to show "3 replies" without another request.
Optional
?view=tree
on a thread's posts to receive the conversation already nested: up to 8 levels deep (deeper answers stay at the eighth level) and at most 500 replies per page; the linear view shows every post.
Read state is a client concern.
The server does not track what each user has read in threads. The client remembers the last
postNumber
it saw in each thread, compares it with
highestPostNumber
and asks for new posts with
?after=
.
Mentions, notifications and subscriptions
Mentions
with
@username
, outside code and block quotes. At most 10 per post; more is rejected with
422 too-many-mentions
. Editing only notifies new mentions.
Notifications
are an inbox with the kinds
reply
,
mention
,
watched
and
report
(the last one only for moderators and admins). There is no
quote
kind: to refer to a post you reply to it, which produces a
reply
notification.
One notification per post and user.
When a post matches several kinds, the highest priority wins:
mention
>
reply
>
watched
.
Who is notified:
never yourself. Read-only users are notified (without read state). Banned and deactivated users are never notified.
Thread subscriptions
have three levels. Creating a thread sets you to
watching
; replying does not change your level; the default is
normal
.
watching
: a
watched
notification for every new post, plus replies and mentions.
normal
: only replies to your posts and mentions.
muted
: nothing from that thread, not even mentions.
Notification excerpts are computed when read, and are empty if the post has been hidden or deleted.
Notifications are kept indefinitely.
Editing and deletion
Edit history
with diffs and revert. Reverting creates a new version and never rewrites history. A thread's title is versioned with its first post: those revisions include
title
, and reverting the first post also restores the title.
History is public, except for deleted posts (nobody sees it) and hidden posts (only moderators and the author).
No edit time limit:
the public history makes it unnecessary.
Pinned threads:
only moderators pin; pinned threads come first in the index, then the rest by creation date, newest first.
Closed threads:
only moderators can reply.
Soft delete:
a deleted post stays as a tombstone (number, author and dates, no body) so numbering is not broken.
Deleting a thread:
the author can only delete it if nobody else has replied (their own replies do not count). With replies, only a moderator can delete it; the author can still delete their first post, which becomes a tombstone, while the thread and its replies remain. A deleted thread returns
404
to everyone except moderators.
Search and feeds
Full-text search
on the server over titles and bodies, accent and case insensitive. Query syntax: words,
"exact phrase"
,
OR
,
NOT
and
prefix*
; anything else returns
400 invalid-query
. Each result has a
highlight
: an HTML snippet with the matches in
<mark>
, safe to insert.
Atom feeds
for the forum and for each thread, discoverable through the
alternate
relation. Entry ids are
urn:uuid:<id>
and the content is the sanitized HTML.
Neither feeds nor search include hidden or deleted content.
Real time
One SSE stream per user
, carrying only their notifications (what you watch arrives as
watched
). There are no per-thread or public streams.
Several connections per user are allowed (several clients at once), up to
max_sse_per_user
.
Events carry the minimum plus links to the resource (see
the example
).
Resume with
Last-Event-ID
for 24 hours; if the id is gone, the server sends
event: reset
. Heartbeat every 15 seconds.
Moderation
Hidden posts
carry a reason. The body and the reason are only visible to moderators and to the author; everyone else sees that the post is hidden.
Public moderation log
with action, reason, moderator and target. Every moderator action is logged, except a moderator editing, reverting or deleting their own posts or threads (that is ordinary authoring).
Reports:
a user sends an internal message to the moderators about a post, a thread or a user. It arrives as a
report
notification to every moderator and admin (inbox and SSE). Reports are not public and do not appear in the moderation log.
Rate limiting:
GET
and
HEAD
count as reads; everything else counts as a write. An open SSE connection counts as one read.
POST /ssh-challenges
and
POST /sessions
also count in the login limit, per IP.
Through the socket, limits apply per user only; over HTTP, per user and per IP. The real IP is taken from
X-Forwarded-For
only when the request comes from
trusted_proxies
.
Responses include the
RateLimit
and
RateLimit-Policy
headers, and
429
comes with
Retry-After
.
Threads as wiki pages
A thread also works as a collaborative wiki page. The first post is the page: its author and the moderators keep it up to date, with no edit time limit, and every change stays in the public history with diffs and revert. Pin it so it stays at the top of the index, close it to keep it clean, or leave it open so the replies act as its discussion page. Like any thread, it is found by the full-text search.
Validation
Every link that sends a body carries a
schema
property pointing to a JSON Schema (draft 2020-12) with the rules: required fields, lengths, formats and allowed values. Clients validate before sending without hard-coding the rules, and the server applies exactly the same ones (
400 validation-failed
otherwise).
Field
Rule
username
^[a-z_][a-z0-9_-]{0,31}$
Thread
title
1 to 200 characters
Post
body
1 to 32,000 characters
about
Up to 1,000 characters
reason
(edit, hide, ban, read-only)
Up to 500 characters
Report
message
1 to 2,000 characters
SSH key
title
Up to 100 characters
email
Email format, up to 254 characters
perPage
1 to 100, default 20
Text (title, body, about, reasons, report message, SSH key title)
No control characters except tab and line breaks, and no bidirectional overrides (
U+202A
–
U+202E
,
U+2066
–
U+2069
): they would drive the terminals of readers
Examples use these identifiers and users: thread
019a2f4e-8b1c-7d2e-9f30-4a5b6c7d8e9f
, post #4
019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e
;
alice
opened the thread,
bob
wrote post #4,
carol
is a moderator.
"Can write" below means an authenticated user with the
user
,
moderator
or
admin
role. Every link that changes something also requires being able to write: a read-only author, for example, gets no
edit
link.
Media type of the target, when it is not the API's own (
text/event-stream
,
application/atom+xml
,
text/x-diff
).
schema
No
JSON Schema for the body the link expects.
Root
Authenticated as
alice
, through the socket:
{"name":"Forum of example.org","_links":{"self":{"href":"/","method":"GET"},"latest":{"href":"/threads","method":"GET"},"watching":{"href":"/threads?filter=watching","method":"GET"},"createThread":{"href":"/threads","method":"POST","schema":"/schemas/thread"},"search":{"href":"/search{?q,author,in}","method":"GET","templated":true},"user":{"href":"/users/{username}","method":"GET","templated":true},"me":{"href":"/users/me","method":"GET"},"notifications":{"href":"/notifications","method":"GET"},"events":{"href":"/events","method":"GET","type":"text/event-stream"},"modlog":{"href":"/modlog","method":"GET"},"alternate":{"href":"/threads.atom","method":"GET","type":"application/atom+xml"}}}
{"username":"bob","about":"Sysadmin. I like **SSE**.","aboutHtml":"<p>Sysadmin. I like <strong>SSE</strong>.</p>","role":"user","deactivated":false,"postsCount":42,"threadsCount":5,"createdAt":"2026-03-02T10:15:00Z","_links":{"self":{"href":"/users/bob","method":"GET"},"threads":{"href":"/threads?author=bob","method":"GET"},"report":{"href":"/reports","method":"POST","schema":"/schemas/report"}}}
rel
Shown when
self
,
threads
Always
report
Can write, and it is not yourself
ban
or
unban
,
readonly
or
unreadonly
Moderator, and the user is not staff
CurrentUser
The
User
of the authenticated user, plus private fields and links.
{"username":"bob","about":"Sysadmin. I like **SSE**.","aboutHtml":"<p>Sysadmin. I like <strong>SSE</strong>.</p>","email":"bob@example.org","role":"user","deactivated":false,"postsCount":42,"threadsCount":5,"unreadNotificationsCount":3,"createdAt":"2026-03-02T10:15:00Z","_links":{"self":{"href":"/users/me","method":"GET"},"threads":{"href":"/threads?author=bob","method":"GET"},"edit":{"href":"/users/me","method":"PATCH","schema":"/schemas/profile"},"sshKeys":{"href":"/users/me/ssh-keys","method":"GET"},"notifications":{"href":"/notifications","method":"GET"},"events":{"href":"/events","method":"GET","type":"text/event-stream"},"logout":{"href":"/sessions/current","method":"DELETE"}}}
rel
Shown when
self
,
threads
,
sshKeys
,
notifications
,
events
Always
edit
Can write
logout
Authenticated with a session token (not through the socket)
unreadNotificationsCount
is absent for read-only users.
Response header:
ETag: "v2"
(version of the first post).
subscription
only for authenticated users.
deletedAt
only for moderators, on deleted threads.
In collections, threads come without
_embedded.firstPost
.
rel
Shown when
self
,
posts
,
alternate
Always
reply
Can write, and the thread is open (or you are a moderator)
edit
Author or moderator
pin
,
close
Moderator (same
href
and method as
edit
, schema
thread-update
)
delete
Author while nobody else has replied, or moderator
subscribe
Can write
report
Can write, and it is not your thread
Post
As seen by
bob
(the author):
{"id":"019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e","postNumber":4,"replyToPostNumber":2,"body":"Add `X-Accel-Buffering: no` to the response. @alice","bodyHtml":"<p>Add <code>X-Accel-Buffering: no</code> to the response. <a href=\"/users/alice\">@alice</a></p>","version":2,"repliesCount":1,"hidden":false,"createdAt":"2026-10-17T18:30:00Z","editedAt":"2026-10-17T18:50:00Z","_embedded":{"author":{"username":"bob","deactivated":false,"_links":{"self":{"href":"/users/bob","method":"GET"}}}},"_links":{"self":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e","method":"GET"},"up":{"href":"/threads/019a2f4e-8b1c-7d2e-9f30-4a5b6c7d8e9f","method":"GET"},"replyTo":{"href":"/posts/019a2f4f-1c2d-7e3f-9a4b-5c6d7e8f9a0b","method":"GET"},"replies":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e/replies","method":"GET"},"version-history":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e/revisions","method":"GET"},"edit":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e","method":"PATCH","schema":"/schemas/post-update"},"delete":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e","method":"DELETE"},"report":{"href":"/reports","method":"POST","schema":"/schemas/report"}}}
Response header:
ETag: "v2"
.
editedAt
is absent if the post was never edited;
replyToPostNumber
and
replyTo
are absent if it answers no post.
Hidden:
everyone sees
hidden: true
; only moderators and the author also receive
body
,
bodyHtml
and
hiddenReason
.
Deleted (tombstone):
only
id
,
postNumber
,
createdAt
,
deletedAt
,
_embedded.author
and the
self
and
up
links.
With
view=tree
, each post carries its replies in
_embedded.replies
.
rel
Shown when
self
,
up
,
replies
Always
replyTo
The post answers another post
version-history
Not deleted; if hidden, only to moderators and the author
In the
version-history
collection, revisions come without
body
.
revert
is shown to the author and moderators, on any version except the current one.
Notification
{"id":"019a2f60-11bb-7e4c-9a5d-6b7c8d9e0f1a","kind":"mention","read":false,"excerpt":"Add X-Accel-Buffering: no to the response. @alice","createdAt":"2026-10-17T18:30:00Z","_embedded":{"actor":{"username":"bob","deactivated":false,"_links":{"self":{"href":"/users/bob","method":"GET"}}},"thread":{"id":"019a2f4e-8b1c-7d2e-9f30-4a5b6c7d8e9f","title":"SSE behind Nginx?","_links":{"self":{"href":"/threads/019a2f4e-8b1c-7d2e-9f30-4a5b6c7d8e9f","method":"GET"}}}},"_links":{"self":{"href":"/notifications/019a2f60-11bb-7e4c-9a5d-6b7c8d9e0f1a","method":"GET"},"target":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e","method":"GET"},"markRead":{"href":"/notifications/019a2f60-11bb-7e4c-9a5d-6b7c8d9e0f1a","method":"PATCH","schema":"/schemas/notification-update"}}}
_embedded.thread
is present when the target belongs to a thread (a report about a user has none).
{"highlight":"Add <mark>X-Accel-Buffering</mark>: no to the response.","_embedded":{"post":{"...":"Post"},"thread":{"...":"Thread"}}}
Error
{"type":"https://tyrtulia.example/errors/edit-conflict","title":"The post was modified by someone else","status":412,"detail":"Your copy is version 2, the current version is 3","instance":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e","_links":{"latest-version":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e","method":"GET"},"diff":{"href":"/posts/019a2f51-02aa-7c3b-8d4e-5f6a7b8c9d0e/revisions/3?format=diff&from=2","method":"GET","type":"text/x-diff"}}}
API reference
Paths are the planned implementation, not the contract: clients discover them by following
_links
from
GET /
. The contract is the relation (
rel
) listed with each endpoint.
Through the Unix socket no token is needed: replace
$API
with
http://localhost
and add
--unix-socket /run/tyrtulia/tyrtulia.sock
.
Access levels:
public
(unauthenticated),
user
(authenticated),
writer
(can write:
user
,
moderator
or
admin
role),
author
(the author, or a moderator),
moderator
.
author
and
moderator
also require being able to write.
Errors common to every endpoint, not repeated below:
400 validation-failed
,
401 unauthenticated
,
403 forbidden
,
403 readonly
(anything except
GET
and logout),
403 banned
,
404 not-found
,
406 not-acceptable
,
413 request-too-large
and
415 unsupported-media-type
(requests with a body),
429 rate-limited
.
Root and authentication
Root
Entry point. Returns the forum name and the links available to whoever asks.
Endpoint:
POST /ssh-challenges
· rel:
sshChallenge
· public
Parameters:
username
(body).
curl -X POST $API/ssh-challenges -H "$JSON" -d '{"username": "alice"}'
Response:
201
with
{"challenge", "message", "namespace": "tyrtulia", "expiresAt"}
.
message
is the text to sign (see
what is signed
). Same response whether the user exists or not. Expires in 2 minutes.
Log in
Sends the signed login message and receives a token.
Endpoint:
POST /sessions
· rel:
login
· public
Parameters:
username
,
challenge
,
signature
(body). The signature is of the challenge's
message
.
REPLY=$(curl -s -X POST $API/ssh-challenges -H "$JSON" -d '{"username": "alice"}')CHALLENGE=$(echo"$REPLY"| jq -r .challenge)MESSAGE=$(echo"$REPLY"| jq -r .message)echo"$MESSAGE"| grep -qx "origin: $API"||{echo"Not this instance: refusing to sign";exit 1;}SIG=$(printf %s "$MESSAGE"| ssh-keygen -Y sign -f ~/.ssh/id_ed25519 -n tyrtulia)
curl -X POST $API/sessions -H "$JSON"\
-d "$(jq -n --arg c "$CHALLENGE" --arg s "$SIG"'{username:"alice", challenge:$c, signature:$s}')"
Response:
201
with
{"token", "expiresAt"}
and
me
and
logout
links. Errors:
401 invalid-signature
,
403 banned
.
Creates a thread with its first post. The author is set to
watching
.
Endpoint:
POST /threads
· rel:
createThread
· writer
Parameters:
title
,
body
(body, Markdown).
curl -X POST $API/threads -H "Authorization: Bearer $TOKEN" -H "$JSON"\
-d '{"title": "SSE behind Nginx?", "body": "Does anyone have it working?"}'
Response:
201
with the
Thread
, including
_embedded.firstPost
, and a
Location
header. Error:
422 too-many-mentions
.
Get a thread
Endpoint:
GET /threads/{threadId}
· rel:
self
,
up
· public
Parameters:
threadId
(path).
curl $API/threads/$THREAD
Response:
200
with the
Thread
and its
ETag
. A deleted thread returns
404 not-found
to everyone except moderators, who receive it with
deletedAt
.
Edit a thread
Changes the title (author) or pins and closes it (moderator).
Endpoint:
PATCH /threads/{threadId}
· rel:
edit
,
pin
,
close
· author / moderator depending on the field
Parameters:
If-Match
(header, required when
title
changes),
title
(author; creates a revision of the first post),
reason
(optional, with
title
),
pinned
,
closed
(moderator) (body).
Linear view by default, or the nested conversation.
Endpoint:
GET /threads/{threadId}/posts
· rel:
posts
· public
Parameters:
after
,
before
(post number),
perPage
,
view=tree
(query).
curl "$API/threads/$THREAD/posts?after=9"
Response:
200
with a collection of
Post
in ascending order. With
view=tree
, pagination applies to root posts (those that do not answer another post) and each one includes its replies nested in
_embedded.replies
, up to 8 levels deep (deeper answers stay at the eighth level) and at most 500 replies per page.
Reply
Adds a post to the thread. Notifies mentioned users, the author of the answered post and whoever watches the thread.
Endpoint:
POST /threads/{threadId}/posts
· rel:
reply
· writer
Parameters:
body
(Markdown),
replyToPostNumber
(optional) (body).
Response:
200
with the
Post
and the new
ETag
. Errors:
428 precondition-required
,
412 edit-conflict
(with
latest-version
and
diff
links),
422 too-many-mentions
.
Delete a post
Soft delete: a tombstone remains. Deleting the first post does not delete the thread.
event: reset
if the
Last-Event-ID
is older than 24 hours: the client reloads through REST. Several connections per user, up to
max_sse_per_user
; beyond that,
429 too-many-connections
. The server also keeps at most 1,000 open streams in total.
Feeds
Forum feed
Endpoint:
GET /threads.atom
· rel:
alternate
(on the root) · public
Parameters:
none.
curl $API/threads.atom
Response:
200
application/atom+xml
with the last 30 threads created, newest first,
ETag
and
Last-Modified
. Each entry is the thread's first post: replies do not change it, and its
updated
is the last edit of that post. Excludes deleted threads.
Thread feed
Endpoint:
GET /threads/{threadId}.atom
· rel:
alternate
(on the thread) · public
Parameters:
threadId
(path).
curl $API/threads/$THREAD.atom
Response:
200
application/atom+xml
with the last 30 posts,
ETag
and
Last-Modified
. Excludes hidden and deleted posts.
Moderation
Ban a user
Sets the role to
banned
, revokes their sessions and blocks their SSH keys.
Endpoint:
PUT /users/{username}/ban
· rel:
ban
· moderator
Parameters:
reason
(body, required; public in the moderation log).
Response:
200
application/schema+json
with the JSON Schema.
Error catalog
All errors use
Content-Type: application/problem+json
(RFC 9457), with
type
under
https://<instance>/errors/
.
type
Status
When
validation-failed
400
The body or the query does not match the rules.
invalid-query
400
Invalid search syntax.
unauthenticated
401
Missing authentication or invalid token.
invalid-signature
401
Wrong SSH signature (same message whether the user exists or not).
forbidden
403
No permission for this action.
readonly
403
Any request other than
GET
or logout from a read-only user.
banned
403
Any request from a banned user.
fields-not-allowed
403
A field in the
PATCH
is not allowed for you; nothing is applied.
cannot-moderate-staff
403
Banning or making read-only a moderator or admin.
not-found
404
Does not exist, or exists but you cannot see it.
not-acceptable
406
Accept
asks for an unsupported media type or API version.
thread-closed
409
Replying to a closed thread without being a moderator.
thread-has-replies
409
The author tries to delete a thread with replies from other people.
key-already-registered
409
The SSH key is already registered.
username-conflict
409
New Unix user with the name of an SSH key account, or with a name another account used before.
username-taken
409
The username exists, existed, or is reserved in the configuration file.
unix-identity-mismatch
409
The UID and Unix name do not match the account.
edit-conflict
412
Stale
If-Match
.
request-too-large
413
The body is larger than 256 KiB.
unsupported-media-type
415
PATCH
without
application/merge-patch+json
, or a body that is not JSON.
too-many-mentions
422
More than 10 mentions in a post.
invalid-public-key
422
SSH key with an unsupported format or type.
invalid-target
422
The report target is not a valid post, thread or user.
precondition-required
428
Missing
If-Match
when editing.
rate-limited
429
Rate limit exceeded (with
Retry-After
).
too-many-connections
429
More open SSE connections than
max_sse_per_user
.
Comparison with BBJ
BBJ
(Bulletin Butter & Jelly, a pun on
bulletin board
and
PB&J
) is a textboard for tilde communities, written in Python with CherryPy and SQLite, with a terminal client (urwid), an Emacs client and a minimal web interface. Tyrtulia keeps its spirit and rebuilds its foundations. These tables are informative: they summarise what Tyrtulia brings compared to BBJ.
Foundations
Area
BBJ
Tyrtulia
API style
RPC: every call is
POST /api/<endpoint>
with a JSON body
Hypermedia REST (HAL): clients only know
GET /
and follow links
Permissions
Clients ask the server (
is_admin
,
edit_query
)
Expressed as links that appear or disappear
Authentication
Unsalted SHA-256 of the password sent on every request; the hash works as the password
No passwords: Unix socket (
SO_PEERCRED
or
getpeereid()
) and signed SSH challenge with revocable session tokens
Registration
Open, optionally with invitation codes
No HTTP registration: Unix users, or SSH key accounts created by the operator
Errors
6 custom numeric codes
RFC 9457 with a catalog of types
Users in responses
Separate
usermap
_embedded.author
Versioning
None
Versioned media type
Pagination
Everything returned at once
Cursor based, posts by number
Concurrency
Last edit wins silently
ETag
and
If-Match
Identifiers and dates
UUID1, floating point timestamps
UUIDv7, ISO 8601 UTC
Text format
Custom bracket language (
[bold: ...]
,
[red: ...]
)
Standard Markdown
Validation
db_validate
endpoint
JSON Schema linked from each form
Configuration
config.json
TOML
Features
Feature
BBJ
Tyrtulia
Anonymous reading
Yes
Yes
Anonymous posting
Yes
No
Reply to a specific post
>>N
quotes in the text
replyToPostNumber
, with
replyTo
/
replies
links and an optional tree view
Mentions
No
Yes
Notifications
No (pending since 2017)
Inbox with
reply
,
mention
,
watched
and
report
Thread subscriptions
No
watching
,
normal
,
muted
Real time
No (polling
message_feed
)
SSE with resume (
Last-Event-ID
)
Edit history
No: edits overwrite the post
Full history with diffs and revert
Edit time limit
24 hours
None (history makes it unnecessary)
Deletion
Deletes the row
Tombstone, numbering preserved
Pinned threads
Yes
Yes
Closed threads
No
Yes
Search
Done by each client over downloaded data
Server-side full text, accent and case insensitive
Feeds
No
Atom for the forum and each thread
Roles
Admin or not
admin
,
moderator
,
user
,
readonly
,
banned
Hidden posts
No
Yes, with a reason
Moderation log
No
Public
Reports
No
Internal message to moderators
Rate limiting
No
Per user and per IP
Profile
Bio, quip, color
About, private email
Import from BBJ
Not applicable
Administration CLI
What stays the same
A small server that is easy to run, built around SQLite.
Several text clients (terminal, Emacs) against one documented API.
Reading without an account.
Validation rules shared between server and clients.
The tilde and smolweb spirit: no images, no attachments, no categories.
Contributing
Contributions are welcome! Please see the
contribution guidelines
for instructions on how to submit issues or pull requests.
I like forums. Debates, questions, answers, opinions, etc. That feeling of an asynchronous community, where answers are thought through and content lasts, is what makes the internet an interesting place. That is why I like reading them and taking part in them. But I had never designed or built one. In my spare time I started sketching one, and I ended up building my humble
headless
, federated forum, with simple, modern technologies and open standards.
And I learned a lot along the way. A forum looks simple, but it hides the classic problems of any server. Who can do what? How do you tell a moderator from a regular user? What happens when you have to ban someone? How do you stop someone from stealing an account? Roles, permissions, bans and security. I had to solve them one by one, and you don't learn that by reading.
The server only offers a solid, documented API. It is REST with hypermedia: the client only knows one route,
GET /
, and discovers everything else by following links. Underneath there are some very boring standards:
RFC 9110: HTTP semantics, with
ETag
and
If-Match
so edits don't overwrite each other.
RFC 9457: errors.
RFC 7396: partial updates with
PATCH
.
RFC 6570: URL templates.
RFC 4287: Atom feeds.
RFC 9562: UUIDv7 identifiers.
JSON Schema: validation.
Server-Sent Events: real-time events.
There is no web interface, no forms, no advanced authentication systems, no file handling (images, attachments, etc.), and nothing that isn't strictly needed to read and write messages.
That is why it is so light. It uses between 60 and 70 MB of RAM and about 50 MB of disk. It only needs Python and SQLite to run.
An API that discovers itself
How does a client know what it can do? The server tells it with links. Look at the same request,
GET /
, made by a guest (simplified):
They are all fantastic, but they are designed to be used from a web browser. They also have an ambitious scope. That is ideal for large communities, but too much for a small, light forum.
That said, Tyrtulia doesn't give up on the browser: it can be indexed by search engines and anyone can read it on the web.
No passwords
No OAuth, no OpenID, no sign-up forms. There are two ways to authenticate:
Your Unix user
: the client talks to the forum through a Unix socket. The operating system tells the forum which user is on the other side. No sign-up, no login. Your user is your account.
SSH key
: the client connects over HTTP and the forum sends it a challenge. The client signs it with your private SSH key and the forum verifies it with your public key.
There is no registration over HTTP. If you have a Unix user on the server, you already have an account: it is created the first time you connect through the socket. From there you can add your SSH keys to log in from outside. If you don't have a user on the machine, the administrator creates an account for you with your public key from the command line. On a tilde, having a Unix user is already the invitation.
Easy to use, secure, and no passwords to steal.
What changes compared to BBJ
The idea isn't mine. I was inspired by
BBJ
, a bulletin board for the terminal built for small communities or clubs (tildes, pubnixes, club servers, etc.). I took its spirit and brought it to my own ground. Still, there are important differences.
First, the foundations.
BBJ
Tyrtulia
API style
RPC: everything is
POST /api/<endpoint>
REST with hypermedia (HAL): only
GET /
is known
Permissions
The client asks (
is_admin
,
edit_query
)
Links that appear or disappear
Authentication
Unsalted SHA-256 hash of the password on every request
No passwords: Unix socket or SSH-signed challenge
Errors
6 custom numeric codes
RFC 9457 with a catalog of types
Pagination
Everything at once
Cursor based
Concurrency
The last edit wins, silently
ETag
and
If-Match
Text format
Custom language (
[bold: ...]
,
[red: ...]
)
Standard Markdown
Validation
A
db_validate
endpoint
JSON Schema linked from each form
Configuration
config.json
TOML
Then, the features I missed.
BBJ
Tyrtulia
Reply to a specific post
>>N
quotes in the text
Linked replies, with a tree view
Mentions
No
Yes
Notifications
No (pending since 2017)
Inbox with replies, mentions and alerts
Real time
No, you have to poll
Server-Sent Events, with resume
Edit history
No, editing overwrites
Full, with diffs and revert
Deletion
Deletes the row
Tombstone: numbering is preserved
Search
Each client searches what it has downloaded
Server-side full text, accent and case insensitive
Feeds
No
Atom for the forum and each thread
Moderation
Admin or not
Roles, hidden posts, reports and a public log
Rate limiting
No
Per user and per IP
What they share is a small server built on SQLite and several text clients against a documented API.
Clients
A
headless
forum needs clients, no way around it.
Tyrtulia TUI
Tyrtulia TUI
is the terminal client, built with Textual.
tyrtulia.el
If you know me a little, you know this package was a must:
tyrtulia.el
.
Tyrtulia Web
Tyrtulia Web
is more a mirror of the content than a client. Its goal is to let search engines index the forum and let anyone read it from a browser.
It has an unstyled theme and another one that mimics my website.
Federation
You can connect to one forum or set up several. Federation happens in the clients: servers don't talk to each other or copy messages between them. Each forum is independent, with its own users, rules and moderators.
Instead of splitting them into tabs or accounts, the clients mix the threads of all of them into a single list.
Of course, you can only write in the forums where you have an account. In the rest you can only read.
I think it's a pretty cool feature, because it keeps the API code much simpler and lets you explore other forums at zero cost.
Try it
It has been fun to design and build. Of course there are more complete forums, more powerful federation systems, and even decentralized systems that leave Tyrtulia in the dust (Aether, Nostr...). But its niche is very specific: a light, closed-access, federated, indexable, secure forum with text clients. For everything else, there are better things out there.
It's been almost 15 years since BioWare co-founders Greg Zeschuk and Ray Muzyka
announced their retirements
from the videogame industry, and they've been resolute about staying out of it ever since: Muzyka continues to head up the
ThresholdImpact
investment firm he founded in 2012 and holds multiple roles at the University of Alberta, while Zeschuk owns
Blind Enthusiasm Brewing Company
, also based in Alberta, where he works with a small handful of other BioWare veterans.
But Zeschuk stepped back into the videogame spotlight, briefly, following last week's
closure of BioWare
by parent company Electronic Arts, telling
80lv
that he got the news hours after it happened because he was on a fishing trip, and by the time he flew home a couple days later, he'd "already shelved it away."
"I wasn't there to share in the moment, which in some ways was nice because I was able to kind of just take my time with it," Zeschuk said. "I’m not saying it doesn't hurt and doesn't suck. And I think it's kind of sad. It's actually, in some ways, saddest for the city of Edmonton, where we're from, because we were kind of this beacon of hope and excitement and possibility. We showed you could do something really special in a place that was not known for doing things like that."
Zeschuk said he had his own "emotional moment" about BioWare years ago, when he left the company; his real concern now is for the people at the studio who now find themselves out of work: "Anyone getting put into the job hunt is a sort of terrifying endeavor and super sad."
He also said he has no regrets about the EA acquisition, saying it offered "stability" as the studio continued to grow and that "we saw there'd be value in being part of something bigger rather than going our own way and taking all the risks." At the same time, he had a feeling when EA was acquired by Saudi Arabia's PIF that it probably wouldn't go well for BioWare, because it doesn't fit into the guns-and-sports model: "BioWare makes these kind of unusual games that take a long time and they're complicated and so it's easier just to pump out sports games every year and that's a better business model in some ways."
Zeschuk was also critical of the idea, seen in the recent wave of studio mergers and AI-powered cutbacks, "that you can just kind of recreate great games with random people and AI. It doesn't work," Zeschuk said.
"The other harsh reality of almost all these companies, the people that go down the business track never really come from the development track. Because none of these people have ever made a game, like, never really sat and made one, like, from start to finish, even a small thing, they just don't know how it works."
From the creators of Edge: A free weekly videogame industry newsletter, featuring in-depth news, expert analysis and exclusive interviews
Zeschuk does know how it works, and he was clear that he's not interested in going back. "For me it was time to go, and I kind of walked out of the games business without any intention of ever returning. I'm happy being a consumer and playing games," he said. "I have my opinions on how to make stuff, but as enticing as it would be, it's just not [worth] the personal struggle and cost and effort required to make a great game. Having gone through the crucible of game development, I'm not going back in there."
Andy has been gaming on PCs from the very beginning, starting as a youngster with text adventures and primitive action games on a cassette-based TRS80. From there he graduated to the glory days of Sierra Online adventures and Microprose sims, ran a local BBS, learned how to build PCs, and developed a longstanding love of RPGs, immersive sims, and shooters. He began writing videogame news in 2007 for The Escapist and somehow managed to avoid getting fired until 2014, when he joined the storied ranks of PC Gamer. He covers all aspects of the industry, from new game announcements and patch notes to legal disputes, Twitch beefs, esports, and Henry Cavill. Lots of Henry Cavill.
100 years ago, the world of mathematics experienced a foundational
crisis: the identification of several paradoxes revealed cracks in the
pillars supporting all areas of math. The most famous of which is
Russel’s paradox, which points to the contradictory “set of all sets
that do not contain themselves”, thereby showing the inconsistency of
naive set theory. This had spurred mathematicians to search for a
rigorous foundation for math without contradictions. In particular,
David Hilbert initiated a program to design a system of axioms that are
complete and consistent, meaning one should be able to derive all
mathematical truth from the axioms, while avoiding paradoxes—“the truth,
the whole truth, and nothing but the truth”. Such a goal was natural for
mathematicians, as it would place their work on firmer grounds. It was
therefore also a seismic event when Gödel dealt the death blow to
Hilbert’s program with his incompleteness theorems: such a complete,
consistent system for math cannot exist!
Although such an impossibility result ought to further deepen the
existential crisis, mathematicians soon recovered and carried on:
improvements to naive set theory, such as Russel’s theory of types, were
proposed; and while such systems still cannot be formally proven to be
consistent due to Gödel’s results, they are largely believed to be
consistent, and it turned out the foundational cracks have had little
effects on the kind of math people care about. In fact, type theory has
grown so mature that it forms the basis of interactive theorem provers
like Lean, making it possible to formalize a growing portion of math in
computer languages, so that proofs can be automatically checked by
programs from the ground up.
Besides the axiomatic system, Hilbert also proposed a related goal in
his program: the development of an algorithm to automatically decide the
truth of any mathematical statement, thereby reducing theorem proving to
the mechanical application of rules. This goal was also shown to be
impossible by Alan Turing, in his landmark proof of the undecidability
of the Halting problem which gave birth to the fertile field of computer
science. It’s rather a poetic contradiction that, in establishing the
impossibility
of Hilbert’s program, Gödel and Turing created
the most powerful computing machines that
can
.
A century has passed since the times of Hilbert, Gödel and Turing.
Against all odds, we are now witnessing the realization of Hilbert’s
“proof machine” incarnated as AI agents armed with mechanized formal
systems found in proof assistants like Lean. This development, as it
turned out, has had a far more devastating effect on the mathematical
community compared to the fall of Hilbert’s program. I am not sure if
Hilbert himself had considered the consequences had his program
succeeded, but von Neumann showed incredible prescience
1
:
When undecidability fails, then mathematics as it is understood today
ceases to exist; in its place there would be an absolutely mechanical
prescription with whose help one could decide whether any given sentence
is provable or not.
I am not a mathematician, but a computer scientist. In computing, the
impact of AI has been less violent, with the majority even cheering it
on, churning out code at an ever-increasing speed. What I sense, though,
is an air of abandonment: it is now becoming standard practice to stop
reading AI-generated code. This is a great tragedy for computer science.
We are one of the very few disciplines where it is possible to get to
the very bottom of things—we can understand exactly what a program does
as long as we have its source code, yet we are now giving this up! We
now have machines that can do everything Hilbert dreamed of, but we have
abandoned what he considered the very soul of science:
We must not believe those, who today, with philosophical bearing and
deliberative tone, prophesy the fall of culture and accept the
ignorabimus
. For us there is no
ignorabimus
, and in my
opinion none whatever in natural science. In opposition to the foolish
ignorabimus our slogan shall be
Wir müssen wissen – wir werden
wissen
("We must know – we will know").
As computer scientists, we should consider what’s happening in math:
refusing to read the code for a program that passes all tests—even
formal verification—is akin to ignoring the proof as soon as AI solves a
theorem in Lean. In fact, they are the same in the literal sense, as a
program is a proof via the Church-Howard correspondence, and that is
exactly how theorem provers like Lean operate. When an AI generated
proof is too long to read, mathematicians don’t give up understanding
it, but take great pains to simplify the proof until it’s digestible. We
must take the same attitude in computing: use abstractions, refactor,
refine scope, invent and adopt higher-level languages, until we can hold
the code in our head.
I don’t know if AI will eventually exceed humans at all activities—I
believe we think with our brains which are just biological computers
with a different architecture—but I know we will end up badly if we give
up thinking to the machines. Consider the growing cases where people
lose their lives by relying on AI for medical advice: the more we take
the machine’s words for granted, the less we are in control of our own
destiny.
We must know – we will know
.
JPEG XS
(
standardized
as
ISO/IEC
21122) is an image and video codec that offers both visually and mathematically lossless quality. It is a special-purpose codec that is specifically designed to allow for low-complexity and
low-latency
implementations.
[
2
]
[
3
]
Target applications of the standard include streaming high-quality content for
professional video over IP
(SMPTE
ST 2022
and
ST 2110
) in broadcast and other applications:
virtual reality
, drones, autonomous vehicles using cameras, gaming.
[
2
]
[
4
]
[
5
]
Although there is not an official acronym definition, XS was chosen to highlight the
extra small
and
extra speed
characteristics of the codec.
Three main features are key to JPEG XS:
Visually transparent compression
: XS compressed content is indistinguishable from the original uncompressed content (passing ISO/IEC 29170-2 tests, also called AIC-2) for compression ratios between 2:1 and 10:1.
Low latency
: The total end-to-end latency, introduced by the JPEG XS compression-decompression cycle, is minimal. Depending on the configuration, XS typically imposes only between 1 and 32 lines of additional end-to-end latency, when compared to the same system using uncompressed video.
Relying on these key features, JPEG XS is suitable for any application where uncompressed content is the norm, yet still allowing for significant savings in the required bandwidth usage, preserving quality and low latency. Among the targeted use cases are video transport over professional video links (like
SDI
and
professional video over IP
), real-time video storage, memory buffers, omnidirectional video capture and rendering, and image sensor compression (for example in cameras and in the automotive industry).
[
6
]
JPEG XS favors visually lossless quality in combination with low latency and low complexity, over data reduction through compression. It is not a direct competitor to alternative image codecs like
JPEG 2000
and
JPEG XL
or video codecs like
AV1
,
AVC/H.264
and
HEVC/H.265
which tend to focus on compression efficiency.
Other important features are:
Exact bitrate allocation
: JPEG XS allows an accurately targeted bitrate to perfectly match the available bandwidth (also referred to as
constant bitrate
).
Multi-generation robustness
: JPEG XS allows for at least 10 encoding-decoding cycles without significant quality degradation.
[
7
]
This feature allows, for example, transparent chaining of multiple devices that recompress the signal, without any significant quality degradation taking place.
Multi-platform interoperability
: The algorithms used in JPEG XS allow for efficient implementations on different platforms, like
CPU
,
GPU
,
FPGA
and
ASIC
. Each of these platform architectures is best exploited when a specific degree of parallelism is available in the implementation. For instance, a multi-core CPU implementation will leverage coarse-grained parallelism, while a GPU or FPGA implementation will work better with fine-grained parallelism. Moreover, the choice of parallelism used in the implementation at the encoder will not affect that of the decoder. This means that real-time encoding and decoding between platforms is possible, without sacrificing the low-complexity, low-latency or high-quality properties.
Support for mathematical
lossless coding
(MLS)
: JPEG XS is also capable of coding images in a mathematically lossless way, to achieve perfect reconstruction at the decoder side (new profile supported by 2nd edition).
Support for
High Dynamic Range
(HDR) content
: The current version of JPEG XS supports bit-depths of up to 16 bits per component, and it provides several parameterizable non-linear transforms to efficiently compress HDR content.
Support for RAW Bayer/CFA compression
: JPEG XS has also the capability to compress
color filter array
(CFA) content, such as RAW
Bayer
content produced by digital cameras. A special color transform, called Star-Tetrix, allows for efficient and direct compression of the original RAW sample values, without the need for converting the Bayer samples to RGB samples first.
[
8
]
Accurate flow control
: A JPEG XS encoder continuously monitors the amount of bits sent out, and adjusts its rate allocation process to neither overflow nor underflow a normatively
[
9
]
defined decoder input buffer.
Video bandwidth requirements are growing as video resolutions, frame rates, bit depths, and the number of video streams are constantly increasing. Although the capacities of video links and communication channels are also growing, there are applications where the reduced data rate afforded by JPEG-XS is advantageous.
Both the broadcast and pro-AV markets are shifting towards AV-over-IP-based infrastructure, with a preference going to
Gigabit Ethernet
links for remote production and faster fiber-based Ethernet for in-house facilities. Given the available bandwidth and infrastructure cost, relying on uncompressed video may be cost-prohibitive if 4K, 8K, increased bit depths for HDR, and higher frame rates need to be supported.
With XS, it is, for example, possible to repurpose existing SDI cables to transport 4K60 over a single 3G-SDI (at 4:1 compression ratio), and even over a single HD-SDI (at 8:1 compression ratio). Similar scenarios can be used to transport 8K60 content over various SDI cable types (e.g., 6G-SDI and 12G-SDI). Alternatively, XS enables transporting 4K60 content over Gigabit Ethernet and 8K60 over 5G or
10 Gigabit Ethernet
, which would be impossible without compression.
The following table shows expected compression ranges for some typical use cases.
Related to the transport of video streams is the storage and retrieval of high-resolution streams where bandwidth limitations similarly apply. For instance, video cameras use internal storage like
SSD drives
or
SD cards
to record high-resolution streams of images, yet the maximum data rates of such storage devices may be below that required to store uncompressed video in real time.
JPEG XS has built-in support for the direct compression of RAW
Bayer
/CFA images using the Star-Tetrix Color Transform. This transform takes a RAW Bayer pattern image and
decorrelates
the samples into a 4-component image with each component having only a quarter of the resolution.
[
11
]
This means that the total amount of samples to further process and compress remains the same, yet the values are decorrelated similarly to a classical Multiple Component Transform.
Avoiding such conversion prevents information loss and allows this processing step to be done outside of the camera. This is advantageous because it allows deferring
demosaicing
the Bayer content from the moment of capturing to the production phase, where choices regarding artistic intent and various settings can be better made; The demosaicing process is irreversible and requires certain choices, like the choice of interpolation algorithm or the level of noise reduction, to be made upfront. The demosaicing process can be power-hungry and will also introduce extra latency and complexity.
Describes the core coding system of JPEG XS. Defines the syntax and, similarly to other JPEG and MPEG image codecs, the decompression process to reconstruct a continuous-tone digital image from its encoded codestream. Provides some guidelines for the inverse process that compresses a digital image into a compressed codestream, the encoding process, but leaves implementation-specific optimizations and choices to the implementers.
Builds on top of Part 1 to segregate different applications and uses of JPEG XS into reduced coding tool subsets with tighter constraints. The definition of profiles, levels, and sublevels allows for reducing the complexity of implementations in particular application use cases, while also safeguarding interoperability. Profiles represent interoperability subsets of the codestream syntax specified in Part 1. In addition, levels and sublevels provide limits to the maximum throughput in the encoded (codestream) and the decoded (spatial and pixels) image domains. Also specifies a buffer model, consisting of a decoder model and a transmission channel model, to enable guaranteeing low latency requirements to a fraction of the frame size.
Specifies transport and container formats for JPEG XS codestreams. It defines the carriage of important metadata, like color spaces, mastering display metadata (MDM), and
Exif
, to facilitate transport, editing, and presentation. Defines the XS-specific ISOBMFF boxes, an
Internet media type
registration, and additional syntax to allow embedding XS in formats like MP4, MPEG-2 TS, or the HEIF image file format.
A reference software implementation (written in ISO C11) of the JPEG XS Part 1 decoder, conforming to the Part 2 profiles, levels and sublevels, as well as an exemplary encoder implementation.
RFC9134 - RTP Payload Format for ISO/IEC 21122 (JPEG XS)
The
RTP payload format
for JPEG XS, in turn, enables the use of JPEG XS in
SMPTE 2110
environments using SMPTE ST 2110-22 for CBR compressed video transport.
Internet Protocol Media Experience
(IPMX) is a suite of open standards and specifications to enable the carriage of compressed and uncompressed video, audio, and data over IP networks for the pro AV market. JPEG XS is supported under IPMX via VSF TR-10-8
[
17
]
and TR-10-11.
[
18
]
GenICam
(Generic Interface for Cameras) is a set of freely available standards developed by the
European Machine Vision Association (EMVA)
to provide a generic programming interface for industrial and machine vision devices. One of its specifications, the Generic Data Container (GenDC), defines a standardized format for transporting image data and metadata independently of the underlying transport interface. Support for JPEG XS encoded image data was added in
GenDC version 1.2
, enabling the use of JPEG XS across GenICam-compliant systems and transport standards that rely on GenDC, such as
GigE Vision
,
USB3 Vision
,
CoaXPress
and
Camera Link
.
The JPEG committee started the standardization activity in 2016 with an open call for a high-performance, low-complexity image coding standard. The best-performing candidate technologies came from intoPIX and Fraunhofer IIS and formed the basis for the new standard. First implementations were demonstrated in April 2018 at the
NAB Show
and later that year at the
International Broadcasting Convention
. JPEG XS was also presented at
CES
in 2019. JPEG XS was formally standardized as ISO/IEC 21122 by the
Joint Photographic Experts Group
with the first edition published in 2019. A second edition was published in 2022,
[
19
]
adding support for direct compression of raw
Bayer CFA
content, lossless compression, and support for 4:2:0 color subsampling. Today, the JPEG committee is still actively working on further improvements, with the third edition published in 2024.
[
needs update
]
This edition adds support for a temporal decorrelation technology in the wavelet domain, called Temporal Differential Coding (TDC).
In 2025, JPEG XS received the EMMY® Award by the Television Academy for technical innovations in the US television industry.
[
20
]
The JPEG XS standard is a
wavelet-based
still-image codec without any frame buffer. While the standard
[
21
]
defines JPEG XS based on a hypothetical reference decoder, JPEG XS is easier to explain through the steps a typical encoder performs:
[
22
]
Component up-scaling and optional component decorrelation:
In the first step, the DC gain of the input data is removed and pixels are upscaled to a 20-bit precision. Optionally, a multi-component generation, identical to the JPEG 2000 RCT, is applied. This transformation is a lossless approximation of an RGB to YUV conversion, generating one luma and two chroma channels.
Wavelet transformation:
Input data is spacially decorrelated by a 5/3
Le Gall–Tabatabai
wavelet filter
.
[
citation needed
]
While a five-stage transformation is performed in the horizontal direction, only 0 to 2 transformations are run in the vertical direction. The reason for this asymmetrical filter is to minimize latency.
Prequantization
: The output of the wavelet filter is converted to a sign-magnitude representation and pre-quantized by a
dead-zone quantizer
to 16-bit precision.
Rate control and quantization
: The encoder determines by a proprietary process
[
22
]
the rate of each possible quantization setting and then quantizes data by either a dead-zone quantizer or a data-dependent uniform quantizer.
Entropy coding
: JPEG XS uses minimalistic
entropy encoding
for the quantized data, which proceeds in up to four passes over horizontal lines of quantized wavelet coefficients. The steps are:
Significance coding
: In the (optional) first pass, the significance of 32 consecutive wavelet coefficients is coded by a single bit.
Bitplane count coding
: In the second pass, the number of non-zero bitplanes of groups of four coefficients each, the so-called
bitplane count
, is entropy coded through a
Golomb
type code. This step may optionally use the bitplane counts of the preceding line as the source for prediction (
differential pulse-code modulation
) and then encode only the prediction difference.
Data coding
: The third pass inserts the raw bitplane values into the codestream without further coding.
Sign coding
: In the last optional coding pass, the sign bits of all non-zero coefficients are inserted into the codestream. If this coding pass is not present, sign bits are included in the data coding pass for all coefficients.
Codestream packing
: All entropy-coded data are packed into a linear stream of bits (grouped in byte multiples) along with all of the required image metadata. This sequence of bytes is called the codestream and its high-level syntax is based on JPEG marker and marker segment syntax.
[
23
]
JPEG XS defines profiles in ISO/IEC 21122-2
[
9
]
that define subsets of coding tools that conforming decoders must support and limit the permitted parameter values and allowed markers. The following table represents an overview of all the profiles along with their most important properties.
In addition, JPEG XS defines levels to represent a lower bound on the required throughput that conforming decoders need to support in the decoded image domain. The following table lists the levels as defined by JPEG XS. The maximums are given in the context of the sampling grid, so they refer to a per-pixel value where each pixel represents one or more component values. However, in the context of Bayer data, JPEG XS internally interprets the Bayer pattern as an interleaved grid of four components. This means that the number of sampling grid points required to represent a Bayer image is one-fourth the total number of Bayer sample points. Each group of
2 × 2
(four) Bayer values gets interpreted as one sampling grid point with four components. Thus, sensor resolutions should be divided by four to calculate the respective width, height and number of sampling grid points. For this reason, all levels also bear double names.
Similar to the concept of levels, JPEG XS defines sublevels to represent a lower bound on the required throughput that conforming decoders need to support in the encoded image domain. Each sublevel is defined by a nominal bit-per-pixel (
N
bpp
) value that indicates the maximum number of bits per pixel for an encoded image of the maximum permissible number of sampling grid points according to the selected conformance level. Thus, decoders conforming to a particular level and sublevel shall conform to the following constraints derived from
N
bpp
:
The following table lists the existing sublevels and their respective nominal
N
bpp
values. Please refer to the standard for a complete specification of each level.
Usage of a TDC profile results in a TDC-enabled codestream that requires support for a frame buffer in the encoder and decoder. In this context, the concept of Frame Buffer Bandwidth (FBB) levels is used by JPEG XS to represent a lower bound on the required throughput that conforming decoders need to support for accessing the frame buffer. From the perspective of the codestream, the FBB level defines the minimal required read and write bandwidths of the frame buffer. This bandwidth is defined by a nominal bit-per-pixel (
B
f
) value that is associated with a valid FBB level and used as an input parameter of the FBB model of JPEG XS. Please refer to the standard for more details.
The following table lists the existing FBB levels and their respective
B
f
values.
JPEG XS contains patented technology which is made available for licensing via the JPEG XS Patent Portfolio License (JPEG XS PPL). This license pool covers essential patents owned by Licensors for implementing the ISO/IEC 21122 JPEG XS video coding standard and is available under
RAND
terms.
[
24
]
↑
T. Richter, J. Keinert, S. Foessel, A. Descampe, G. Rouvroy and J. -B. Lorent, "JPEG-XS -- A High-Quality Mezzanine Image Codec for Video Over IP," in SMPTE Motion Imaging Journal, vol. 127, no. 9, pp. 39-49, Oct. 2018, doi: 10.5594/JMI.2018.2862098
link
.
↑
Keinert, Joachim; Lorent, Jean-Baptiste; Descampe, Antonin; Rouvroy, Gaël; Fößel, Siegfried.
"Introduction to JPEG XS"
.
IBC
.
Archived
from the original on 2023-04-02.
↑
Richter, Thomas; Keinert, Joachim; Descampe, Antonin; Rouvroy, Gael; Willeme, Alexandre (December 2017). "Multi-generation-robust Coding with JPEG XS".
2017 IEEE International Symposium on Multimedia (ISM)
. pp.
6–
13.
doi
:
10.1109/ISM.2017.12
.
ISBN
978-1-5386-2937-6
.
S2CID
32986247
.
↑
T. Richter, S. Fößel, A. Descampe and G. Rouvroy, "Bayer CFA Pattern Compression With JPEG XS," in IEEE Transactions on Image Processing, vol. 30, pp. 6557-6569, 2021, doi: 10.1109/TIP.2021.3095421
link
.
↑
A. Descampe et al., "JPEG XS--A New Standard for Visually Lossless Low-Latency Lightweight Image Coding," in Proceedings of the IEEE, doi: 10.1109/JPROC.2021.3080916
PDF
.
On October 16, 2026, China’s Ministry of Commerce (MOFCOM) and Ministry of Science and Technology (MOST) issued Announcement No. 48 of 2026. The announcement proposes adding 40 technologies to the
Catalogue of Technologies Prohibited or Restricted from Export
, 23 of them in artificial intelligence, and it imposes an interim pre-export registration requirement that is already in effect [1][2]. China’s catalogue has touched AI before: the 2020 revision added personalized recommendation and AI interaction interface technology [6]. Unlike those 2020 additions, the draft reaches frontier model weights, agent architectures, and AI-enabled offensive cyber capability, with explicit technical thresholds for each [1].
The announcement contains two instruments with different legal status. The catalogue additions are a draft open for comment until November 15, 2026, while the registration requirement applies now, including to exports that have started but are not yet complete [1][2]. Within the draft, model weights are restricted at frontier scale. Weights, key checkpoints, source code, architecture, and training and inference optimization know-how are restricted if a model meets any one of three tests: at least 10^26 FLOPs of training compute, at least 2 trillion parameters, or a global top-10 position on public evaluation rankings, global call volume, or comparable measures at the time of export [1][3]. Agents are controlled independently of the model. Planning, multi-agent coordination, tool use, and game-theoretic decision technologies are restricted above set benchmarks, and high-performance agents built for critical infrastructure, defense, or core government use are prohibited outright [1].
The line between service access and custody determines most enterprise exposure. MOFCOM says API, app, web, SaaS, and lightweight on-device use do not count as exporting a large model. Deploying a restricted model locally outside China does count, though automatic licensing may apply on a Chinese company’s own overseas cloud [2]. Multinationals also face exposure through licensing deals and changes of ownership or control, not only through product exports, and analysts read the registration form as potentially capturing intra-group code sharing as well [2][3][4]. Any transfer of ownership, licensing for overseas use, or shift of control to a foreign entity can trigger registration [2][4].
Background
China regulates outbound technology transfer under two overlapping regimes. The first is the technology-trade regime under the Foreign Trade Law and the Regulations on Technology Import and Export Administration, which governs the catalogue being revised here. The second is the dual-use export-control regime under the 2020 Export Control Law [5]. MOFCOM’s spokesperson described the revision as routine and grounded in Article 20 of the Foreign Trade Law. The spokesperson stressed that it “does not target any specific country, region, or entity” and that the aim is to prevent uncontrolled diffusion while protecting legitimate cooperation [2]. The catalogue has been revised several times since 2001. The August 2020 revision, which added recommendation-algorithm technology, coincided with the proposed sale of TikTok to a U.S. buyer and was reported as likely to complicate that deal [3][6].
Across all three sectors, Announcement No. 48 proposes 14 prohibited and 26 restricted items: 23 in AI, 5 in biomedicine, and 12 in photovoltaics [2]. In China’s technology-trade framework, prohibited technologies may not be exported at all, while restricted technologies require a license from commerce and science authorities. ChinaTalk notes that licensing is administered by provincial authorities, although central authorities have intervened in prominent cases [3]. The official annex groups the AI items into large models, agents, compute clusters, chip design, and information-processing technologies. It also excludes from management any technology that has already entered the public domain through open-source release or academic publication [1].
The interim registration requirement is the part that applies today. From the announcement date until the revised catalogue takes effect, exporters must register through MOFCOM’s unified business platform before exporting any listed technology. Registration covers four triggers: transfer of technology ownership abroad, licensing of domestic technology for overseas use, a change in ownership or control that makes a foreign entity the owner or controller, and other cases MOFCOM designates [2][4]. Exporters must decide for themselves whether a technology falls within scope. The registration form asks for recipient details, destination, technology description, end use, and estimated value [2][4].
The timing also matters for enterprises that already track U.S. controls. CSA’s prior research documented the first U.S. use of export-control authority to suspend a broadly deployed commercial frontier model [9]. Announcement No. 48 means frontier AI supply chains now face formal outbound controls from both of the largest AI-producing jurisdictions. ChinaTalk observes that the 10^26 FLOP threshold mirrors the figure used in the 2023 U.S. executive order and the since-rescinded 2025 U.S. AI Diffusion rule. The authors also estimate that no released Chinese model currently exceeds it [3].
Security Analysis
What the AI Annex Actually Controls
The annex is granular, and its structure matters for scoping. The table below summarizes the AI items drawn from the official draft annex [1], with interpretive notes from published analyses [3][4].
Category
Control
Key Thresholds or Conditions (draft)
Large models trained on China’s “core data”
Prohibited
Core data scope defined by the Data Security Law and related regulations
Large models trained on China’s “important data”
Restricted
Excludes trace use (≤0.1%) of desensitized important data that does not materially improve capability or reappear in outputs, and lawfully public data
Core large-model technologies (weights, checkpoints, source code, architecture, training and inference optimization)
Restricted
Any one of: ≥10^26 FLOPs; ≥2 trillion parameters; global top-10 position on public evaluation rankings, global call volume, or comparable measures at the time of export
High-performance agents for critical infrastructure, defense, or core government
Prohibited
Agent meets the restricted agent thresholds and is specially or mainly designed for those uses
AI malicious code generation, adaptive evasion, and variant generation
Prohibited
Attack-success and automation thresholds, where usable against high-sensitivity systems
AI-assisted vulnerability discovery
Restricted
Any of six tests, including ≥75% on authoritative benchmarks, 10x senior-expert efficiency, or autonomous chaining of 2+ low or medium flaws into a working exploit
AI detection technology specially designed for cross-border data security
Prohibited
Detection accuracy and recall ≥95%
Deepfake detection; steganography; voiceprint; Chinese and minority-language translation
Restricted
Accuracy, recall, latency, and fidelity thresholds (for example, ≥98% for deepfake detection)
AI chip and memory chip design; large compute cluster technologies
Restricted
Process node ≤14 nm and other alternative criteria for AI chips; interconnect ≥800 GB/s, fault tolerance, and heterogeneous scheduling for clusters above 10,000 accelerators
Three features of this structure deserve attention from security and compliance teams. First, the “global leading” test is dynamic. A model that falls below the compute and parameter thresholds can still become restricted if its public evaluation rank, global call volume, or similar measures place it in the top 10 when the export occurs [1]. ChinaTalk calls this criterion “by far the most up-for-interpretation,” noting that the rule names no ranking body [3], and it means the compliance status of a given model version can change after deployment. Second, the agent items are defined by system behavior rather than by the underlying model. An orchestration framework that integrates hundreds of tools could meet the tool-use threshold even when it runs on a model well below the frontier thresholds, an issue Ray Sun identifies as unresolved [5]. Third, the cyber items carry direct security consequences: Chinese-origin AI red-teaming, vulnerability-discovery, and detection tools that meet the thresholds would require a license to leave China, and offensive code-generation capability could not be exported at all.
Deployment Models and Where the Line Falls
MOFCOM’s spokesperson framed the central test as whether restricted technology moves from inside China to overseas [2]. That framing produces a clear distinction between consuming a model as a service and taking custody of it, as summarized below.
Deployment scenario
Treatment under MOFCOM guidance
API calls, apps or websites, SaaS for overseas customers
Not within large-model export management [2]
Lightweight on-device deployment
Not within large-model export management [2]
Local deployment of a restricted model on infrastructure outside China
Within export management; license required [2]
Deployment on a Chinese company’s self-operated overseas cloud or compute center
Automatic licensing (“report and export”) may apply [2]
Open-sourcing newly developed restricted technology
Automatic licensing after reporting [2]
Technology already public via open-source release or academic papers
Outside management scope [1]
For enterprises, the practical consequence is that the deployment model the draft captures, local self-hosting, is also the one often chosen for data-protection reasons. Organizations that chose to self-host a Chinese frontier model in their own environment, rather than call a Chinese-hosted API, did so to keep sensitive data inside their own boundary. MOFCOM places local deployment outside China within export management, which means those self-hosting arrangements require the Chinese supplier to hold a license [2]. ChinaTalk reads this as also covering on-premises licensing deals and managed deployments in which a vendor runs an agent’s reasoning loop inside a customer environment [3]. Because licenses can be revoked or conditioned, ChinaTalk suggests this may lead some non-Chinese users to reconsider such dependencies [3]. The registration requirement already covers exports in progress, so ongoing weight-delivery or model-update arrangements may be affected now rather than after the catalogue is finalized [2].
The treatment of open weights depends on whether a model is already public. Weights that are already public are outside the controls [1]. Open-sourcing a
new
restricted model, however, falls under automatic licensing that still requires reporting [2]. Commentators note that “open source” is not defined, and that it is unclear how restricted-use licenses would be treated [3]. Organizations that depend on successive open-weight releases from Chinese labs should therefore treat future release cadence, not just current availability, as a variable outside their control.
Exposure Through Corporate Structure
The exposure most likely to surprise multinationals sits in their own organizational charts. Registration is triggered by licensing technology for overseas use and by changes of ownership or control, not only by sales to third parties [2][4]. A multinational with China-based R&D teams building agent frameworks, vulnerability-discovery tooling, or fine-tuned models may therefore need to register before sharing that work with affiliates outside China. ChinaTalk reads the registration form’s field on exports made outside formal contracts as potentially capturing intra-group code sharing and open-sourcing [3]. This reading is an inference from the form rather than an official statement, but it is a reasonable basis for conservative scoping.
The data-based items add a second pathway. A model fine-tuned in China on operational data that qualifies as “important data” under the Data Security Law would be restricted, unless the de minimis carve-out applies. A model trained on “core data” would be prohibited [1]. Organizations that classified China-resident data under the Data Security Law for cross-border data transfer purposes may not have applied the same classification to training and fine-tuning datasets, since those obligations previously governed data transfer rather than model transfer.
Finally, recent precedent suggests that redomiciling does not remove Chinese jurisdiction. In January 2026, MOFCOM announced an assessment of Meta’s acquisition of Manus, a Singapore-registered AI agent company whose products were developed in China. The assessment examined whether the relocation of staff and technology required export authorization [7]. Announcement No. 48 now gives that line of scrutiny explicit, agent-specific thresholds, which suggests M&A due diligence on AI targets with China-origin development history should include an export-control workstream.
Recommendations
Immediate Actions
Organizations should first inventory every Chinese-origin model, agent framework, and AI security tool in use and record how each is consumed: as an API or SaaS service, as a lightweight on-device component, or as locally deployed weights or code. The local-deployment category is the priority, because it is the one the draft places within export management [2]. For each locally deployed asset, teams should ask the supplier in writing whether it has registered the export under Announcement No. 48 and whether it expects to need a license once the catalogue is final.
Legal and engineering teams should jointly review technology flows out of China-based subsidiaries, joint ventures, and R&D centers since October 16. The goal is to identify transfers that may already require registration, including code repositories, model checkpoints, and agent frameworks shared with overseas affiliates [2][4]. Organizations with a material interest in the final scope should prepare comments before the November 15, 2026 deadline, through the email, fax, or postal channels published in the announcement [1][4].
Short-Term Mitigations
Over the next one to two quarters, organizations should treat restricted-category Chinese models as regulated dependencies with documented fallbacks. Vendor contracts for locally deployed models should allocate responsibility for Chinese export registration and licensing, specify notice obligations if a license is denied, conditioned, or revoked, and address whether updates and new checkpoints are covered by the same authorization. Because the “global leading” test depends on rankings at the time of export [1], teams should not assume that a model version cleared today will remain uncontrolled for later updates.
Security teams that rely on AI-assisted vulnerability discovery or deepfake detection tools developed in China should map those tools against the restricted thresholds and identify alternatives. A licensing interruption to a detection capability is a security-control gap, not only a procurement issue. Data governance teams should also extend Data Security Law classification work to cover training and fine-tuning datasets held in China, so that “important data” and “core data” exposure is known before models trained on them are shared abroad [1].
Strategic Considerations
Announcement No. 48, if finalized substantially as drafted, would mean that frontier AI is treated as controlled technology by both the United States and China, with each regime drawing its lines differently. Recent U.S. measures have emphasized compute and, in the Fable 5 case, access to a deployed model, although the since-rescinded 2025 AI Diffusion rule also reached model weights [3]. The Chinese draft centers on custody of weights, know-how, and agent architectures. Enterprise AI governance programs should model export-control disruption from either jurisdiction as a standing risk scenario and design architectures that can swap models without rebuilding agent workflows.
Organizations should also recognize that the draft treats agents as a distinct regulatory object. Defining controls by planning depth, concurrency, and tool count means that internal agent platforms, and not only foundation models, may be subject to cross-border technology rules. Agent inventories should record these characteristics alongside traditional security attributes. Finally, because the final catalogue may narrow, broaden, or shift after consultation [3], and because ChinaTalk notes the comment window closes shortly before an expected leaders’ meeting at APEC [3], a timing that could bear on the final scope, this analysis should be revisited once MOFCOM publishes the final text.
CSA Resource Alignment
CSA’s most directly relevant prior work is
Frontier AI Export Controls: Enterprise Governance After Fable 5
[8]. That note argues for treating frontier model access as a contingent operational dependency that belongs in vendor management and business continuity planning. Announcement No. 48 extends that argument to a second jurisdiction and to a different control point. The U.S. action examined in that note interrupted service access, while the Chinese draft leaves service access untouched and instead governs weight custody and local deployment. Enterprises that built continuity plans around the U.S. scenario should check whether those plans also cover a supplier-side license failure for self-hosted models.
AI Model Export Controls: The Fable 5 Precedent
[9] documented how a single regulatory directive can remove a widely deployed model with little notice. Its recommendation for multi-model fallback strategies applies directly to organizations that depend on Chinese frontier or open-weight models, particularly given the dynamic “global leading” threshold described above.
Sovereign AI Access Controls and Enterprise Concentration Risk
[10] provides the broader framing. It treats national control over AI capability as a source of concentration risk, and the Chinese draft is a concrete instance of that risk materializing on the supply side.
For control implementation, the AI Controls Matrix (AICM) v1.1 [11] provides the relevant control objectives in its supply chain management, transparency and accountability, and governance, risk and compliance domains, including supplier inventory, third-party assessment, and regulatory change management. For agent-specific exposure, MAESTRO [12] offers a layered threat model that can help teams document the agent capabilities, such as tool integration and multi-agent coordination, that now carry export-control significance.
The Hamburg Rathaus tells the story of its city in more than a hundred small details, and this is my walk through them.
The Rathaus was one of the first things I photographed in Hamburg. The picture is still in my camera roll, taken on 6 July 2017, five days after I arrived for an internship. Like most visitors, I was impressed by its size and its tower, but what I kept coming back to was the roof. Why was it turquoise, and why were so many other roofs in the city the same colour?
My first Rathaus picture
The answer turned out to be chemistry. The roof is copper, and copper slowly reacts with oxygen, water and carbon dioxide in the air. Over the decades this forms a green layer called patina, so the roof I was looking at wasn't the colour the builders had seen.
That was the first time the Rathaus made me curious, but not the last. On a city tour with
Robin and the Tour Guides
, our guide pointed out the emperors on the façade, the Phoenix and the Hanseatic coats of arms. We only heard a little about each, but it was enough to make me look at the building differently.
Later, I decided to build a digital 3D model of the Rathaus, which meant photographing it from every side. Zooming in, I kept finding faces, figures and coats of arms I had never noticed from the square. Reading about them one by one, I realised the Rathaus can be read like a book about Hamburg's history.
In this article, I'd like to take you around the building and share what I found.
A building that rose from the ashes
The Rathaus we see today isn't Hamburg's first. It's probably the sixth, and fire runs through the story of the ones before it. An early Rathaus, built after the old town and the new merchant town joined in 1216, was most likely lost in the city fire of 1284. Around 1290, a bigger brick Rathaus was built next to the Trostbrücke, and it stood there for more than 550 years.
The old Rathaus, 1700
Then, in the night of 5 May 1842, the Great Fire broke out. The flames spread quickly through the narrow streets, and on the morning of 6 May the people of Hamburg blew up their own Rathaus with 800 pounds of gunpowder, hoping to stop the fire. It didn't work, and the fire went on for days. Today the Haus der Patriotischen Gesellschaft stands where the old Rathaus once was.
It took 55 years until Hamburg had a new Rathaus. The foundation stone was laid on 6 May 1886, exactly 44 years after the old one was blown up, and the building opened in 1897. Its front is 111 metres wide, and its tower is 112 metres high, almost exactly as tall as the building is wide.
The fire is remembered right on its front. High on the tower, between the clock and Hamburg's coat of arms, sits a Phoenix, the bird that rises again from its own ashes, with wings more than 2 metres wide. Below it, a gilded medallion shows the old Rathaus and the word "Resurgam", meaning "I will rise again". In 1892, exactly 50 years after the fire, the Phoenix was pulled to the Rathaus by four black horses.
The Phoenix
In the summer of 1943, the bombing raids of Operation Gomorrah destroyed large parts of Hamburg. This time the Rathaus was spared. It was one of the few historic buildings in the centre to survive, and that's why we can still read every detail on its façade today.
The emperors on the façade
Above the main entrance, two emperors stand on either side of the balcony: Charlemagne on the left and Friedrich I Barbarossa on the right. Between them, in the background, Hammonia looks down from a golden mosaic made in 1896, holding out a wreath. She is the goddess who stands for Hamburg, much like Britannia does for Britain, and she represents the city's values: freedom, peace, trade and prosperity.
Hammonia
It might seem strange that a city so proud of its freedom puts emperors on its town hall. But Hamburg never had a king of its own. Over the centuries, the Dukes of Saxony, the Counts of Holstein and later the Kings of Denmark all claimed power over the city, and Hamburg's answer was always the same: it answered only to the emperor, the head of the Holy Roman Empire. When Denmark claimed Hamburg in 1618, the city added emperor statues to its old Rathaus to make that point, and the new Rathaus carries on the tradition.
Around them are a few more details. Above Charlemagne is a small coat of arms with a black eagle on gold, the eagle of the empire. Above Barbarossa are three black lions on gold, the arms of his family, the Staufer. And on the columns next to the statues are the letters SPQH, short for Senatus Populusque Hamburgensis, "the Senate and the People of Hamburg". It's a copy of the SPQR of ancient Rome, and a reminder that even standing next to the emperors, the city was run by its own citizens.
Charlemagne (left) and Barbarossa (right)
Charlemagne (768–814) is where Hamburg's story begins. He was crowned emperor in 800, and in his time the area around the Alster and the Elbe came under Frankish rule. According to old stories, he also started the first church here. So it makes sense that the whole line of emperors starts with him.
Barbarossa (1152–1190) is there for a very different reason. On 7 May 1189, he is said to have given Hamburg the right to sail down the Elbe to the North Sea without paying any tolls. This is seen as the birth of the port, and the city still celebrates it every May with the Hafengeburtstag, the harbour's birthday.
The twist is that the document is a fake. Count Adolf III, who ruled Hamburg at the time and joined Barbarossa on his crusade, most likely did get some kind of promise from him. But Barbarossa died on the crusade, and there was no written proof. Decades later, the people of Hamburg wrote the charter themselves, with everything they wanted in it, and got it sealed with the seal of Barbarossa's grandson, Friedrich II. Historians have known this since 1907, but the city celebrates anyway. I find it funny that one of the biggest ports in the world partly owes its start to a clever trick.
Hamburger Barbarossa-Freibrief 1189. Von Uwe Rohwedder - Eigenes Werk, CC BY-SA 4.0
The other 18 rulers stand in the niches between the first-floor windows, and Charlemagne and Barbarossa split them in time. Charlemagne leads the older rulers on the left, and from him the line moves forward in time until Lothar III, who died in 1137. Barbarossa, who became king 15 years later, leads the newer ones on the right, and from him the line continues all the way to Franz II, the last emperor, in 1806. Strictly speaking, not all of them belong to the Holy Roman Empire: the first five ruled before the empire began in 962, in the Frankish empire, but in the 1890s they were seen as the start of the same line.
When I was photographing them, I noticed that the emperors stand behind thin nets. At first they made them look a bit trapped, but I guess the nets are there to keep the pigeons away. These men once ruled large parts of Europe, and today they need protecting from the pigeons on the Rathausmarkt.
Left of the balcony, from Charlemagne outwards
Ludwig the Pious (814–840)
: Charlemagne's son, who made Hamburg the seat of an archbishop around 831.
Ludwig the German (843–876)
: Charlemagne's grandson and king of the eastern Franks, in whose time Vikings burned Hamburg in 845.
Konrad I (911–918)
: Duke of Franconia, chosen as king after Charlemagne's family line died out in the east.
Heinrich I (919–936)
: The first king from Saxony, the region Hamburg belonged to, who fought back the Danes in the north.
Otto I, the Great (936–973)
: Heinrich I's son, whose crowning in Rome in 962 marks the start of the Holy Roman Empire.
Otto II (973–983)
: Otto I's son, during whose last year a Slavic uprising reached Hamburg and burned the town.
Konrad II (1024–1039)
: The first emperor of the Salian family, who added the Kingdom of Burgundy to the empire.
Heinrich III (1039–1056)
: Konrad II's son, so powerful that he removed three rival popes in 1046.
Lothar III (1125–1137)
: Duke of Saxony and later emperor, who gave the area around Hamburg to the Counts of Schauenburg.
The Emperors
Right of the balcony, from Barbarossa outwards
Heinrich VI (1190–1197)
: Barbarossa's son, who added Sicily to the empire through his marriage.
Friedrich II (1212–1250)
: Barbarossa's grandson, who gave Hamburg to the King of Denmark in 1214 and whose seal is on the fake charter.
Rudolf of Habsburg (1273–1291)
: The first Habsburg king, who brought order back after years without a strong ruler but was never crowned emperor.
Karl IV (1346–1378)
: King of Bohemia, who made Prague his capital and wrote the Golden Bull of 1356, the rules for choosing kings.
Maximilian I (1493–1519)
: Known as "the last knight", during whose rule Hamburg was officially counted as a free imperial city in 1510.
Karl V (1519–1556)
: Maximilian's grandson and King of Spain, in whose time Hamburg became Protestant in 1529.
Maximilian II (1564–1576)
: Karl V's nephew, during whose rule the English Merchant Adventurers settled in Hamburg in 1567.
Joseph II (1765–1790)
: Maria Theresa's son, in whose time Denmark finally accepted Hamburg as a free imperial city in 1768.
Franz II (1792–1806)
: The last Holy Roman Emperor, who gave up the crown in 1806 under pressure from Napoleon.
The virtues above the emperors
Hamburg gave Charlemagne and Barbarossa the place of honour, right above the main entrance. But above the emperors, the builders placed what mattered even more to them: Hamburg's own coat of arms and the four civic virtues.
The coat of arms shows a white castle with three towers on red, with a cross and two stars above it, held up by two stone lions. You don't often see the coat of arms in this full form, with the lions and the helmet on top. The last time I saw it was embossed at the top of a letter I received from the Mayor of Hamburg, congratulating me on becoming a German citizen and on choosing Hamburg as my home. The interesting thing is that the letter talks about freedom, tolerance and trade, almost the same values Hammonia stands for.
From the letter
Below it sit four bronze women, the virtues that, according to the builders, every Hamburg citizen should have. Courage wears a winged helmet and holds a sword, Piety raises a large cross, Unity has a sword lying across her lap, and Wisdom holds up a torch, standing for knowledge and good judgement.
Between them, in gold letters, runs the city's motto: "Libertatem quam peperere maiores digne studeat servare posteritas". It means "May the young generation be worthy of keeping the freedom that the old generation won." It sums up the whole building in one sentence: Hamburg is a free city, and its citizens have to look after that freedom.
The citizens above the emperors
Above Charlemagne and Barbarossa sit the virtues. Above the other 18 emperors, the builders went a step further and placed the citizens themselves. Over each of the 28 first-floor windows is a small triangular gable with a stone bust of a profession, from fisherman and baker to judge and banker. So along the whole façade, the people who made the city work sit higher than the rulers of the empire.
The sculptures modelled on real people
All 28 busts were made by the sculptor Aloys Denoth, and half of them were modelled on real people, from craftsmen to some of the best-known names of Hamburg at the time:
President of the city parliament
: Johann Georg Mönckeberg, holding a bell and a bill. Hamburg's main shopping street, the Mönckebergstraße, is named after him.
Mayor
: Carl Friedrich Petersen, who led the Rathaus building commission, holding a model of the Rathaus.
Shoemaker
: Master W. Knieriem, a shoemaker from St. Pauli.
Sculptor
: Engelbert Peiffer, Denoth's fellow sculptor, who carved the coat of arms with the two lions on the tower and the statue of Heinrich VI.
Locksmith
: the master locksmith Hermann Eggers.
Cabinetmaker
: Julius Rud. Loose, who had a workshop at Pulverteich.
Potter
: the master potter A.D.C. Warnstedt.
Painter
: the landscape painter Valentin Ruths, many of whose works are still in the Hamburger Kunsthalle.
Engineer
: Franz Andreas Meyer, the city's chief engineer, holding a model of a bridge. He planned the Speicherstadt and built bridges still in use today, like the Trostbrücke.
Judge
: Ernst Friedrich Sieveking, the first president of the Hanseatic Higher Regional Court, after whom the Sievekingplatz is named.
Scholar
: Justus Brinckmann, the founder of the Museum für Kunst und Gewerbe, holding objects from his collection. A cup he was holding has broken off.
Teacher
: the geographer Georg von Neumayer, after whom Germany's research station in Antarctica is named.
Architect
: Wilhelm Hauers, one of the architects of the Rathaus itself, holding a model of St. Johannis church in Harvestehude, which he designed.
Fund broker
: Eduard Behrens, a banker and art collector who gave paintings to the Hamburger Kunsthalle.
The other half show the everyday trades of the city: the fisherman, skipper and merchant, the tailor, baker, butcher, farmer and gardener, the shipbuilder, carpenter and mason, the pastor, a brewer or innkeeper with a cigar, and the chimneysweep.
The rest of the sculptures
What surprised me most is how detailed they are. The baker holds a cake, the butcher has a pig's head, and the judge holds a law book. The sad part is that you can hardly see any of this with the naked eye. This is where you need a phone with a good camera: at 4x zoom, you can see what each of them is holding.
The coats of arms of the Hanseatic cities
Between the busts of the citizens and the emperors below them, each of the 28 gables carries a small coat of arms. Each one stands for a city of the Hanseatic League, or for a city that traded closely with it. Together they run around the building, from the Große Johannisstraße side, along the Rathausmarkt, to the Alter Wall side.
The Hanseatic League, or simply the Hanse, was a network of merchant towns that worked together to protect their trade. From the 13th to the 17th century it connected around 200 towns, from the Netherlands to the Baltic, with Lübeck as its leading city. The Hanse is so interesting that it would need its own article, but you can still see its traces in Hamburg today. The city's official name is the Free and Hanseatic City of Hamburg, and that's why Hamburg's number plates start with HH, for Hansestadt Hamburg.
Two of the coats of arms are bigger than the others: Lübeck on the left side of the front and Bremen on the right. They sit below the two most important busts, the president of the city parliament and the mayor, and it's probably no accident. Lübeck was the leading city of the Hanse, and when the league slowly faded away, Lübeck, Bremen and Hamburg were the last three cities that kept its name. Like Hamburg, they still call themselves Hanseatic cities, and their number plates still start with HL and HB.
Not every city on the façade was a full member. Amsterdam and Utrecht, for example, were trading partners rather than members. Florence is far from the North Sea, but its bankers, like the Medici, had offices across Europe, including in Bruges, one of the Hanse's main trading posts. A few of the coats of arms are not fully confirmed, Florence among them.
The coats of arms of cities
Here are all 28, grouped by where the cities are:
The German Baltic coast
: Lübeck, Wismar, Rostock and Stralsund
Lower Saxony and Bremen
: Bremen, Lüneburg, Hannover, Braunschweig, Goslar and Osnabrück
Saxony-Anhalt and Berlin
: Salzwedel, Stendal, Magdeburg and Berlin
Westphalia and the Rhineland
: Münster, Soest, Cologne and Aachen
The Netherlands
: Kampen, Stavoren, Amsterdam and Utrecht
Around the Baltic Sea
: Danzig (Gdańsk, Poland), Toruń (Poland), Königsberg (Kaliningrad, Russia), Riga (Latvia) and Tallinn (Estonia)
Italy
: Florence
The senators' families
Below the balcony that runs under the emperors, above the arched ground-floor windows, there is another set of coats of arms carved into the stone. These belong to Hamburg's own families. Until 1919, Hamburg was run by a Senate made up of men from the city's richest and most powerful families. They were chosen for life, not by a public vote, and 29 of these families have their coats of arms here.
The coat of arms of families
You might not know their faces, but you probably know their names, because many of them live on in the city's streets, squares and parks:
Mönckeberg
and
Petersen
: the two mayors we already met among the busts.
Versmann
: Mayor Johannes Versmann, who led the talks that brought Hamburg into the German customs union in 1888. The Versmannstraße in HafenCity is named after him.
Hertz
: Senator Gustav Ferdinand Hertz was the father of the scientist Heinrich Hertz, after whom Hamburg's TV tower, the Heinrich-Hertz-Turm, is named.
Hachmann
: Mayor Gerhard Hachmann, after whom the Hachmannplatz next to the main station is named.
Burchard
: Mayor Johann Heinrich Burchard, after whom the Burchardplatz next to the Chilehaus is named.
Lappenberg
: Johann Martin Lappenberg, who looked after the city's old records and wrote its history. The Lappenbergsallee in Eimsbüttel is named after him.
Hayn
: Mayor Max Theodor Hayn, whose summer garden is today's Hayns Park in Eppendorf, where about 130 Alster swans spend the winter.
Gossler
: A family of bankers connected to the Berenberg bank, one of the oldest banks in the world. Goßlers Park in Blankenese was once the family's land.
Amsinck
and
Sillem
: Two trading families from the Netherlands, remembered in the Amsinckstraße and the Sillemstraße.
The others are the Abendroth, Bartels, Chapeaurouge, Geffcken, Kähler, Kunhardt, Lehmann, Merck, Möring, Moller, O'Swald, Roosen, Roscher, Schemmann, Schröder, Stahmer and Stammann families.
The coat of arms of families
Hayns Park is one of my favourite parks for a walk on a nice day. I usually go there after stopping by the French café
Éclair au Café
. I had always wondered where the park's name came from, and now I know.
The saints on the gables
Higher than anything else except the tower, nine saints stand on top of the large roof gables. Two of them hold the front corners facing the Rathausmarkt, St. Michael on the Große Johannisstraße side and St. Catherine on the Alter Wall side, and the others are spread around the sides and the courtyard. They are also the hardest figures on the building to see. From the ground they are just small shapes against the sky, and only at 8x zoom do they turn into figures with faces and the objects they hold.
Hamburg had been Protestant for more than 350 years when the Rathaus was built, so the saints aren't there to be prayed to. They stand for places. For centuries the city was divided into parishes, areas that each belonged to one church, and these parishes were also the basis of its politics: citizens met, voted and paid their taxes by parish. So the nine saints are really a map of old Hamburg.
Five of them stand for the city's five main churches, which you can still visit today: St. Peter for St. Petri, Hamburg's oldest parish church; St. Nicholas for St. Nikolai, the church of the merchants and sailors, whose ruined tower is now a memorial to the victims of war; St. Catherine for St. Katharinen by the harbour; St. James for St. Jacobi; and St. Michael for St. Michaelis, the "Michel", Hamburg's best-known church.
St. Peter, St. Nicholas, St. Catherine, St. James and St. Michael
The one that surprised me most was St. Michael. I know the Michel well: I've passed it many times and even lived close by for more than a year. But I had never imagined the saint behind the name, an archangel with big wings.
Two more stand for the old suburbs outside the city walls, St. Pauli and St. Georg. Both are still districts today, and it's interesting how far they have moved from the saints who gave them their names. St. Pauli, named after the apostle Paul, is known for the Reeperbahn and its nightlife, and St. Georg, named after the knight who killed the dragon, is known as the heart of Hamburg's gay scene.
St. Paul and St. George, followed by St. Mary Magdalene and St. John
The last two stand for monasteries that are long gone: St. Mary Magdalene for the monastery that stood where the stock exchange is today, and St. John for the St. Johannis monastery near today's Rathaus, which still gives its name to the Große Johannisstraße.
The Orators and Justitia
The Rathaus is home to two different bodies: the Bürgerschaft, the city parliament, and the Senate, the city government. The parliament sits in the wing on the Große Johannisstraße side, and the Senate in the wing on the Alter Wall side. The figures on the two side façades show what each of them is about.
On the parliament side stand two orators, both made by the sculptor Friedrich Offermann. One is a young man in a Roman robe with the words "Plus ultra", meaning "More beyond". He stands for progress. The other is an old man with a book and a document and the words "Parta tueri", meaning "To protect what has been gained". He stands for conservatism. Together they show the two sides of every debate in a parliament: moving forward and keeping what already works.
The Orators
On the Senate side sits Justitia, the figure of justice, also by Offermann. Today we would expect justice to sit with a court, not with the government. But until 1860, the Senate was not only Hamburg's government but also its highest court, so justice belonged right there.
Justitia
The courtyard
Behind the Rathaus is a courtyard it shares with the stock exchange, open to everyone from the Große Johannisstraße and the Alter Wall. It's much quieter than the Rathausmarkt, and the figures here tell an older story.
The early founders
In niches about 12 metres up stand six men who shaped Hamburg long before it became a free city: three archbishops, who made it a centre of the church, and three rulers, who helped it become a trading town. Some of them we already met through the emperors on the front.
Ansgar
(archbishop 831–865): The first archbishop of Hamburg, appointed under Ludwig the Pious and sent to bring Christianity to Denmark and Sweden. He is often called the "Apostle of the North".
Adaldag
(archbishop 936–988): He led the archbishopric of Hamburg and Bremen for over 50 years, with the protection of Otto I.
Adalbert
(archbishop 1043–1072): Heinrich III's close adviser, under whom Hamburg-Bremen became one of the most powerful church seats in the empire.
Henry the Lion
(Duke of Saxony 1142–1180): Hamburg lay in his duchy, and he helped the trading towns of the north grow, above all Lübeck.
Adolf III of Holstein
(count 1164–1203): Around 1188 he started a new settlement for merchants next to the old town, and he is the count who got Barbarossa's promise of free trade on the Elbe.
Adolf IV of Holstein
(count 1224–1239): He ended Danish rule over Hamburg at the Battle of Bornhöved in 1227 and founded the Maria-Magdalenen monastery, the one remembered by St. Mary Magdalene on the gables. Later he became a monk himself.
Higher up, on four smaller gables, stand bronze shield-bearers, most of them in 16th-century armour, each holding a halberd and a shield. The round medallions below them carry four names: Bruegge, Nowgorod, London and Bergen. These are not member cities but the Hanse's four big trading posts abroad, called Kontore. In each of them, Hanseatic merchants had their own rules and trading rights, and everywhere except Bruges, their own walled quarter with warehouses and living quarters.
Bruges
: The trading centre of Flanders, where the merchants bought Flemish cloth and met traders from France, Spain and Italy.
London
: The Steelyard on the Thames, where they traded English wool and cloth, until Queen Elizabeth I closed it in 1598.
Bergen
: Bryggen, the harbour quarter at the centre of the trade in dried cod from northern Norway. Its wooden buildings still stand today.
Novgorod
: The Peterhof, where they bought furs and wax from Russia, until Ivan III of Moscow closed it in 1494.
In the left corner of the courtyard is a doorway that was meant for weddings. Above it sits Hymen, the god of marriage, with two torches, and on either side stand Adam with a dog and Eve with an apple and the snake. The figures were made in 1896 for a wedding office that was supposed to move into the Rathaus, but it never did, so the gate has never been used for a single wedding. Couples can marry in the Rathaus today, in the Phoenix Hall once a month, just not through this door.
The wedding door
In the right corner, above the kitchen window, sits a small carved kitchen boy, licking a spoon and holding a fork. It's one of the smallest details on the whole building and very easy to walk past.
The kitchen boy with a spoon
In the middle of the courtyard stands the Hygieia fountain. It was built in 1895–1896 to remember the cholera epidemic of 1892, when more than 8.000 people in Hamburg died, mostly because of unclean drinking water. The first plan was to show Mercury, the god of trade, but after the epidemic the city chose Hygieia, the Greek goddess of health, instead.
Hygieia stands at the top, holding a small bowl of clean water, with a dragon at her feet that stands for the defeated cholera. Around the basin below sit six bronze figures showing how people use water, such as a man with a ship for shipping, a man with a fish for fishing and a woman with a jug for drinking water.
The Hygieia Fountain
The best detail is hidden in the base. Between the figures hang bronze faces, each above an arched opening with a grille. These openings are the air intakes of the Rathaus ventilation system: air was drawn in over the water, which helped cool the building. So the fountain is not only a memorial, but also part of the building's air conditioning.
Standing in the courtyard, I always imagine it busy with senators and merchants, talking next to the fountain between a meeting in the Rathaus and a deal at the stock exchange. You have to admire how transparent the city is: the government and the stock exchange share one courtyard, so at least nobody can say they didn't know whose interests the decisions were made for.
Still more to see
Even after all this, there are many details I haven't covered, and I haven't even gone inside: the Rathaus has 647 rooms, and its halls and staircases are full of paintings and carvings that would need an article of their own.
I'm not sure there is another town hall in Germany that tells the story of its city in this much detail, from the emperors and saints to the fishermen, bakers and a kitchen boy licking a spoon. Almost every figure was chosen for a reason, and many of them point to something you can still see in Hamburg today.
While researching this article, one book kept coming up again and again:
Das Hamburger Rathaus
by Heinz-Jürgen Brandt, from 1957, which describes the building's history, architecture and decoration in detail. When I tried to buy it, I found out it's no longer in print, and in the end I ordered a second-hand copy for a much higher price, which I'm really looking forward to reading. There is a newer book,
Das Hamburger Rathaus: 125 Jahre – 125 Geschichten
by Rita Bake and Michael Zapf, published for the building's 125th birthday in 2022, but I'm not sure it covers the façade in as much detail as the old one.
This wasn't the first time I've had this problem. Whenever I want to go deeper into a part of Hamburg's history, the best books are often out of print, and second-hand shops are the only way to find them. When I first wanted to learn about the city's history, I couldn't even find a single book written in English.
I'm not sure the city is aware of this, but I believe that if these stories aren't told, they will slowly get lost. And if people don't know the stories behind the places they pass every day, they probably won't feel as connected to their city. This article is my small attempt to keep some of them alive.
I want to give people a little bit of insight into what the optimization process looks like for Godot’s renderer. From
the outside, rendering optimization can look a little bit like black magic. I want to show a few examples of some
optimizations that we did this year to highlight that optimizing rendering code isn’t as scary as it sounds.
Bottlenecks
In this article I will only be looking at optimizing CPU code. When working on renderers, you need to be very careful
about optimizing code both for the CPU and for the GPU, because your ultimate performance will be dictated by the slower
of the two processors. No amount of CPU optimization will save you from inefficient shaders.
In writing a renderer you also make tradeoffs all the time to benefit one or the other. For example, batching in 2D is a
technique that hurts GPU performance a little, but improves CPU performance a lot. Since 2D games tend to get CPU
bottlenecked before they get GPU bottlenecked, batching ends up being a net positive in most cases. Conversely, 3D games
tend to be GPU bottlenecked more often, so in Godot we do our occlusion culling on the CPU in order to take some of the
load off the GPU.
Regardless of your ultimate balance between CPU and GPU work, it is always beneficial to optimize the engine since any
optimization to the engine leaves more space for game developers to use the CPU and GPU and saves battery power for
power-constrained platforms.
Methodology
The process of optimizing CPU performance roughly looks like:
Identify the performance bottlenecks/hotspots
Understand why the bottleneck is where it is
Investigate solutions
Re-measure performance
Repeat
Identify the performance bottlenecks
This is the hardest part of performance optimization. When your game or application starts running poorly it can be
difficult to track down exactly why. Often once you see where the poor performance is coming from, you immediately
understand why performance is bad. This is often the case when you accidentally call an expensive API in a hot loop, or
you leave in some debug code that should never have seen the light of day.
For us, as engine developers, we have the added challenge of needing to optimize for a wide range of potential games. We
don’t have just one target, we have thousands of moving targets and we only know there is a serious problem when people
make bug reports. That being said, we can identify and make many improvements just testing using our existing demos and
open-source games.
The best tool for identifying CPU bottlenecks is a CPU profiler. They come in many shapes and sizes. Godot even has two
built into the editor (one for profiling GDScript code and another for profiling the renderer exclusively). However,
when working on engine code itself, you need to use an external profiler. Godot has support for building with
Tracy
to do tracing profiling,
but you can also use an external sampling profiler.
My tool of choice is
Superluminal
which is an external sampling profiler. It attaches to a
running instance of Godot and records profiling data as you run the engine. Then it provides a very nice visualization
of what code is running when. It also recently got
Linux support
! The
examples below in this article will be using Superluminal.
Understand the bottleneck
As mentioned above, sometimes it is easy to understand the bottleneck. Especially when the poor performance is due to a
bug and not a poor design choice. Oftentimes understanding the bottleneck requires understanding the surrounding code.
Ultimately performance optimization boils down to “don’t do things that don’t need to be done”. Understanding the code
and understanding the system is very helpful to understanding what things need to be done and what things are
unnecessary.
Investigate solutions
Typically, once you understand the bottleneck, the solution will seem obvious. But sometimes you need to try a few
different things before you figure out exactly what is required. For example, you might see a heavy loop with a lot of
calculations in it and initially think the solution is to do less calculations per iteration. But it could turn out that
the problem comes from memory bandwidth, so re-structuring your data might yield better performance gains.
Ultimately, you will make changes that directly reduce the impact of the bottleneck.
Re-measure performance
You always need to re-measure. Computers are very complex, game engines are very complex, sometimes the thing you think
will be faster is not faster. Optimizations are often counter-intuitive. For example, adding a branch in a for loop to
skip some calculations intuitively seems like it will always be faster, but in many cases it stops the compiler from
vectorizing the loop and thus leads to worse performance. Or, a more common example, you may cache a value to avoid
calculating something twice, but if your bottleneck in that code path is memory reads, then caching the value will make
your code slower and you would have been better off doing the work twice.
Unless you are an absolute C++ compiler genius, you likely can’t guess the exact result of your code change on
performance. So it is always better to re-measure and confirm your results.
Case Study #1: Polygon2D
This PR came out of a conversation I had with Aurélien Condomines who is working on
Heidi’s Legacy: Mountains Calling
. He noted that he found the animation code
in Godot to be very slow which was limiting his ability to add content to his game. He found that by having 20 or so
animated characters, performance would start to fall off in his game.
I was surprised at those numbers and felt that something was wrong. Our animation code shouldn’t be a bottleneck with
only 20 or so animated characters so I decided to investigate. I asked him to put together a minimal project with a
bunch of animated sprites on screen so I could profile it.
This first hint is that he was using an AnimationPlayer to animate the vertices of a Polygon2D that was displayed on a
Viewport on a Sprite3D in order to have nice looking 2D animations in a 3D world.
That alone sounds a bit unique, but it isn’t something that should cause any significant problems. So our next step is
to boot up Superluminal and see what is going on.
I captured a few seconds of the demo running and then zoomed in on a frame.
Note: all testing of this demo scene is done with a Ryzen 5 9600X CPU.
Since it isn’t totally clear in the screenshot, here is a breakdown of where we are spending our time:
4.6 ms
: AnimationMixer. That is slower than I would like to see, but there are a lot of animation tracks in this scene since every vertex is being animated, so it isn’t totally unexpected.
15.7 ms
:
Polygon2D::_notification()
. I will say more about this below.
11.7 ms
: Drawing the scene. Of this
3.3 ms
is spent creating vertex arrays, and
5 ms
is spent freeing vertex arrays. So we only spend about
3.4 ms
actually rendering the scene.
With just this high-level overview, we can immediately tell that there are two things going wrong:
Polygon2D is doing something unexpected.
Whatever it is, it is resulting in vertex arrays getting created and freed every frame.
So let’s take a close look and see where the time is spent in Polygon2D. We have two options, we can just zoom in on the
timeline, which looks like this:
Or we can use the call graph below to get a more detailed overview:
The call graph orders the function calls inside the chosen function by how expensive they are. You can see that most of
the time is spent adding a new surface and freeing the previous one. That aligns with what we already saw in the
rendering code. Now we know that the Polygon2D class is responsible for the poor performance even at rendering time.
Now, we can look over at the “Source and Disassembly” panel to see exactly what lines of code are responsible:
This code is responsible for building the internal mesh for the Polygon2D. This profile is telling us that this code is
running every frame and it is very slow. After putting everything together, its no surprise the mesh is being rebuilt
every frame, since the vertices are changing on the CPU every frame!
Based on our profile result, we know that is a really slow process. So the question is, can we do something about it?
Of course!
Godot exposes a low-level API to manually update vertex data. This can be extremely helpful when you know the number of
vertices in a mesh is going to stay the same. Instead of freeing the old mesh, allocating a new one, and then uploading
the vertex data to the new mesh, we can simply upload the vertex data to the existing mesh.
That’s what I ended up doing in
my pull request
. Most of the work in
the PR was identifying which cases were suitable for updating the mesh, and which cases needed the mesh to be recreated.
This optimization roughly tripled the performance of animated Polygon2Ds, making the technique that Aurélien was using
totally viable.
Taking a look at a single frame after my PR, we have gone from
35 ms
per frame to just
13 ms
:
Rendering now accounts for about 50% of the frame time, animation takes about 30% and the Polygon2D update takes about
20%.
This same bit of code could be optimized much further of course. For one, we are re-uploading the entire mesh data each
frame that any vertex moves. A potential optimization would be to only update the region of the mesh that changed. That
way if you have an animation that only impacts a few vertices, you don’t pay the cost for the full upload.
However, such an optimization would need to be guided by a game or scene that would actually benefit from partial
updates. This game basically animates every vertex every frame, so the tracking required to do partial updates would
likely be a net negative. Remember, to properly optimize something you need to be able to test your optimization! If you
don’t have a test case, you can’t do an optimization properly.
Ultimately, the small and safe optimization I did yielded enough performance gains to go from 28 FPS to 83 FPS on my
device. So it was enough for now. If someone was really interested and had the time, performance could be pushed
further. Personally, I’m most curious about where we are spending time in the rendering frame now since that accounts
for 50% of the frame time.
Case Study #2: Optimizing SPIRV to DXIL Transpilation
Earlier this year,
Asilkan
was investigating our shader compilation pipeline on the D3D12 backend. Our current method for compiling shaders with D3D12 is described in this
earlier blog post
from
Pedro
. In short it is:
Compile GDShader to GLSL using built in shader compiler
Compile GLSL to SPIRV using GLSLang
Transpile SPIRV to DXIL using Mesa’s NIR converter
Submit DXIL to GPU driver to be compiled to binary
By far the slowest part of the process is the transpilation step. Further, it leads to a noticeable difference in
loading times when switching from Vulkan to DXIL since Vulkan can use SPIRV directly.
To optimize this, Asilkan ran a trace through Superluminal and analyzed the results. From a very high level you can
already see that there are significant gaps where only a couple of threads are active. Gaps in thread execution mean
that things are taking longer than they theoretically should. Ideally our loading process would fully saturate all cores
in order to load as fast as possible. That’s the goal anyway.
Right off the bat, you can see something is wrong. Green means the thread is doing work. The height of the green bar is
basically how hard the thread is working. Red means the thread is stalled waiting for something.
The beautiful thing about a profiler like Superluminal in a situation like this is you can zoom in and see what is
happening. Here, Asilkan was able to see that these gaps were all caused by threads requesting more memory from the OS.
While the work was heavily multithreaded, all threads ended up waiting on each other while they waited on the OS to
allocate more heap memory.
So what if we just didn’t do that?
What Asilkan ended up doing is giving each thread their own heap and getting them to allocate from that
instead of frequently asking the OS for more memory from a single global heap. The end result is the threads are no longer stuck waiting for the
OS and their execution now looks like:
This optimization saved 11 seconds of load time in the TPS demo. Those 11 seconds were purely wasted time where the CPU
ended up stalled doing nothing.
Conclusion
I hope this post has helped show what we actually do to optimize components of the renderer. The process isn’t super
complicated, especially when you have the right tools on hand. In both cases the solution was very apparent once the
problem was properly identified.
F
or more than a century, the mining town of Lead, in the Black Hills of
South Dakota
, attracted workers with a taste for gold. Now, a flagship experiment being built in the old mine is drawing scientists in search of more intriguing riches: answers to the deepest mysteries of the universe.
Construction of the Deep Underground Neutrino Experiment (Dune) in the former Homestake mine is due to start on Monday. Engineers will begin assembling four giant detectors inside caverns blasted from rock a mile underground.
Once switched on, the detectors will quietly spark and flicker as ghostly particles called neutrinos hurtle through them. Some will arrive from the sun and distant supernovae, others in an intense beam sent through 800 miles (1,300km) of rock from a government laboratory near Chicago, Illinois.
Project members visit the old mine, where construction is soon to start. Photograph: Landin Burke, Destyn Humann/Fermilab
Observing the neutrinos could uncover entirely new science and shed light on cosmic spectacles, such as the death of stars and the birth of black holes. Scientists are particularly hopeful of solving one of the most enduring puzzles in nature: why everything around us is made from matter and not its “evil twin”, antimatter. In other words, why we, and why anything, is here at all.
Guardian graphic. Source: Dune science, UKRI Science and Technology Facilities Council, Fermilab. *Sandford Underground Research Facility. Note: curve of Earth to scale, all other elements exaggerated. 800 miles = 1,300km
“It’s a super exciting time to be working on this,” said Dr Kate Shaw, an experimental particle physicist on Dune from the University of Sussex. “Neutrinos do not make sense yet and they could open a door to new physics.”
Neutrinos are the most abundant matter particles in the universe, outnumbered only by photons, the massless particles of light that illuminate the universe. They are produced in natural nuclear reactions, such as those that power the sun, in radioactive decay and in particle accelerators.
Even by physicists’ standards, neutrinos are oddballs. The chargeless particles barely interact with other matter: about a trillion pass unnoticed through your body each second. Neutrinos come in three varieties, all with minuscule but different masses. Weirdly, they change from one variety to another, a process called oscillation, as they travel. “It’s a very strange behaviour,” said Shaw. “It’s like me throwing you an apple and it turns into a banana.”
Lead has a history with these peculiar particles. In the 1960s, the physicist Ray Davis followed miners down the shaft and built a detector from a 100,000-gallon tank filled with dry-cleaning fluid. Shielded so deep beneath the ground, the detector became the first to record neutrinos from the sun as they struck chlorine atoms in the fluid and turned them into argon.
The mine closed in 2001 and transitioned, thanks to visionary forethought, into a dedicated subterranean laboratory, the Sanford Underground Research Facility (Surf). It is already home to the Lux-Zeplin (LZ) experiment which last month reported the
first potential detection
of dark matter.
The Lux-Zeplin dark matter experiment main detector in a surface lab before installation underground. Photograph: Matthew Kapust/Sanford Underground Research Facility/Reuters
“We’ve gone from mining the Earth to helping scientists explore the universe,” said Robin Lucero, Lead’s city administrator, whose family has lived locally for generations. “We’re incredibly proud and lucky that Dune is happening here, and Surf, which has other experiments like LZ. Having world-class science brings new opportunities, new people and new possibilities for our local businesses and especially our young people.”
Unlike Davis’s apparatus, the Dune experiment is enormous. Made from more than 4,000 tonnes of steel, the detectors will stand up to five-storeys high and hold a combined 70,000 tonnes of liquid argon that must be chilled to -200C. Assembling the detectors should take nearly three years. Filling them with liquid argon, another year.
“This is a heavy duty engineering operation,” and a partnership with the skilled local workforce, said Prof David Newbold, of the UK’s Science and Technology Facilities Council and a Dune spokesperson. “It’s like building a ship underground.” The UK is the largest international partner in the US-led experiment and is making core detector components, critical computing to reconstruct events in the detectors, and the graphite target needed to generate neutrinos at Femilab, a particle physics facility in Illinois.
The LBNF-Dune underground site. Photograph: Landin Burke, Destyn Humann/Fermilab
Once operational, Dune will study an intense beam of neutrinos sent from Fermilab through 800 miles of the Earth’s crust and mantle. Most neutrinos will shoot straight through, but every so often one will strike an argon atom, creating a flicker of light and a pulse of electric charge. Analysing these will reveal intimate details about the mysterious particles.
One big question surrounds the masses of different neutrinos. Which is lightest and which heaviest? The answer could reveal fundamental new forces of nature. And in an attempt to answer the matter v antimatter question, Dune will look for subtle differences between neutrinos and their antimatter twins, antineutrinos.
Beyond the beams of particles sent from Fermilab, Dune will also study neutrinos arriving from space, carrying with them fresh information on stellar explosions and the formation of black holes.
Earlier this month, the Nobel prize for physics honoured the Belgian researcher Francis Halzen, who used
a cubic kilometre of Antarctica
to detect high-energy neutrinos from deep space. But detection is far from the end of the story. “This really is going to be a golden era,” Newbold said. “There’s a lot more of that to come.”
Ian Sample
is science editor of the Guardian. Before joining the newspaper in 2003, he was a journalist at New Scientist and worked at the Institute of Physics as a journal editor. He has a PhD in biomedical materials from Queen Mary's, University of London. Ian also presents the
Police use facial recognition to scan one million faces in WA trial leading to 122 arrests
Guardian
www.theguardian.com
2026-11-02 01:12:52
Deployments of surveillance vans over five months resulted in nine false alerts and led officers to engage with 146 registered sex offendersGet our breaking news email, free app or daily news podcastMore than a million people have had their faces scanned in a trial of real-time facial recognition te...
More than a million people have had their faces scanned in a trial of real-time facial recognition technology in
Western Australia
, resulting in 122 arrests and nine incorrect alerts.
WA police said under the five-month trial, which concluded on Saturday, the technology scanned 1,380,882 faces after 100 deployments across 43 regional and metropolitan locations.
Alongside the 122 arrests, the alerts led officers to engage with 146 registered sex offenders and locate 10 people for a welfare check.
The deployments included three operations at the Perth royal show, where 223,749 faces were scanned, resulting in 15 alerts, three arrests and seven “interactions” with registered sex offenders.
A police spokesperson confirmed the system generated nine incorrect alerts against the total 1.38m scans, meaning there was an error rate of 0.00065%.
Addressing the false matches at a media conference on Sunday, the WA police commissioner, Col Blanch, confirmed officers engaged with each individual.
“I want to be very clear on those false alerts,” he said. “No one was arrested. No one made a complaint. Each one did result in an interaction with that person and asking them relevant questions.
“It’s a very, very, very low number. It’s not zero. We’d love it to be zero. But I think the important thing for us is to engage with our community just like we would if a human thought someone was a wanted person.”
Blanch acknowledged the technology was “not perfect” and that authorities “knew this going into this trial”, but maintained that officers followed up on alerts in the same way they would if a member of the public raised suspicions.
However, civil liberties advocates have criticised how the WA police framed the error rate.
The chief executive of Civil Liberties Australia, Bill Rowlings, said WA police were “severely gilding the lily” by claiming the incorrect alert rate was 0.00065% and suggested it should instead be calculated by comparing how many of the arrests resulted in convictions.
“We will only know what the incorrect rate is when the 122 people arrested have their day in court,” he said.
“If half the 122 arrested … are not convicted, the ‘incorrect’ rate will be 50%,” he said in a statement, saying police had performed the almost 1.4m scans “with no reasonable suspicion whatsoever”.
Sarah Schwartz, legal director at the Human Rights Law Centre, said every false match is “a real person who may be stopped, questioned or investigated because an algorithm got it wrong”.
“The consequences of those mistakes have real consequences for the people of Western Australia.”
She questioned if it was “proportionate in a democratic society” to conduct “biometric surveillance on an extraordinary scale” of more than a million people – and raised particular concern over the deployment of surveillance vans at public demonstrations.
When the trial was announced, the state’s independent privacy watchdog, the office of the information commissioner WA (OIC WA), said it was not involved in implementing the trial, warning of “collective privacy harms”
in a statement
. The OIC WA was contacted for comment.
The police surveillance vans have now been removed from public deployment while the trial is evaluated.
Photograph: Aaron Bunch/AAP
On Sunday, Blanch said the trial “focused on using modern technology responsibly to identify people who presented a risk, while allowing the vast majority of people to go about their day without any impact”.
Under the trial guidelines, cameras mounted on or positioned near marked police vehicles scanned faces in designated public areas, matching them against an alert list drawn from police databases. Police said images of people who weren’t on the list were pixelated in real time and deleted immediately without being stored.
The trial previously
triggered backlash
over concerns about privacy and algorithmic bias. Experts say the technology could disproportionately affect marginalised groups who are already overrepresented on police watchlists, with facial recognition algorithms historically having higher false-match rates for people with darker skin tones.
Schwartz said the governance framework surrounding the technology remained opaque.
“WA police have provided remarkably little detail about the safeguards governing this technology,” she said. “Their publicly available policies on the use of this technology appear heavily borrowed from overseas models, while leaving critical questions unanswered about misuse, bias and accountability.”
The surveillance vans have now been removed from public deployment while the force conducts an internal evaluation into how the technology performed on the ground and its community impacts, including “reviewing the safeguards and whether all requirements were met”.
Only Israel’s Palestinians Vote Against Genocide
Portside
portside.org
2026-11-02 00:52:45
Only Israel’s Palestinians Vote Against Genocide
Ira
Mon, 11/02/2026 - 00:52
...
Only Israel’s Palestinians Vote Against Genocide
Published
Yousef Jabareen, leader of Israel's Arab-led Joint List political alliance, casts his ballot in the town of Umm al-Fahm on October 27, 2026. | Ahmad Gharabli / AFP via Getty Images
t was heralded by many as “the most consequential
election
in Israeli history,” and by rights it should have been. Under Benjamin Netanyahu’s outgoing far-right government, Israel plunged into a severe constitutional crisis, suffered the worst military setback in its history, carried out a genocide in Gaza and two failed wars in Iran, and become the
most hated
country in the world.
And yet, as
Haaretz
editor Aluf Benn noted in his
postmortem
, Israeli Jewish voters proved themselves imperturbable. Support for aggression throughout the region remains nearly universal among Jewish voters, who have once again sorted themselves again into pro- and anti-Netanyahu blocs on a basis that, while reflecting real
class divisions
, is expressed in familiar culture-war terms, producing yet another frustrating stalemate.
There is, however, one dramatic exception to this uncanny stasis: Israel’s Palestinian minority, which responded to unprecedented levels of racist incitement with the highest turnout on record.
Of over eight hundred thousand votes cast by Palestinian citizens, 37 percent went to the Islamist United Arab List (UAL, also known as Ra’am), and 51 percent to the Joint List, which unites communists, left-liberals and independents. As a result, nineteen out of 120 seats in the Knesset, which are allocated proportionally, have gone to Israel’s only two non-Zionist parties, each of which also has one Jewish representative. (Support for these parties among Jews is numerically insignificant, though hundreds of Jewish
academics
signed a petition in support of the Joint List.)
In stark contrast to the violence and malaise that have gripped Israel’s Palestinian community following the rise of state-tolerated
organized crime
, the contest between the two parties was peaceful and restrained. This is all the more remarkable given the clear ideological divide between them.
The Joint List couples the demand for civic equality for Palestinian citizens with a clear
rejection
of Israel’s annihilationist policy in Gaza and its ethnic cleansing in the West Bank. Meanwhile, the UAL — a prominent member of the
original
Joint List — has turned to a sectoral strategy
modeled
on that of Israel’s ultra-Orthodox Jewish parties. Following his 2021 split from the Joint List, UAL leader Mansour Abbas
courted
Netanyahu and then served as a junior partner in the government that briefly ousted him in 2021–22.
This time around, the anti-Netanyahu bloc has drawn the bar for cooperation at a level meant to exclude even the UAL. Its leader du jour, Lieutenant General Gadi Eisenkot, clarified that any party seeking to join him in government would have to declare Hamas — which still is hanging onto power in Gaza — a “terrorist organization which must be annihilated.”
As Israel Defense Forces chief of staff, Eisenkot elaborated the infamous “Dahiya doctrine,” which recommends mass killing of civilians in areas dominated by hostile forces. In the current context, his shibboleth cannot be read as anything but a demand for active consent to ongoing genocide.
The hostility of the Zionist opposition to any form of solidarity with Gaza was driven home when all its parties, including the liberal
Democrats
, backed an election board decision to bar Joint List candidate Sami Abu-Shehadeh from running on the basis of a vaguely celebratory article written on October 7 and subsequently repudiated. (Genocidal expressions by right-wing candidates, it goes without saying, were not censured in the same way.)
Against Abstention
G
iven the depth of the racist rejectionism Palestinian citizens have suffered from both Zionist blocs and the eager complicity of both in colonial violence, a turn to abstention would not be unexpected. But Israel’s Palestinian citizens do not have the luxury of
ignoring
the differences between these blocs, small as they may be.
Thus, Netanyahu’s finance minister and West Bank “viceroy” Bezalel Smotrich has empowered settler militias to carry out a campaign of ethnic cleansing that might be halted, if not reversed, under an Eisenkot administration. His ghoulish national security minister, Itamar Ben-Gvir, has made a fetish out of
torturing
and
executing
Palestinian prisoners.
High Palestinian turnout has severely weakened extreme-right parties led by the two and
prevented
two others from crossing the representation threshold, depriving Netanyahu’s coalition of its majority. Voters are justified in hoping a new government will dial back its ferocity on both fronts as well as tackling organized crime more seriously. For Palestinians, these are questions of life and death, if not of imminent decolonization.
Under these complex circumstances, voters have rewarded the Joint List for striking a balance between principle and pragmatism. Sociologist Maha Sabbah-Karkabi, who will be joining the Knesset as a Joint List representative, stressed the importance of Palestinians’ show of collective power. “By fully utilizing their electoral potential,” she told me, they “have repositioned themselves at the center of Israel’s political landscape.”
However, Sabbah-Karkabi added, the Joint List now confronts a “climate of delegitimization of Arab political participation coming from different parts of the political spectrum,” rendering the question of influencing government policy “much more complicated.”
Delegitimization has come swiftly from Netanyahu’s propaganda organs, which quickly coined the felicitous oxymoron “
ballot box terrorism
” while weakly claiming Qatari interference on the Joint List’s behalf. (Netanyahu’s advisers, of course, are the ones who have
worked
for Qatar.) But while Eisenkot and most of his allies obstinately continue to refuse any alliance with Palestinian parties, the Democrats are finally beginning to
peel away
from this boneheaded consensus.
For the Joint List, the situation presents opportunities as well as risks. Sabbah-Karkabi and Amjad Shbita, general secretary of the Democratic Front for Peace and Equality (the largest of the Joint List’s components), agree that seeing Netanyahu and his far-right government replaced is an urgent priority. Unlike the UAL’s deputies, however, they are also committed to Palestinian national solidarity across borders. The electorate has endorsed this commitment in no uncertain terms.
The ball is now in the court of Eisenkot’s anti-Netanyahu bloc. To avoid a return election, he will have to break one of two campaign promises: not to enter a coalition with Netanyahu, and to avoid relying on “Arab votes.” If he reneges on the former vow — perhaps under cover of a new war with Iran, conveniently started by Donald Trump and Netanyahu in the coming weeks — he will gift the latter with a renewed sheen of invincibility and the international legitimacy he has done so much to lose.
To end Netanyahu’s rule, Eisenkot would have to break the racist taboo on negotiation with the Palestinian public’s representatives. Though it will not join an Eisenkot government, Shbita told me, the Joint List would be willing to provide it with a parliamentary safety net, preventing Netanyahu from getting another shot at power by bringing the government down with a vote of no confidence.
Even if Eisenkot lacks the courage to negotiate such an agreement, Shbita added, “we would consider enabling a minority government by not voting it down until Netanyahu’s departure from the political scene becomes a certainty.”
Following the “most consequential election in its history,” the political scene in Israel continues to look unbearably bleak. Progressives elsewhere have no reason to let up the pressure for broad sanctions and a total arms embargo. Nevertheless, they can take inspiration from the many Palestinian (and a few Jewish) citizens of Israel, who by voting for the Joint List have expressed their steadfastness —
sumoud
— in the struggle for democracy, justice, and peace.
Dr Matan Kaminer
is an anthropologist, an activist on the Israeli radical left, and a lecturer at Queen Mary University of London.
is a leading voice of the American left, offering socialist perspectives on politics, economics, and culture. The print magazine is released quarterly and reaches 75,000 subscribers, in addition to a web audience of over 3,000,000 a month.
Subscribe to Jacobin
today, get four beautiful editions a year, and help us build a real, socialist alternative to billionaire media.
From 18th-Century False Ballots to the Second JFK and Double Dan Sullivans, a Brief History of Spoiler Candidates and the Unsuccessful Legal Challenges to Them
Portside
portside.org
2026-11-02 00:39:28
From 18th-Century False Ballots to the Second JFK and Double Dan Sullivans, a Brief History of Spoiler Candidates and the Unsuccessful Legal Challenges to Them
Ira
Mon, 11/02/2026 - 00:39
...
The campaign of U.S. Senate candidate Mary Peltola of Alaska, a Democrat, has been rocked by accusations it recruited a spoiler candidate for the ballot. | AP/Mark Thiessen
Less than a week before Election Day, The New York Times
reported
that people with links to Alaska Democrat Mary Peltola’s U.S. Senate campaign and the Democratic Senatorial Campaign Committee had recruited a candidate to the race whose name is nearly identical to that of the incumbent Republican senator, Dan Sullivan, whom Peltola was challenging. The plan aimed to siphon votes from the incumbent senator and help Peltola win, according to the reporting.
19th-century voting usually involved casting a ballot printed by political parties, such as this electoral ticket for the Free Soil Party from 1848.
Smithsonian National Museum of American History
For most of the 1800s, American elections were conducted with ballots designed and printed by the parties. There were no official ballots printed by the state.
In those days, the addition of a rival name to a party ballot by an insidious printer could swing countless votes to the opposing party. The tactic was well known, and voters were often warned to look out for the “
Spurious Ticket
.”
By the 1890s, the government-printed ballot had gained widespread adoption and ended the practice of counterfeit tickets. But that would not be enough to deter those who wanted to deceive voters.
In John F. Kennedy’s first congressional primary, his opponent was a city councilor named Joseph Russo. An aide to Kennedy
reportedly paid a janitor
with the same name to enter the race. The real Russo complained, but in the end, Kennedy took more votes in the 1946 race than the Russos combined.
Eight years later, the name Kennedy worked the other way, when a
Gillette stockroom supervisor named John Francis Kennedy
, no relation, was elected as a Democrat to the position of treasurer and receiver-general of Massachusetts, owing in large part to the familiarity of his name. Names can move votes, so namesakes can be worth recruiting.
Spoilers as surrogates
Some spoiler candidates are not necessarily meant to win votes. Sometimes they are there to amplify a message.
Days before the election, Don Sprouse, an independent running for office for the first time,
held a press conference
and
declared that Heller was unfit
for office as he had “not made clear to the voters he is not a Christian” and was “not as qualified to represent the Christian people.”
Although Campbell condemned the remarks and Sprouse received only 1,693 votes, Heller lost the race. The allegations, still unproven, of Sprouse
being a “stalking-horse”
for Campbell only surfaced years later, but they
continued to dog Campbell
when he ran for governor in 1986.
Sprouse wasn’t going to spoil the election with the votes he could win, but he could be a spoiler with his message, an antisemitic appeal that Campbell would not make directly.
A spoiler candidate who made antisemitic remarks in a 1978 South Carolina campaign for the House of Representatives became a campaign issue eight years later.
New York Times archive
Candidates who don’t even know they’re running
A different type of ballot shenanigan skips the recruitment and simply puts candidates on the ballot.
This was the case in 2010 when two Oakland County, Michigan, Democratic officials put candidates on the ballot
under the Tea Party label
in three races. The candidates didn’t even know they were running.
In this case, the law did come back to bite. The county party chair and operations director were
charged with perjury and forgery
, but the crime was forging signatures, not their strategy.
Some state election codes specifically discuss false declarations or forged election paperwork, but many don’t contemplate this type of scheme. Forgery and perjury, however, are illegal across the country, and it’s almost impossible to put an unwitting person on the ballot without one or the other.
Alaska: No barriers
In Alaska, this tactic can be used cheaply. There is no petition requirement to get on the ballot, just a $100 fee. Moreover, the parties can’t vet candidates in the primary.
The state’s unique top-four primary system
lets candidates choose their own party label.
Petersburg Sullivan filed to run as a Republican in the Senate race just three days before the deadline; state officials tried to bar him from the ballot. In June
a judge said
there was nothing in Alaska law to prevent him from appearing on the ballot, and
the state Supreme Court subsequently agreed.
Since he couldn’t be removed, the state Division of Elections took unusual steps on their ballot design to help distinguish the two candidates. They listed Sen. Dan S. Sullivan as a Republican, marking him as the incumbent, and
left Daniel J. Sullivan without a party designation
. This had not been done for previous incumbents in at least a decade.
What the law can do
Assuming there is no forgery of documents with an unwitting and unwilling spoiler candidate, there is little the law can do. History shows us that the real legal exposure in these cases usually lies where it most often does in politics: money.
In a 2020 Miami, Florida, race, a former state senator turned political operative was
convicted of three felonies
related to efforts to recruit and pay a decoy candidate with the same surname as the Democratic incumbent. In another Florida case, a local Republican Party chair was convicted in connection with a
scheme to plant a progressive alternative
to siphon votes from the Democratic candidate. In both cases the prosecutions focused on campaign finance crimes, not simply on putting the spoilers on the ballot.
The state of Washington is an outlier, where it is
a felony to run under a rival’s surname if the intent is to confuse voters
. In 2024, two Bob Fergusons were recruited by conservative activist Glen Morgan to run for governor against state Attorney General Bob Ferguson. Both opted to drop out of their race under threat of prosecution.
There is nothing comparable to these laws in Alaska, however, and whether laws like this could prevent this type of thing in a federal election is still untested. Under the Supreme Court’s decision in
U.S. Term Limits v. Thornton
in 1995, states cannot add their own qualifications for Congress.
Read your ballots!
The advice
from 1860 – “Watch your tickets on election day. There’s a scheme on foot …” – when
newspapers urged voters
to read their tickets before casting their vote, is still relevant long after the demise of the party ticket. It is unlikely that the law can solve this problem, because to do so would be to wade into the First Amendment waters of why people run for office. Regulating thought and intention is a dangerous proposition under the Constitution.
The old answer is the best one. Self-government depends less on clever rules and more on citizens paying attention. Read your ballots!
Carney Hospital, which closed due to financial neglect from former owner Steward Health Care. | Danielle Parhizkaran / Getty
For such a busy bee, the private-equity oligarch Jeffrey Leerink has really managed to fly under the radar. No major media outlet in the Boston area, where he operates, has seriously investigated his role in any number of headline-grabbing scandals. You’ve probably never even heard of him, but he was a key player in the collapse of Silicon Valley Bank (SVB) and the financing of the now-bankrupt for-profit hospital system Steward Health Care. Now he’s hard at work trying to expand an ecologically disastrous private-jet airport.
In 2019, Leerink sold his private investment bank, Leerink Partners, to the infamous SVB
for $280 million
and stayed on to manage the bank’s securities arm. The rest is history: SVB’s March 2023 collapse and subsequent federal seizure was the largest bank failure since the 2008 economic meltdown. The whole sordid affair was a
textbook example
of a disaster that could have been averted had Congress not capitulated to the bankers that lobbied for deregulation.
As Leerink leaped from the sinking SVB ship, reestablishing Leerink Partners, he plundered from the fields of life sciences and healthcare. He handled financing for Steward Health Care, cofounded by the
oligarch Ralph de la Torre
. A cardiac surgeon, de la Torre was formerly CEO of Caritas Christi, a nonprofit hospital network of six acute-care facilities in eastern Massachusetts owned by the Archdiocese of Boston. In 2010, he brought in the private-equity fund Cerberus Capital to acquire the hospital network for $246 million and convert it to a for-profit system.
Rebranded as Steward Health Care with de la Torre as CEO, it
acquired dozens of hospitals over the next decade
, becoming one of the largest for-profit hospital networks in the US. Like other private-equity ventures in healthcare, Steward squeezed value from patients, workers, and communities, resulting in
patient endangerment
and
bankruptcy
. It sliced and diced assets, putting real estate into a separate venture called Medical Properties Trust (MPT) that charged the hospitals huge rents, contributing to their insolvency.
In 2020, Cerberus sold Steward to a physicians’ group with $335 million in financing from MPT. Steward’s leaders celebrated, paying themselves $111 million in dividends, with de la Torre treating himself to not one but two yachts with his windfall.
Within a year, Steward Health Care was headed toward bankruptcy, unable to afford the real-estate payments to MPT.
The Boston Globe
, which assigned its investigative
Spotlight team
to chronicle the unraveling of Steward, failed to mention the Leerink connection. But he was there.
When the Tennessee-based for-profit operator HCA Healthcare announced its plan to purchase five Utah hospitals from Steward, which ultimately was never completed, Leerink acted as an “exclusive financial advisor to Steward Health Care,” according to HCA’s website. (Leerink’s own website made a similar announcement in 2023, only to delete the post as Steward teetered toward bankruptcy.)
Now Leerink is spearheading an effort to expand the hangar space for luxury private jets at Hanscom Field, New England’s largest private-jet airport, located 20 miles northwest of Boston, which hosts an estimated 38,000 annual private-jet operations. A
study
that I cowrote found that nearly half of private-jet flights departing from Hanscom are for recreational or luxury travel. Leerink’s hangar expansion would facilitate an estimated additional 6,000 flights a year, releasing a “climate bomb” of emissions,
according to Elizabeth Turnbull Henry
, the president of the Environmental League of Massachusetts. To lead the project, Leerink has teamed up with Michael Argiros, a local Cadillac dealer. Neither has any track record in aviation.
A mighty grassroots coalition, Stop Private Jet Expansion at Hanscom or Anywhere, brings together 120 organizations from the region to oppose the expansion. It argues that
private-jet pollution is indefensible on a warming planet
—and that just as we should halt construction of new fossil-fuel infrastructure, we should stop incentivizing private-jet travel, one of the most polluting activities on the planet.
These and other broad objections from local communities and their elected representatives haven’t stopped Leerink and his team from forging ahead with the project. In their 2024 application for a state environmental review, they claimed that their project would actually reduce emissions by decreasing the frequency of “ferry flights,” in which pilots fly short distances to reposition the aircraft and park it. Massachusetts state Senator Michael Barrett lambasted them for this claim at a public hearing: “Please don’t erect these bogus environmental rationales for something that has nothing whatsoever to do with relieving the crisis that faces us in terms of climate. Just be honest about it. This is all about becoming a little richer yourselves by helping people even richer than you.” The activists won the first round of the state regulatory battle, with the Commonwealth of Massachusetts
rejecting
Leerink’s initial environmental plan. At a September 2026 hearing, Massport shared that Leerink and the developers are aiming to submit a supplemental plan and environment statement by the end of 2026.
One of Hanscom Field’s frequent fliers is Leerink’s fellow oligarch John Henry, owner of the Boston Red Sox and
The Boston Globe
. The newspaper has conspicuously avoided probing Leerink’s role in the Steward Health Care debacle or the much-derided hangar expansion. Instead, it has published multiple puff pieces about him, including a
July 2023 article
exalting his recovery after the collapse of SVB. The
Globe
’s business reporter Jon Chesto fawned, “It’s hard to overstate how important Leerink and his team—with their deal-making skills and financial expertise—have been to the emergence of Greater Boston as the world’s epicenter of life sciences.”
Why has Boston’s biggest newspaper shied away from examining Leerink’s newsworthy involvement in the self-dealing of the superrich? We can only suspect that deep solidarity among oligarchs is forged in private-jet passenger lounges across the planet.
Founded by abolitionists in 1865, The Nation has long believed that independent journalism has the capacity to bring about a more democratic and equitable world. Our writers shift paradigms and open minds. Our deep investigative reporting launches congressional hearings, forces policy change, and shapes news cycles. Instigating progress: It’s not only our legacy, it’s our continued commitment to future generations of torchbearers.
On his “Swing Around the Circle” in summer 1866 to drum up support for his party before that year’s midterms, President Andrew Johnson denounced Congress as an illegal body, called certain Republican members of Congress traitors who should be hanged, and compared himself both to his martyred predecessor Abraham Lincoln and to Jesus Christ.
In those days, the Democrats were the party of slavery and secession, while the young Republican Party stood firm on the Declaration of Independence. Johnson was a former Democrat from Tennessee, placed on the 1864 “National Union Party” ticket to emphasize unity in the fight to save the United States, and became an accidental president after Lincoln’s assassination. Johnson had broken from his Confederate neighbors because he was happy to see the end of the institution of human enslavement even though he had enslaved people himself until 1863. He believed slavery enabled a few rich men to monopolize the nation’s wealth, shutting out poorer white men like him.
When he became president in April 1865, in the last days of the Civil War, he supported the ratification of the Thirteenth Amendment to the Constitution. But that was as far as he went with the Republican plans for a future in which every man, regardless of race, would have the right to the fruits of his own labor. Johnson opposed the party’s growing support for Black rights and intended to reconstruct the United States on the basis of white supremacy.
In one of the nation’s most unfortunate series of events, Johnson took office while Congress was out of session. It would not reconvene until December. After Lincoln’s death, Republican members of Congress rushed to Washington, D.C., expecting Johnson would call them into special session as Lincoln had done in summer 1861, but Johnson declined to do so, intending to “restore” the nation on his own terms.
Johnson set out to resuscitate the prewar political system dominated by the Democratic Party, with himself at its head. Over the course of the summer, he pardoned all but about fifteen hundred former Confederates, either by proclamation or by presidential pardon, returning them to positions of authority in southern society.
He required only that southern states ratify the Thirteenth Amendment, declare secession illegal, and default on Confederate debts before they could be readmitted to the Union. Since he pardoned ex-Confederates and refused to endorse Black voting, Johnson guaranteed that the same men who had taken the South out of the Union would be responsible for bringing it back in.
Southern leaders half-heartedly met the president’s demands, then joined him in trying to re-create the world of the Old South. Within each state, through a series of laws together dubbed “Black Codes,” they bound formerly enslaved people to white employers and guaranteed the dominance of white people in the South. In Mississippi, for example, Black men could be arrested and fined if they did not sign yearlong contracts, then “hired out” to anyone who paid their fines—a system that essentially threatened uncooperative Black Americans with new language for an old system of forced labor. Nowhere could a Black person testify against a white person in court, so they had no recourse against theft, rape, assault, or even murder.
Republicans looked on, aghast, as Johnson and white southerners took only months to erase the hard-earned fruits of victory in the Civil War. Adding insult to injury, the 1870 census would count Black Americans as whole persons rather than three-fifths of a person, as they had been counted before the war, meaning the South’s representation in the House of Representatives would increase. If Johnson’s plan were permitted to go forward unchallenged, wealthy southern whites would be more powerful after the Civil War than they had been before it.
When Congress reconvened in early December, Johnson cheerfully informed the members that the country had been reconstructed. All that remained to do, he announced, was for Congress to admit newly elected southern representatives—including many of the same men who had taken the South out of the Union—to their seats. They were already in Washington, marveling at the changes the war years had wrought on what only four years before had been a sleepy southern town.
Republicans had no intention of allowing the South to win the peace after losing the war.
Congress refused to seat the South’s representatives and appointed a committee of fifteen members to come up with a better plan than Johnson’s for rebuilding the nation. The Committee of Fifteen spent months hearing testimony from southerners, white and Black, about conditions in their war-torn region. Finally, on the last day of April 1866, members of the committee introduced the joint resolution that would become the Fourteenth Amendment to the Constitution.
The amendment recognized that every man was equal before the law and that any person born or naturalized in America—except “Indians not taxed”—was an American citizen. To address the problem of the upcoming 1870 census, the committee proposed that congressional representation for the southern states be based on population, as with all states, but that southern representation be reduced in proportion to the number of men a state refused to allow to vote.
The committee prohibited from holding elected office anyone who had sworn to support the Union and then broken that oath in order to join the Confederacy. The amendment also closed loopholes that southern states had found to protect their war debt. Finally, like the amendment that preceded it, the Fourteenth Amendment increased the strength of the federal government by giving Congress the power to enforce its provisions.
Both chambers of Congress agreed to the amendment on June 18, 1866, and sent it off to the states for ratification. Congress required southern states to ratify the amendment before being readmitted to the Union.
Johnson hated the proposed Fourteenth Amendment. He hated its broad definition of citizenship; he hated its assertion of national power; he hated its disfranchisement of the southern leaders he backed. Most of all, though, he hated that it offered a moderate route to reunification that could be supported by most Americans. If it were ratified by the states, it would destroy his chances of rebuilding a dominant national Democratic Party.
Johnson told southern politicians to ignore Congress. Democrats would win the 1866 midterm congressional elections, he assured them, and once back in power, they would reject Republican “radicalism” and allow Johnson’s plan for reconstruction of the Union to proceed.
As he asserted his vision for the country, Johnson egged on white supremacist violence. In July, white mobs attacked a Unionist convention in New Orleans where delegates had called for taking the vote away from ex-Confederates and giving it to loyal Black men. The rioters killed thirty-seven Black people and three white delegates to the convention.
By then, Johnson had become as unpopular as his policies. Increasingly isolated, he defended his plan for the nation as the only true course. In late August he broke tradition to campaign in person, an act at the time considered beneath the dignity of a president. He set off on a railroad tour, known as the Swing Around the Circle, to whip up support for the Democrats before the election.
Speaking from the same set of notes as the train stopped at different towns and cities from Washington, D.C., to New York, to Chicago, to St. Louis, and back to Washington, Johnson complained bitterly about the opposition to his reconstruction policies, attacked specific members of Congress as traitors and called for them to be hanged, and described himself as a martyr like Lincoln. And, noting the mercy of his reconstruction policies, he likened himself to Jesus.
It was all too much for voters. The white supremacist violence across the South horrified them, the restoration of power to southern whites infuriated them, the reduction of Black soldiers to quasi-slaves enraged them, and Johnson’s attacks on Congress alarmed them. Johnson seemed determined to hand the country over to its former enemies to re-create the antebellum world that northerners had just poured more than 350,000 lives and $5 billion into destroying, no matter what voters wanted.
Johnson’s extremism and his supporters’ violence created a backlash. Northerners were not willing to hand the country back to the Democrats who were rioting in the South and to a president who compared himself to Jesus. Rather than turning against the Republicans in the 1866 elections, voters repudiated Johnson. They gave Republicans a two-thirds majority of Congress, enabling them to override any policy Johnson proposed.
And, in 1868, the states ratified the Fourteenth Amendment to the Constitution, launching a new era in the history of the United States.
Historians are fond of saying that the past doesn’t repeat itself; it rhymes.
To understand the present, we have to understand how we got here.
That’s where this newsletter comes in.
I’m a professor of American history. This is a chronicle of today’s political landscape, but because you can’t get a grip on today’s politics without an outline of America’s Constitution, and laws, and the economy, and social customs, this newsletter explores what it means, and what it has meant, to be an American.
These were the same questions a famous observer asked in a book of letters he published in 1782, the year before the Treaty of Paris ended the Revolutionary War.
Hector St. John de Crevecoeur called his book “Letters from an American Farmer.”
Like I say, history doesn’t repeat itself, but it sure rhymes.
Patrick Stewart, Actor: “There’s Nothing More Fun Than Being at a Dinner Party in Beverly Hills, Surrounded by Elegant People, and Telling Them You’re a Socialist”
Portside
portside.org
2026-11-01 23:48:48
Patrick Stewart, Actor: “There’s Nothing More Fun Than Being at a Dinner Party in Beverly Hills, Surrounded by Elegant People, and Telling Them You’re a Socialist”
Marti
Sun, 11/01/2026 - 23:48
...
Patrick Stewart, Actor: “There’s Nothing More Fun Than Being at a Dinner Party in Beverly Hills, Surrounded by Elegant People, and Telling Them You’re a Socialist”
Published
Patrick Stewart | EFE
Patrick Stewart
, an actor recognized for playing Professor X in ‘X-Men’ and Captain Jean-Luc Picard in ‘Star Trek’, died at the
age of 86
this past Thursday, October 29. The actor’s death occurred after battling Alzheimer’s disease and Lewy body dementia.
The death was announced through the British actor’s own social media. It was reported that
“he was surrounded by the warm embrace of his family and friends,
surrounded by live music, prayers, tears, laughter, and that deep emotion that only he was capable of inspiring, after a lifetime of devotion to all of us”.
Now that only his memory remains, many users have shared some of his most striking reflections. Whenever he has had the opportunity, Stewart has defined himself as a “socialist.” On Jimmy Kimmel’s show, he opened up and explained the reasons that led him to have this political leaning. The origin, according to his account, came
when he was barely five years old, after the Second World War.
Patrick Stewart, on his view of socialism in the United States: “It’s a lot of fun”
“My first act of civil disobedience took place when I was five years old, during the general election held in 1945, after the Second World War. To everyone’s surprise, Winston Churchill lost the election and Clement Attlee came to power. Since then,
I have been a man of the left”
, he said.
In addition, he shared an anecdote that sparked the audience’s laughter: “This political leaning can sometimes cause one problem or another in the United States, but it has also given me some very amusing moments.
There is nothing like sitting down at the table at a dinner in Beverly Hills, surrounded by elegant and distinguished guests, and announcing to them that one is a socialist.
The reaction is usually priceless”.
Billionaire Tax: How a Union Leader’s ‘Crazy’ Idea Became the Defining Issue of California’s Midterms
Portside
portside.org
2026-11-01 23:15:49
Billionaire Tax: How a Union Leader’s ‘Crazy’ Idea Became the Defining Issue of California’s Midterms
Marti
Sun, 11/01/2026 - 23:15
...
One year ago, the idea of a billionaire tax in
California
was still in its infancy.
It wasn’t yet one of the most contentious ballot measure fights in state history – pitting Democrats against Democrats, drawing national attention from political heavyweights, and netting hundreds of millions of Silicon Valley dollars to fight it. Back then, it was a bold and desperate answer to Donald Trump’s healthcare cuts.
Trump had just signed the One Big Beautiful Bill Act, which dictated a trillion dollars of Medicaid, Affordable Care Act and other healthcare funding
be cut over the next decade
to offset sweeping tax cuts. When the bill goes into effect in January, that could mean
more than 3.4m Californians
will lose their coverage, including millions of children.
“That’s why we were willing to do something that lots of people said was crazy,” said Dave Regan, the divisive architect of the billionaire tax and president of the healthcare union SEIU-United Healthcare Workers West (SEIU-UHW). “To us, what’s crazy is doing nothing and watching millions of people lose coverage.”
Now, with the midterms less than a week away, California voters will decide on whether to pass the ballot measure known as Proposition 40. The proposal would levy a one-time 5% tax on billionaires living in the state as of 1 January 2026, the proceeds of which would fund healthcare programs, along with some education and food assistance.
The initiative has its critics, even among Democrats, who point at its structure as a one-off tax that would only fund certain healthcare programs. But its most vocal critics have been billionaires.
The idea of taxing the Golden State’s ultra-wealthy has snowballed into a messy brawl over the last year. Progressive politicians have become bedfellows with tech billionaires, opposition ballot measure campaigns have been mounted, Signal chats leaked, and boatloads of money have been spent.
Tech billionaires have treated the proposal as an existential threat, dumping upwards of $356m into “independent expenditure-only” political action committees (Super Pacs) to stop it. In the first two weeks of October, according to the most recent campaign filings, the opposition campaign spent more than $42.8m – surpassing $2.5m per day. That money
went to an army of consultants
, who have been tasked with lobbying politicians, blanketing the airwaves with ads and placing op-eds in Latino newspapers across the state.
Attendees cheer Bernie Sanders at a rally in Los Angeles in February in support of the billionaire tax.
Photograph: Patrick T Fallon/AFP/Getty Images
“This looks like it’s going to be the most expensive ballot measure ever,” said David McCuan, a political science professor at Sonoma State University. “Not in California history, but in the history of the world.”
Meanwhile, the far-less-funded pro-Prop 40 camp has relied on viral social media campaigns from influencers and actors, including the
artist Shepard Fairey
and
actor Mark Ruffalo
, and raucous rallies headlined by leftwing politicians. During one packed event in San Francisco earlier this month, Vermont’s Bernie Sanders said: “The whole country, and frankly people from around the world, are paying attention to what is happening here.”
While Prop 40 is just one ballot measure election in one state, it is causing ripple effects nationwide. It comes at a time when affordability is top of mind and the
disparity of wealth
in the US hasn’t been so stark
since the Great Depression
. McCuan said the billionaires’ big spend was partially due to a fear of contagion to other states. Likewise, supporters say they hope the proposed tax is a harbinger for referendums on wealth inequality across the country.
Regan and his union got started on their endeavor last summer. After Trump signed his bill on 4 July, Regan’s team began making calls. They put together a working group of policy experts, economists, lawyers, researchers and political consultants, who came up with the one-time 5% tax to keep hospitals open and union healthcare workers in their jobs.
Regan is known for using the ballot measure process as a political tool to strongarm deals with lawmakers and push causes important to the SEIU-UHW. Since 2012, the union has
sponsored 48 initiatives
. The strategy with the billionaire tax was to sharpen the focus on inequality.
“We worked to come up with something that could draw enough support from voters, and that could avoid a catastrophe,” Regan said. “We just bore down on that.”
When their plan became public after the October 2025 filing deadline, the billionaires hatched a working group of their own.
‘If there’s an impediment, just buy it’
Chris Larsen, a cryptocurrency mogul, hosted a Christmas party in northern California’s swanky Marin county last December, according to
Bloomberg
. It was in a treehouse tucked into a grove of towering redwood trees. The singer Janelle Monáe performed and California governor Gavin Newsom was there, along with Google co-founder Sergey Brin.
Newsom’s relationship with Brin goes back decades to when he was mayor of San Francisco. The governor, now a presumed presidential candidate, had
flown on Google’s jet with Brin
, and the two men attended each other’s weddings. But a lengthy exchange they reportedly had at Larsen’s party was far from collegial.
Brin confronted Newsom about the proposed billionaire tax, according to people who spoke with Newsom, Bloomberg reported. The
Google
co-founder, currently worth about $245bn, told the governor it was forcing him to leave California.
Newsom and Brin did not respond to the Guardian’s detailed list of questions.
By that point, more than a dozen tech executives and prominent Silicon Valley venture capitalists had reportedly already joined an encrypted Signal group chat expressly dedicated to burying the billionaire tax, the
San Francisco Standard
reported. Brin and Larsen were members, along with venture capitalist Marc Andreessen, PayPal co-founder Max Levchin, Stripe chief executive Patrick Collison and investor Ron Conway, according to people who saw the chats.
Larsen and Levchin declined to comment. Brin, Andreessen, Collison and Conway did not return requests for comment.
Google co-founder Sergey Brin at a conference at the renamed Donald J Trump Institute of Peace in Washington earlier this month.
Photograph: Evan Vucci/Reuters
The group brainstormed ideas for fighting the tax. They
reportedly
met every Tuesday for months via video call, debated fleeing the state, and tussled over how much to trust Newsom because of his position as a Democratic governor of a blue state. Newsom has long opposed state wealth taxes and quickly came out against the billionaire tax, repeatedly saying it would “drive a race to the bottom” and stifle innovation as the ultra-wealthy leave.
While a handful of those ultra-wealthy have left, the vast majority of the
state’s 200-plus billionaires
have remained – including Nvidia CEO Jensen Huang, who said
he’s “perfectly fine” with the tax
. Initially, Meta CEO Mark Zuckerberg threatened to leave California, but filings with the Securities and Exchange Commission last week
revealed he struck a deal
with his company’s board to help him foot the bill if Prop 40 passes in exchange for staying in the state.
Among others who indicated they’re departing are Palantir co-founder Peter Thiel, Trump’s former AI and crypto czar David Sacks, and Google co-founders Larry Page and Brin. Last December, Brin
moved to a $42m mega-mansion
on the Nevada side of Lake Tahoe, just steps from California’s border.
Brin is the de facto ringleader of the Prop 40 opposition campaign. Through a Super Pac called Building a Better California, he’s contributed at least $102m to the cause – more than any other donor. Brin worked with political operatives and had his family office map out plans to trounce the tax, according to the
New York Times
. Brin is also a frequent guest at the White House and was
awarded the National Medal of Science
by Trump earlier this month.
Abby Lunardini, a spokesperson for Building a Better California, said: “Californians deserve to know the facts about a measure that would permanently gut the state’s budget, tax base, and economic advantage, all while making the problem it’s supposed to be fixing – healthcare funding – worse.”
Another major early donor to the opposition campaign was Thiel, who gave $3m to the California Business Roundtable Super Pac at the end of December. Larsen has donated more than $23m through various Super Pacs, and former Google CEO Eric Schmidt has given more than $42m, according to campaign finance filings.
At least some of the donations were in cryptocurrency, which had to be converted to dollars under state rules, documents show. Campaign expenditure filings for the tech Super Pacs reveal more than $40,000 has been spent on crypto processing fees alone. Company stock shares given by founders like DoorDash’s Tony Xu reportedly
comprised another significant chunk
.
Peter Thiel leaves the presidential palace in Buenos Aires after a meeting with Argentina’s rightwing president Javier Milei in April.
Photograph: Matias Baglietto/Reuters
The opposition campaign’s initial goal was to ensure Prop 40 didn’t make the ballot. They already had Newsom
publicly vowing
to defeat the proposal. The next step was to go after the signature-gathering process. Anyone can put forward a ballot measure in California if they can gather enough signatures by the filing deadline. Signature-collection firms charge per signature, and the price fluctuates each election based on demand.
Opponents to the billionaire tax came up with a strategy to kneecap SEIU-UHW’s signature-gathering process by introducing several ballot measures of their own to
boost the price of each signature
– the going rate went from $4 per signature to a record-breaking $36.
The venture capitalist Conway spelled it out in a
podcast
in March: “Our job is to get Gavin to negotiate this so that it doesn’t get to the ballot. So maybe they don’t get the signatures.”
When the SEIU-UHW used mostly volunteers to amass their signatures, the opposition campaign reportedly attempted another plan – purchase the firm the union used for signature-collection, so that those signatures couldn’t be used, according to Regan. The same idea was floated in the Signal chat, according to the San Francisco Standard, with one participant reportedly messaging: “That’s what you would do in business: If there’s an impediment, just buy it.”
Fred Kimball is the president of the signature-gathering firm Kimball Petition Management, which worked with the SEIU-UHW and pioneered this type of business. He said that not only did his company endure denial-of-service cyberattacks, which stalled their signature validation process, but he received two calls from unnamed individuals inquiring about a sale.
“It never got any further than that,” Kimball said. “The requirement would be that I’d have to turn over the signatures. Once you do that, it’s all over … My dad started this industry, there’s no way I’d do that.”
By the end of April, well before the signature filing deadline in June, the SEIU-UHW took a victory lap, announcing
they’d gathered 1.55m signatures
– nearly double the necessary amount to qualify for the ballot.
“This is a David versus Goliath battle,” Mayra Castaneda, an executive member of SEIU-UHW, said at a press conference at the time. “David has just won the first round.”
The showdown
After the union’s signature-gathering triumph, the responsibility to kill the billionaire tax rested with Newsom, at least in the eyes of billionaires. Newsom’s objective was to
reach a compromise
with the SEIU-UHW to withdraw its proposal before the June deadline. The governor and Regan had been in talks since December, Regan said, but they remained at odds in finding a solution to the impending healthcare crisis.
Newsom had brought together a coalition to negotiate a deal, enlisting other unions and interest groups to
oppose the tax
. The governor also tried to get Regan to his side, according to Regan. He said Newsom’s chief of staff invited the union boss to a press conference, in which the governor would credit Regan for a
national wealth tax proposal
. Regan said Newsom’s national plan lacked teeth, so he declined.
SEIU-UHW offered a counter proposal: they’d drop the one-time tax from 5% to 2% if Newsom endorsed it. In a
letter to the governor
, the union wrote: “Let’s save patient lives together.”
“But that was a non-starter,” Regan said. So SEIU-UHW announced in late June that it was advancing to the ballot, saying: “David won the second round against Goliath”.
Gavin Newsom at a Democratic rally in San Francisco last week.
Photograph: Heather Diehl/AP
Two billionaire-backed countermeasures also made the ballot, however: Proposition 41 and 42. Prop 41 directly targets the billionaire tax, while Prop 42 is a general proposal on limiting certain new taxes and would ban any taxation on billionaire wealth. If either measure receives more yes votes than Prop 40, it will stop the billionaire tax from taking effect – even if it passes. Regan calls them “poison pills”.
After that, the battle entrenched. Flush with tech money, which kept rolling in, the opposition side grew quickly, landing dozens of endorsements. Planned Parenthood Affiliates of California, the lobbying branch of the healthcare group, joined the cause, as did the influential California Teachers Association, the California Medical Association, and a host of building and trade unions and public safety associations.
The newly formed coalition launched an ad campaign calling Prop 40 a “dangerous wealth tax” and “reckless experiment”. The campaign received endorsements from several prominent Democrats, including Xavier Becerra, the gubernatorial candidate; Katie Porter, a former congresswoman; and Daniel Lurie, the mayor of San Francisco. While Porter contends the tax doesn’t fix the underlying system, Lurie argues the tax would
drive revenue out of California
. Becerra says billionaires should pay their fair share, but Prop 40 isn’t how to do it.
When asked about No on 40’s campaign strategy, a spokesperson for the campaign instead shared a comment from Jack Chou, the president of the California Medical Association, who said the cause has gained endorsements because the measure “permanently damages our economy and budget while failing to fix our healthcare system, improve care, or lower costs.”
The Prop 40 campaign looked doomed as its core allies, Democrats and unions, were jumping ship. But then things took a turn.
In August, the California Democratic party held a board meeting in San Diego, where they voted on endorsing Prop 40. At first, the measure lost. But, according to people familiar with the matter, Congresswoman Maxine Waters stepped in and condemned her colleagues for not supporting the tax. A reconsideration was introduced and the votes swung in favor of Prop 40.
State filings show that the opposition campaign had donated tens of thousands of dollars to the California Democratic party, along with the Democratic parties of San Diego, Orange county and Los Angeles county. Despite the donations, all of these Democratic parties have endorsed the billionaire tax.
Members of the SEIU march during a one-day strike in Sacramento last week.
Photograph: Jose Luis Villegas/AP
Ro Khanna, the Democratic congressman who represents the district that includes Silicon Valley, has been a
staunch supporter of Prop 40
. “It makes no sense to me why any democratically elected official or traditional Democratic allies wouldn’t be supporting this,” he said during a June press conference. “People want folks who are going to stand up to the billionaires and the trillionaires.”
Another turning point was the endorsement of the powerful California Federation of Labor Unions AFL-CIO, which lists 1,300 unions in its roster. During its biennial convention in August, its members voted overwhelmingly to endorse Prop 40.
Lorena Gonzalez, the president of the federation, described the scene on the floor of the convention center that day. One-by-one, union healthcare workers stood up and spoke passionately about the need to fund hospitals, maternity wards and emergency rooms to ensure they don’t close in the wake of Trump’s One Big Beautiful Bill Act, she said.
“Once you heard that, and heard that billionaires got tax breaks from the big BS bill and that Medi-Cal got cut, it just became a no-brainer,” Gonzalez said. “I’m not going to be a labor movement that bows down to the establishment and is scared of taking on billionaires.”
Mudslinging abounds and record-breaking spending
As the election nears, California residents have witnessed the two campaigns vie for attention and relentlessly demonize each other.
Last week, after Newsom
endorsed the billionaire’s Prop 42 countermeasure
, which would stop the state from ever passing a wealth tax, the Prop 40 campaign released an attack ad on the governor. Likewise, the opposition campaign sent voters across the state a glossy four-page mailer that read: “You’re Next.”
Since July, the tech Super Pacs have upped their donations to the committees opposing Prop 40 and supporting Props 41 and 42. The three campaigns have received a total of more than $379.5m – making it the most expensive opposition campaign in state history. “We’ve said for months we’re going to get carpet-bombed by these guys,” Regan said. “Their so-called strategy, there’s only one prong to it: we can buy anything we want.”
Along with the flood of spending and negative ads, accusations and personal attacks have been flying in all directions. Much of this has been focused on Regan, who’s been
criticized in Democratic and union circles
for being an aggressive bulldog and spending his union’s resources to push ballot measures.
In one specific instance, Courtni Pugh, a former executive director at the SEIU, accused Regan of physically assaulting her in 2009,
according to the Los Angeles Times
. She said he kicked open her office door, pinned her against a wall and screamed that she was a “dumb ass”. The SEIU investigated the allegations this year and wrote in a report that Pugh’s claim was substantiated. Regan denied the alleged abuse, claiming that the allegations against him are retaliatory for his support of Prop 40. (The union investigation did not substantiate Regan’s retaliation claim.) Law enforcement did not investigate the abuse allegations and no charges were brought.
Pugh is now serving as a
senior advisor to Becerra
and working as a consultant for the No on 40 campaign. Regan told the Guardian: “Courtni Pugh, at Hilltop Solutions, wants to invent accusations about me from 17 years ago on the payroll of the billionaires.”
Pugh said Regan’s response “is offensive to the people – especially the women and women of color – who have had to endure his threatening behavior”. A spokesperson for Becerra said Pugh has not advised him on his position on the wealth tax.
Dave Regan, the SEIU-UHW president, during a ‘Yes on 40 for Ballots Over Billionaires’ rally in San Francisco.
Photograph: Jason Henry/Bloomberg via Getty Images
The newest Democrat to vocally oppose Prop 40 is former congresswoman Porter, who’s known for taking corporate executives to task. She has
written op-eds
in local papers, posted
videos
on social media, and is quoted in ads and mailers saying: “Prop 40 does nothing to reform our healthcare system.” After losing a gubernatorial bid to Becerra earlier this year, she said she was retiring from politics. But she’s waded back into the fray with Prop 40. Late last month, Newsom also
appointed her
to a paid position on the board of governors for the state’s community colleges.
Despite the controversy and criticism, the SEIU-UHW has brought notable figures to its side. Last month, six Nobel prize-winning economists wrote an
open letter
endorsing Prop 40. Doing the math, they wrote that the combined wealth of California’s billionaires is $2.3tn, equal to the entire annual income of all 20m taxpayers in the state. They pointed out that billionaires’ net worth has grown nearly 150% over the past couple of years, fueled by the AI boom. “This tax only partially plays catch-up, but it is a move in the right direction,” the economists wrote.
Now it’s up to the voters, who appear to be unswayed by the onslaught of messaging. A
Politico poll
last week showed that 84% of registered voters said they’d seen campaign ads involving Prop 40 and the majority of them weren’t persuaded into changing their positions. Despite that, the measure has fallen in the polls. While it enjoyed a seven-point lead over the summer, it’s since fallen to a one-point lead.
For Regan, win or lose, he said the idea of taxing billionaires isn’t going away.
“I think this campaign has already put this issue on the national agenda in a very significant way,” he said. “This issue is going to be at the center of the nominating process for a Democratic candidate in 2028.”
We are pleased to announce the winning sequences in the OEIS 400k celebration. Well over 100 nominations were received, covering a great many topics collectively showing that the spirit of the OEIS is alive and well.
The selection committee consisted of
Sean A. Irvine
,
Jonas Karlsson
,
Hugo Pfoertner
, and
Neil J. A. Sloane
. Committee members abstained from providing evaluations on sequences they authored or were closely involved with. The selection committee would like to thank all the people who took time to nominate a sequence.
Winner
A400000
(previously
A399633
)
Eric Fox
The city-building sequence: a(n) = the height h of the shortest building which can see exactly h earlier buildings.
The height of a building in Eric Fox's City-Building sequence is determined by how many existing buildings it can see. This impressed all the judges as by far the most brilliant invention of 2026. There are already 25 other sequences that reference it.
Runners-up (new A-numbers were assigned by lottery)
A399995
(previously
A394611
)
Joshua B. Weinstein
a(1) = 0. For n > 1, a(n) is the number of ways to write a(n-1) as the sum of two earlier terms with distinct positions in the sequence.
A simple definition leading to a frustratingly difficult unsolved problem.
A399996
(previously
A400866
)
Dmitry Kamenetsky
Number of free polyominoes with 2n cells that can be dissected into two congruent polyominoes.
The best polyomino sequence of the year.
A399997
(previously
A397960
)
Wujie Gu
a(n) is the maximum number of distinct positive differences of an n-element subset of {0, 1, ..., n*(n-1)/2}.
An instant classic extremal set-theory problem.
A399998
(previously
A400887
)
Rok Cestnik
a(n+1) = a(n-a(a(n))^2) + 1, a(0) = 0.
A new twist on the classic Hofstadter recurrences.
A399999
(previously
A400462
)
Ed Pegg Jr
Number of ways to cut a 30-60-90 triangle into n 30-60-90 triangles.
One of several triangle dissection problems, lovely pictures.
A400001
(previously
A400663
)
Felix Huber
a(n) is the number of congruence classes of nondegenerate triangles whose vertices are points of an n X n X n grid.
One of several excellent Huber sequences relating to triangles.
A400002
(previously
A400976
)
Aidan Markey
Start with a set of two adjacent points in a hexagonal lattice. At each iteration, add to the set the lattice points which are adjacent to exactly two points already in the set, but only if those two points are adjacent to each other. The sequence gives the number of points in the set after n iterations.
An unusual sequence of clusters.
A400003
(previously
A399179
)
Michael S. Branicky
and
N. J. A. Sloane
The sequence of binary decisions that defines the unique infinite path in the base-10 comma child graph.
Previously MSB/NJAS had to use Koenig's Lemma and the Axiom of Choice to show that it was possible for the kangaroo to escape death. Chris Randall and Paul Roberts showed without using AI that there is a unique path with a simple recurrence.
A400004
(previously
A401021
)
Scott B. Farrar
Half the minimum surface area of a rectangular box with positive integer side lengths and volume at least n.
A new problem with a classic feel.
A400005
(previously
A395645
)
Rémy Sigrist
Consider the square spiral with its cells numbered starting at 0. Three players, Red, Green and Blue, take turns and place knights at the smallest unoccupied cell not attacked by an existing knight of a different color. Sequence lists squares occupied by a Red knight.
A representative of many astonishing sequences of knights on an infinite chessboard, a popular topic after a Numberphile video appeared.
Honorable mentions (sorted by existing A-number)
These are a further ten nominated sequences which the committee found particularly attractive or interesting. These sequences have all received the keyword “nice” but retain their original A-numbers.
A394184
(
Charles L. Hohn
) Number of primitive tessellations of paths of length n in the square lattice, reduced for symmetry (with rotations, reflections, and translations not counted as distinct).
A396596
(
Hal M. Switkay
) Natural numbers k such that the sum of the x-th powers of the divisors of k achieves a record, for some x <= 0.
A397182
(
N. J. A. Sloane
) a(n) = maximum number of regions the plane can be divided into by drawing n constrained long-legged A's
A398965
(
Jens Ahlström
) Squares visited by a "Carpenter" (Dabbaba + Knight) moving on a spirally numbered board always to the lowest available unvisited square.
A399136
(
Stephen J. Pursey
) a(n) is the smallest number of cells in a polycube that has a cavity consisting of exactly n cells.
A399946
(
Eliott Grillet
) Maximum terminal value in the Gilbreath reduction of graceful paths of order n.
A400615
(
Jakob Süskind
) Maximum number of edges in the Hasse diagram of a lattice with n elements.
A400939
(
Zlatko Damijanic
) The halving Recaman walk: a(1) = 1; for n > 1, a(n) = a(n-1)/2 if a(n-1) is even and a(n-1)/2 has not already appeared, otherwise a(n) = a(n-1) + n.
In the
first part
of this article, we have taken a look at two attempts to decide whether a given string is a palindrome. Both implementations were written in “Classic C++” and didn’t feel great: the code was difficult to read, hard to understand and consequently the defects were difficult to detect. Now it’s time to continue the story and to take a look at a better solution, the “Modern C++” solution.
Yes, it’s that simple: a single function call to the
std::equal()
algorithm. We compare the first half of the range represented by forward iterators with the second half of the range represented by reverse iterators.
“But Klaus, wait a second: this
std::equal()
algorithm is ages old! This has been part of C++98! This isn’t modern, is it?”
If this is what you are thinking, then I have some important information for you: modern C++ is not about new features or new C++ standards. It is about writing correct, fast and safe code by default. It is about a certain mindset. So to prove my point, ask yourself the question from before: are you certain that there are no safety or security related issues?
…
There are none, right? And it is so easy to verify: no
if
statements — in fact, no conditionals at all, no pointers or indices, no explicit loop. It’s just a single, simple function call. So this code is correct by inspection. It is correct by default. And it is simple. And fast.
There is even a newer, even more elegant solution:
This is the C++20 ranges-based solution. We are using the
std::ranges::equal()
function on the string and its reverse representation directly. We don’t even need iterators anymore. Admittedly, we are now comparing the entire range and not just the first and second half. However, performance aside, the simplicity, the elegance, and the correctness by default are amazing! This is what we refer to as “Modern C++”.
The Real Problem
“OK, Klaus, I agree this is simpler and correct by default. But does this really mean that the loop-based solution is bad? After all, it works, doesn’t it?”
You are correct, it “works”, ignoring the UB for single-character strings. But it is more complicated, requires more effort to verify, takes more time to write, is more difficult to read and understand, requires considerably more testing and just isn’t correct by default. All of which means that, in the end, it is considerably more expensive, too. There simply isn’t a good enough reason to write it in this classic style anymore. This is exactly what I perceive as the real problem in our community.
Too many C++ developers have not embraced modern C++ yet and still use the dangerous and more error-prone constructs by default. Too many C++ developers don’t know about the modern ways of doing things or, for whatever reason, still prefer to write the more explicit code. Because of this, they work by default in the “unsafe mode” of C++.
To me this feels like an educational problem, a teaching issue. For too many decades people have written this classic C++ and “it was OK”. By now people feel too comfortable, feel too familiar with this style. This is the real problem.
Unfortunately, this problem is currently getting scaled up significantly: With the use of AI tools, people generate this kind of classic C++ automatically. And yes, I have got proof: I asked ChatGPT to generate an
is_palindrome()
function for me (see Figure 1). Guess what: it gave me almost exactly the function from the original LinkedIn post. And no: it didn’t even consider suggesting the use of an algorithm.
Figure 1: ChatGPT’s solution for an
is_palindrome()
function
So why don’t these modern tools use the modern way? Well, despite the name, AI has nothing to do with intelligence. It is a statistics-based tool. If more people use the classic way, then statistically this is the “correct” solution. So it is not a big surprise that AI tools will rather write a
for
loop for you instead of using an algorithm.
I should be honest and state that this is not true for every AI tool on the market. Some are better than others, some actually suggest using an algorithm. However, this unfortunately doesn’t resolve the problem: AI tools may suggest outdated solutions, which are not best practice anymore.
Luckily many people in the community know better than the AI tool and know about the modern solutions. Let me use this opportunity for a shout-out to some people that quickly posted a good and modern solution to the original LinkedIn post. Many thanks to Alex Dathskovsky and Sergii Rybin for posting an algorithm-based solution. Honestly, we need more people like you that help to spread the information on how things should be done today. Thank you very much!
What’s next?
So what else can we do? I for one can do my part as a C++ trainer: I can tell people. I can teach people. I can show people. And this is what I want to do here in Rainer Grimm’s blog: in the coming months I want to talk about the modern way of doing things in C++. I want to demonstrate how we work today, in the hope that this makes a difference, in the hope of establishing the modern C++ mindset and of maybe (hopefully!) contributing to the general realization, even beyond the C++ community, that C++ can be incredibly safe, if we just use it as it is supposed to be used.
In the following months, I’ll write about bounds safety, type safety, lifetime safety, initialization safety, and — as a special add-on — about undefined behavior. And I will demonstrate that with very little effort, you can avoid all of these safety problems easily and by default.
Thanks a lot to my
Patreon Supporters
: Matt Braun, Roman Postanciuc, Tobias Zindl, G Prvulovic, Reinhold Dröge, Abernitzke, Frank Grimm, Sakib, Broeserl, António Pina, Sergey Agafyin, Андрей Бурмистров, Jake, GS, Lawton Shoemake, Jozo Leko, John Breland, Venkat Nandam, Jose Francisco, Douglas Tinkham, Kuchlong Kuchlong, Robert Blanch, Truels Wissneth, Mario Luoni, Friedrich Huber, lennonli, Pramod Tikare Muralidhara, Peter Ware, Daniel Hufschläger, Alessandro Pezzato, Bob Perry, Satish Vangipuram, Andi Ireland, Richard Ohnemus, Michael Dunsky, Leo Goodstadt, John Wiederhirn, Yacob Cohen-Arazi, Florian Tischler, Robin Furness, Michael Young, Holger Detering, Bernd Mühlhaus, Stephen Kelley, Kyle Dean, Tusar Palauri, Juan Dent, George Liao, Daniel Ceperley, Jon T Hess, Stephen Totten, Wolfgang Fütterer, Matthias Grün, Ben Atakora, Ann Shatoff, Rob North, Bhavith C Achar, Marco Parri Empoli, Philipp Lenk, Charles-Jianye Chen, Keith Jeffery, Matt Godbolt, Honey Sukesan, bruce_lee_wayne, Silviu Ardelean, schnapper79, Seeker, and Sundareswaran Senthilvel.
Thanks, in particular, to Jon Hess, Lakshman, Christian Wittenhorst, Sherhy Pyton, Dendi Suhubdy, Sudhakar Belagurusamy, Richard Sargeant, Rusty Fleming, John Nebel, Mipko, Alicja Kaminska, Slavko Radman, and David Poole.
When Ken Thompson returned to Bell Labs in 1976, Bill Joy and Chuck Haley began exploring the Unix kernel themselves.
Hmm... we could speed things up here in the kernel.
Other universities want copies of our Pascal compiler.
Let's bundle it with ex and send them a tape.
In early 1977, Bill Joy put together the Berkeley Software Distribution, or BSD, bundling the Pascal system with ex, a line editor developed from em. Early BSD was not a standalone operating system: it was a collection of software to install on an existing Unix system. About thirty copies were sent out over the following year.
[1]
At the time, an AT&T Unix source license was needed to get BSD. Berkeley charged only for the tape and distribution.
[2]
If you already have a Unix source license, it's just the tape and postage.
Berkeley already had a tradition of sharing research software. BSD carried it on. Berkeley's Free Speech Movement, hippie culture, and antiwar activism are also said to have influenced Joy's decision to charge so little.
[2]
Universities and research institutions could obtain Unix, including its source code, for a modest licensing fee. This also helped the system spread.
[2]
[6]
We have Unix, but the antitrust agreement bars us from the computer business... We can't sell it as a product, either.
The second distribution: 2BSD
After the first BSD distribution, Joy improved the Pascal system based on user feedback. He bundled it with vi (ex's visual mode), the C shell, and other tools to form the Second Berkeley Software Distribution, or 2BSD. Nearly seventy-five tapes were shipped over the following year.
[1]
[5]
Joy did more than fix bugs. He handled every distribution request, copied BSD onto tapes, and mailed them out himself.
Oh, you need a copy of BSD?
Another university wants a copy.
Tune the kernel. Fix the bugs.
A VAX arrives at Berkeley, 1978
Professor Fateman's group is getting a VAX. It's 32-bit, with a whole megabyte of memory.
Really? We should put Unix on it.
They've already got VAX Unix from Bell Labs. It's called 32/V.
My adviser asked me to add virtual memory to VAX Unix. Could you look over the code?
Babaoglu was a Berkeley graduate student working with Professor Domenico Ferrari. Joy helped him integrate the virtual-memory code into 32/V and debug it.
[1]
[4]
At the time, 32/V did not use the VAX hardware's virtual-memory capabilities. A program's address space was limited by physical memory, but Professor Richard Fateman's work on the computer algebra system
Macsyma
needed more. Virtual memory uses disk storage to give programs an address space larger than physical memory.
[1]
Macsyma, developed at MIT beginning in the late 1960s, worked with symbolic expressions. It could factor expressions, compute derivatives, and perform integration.
Debugging continued through the holidays at the end of 1978.
Not going away for the holidays?
Might as well debug this while fewer people are using the VAX.
By January 1979, most of the bugs had been fixed.
[1]
Let's try Unix with virtual memory.
Finally. It runs without crashing.
Joy ported ex, vi, the C shell, and other 2BSD tools to the VAX, while Peter Kessler and Marshall Kirk McKusick ported the Pascal system. These programs were bundled with the standard 32/V utilities and the virtual-memory kernel that Babaoglu and Joy had integrated and debugged. The result was 3BSD, which began shipping in December 1979.
[1]
At last, we can send these out.
ARPANET had been connecting research institutions funded by the Defense Advanced Research Projects Agency, or DARPA, since 1969.
[3]
But the sites used different computers and operating systems, making software expensive to port or rewrite. DARPA became interested in Unix's portability, and took notice of Berkeley's successful distribution of 3BSD to research institutions.
[1]
"These tools have made it very easy for bad actors to crowd out good information..."
Image credit:
Eurogamer / GameDB
Recently, I was writing a piece about Nighthawks and wanted to remind myself of some basic information about the companion characters. As I have a thousand times before, I put the question into Google and clicked the first result: the Nighthawks Wiki.
Of course, wikis aren't necessarily reliable, especially fan-run projects that might not have enough engaged individuals to immediately spot and fix errors. But the "Nighthawks Wiki" that I landed on immediately and obviously had deeper issues. Firstly, it wasn't a wiki at all. Rather than being editable by anybody, this website only invited "corrections" by email. Secondly, it was full of false information that didn't seem likely to have been added by fans getting their facts slightly wrong, like listing companions and skills that don't exist. It also regularly included unrelated information about the 1981 movie and the Edward Hopper painting, both of the same name, and listed them in its "sources" even when they had no relevance to the page in question.
I don't like to accuse anybody of using generative AI, because its hallmarks are difficult to pin down and are often misattributed to the LLM rather than the human writers it rips off – ask any of us how much we enjoy the em-dash, for example. But, interest piqued by the rate and intensity of apparently made up facts, I managed to trace the domain name to a website building service called Spaceship, which also offers an "AI Website builder." (I emailed the address on the "wiki" for comment, but the message bounced.)
The unofficial Nighthawks wiki in question. |
Image credit:
Eurogamer / Nighthawks.wiki
A quick search turned up many more similar sites, some of which were actually editable as wikis and others which only called themselves by that name. They typically targeted indie games without gigantic playerbases – the sort of game that people might seek out help with and not find a thriving community ready to provide it.
It's a problem that Tim Bender, CEO for publishing company Hooded Horse, knows well. Unlike most publishers, Hooded Horse creates wikis for their own games, working with fans to make sure they're well-populated with useful information. But lately the company has had to "play whack-a-mole" with AI-generated sites competing for the search results and providing inaccurate information. One of these, for example, appeared for their game Corsair Cove, but was actually part of a
more than 20 game scattershot collection
operated by a site called GameDB, which the creator
admitted to using generative AI
to make. "[Generative AI] enables a run of basically commercial enterprises to mass produce and try to flood the search engine results of games to soak up their traffic," says Bender. Attempts to reach GameDB for comment via their public contact email resulted in an error, stating "the remote server is misconfigured".
And the combination of elbowing out legitimate wikis and being full of incorrect information ends up with a lot of frustrated players. "The whole purpose of putting out a wiki is so when [players] search for help, they end up at a site that helps them. And instead we continually hear from players who stumbled up on these things and got confused and are wondering what's going on," says Bender.
That collection of genAI wikis includes one for the 100-player FPS Wardogs; one of at least five that FlamingMojo, a member of the Independent Game Wiki Network, saw appear in the days after the game's release in September. And he says that the frustration the genAI-created sites causes players also impacts those who are trying to do the legitimate work of creating and maintaining these volunteer-led resources: "It's hurt our reputation, as players have been misinformed but we haven't created or updated the relevant page yet so they bounce off [wikis] entirely."
The wiki is full of irrelevant information, such as casting info for the 1981 film of the same name. |
Image credit:
Eurogamer / Nighthawks.wiki
Fanwikis are already a contested part of games culture. Hosting sites like Fandom dominate the space, and have themselves
introduced AI features
that had to be rolled back due to their inaccuracies and unpopularity. Many groups have decided to leave these hosts, but those who go independent often
struggle to show up
in Google's search rankings underneath the competition from the umbrella sites.
Whether on a hosting site or independent, fans rarely make any money from the effort they put into their wikis. Either they sacrifice any ad revenue to the bigger platforms, or they pay for their own hosting, which is rarely covered by ads or donations. FlamingMojo, for example, pays for the
Dune: Awakening
Community Wiki out of his own pocket and doesn't run any ads on it.
But counterintuitively, the low profitability of these kinds of sites only leads to worse outcomes when generative-AI versions are concerned. "There's not that much ad revenue, but there's going to be a little bit and that's all it takes," says Bender. "Because the barrier is so low, why not just spew out a mountain of it and flood the search results?"
It's also possible the generative-AI sites are sometimes aiming for outcomes beyond a little ad revenue. "Some of them, you don't know what data they're collecting," says Bender. In at least one case, a generative-AI site was pointing users to download additional software that offers a paid subscription. The software had been reviewed poorly online and opening the website caused my malware software to kick in and warn me it had been flagged as "potentially unsafe." (None of the sites I attempted to contact for comment responded to my emails.)
Editors Note: when a Eurogamer editor attempted to visit the Nighthawks wiki in question, it repeatedly redirected to various spam URLs, though this author was not able to replicate the issue. |
Image credit:
Eurogamer / Nighthawks.wiki
Independent wikis also now shoulder a double burden, as generative AI scrapers send huge amounts of traffic their way, potentially affecting site stability.
Publicly available
internal data from the Dune: Awakening Community Wiki shows 7m requests in a 30 day period from AI crawlers. (5m of these come from Meta alone. While FlamingMojo is able to tame other scrapers using files that direct them to real pages rather than brute-force trying every link, he's found that Meta simply "hammers every page multiple times per second" with no care for "any sensible ratelimits.") These scraping waves hit smaller wikis particularly hard, as their maintainers aren't necessarily expecting huge amounts of traffic and don't have the compute capacity to handle huge spikes in visitors.
"Independent gaming wikis are in a difficult place," says FlamingMojo. As well as competition for SEO, both from corporate sites and now genAI, he says server costs have increased between three- and ten-fold, again due to the impact of genAI. Building a group of people who want to volunteer their time and energy to keep a resource updated isn't easy, although the Dune: Awakening Community Wiki has found success through
gamifying contributions
, in affiliation with the developer giving out in-game rewards.
And, Bender points out, all of this is happening within a much wider context of genAI's impact on the availability of information. "In a more fundamental way … these tools have made it very easy for bad actors to flood things and crowd out good information which people put dedicated time into. This is just one example of a much broader struggle that, honestly, whatever frustration I feel for this on my end, I'm sure you [journalists] feel at a thousand times that."
But it's possible that, by being one part of the wider issue, fanwiki culture could also hold a sliver of the bigger solution. "The way we survive the AI pollution is with community," says FlamingMojo. "Many people chipping in a small bit, and feeling a sense of shared ownership."
AI Can Write More of the Code. We Still Have to Engineer the System
When people say AI has transformed software development, I want to know which part they mean. Producing an implementation? Exploring a design? Deciding whether a system solves the right problem?
Those activities are connected, but automating one does not settle the others. My argument is that agents can take on more implementation work, and assist with analysis, design, and testing, while engineers remain responsible for deciding what to build and what evidence is sufficient to accept it.
That changes the work. It does not make the responsibilities disappear.
I am wary of describing this as replacing the person who churns out code. It reduces developers to typing, understates what agents can assist with, and turns task automation into a claim about jobs. Claims about workforce replacement require evidence beyond an impressive demonstration.
The practical question interests me more: how do we use these tools while keeping our decisions visible and our feedback useful?
Give decisions somewhere to live
A conversation is convenient for exploring options. It is a fragile place to keep the only record of an architectural decision.
Model context is bounded. A long session may be summarized, compacted, or resumed with selected information. The summary can preserve the conclusion and lose the condition that made it reasonable. “Use a local cache” is different from “use a local cache because this screen must remain usable without connectivity.”
Anthropic's
account of long-running agents
describes continuity problems across context windows and an approach using incremental work and persistent progress artifacts. It also reports that compaction can leave unclear instructions. This is a vendor engineering report about a particular setup, rather than a universal comparison of development workflows.
My preference is to give durable decisions a small, reviewable home. A
PROJECT_GUIDE.md
might contain:
The product's purpose and current scope, including explicit exclusions.
The important system boundaries and the reasons behind architectural choices.
Quality expectations and the commands used to check them.
Links to deeper decisions when those decisions need more space.
The filename has no special power. An agent will not necessarily load it automatically. I would connect it through the tool's supported project instructions or explicitly include it in the task, then check the agent's interpretation.
For each increment, I would supply the relevant files and ask for a brief account of the intended change, constraints, and acceptance cases. A paraphrase is an early opportunity to catch misunderstanding; the implementation still has to demonstrate compliance.
Simon Willison's
June 2025 discussion of context engineering
helps frame the problem of choosing information for the next model step. My narrower recommendation is to make that selection deliberate and inspect its consequences.
Markdown alone guarantees neither retrieval nor obedience. Sending every document on every task can increase context and cost. Selective retrieval may reduce input, but maintaining documents, retrieving them, and correcting mistakes also consume resources. Any claim of savings needs a measurement of the complete task.
Specify the next behavior
Consider a hypothetical mobile search screen. This is an illustration, not a report from a completed project.
The story could be: “As a user, I want to search saved items so that I can find an item without scrolling through the entire collection.” For the first increment, the scope is local search over existing items. Remote search and changes to persistence are excluded.
Before implementation, I would make these acceptance cases explicit:
A matching query displays the expected items.
A query with no matches displays an empty result state.
Clearing the query restores the complete collection.
If filtering is asynchronous, an older result cannot overwrite a newer query's results.
These cases expose a technical question: does filtering need asynchronous work at all? The agent can help assess that decision. We should resolve it against the expected data size and responsiveness requirements, rather than introducing concurrency because a template expects it.
For this example, I might choose unidirectional data flow, explicit UI states, and a separation between filtering logic and rendering. That makes state transitions easier to inspect. Another system may justify a different structure; this is an engineering choice with a purpose.
The project guide should preserve that purpose. Feature-level notes should explain the behavior and any new decision. They do not need to reproduce the entire architecture.
After the increment, I would inspect the diff, run the relevant checks, and exercise the screen. If the agent rewrote persistence despite the scope restriction, passing tests would not make that expansion acceptable. If the empty state looks correct but the search matches the wrong field, a screenshot would not establish correctness either.
Make generation answer to verification
Agent output is probabilistic. Acceptance should depend on evidence that we can inspect and, where practical, reproduce.
A local verification command can run formatting checks, static analysis, compilation, and focused tests. These checks answer different questions. A formatter checks conventions; a compiler checks language and type rules; tests evaluate selected behavior under selected conditions. None establishes that we chose the right product behavior.
Controlled verification also needs controlled inputs. Tool versions, fixtures, and execution conditions matter. Tests involving timing, devices, networks, or external services may be flaky. Calling a test suite deterministic does not make it so.
Even a perfectly repeatable check can verify the wrong thing. A search test might assert that some result appears without checking which result it is. A generated test can faithfully encode the same misunderstanding as the generated implementation. I would review the assertion against the acceptance case and use deliberately chosen examples, including a failure case.
Running the app adds another kind of evidence: whether the behavior makes sense through the actual interface. When performance matters, define the measurement before declaring success—device, dataset, scenario, and threshold. A quick impression of responsiveness is useful feedback, but it is not a latency result.
Hooks can make local checks convenient and catch mistakes earlier. Their coverage depends on what triggers them and whether they can be bypassed. Where a team requires enforcement, server-side checks and repository protections need their own configuration. A local hook is not equivalent to an enforced remote policy.
The engineer still decides whether the evidence supports acceptance, revision, or rejection. An agent can help gather that evidence; it cannot make an incomplete test suite complete by announcing success.
Documentation has to earn its maintenance
I become skeptical when a small change requires a long chain of generated documents before anyone can see working behavior. Each document becomes something to review, reconcile, and keep current. Producing text cheaply does not make understanding it cheap.
That is a criticism of proportionality and feedback cadence. Spec-driven development has multiple meanings. Birgitta Böckeler's
2025 examination of Kiro, spec-kit, and Tessl
distinguishes approaches that write specifications first, retain them during maintenance, or treat them as the primary source. Those differences matter when discussing costs and control.
François Zaninotto's
critique of document-heavy workflows
raises concerns about review overhead and instruction following through reported examples. Those cases are useful prompts for investigation. They do not establish that specifications generally cause worse outcomes.
An incremental specification can support rapid feedback. Agile does not mean avoiding documentation: its
principles
emphasize frequent working software, technical excellence, and adaptation. A short acceptance case can serve those aims; so can a detailed design when the risk warrants it.
Stronger specifications can be valuable for safety constraints, regulated work, public interfaces, migration compatibility, and coordination across teams. In those settings, a concise guide is an entry point to the necessary detail. It cannot replace that detail.
My test for a document is practical: what decision, implementation, or verification does it support, and who will maintain it? I would keep the explanation of a compatibility constraint even if it takes several pages. I would remove repeated prose that neither changes a decision nor helps someone evaluate the work.
Matt Pocock's
collection of engineering skills
offers another example of packaging guidance into adaptable, composable pieces. I find that direction useful. Its stated design goals are not evidence that a collection of skills outperforms another workflow.
Keep the acceptance decision explicit
A small increment can still contain a bad assumption. Its advantage is that it gives us an earlier, more bounded opportunity to discover one.
The loop I would advocate is straightforward: define the next behavior, supply the context it needs, check the interpretation, inspect the change, collect evidence, and decide whether to accept it. Update durable guidance when a decision changes. If review becomes unmanageable, reduce the increment or improve the checks before asking the agent to produce more.
This approach has limits. It depends on the engineer's understanding, the quality of the evidence, and the cost of keeping guidance current. This article presents a working position, not a controlled study or a demonstrated productivity gain. Establishing such a gain would require comparable tasks and accounting for review, rework, defects, and maintenance—not just generation time.
I am comfortable delegating more of the work when I can explain what was requested, what changed, and why the evidence supports accepting it. That is the engineering standard I want to carry into AI-assisted development.
Which decisions in your workflow remain visible after the conversation ends—and what evidence tells you that the agent respected them?
Authorship note: The initial draft was generated with AI assistance from Armando Picón's arguments and a source review. Armando reviewed the article and approved publication. The Spanish translation was produced with AI assistance. The mobile example is hypothetical.
Bones may release proteins that slow aging and extend life span
OccupySaaS is a protest movement written in code. Laid-off engineers, armed with AI and public documentation,
are building
clean-room, open-source, drop-in replacements
for the platforms that
rent our own industry back to us — one seat at a time.
We are the
1,000,000
The grievances
The software built
by the 99%
belongs
to the 99%.
In 2011, Occupy Wall Street asked who the economy was really working for. In 2026 we're asking the same
question about the cloud. The tools every company runs on are now toll booths — and the
people who built them were shown the door.
Per-seat pricing is a
tax
on growth
Every hire, every intern, every contractor — another line item. Infrastructure shouldn't charge rent by the head.
Lock-in is not a
feature
Proprietary APIs, export walls and surprise price hikes keep customers captive. Compatible open alternatives set them free.
They cut us. We
commit
.
A million engineers were let go while the platforms kept growing. We still know how to ship. Now we ship it in the open.
The method
Clean room.
AI rebuild.
Drop-in.
We don't leak, decompile or copy anyone's code. Every project is rebuilt from public documentation and
open standards, with AI doing the heavy lifting on the build side of a strict firewall — then
proven compatible against the vendors' own public SDKs.
Read the public record
Public API references, open standards (OAuth 2.0, OIDC, REST) and published SDK behaviour become a written specification. Nothing confidential, nothing leaked, nothing from former employers.
Raise the firewall
Spec writers and builders stay separated. Each repo carries a
CLEAN_ROOM.md
policy, and provenance is recorded for every fixture and reference.
Rebuild with AI
AI agents implement from the spec alone — contracts first (
openapi.yaml
), then code, then tests — under human review at every merge.
Prove compatibility
A compatibility laboratory runs the vendors' own public client SDKs against our server. If your existing code doesn't work unchanged, it isn't done.
Release it to everyone
Apache-2.0. Self-hostable. Pluggable storage. No telemetry you didn't ask for, and no seat counter.
Allowed in →
Public API docs
Open standards & RFCs
Public SDK behaviour
Black-box test results
Clean room
× Never allowed
Leaked or proprietary source
Decompiled binaries
Former-employer confidential info
Copied UI assets or trademarks
First releases
Two towers
occupied.
Identity and CRM: the two subscriptions almost every company pays for and almost nobody loves.
Both are early, both are public, and both welcome contributors today.
Early · in the open
Go
Zero
Authy
A self-hostable identity server that speaks the Auth0 Authentication API. Point an existing Auth0 SDK at
a ZeroAuthy tenant domain and it works with no code changes.
A free, open-source Salesforce replacement with a tenant-isolated Go/PostgreSQL core, a Vue record
editor, and a compatibility laboratory that checks behaviour against Salesforce's public API contract.
Persistent Account & Contact CRUD
Object
describe
and limited queries
External-ID upsert & bearer authentication
OpenAPI 3.2 REST contract & offline test lab
# run the compatibility laboratory$ npm ci --prefix compat/clients/node --ignore-scripts
$ npm --prefix compat/clients/node run check
PostgreSQL
Multi-tenant
REST
i18n
Apache-2.0
The occupation map
Who's
next?
Every category on this board is a monthly invoice somewhere. The movement picks the next target in public —
bring a spec, a use case, or a grudge.
● Occupied
Identity & Auth
ZeroAuthy — Auth0-compatible login, tokens and tenants.
● Occupied
CRM
SalesCompel — Salesforce-compatible records and REST API.
○ Proposed
Docs & Wikis
Team knowledge bases that don't lock your writing behind a seat licence.
○ Proposed
Issue Tracking
Tickets, sprints and boards with a compatible API for existing integrations.
○ Proposed
Feature Flags
SDK-compatible flag evaluation you can run next to your own services.
○ Proposed
Observability
Logs, metrics and traces without per-gigabyte ransom notes.
○ Proposed
Customer Support
Help desk, inbox and knowledge base, priced at zero agents.
Proposed categories are open for discussion, not commitments. Names of existing products are used only to describe compatibility.
Lineage
From Zuccotti Park
to
git push.
Occupy Wall Street
began on
17 September 2011 in Zuccotti Park, New York City, with a simple frame: the 1% and the 99%.
It had no leaders, made decisions in open assemblies, and spread to hundreds of cities.
OccupySaaS borrows that spirit and swaps tents for terminals. No marches, no property damage —
just working software, published under an open licence, that anyone can run.
“We are the 99%.”
Occupy Wall Street begins
Protesters set up camp in Zuccotti Park in Manhattan's Financial District.
The movement spreads
Occupy camps and general assemblies appear in cities around the world.
Zuccotti Park is cleared
The camp ends; the language of the 1% and the 99% doesn't.
The great tech layoffs
Wave after wave of cuts across the industry while subscription prices keep climbing.
ZeroAuthy goes public
The first clean-room rebuild opens its repo under Apache-2.0.
SalesCompel joins the occupation
The second tower. More to come — decided in the open.
Join
Occupy
everything.
You don't need permission, a badge or a manager. You need a laptop and an afternoon.
Write code
Pick an open issue, follow the clean-room policy, open a pull request.
Self-host it
Run a release against your real SDKs and tell us exactly where it breaks.
Write specs
Turn public docs into precise, testable specifications for the builders.
Make noise
Star the repos, share the site, and tell your CFO what you're paying per seat.
Is this legal?
Clean-room reimplementation is a long-established way to build compatible software: you work only from
public information and independently write your own code. Interoperating with an API is not the same as
copying a product. Each project publishes its clean-room policy and source provenance so anyone can check
the work. This page isn't legal advice; if you're adopting a project for a business, talk to your own counsel.
Are you affiliated with Auth0, Okta or Salesforce?
No. OccupySaaS and its projects are independent and not affiliated with, endorsed by, or sponsored by
any vendor named on this site. Product names are used only to describe compatibility.
Can former employees of those companies contribute?
Yes, but never with confidential knowledge from a former employer. Contributors work only from public
material, and anyone with relevant inside knowledge should stay on the public-spec side of the firewall
or sit that project out. When in doubt, read the project's
CLEAN_ROOM.md
and ask first.
Why use AI to build it?
Because the giants had thousands of engineers and we have a movement. AI agents let a small group turn a
precise spec into working code quickly. Humans write the specs, review every change, and own the result,
and compatibility tests against real SDKs decide what counts as done.
Why Go?
Portability. A Go project compiles to a single self-contained binary, so self-hosting means dropping one
file onto a server or into a Docker or Podman image — no installer, no runtime to set up, no tree of
shared libraries to keep in sync. The same toolchain cross-compiles for Linux, macOS, Windows, x86 and ARM,
and the result fits in a minimal container image. If you're leaving a SaaS vendor, the replacement should be
this easy to carry.
Why not Rust?
Nothing is wrong with Rust, and we're not against it. Go was picked for the first projects because it makes
the one-file, container-friendly drop-in the default with very little build setup. Rust can get there too;
Go just gets there more simply for the kind of servers we're building.
Rust recreations of these projects are welcome — they just haven't been made yet. The specs, API
contracts and compatibility tests are language-neutral, so a Rust port can be checked against the same
SDKs. If you want to start one,
open an issue
and say which project.
Is it production-ready?
Not yet. Both first projects are early and say so plainly in their READMEs, along with exactly which
features work today. Try them, break them, and file issues.
What does it cost?
Nothing. Every project is released under the Apache License 2.0. Run it, fork it, sell services around it.
The only thing we ask is that you contribute back what you can.
Kernel prepatch 7.4-rc1
Linux Weekly News
lwn.net
2026-11-01 19:17:08
The 7.4-rc1 kernel prepatch is out, and
the merge window is closed for this development cycle.
This is a *BIG* merge window. I think the reason is the by now
obvious one, but I'm actually taking it as a good sign: I'm *much*
happier with the flood of AI triggered work going into the merge
wind...
The
7.4-rc1
kernel prepatch is out, and
the merge window is closed for this development cycle.
This is a *BIG* merge window. I think the reason is the by now
obvious one, but I'm actually taking it as a good sign: I'm *much*
happier with the flood of AI triggered work going into the merge
window than it showing up mainly as extra fixes during the -rc
period. So I'm hoping that this merge window being larger than
usual is a sign that the AI fixes are becoming more part of the
"normal development" than a flood of fixes that go into the whole
stabilization phase. I think that's a much healthier model.
There were 18,591 non-merge changesets brought in during this merge window
— this is not just a record, but it nearly matches the patch volume during
the entire (record-setting) 7.3 development cycle. So far, 2,678
developers have contributed to 7.4; 625 of them are first-time kernel
contributors. Subscribers can find more details in
the KSDB 7.4 page
.
The LWN site is currently under high scraper load, so comment
display has been suppressed for anonymous users.
If you are a
human, you may read the comments by clicking the button below:
Note
: you can avoid this step in the future by logging
into your LWN account.
The Megadonors Who Have Poured $1.2B into the Midterms
I've been getting the occasional confusing Python dependency error since switching over to Python 3.15 and the rule I'm having to remember is:
If in doubt, delete the uv.lock file
Most of my projects are libraries which don't include a uv.lock file, since it's rude for a library dependency to lock v...
I've been getting the occasional confusing Python dependency error since switching over to Python 3.15 and the rule I'm having to remember is:
If in doubt, delete the uv.lock file
Most of my projects are libraries which don't include a
uv.lock
file, since it's rude for a library dependency to lock versions for anything else that uses it.
Running
uv run pytest
etc tends to leave
uv.lock
files lying around, which are listed in
.gitignore
and so do not get committed to my repositories.
But... sometimes that
uv.lock
pins a version of a dependency that doesn't yet have a Python 3.15 wheel. An example from earlier today:
error: Failed to build `watchfiles==1.1.1`
cause: Call to `maturin.build_wheel` failed
error: the configured Python interpreter version (3.15) is newer
than PyO3's maximum supported version (3.14)
Deleting that file and running
uv run pytest
again usually fixes the problem.
U.S. workers took home their smallest share of capital since at least 1947
As corporate earnings soar and U.S. GDP balloons, the American workforce isn’t feeling the same boom. American workers are taking home less of the country’s overall income, data from the Bureau of Labor Statistics shows, and employment in the U.S. is set to continue to slow.
Labor share, or the portion of the U.S.’s economic output that workers receive through salary and wages, decreased to 52.8% in the second quarter of 2026, its lowest level since the BLS started recording this data in 1947, according to its
labor productivity and costs report
. In the previous quarter, labor share was at 53.6%. This decade, the labor share average was about 55.6%.
That’s despite corporate earnings skyrocketing, with profits for Fortune 500 companies
hitting a record $2.1 trillion
in 2025. The
U.S. GDP grew 3.4%
in the third quarter last year, exceeding economists’ predictions.
That growth has not only come at the expense of how much of the pie of wealth workers are taking home, but also how many Americans are in the workforce, economists warn.
“That decline in the share of labor has got to be either falling earnings or falling numbers of people,” Raymond Robertson, a labor economist at Texas A&M’s Bush School of Government, told
Fortune.
“The falling share of income is having to do with the shift towards capital.”
Indeed, there are growing signs that as national income balloons, the U.S. workforce is deflating. Unemployment
ticked up to 4.2%
in September, and employers added just 584,000 jobs in 2025 compared to 2 million added in 2024. The U.S. added 29,000 jobs in September, about one-third of economists’ projections.
The bifurcation of corporate victories and weak labor data raises concerns among economists of
jobless growth
jeopardizing the U.S. workforce, as well as a
K-shaped economy
, where the rich get richer while the poor get poorer, becoming more exaggerated.
“Data right now is very mixed,” Robertson said. “But I think it also all consistently points to this idea that things are getting worse for workers and much better for billionaires.”
What is causing jobless growth?
Robertson attributes weakening labor share averages to the rise in automation, which he noted is displacing workers, with productivity—a metric essentially measuring worker output—continuing to rise. Second-quarter GDP data showed nonfarm productivity growth
grew 1.4% last quarter
, though the rate has slowed from 2.2% in the same quarter a year ago.
“All these things, bit by bit, are replacing people, and they’re concentrating income and their share of capital,” he said.
Goldman Sachs
analysts
Joseph Briggs
and
Sarah Dong
estimated in a report, based on Department of Labor job numbers, that AI automation could
displace 25% of all work hours
. They predicted that over the course of the AI adoption period, a 15% increase in AI-driven productivity would displace 6% to 7% of jobs, and, at its peak, a 1 million increase in unemployed workers, though job displacement as a result of AI has
not shown up meaningfully
in any economic data.
The displacement could be substantial, the analysts said, but said the impacts of automation will be tempered by a wealth of new jobs created as a result of the technological changes.
Automation is expected to be a boon to corporate profits and GDP, projected to boost GDP by 1.5% by 2035, according to a
Wharton brief
published in September 2025. Early signs indicate AI could drive productivity gains, with around 80% of employees reporting AI makes them faster at specific tasks, but only 37% of companies seeing an impact of that productivity on operating profits, according to
McKinsey data
, and AI operating costs continue to constrain its deployment.
Robertson added that growing unemployment, which he expects to see rise over the next few months, keeps wages down, allowing margins and profits to expand.
To be sure, the recent productivity surge has been an “
open question
,”
Morgan Stanley
economists have argued, not unanimously attributed to increased adoption of AI or automation. The analysts suggested this increase would be cyclical, or vestiges of pandemic-era habits of companies making more from less.
An
Oxford Economists research brief
from earlier this year suggested companies are disguising overhiring-related layoffs as a result of AI, but said automation-related workforce reductions have not yet happened en masse. Additionally, while unemployment has been ticking up over the past year, it is still relatively low.
Why is an immigration crackdown backfiring on U.S. labor
?
Mark Regets, senior fellow at National Foundation for American Policy (NFAP), sees a different reason for a slowing workforce. He told
Fortune
President Donald Trump’s
immigration crackdown
has not done what Trump administration officials, such as White House Deputy Chief of Staff Stephen Miller, said it would in
increasing the number of U.S.-born workers
. Instead, according to Regets, Trump’s immigration policies have not only decimated the foreign-born workforce, but has also created fewer opportunities for domestic-born workers to find jobs.
“The data is raising huge red flags that we are losing immigrants of all types that we otherwise would be advancing America’s economy,” Regets said.
Recent
BLS data
is counter-evidence to Miller’s argument that harsher immigration policy would grow the U.S. workforce. The labor participation rate for foreign-born workers fell from 66.3% to 66% from September 2025 to September 2026, and labor participation for native-born workers has similarly fallen from 61.6% to 60.9% over the same period. Fewer immigrant workers may actually make it harder for U.S.-born individuals to find work.
“A company unable to find the workers it needs for some roles could shut down operations rather than continuing,” Regets said.
He noted skillset diversity in a workplace could boost productivity and justify employing more people. Greater immigration can also increase consumer spending and stimulate businesses, as well as encourage businesses to take advantage of ample labor market availability and seek out their labor instead of offshoring jobs.
How does the U.S. reverse a shrinking labor force
?
While friendlier immigration policies could help reverse an exodus of foreign-born workers, Robertson said addressing the workplace automation push would be key to growing the U.S. workforce.
“There are trades that are technology-assisted,” he said. “Those are going to be in higher demand, but you really still have to have a significant investment in skills.”
The young generation of workers are already prepared to adapt to a changing labor landscape. Gen Z are
flocking to trade schools
in hopes of a finding a job as a carpenter or welder not so easily outsourced by AI, and in 2024, enrollment in vocation-based community colleges
increased 16%
, according to data from the National Student Clearinghouse.
Companies have taken it upon themselves to
provide reskilling opportunities
to employees. A CareerTrainer
survey
from June found that 72% of hiring managers intended to reskill employees. While the U.S. Department of Labor
updated guidelines
to encourage states to adapt workplace development systems, Robertson argued the government hasn’t done enough in several decades to imbue the workforce with necessary skillsets for future jobs.
“Democrats and Republicans have not significantly invested in training [or] the retraining or active labor market programs that you need to match workers to jobs,” Robertson said. “That’s the obvious solution.”
Without changes, economists see the pattern of an employment slowdown continuing, but with greater concern about the ability for the U.S. economy to sustain growth.
“We need job growth to have a growing economy, and I think we need job growth to pay our debts,” Regets said. “I don’t know how you have job growth with a shrinking labor force.”
A version of this story was published on Fortune.com on Jan. 13, 2026.
TL;DR
: LaTeX in Org mode now go
whoosh
, all pretty like. Install the development branch of Org mode to use it.
After grumbling about the inadequacy of LaTeX previews in Emacs – and specifically Org mode – for over a decade,
Tecosaur
and I decided to take the leap and rewrite the feature from the ground up. The result is a preview system for LaTeX math that can provide near-WYSIWYG feedback but retains the properties we expect in the Emacs environment: text-first, extensible and composable.
It’s not much code, but it contains many moving pieces and makes contact with several corners of Emacs. Now that the feature is part of Org mode, I thought I would take this opportunity to document how it works from the ground up, and use it as a launchpad to explain relevant aspects of Emacs’ event loop and its unofficial package API
Emacs doesn’t really have a “package API”, since you build Emacs packages from the same elements as Emacs itself.
.
This article is not an exhaustive demo of all the shiny new features, although there are a few videos peppered throughout. It’s also not the user manual (not yet updated on orgmode.org) or a tutorial (see Tecosaur’s
This Month in Org
writeup).
Instead, it aims to provide an overview of the goals and design of the library, along with some explainers of relevant aspects of the Emacs API and tips to write your own features. This write-up is for you if you are interested in
a broad understanding of LaTeX preview generation (how and why),
learning about Emacs’ event loop, subprocess interaction, overlay-based display and UI, or
a case study on building a non-trivial Emacs “application” from scratch.
What is
org-latex-preview
?
This is a feature included with Org mode to preview LaTeX math snippets as images in Emacs buffers. The contents of LaTeX fragments and environments in Org mode are replaced with rendered images.
It has existed for decades at this point, but in a form that I consider barely usable; this rewrite overhauls the code base and adds a number of quality-of-life features. There is a 13 minute demo that can give you a good idea of the new capabilities of Org mode:
The bullet points, if you prefer:
Preview generation is fully asynchronous
Or as asynchronous as Emacs in its current form allows.
, with images placed continuously as they are generated
Bulk generation: Preview 200 to 800 snippets per second
Previews blend in correctly with surrounding colors
Automatically match the current Emacs theme (even through theme changes)
Previews are aligned with and scaled to match the font baseline and size
Previews auto-generate for newly inserted LaTeX
Previews auto-update as you change the underlying LaTeX
Preview as you type: Live-previews of LaTeX fragments
Consistent (correct) equation numbering across the document
LaTeX compilation error reporting for previews
And a couple more quick demos:
The rest of this article is mostly about how it works, and briefly why. As such, it has a “fractal” level of detail. There is information intended
to give the casual reader a bird’s eye view of LaTeX preview generation,
to give the motivated reader a detailed look at its many pieces,
for the potential new Emacs package author looking for pointers on aspects of Emacs’ API,
for folks wanting to develop a similar feature for applications
other
than Emacs,
and information that is of no use whatsoever except to the future maintainers of the
org-latex-preview
library.
To that end, many details are tucked away in blocks like the following. You can safely ignore them in favor of the top-level headings, images and video demos.
Implementation details
Blocks like this describe the concrete implementation of aspects of the code.
This is probably only interesting if you are looking to dive into the code itself, or if you are looking to write a similar package.
Emacs API notes
Blocks like this are for additional notes on aspects of Emacs’ API, including the odd code snippets.
Warning: Here be idiosyncratic takes.
Org Mode notes
Blocks like this describe other Org features relevant to LaTeX preview generation.
Command output
Blocks like this describe CLI commands associated with this process, or their output.
Isn’t this a narrow feature to spend so much time on?
Indeed, this feature
requires a LaTeX distribution installed,
works only with Org mode (for now),
and is glued together from several processes and grossly inefficient in the larger scheme of things.
It’s also finicky, as all hyper-configurable software is.
LaTeX was not built with the aim of minimizing latency. Having to start multiple new processes even for a one-character change places a hard cap on what’s possible in terms of real-time feedback. Implementing an equivalent preview feature for a system designed for modern computers, like Typst, is trivial in comparison.
Moreover, as popular as Org mode is, it’s a small player in the space of markup languages. The fraction of Org and LaTeX users who would prefer to use Org mode to write LaTeX is smaller still. And while Org attempts to integrate well with LaTeX, navigating the mishmash of Org and LaTeX syntax presents an extra learning curve on top of learning to use the two markup languages individually.
But it also has a few things going for it.
First, LaTeX is ubiquitous, and Org mode is a far more convenient format for writing than raw LaTeX. There is significantly less visual noise (think Markdown vs HTML), and Org provides an escape hatch for adding LaTeX verbatim. Org’s simple interface for shuffling text around, monitoring task progress and executing code blocks makes it easy for eventual LaTeX documents to begin as Org files. Since everything in Emacs composes with everything else, your system for (say)
fast LaTeX input
transfers over to Org mode seamlessly.
Second, the fact that Org is versatile as a publishing tool implies that some benefits of better LaTeX support transfer over to other formats too, like equations in HTML or ODT exports.
There is a wide swath of applications built on Org mode that benefit from LaTeX math input, such as Org Roam and Org Noter. For these applications converting Org documents to LaTeX is not the goal, as they are not primarily publication tools: they stick around as private digital archives, and better LaTeX support is a bonus to the user’s expressivity.
Real-time feedback when editing LaTeX snippets is a categorical improvement to the user’s experience. I did not realize this when I started work on
org-latex-preview
, but now I wouldn’t want to LaTeX without it.
It baffles me that web LaTeX editors (Overleaf etc) don’t provide this out of the box, especially as they target novice LaTeX users.
.
Finally, we
can
keep Org mode out of the picture:
org-latex-preview
can technically work anywhere in Emacs, and in every major-mode. it’s just not there yet
(See
below
)
.
But perhaps the most germane reason is that Emacs users don’t shy away from settling comfortably into tiny, cramped and poorly insulated niches. Comes with the territory!
The Objectives
Never Block Emacs
Emacs users are closely acquainted with its frequent unresponsiveness. You’re never more than one minor-mode or mode line indicator away from causing Emacs to block or lag. While it’s not a big ask to wait a second for the result of an action
you
initiate, the unresponsiveness often manifests as short random freezes, which is unacceptable. But there’s more. Beyond the annoyance and inconvenience, poor responsiveness denies us an entire class of uses: real-time feedback is everything for interactive applications.
Owing to both its design and age, Emacs is not very well suited for asynchronous applications. Certain patterns of asynchronous usage are viable (see below), but concurrency support is limited and parallelism is absent: no worker threads, channels, async-io, or anything of the like.
Generator-based async in Emacs lisp
I lied. In addition to Elisp threads, there exist prototype libraries for async-io (
Chris Wellons
) and channels/CSP (
Andrey Listopadov
) based on Emacs’
generators library
, which in turn is implemented using continuation passing. But generators as provided by this library do not have anywhere near the level of robustness or support you need to build actual applications: no introspection, error handling or debugging support, poor performance, and just overall jankiness. I’ve tried and lived to regret it –
Elfeed Tube
is a real mess.
The
Futur
package is another promising swing at an async-io style interface, but it’s new and I haven’t been able to try it yet.
In any event, a more responsive
org-latex-preview
would require parallelism and not just concurrency. There is John Wiegley’s
async library
(and packages inspired by it, like
session-async
) that spawn a child Emacs that runs in parallel, but communication is limited and the overhead is sizeable, so this is a bit of a bludgeon.
The upshot is this: we want
org-latex-preview
to be as asynchronous as possible within Emacs’ effectively single-threaded event loop model. The goal is to minimize how long the user has to wait, in terms of both the overall run time and the fraction of that for which Emacs is unresponsive. We end up having to trade these off against each other.
Go Fast (relative to LaTeX and Emacs)
A close second concern is preview generation speed. This is desirable both in the large and the small. Previewing upwards of a hundred LaTeX math fragments in a math-heavy Org document used to be an exercise in patience. Now it takes about one second, and can be much faster with some trade-offs.
LaTeX Fragments
Old preview system (s)
org-latex-preview
(s)
Speed up
35
12.96
0.43
30x
70
22.59
0.51
44x
160
33.99
0.68
50x
330
84.95
1.32
64x
660
139.73
1.96
71x
1000
194.16
3.24
60x
Figure 1:
LaTeX preview generation times under the slowest settings. The vertical axis is time, and along each vertical line the red chunks correspond to intervals for which Emacs is blocked. We still manage to process a file with 250 LaTeX fragments in about a second with minimal impact on Emacs’ responsiveness.
Provide Real-Time Feedback
Speeding up bulk generation is at most a boost in convenience. You don’t regenerate a large buffer’s worth of previews more than once every few sessions, and the resulting images are cached anyway. The more interesting aspect is that speed enables a close to real-time feedback loop that wasn’t available before. This is especially useful when you’re deep inside a messy structure like an array:
No more counting field separators. No more exiting, regenerating and re-examining snippets. It’s the best parts of WYSIWYG and visible markup, together.
Look Pretty
Even in the niche of academic types who use Org mode with LaTeX, there are very different orientations. There’s Org the transient format, where an Org document starts out as a rough sketch, a thought bubble translated to the screen, and soon transitions into a TeX file that becomes the source of truth. There’s Org the intermediate publication format, where the document lives as an Org file, but its value is in the web page or PDF document it eventually becomes. For the adherents of this style the appearance of the Org document is irrelevant.
Finally, there’s the class of uses described
above
, where the desired output is the Org document itself, or a cross-linked collection of them. These are folks who care about the aesthetics of Org documents and often suffer a thousand cuts in trying to make Emacs look “modern”.
One of the objectives of
org-latex-preview
is to ease their self-imposed burden somewhat: LaTeX previews should blend in perfectly with the surrounding text. This means
preview image sizing that is consistent with surrounding text,
with perfect baseline alignment at any font size,
and Emacs theme-aware colors that change automatically with the theme.
Getting the fonts to match is possible too, but requires some degree of compromise on the part of the user
You could start by using the variable-pitch Latin Modern font in your Org documents.
.
Each of these is a long-standing issue with LaTeX previews in Org, and Stack Overflow and the usual haunts have decades of discussions on how to fix them, with band-aid after band-aid proposed as solutions. We took an elephant gun to this problem, starting by throwing out the giant mess that was Org’s accreted LaTeX preview code of the past two decades.
Here’s a demo of theme-aware LaTeX preview colors:
Windfalls: Consistent numbering, error reporting
Finally, there are a few features we picked up along the way because they came free.
Recording compilation errors was a simple matter since we scan the LaTeX output anyway.
Consistent equation numbering was expensive in the past because numbering changes can propagate across the document, and parsing the Org document and LaTeX fragment compilation were both slow. Recent changes to Org have made the former a non-issue (see the
parsing section below
), and the
performance scaling
of our bulk LaTeX runs has fixed the latter.
There are several more minor features along for the ride, such as improvements to editing LaTeX with
org-edit-special
(
C-c '
).
The Emacs API: An Overview
As mentioned in the introduction, the LaTeX preview system draws on a few different facets of the Emacs API: mainly async process interaction (the “backend”) and overlays (the UI or “frontend”). The live-updating behavior leans heavily on hooks and timers, and we join these pieces together with very simple Lispy data structures, like alists and plists.
We also (naturally) use a few Org mode specific APIs, including ones developed by us for this purpose. We cover these further below as we encounter them.
The Substrate: Emacs’ Event Loop
If you are designing an Emacs application that interacts with the outside world, it helps to start with a mental picture of Emacs’ event loop. To a first approximation, the main loop does only four things:
Read input events: This includes keyboard, mouse and touchscreen events, but also some OS signals. This is the user’s interface to Emacs.
Evaluate commands and redisplay: This is the heart of the Lisp evaluator, and Emacs’ interface to the user.
Together these two connect the user and Emacs. If the main loop did only these two things Emacs would still qualify for the title of “Lisp evaluator masquerading as a text editor”, but it wouldn’t be nearly as useful. For that we need it to
Read subprocess output: This includes output from “regular” processes like shell commands or LaTeX, but also network requests. This connects Emacs to the world
(your computer and the network)
, and vice-versa. Command shells, LaTeX previews, LLM clients and most IDE features work because of this.
The last event loop function combines the above three, but across time:
Run timers: Schedule code for the future and run them when timers are ripe. Somewhat tentatively, I’m including garbage collection here.
This design explains why timers in Emacs are not reliable in the absolute sense. The timer runs not when it’s ripe, but when it’s ripe
and
when the main loop gets around to it. Nevertheless, scheduling code provides a much-needed degree of flexibility to a rigidly ordered task list.
“To a first approximation”
The way these four tasks are actually organized in the main loop code is less pleasing:
Also, Emacs is obviously doing things outside of the event loop as well. Font caches are compacted, Emacs is responding to the compositor and some OS signals, interacting with dynamic modules, and so on. In the currently experimental MPS branch, some garbage collection activities also occur in the background. These are invisible to the Lisp evaluator and thus out of scope when learning to build an Elisp package.
The Backend: Subprocess Interaction
This design suggests that
Emacs can only do one thing at a time
As of Emacs 31
, so that designing a responsive application is a steep ask. But the situation is more nuanced:
Yes, the Lisp evaluator holds a global interpreter lock. At any point in time Emacs is doing exactly one of the following: painting the screen (redisplay), updating syntax highlighting (fontification), running hook functions or timers, responding to window changes, handling keyboard input, and – crucially for us – interacting with subprocesses. Every action is thus
blocking
.
The only
asynchronous
interaction possible is by starting an OS-level subprocess
Another lie, since we have
dynamic modules, covered below
.
. Emacs continues to run independently from the subprocess, but it still has to handle its
STDOUT
and
STDERR
streams and respond to process signals. When it is doing this, it cannot pay attention to your input or anything else. Again, Emacs can only do one thing at a time.
Starting asynchronous subprocesses
There are multiple commands to start an asynchronous subprocess from Emacs (see
M-x shortdoc
process
), but
start-process
covers most common use cases.
Processes are (optionally) associated with a buffer where process output is inserted. To send a subprocess some input, there’s
process-send-string
and
process-send-region
. Responding to process output and signals is the responsibility of process filters and sentinels (
next block
).
kill-process
does what you’d expect. All of these functions take the process object that
start-process
returns as an argument.
Except for some
fiddly details
, that’s essentially all there is to the process interaction API. There are several variants of
start-process
available – the synchronous
call-process
, the shell versions
start-process-shell-command
and
call-process-shell-command
, and so on. The only one that does something different is the primitive
make-process
, which is a more extensive version that can (additionally) create network processes, send data through TCP or Unix sockets and so on.
Emacs will try to prioritize user input events. To do this it can delay tasks like reading from subprcesses if necessary.
Idle timers
are another example of this. Some actions are thus
delayable
, but they block Emacs when they (eventually) run. There might be a penalty to pay afterwards, for instance when there’s more subprocess output to process as a result of delaying the task.
Process filters and sentinels: Handling process output
By default, Emacs inserts the output from a subprocess into the process buffer, or ignores it if there is no process buffer. (Crucially, it still has to read the output from the process pipe.) This is good enough for simple tasks. To use or transform subprocess output in more specialized ways, you can write an elisp function that is used as a
process filter
. It is called whenever Emacs reads new process output. In the language of shell commands, I think of process filters like this:
Process filters can be tricky to write if you want to parse structured process output, because the amount of data that Emacs reads from the pipe can be arbitrary. Unless latency is a concern, it’s often preferable to work with
process sentinels
for this reason. Sentinels are the second tool Emacs makes available for reacting to processes. A process sentinel is an elisp function (callback) called whenever the process changes state or raises a signal, including when it terminates. Practically speaking, I think of them like this:
They are typically used for cleaning up after the process, and updating Emacs’ state depending on how the subprocess ends. But you can use them to access the process buffer containing all the process output, and thus do whatever you would in the filter.
Filters and sentinels can be attached to a subprocess object created with
start-process
using
set-process-filter
and
set-process-sentinel
respectively.
Many blocking operations can be interrupted by signaling a
quit
: this is what pressing
C-g
does, with some variations, in most contexts. Crucially for us, however, quitting is suppressed when Emacs is handling subprocess output or signals
This is to keep your
C-g
from doing something unpredictable to “background” code that you didn’t initiate.
. If Emacs is blocked because it’s reading a large data dump… well, tough luck.
Interrupting code
keyboard-quit
(
C-g
) is a
specific
input that interrupts
any
code, with a few exceptions.
The obverse of this is to wrap code in a
while-no-input
clause – now
any
input will interrupt the processing of
this
specific block. This can be used to do background processing in small chunks without making Emacs unresponsive, but there are subtleties involved that I’m not familiar with.
Despite these carveouts, a package that interacts heavily with external processes faces a latency-throughput tradeoff. The more time Emacs spends starting processes and managing their output, the less attention it can pay to your input. Designing Org’s LaTeX preview thus involves finding an acceptable balance.
Figure 2:
(Top) Emacs dealing with a “fat” high-throughput process, where it spends a chunk of time reading subprocess output. The time between reading your input and running the resulting command is thus longer than (bottom) the “thin” low-throughput case.
It is
significantly
easier to write a large Emacs library than an equally versatile plugin for most plugin systems built around multi-threaded GUI applications. I say this as a non-programmer who has only succeeded at the former. We lean on the interpreter lock in many ways, choosing
simple data structures
and
control flow
that wouldn’t work if Emacs was multi-threaded. But most of Emacs wouldn’t work if Emacs was multi-threaded, anyway. In for a penny…
The Frontend: Displaying images over text
Overlays
are a way to alter the appearance or behavior of regions of buffer text without modifying the text itself. Displaying images is one of their most common uses: LaTeX preview images are overlaid on top of the source fragments.
Once you have the preview images in hand, placing them in overlays and managing their state is essentially all the LaTeX preview code does.
In fact, overlays account for much of the display trickery in Emacs: Images over text? Overlays. “Virtual text” like in
whitespace-mode
? Overlays.
Inline results
of code evaluation: overlays.
Multiple cursors
? Those are overlays. Emacs has only one cursor.
Making overlays
For a quick look into the overlay API, run
M-x shortdoc
overlay
.
The even shorter version: We can create an overlay
ov
from buffer position
BEG
to
END
with
(setqov (make-overlayBEGEND))
and move it around with
(move-overlayovNEW-BEGNEW-END)
To avoid creating garbage, it is preferable to reuse overlays when possible instead of throwing them away with
(delete-overlay ov)
and starting over.
move-overlay
can move them to other buffers too.
The only other tricky question is what happens to the overlay when text is inserted at its left or right boundaries. Does it expand to contain the text? This is left as an exercise for the reader.
These alterations are specified by adding or modifying the overlay’s “properties”. Like
symbol properties
, overlay properties are a way to store arbitrary information along with the overlay, and
org-latex-preview
uses them to keep track of state, such as whether the preview image is out of date because the LaTeX fragment has been edited, and to store LaTeX compilation errors for the corresponding fragment.
Overlay properties
To set and retrieve overlay properties, we can use
(overlay-putovpropvalue)
(overlay-getovprop)
where the property
prop
is any elisp symbol. There is also an
overlay-properties
function to access the full list. Once again,
M-x shortdoc
overlay
has us covered.
Some properties have special meanings as interpreted by Emacs’ display engine.
display
, the most common property, changes how the overlay region appears on screen, and this is typically set to the image we want to see in place of the text.
But several more interesting modifications are possible with overlays, such as changing the face (font+style) of the underlying text, making it invisible, making it intangible to the cursor, or specifying keymaps that are activated when the cursor is within the bounds of the overlay.
Fun with overlay properties
The only other detail we need is the
list of overlay properties
that Emacs considers special. The
display
property can be used to show an image instead of the text “below” (or “under”) the overlay, for example:
You can use this to set up a “watcher” – perhaps run a process when a region inside a buffer changes
If you want to set up a watcher on the contents of the entire buffer as opposed to a piece of it, you should avoid overlays and look at the hook
after-change-functions
instead. Be warned: it’s easy to slow down Emacs to a crawl this way if you’re not careful.
! This is not hypothetical:
preview-latex
does this to
re-dump the LaTeX preamble
of TeX files (in the background) as you edit them.
Add a highlight and tooltip (“hover text”) when the mouse is on the overlay:
(overlay-putov'mouse-face'highlight)
(overlay-putov'help-echo"You're on an overlay, pal.")
Open the displayed image externally when pressing
RET
with the cursor on it:
This works by defining a
keymap
overlay property with the required function. (On non-Linux systems, you will want to change
"xdg-open"
to an equivalent command-line utility.)
This image stays centered as you resize the window:
This one needs a little explanation: The
before-string
overlay property displays an object before the overlay itself. The object we choose to display is some propertized text with a
display
text-property
. The
display
text-property provides the means to
insert spaces of specified widths
, and a symbolic way to refer to the text margins, window center and horizontal window boundary positions. Combining the two gives us (consistently) centered images in Emacs.
The Reactive UI: Hooks and Timers
In its basic form, the LaTeX preview feature involves no reactive behavior: there is one command to overlay preview images on LaTeX fragments, another to clear those overlays, and nothing else.
But this is a far cry from our goal of real-time feedback. For LaTeX previews to be automatically generated and updated as we type, we need both timers and
hooks
.
Hooks are variables provided by Emacs where we can register code to run at certain points. In this case, we want to react primarily to two kinds of user inputs:
Cursor motion, so that entering a previewed LaTeX fragment reveals its source and leaving it re-displays the image.
Buffer changes, so that a preview generation subprocess automatically fires when a new LaTeX fragment is typed in or an existing one is edited.
In turn,
org-latex-preview
exposes hooks of its own for consumers to run additional processing. These fire when LaTeX subprocesses finish, when overlays are created, and so on.
Timers are complementary to hooks, used to space out or limit the consequences of hooks firing. They are used
to keep these hooks from firing
all the time
, which would create redundant subprocesses with each edit, locking up the main loop. They provide the means to implement
throttling and debouncing
.
to delay clean-up tasks like deleting intermediate LaTeX output until Emacs is idle.
The structure of
org-latex-preview
Besides basic Elisp data structures, that’s everything we need from Emacs to implement non-blocking, fast LaTeX previews. Assuming you have a handle on what
org-latex-preview
does, we can start peering into the how.
How it works: control flow
Here is a high-level overview of the control flow of LaTeX preview generation:
Figure 3:
Control flow for Org mode’s LaTeX preview system.
This is a rough picture of how LaTeX previews are generated in Org mode, subdivided into tasks
The made-up term “task” corresponds to a handful of functions devoted, in our heads, to one purpose.
. Each action belongs to one of three categories:
Blocking
tasks: These must run synchronously and thus require the user to wait while Emacs is unresponsive. We want this to be minimal and/or done ahead of time.
Delayable
tasks: These tasks can be delayed if more immediate actions – such as accepting user input – are pending. However, once such a task is started it runs synchronously and we’re locked in. We want this work to be chunked into small independent pieces.
Asynchronous
tasks: These are subprocesses spawned by Emacs and managed by the OS. Emacs monitors the process status and handles process output in a delayable fashion, but is otherwise unencumbered by them.
When a LaTeX preview command is issued, the first chunk of processing (the
PREPARE
block)
sets up overlays in the buffer that “cover” the fragments, with the intent to add preview images to them at the end,
creates a TeX file containing the relevant LaTeX fragments from the Org buffer, along with a preamble. This preamble is optionally precompiled, which can speed up processing considerably.
It then starts a two step process chain (the
RUN
block) that runs asynchronously. The end result of a successful run is a preview image or series of images.
The
PROCESS FILTERS
block: From this point on until either process ends, Emacs checks in periodically through the associated process filter, acting on the
STDOUT
to capture various bits of preview metadata. It also keeps track of images that have finished processing, and (synchronously) caches them and adds them to the overlays in the buffer for these fragments. This continuous update is doubly useful:
Previews start appearing in the buffer much faster, with the window viewport being updated first.
This processing happens in small chunks instead of all at once at the end, so Emacs stays more (even if not fully) responsive to input.
The
POSTPROCESS
block: After the subprocesses have finished, Emacs runs their process sentinels, which clean up the detritus of this process chain, or reset the buffer if the preview run failed.
And speaking of detritus…
How it works: file I/O
This figure shows the same process but with an emphasis on disk I/O. The primary chain is
TeX file
→
DVI/PDF output file
→
SVG/PNG preview images
.
Of the file categories involved in the process, we care about two: the “format dump file” produced by precompilation, and the preview images. Both are cached and persisted across Emacs sessions. All other produced files are deleted by the POSTPROCESS block.
How it works: data flow
Displaying previews that are color-, alignment- and size-aware, along with associated information like errors, requires a considerable amount of bookkeeping. Since this information arrives piecemeal and processing happens asynchronously in chunks, we also need some way of storing and passing along this context. There are perhaps clever ways to do this, but we use a straightforward solution – a (dynamic) global plist that associates each active process chain with its data (see
org-async
below).
All of the data generated and required for the many tasks in our process chain are captured in a single mutable container, referred to here – for convenience – as “Extended Info” (
extended-info
). This data includes
User preferences for the preview run, such as the LaTeX compiler in use,
Input LaTeX strings to be rendered,
Input file paths, output image paths and image cache details,
Emacs’ context like the current buffer – which is important since we need to restore it before we can run asynchronous callbacks,
Overlay positions and properties,
Compilation errors,
Computed preview geometry and fontsize for alignment,
and more besides.
extended-info
: implementation
“Extended Info” is implemented as a plist (property list). Actually, almost every compound data structure
org-latex-preview
uses is either an alist or a plist, which is just a linked-list under the hood.
extended-info
is also a tree since a few of its members are plists themselves (
see below
).
We used property lists at first since they’re mutable and schema-less, and the design of the library was very fluid. Unfortunately we are now well past the point where we could switch to something more structured like
cl-defstruct
, so plists it is.
If it’s any consolation, most of Org works with large property lists too. Org’s export system passes around a big
plist
bucket for communication in much the same way we do here. Access can get expensive as they grow, but at least passing them around is cheap. It’s… not much of a consolation.
apply
yourself: property lists as arguments
plists have one other coincidental(?) advantage: You can splice them in as arguments to a
cl-defun
.
Consider:
(cl-defunfoo (&key (buf (current-buffer)) (valid-pt) (pred#'ignore)
&allow-other-keys)
;; body of foo )
This function has the signature
(foo:bufsome-buffer:valid-pt:pred#'always)
which is just a list with the name of the function followed by… a plist. So we can build up the plist programmatically and use it directly as
foo
’s argument list:
’tis a silly thing but oh, so satisfying. In fact, this is a convenient way to feed a job spec to
org-async
(
see below
).
How
extended-info
is populated
The setup functions in the
PREPARE
block initialize this bucket with data from parsing the buffer – i.e., LaTeX fragment locations, equation numbers, etc. From then on, each task updates
extended-info
with progressively more information. The structure of
extended-info
and this flow of information into it is visualized in this “data flow” diagram:
Contained in “Extended Info”: “Processing Info” contains information about and relevant to the external process (LaTeX and the image converter). “Fragment” is a list of metadata for fragments to be previewed, including the overlays that will eventually display the preview images. All of this preview metadata is obtained from the outputs of the latex and image converter runs via the process filters.
Note that Emacs’ interpreter lock precludes data races and other concurrency issues, since only one function modifies
extended-info
at a time.
How
extended-info
is accessed
The above chart only captures the flow of data in one direction: from the sources of computation into the
extended-info
container. Tasks that don’t modify
extended-info
still need to read it to do their jobs. For example,
Clean up temporary files
needs the paths to the temporary files.
Cache images and metadata
needs the generated image paths, as well as all the geometry metadata.
Most functions in the process chain are passed
extended-info
, or a subset thereof, as an argument.
Once again, we get away with it because there are no concurrency problems when there is no concurrency. This design pattern of passing around a single heap-allocated container across setup functions and callbacks works for all kinds of applications in Emacs. This is exactly how Org’s (synchronous) export system manages information as well.
org-latex-preview
Pieces and the Org API
This is where we get into the weeds. If you’re here for the bird’s eye view, you can skip ahead to
Live previews
now.
Org’s syntax is
extensive
, and parsing Org files used to involve a giant mess of regular expressions. Thankfully the mess is now handled by the
org-element
library, which provides a simple and powerful API for querying or acting on Org’s AST. A fair amount of work has gone into making it perform well under the constraints of Emacs’ single-threaded event loop
Besides caching its state, it uses
timers
to schedule parsing work outside of the times the user is interacting with Emacs.
.
Crucially for us, Org has first-class support for LaTeX environments and fragments, and this greatly simplifies parsing and metadata handling. This support is what makes using LaTeX in Org a pleasant experience in the first place.
The
org-element
library
Added in late 2021, the
org-element
library greatly simplifies the process of working with Org documents.
I have a minimal working knowledge of the library, so this block covers only the aspects of
org-element
that we use in
org-latex-preview
.
Org’s syntax distinguishes between “elements” and “objects”, where anything “larger” than a paragraph is an element, and any syntax that could be contained within a paragraph is an object
Incidentally, the default
mark-paragraph
binding runs
org-mark-element
in Org mode, which selects the smallest logical element is at point instead. Try
M-h
at different locations in an Org buffer to see what element you’re on!
.
This distinction matters to us because LaTeX text to be previewed can be either. As you might expect, inline math (
$...$
or
\(...\)
) is an object, and LaTeX environments (
\begin{foo}...\end{foo}
) are elements, and they differ in the way we must display the preview images.
Every node in the AST as parsed by
org-element
is made available as a plist whose keys are properties of the element. These properties include the start and end positions of the element and its contents (
:begin
,
:end
and
:contents
), as well as metadata like properties and keywords. For example, every element can be named with a
#+name:
keyword.
These are the only things we need from
org-element
as far as parsing goes.
The objective of the parsing step is to find the bounds of LaTeX fragments in the buffer (or in some region). In principle, we could do this with
org-element
:
But this would be too slow, and would allocate far more memory than we need. Instead, we use a simple regexp scan of the buffer to find the bounds of LaTeX fragments and environments, and
org-element
only to gather specific metadata about these elements.
Many editor plugins (and web apps) implement LaTeX previews, usually via
MathJax
or
KaTeX
. KaTeX is fast, and both function well for web pages with limited needs. But they implement ersatz support for a limited number of LaTeX packages, and cannot consume an actual “full-fat” LaTeX preamble of the kind you might use when writing a research paper. There is no
\usepackage{}
macro or TikZ support, for example.
By comparison,
org-latex-preview
is backed by a full LaTeX engine and LaTeX distribution, so nothing is off the table. Of course, this means we need a way to construct the preamble and feed it to the LaTeX engine.
In addition to Emacs-wide settings (like
org-latex-preview-preamble
and
org-latex-packages-alist
), document-specific LaTeX headers can be specified with the
#+latex_header:
directive:
The combined preamble must be constructed when exporting the Org document to LaTeX, and this machinery lives in the
ox-latex
library.
org-latex-preview
simply calls into it for the preamble. Among other benefits, this means the preamble used for the preview run is identical to the one used for LaTeX exports
unless you
want
them to differ, because of course Org provides ways to do that
, so these truly are
previews
.
The
org-export
(
ox
) library
The
ox
library and its children, like
ox-latex
provide an interface to convert Org documents to any other format, like Pandoc does.
While the specifics (like collating a LaTeX preamble) are output-format dependent, the basic method is uniform and simple. An exporter maps each Org syntax
element and object
to a function that creates the equivalent in the output format, like image links to HTML
img
tags, or
#+begin_quote ... #+end_quote
blocks to HTML blockquotes elements. The fact that the Org syntax supports LaTeX environments natively helps with LaTeX exports.
Of course, when a clean mapping like this is not possible, you can work on the buffer directly and rip out and replace raw text. Emacs, huh.
This does make
ox-latex
one of the few dependencies of
org-latex-preview
, which is regrettable given the larger aim of
providing LaTeX previews across Emacs
.
To process a preamble, the LaTeX engine needs to tokenize and execute tens of thousands of lines of macro source. For a complex preamble, the kind you would use for a book, this can take several seconds on a modern processor.
Since
org-latex-preview
aims to provide close to real-time feedback as you type, we resort to a trick from the 1990s: we serialize the entire engine state – the string pool, macro bodies as token lists, font data, engine execution parameters and so on – into a single binary memory image. This process, called
format dumping
or
precompilation
, was automated for document preambles by David Carlisle’s
mylatex.ltx
, modernized as the
mylatexformat
package by Florent Chervet.
The upshot is that precompilation saves a significant amount of LaTeX startup time, reducing and (almost) flattening it across all preamble sizes.
Precompilation relies on a number of hacks and only works because pdftex has changed little for decades. Unfortunately, precompilation is not supported at all by xelatex and only partially by lualatex
preambles that load OpenType fonts or Lua code are unsupported
, so
org-latex-preview
is slow with the Unicode engines.
And speaking of being stuck with an old architecture, depending on pdflatex implies that we have to create and feed it a TeX file for each batch of previews
pdflatex and friends do accept input via STDIN, but this did not work out for finicky reasons.
.
As you might expect, the generated TeX file contains a reference to the precompiled dump file and a listing of all the LaTeX fragments to be previewed. Instead of placing them into the document separated by something like
\pagebreak
, we place them inside
\begin{preview}
and
\end{preview}
blocks provided by the
preview.sty
CTAN package.
preview.sty
is authored by David Kastrup, who also wrote the Emacs-side
preview.el
package that’s part of AUCTeX.
preview.el
was the inspiration for
org-latex-preview
– it was clear at least that previews in Org mode could be much, much faster.
preview.sty
does three critical things for us:
It places each preview snippet (enclosed within
\begin{preview}
and
\end{preview}
blocks) on a separate page in the LaTeX output;
it finds the boundaries of the generated text and crops the page down to them;
and it reports the font size, image size and baseline position in the LaTeX process’ output stream.
These modifications were technically doable through a combination of other packages (Org’s old implementation used the tightpage option of the preview machinery), but in cumbersome and error-prone ways.
This makes
preview.sty
an external dependency of
org-latex-preview
. Fortunately this package is included by default even with minimal installs of all common LaTeX distributions.
The process runner
The pipeline from the LaTeX fragment text in the TeX file to images suitable for preview consists of two steps. A LaTeX engine (pdflatex, xelatex or lualatex) produces DVI, XDV or PDF output, which is fed to an image converter (dvisvgm, dvipng or imagemagick) to produce PNG or SVG images. This is the
RUN
block of the
control flow diagram
.
This involves a fair bit of bookkeeping, and Emacs’ callback-based async process management (via “process sentinels” and “process filters”) isn’t the most ergonomic tool for this. In the absence of reliable async-io support, we use
org-async
, a declarative process chain manager now bundled with Org mode.
The
org-async
library
Because we need to run a chain of processes with different callback behaviors depending on the exit statuses,
org-latex-preview
uses
org-async
, a process queue manager that wraps Emacs’
start-process
, process filter and process sentinel API. It handles a lot of the bookkeeping for us, providing for
declarative specification of a tree of processes that branch on the exit status of a process,
including the ability to register multiple callbacks as required,
keeping track of the context (the
extended-info
bucket in our case) and passing it to the filter and sentinel,
and limiting the pool of concurrent subprocesses and handling timeouts.
The global variable that keeps track of the context data between async calls is managed by
org-async
.
The following code might be out of date, please refer to the Org manual.
An example of an
org-async
job spec might look like the following (simplified for clarity):
org-async-call
also provides other conveniences like timeouts, retries and concurrent subprocess limits.
An example of this bookkeeping is that the pipeline isn’t always serial. When generating PNG images with dvipng, we start LaTeX and dvipng in parallel. Dvipng processes pages of the DVI file as they appear, while the LaTeX process is still churning them out. (Generating PNG preview images for a large Org buffer is much faster than the other routes.)
Tracking with process filters
On the other side of the process invocation are the “process filters”, which are callbacks
called by the event loop
whenever Emacs reads chunks of subprocess output. This is the
PROCESS FILTERS
block in the
control flow diagram
.
A shell metaphor for process filters
Following the
shell metaphor for process filters
, a simple way to think of a process filter for an Emacs subprocess is as a sink at the end of a Unix pipe:
some-process 2>&1 | process-filter
In the LaTeX preview case, the picture is a little more complex as there are multiple processes:
In
org-latex-preview
’s process filters we look for the output (DVI or image) file paths, but also a variety of metadata that preview.sty and dvisvgm emit. This includes geometry information like the width, height and baseline alignment of the image, and any compilation errors.
pre-processing DVI file (format version 2)
processing page 1
applying bounding box set by preview package (version 13.1)
width=346.003754pt, height=15.362961pt, depth=.501877pt
graphic size: 346.003754pt x 15.864838pt (121.606411mm x 5.575853mm)
output written to /tmp/org-tex-ang3Se-000000001.svg
1 of 1 page converted in 0.16035 seconds
[1 (preview-latex version 13.1) (preview-latex tightpage option detected, will use its bounding box) depth=1 height=29]
Fine-tuning Emacs’ process interaction parameters
“Whenever Emacs reads chunks of subprocess output…”
This phrasing omits some important but fiddly details about how often and how much output is read at a time, in much the same way as the Unix pipe
|
symbol does.
While the event loop is switching between
its four main tasks
continuously, there are (only) a couple of knobs available at the Elisp level to manage how Emacs reads subprocess output.
read-process-output-max
limits how much it reads in one event loop turn. Setting it to a high value means output from chatty processes will be read sooner, but at the expense of blocking user input. This is, essentially, Emacs’
latency vs throughput
curse in a nutshell.
process-adaptive-read-buffering
tries to delay reading from subprocesses that produce very small chunks of data at a time, as this is inefficient. In practice turning this on has tended to make things worse instead of better when running most subprocesses.
Trying to optimize this for
org-latex-preview
has proved to be… difficult. The rates and sizes of output emitted varies significantly between different LaTeX engines and image converters:
Figure 4:
Intervals for which Emacs is blocked when previewing using SVG and PNG images. The vertical axis is time, and the solid chunks are periods where Emacs is unresponsive.
They differ even between different versions of the same program across Texlive versions:
Figure 5:
A time-aligned flamegraph profile showing process filter calls for the LaTeX and image converter processes when using Texlive 2022 and 2023. In the aggregate, the LaTeX and dvisvgm binaries shipped with Texlive 2023 aren’t faster or slower than the former. They just have a different profile across time to which it is difficult to adjust.
org-latex-preview
settles for a broadly acceptable set of defaults in this regard.
But the main reason we use process filters here is that we can place images in the buffer as they are generated (and read in from the process STDOUT by the event loop), and don’t have to wait for the LaTeX or image converter processes to end.
Cleaning up with process sentinels
The unfortunate result of being forced to do file I/O at each step of the pipeline is that we are left with a trail of intermediate files that need to be swept away
Instead of, say, a block of shared memory that can simply be freed.
. This is the
POSTPROCESS
block in the
control flow diagram
, accomplished in the “process sentinels”.
A process sentinel is a callback invoked by the event loop when the process changes status. In principle this includes any status change (exit, signal, stop/continue), but in practice we only care about it after the process exits.
Clearing the intermediate detritus of the compilation is not an urgent task, so
org-latex-preview
’s process sentinels actually schedule this on an idle timer. Their other major function in this design is to diagnose process failures and restart the job when the cause looks transient.
LaTeX preview generation is an order of magnitude (or two) faster than what Org did before, so caching preview images might seem like a secondary concern. But
org-latex-preview
still caches images, for a few reasons:
The same state may be encountered dozens of times when editing a fragment with live previews.
We would like to avoid rerunning LaTeX in contexts where the Org document itself is the final output, such as in-Emacs slideshow programs.
When exporting to other formats (HTML or ODT),
org-latex-preview
is responsible for generating the images to be bundled with the web page or document.
In addition to the text of the LaTeX fragment itself, preview images are specific to the LaTeX preview pipeline configuration, the preamble, the Emacs theme and even the fragment’s position in the buffer when equation numbering is turned on. To cache previews reproducibly, all of these inputs are hashed together as the key to the cached image.
The
org-persist
library
While you can specify where the preview images should be cached relative to the document,
org-latex-preview
defaults to using
org-persist
, a general-purpose caching library that is part of Org mode.
org-persist
is used for various caching duties by Org mode, such as resources fetched from the Internet and the partial syntax trees of parsed Org buffers.
Caching LaTeX preview images actually involves more than copying a file somewhere and serializing a hash table to disk. We also need to store, as Elisp data, the image display geometry information, including the baseline alignment height. The
org-persist
API supports caching different container kinds for exactly this purpose.
Finally, with sufficient time (or a large enough corpus) the cache of LaTeX images can grow to gigabytes in size, so cache entries are set to expire after a month, another function served by
org-persist
’s built-in garbage collection.
Displaying images and overlay management
This covers the in-buffer UI and affordances of
org-latex-preview
, indicated by the
Create overlays
and
Place LaTeX previews
blocks in the
control flow diagram
.
Beyond what is covered in
displaying images over text
, displaying the image itself is simple: the image just needs to be resized and aligned according to the recorded geometry. The overlays themselves are set up before starting the LaTeX run, and a series of overlay properties are set or unset over the course of the process.
If an existing image is being updated, the overlay optionally gets a “face” property indicating that the preview is out of date.
If this is a fresh preview, an overlay property marks it as “processing”, displayed in the fringe.
If any compilation errors are captured, they’re placed as tooltip text via an overlay property. Via the SVG’s
currentColor
attribute, we also color the overlay image red to indicate that there was an error.
These are basic but (hopefully) intuitive indicators of the state of the preview.
Live previews: Hooks galore
The breakdown provided so far does not explain the claims we made in the objectives section, like real-time (or at least fast) feedback as you type. As seen in the demos above, there is one more element to making the previews appear “live” and reactive to cursor movement or buffer edits.
For the seamless hide/show behavior:
Moving the cursor into the region covered by the overlay should “open” the overlay, reveal the underlying text and move the image next to or below the LaTeX fragment.
Moving the cursor out of the region should do the reverse, and “close up” the overlay.
For real-time feedback:
Editing a fragment with a visible preview marks the preview out of date, so the image should either disappear or be updated.
If editing displaymath and the numbering changes, all other numbered environments in the buffer should be updated as well.
Editing a previewed fragment in another buffer (with
org-edit-special
,
C-c '
) should cause the preview to be updated as you type.
When typing or pasting text, LaTeX environments need to be detected and parsed, and the preview pipeline started in real time.
All of these are achieved using hooks, which is ultimately the only mechanism we have to trigger reactive behavior in Emacs. The hide/show behavior involves tracking the cursor position via
pre-command-hook
and
post-command-hook
(made buffer-local). Buffer edits are tracked via
after-change-functions
, but with several fast-fail heuristics to avoid slowing down Emacs
The difficulty of identifying, in real time,
$
and
$$
as LaTeX delimiter starts, delimiter ends, or as actual dollar signs is why live previews require the unambiguous
\(...\)
and
\[...\]
delimiters, even though the dollars are supported in non-live usage.
. The list goes on, since we need to track Org’s folding states, as well as changes in other buffers when using
org-edit-special
.
The other strategy we’re forced to employ, given the event loop’s limitations, is to throttle and debounce pipeline calls, batching edits and avoiding race conditions between in-flight LaTeX runs – the interpreter lock protects Elisp state, but independent subprocesses will happily interleave and clobber each other’s output.
Finally, this feature is the source of dozens of edge cases, such as editing fragments inside folded regions (during global text replacement, say), or the many invalid intermediate states of a LaTeX fragment causing the preview to fail repeatedly. Knocking down these bugs while keeping this path fast was a substantial portion of the effort over the years this library was developed
Now that it’s part of Org mode and seeing more widespread use, there are no doubt several more edge cases waiting to be reported!
.
LaTeX previews everywhere in Emacs
A proof of concept: the design of
org-latex-preview
makes it possible to preview LaTeX outside of Org mode. Here’s an example previewing LaTeX-formatted comments in a prog buffer (Clojure, here):
In fact, you can use the same code to preview LaTeX fragments literally anywhere in Emacs, including, delightfully, the Emacs calculator. This (18 minute) demo runs through a bunch of contexts outside of Org mode showcasing live LaTeX previews:
While this feature exists only as a prototype, it’s simple to implement and extend. To see why, let’s look again at the
preview process breakdown
. All tasks here are agnostic of Org mode save for two:
Collect LaTeX fragments
: currently handled by
org-element
,
Create preamble
: currently assembled by the
org-export
(
ox-latex
) library.
To use LaTeX previews outside of Org mode, we only need providers for these two tasks:
A parser that can identify the boundaries of LaTeX fragments in the buffer
Technically you could reuse
org-element
in non-Org buffers for this, but there will be many false negatives and occasional false positives.
. This does not need to be a full-fledged parser – just a regexp scan of the buffer looking for
\begin{
or
\[
can suffice. This is how the above demos were created.
A LaTeX preamble generator appropriate for the buffer. This can just be a constant string (instead of a function) that includes a reasonable set of LaTeX packages. I used Org’s default LaTeX preamble string everywhere in the above demos.
The Emacs-wide API for
org-latex-preview
This description might be out of date, please refer to the Org manual.
org-latex-preview
tries to separate these independent concerns by providing the function
org-latex-preview-place
, that generates previews anywhere in Emacs given the above information. Previewing LaTeX fragments is as simple as:
where
entries
encodes the start and end positions of the fragments, and
preamble
is the LaTeX preamble to use.
;; entries is a list of lists;; Each entry is (start end),;; or (start end contents)((13421388)
(19112104)
(22732295"\\( \\epsilon + 3 \\)"))
;; preamble is a preamble string;; See `org-latex-preview-preamble'"\\documentclass{article}
[DEFAULT-PACKAGES]
[PACKAGES]
\\usepackage{xcolor}"
The positions in
entries
would be obtained from your parser for the major mode. For
preamble
you can just reuse or tweak
org-latex-preview-preamble
in a pinch. See the function documentation for more details.
Requiring the
org-latex-preview
library will in turn load a few Org libraries (
org-element
,
org-persist
,
ox
and
org-macs
), but it does
not
load all of Org. We tried hard to keep from loading the 16,000-line behemoth that is
org.el
.
As of late 2026, abstracting the library from its major-mode adapters remains on the TODO list.
How a modern LaTeX preview system
should
work
When we started
org-latex-preview
in 2021, live-updating previews were not available anywhere except one LaTeX formula-editing website and, to a lesser degree,
LyX
. We often wondered why, since
preview.sty
did most of the work for you and the editor side was pretty simple to implement. If
org-latex-preview
were a thrown-together external Emacs package providing many of the same features (sans polish), it could have been written over a couple of weekends. I know this because
I wrote it
.
Twice
.
Over the years, however, this feature has grown in ubiquity and there are plugins for most note-taking (Obsidian) and editing software (Neovim, VS Code) that provide this. Emacs itself has at least four active implementations of live LaTeX previews now, and a frontier LLM can probably implement a basic version for your preferred platform with one prompt.
Even so, Emacs remains the oldest platform to implement this feature, and does so in the most complete and mature form
As of 2026.
. Considering I spent a lot of this write-up complaining about the limitations of the Elisp runtime, this is remarkable.
But creating temporary files after every buffer change and spawning separate LaTeX processes remains a terrible way to do real-time updates. Precompilation is another hack, unsupported by LaTeX engines besides pdflatex. While
org-latex-preview
is a massive improvement over the previous LaTeX preview implementation for Org mode, it’s still the Elisp equivalent of a Bash script that got out of hand. Quite simply, this is the wrong level at which to address the problem. A better way to implement continuous previews would be to talk to a managed process over a pipe or socket. The right approach here is to work at the level of the TeX engine, as
TeXPresso
does.
But it would be even better to avoid inter-process communication entirely and use an
Emacs dynamic module
, linking with a modern LaTeX library that does incremental (instead of batch) compilation. “Lightweight” alternatives like KaTeX or MathJaX don’t cut it, since we need a full-fat LaTeX compiler, macros and all. One possible candidate for this is
realtime-tex
, the
rtex
library, which embeds LuaLaTeX (the most modern of the LaTeX engines) and compiles incrementally and effectively instantaneously
Although I would be inclined to wait for the project to mature.
. This would be a quality improvement almost as drastic as the one
org-latex-preview
makes over its predecessor.
Of course, a dynamic module linking an incremental compiler is a completely different architecture from what we have now, and would share at most the image display and UI layer. So projects like this and the inevitable official Typst preview package for Org are tabled for another day. Maybe you’ll be the one to implement them?
AI delivers 'zero financial payoff' to most companies, Apollo economist says
AI hasn’t benefited the bottom line of any big corporations outside the tech industry, according to
a note
recently published by the chief economist at Apollo Global Management (the private equity giant that used to own TechCrunch).
Apollo’s Torsten Slok focuses on what he calls the S&P 493 — the S&P 500, minus Alphabet, Amazon, Apple, Meta, Microsoft, Nvidia, and Tesla — finding that profit margins and revenue per worker appear to be flat, with no evidence that AI is improving productivity.
“Despite massive investments and widespread hype around artificial intelligence, the technology has delivered zero financial payoff to corporate America outside of tech,” Slok said.
Is this simply a matter of non-tech companies lagging behind the tech giants? Slok sounded skeptical, arguing that it will be tough for corporate America to “turn increasingly cheap intelligence into higher revenue growth and higher profit margins.”
“AI may be a great calculator, but in business, intelligence is mostly judgment,” he wrote, later adding, “AI does not understand accountability, trust, culture, unwritten rules, ethical trade-offs or fairness, all of which shape reputational risk and ultimately drive revenue growth and profit margins.”
Apple captures 50% of consumer revenue of the top half of the S&P 500. Market offers no other exposure to consumers.
When asked what they do, Apple says they create great products. Products that are designed to be useful and easy to use. “Building incredible products that make their lives better” is how John Ternus, the CEO put it recently. The “user” is usually expected to also be the “buyer”. It is therefore almost tautological to say that Apple is a consumer company.
Yes, there are businesses and other organizations buying Apple products, but they are not the
target
buyers. There is no “B2B” division. There are no “Enterprise” products. The “Pro” products are useful to professionals as well as consumers and they work for organizations but the Pro products are not designed to satisfy a purchasing manager. They are designed to be requested by if not purchased by the end user.
When looking at other large companies today, I noted that most of them are focused on selling to institutions rather than consumers. I seem to remember that there were more consumer companies among the largest market caps than what I see present today.
To test my memory, I retrieved a list the top 20 companies by market capitalization in 2010.
I assigned a sector to each and highlighted those which were not Tech companies
1
. Within the list, beside Apple, there are notable consumer products companies such as Nestlé, P&G, J&J, and Coca-Cola. China Mobile served consumers directly and Walmart and Amazon faced consumers as mass retailers.
Therefore, in 2010, 8 out of 20 of the largest companies were consumer-focused and of the others, there were significant portions of the businesses which were consumer-focused. For instance, Energy and Banking stalwarts such as Exxon and JPMorgan Chase had retail operations. Overall perhaps half the index was involved in some way with consumers. Only IBM, Oracle and SLB (Schlumberger, an oilfield services company) were strictly B2B companies.
Now let’s look at the top 20 of 2026.
I again assigned a sector to each and highlighted those which were not Tech companies
2
. Within the list, beside Apple, there are no consumer products companies. Walmart and Amazon remain in the index and face consumers as mass retailers, however Amazon, as we shall see, relies mostly on the AWS cloud service business for profitability and growth. The number of tech companies grew from 6 to 14 out of 20 and the combined share of capitalization went from about a third to 83%.
To sharpen the focus, I added a column of trailing revenues for each company and an assessment of the percent of sales that can be attributed to consumer purchasing
3
. Consumer sales total $850 billion or 15% of total index revenues. We can therefore conclude that of the top 20 largest market caps (which make up 25% of the entire
global
market capitalization
4
) 85% is derived from sales to businesses or government.
A large number of companies offer 0% consumer exposure: Nvidia, TSMC, Saudi Aramco, Broadcom, Meta (revenues), Eli Lilly, Berkshire Hathaway, Micron, AMD, SK hynix.
For companies which have a mix of business and consumer (B2B and B2C), the percent of business sales has been growing rapidly in the last three years.
The following graph shows Samsung, where the DX division is consumer products.
Samsung’s consumer share of sales has shrunk from 69% to 21%.
For Amazon, the following is how it reports its sales, with AWS shown as the non-consumer side.
For the third quarter, the share of AWS increased from 16% to 23% over three years. AWS operates at an operating margin of roughly 35% while North America Retail operates at a much lower operating margin of roughly 4.5% to 6.9%. International Retail operates at an operating margin of around 2.9%. Traditional retail and product sales run on thin single-digit operating margins (or near-break-even when separated from higher-margin embedded services like advertising). Therefore AWS or cloud is truly the core of profit and growth at Amazon.
For Microsoft, the picture of B2C vs. B2B is similar:
The B2C oriented “More personal computing” which targets end-users, consumers, and gamers has shrunk to about 15% from 25% in three years.
At Google the picture is similar. Though subscriptions, platforms and devices are down slightly from 13% a year ago to 11% today, Google Cloud has more than doubled as percent of revenues. The real erosion is happening in Search where growth has stalled and anecdotal evidence exists that advertisers are being given worse terms thus decreasing end user search quality.
Clearly, if there is any value in Google’s shares today it’s in whatever Cloud is offering.
To illustrate this even more sharply, at a company-level, the following graph shows the consumer sales vs. market capitalization of the top 20 companies.
Apple is the one outlier.
Apple is over 50% of the total consumer revenue of the top half of the S&P 500.
Put another way, all other S&P 500 consumer sales
put together
create less value than Apple alone.
This anomaly suggests a curious and possibly unprecedented situation. From an investor’s point of view, Apple is essentially the only consumer investment worth considering.
To wit, if you want to gain exposure to consumers, own Apple. If you don’t believe consumers are a source of value then exclude Apple from your portfolio.
There are some significant implications.
Apple is the last and only consumer play, moreover…
Subscribe to
Asymco One
to read the rest of this and other in-depth analysis and commentary.
The total market capitalization of the top 20 in 2010 was almost exactly $2 trillion. The Tech total was $631 billion
↩︎
The total market capitalization of the top 20 in 2016 is almost $40 trillion. The Tech total is $33.1 trillion, an increase of 5145% over 16 years.
↩︎
Retailer percentages are margin only as sales otherwise are pass-through.
↩︎
The top 20 companies in the S&P 500 represent approximately 42.4% of the index’s total market capitalization
↩︎
in recent times i've been asked a lot the question of "what ai model did you find to use this?", practically for all of the articles on this blog. the answer has always been my brain, i have never used ai for any kind of research i published, ever.
the next question is often why, followed up with "you would be so unstoppable with claude mythos" this article should explain why i don't like using ai for my research, and my general opinions on llms in both development and cybersecurity
why i don't use ai
it's as simple as i do this for fun, and i don't find giving an ai all the fun at poking at something instead of doing it myself fun. put it another way, i like using my brain
i like coding, i like staring at devtools, binary ninja, or whatever else for hours, i like exploring weird library source code and tracing it with nothing but github code search and a dream, it is fun to me. i don't do anything on this blog for money, i do it for fun, the money part is just a nice side effect
but why don't you just offload the tedious bits to ai?
a lot of the tedious bits are still fun, there's some that aren't, but i also cannot take credit for something an llm did for me, and i am wildly uncomfortable with publishing something an llm did for me on my personal blog as a writeup
this blog is mine, all the work on it is mine, or is shared research with other people that are explicitly credited. i cannot take credit for something an llm did for me, and i also cannot credit an llm properly, as they aren't a person
using an llm makes me lose my mind
i truly have tried to use llms before, just to try it out. nothing i was going to publish as research, but i really couldn't and it left me never wanting to use an llm again
i don't know how more people don't experience this, but staring at an llms output be confidently wrong eventually to get there in the process, or looking at the commands its running and see some really stupid things its doing drives me crazy to the point i will just stop it before it answers, i cannot handle something being that confidently wrong (even though it'll probably still get there in the end)
i'm not denying these models are powerful
llms are quite powerful machines, we've seen that as of recent, that's for sure, they are still wildly marketed as being more powerful then they are, but i cannot argue that they are infact not powerful, because they are
llms still need guidance
if you threw an llm at the linux kernel and told it to find an rce without any other guidance, it probably couldn't. you still need skill and experience to guide an llm at the current state
all of these vulnerabilities that we've been panicking over have been found by skilled researchers using llms, likely at a big firm to give them all the tokens they want. because llms still need guidance and not anyone can just do this, no matter what openai or anthropic try to tell you
marketing and php2shell
ai companies have hooked onto cybersecurity as their marketing strategy and it shows, a good example is php2shell. where three companies:
calif (research firm)
openai (you know who openai is)
wiz (research firm)
all collaborated on it and released their own blogposts, all seemingly with different narratives
calif went with the narrative of an ai found these php vulns with the help of a human guiding it, while openai went with the agent uncovered critical vulnerabilties, and wiz went with a middle point, still pretty openai-like
"patch the world"
openai talked a lot about how they were helping maintainers patch these issues, and mitigate them faster and for more people. which i'm sure they did to an extent, but i don't think they did all they could
they could easily backport these security fixes to older versions of php with the help of ai if they wanted to, it's not like they don't have the compute, they do, they have a lot of it, they're openai
dropping poc videos before patch
this has been a common trend in the recent days of ai-based cybersecurity where researchers and otherwise companies, drop poc videos of unpatched 0days on twitter, for some reason
this is so clearly marketing, and it does nothing but make everything worse in terms of security. these poc videos contain quite a lot of useful information, no matter how much you hide (which, a lot of them don't hide much) and just do nothing but fearmonger
hashes are also apart of this, hashes have always been a way to claim proof over a vulnerability or generally acomplishment before it becomes public, not as a way of fearmongering.
you shouldn't include the name of the product next to a hash, that is just not what a hash exists for.
but a lot of companies and generally people now drop them like candy, and it causes panic for no reason. at least with a poc video you know what the vuln is, with just a hash and a product name you don't even know that, you just know some vuln exists
tailworm
this was a less big story, but still pretty big. on bluesky, a "security researcher" claimed to have found a 0day in tailscale. they did this by dropping a hash with the caption of "tailscale privilege escalation"
obviously, this got people very very worried. what could tailscale privilege escalation mean? could someone bypass all ACL rules, what can happen?
don't worry! cause we slowly get information of the vulnerability fed to us over a bluesky thread
so, what was the point of that? you posted barely any information, posted more but cryptic information an hour after, and then an hour after you just gave away where the vulnerability is?
i assumed that there was some responsible disclosure happening here, and that's why the hash got tweeted, etc.
would you believe me that this wasn't even responsibly disclosed?
next day, the (single-person) company this person owns drops this website on bluesky:
a full website, obviously vibe-coded, with all of their findings, a lot of which completely do not matter or just aren't critical. while the "privilege escalation"
was
real, it required you to already have acl rules to a machine, and have it's tailscale ssh enabled
the vulnerability was just the fact that you could privilege escalate on that machine specifically to root. that's it. that's what was fearmongered about so much, and then they dropped it publicly next day, without even contacting tailscale to let them patch it
it was given a fancy name, a catchphrase, even. but no one thought to responsibly disclose it, instead posting it on a social media platform
the reason this happened was pretty clear: it was to advertise the company, not to help make the world better. it was to make everyone think about the company who exposed critical issues in tailscale, and maybe ask them for an audit
it's not all ai finding these vulnerabilities
the biggest of them all, in my opinion, was eMudhra. you know, the company who had their CA private key as the number Four. this wasn't found by an ai, but just a human using a old ass library made by google called
paranoid_crypto
to scan through keys for fun out of curiosity
that's whats beautiful about it in my opinion, the biggest thing of them all was missed by all these ais but rather was just by a human just doing random shit
conclusion
i think that people have been way too lost in this wave of ai everything, that they are disregarding doing things themselves by a whole lot. it is still extremely fun to just, write code yourself, debug things yourself, stare at minified js bundles for hours at a end, cycle through random interesting looking subdomains, it's all fun! and i feel people have forgotten that a bit
while an ai "can just do things", it is fun to do them yourself. i totally get using ai for work, as we live in a world where software engineers are expected to move faster then ever, and in my opinion, that isn't a good precedent, but it's what we have to live with
personal projects are the place where you get to have fun, do something you enjoy. and i think giving those to an ai even just ruins your enjoyment of creating things
for me, at least, the process of getting somewhere is more interesting then the end product. cause i enjoy learning along the way and i enjoy banging my head against the wall on a primitive that goes nowhere
i mess with a lot of things, and i find a lot of things. but almost none of them go on this blog, simply because i have a feeling i need to one-up myself with every new blogpost, and i keep challenging myself to find something even funnier and also more critical at the same time. it makes me learn a
lot
more then i already knew
writing code itself is fun, and an ai writing it for you even if you gave it a plan on how to write it, you are missing out on
so
many learning opportunities by not writing it yourself, because there's no such thing as a perfect infrastructure plan, things change as you go, while you are in the flow of writing code
go reverse engineer that old iot device, mitm that weird looking chinese automatic pet feeder that requires an internet connection with the number four, make a new frontend for a website you use a lot that has a really bad frontend. you learn a lot by tinkering
The actor
John Leeson,
the voice of K-9, has died at the age of 83.
John Leeson played the voice of the Doctor's robot dog in 54 episodes of Doctor Who. His characterisation of the little robot dog was so skilled and so full of fun that he helped make what was due to be a temporary character into a fully fledged member of the TARDIS crew, much loved and remembered by fans. It is a tribute to the actor that his cry of
Affirmative
is as much associated with Doctor Who as the Daleks' cry of
Exterminate
. He endowed the character with a life force that saw K-9 return to the series and its spinoffs many times over the years.
John Leeson was born
John Ducker
in 1943. In the 1950s, he joined the Leicester Dramatic Society before winning a place at the Royal Academy of Dramatic Art in London. At RADA, he adopted the stage name John Leeson, derived from the surname of a wealthy godmother whom he believed was funding his tuition.
After graduating, he worked in repertory and pantomimes, including a production of
Toad of Toad Hall
, in which he met his future wife.
His first work in television was as a walk-on in a BBC play
The Wedding Feast
in 1965, followed by
The Spanish Farm
and
Dad's Army
in 1968. He played the original Bungle the Bear in the children's series
Rainbow
.
In 1977, he was asked to play two voices in the fourth Doctor story,
The Invisible Enemy
. One was the voice of the Nucleus, the other a fun character, Professor Marius's robot dog K-9. Because the electronic prop was unavailable for the rehearsal period, Leeson got down on his hands and knees and physically ran around the rehearsal room acting as K9.
His voice for the character was so successful that the producer
Graham Williams
decided to make the character a regular and put Leeson on contract to provide the voice.
Leeson would stay with the series for the next two years, appearing on screen as the character Dugeen in the story
The Power of Kroll,
a story where K-9 does not appear. He left at the end of
The Armageddon Factor
, fearing that he would lose work if he remained firmly off-screen.
The character of K-9 remained in the series, voiced by
David Brierley
for season 17, but for season 18, Leeson was back behind the microphone, guiding the character until it was written out at the end of the 1981 story
Warriors' Gate.
K-9 and Leeson would be back at the end of the year in the spin-off starring
Elisabeth Sladen
,
K-9 and Compan
y and in the 20th Anniversary special
The Five Doctors
. in 2006, when Russell T. Davies was writing an episode,
School Reunion,
dealing with the return of Sarah Jane Smith, he also brought back John Leeson as K-9. Over the next 15 years, Leeson would reprise his portrayal of K-9 on many occasions, including in
The Sarah Jane Adventures
, the
K-9
series and many Big Finish audio adventures.
Away from Doctor Who, John Leeson made many memorable appearances on British television, including roles in
Blake's 7, Jigsaw, Crown Court, The Barretts of Wimpole Street, Minder, Vanity Fair
and
Doctors
.
John Leeson died peacefully earlier today after a short illness. He is survived by his wife
Judy Griffiths
and their son, filmaker
Guy Ducker
An easy way to try local AI with braindead-local-intelligence
I wanted a simple way to try local AI on my computer when I need to experiment, both a LLM to chat with (with web search and image recognition) and a model to generate or modify images. Most guides require installing a lot of things and choosing models among thousands of files, so I made a small project doing it for me.
The name is a bad pun, "braindead" because you just start it to try without much thinking about it, and "local intelligence" because it runs some "intelligence" on your own computer :)
In this blog post, I present the project and how to get started with it.
The project runs a few containers with podman: a LLM server compatible with the OpenAI API, an image generation server, SearXNG for the web search and Open WebUI as a web interface. Everything is behind a single address, protected by a token if you need to expose it to an external service.
A feature I really like is that you do not have to choose between the LLM and the image model, or to switch from one to the other. Only one model is loaded in memory at a time, the right model is loaded when a request arrives, and it is unloaded after a few minutes without any request, so your GPU memory is free again when you do not use it.
There is a list of ready to use models in the documentation, chosen for regular hardware people have at home and for different needs: small text models, models understanding images, image generation models, and two models able to modify an existing image. Changing the model is a matter of copying a block of configuration and restarting, the clients do not need to change anything.
On the software side, you only need a Linux system with podman 5.0 or newer using rootless mode (everything runs under your own user, no root involved). The project currently targets AMD graphics cards, it was never tested with NVIDIA cards because I do not have one.
On the hardware side, you still need a GPU with some memory (VRAM), the models were tested up to 12 GB of VRAM, and you also need RAM because the biggest models run partly in RAM. The list should allow you to try a few things without a powerful gaming computer, although the smallest models need something like 4 GB of VRAM and 8 GB of RAM. I use a computer I built in 2021 with 32 GB of RAM and a Radeon RX 6700 XT (released in January 2021), it still works very well.
The project has its own documentaiton, but here are the main steps.
Your user needs to access the GPU, and to be in the groups
render
and
video
, then log out and log in again:
sudo usermod -aG render,video $USER
From the project directory, install it and check that everything is fine, the second command should list your AMD graphics card:
bin/ia install
ia doctor
The copy/paste commands will download the models in
~/.local/share/ia-locale/models
, the default ones weigh about 8 GB. Then start the services, it can take several minutes to download the containers for all runtimes:
ia start
The command
ia info
displays the address, the token and the model names to use in a client compatible with the OpenAI API.
The simplest way to use it is the web interface Open WebUI at http://127.0.0.1:3000, only reachable from the computer itself. You can chat, send a picture, and enable the web search or the image generation from the
+
menu in the input box.
I got good results so far with Qwen3.6 35B-A3B, coupled with the web search it does a pretty good job. It is slow though, it "thinks" before answering, and as half of the model runs in RAM, an answer can take a few minutes, but it runs locally. Having a 35B model with web search allows it to be smart enough to gather information and "understand it", without having to know everything and serve outdated information.
I also use it with Camille, a Mattermost bot used within a community of friends, which replies when pinged. Everyone configures it as they want to use the model of their choice, it uses two different models depending on the request depending if it is for text or image. Mistral have a free API that allows to use ministral on it, it is a bit too dumb though, but nice if you want to give it a try.
Do not expect miracles, this is slow and your computer will get hot and noisy while crunching for a reply. This will always be very far from the LLMs as a service that use much more advanced models and a context window (the amount of text a model can handle at once) way bigger than what you can run at home. Even spending a few dozen of thousand of euros or dollars would not approach it.
But it is a nice way to play with local AI and see what your hardware can do, and nothing leaves your computer except the web searches.
As stupid as small models might sound, they could still be used for some tasks, for instance if you just need it to understand a context and say
yes
or
no
, this can be well integrated into a CI or maybe filtering local SPAM. If you can run the 35B model, it opens to a lot more of cool stuff.
Work on GEFS (the Good Enough File System) for OpenBSD progresses
As
previously
reported, Ori Bernstein is working on GEFS, the "Good Enough File System" imported to
OpenBSD
, and his
EuroBSDCon 2026
presentation
was very well received. In fact, the video is among the most frequently played from the conference.
Now Ori has followed up with more evolved code that includes several important steps towards makin the new file system fit for general use. In a
message
to
tech@
, Ori writes:
List: openbsd-tech
Subject: Re: GEFS on OpenBSD: A very early preview
From: ori () eigenstate ! org
Date: 2026-10-31 17:08:42
So, it's probably time for a few updates. First
off, thanks to Martijn, and everyone else who
tried it out and sent patches.
Everything is updated and rebased, and the diff
has been regenerated with all of the lastest.
I've gone through a the error handling code. In
theory the filesystem will handle errors mostly
correctly. There are probably still places where
theory diverges from reality. I've also forward
ported most of the important bugfixes from 9front.
The file system now leaves the first FS block
unused by default -- we can put disklabels there,
which makes it hard to put the superblock there as
well.
Some userspace tools have been done (you can now
mkfs on openbsd!), but snapshot control hasn't.
I'm thinking we have a userspace 'gefs' program
that lets you handle this, trying to keep the
interface similar to the 9front ctl file. The
commands below should suffice:
gefs debug [0|1]
gefs usereserve
gefs retain snap timespec
gefs snap name newname
gefs fork name newname
The consistency protocol is still the biggest
missing piece; I've been reading the old code
for softupdates, since it had the same flavor
of problem.
what gefs wants, in the ideal case, is some way
to put ordering barriers in the commit phases,
while still accepting new block i/o, something
like the following sequence of steps:
0) normal use, generate dirty blocks and
write back in any order.
holding mutlk {
1) barrier(), ensure that all blocks
written before the commit will be
on disk before the commit finishes
2) enqueue free list/deadlist data
3) prepare arena header/footer + superblock
}
4) enqueue arena headers
5) barrier()
6) enqueue superblock
7) enqueue arena footers
8) wait for enqueued blocks to hit disk
9) free blocks no longer used by previous gen
on 9front, I have a priority queue of blocks that
are sorted by (gen, addr) tuples, which means that
writes within a generation are flushed to disk as
sequentially as they can be, while not crossing
generation boundaries.
I'm currently thinking I'll implement the same in
OpenBSD -- launch syncer kprocs that do synchronous
I/O to the block device. If we kick things out of
cache and write it back to disk early, that should
generally be safe -- The only blocks the buffer
cache must not reorder are the arena headers and
superblocks, so a flag to pin them unwritten may
be all that's needed here.
I'm open to other ideas, though.
Quoth ori@eigenstate.org:
Followed by a quoted part of the
previous message
that has the links to the code and documentation.
If you are as excited as we are about this new file system development and are in a position to test, please do.
FBI links $6.3 billion Binance crypto heist to North Korean hackers
Bleeping Computer
www.bleepingcomputer.com
2026-11-01 14:57:53
The FBI has confirmed that North Korean state-sponsored hackers known as TraderTraitor were behind the $6.3 billion crypto-theft attack on Binance, making it the largest cryptocurrency heist on record. [...]...
The FBI has confirmed that North Korean state-sponsored hackers known as TraderTraitor were behind the $6.3 billion crypto-theft attack on Binance, making it the largest cryptocurrency heist on record.
In a security alert published Saturday, the agency attributed the attack to TraderTraitor, a North Korean hacking group also tracked as Lazarus Group, Jade Sleet, and APT38, linked to numerous cryptocurrency thefts.
"The Federal Bureau of Investigation (FBI) is releasing information advising the Democratic People's Republic of Korea (North Korea) was responsible for the theft of more than $6 billion USD in virtual assets from cryptocurrency exchange Binance, on or about October 29, 2026,"
the FBI said
.
This official attribution follows blockchain intelligence firms TRM Labs and Chainalysis, which had already linked the theft to TraderTraitor.
As
previously reported
, Binance linked the breach to a compromised DevOps engineer's laptop, which allowed the attackers to access a service container and submit unauthorized withdrawal transactions.
The attackers stole approximately $6.36 billion in cryptocurrency across ten blockchains, with most of the funds transferred in just 20 minutes.
The FBI says it has observed TraderTraitor converting stolen assets primarily into Ethereum (ETH), which it expects will be further laundered through decentralized finance (DeFi) platforms and instant cryptocurrency swap services.
The FBI, U.S. Secret Service, and IRS Criminal Investigation are now asking cryptocurrency exchanges, blockchain operators, and other virtual asset service providers to block transactions linked to addresses the hackers used to launder stolen funds.
The agency also published a list of 628 cryptocurrency addresses associated with the theft, including 608 Ethereum addresses, 19 Bitcoin addresses, and one Solana address.
One tactic used by North Korean Lazarus hackers is a long-running campaign known as '
Operation DreamJob
,' in which they impersonate recruiters and approach software developers and cryptocurrency employees with fake job offers to trick them into installing malware.
Both the DMM and Bybit attacks involved compromising computers belonging to employees or developers with access to cryptocurrency infrastructure.
In the DMM Bitcoin attack, hackers impersonated a recruiter on LinkedIn and tricked an employee at the cryptocurrency wallet company Ginco into running malicious Python code disguised as a pre-employment test.
The Bybit heist also stemmed from a compromised Safe{Wallet} developer's computer, which let hackers access the wallet platform and manipulate a legitimate transaction request by a DMM employee, redirecting $1.5 billion in cryptocurrency to wallets they controlled.
More recently, the April 2026 Drift attack involved a six-month social engineering operation in which individuals posing as representatives of a cryptocurrency trading firm met Drift contributors in person at multiple industry conferences.
The attackers spent months building relationships, and are believed to have compromised employees' devices through a malicious code repository or a TestFlight application.
In a statement shared with BleepingComputer today, Binance said it was preparing to resume the first phase of withdrawals on November 1 between 06:00 and 09:00 UTC, as it continues working to restore services and cover the stolen assets.
The exchange says it has now covered approximately 95% of the impacted amount, up from 90% in its previous update. The remaining 5% consists of less-liquid tokens that Binance is continuing to purchase through market and over-the-counter channels.
Binance also said that BTC, ETH, USDT, BNB, SOL, USDC, XRP, USD1, RLUSD, and ZEC balances are fully backed, and that deposit services remain unaffected by the incident.
"Our commitment remains unchanged: Binance will bear the financial impact of the stolen funds. These losses will not be passed on to users," Binance told BleepingComputer.
"Resuming withdrawals is a major milestone, but our work continues. We will continue strengthening our safeguards, completing the recovery process, and providing updates as progress continues."
I haven’t owned a TV in years and would like one that respects my privacy. Are there any TV’s which do this well? I am looking for something in the 60”to 80”range and would like a 4k display with at least 60hz display. I don’t mind paying a premium.
I have heard good things about Apple TV as a better alternative to most smart TV native apps. Would some TV’s pair better with this than others for privacy? I don’t want the TV doing anything other than being the display for Apple TV.
I’ve been paying for Proton for many years, and it’s what I recommend when someone asks how to get off Gmail. This week 404 Media published an investigation into how Proton’s leadership is pushing AI inside the company. The detail that made the rounds came from the security Slack channel: an engineer asked for a few days to pentest new code before shipping it, and CEO
Andy Yen answered
that it was
“bullshit and loser thinking”
, then told the team to stop
lovingly handcrafting code
and use AI more.
For the first time, I’m not sure I’d still recommend them.
A lot of the reactions treat this as an AI story, as if any AI-written code poisons the well, and I think that misses the point. Engineers using AI to write code is the new norm. It’s fine as long as there are guardrails around it: someone reviews what the model produced, it gets tested, and someone is
accountable for what ships
.
What the reporting describes is a CEO telling an engineer, in front of colleagues, that raising a security concern was
“loser thinking”
. Staff were told their AI usage
could decide whether they kept their jobs
, and when that plan was dropped after it leaked, Yen said it had
already served its purpose
. None of this is about AI. Plenty of companies are rolling out coding agents right now without humiliating anyone, so if Proton can’t, the problem sits with its leadership.
The tradeoff Proton can’t make
For most companies, trading some caution for speed is a reasonable call. Proton is a privacy company, though, and what its users pay for is the confidence that the people building it are careful. Shipping slowly and carefully is part of what they’re selling. A bad release from a B2B vendor can be
catastrophic
, but for a company like Proton, a bug that exposes users’ mail would be fatal.
One of the employees defending the company on Reddit
added some context
that was meant to help: Yen was fine with pentesting, as long as it didn’t slow down the pipeline.
This is one of a few employees pretending to courageously dissent, but in reality agreeing with the official corporate line. It reads like astroturfing and a
clumsy attempt
at crisis management. No employee who shared on Reddit disagrees with how things were handled or with where the company is going.
Beyond the clumsiness, it shows a leadership that now treats the constraints that made Proton worth paying for as the thing slowing it down.
I’d like to keep recommending Proton, but that depends on the engineer asking for a few more days of testing still winning the argument against a leadership obsessed with velocity and arbitrary release dates.
Nurses Sound Alarm as 'Uber for Nursing' Apps Push to Deregulate Healthcare
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
Generated by cloudfront (CloudFront)
Request ID: kQUCzK27nkYOOSEyXQykUJaBRapvmmwIC2lnSmNSHiutUclUKB97-g==
Investors pull billions in funds from Binance after record $6.3bn hack
Guardian
www.theguardian.com
2026-11-01 13:53:25
Cryptocurrency exchange has begun to allow withdrawals after theft believed to have come from North Korea Investors have pulled $3.6bn (£2.7bn) in funds from the cryptocurrency exchange Binance as it recovers from a record hack in which billions were stolen from customer accounts. The platform was h...
Investors have pulled $3.6bn (£2.7bn) in funds from the cryptocurrency exchange
Binance
as it recovers from a record hack in which billions were stolen from customer accounts.
The platform was hit by a $6.3bn theft on Thursday, which has since been described as the biggest heist in history. Experts identified
North Korea as the probable culprit
.
Binance initially froze withdrawals as it investigated the breach but on Sunday announced it had
begun a phased reopening
.
By Sunday afternoon, the platform had processed $4.6bn in withdrawals and $1bn in deposits, resulting in a net outflow of about $3.6bn – representing 2.1% of its $171bn in total assets under management.
The numbers were confirmed by the company in its latest proof of reserves snapshot. A full report is expected on Monday.
Binance said: “After an initial spike, withdrawal volumes have declined and are stabilising as expected, while deposit volumes have remained substantial.”
The crypto exchange told its customers that market conditions were stable and that its team was “testing rigorously the remaining networks and assets”.
“We know how important safe and reliable access to your funds is and will share details of the next phase as soon as we can,” it said on X.
In the first phase Binance restored withdrawals for 15 networks and assets, including ethereum, USDT and USDC tokens on the ethereum blockchain, as well as some others held on the BNB Smart Chain blockchain. Later on Sunday, dealing in the Dogecoin and Avalanche currencies resumed on their own platforms.
The platform said it had replenished 90% of the amount stolen. The remaining 10% of tokens were less liquid, meaning they may take longer to buy, the company said in a post on X on Saturday.
Richard Teng, Binance’s co-chief executive, said separately that the company was “committed to covering the remaining 10%” and users would not bear the financial impact of the hack.
On Saturday the FBI confirmed earlier suspicions from private investigators and analysts that a North Korea-affiliated group called TraderTraitor was behind the hack.
TraderTraitor is believed to be part of the Lazarus Group, which gained notoriety in 2014 when it was accused of
hacking into Sony Pictures
as revenge for The Interview, a film that mocked the North Korean leader Kim Jong-un.
“The Federal Bureau of Investigation is releasing information advising the Democratic People’s Republic of Korea (North Korea) was responsible for the theft of more than $6bn in virtual assets from cryptocurrency exchange Binance, on or about October 29,” the law enforcement agency said.
The FBI added that it had observed “TraderTraitor actors” converting stolen assets into ethereum, and expected those assets to be laundered further by converting them using decentralised finance and instant swap services.
It said it was urging private sector entities “to block transactions with or derived from addresses TraderTraitor actors” were “using to launder these stolen assets”.
Teng said on Friday that the breach was a “complex attack” involving a compromised laptop and malicious software designed to delete evidence of its activity, making it harder to trace and investigate.
Binance said it had set aside about $6.34bn of company funds to cover the losses. “Our commitment remains the same: Binance will fully cover the financial impact of the funds stolen in this incident,” the company said.
The money spent rebuying tokens came from using part of its $1bn secure asset fund for users, a bitcoin reserve it holds to protect customers in the event of security breaches, as well as some of its futures insurance fund and own corporate treasury, Binance said on X.
The hack came as many observers expressed worries that the ever-expanding capabilities of artificial intelligence models could compromise cryptography and IT security protocols.
Bitcoin plunged nearly 5% after the news of the hack, while ethereum dropped nearly 6% and BNB, native to Binance, tumbled nearly 10%.
The sheer scale of the hack has put a
spotlight on the capacity of North Korea
– which faces extensive international sanctions over its nuclear and missile programmes – to find illicit means to capture revenue.
US Jobs Pickup May Keep Fed on Track to Hike Rates Again
We've detected unusual activity from your computer network
To continue, please click the box below to let us know you're not a robot.
Why did this happen?
Please make sure your browser supports JavaScript and cookies and that you are not
blocking them from loading.
For more information you can review our
Terms of Service
and
Cookie Policy
.
Need Help?
For inquiries related to this message please
contact
our support team
and provide the reference ID below.
Pub Address: The Fellow, 24 York Way, London, N1 9AA.
Date: Sunday, 15/11/2026
The idea is to just meet and have a chat really.
I have submitted a request to book a table. Hopefully they approve but if not we can just find an un-booked table.
I am posting the meetup prior to receiving their approval...
Pub Address: The Fellow, 24 York Way, London, N1 9AA.
Date: Sunday, 15/11/2026
The idea is to just meet and have a chat really.
I have submitted a request to book a table. Hopefully they approve but if not we can just find an un-booked table.
I am posting the meetup prior to receiving their approval, so that those of you who live outside London can (hopefully) bag a cheaper train ticket by booking 2 weeks in advance.
I've created a London Lobsters
Whatsapp group
too, where we can communicate on the day to share the table number/updates.
Please do join it.
National Grid to site biggest substation in UK near planned Teesside datacentre
Guardian
www.theguardian.com
2026-11-01 13:37:42
New electricity generating capacity would enable north-east to attract investment for large-scale AI infrastructure National Grid has taken a step towards building Britain’s biggest electricity substation on Teesside, which could supply power to a vast neighbouring datacentre as the UK races to expa...
National Grid has taken a step towards building Britain’s biggest electricity substation on Teesside, which could supply power to a vast neighbouring datacentre as the UK races to expand its AI infrastructure.
Last week, the company agreed to purchase 29.5 hectares (73 acres) of land at Dorman Point on a site owned by Teesworks, a local private-public partnership. The deal was first reported by Sky News.
The deal followed reports last year that the UK government had picked Teesworks as the site for a new artificial intelligence datacentre, and planning applications had been submitted for a project that would span nearly 500,000 sq metres.
The deal “reflects the constructive work between all those involved to enable this vital infrastructure … and will support the growth of AI and digital technologies, and enable the region’s next phase of industrial and economic development”, Matt Pearce, the head of land at National Grid, said.
“We are committed to working closely with our partners to progress the development as quickly as possible,” he added.
Ben Houchen, the mayor of the Tees Valley combined authority, told Sky News: “Everyone who has tried to build a datacentre in this country knows the biggest problem is getting enough power to the site quickly.
Ben Houchen: ‘If Britain is serious about having its own AI capability … it needs somewhere to physically put it.’
Photograph: Mark Pinder/The Guardian
“While we’ve been stuck in the queue, the US, the Gulf and parts of Europe have been getting on with it and hoovering up the investment. Having up to 3.2GW of capacity here means we can finally compete for those projects.
“If Britain is serious about having its own AI capability, with sensitive data stored safely on British soil, it needs somewhere to physically put it, and I’d much rather that was Teesside than Texas or Abu Dhabi.”
Last week, the government published a draft national policy statement on datacentres, describing them as “as fundamental to our economy as the railways and electricity networks on which we have long depended”.
“We must take control over the development of datacentres in this country and ensure that the country and host communities benefit from them,” it said.
However, the rapid construction of datacentres in the UK has been met by strong opposition, with campaign groups protesting against planned developments in places ranging from
Auchtertool
, a village north of Edinburgh, to
Brick Lane
in east London.
The London-listed infrastructure services group Balfour Beatty was reported to be in line to deliver the Teesworks substation for National Grid, with the utilities firm Steel River Power (SRP) also heavily involved in the project.
Balfour Beatty and SRP were approached for comment.
[This is a guest post by
Frank Calegari
. This blog post was initially written in a different file format and converted using AI. — T.]
Why we should not be sad
The release by OpenAI on October 6 of over 700 research papers was a watershed in the history of mathematics. AI is now undeniably capable of making groundbreaking contributions across many areas of mathematics. So why are some mathematicians so unhappy?
Some of it is the apparent disregard by AI companies for the practice of mathematics. The OpenAI papers — despite some tremendous ideas — are atrociously written and violate our community expectations of attribution. Some of them are even wrong. We have much more control over how we hire mathematicians, publish papers, and train students than over the behavior of trillion-dollar companies, and that is where our efforts would be better spent.
Much of the reaction seems to reflect a feeling that the subject that they love is under attack, and is being “hollowed out”. The recent OpenAI dump made barely a dent on the general problems of functoriality and reciprocity in the Langlands program. But even if it
had
solved every problem in the Langlands program, to imagine that would have “solved number theory” would be as naive as saying that the construction of the periodic table solved chemistry. Mathematicians should do a better job explaining that when we talk about the central problems in any area, we are merely discussing the most natural boundaries of our understanding — and that unlocking them is not the end but the first step to finding new playgrounds to explore. Of course, this takes time. I looked for areas of mathematics which were “harder hit”. It is hard not to be struck by Hugo Duminil-Copin’s
lament
:
Not a single one of the major open problems I have publicly mentioned throughout my career (whether in a talk, a lecture, an article, or even a grant proposal) was left untouched by the announcement. Everything has been claimed to be proved.
I expected to see a few of them in the list. But not all of them. Not all at once. Not with such nonchalance.
How would I respond to that?
The classification of finite simple groups.
I can’t say much about percolation theory, but there is another historical precedent I think mathematicians should take note of. In 1972, Dan Gorenstein announced a program towards the classification of finite simple groups. After a flurry of activity over around a decade, the “classification” was announced by Gorenstein in 1983 (somewhat prematurely, with the case of quasithin groups not being completed for another two decades by Aschbacher and Smith).
The classification of finite simple groups was in many ways a huge success, but it was also a failure in that it — apologies to my group theorist friends and collaborators — cooled a lot of the interest around the subject. The “famous problem” had gone, and mathematicians on the whole turned their attention elsewhere. And yet, there remain very rich questions in finite group theory and even about the classification itself (to name just one: John Conway’s favorite question is
why does the Monster group exist?
). Perhaps for some fields, the OpenAI dump feels like how a group theorist may have felt in 1970 if the classification was dumped in their office, not in the form of the actual papers, but printed out on a spirit duplicator, smudged a little, then spread randomly like confetti all over the floor.
Right now, we need efforts like that of Gorenstein, Lyons, and Solomon and their work on the revised classification. If this was all we had to do, then it would be a fairly thankless task. But we also have the responsibility not to pronounce the subject “done” but rather — with these new tools available — ripe for further investigation. When mathematicians lament the winding paths missed by the construction of a galactic bypass, they need to realize that those old paths still exist and seek them out. And we need to reward that behavior when it comes to offering people jobs.
The promise of AI, and the (dis)-unity of mathematics:
Since I was 15, I’ve wanted to know what
was. First, there was the childhood fantasy of finding a formula that Euler missed. But soon one learns more: how Grothendieck’s period conjecture, together with a sufficiently fine understanding of the geometry of mixed Tate motives (following work of Don Zagier, Sasha Goncharov, Francis Brown, and others) gives a completely satisfying structural, conjectural explanation of what’s going on. The irrationality of
, or even its transcendence, is a mere shadow of the actual interesting underlying problem. Apéry’s proof of the irrationality of
has inspired me for over
years not because of its conclusion, but for how it reveals the tiniest hints of some still secret story relating algebraic
-functions to arithmetic.
I hope AI can help mathematicians break out of the straitjacket of being constrained by the first natural conjecture they cannot prove, and work on unraveling the deeper mysteries woven throughout our subject. In the last thirty years, progress in mathematics by humans has been remarkable. But at the same time, I have felt a growing unease; the number of times I have seen mathematicians of all levels being unable to explain to someone even slightly outside their field why what they are doing is interesting has been distressing. The natural grooves of our profession have very much been bent toward more narrow technical expertise than a bigger vision. AI allows us an amazing opportunity to broaden ourselves as mathematicians.
I have tried (not always successfully) to be broad as a mathematician. I had some help by the fact that, while I knew since I was 17 that I wanted to work in algebraic number theory and the Langlands program, I grew up in the crucible of Melbourne University, which was a home to many Thurston students and low-dimensional topologists. This meant I was bathed in the sweet liqueur of hyperbolic 3-manifolds. That exposure has helped my career immeasurably. As with any two deep areas of mathematics, hyperbolic three-manifolds and arithmetic are much more closely connected than would first appear. But it’s still hard to be an expert on both, let alone pursue a crazy idea combining them. Let me get technical for a moment. For every imaginary quadratic field
, there are many commensurability classes of compact arithmetic hyperbolic three-manifolds indexed by non-split quaternion algebras
. Manifolds
in each of these commensurability classes give rise to elements in
via the decomposition of
into hyperbolic tetrahedra, and their volumes are always rational multiples of
. I have always had this idea that one could conceive of elements in these different commensurability classes as forming some type of geometric Kolyvagin system (“Heegner points”) associated to
, which one could yoke together by some unknown integral arithmetic version of the Jacquet–Langlands correspondence. This is not a question I have pursued, for two reasons. First, there were so many underlying conjectures involved that any result I could prove was going to be conditional on many of them, something that’s not always appreciated by the mathematical community. Second, assembling all the technical background to formulate such an idea precisely would have taken a huge amount of background work. Now, with AI, the barriers to trying such a problem are much lower. It’s not trying to prove a theorem. It’s trying to explore some completely new direction that may or may not be nonsense. There are so many problems in mathematics that are still so fresh and vital but were previously untouchable that the opportunity to approach them is massively exciting. We need to be more ambitious, to create bold research plans that weren’t even conceivable a year ago.
This is not to say that everyone must (or must not) use AI. But we have to rethink and readjust what’s important. Proofs of major conjectures, using AI or otherwise, are going to remain part of the new mathematics. But working toward broader programs (new and old) and structural understanding is going to play a more fundamental role. Christian Szegedy raised the
following point:
Mathematics prided itself on being meritocratic, with deep reverence for the explorers: the people who could climb what nobody else could. That specific kind of heroism will matter less, and other skills will matter more: choosing which terrain is worth exploring, steering fleets of airplanes, and explaining what they find to the rest of the world.
I’m ready to board the plane, because I want to see what’s there. At the same time, I would add that people who previously have been walking around on earth with their eyes closed are not going to recognize anything however high they fly; I still believe that many of the skills that mathematicians have now will not only be transferable but vital to the new world order.
The difference in reaction between the optimists and the pessimists is that, for the pessimists, the game is over. For the optimists, the game has just changed. I acknowledge that there are aspects of our mathematical culture that we treasure that won’t survive, and it is only human to grieve that loss. But there now exists an amazing new tool to study mathematics. To turn away and say that the community of mathematics as it currently exists is more important than mathematics itself would be a terrible mistake.
The ugly truth.
For all that everyone has been saying on this blog and elsewhere about how mathematics is about understanding, it takes one look at the arXiv to see many people in our profession rushing out AI-assisted papers in a somewhat unsightly rush to be the “first” to prove things. As many others have noted, there is quickly becoming a gulf between the ability to produce novel proofs and the ability to either demonstrate understanding or push the subject in new and interesting directions. As people crank out more and more results, the Leiden declaration is being honored more in name than in spirit, and it seems like “business as usual” except with a much more compressed time scale and the constant threat of being scooped hanging over people’s heads.
To me, someone who chooses a worthwhile problem, obtains a proof completely using AI, understands it thoroughly, and uses it to develop interesting further questions or directions has made a substantial mathematical contribution. I would give that person substantial credit even if they supplied no new ingredient in the original proof. Choosing worthwhile problems may end up mattering more than producing proofs.
Our obsession with conjectures tends to greatly undervalue the exploratory work that makes answering (or even asking) these problems possible. We are in danger of making that mistake a hundredfold now by encouraging or rewarding the rush for everyone to press the button to get these new results out as soon as possible. I would encourage all senior mathematicians (with the privilege of tenure) to take up the work of creating future problems and exploring new ideas rather than simply using AI to answer existing questions. If you, as a senior mathematician, are stressed about being scooped by AI, then you are not being ambitious enough.
So what are we actually going to do?
Think pieces are ultimately pointless unless they propose some mechanism for actually affecting how people act. The tools that are going to be the most effective are going to be economic rather than rhetorical. Mathematicians want recognition, mathematicians want jobs, mathematicians want grants. The tools that we currently have to affect this are hiring and accepting papers at journals.
Let me offer some very basic concrete suggestions, followed by a few details.
Evaluate all work as if it is AI-assisted.
Make hiring much more about personal interactions rather than a paper file. Interview postdocs and graduate students in person.
Demand more from journals, and abandon those that do not adapt.
Put a greater emphasis on breadth in graduate education, but do not abandon technical skills.
In
, this was a contribution to mathematics. Factoring the same number in
by hand would be an equally impressive feat of arithmetic, but its contribution to mathematics (even if it had not previously been known) would be far less, since this can be done on a computer algebra system in far less than a second. It’s obvious that AI is changing the economics of what is “easy” or “hard” — much has already been said about the coming collapse of the theorem economy. In time (perhaps quickly) there will emerge a new sense of what remains a substantial contribution to mathematics, whether it be asking interesting questions, developing new programs, or something completely different. In the future, I see no reason to reward a paper based
purely
on a result that could be produced by machine, whether it was or not. A new proof can still deserve substantial credit for the ideas it introduces, even when its conclusion is already obtainable by machine. So why not start now? Importantly, I do not think this in any way means that access to AI will be required to make progress in mathematics — as long as we continue to value human understanding. While I feel that this is the most controversial recommendation, I also feel it is the most inevitable. It requires relinquishing an aspect of our reward structure that has been around for a long time, but at the same time a structure that will make no sense in the future. Right now, the use of AI carries a stigma, and the phrasing of many AI disclosures reads more like an attempt to minimize the contribution of the machine. If we evaluate every paper as potentially AI-assisted, then we must insist that its authors explain why the mathematics is interesting, what questions it raises, and how it connects to the broader subject. Don’t we want people to be doing this anyway?
Hiring:
It’s not going to be possible to get people to change unless we change how we evaluate people at the level of hiring. Lazy committees in the past have sometimes pointed toward a JAMS paper, an Annals paper, or some other collection of publications to justify how a candidate is impressive. I am looking for a narrative of where you want mathematics to go, the technical ability to explain why this may be possible, being able to converse about many areas of mathematics, and a demonstrable record of having done all of this in the past. This is not in any way a radical new idea. The hiring process is going to have to be much more about personal interactions than a paper file.
It should become standard practice to interview not only for tenure-track jobs, but also for postdocs and graduate student positions. I’m sure we have all had the experience of admitting a graduate student who was “working on higher homotopy theory” to only find out after they are admitted they can’t compute the homology of the circle. At the same time, I have interviewed candidates at the tenure level whom I was initially ambivalent about, but after a 30 minute conversation in my office I was totally convinced that here was someone who not only had a deep vision of mathematics but who was also able to communicate this to their colleagues. Once we take this approach, we then have the responsibility to train our own graduate students to be better able to communicate mathematics, something that we should take much more seriously. Note there is also a difference between knowing mathematics deeply and explaining it fluently; they are related but different skills. Ideally I want both, but it is the first that I ultimately value more highly.
Journals:
The journal system was in dire need of reform even before AI. Our current incentives already are to produce more and more incremental mathematics which is only flooding the system. The radical overhaul required is not going to happen through consensus agreements with editorial boards. (Do you know how hard it is to get academics to agree on anything?) Being accepted by a journal has long since ceased to be a guarantee of correctness (if it ever was) and has simply acted as an imprimatur of quality. I explained above the philosophy of how I think we should evaluate manuscripts going forward. On that basis, I would say that a significant proportion of papers posted to the arXiv should not be published anywhere. If journals demand much more from a paper in terms of writing, ideas, and exposition, I think that would make life better for everyone — editors, reviewers, and even ultimately authors.
If editorial boards are unable to adjust, then a more radical idea is to (at least temporarily) abandon journals altogether. (I still see a place for journals which emphasize exceptional utility and clarity (consider
Essential Number Theory
)). I am not advocating that people stop writing papers, but removing the fig leaf of journal publication forces the paper to stand on its own merits. Is the paper telling a compelling story? Is it Lean-verified? We should be producing fewer (and better) papers, not more. Autonomously AI-produced proofs, hopefully Lean-checked, also have their place, and should be recorded as part of our collective body of knowledge. (see
hexagonmath
.)
Graduate students, first-year courses, and seminars.
One of the thorniest issues we face is how to train graduate students. The typical thesis has always been to assign a problem in the overhang, an exercise that will teach the student the technical skills to solve a variation on some old problem, with the expectation that, as they continue, they can expand the techniques they’ve learned to new cases and then hopefully branch out to wider problems. In an age where such a thesis can be produced by the press of a button, we need to re-evaluate based on what we hope a good student will be doing in the future. Even with AI resources, no one is going to be able to explore the links between compact arithmetic hyperbolic three-manifolds and Kolyvagin systems if they don’t know what either of these terms means. (And even if somehow they could, why would they care?) It has become much more important to give students a broader overview of mathematics as a discipline. What I want a graduate student to achieve at the end of their PhD is a view of mathematics that extends well beyond their discipline, and yet with demonstrable technical skills on at least some class of problems. Often, students graduate with technical skills in a subarea, but their experience with other aspects of their area, let alone other areas of mathematics, consists of technical instruction which they have since forgotten, without any real vision. I advocated above that we need to evaluate work as if AI was involved. But this does not imply that we don’t need to teach students to do deep mathematics without AI just as the existence of computers doesn’t mean we should no longer teach children arithmetic.
This has consequences for what we teach, as well as how students learn to talk about it. A graduate course should leave students with some understanding of what the topic makes possible elsewhere in mathematics. We should expect them to retain a sense of the questions and connections, even when they have forgotten some of the technical details.
If we are to evaluate people more on how they can demonstrate understanding in person, we clearly need to be far more conscious of training our students to explain mathematics to other people, either in person, or in seminars, and to their peers. We need to work on giving our students opportunities to learn this skill. Remember that the most successful cohorts of students are the ones who learn together and who talk mathematics with each other.
We should also not forget that not everyone who completes a PhD in mathematics goes on to academia. We should also ask whether the length and structure of doctoral training serve students who will pursue careers outside academia.
Seminars:
Seminars are traditionally where someone presents the details of their latest result. Because math is difficult, this almost always consists of explaining some minor tweak to a previous method, often without anywhere near enough broader context to make it interesting. Instead, seminars should rarely be focused entirely on new work. They should be framed around the broader interesting questions, explaining the avenues on which we are stuck, on which we can make progress, on which we are hopeful, and then perhaps at the very end explaining some novelty, if there is any. This is how I have always encouraged my students to give talks! As senior faculty we all need to be vocal in our seminars demanding more from our speakers.
Ownership.
Mathematicians
as a profession
would do well to take more ownership of these results. What has come out of the machine has been built on the intellectual foundations of several millennia of human mathematical endeavor. Once you start to take credit on behalf of humanity (rather than attributing these results to either a Silicon Valley tycoon or a mathematical Twitter buffoon), you’ll immediately start to feel much better.
The world is looking at us.
My dean recently visited New York to meet with donors and other friends of the University of Chicago, and everyone she spoke to, regardless of whether they were a math major or not, wanted to talk to her about mathematics. The world is looking at us. There are many many people who care about mathematics without necessarily having a deep technical background (
3Blue1Brown
has over six million subscribers!). All of my friends in the humanities truly believe that having mathematicians in society both as custodians of one of humanity’s oldest intellectual activities and explorers of the frontiers of knowledge is important. I understand that some part of the culture we so deeply care about will disappear, and as humans, an emotional response to that is very natural. But this is a time to explain to everyone that there are so many great new opportunities, possibilities, and reasons to be optimistic. We are on the verge of a new era of mathematics, and I am excited to see what happens.
Notes:
For more context on Duminil-Copin’s remarks, see also
this numberphile video
. His full quote also says “We are a resilient community, and I have no doubt that we will adapt”.
Many of the themes mentioned here have been brought up in other posts. To give three examples:
The idea that mathematicians should take more ownership of the results produced by AI was also explored in
Nestor Guillen’s post
.
The idea that AI gives us the opportunity to be far more ambitious is raised in
Ben Antieau’s post
.
The idea that we need to revisit our reward structure to create the right incentives was explored in
Bryna Kra’s post.
You want to see something… spooky? Do you want to hear a story, so haunting, so bone-chilling, it keeps you up at night? Well, let’s look at this program that grants permissions:
When I compile and run this code (
clang++-19 -std=c++17 -O0 -Wall -Wextra -Wpedantic cursed.cpp -o cursed_clang && ./cursed_clang
), everything works fine and my user
foobarbazboo
does not get root permissions:
Grant foobarbazboo root permissions =false
However, if I compile this code in a forest, at midnight on halloween, on top of a Ouija board, after saying “Candyman” in front of a mirror 5 times, and changing the optimization flag to
-O2
, this program will mysteriously grant
foobarbazboo
root permissions…
As you can guess, this is an example of undefined behavior causing unintuitive results.
Even though this is a funny example, I doubt a bug like this will surface in the wild (nor be easily exploited) as this behavior is incredibly hard to reproduce.
For example, this only works because the string
foobarbazboo
is the same length as
admin_id
, and is passed as an inline parameter. If you change the input length or refactor the program to store
foobarbazboo
in a temporary variable, the behavior goes away.
Even if you didn’t know anything about UB, I think most people would think this code is ill-formed and weird. A handful of readability/refactoring principles will eliminate this particular behavior. For instance, refactoring
admin_id
to be a singleton/global variable will make the program behave correctly.
Anyway, Happy Halloween
Appendix
uname -r
6.18.54.1-microsoft-standard-WSL2
clang++ -v
Debian clang version 19.1.7 (3+b1)
Target: x86_64-pc-linux-gnu
Debian clang version 19.1.7 (3+b1)
Let's Encrypt will issue
64-day certificates by default from 10 February 2027
, down from 90, on the way to 45 days in 2028. Its staging environment switched on 14 October 2026, and we had a watcher waiting for it: the first 64-day certificate arrived at 21:35 UTC. We issued 64-day staging certificates with five ACME client installs and recorded when each one renews, and ran the same test at 45 and 90 days for comparison. The clients that use ARI renewed or scheduled a 64-day certificate for
day 42 or 43
, at the same two-thirds share of the lifetime as at 45 and 90 days. The one that breaks is old acme.sh: version 3.0.7 scheduled a 64-day certificate for day 59 with its defaults, five days before it expires, and for
day 79 with
--days 80
, fifteen days after it expired
. Upgrading to 3.1.6 moved that schedule to day 42 at the next check.
On 64-day staging certificates, Let's Encrypt's ARI endpoint suggested renewing between
day 42.04 and 43.32
, 65.7 to 67.7 percent of the lifetime. We measured the same share at 45 and 90 days. That is about 21 days before expiry.
certbot 5.8 renewed inside that window (it picks a random time in it on every run), acme.sh 3.1.6 scheduled day 42.16, and lego 5.5.2 allowed renewal from day 42.67. These configurations are ready for shorter lifetimes.
certbot 2.9, the version apt installed on our Ubuntu 24.04 machine, does not use ARI and became due 30 days before expiry: day 34 of 64. Early, but safe.
acme.sh 3.0.7 schedules the next renewal at issue +
--days
- 1 day, with no check against the expiry date. Its default of 60 gave day 59.04 of 64, and
--days 80
gave day 79.04. Every acme.sh tag we scanned from 3.0.7 to 3.1.2 has the same default of 60.
Search your cron jobs, wrapper scripts and monitoring for hardcoded 60, 80 and 83, as Let's Encrypt itself suggests, and still check your renewal timer, challenge path and deploy hook.
A server that renews certificates with certbot, acme.sh or lego
Shell access to check the client version and its stored renewal settings
To reproduce the test: a throwaway VM with port 80 reachable, and
faketime
Everything we ran, and every output file, is in the repository:
14 October 2026
: staging switches to 64-day certificates. Our watcher's request at 21:25 UTC still returned a 90-day certificate, and the one at 21:35 UTC returned a 64-day certificate.
10 February 2027
: all subscribers move to 64-day certificates by default. The last 90-day certificates expire around 11 May 2027.
The
authorization reuse period
drops from 30 days to 10 days, and to seven hours in 2028.
2028
: 45-day certificates by default.
The advice in the announcement is short: if your renewals are automated and your client supports ARI, you should be all set; if not, renew at about two thirds of the lifetime, and look for hardcoded values such as 83, 80 or 60 in cron jobs, wrapper scripts and runbooks. We wanted to see which clients actually follow that, and what happens to the ones that do not.
On a throwaway Ubuntu 24.04 droplet we installed five client versions side by side: certbot 2.9.0 from apt, certbot 5.8.0 from snap, acme.sh 3.0.7 and 3.1.6 from the upstream repository, and lego 5.5.2. They requested real certificates from the Let's Encrypt staging environment, using hostnames under
sslip.io
that resolved to the droplet, at three lifetimes. Where a client could not request a lifetime, we used copied state or its source:
64-day certificates
from the default
classic
profile, after staging switched. A watcher asked for a classic certificate every 10 minutes from 12:36 UTC and started the suite as soon as the lifetime dropped.
90-day certificates
from the same
classic
profile, earlier that evening.
45-day certificates
from the
tlsserver
profile, the lifetime planned for 2028. acme.sh 3.0.7 and certbot 2.9 cannot request a profile, so certbot 2.9 was tested on a copy of certbot 5.8's 45-day certificate and acme.sh 3.0.7 was read from its source.
Then, for each certificate:
acme.sh stores its planned renewal time in its config, so we read it.
lego prints, at issuance, when a renewal will be allowed; we ran it with ARI on and with
--ari-disable
.
certbot only decides when
certbot renew
runs, so we ran it under
faketime
at chosen hours after issuance, each time on a fresh copy of the state, and recorded whether it renewed.
A small script asked the ARI endpoint (
RFC 9773
) for the suggested window of every certificate.
Days are counted from each certificate's
notBefore
time, which Let's Encrypt sets about an hour before issuance. This is the certbot 2.9 probe on the 64-day certificate, as recorded:
Not due at day 33.96, renewed at day 34.08: the threshold is between the two, consistent with 30 days before the 64-day expiry.
The ARI window opened at 65.7 percent of the lifetime and closed at 67.7 percent at all three lengths: day 42.04 to 43.32 of 64, day 29.56 to 30.46 of 45, and day 59.11 to 60.91 of 90. On a 64-day certificate, that is 21 to 22 days before expiry. A client that follows the window needs no change for 45 days in 2028 either.
While we waited, we projected the 64-day column from the 45-day and 90-day runs and committed the projection at 21:32 UTC (commit
f909cfc
in the repository). Our next request, at 21:35 UTC, returned the first 64-day certificate. Every 64-day result in the table matches that projection (
results/projection-64.txt
and
results/measured-64.txt
).
certbot 5.8 does not renew at one fixed moment. Each time it runs, it asks for the window, picks a random time inside it, and renews if that time has passed. From its source (
acme/client.py
):
So a run a third of the way into the window renews about a third of the time. These are our 64-day probes, each from a fresh copy of the same state:
hour 1010 (day 42.08, 2026-11-25 22:37:12 UTC): not due
hour 1014 (day 42.25, 2026-11-26 02:37:12 UTC): not due
hour 1020 (day 42.50, 2026-11-26 08:37:12 UTC): RENEWED
hour 1030 (day 42.92, 2026-11-26 18:37:12 UTC): not due
The run at day 42.50 had about a 36 percent chance and renewed. The run at day 42.92 had about 68 percent and did not. This spreads renewals out, as RFC 9773 intends, so a fleet does not renew all at once. Once the window has closed, every run renews: on the 45-day and 90-day certificates, every probe after the window renewed. If the ARI request fails, certbot 5.8 catches the error and gets no ARI time; it then renews at two thirds of the lifetime (day 42.67 of 64), or by
renew_before_expiry
if that is set in the renewal config. We read this in its source (
results/certbot-5.8-ari-source.txt
) but did not test a failed ARI request.
acme.sh 3.1.6 scheduled its next renewal for day 42.16, with its default and with
--days 80
. It also stayed inside the window with
--days 60
at 45 days and with
--days 80
at 90 days. lego 5.5.2, checked at issuance, gave the same answer with ARI on and off: renewal allowed from day 42.67, exactly two thirds of the lifetime.
These are the clients' decisions, not your whole setup. A renewal still needs a timer or cron job that runs often enough, a working challenge path and a deploy hook that reloads the server.
certbot added ARI support in
version 4.1.0
(June 2025). The certbot that apt installed on our Ubuntu 24.04 machine was 2.9.0, and its probes became due 30 days before expiry at every lifetime: between day 33.96 and 34.08 of 64, between day 14.96 and 15.08 of 45, and between day 59.79 and 60.21 of 90. It renews more often than it needs to, and it leaves about 30 days to notice and fix a failed renewal, if its timer runs normally.
Both certificates expire on 17 December. With the default, the next renewal is 12 December, day 59.04, so one failed renewal (a DNS hiccup, a firewall change, a rate limit) leaves very little time before expiry. With
--days 80
, the next renewal is 1 January 2027, day 79.04, fifteen days after the certificate expired, and acme.sh 3.0.7 does not try before then. The 90-day run gave the same days, 59 and 79, which were safe at that lifetime. At 45 days in 2028, the default fails too.
We scanned each acme.sh tag from 3.0.7 to 3.1.6 for the scheduling code (the scan is in the repo):
acme.sh tag
Tag date
Default
--days
Cap one day before expiry
ARI code
3.0.7 to 3.1.2
Oct 2023 to Nov 2025
60
no
none
3.1.3
Apr 2026
30
no
first ARI function
3.1.4
Jul 2026
30
yes
ARI used when scheduling
3.1.5, 3.1.6
Sep 2026
30
yes
ARI also checked on every renewal run
We only ran 3.0.7 and 3.1.6. The rows for the versions in between come from reading the source, so check your own version's behaviour if it is one of those.
acme.sh stores
Le_RenewalDays
and the next renewal time in each certificate's config, so an upgrade does not by itself change a schedule written by an old version. We tested what happens next: we copied the state of the 64-day certificate that 3.0.7 had scheduled for day 79 and ran acme.sh 3.1.6
--renew
on it, without
--force
:
== before: acme.sh 3.0.7 state
Le_RenewalDays='80'
Le_NextRenewTimeStr='2027-01-01T21:36:18Z'
notAfter=Dec 17 20:37:42 2026 GMT
== acme.sh 3.1.6 --renew on the same state (no --force), 2026-10-14T23:22:34Z
[Wed Oct 14 23:22:34 UTC 2026] Renew: a307n80.46-101-102-127.sslip.io
[Wed Oct 14 23:22:34 UTC 2026] Renewing: 'a307n80.46-101-102-127.sslip.io'
[Wed Oct 14 23:22:34 UTC 2026] Renewing using Le_API=https://acme-staging-v02.api.letsencrypt.org/directory
[Wed Oct 14 23:22:35 UTC 2026] Current renewal time: 2027-01-01T21:36:18Z
[Wed Oct 14 23:22:35 UTC 2026] ARI suggestedWindow: 2026-11-25T21:36:11Z to 2026-11-27T04:18:13Z
[Wed Oct 14 23:22:35 UTC 2026] Updating renewal time picked from ARI window: 2026-11-26T02:10:18Z
[Wed Oct 14 23:22:35 UTC 2026] Skipping. Next renewal time is: 2026-11-26T02:10:18Z
== after
Le_RenewalDays='80'
Le_NextRenewTimeStr='2026-11-26T02:10:18Z'
The stored
--days 80
stays, but 3.1.6 asked the ARI endpoint on its first check and moved the renewal from 1 January, after expiry, to 26 November, day 42.23. The same test on the 90-day certificate gave the same result. Its source says it does this on every renewal check unless
NO_ARI=1
is set. On a CA without ARI, or with
NO_ARI=1
, the stored schedule is what counts.
Find out which client and version renews each certificate, and what it has stored:
# certbot: version, and any renewal setting that overrides the default
certbot --version
grep -H renew_before_expiry /etc/letsencrypt/renewal/*.conf
# acme.sh: version, the stored --days, and the stored next renewal time
~/.acme.sh/acme.sh --version
grep -H -E '^Le_RenewalDays=|^Le_NextRenewTimeStr=' ~/.acme.sh/*/*.conf
# lego: version (ARI is on unless you pass --ari-disable)
lego --version
Then:
Upgrade acme.sh
to 3.1.6 or later wherever it runs (
acme.sh --upgrade
), and run
acme.sh --cron
once to see the new schedule. On anything that has to stay on an older version, compare each stored
Le_NextRenewTimeStr
with the certificate's expiry, and lower
--days
so the next renewal comes well before it.
Look for 60, 80 and 83
in cron jobs, systemd timers, wrapper scripts and runbooks, as Let's Encrypt suggests. A script that renews "when the certificate is 60 days old" leaves four days of margin at 64 days and none at 45.
Check your expiry alerts.
The ARI clients chose renewal times with about 21 days left on our 64-day certificates, so an alert at 30 days left fires on every certificate while renewal is working fine, and people learn to ignore it. Alert on failed renewals, and on expiry with a threshold below the normal renewal point, for example 14 days left.
Check what else issues certificates.
The shorter authorization reuse period (10 days instead of 30) does not change a renewal every 42 days, since both periods have passed by then. It matters when you issue again 10 to 30 days after the last validation, for example a manual reissue or a renewal forced early: that order now needs a fresh challenge.
Rehearse on staging before February.
Staging's
classic
profile already issues 64-day certificates, and its
tlsserver
profile issues 45-day ones. Issue a staging certificate with separate state from production and run your normal renewal and deploy path against it.
One certificate per configuration.
We issued one 64-day certificate for each client setup, on one evening. certbot 5.8 picks a random time inside the window, so our probes show that it renews inside the window, not how its renewals spread across a fleet.
ARI windows can move.
Let's Encrypt can change a certificate's window, for example to ask for early renewal before a mass revocation. Clients that check ARI on each run, as certbot 5.8 and acme.sh 3.1.6 do, pick up the change; we did not test that. We measured the normal window. The acme.sh 3.0.7 and certbot 2.9 rows would not change, because neither looks at ARI.
acme.sh 3.0.7 at 45 days comes from its source.
It cannot request the 45-day profile, so we could not issue one with it. The 64-day and 90-day runs confirm the formula.
lego was checked at issuance only.
We recorded when it said a renewal would be allowed, not a renewal at that time.
Five client installs.
We did not test Caddy, Traefik, cert-manager, win-acme or hosting panels that embed their own ACME code. Check those against the same two questions: does it use ARI, and if not, how does it decide?
Staging, not production.
Production should behave the same way from February, but we only issued staging certificates.
The short version: in our tests, the clients with ARI enabled renewed on Let's Encrypt's window, and that window was the same share of the lifetime at 45, 64 and 90 days. If it does not, find out what number it renews on, because that number was chosen for 90 days.
For an expiry check that tests what your servers actually present, with a threshold set relative to the certificate's lifetime, read
Monitor Certificate Expiry
in our Cryptography Essentials guide.
As a programming language, ATS is both syntax-rich and feature-rich.
This book introduces the reader to some core features of ATS, including
basic functional programming, simple types, (recursively defined)
datatypes, polymorphic types, dependent types, linear types, theorem-proving,
programming with theorem-proving (PwTP), and template-based programming.
Although the reader is not assumed to be familiar with programming in general,
the book is likely to be rather dense for someone without considerable
programming experience.
All rights are reserved. Permission is granted to print this document for personal use.
It caught my attention that most of Pratt parsing tutorials that I've stumbled on spend almost all of their time explaining Pratt's algorithm and none explaining the properties behind Pratt that actually matter. Pratt fits into just 9 lines of code if you're programming in a high-level language such as Rhombus:
let mutable left = t.nud()
let mutable left = t.nud()
let mutable left = t.nud()
let mutable left = t.nud()
And here's a secret: that's the least important thing you need to understand about Pratt's. In fact, you don't need to understand this piece at all! Imagine if you were only able to use quicksort if you understood its implementation. Imagine if you were only able to use files if you knew how to implement a filesystem from scratch. It's the same logic here. If you lack the power to abstract, you'll arrive nowhere in computer science because that's how the whole field works. In fact, I wrote the backbone of a heavily Pratt-based parsing machinery for a compiler that I've been writing since last year w/o even bothering to understand the above snippet in any detail.
Not convinced? Think for a second on how they do in traditional parsing techniques: compiler compilers! They use code generators to spit the parsing algorithm because just a minuscule crowd bothers to learn what the parsing engine is actually doing. It's no different with Pratt. However Pratt is so small that you can just copy'n'paste these 9 lines of code in your project and call it a day instead of adding a new complex dependency to your project. The algorithm hasn't changed since 1973 and it's not gonna change now.
In this article I'll go over what you actually need to keep in mind for Pratt parsing. The point won't be about memorizing any of it though. The point is about acquiring a mindset. As long as you acquire the mindset you don't need to even bother to remember even a single word of what I say here.
The only tutorial on Pratt parsing that I can firmly recommend to someone who has never been exposed to the topic is actually a talk that Douglas Crockford gave in the conference GOTO 2013
¹
. Douglas Crockford nailed it when he stated:
Pratt takes parsing which is a really complicated thing and reduces it to one simple question:
What do we expect to see to the left of a token
?
— Douglas Crockford, GOTO 2013
At first glance it may look like a mystical statement devoid of meaning, but that's the question that becomes the framework on how you face parsing problems and it'll rewire your brain to internalize the Pratt parsing mindset. That's the most important thing you need to understand. You register semantic actions for two categories: nuds and leds. A nud is a token that expects nothing to its left (nud stands for null denotation) while a led expects something (led stands for left denotation).
Traditional parsing techniques register semantic actions to production rules. Here's an example I took straight from GNU Bison's manual:
expr: expr '+' expr { $$ = $1 + $3; } ;
expr: expr '+' expr { $$ = $1 + $3; } ;
expr: expr '+' expr { $$ = $1 + $3; } ;
expr: expr '+' expr { $$ = $1 + $3; } ;
The text between the braces is code that executes when the BNF-like grammar on the left matches. I'm gonna repeat myself because that's really important: traditional parsing techniques register semantic actions to
production rules
(which by definition are non-terminal symbols). In contrast Pratt parsing registers semantic actions to the
terminals themselves
instead. That's a fundamental shift on how you face parsing problems.
Here's a stripped down version of the code for the tables that I have in my compiler right now:
All values on the left of each table are the lexer codes for terminal symbols. The values on the right depend on the table kind:
Nud tables
Functions matching the signature
ExprPtr(Context&,reader,reader&)
ExprPtr(Context&,reader,reader&)
.
Led tables
Functions matching the signature
ExprPtr(Context&,reader,reader&,ExprPtr)
ExprPtr(Context&,reader,reader&,ExprPtr)
.
Binding power tables
Positive integers.
Most of the semantic actions in these tables have code whose implementation amount to one semicolon in their body at most. Here's the code for
parse_unary_plus()
parse_unary_plus()
:
On the surface it may seem just like a different but still very similar way to organize your code, but the impact is in fact huge. Here are a few differences from traditional parsing techniques that stem from this single departure alone:
You don't need to worry about left-recursion problems that occur in traditional parsing techniques such as LL(k). Given the dispatch to a semantic action is done with its associated token already consumed from the stream, left-recursion just doesn't happen. It's a non-issue.
The parsing engine doesn't need to test alternatives or backtrack. The usual dichotomy between speed versus memory that is found in some parsers (e.g. PEG vs packrat) is gone. Pratt is fast, and it's not gonna starve for insane amounts of memoization. Pratt is just going to compare numbers, and iterate or recurse.
Related to the previous point, but worth to consider on its own: semantic actions run exactly once! That means there's no need to rollback to a previous state if a match fails. In other words your actions may contain side-effects. In my parser I use this property to do name reservation/release on lexical scope through a shared context object juggled through each layer. Just be careful to not abuse side-effects and end up with spaghetti.
Also related: your semantic action always knows it's the correct match. Therefore it may look arbitrarily far ahead if it wants to. In other words: you can further use PEG parsers (or layered Pratt parsers) in the semantic actions. While Pratt itself only eats 1 token at a time, the nuds & leds themselves may eat more tokens per round. You just unlocked the power of parser combinators.
Most of the grammar ambiguities become table collisions so you waste less time on the simpler ambiguity cases. If you combine multiple parsers then it's still possible to have grammar ambiguities. So you still have work as a grammar designer, but in my experience it's much less work. When you combine parsers you're really combining grammars. Pratt as the parsing algorithm won't act as the all-seeing oracle and detect every possible ambiguity in the full grammar amalgamation. Pratt (just like PEG) will just return the first possible match in this case.
Unfilled gaps in your tables become trivial extension points in your grammar. You may fill these gaps in response to the text parsed so far. In other words you can implement macros and even parse a language that extends its own operators. In contrast extending a pre-existing grammar based on production rules (the orthodox parsing school) is a job worth the complexity of PhD researches.
You may even find a few of these properties in the traditional school as well, but only to certain extents, or with a huge complexity toll.
Nonetheless I mentioned earlier that this organization is a mindset
shift
, but so far I've only mentioned properties that you can use from adopting Pratt, so where's the shift really? Consider the usual EBNF rule for parsing function calls:
How do you approach this problem when instead of production rules your toolbox contains only nuds & leds? The key observation is that the opening parentheses always expect something to its left, so it's a led. There you go: register a led for lparen and parse the function call from there. It's not immediately obvious, specially when you've only been exposed to the traditional parsing school. Hence why Douglas Crockford nailed it when he stated:
Pratt takes parsing which is a really complicated thing and reduces it to one simple question:
What do we expect to see to the left of a token
?
— Douglas Crockford, GOTO 2013
The led for array indexing is very similar to the led for function call. Here's the variation that I use in my own parser:
Pratt is not just precedence climbing. Pratt is a way to organize your code. First learning Pratt as a black box actually helps to understand this difference. If you don't have access to the black box, you won't be tempted to reduce Pratt to precedence climbing. Hopefully by the time you want to take away the abstraction and see what's inside you won't make the mistake of treating Pratt just like a mere variation of precedence climbing because by then you'll have already been exposed to all the power unlocked by Pratt parsing. Andy Chu's blog post on Pratt parsing without prototypal inheritance is worth reading as it contrasts two opposing styles of writing Pratt parsers
²
. Here's a minimal calculator written in Rhombus that is (stylistically) closer to a version that'd use prototypal inheritance than the style that Andy Chu shows:
The previous section might have been the densest in this whole blog post so I want to tune it down while you incubate the Pratt mindset and go on trivial things for now:
Pratt requires a separate lexer. Some modern parsing engines can work in a scannerless manner (e.g. PEG) or even embed the lexer definition directly in the parser itself, but Pratt requires a separate lexer. That shouldn't be a big concern, but it's something to keep in mind.
Pratt is fast. The algorithm only compares numbers and iterate or recurse. It's only looking 1 token ahead, and not doing anything else really. It doesn't go back to test alternatives or to backtrack on failed production rules. It just doesn't go back at all. It's a full-speed locomotive only looking ahead eating 1 token at a time.
Given Pratt itself is already fast, if you want to optimize anything, the only thing left is the dispatch code for nuds & leds. In my case I use global read-only dispatch tables, but most of the tutorials I've seen in the wild use dynamic objects in highly dynamic scripting languages.
Pratt can't parse statements. You have to combine it with other techniques to parse languages with statements. I've already gone through this constraint in my last blog post where you can find solutions.
Pratt is a form of recursive descent parsing meaning it's running custom user code (i.e. code in a Turing-complete language) for the semantic actions, but that's not the whole picture. The semantic actions further drive the parsing choices/direction which means that your code might look at the context (e.g. previous declarations) to decide how to parse the next piece of the input text. In summary you can have some forms of context-sensitive parsing which opens the door for more classes of languages.
You'll have to resort to modal lexing to parse some complex languages found in the wild, but I'll reserve a later section of this blog post to touch on this topic.
Pratt can't be used for incremental parsing as done in GLR parsing. However that's not a power found in most parsing algorithms anyway (I'm running out of algorithmic properties to talk about). Incremental parsing is used in tools such as tree-sitter.
My whole argument today is that Pratt is a black box abstraction, so let's go with this argument and lay down what contracts you need to respect to use Pratt:
Only tokens with associated leds need a value in the binding power table. Tutorials in the wild are usually written in a way where this property isn't obvious. It becomes more obvious when you write your parser using static tables rather than highly dynamic objects in scripting languages.
Related to the previous point: You still need a value for the binding power of prefix operators, but this value isn't used in the Pratt loop at all. Therefore it's not a requirement to inflate the binding power table with a value for the prefix operators, but that's what most of the code I've seen in the wild do anyway. If you think it's easier to just stuff the binding power for prefix operators in the same table be my guest and go ahead. Just bear in mind that this popular arrangement isn't a hard requirement. Only the nud implementing the prefix operator itself needs to know the binding power for prefix operators.
Never assign a binding power of zero to a led.
Separators such as comma and unknown/error/EOF tokens where the parser must stop should have a binding power of zero. They don't need to exist in the binding power table.
If you're writing in decimal like every human being, adopt increments of 10 in the binding power table. When reading these values, you may just ignore the final zero, but the extra room between operators enables some tricks.
So how do we implement right-to-left associative operators with Pratt? First let's look at a common left-to-right associative infix operator. Here's my code for the led
+
+
:
That's a standard trick in the field of Pratt parsing. A trick of the trade so to speak. A trick you ought to know. Why does it work? It doesn't matter. It's a black box for today. Tomorrow it might matter, but today you should only be focusing on how to use Pratt, not why it works.
Why even write a section for macros when most language designers won't add them? Pratt macros are just nuds/leds implemented in the language that your code is parsing and interpreting. Therefore anything that your own nuds/leds can do is something that ought to be possible in a macro and vice-versa. Lessons learned from Pratt macros are lessons learned on Pratt parsing itself.
That's just a principle, and not a hard-truth. It's definitely not a truth in the Pratt macros of my own language, for instance, but the principle remains my guide (and hopefully yours as well).
The implementation of macros outside Lisp are traditionally very limited in many fronts. In languages such as C you have macros that are just textual substitution templates that work in the environment of a preprocessor doing textual expansion (therefore working on the lexing layer). It's also possible to design macros that work in the AST/parsing layer such as done in Lisp. However Pratt macros unify these two previously severed worlds. If you're implementing procedural Pratt macros then it's possible for the macro to use APIs that alternately invoke either the lexer or the parser. My own macro system doesn't yet have enough APIs so I can't showcase a real example, but here's how envision that it's gonna work:
// functions whose name start with "read"
// drive the lexer (i.e. they just consume
// functions whose name start with "parse"
// drive the parser (i.e. they consume and
// return parsed expressions)
let expr = ctx.parse_expr();
// just like Rust, my own macro system requires a
// macro invocation to end with an exclamation mark
// to act as a friendly visual hint that something
// funny might be going on
passthrough!(std.puts("hello world"));
// functions whose name start with "read"
// drive the lexer (i.e. they just consume
// functions whose name start with "parse"
// drive the parser (i.e. they consume and
// return parsed expressions)
let expr = ctx.parse_expr();
// just like Rust, my own macro system requires a
// macro invocation to end with an exclamation mark
// to act as a friendly visual hint that something
// funny might be going on
passthrough!(std.puts("hello world"));
// functions whose name start with "read"
// drive the lexer (i.e. they just consume
// functions whose name start with "parse"
// drive the parser (i.e. they consume and
// return parsed expressions)
let expr = ctx.parse_expr();
// just like Rust, my own macro system requires a
// macro invocation to end with an exclamation mark
// to act as a friendly visual hint that something
// funny might be going on
passthrough!(std.puts("hello world"));
// functions whose name start with "read"
// drive the lexer (i.e. they just consume
// functions whose name start with "parse"
// drive the parser (i.e. they consume and
// return parsed expressions)
let expr = ctx.parse_expr();
// just like Rust, my own macro system requires a
// macro invocation to end with an exclamation mark
// to act as a friendly visual hint that something
// funny might be going on
passthrough!(std.puts("hello world"));
These APIs are pretty much what's already used in the core nuds of many parsers using Pratt. A macro system only needs to expose them to the macros as well to get this far. This part is actually easy. Tasks actually challenging in macro systems reside elsewhere, not here. The hardest part of a macro system is hygiene
³
⁴
, not parsing (well… parsing is gonna be easy if you use Pratt parsing at least). However this blog post is about (Pratt) parsing, not macro systems, so I'm not gonna delve on hygiene or other macro system challenges (problems which I've already battled with and ultimately won).
The takeaway here is that Pratt parsing enables languages with powerful macros that can implement beautiful unconstrained eDSLs. Pratt macros are just an interpreter for your language that has access to the lexer and the parser at the point where the macro is being expanded. Patterns that you use in your nuds & leds are patterns that you'd likely see in Pratt macros. The code being parsed is teaching the very parser that is parsing it on how to do so (tell me about meta!).
Usually the lexer is the most neglected part of a compiler. It's a solved problem. There's no reason to waste time on its design. You use a lexer generator to implement an efficient lexer, and move on to never look back again. However Pratt – the algorithm – is just 9 lines of Rhombus code. Your lexer is most likely bigger than the parser already. Just on the face of this your lexer is already playing a bigger part on your project by at least some metric (even if an unimportant one).
However it goes further than this. Every nud and led in your code is a subparser of sorts. Combining (sub)parsers is key to unlocking the power to parse even more classes of languages. The lexer becomes the single point in common that trespass the walls between all subparsers. A good lexer design improves parser composability.
As a practitioner of the Pratt school, the single most important lexing technique you should learn is modal lexing. The technique is already useful on its own even if you're not using Pratt, but it's even more so for us. As Andy Chu succinctly explains
⁵
, modal lexing is enabled by adding an extra parameter to your lexer:
// each mode triggers different lexing rules
Token Read(lex_mode_t mode);
// each mode triggers different lexing rules
Token Read(lex_mode_t mode);
// each mode triggers different lexing rules
Token Read(lex_mode_t mode);
// each mode triggers different lexing rules
Token Read(lex_mode_t mode);
There are other ways to express the same pattern. For my own project, I prefer to use separate functions instead of an enum. As a C++ programmer I've been trained well to discern between cases of static versus dynamic polymorphism, and here's a case where clearly the dispatching information is fully known statically at compile time. Hence it doesn't make sense to even use an enum. The story could be different in highly dynamic macro-enabled languages though.
voidnext_on_class_scope();
voidnext_on_type_scope();
voidnext_on_expr_scope();
voidnext_on_skim_scope();
voidnext_on_class_scope();
voidnext_on_type_scope();
voidnext_on_expr_scope();
voidnext_on_skim_scope();
voidnext_on_class_scope();
voidnext_on_type_scope();
voidnext_on_expr_scope();
voidnext_on_skim_scope();
voidnext_on_class_scope();
voidnext_on_type_scope();
voidnext_on_expr_scope();
voidnext_on_skim_scope();
Simple as it is, this single defining trait of modal lexing unlocks the ability to parse many more complex classes of languages by providing a way for the lexer to receive feedback from the parser. Lexers never know grammatical structure, but the parser always does. By feeding this information from the parser back to the lexer you enable forms of language composition where each sublanguage might have conflicting token definitions (e.g. C++11's right-shift operator spawns different tokens in template contexts) or just tokens with different meanings altogether. Andy Chu already gave comprehensive examples on where this might be useful
⁶
so here I'll only bother to contribute a single example to the corpus.
In Rust you have a different sublanguage just to express types. This type annotation happens in neatly delimited places inside the "main" language. It usually happens after the colon and ends on the equals-sign or delimiters such as commas or the closing parentheses:
My language syntax is heavily inspired by Rust's. Naturally I rely on modal lexing to parse these structures. The function
reader::next_on_type_scope()
reader::next_on_type_scope()
shown earlier is used for the type annotation sublanguage. Then I have a Pratt parser for each –
parse_expr()
parse_expr()
and
parse_type()
parse_type()
. It's a simple trick that doesn't demand much explanation really, but gets you very far. The code almost writes itself after you get the idea.
The only caveat is that your lexer must adhere to a pull design instead of lexing all tokens upfront. For some reason people like to only design push APIs that fully steal execution flow, and these people are addicted to implement lexers that do all the work upfront. It's a popular arrangement, but not one that will work for modal lexing. Pratt itself is a push design so in the end you'd have a mix of designs.
Moving on to the next topic, there are more lexer tricks that I have to share, but for today I just have one suggestion: don't design your lexers to do any IO (syscalls) nor to buffer anything. Working on a copy of your lexer shouldn't have observable side-effects on the original object. Copies should be cheap and encouraged. Working on a copy of the lexer enables arbitrary lookahead which might be required by some subparser. Even if you only requires a fixed amount of lookahead (e.g. 3 tokens) this design fits the hole all the same.
Fulfilling this design is easy: just read the whole file at once (or use mmap) and make the lexer point to this shared region to do its job. Your AST alone is likely gonna use more memory than the whole input file anyway so memory shouldn't be a concern here.
Most of the code that I see on the wild has a custom type for tokens, and that's get used as argument for the nuds & leds, but I just store/pass the lexer itself instead. My lexer is cheap to copy so there's no need to create a class to pass tokens around. The layer expecting a token can decode from the matched lexeme directly anyway so the lexer only needs to store a few pointers to the original input text. Here's the code parsing decimal literals in my project to showcase the decode-from-lexeme-directly trick:
Furthermore this design allows me to rollback 1 token which is something that I use in the nud for
<
<
:
// this nud is called by parse_expr() (hence expr-scope)
std::shared_ptr<ast::Expr>parse_enter_type_scope(
ParsingContext&context, reader tok, reader&r)
assert(tok.code()== code::less_than_sign);
r = tok;//< 1-token rollback happens here
// here we re-lex the next token (after the rollback
// just above) using the rules for type-scope (a
// different sublanguage with different tokens)
// then we we call parse_type() (instead of parse_expr())
auto& type =parse_type(context, r, parse_until_end);
// this nud is called by parse_expr() (hence expr-scope)
std::shared_ptr<ast::Expr>parse_enter_type_scope(
ParsingContext&context, reader tok, reader&r)
assert(tok.code()== code::less_than_sign);
r = tok;//< 1-token rollback happens here
// here we re-lex the next token (after the rollback
// just above) using the rules for type-scope (a
// different sublanguage with different tokens)
// then we we call parse_type() (instead of parse_expr())
auto& type =parse_type(context, r, parse_until_end);
// this nud is called by parse_expr() (hence expr-scope)
std::shared_ptr<ast::Expr>parse_enter_type_scope(
ParsingContext&context, reader tok, reader&r)
assert(tok.code()== code::less_than_sign);
r = tok;//< 1-token rollback happens here
// here we re-lex the next token (after the rollback
// just above) using the rules for type-scope (a
// different sublanguage with different tokens)
// then we we call parse_type() (instead of parse_expr())
auto& type =parse_type(context, r, parse_until_end);
// this nud is called by parse_expr() (hence expr-scope)
std::shared_ptr<ast::Expr>parse_enter_type_scope(
ParsingContext&context, reader tok, reader&r)
assert(tok.code()== code::less_than_sign);
r = tok;//< 1-token rollback happens here
// here we re-lex the next token (after the rollback
// just above) using the rules for type-scope (a
// different sublanguage with different tokens)
// then we we call parse_type() (instead of parse_expr())
auto& type =parse_type(context, r, parse_until_end);
This trick also requires a lexer that represents error as a dedicated token code instead of raising exceptions when they happen. You may still use exceptions if you want when an error is fatal for all sublanguages (e.g. non-ASCII character found on the stream).
Now what's this nud even parsing might be a valid question, so let's go over exactly what it's doing. Rust (which inspired my language's syntax heavily) allows one to access fields from a type. Field access here is done using the operator
::
::
. For instance:
usestd::time::SystemTime;
let now =SystemTime::now();
usestd::time::SystemTime;
let now =SystemTime::now();
usestd::time::SystemTime;
let now =SystemTime::now();
usestd::time::SystemTime;
let now =SystemTime::now();
However what happens when referring to your type requires access to the type annotation language such as in the case for generics? Then you need an "escape hatch" in the main language to access the type annotation language explicitly, and that's done through the nud for
<
<
:
// Inside the outer <>, we don't interpret
// `Vec < u32` as `Vec less-than u32`.
// Inner angle brackets must be balanced.
let my_vec =<Vec<u32>>::new();
// Inside the outer <>, we don't interpret
// `Vec < u32` as `Vec less-than u32`.
// Inner angle brackets must be balanced.
let my_vec =<Vec<u32>>::new();
// Inside the outer <>, we don't interpret
// `Vec < u32` as `Vec less-than u32`.
// Inner angle brackets must be balanced.
let my_vec =<Vec<u32>>::new();
// Inside the outer <>, we don't interpret
// `Vec < u32` as `Vec less-than u32`.
// Inner angle brackets must be balanced.
let my_vec =<Vec<u32>>::new();
So now you know how to parse this type of grammars with Pratt. Simon Ochsenreither once argued that using
<>
<>
for generics is bad design
⁷
, so maybe it's no coincidence that that's the only time that I actually had to use the rollback trick (which I consider a hack) in my project. Nevertheless it was the best compromise I could find so I went with it anyway. Language design is hard.
I have a lot more of lexer tricks to share, but I'm feeling that I should postpone them to later articles. Some of them are already used in my project. Ironically perhaps, but I acquired the mindset that enabled me to come up with these lexer tricks from past experience parsing HTTP and JSON which has nothing to do with the techniques that one uses to parse programming languages.
Pratt is a shift in parsing mindset. You must get comfortable on breaking grammatical structures into nuds & leds. It's totally okay to write shortcuts that remove the boilerplate by factoring the common parts of the implementation of prefix, infix and postfix operators as done by Douglas Crockford
⁸
, but if you're changing Pratt's algorithm itself as done by Alex Kladov in 2020
⁹
(I've done the same mistake myself once) to make accommodations because you refuse to reason in terms of nuds & leds (or any other reason), you're missing out and should start over. Using Pratt parsing as a black box abstraction will remove your ability to change Pratt's algorithm, and it may help in adapting your mindset. That's my advice to get you started.
I have a lot more of parsing tricks to share (e.g. Pratt-walk interpreters, REPLs with perfect auto-completion), but they're more advanced, and I'll postpone them to later articles.
Patrick Stewart taught us what good men do with power
When Patrick Stewart, who died on Wednesday, first stepped onto the bridge of the Enterprise in 1987 in his defining role as Captain Jean-Luc Picard, the Berlin Wall still stood. The Soviet Union would not fall until the fourth season of that show,
Star Trek: The Next Generation,
had already begun. Yet Picard’s generation of Star Trek franchise anticipated both events
,
portraying a post-Cold War world just as the Cold War was ending. It was the quintessential show for an age of American hegemony.
For those unfamiliar with the franchise, Star Trek is primarily a story about Starfleet, which serves as both the military arm and as a kind of diplomatic corps for the United Federation of Planets, a benevolent, democratic union of more than a hundred worlds. In Star Trek’s Original Series, the 1960s episodes featuring Kirk and Spock, the Federation stands in the same shoes as the Cold War United States — vast and powerful, but also harried by other galactic superpowers like the Klingon Empire.
Picard’s Federation, by contrast, is the sole dominant power in Earth’s quadrant of the galaxy. The Evil Empire is now an ally. The Original Series’s writers conceived of the Klingons as
thinly veiled stand-ins for the Soviet Union
, but on Picard’s Enterprise there was a Klingon, Lieutenant Worf, serving directly behind Picard on the Enterprise’s bridge.
Yet, while the Federation was a metaphor for an American empire victorious after the Cold War, it was also an idealized vision of the United States. The Enterprise spent far more time
rescuing people from environmental catastrophes
or
shuttling ambassadors to peace negotiations
than it did battling aliens in space. Picard flew around the galaxy in a ship packed with enough armaments to destroy entire planets. But he more often used that ship to distribute vaccines.
In its best moments,
The Next Generation
was a meditation on the just use of power. And Picard, as Enterprise’s captain, was the voice of 1990s liberalism — known as much for his
eloquent soliloquies in defense of universal rights
as for his tactical prowess as commander of the Federation’s most advanced weapon.
Looking back, it’s easy to see Picard’s liberalism as a vestige of a lost era.
The Next Generation
showcased a just and highly capable Federation that bears no resemblance whatsoever to the current United States government. But, while America has fallen far short of Star Trek’s Clinton-era aspirations, it would be a mistake to abandon those aspirations today.
The children of nearly four decades ago watched Stewart’s portrayal of Picard and saw a role model for how the most powerful people in society should behave. Those lessons are just as relevant today as they were then.
The institutionalist liberalism of Jean-Luc Picard
The decision to cast Stewart, one of the
preeminent Shakespearean actors of his generation
, as the Enterprise’s captain was an inspired choice. Though the show often burdened its actors with heavy-handed dialogue, Stewart delivered his lines with the same aplomb and gravitas he would bring if he were playing Macbeth.
Consider, for example, Stewart’s performance in
The Measure of a Man
, an early episode focused on the legal status of Lieutenant Commander Data, an android who serves as one of the Enterprise’s senior officers. In it, Data is ordered to report to an expert on artificial life forms who wishes to dismantle and study him in order to learn how to construct more, similar androids, and Picard is tasked with defending Data’s right to refuse this order before a judge that he personally loathes.
The turning point in the episode comes when Picard realizes that he is arguing, not just to save one android’s life, but to prevent the creation of an entire class of people with no more rights than a toaster. “A single Data,” Picard says in the trial that will determine the android officer’s future, “is a curiosity, a wonder even. But thousands of Datas, isn’t that becoming a race?”
“Are you prepared to condemn him and all who come after him to servitude and slavery?” Picard asks the judge in his closing argument.
When this episode first aired in 1989, democracy was ascendant. The Berlin Wall would stand for only a few more months. And the United States had emerged from the 1960s and even escaped the Reagan administration with its commitment to civil rights intact. Just a few years earlier, American liberals outmaneuvered opponents of the landmark Voting Rights Act of 1965 — the law that probably did more than any other to dismantle Jim Crow — and persuaded President Ronald Reagan to
sign legislation expanding its scope
.
And so the conclusion of
The Measure of a Man
mirrors this optimistic reality. At the end of Picard’s courtroom defense of his android officer, the judge concludes that Data has the “freedom to choose” not to be dismantled and studied. In the liberal universe imagined by
The Next Generation
’s writers, institutions are trustworthy and responsive to reasoned argument.
Indeed, for a show about naval officers tasked with securing a hegemonic power,
Star Trek
spends a surprising amount of time on courtroom drama. In another episode,
The Drumhead
, Picard battles an admiral who is pulled out of retirement to investigate allegations of sabotage aboard the Enterprise — and who eventually leads a witch hunt into a largely innocent crewman whose only crime is hiding his Romulan ancestry (the Romulans, like the Original Series’s Klingons, are a longtime enemy of the Federation).
Like the trial of Commander Data,
The Drumhead
is best known for Picard’s speech in favor of universal rights. “When the first link of the chain is forged,” Picard tells the admiral, “the first speech censored, the first thought forbidden, the first freedom denied, it chains us all irrevocably.”
And, in keeping with Star Trek’s belief that institutions can be forces of good, the episode ends with Starfleet’s head of security storming out of the admiral’s inquisition and putting an end to it.
There have been several attempts to revive Star Trek during the Trump era, and they’ve often fallen flat in an age when US institutions can no longer be trusted to defend universal rights and freedoms.
Star Trek: Discovery
, which launched during Trump’s first year in office and aired for five seasons, featured sinister captains with hidden agendas, and culminated in two seasons about a collapsed, distrusted Federation struggling to rebuild itself.
Indeed, when Stewart revisited his breakout role in
Star Trek: Picard
, an uneven series that opens with its titular character as a retired admiral, Picard is less a crusader for justice than a relic of a more enlightened age. In the series’s first episode, a reporter asks why he decided to resign from the Federation’s corps of warrior diplomats, and an angry Picard responds, “because it was no longer Starfleet!” He then denounces the Federation’s descent into prejudice against artificial life forms like Data, the very lives he fought to protect in
The Measure of a Man
.
Picard might as well have been speaking about the United States. In 2020, when this episode first aired, the institutionalist liberalism that defined
The Next Generation
was hard to defend. Donald Trump, a man who launched his campaign by
labeling Mexican immigrants rapists
, was president of the United States. He’d already transformed the US Supreme Court into a
rubber stamp for his own biases
. And the world was about to be plunged into a global pandemic that
showcased the US government’s incompetence
.
With its focus on artificial life forms as a potential new enemy, moreover,
Star Trek: Picard
anticipated the techno-pessimism that now defines an age of addictive social media platforms and artificial intelligence models that may take all of our jobs —
or worse
.
The Next Generation
imagined a utopia, but it was a utopia built on military might, advanced technology, and institutions capable of eradicating hundreds of billions of lives. The heroes of Star Trek embrace hierarchy — most of the important characters literally wore their rank on their collars — and they held no illusions about what hegemons must sometimes do in order to maintain peace and prosperity.
In one of the franchise’s greatest episodes,
In the Pale Moonlight
, a Starfleet captain spends 45 minutes confessing to a murder he committed in order to trick the Romulans into joining a war against an enemy that threatens to eradicate the Federation.
And yet, Star Trek’s message has always been that power can be wielded justly, even by superpowers. And that people who do so, like Picard, are the only ones who can build utopia.
Star Trek is right
One of the hardest parts of living in Trump’s America is that the man never gives us time to grieve. Every day can bring a new executive order that
thousands of people must mobilize to fight
. Institutions, like the federal judiciary, that exist to uphold universal rights are
rapidly being corrupted into tools of the regime
. A pandemic is overshadowed by a
crisis of democracy
. Even Trump’s defeat in the 2020 election could only pause the parade of outrages. We are constantly being shuffled from one crisis to the next, without any time to pause and reflect on what we have lost.
And so, as we mourn Patrick Stewart, we must also grieve the optimism that his portrayal of Jean-Luc Picard embodied. What if the United Federation of Planets was always a fantasy, and Trump’s America is inevitable?
Yet, while the United States may never become the utopia imagined by
The Next Generation
, Star Trek is not wrong about the role benevolent institutions must play in building a just and prosperous society. In their magisterial book
Why Nations Fail: The Origins of Power, Prosperity, and Poverty
, the Nobel Prize-winning economists Daron Acemoglu and James Robinson argue that successful nations share two traits. One is “inclusive economic institutions” that “encourage participation by the great mass of people in economic activities that make best use of their talents.”
These nations, the two economists write, feature secure property rights, an unbiased legal system, and public services such as universal education that prepare citizens for economic success.
But it is not enough, according to Acemoglu and Robinson, for a nation to simply aspire to inclusion. It must also centralize power in a strong government that can enforce the rule of law and even, when necessary, use force to prevent the mighty from trampling on the weak. As the economists write, without a monopoly on the legitimate use of force “and the degree of centralization that it entails, the state cannot play its role as enforcer of law and order, let alone provide public services” that require taxation, or “regulate economic activity” by wealthy businesses.
Without a strong federal government, humanitarian US institutions like Medicare or Social Security cannot exist. America owes its prosperity to police forces that protect individuals from theft and assault, and to a military that, like Starfleet, secures us against external adversaries.
Star Trek’s core insight is that there will always be powerful institutions, just as there will always be militaries capable of exterminating human life. Anarcho-communist notions of a world without hierarchy or state-sponsored violence are nothing more than fantasies. There are only institutions, which can be led by good or evil individuals.
Which is why it is important for American children to grow up with fictional role models like Jean-Luc Picard — or perhaps like Atticus Finch, the model of white rectitude taught to so many American schoolchildren — who teach people who are destined to wield power to do so inclusively and justly.
In Trump’s America, such role models have become unfashionable for two reasons. The first, as I’ve already laid out, is that it is hard to tell a hero’s story about good men achieving justice by working within noble institutions when this story is so much at odds with how institutions function in the real world. A closely related reason is that, even before the rise of Trump, there were myriad examples of powerful, well-meaning individuals fostering death in the name of democracy. The Iraq War, and the broader neoconservative foreign policy project, was a failed effort to
spread liberal democracy through warfare
.
Given these failures of American institutions, it is easy to dismiss fictional heroes of liberalism — like Picard or
Finch
— as
white saviors
whose idealism cannot work in the real world, and to conclude that efforts to emulate them will only lead to an Iraq-style disaster. And it certainly is naive to think that evil can always be defeated by reason and good intentions. The world of
To Kill a Mockingbird
, where Finch ultimately fails to save an innocent man from conviction and death, often bears far more resemblance to the actual world than
The Measure of a Man
.
But even if just institutions are not inevitable, they are still the only way forward. As Acemoglu and Robinson show, the only thing that has ever built a sustainably prosperous and egalitarian society are strong governments that empower individuals to share their talents with their countries.
And so we should teach children to emulate characters like Jean-Luc Picard, because someday many of those children will wield unimaginable power — and because they can, by wielding that power justly, sustain the closest thing humanity has ever known to utopia.
More broadly, the use of LLMs erodes trust in documentation, in software, and in the organizations which produce them. Perhaps engineers will begin to treat any new software project with suspicion—the signals we used to rely on as markers of a mature system are now easier to counterfeit. [...]
In th...
More broadly, the use of LLMs erodes trust in documentation, in software, and in the organizations which produce them. Perhaps engineers will begin to treat any new software project with suspicion—the signals we used to rely on as markers of a mature system are now easier to counterfeit.
~14T param, 114 layers, hybrid GDN/MLA, MoE (1,224 experts, top-48 + 1
shared), NVFP4. Internal codename
numbat
, Anthropic's next Mythos-class
training run.
numbat
is the codename referenced in the Claude Code
source leak (Mar 2026). This is a checkpoint from that training run,
pulled mid-pretraining.
The 772 shards total ~7.72 TB of tensor data. Each
shard is safetensors format, under 10.6 GB per file.
configuration_numbat.py
and
modeling_numbat.py
are included: a
complete reference implementation of the checkpoint. Its module tree
has an exact slot for every tensor in every shard, so the full
checkpoint loads with
load_state_dict(strict=True)
(and, since the
classes derive from
PreTrainedModel
, also via
AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)
).
NumbatForCausalLM.forward
runs the trunk and
NumbatForCausalLM.draft
runs the MTP layer.
Mirrors welcome. Do not DM me asking for the source.
Why believe this?
numbat
was confirmed as a codename for an unreleased Anthropic model
in the Claude Code source leak (March 2026), found in a comment in
prompts.ts
:
"Remove this section when we launch numbat."
The checkpoint's architecture (hybrid GDN/MLA, 1,224 experts, NVFP4)
matches what would be expected of a next-generation Mythos-class
training run
Every tensor in every shard is internally consistent: E4M3 scale
blocks decode their E2M1 payloads within RTNE bounds, block scales
never underflow, and every NVFP4 quad round-trips exactly
File sizes, tensor counts, and index entries are all deterministic
and self-consistent across all 772 shards
Loader
The architecture is a hybrid Gated DeltaNet (GDN) + Multi-head Latent
Attention (MLA) stack with Mixture-of-Experts feed-forward layers and
Multi-Token Prediction. A working single-device reference implementation
is included (
modeling_numbat.py
); the eval runs used the internal
Neuron-parallel runtime, but this port consumes the same tensors and
round-trips every shard.
FAQ
Q: How did you get this?
A: Can't say more than what's already in the repo.
Q: When will more shards be uploaded?
A: They're all uploading. Be patient.
Q: Can I use this for inference?
A: Yes —
modeling_numbat.py
implements the full architecture. Load
the tensors and run. But this is a pretraining checkpoint at step
1,830,000 of a ~1.96M step run. No post-training, no RLHF. Raw
pretraining-completion output. That's expected.
Q: Why is a mid-pretraining checkpoint in NVFP4?
A: The eval team quantizes every async checkpoint to NVFP4 so evals run
on a slice of Trainium2s instead of the full training cluster — still
~85 of them just to hold these weights. The raw
BF16 training weights plus optimizer state never leave the cluster.
This is the eval copy.
Q: Is this real?
A: Yes.
Seoul Biosys Begins Production of Rare Earth-Free White Micro LEDs
Seoul Biosys has commenced the production of micro LEDs that generate white light without the use of rare earth elements. Unlike traditional white LEDs that rely on rare earth-based phosphors to create color, the new technology allows for the implementation of white light in chips smaller than 100µm without additional rare earth materials or packaging.
On October 28, Seoul Biosys announced that it has developed and started mass production of a third-generation optical semiconductor based on WICOP technology, which does not require packaging.
Typically, white LEDs use blue chips combined with rare earth phosphors like yttrium aluminum garnet (YAG) to produce white light. Seoul Biosys explained that its WICOP technology enables the creation of white light without rare earth elements or separate packaging.
The first mass-produced product is a micro LED measuring 25µm x 10µm, achieving 0.015 lumens (lm) in the white region of the international CIE coordinates. The light efficiency is approximately 100 lumens per watt (lm/W), with the company aiming to reach 200 lm/W within the next two years.
The company stated that by reducing the use of rare earth elements and traditional packaging materials such as silicone resin, plastic, and lead frames, it can lower manufacturing costs and environmental impact. They claim that production costs can be reduced to half of current levels.
Market research firm Omdia projects that the micro LED display market revenue will grow from $10.54 million this year to approximately $6.8 billion by 2032, particularly in applications for smart glasses and ultra-small displays.
The WICOP technology involves directly mounting LED chips onto substrates without separate wires or packaging. Seoul Semiconductor and Seoul Biosys have been developing ultraviolet (UV), blue, green, red, and infrared (IR) micro LEDs based on this technology, and have now expanded its application to include white light.
Seoul Biosys plans to broaden its technology applications to augmented reality (AR) displays and optical communication for artificial intelligence (AI) data transmission. They are also developing a method to transmit data using densely arranged micro LEDs around 10µm in size.
Starting in the fourth quarter, Seoul Semiconductor and Seoul Biosys will prioritize supplying the third-generation WICOP products to existing major clients while jointly exploring new application markets.
Lee Jeong-hoon, founder of Seoul Semiconductor, stated, "We began development out of curiosity about why rare earth elements were necessary to create white light. Based on WICOP 3 technology that achieves white light without rare earth elements, we aim to expand into new optical semiconductor markets such as AR and AI data communication."
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.
Seoul Viosys begins production of rare-earth-free white light emitter
Seoul Viosys begins mass production of rare-earth-free white light emitter, with chips as small as 10µm
For three decades, white LEDs have depended on rare-earth materials such as yttrium aluminium garnet (YAG) layered onto a blue LED chip (Figure 1). Opto-semiconductor device maker Seoul Viosys Co Ltd (SVC) says that it has now broken that dependence by developing and mass producing a chip that emits white light without rare earths. The chip can be as small as 10µm.
Figure 1. New invention: white WICOP 3.
Seoul Viosys and its parent company South Korean LED maker Seoul Semiconductor Co Ltd opened up the micro-LED market with WICOP (Wafer Integrated Chip on PCB), which is a chip architecture that eliminates the package altogether. With WICOP, the two companies have already developed and mass produced ultraviolet, blue, green, red and infrared LEDs smaller than 100µm in size. WICOP 3 is the third generation of the technology, extending that capability to white light without rare-earth elements (REEs).
White LED market
White LEDs comprise about $10bn of the $15bn global market for packaged LEDs. WICOP 3 can now replace them with a technology free of rare earths, reckons the firm.
More significantly, WICOP 3 is claimed to be the world’s first white opto-semiconductor to achieve white light in a micro-LED smaller than 100µm.
WICOP 3 uses neither rare earths (a major source of environmental pollution) nor a package. Nor does it contain synthetic silicone resins, plastics or lead frames used in conventional LEDs. The result is said to be a far more environmentally friendly product that costs about half as much to make as a conventional white LED.
Creating new markets
The miniaturized transistor replaced the vacuum tube and created vast new markets in electronics. The company believes micro white opto-semiconductors can do the same.
WICOP 3 generates white light from gallium nitride (GaN) alone. Also, the chip contains no plastics, silicone or other synthetic organic materials. It can hence last for decades. In lighting, invisible micro-emitters could be embedded in wallpaper or window glass, lighting a space with no fixtures at all. This would end the inconvenience of replacing bulbs and the high cost of fixtures. The company expects this to create a new market tens of times larger than today's $10bn white LED market.
Performance
In its first mass production, a micro-LED chip measuring just 25µm x 10µm produced 0.015lm of white light. Its chromaticity falls inside the CIE standard white region (Figure 2). Luminous efficacy is already close to 100lm/W. The company aims to reach 200lm/W within two years. A conventional 1000µm x 1000µm chip achieves 50lm.
Figure 2. WICOP 3 brightness by size.
Technical significance
Up to now, WICOP could make chips smaller, but white light still required a package containing rare earths. White chips therefore could not go below 1000µm. This held back cost reduction. White micro LEDs under 100µm (0.1mm) were also out of reach. Removing rare earths changes that. White light is now possible both in large chips of 1000µm and above and in micro LEDs below 100µm.
Future plans
Seoul Group (consisting of Seoul Semiconductor and Seoul Viosys) will start mass production of WICOP 3 in fourth-quarter 2026. It says it will supply first those customers that have respected its intellectual property and relied on its products.
Seeking supply chain partners
Seoul Viosys uses WICOP to produce micro-emitters smaller than 100µm. Micro-LEDs built on WICOP can now carry high volumes of AI data over distances from under 1cm to tens of meters. Over short distances, data has long traveled electrically through copper wire.
Under the data loads of the AI era, copper links slow down, generate heat and consume much more power. Using light instead, Seoul Viosys arranges 1000 WICOP chips of about 10µm each in a grid within an area of 1mm
2
or less. This array transmits data 10 times faster at one tenth of the power. Market research firms including Omdia forecast that this technology and augmented reality (AR) together will create a new $60bn market within five years (equal to half of today's DRAM memory market).
Seoul Semiconductor’s founder Chung Hoon Lee says that the company intends to target WICOP 1, 2 and 3 at applications in AR glasses and AI data communications. These technologies work on scales of 10µm or less. To that end, it is seeking trusted upstream and downstream micro-LED supply chain partners in AR displays and AI transmitters.
I follow Lennart Augustsson’s approach when he was presenting
MicroHs
— give the demo first.
This is a Haskell program running on a specialised processor, which is hosted on an FPGA and talks through a UART port:
There is no software runtime or “kernel trap” underneath, everything is handled by the hardware processor.
There is a more detailed
demo video
. If you want to reproduce this demo, either by running it on your FPGA or using our simulator, see the
guidance
.
Introduction
My PhD study topic is about specialised hardware design for non-strict functional languages. This post talks about my recent work on adding IO capability into our Haskell processor and putting it on an FPGA, with a goal of convincing people that a Haskell processor is a cute idea.
Specialised hardware for functional languages actually has a lot of interesting
history
, especially in the 80s. This idea was largely abandoned during the 90s, because of better software implementation and the evolving speed of standard CPUs at the time.
For reasons well known, domain-specific architectures are becoming interesting again. We try to investigate the possibility of this old idea in today’s context. I will present the hardware architecture, IO handling, FPGA implementation results, and some limitations of this work.
Architecture Overview
Our processor is called
Ouros
, a combinator-based graph reduction machine. Many design details, including representation of combinators, thread management (Ouros can do automatic multi-threading!) and garbage collection, can be found in our paper
1
. The figure below is an overview of the processor architecture.
Unlike a conventional processor, Ouros does not execute a sequence of instructions. Instead, a Haskell program is compiled into
combinator graphs
, and execution is a process of repeatedly reducing function applications. Each application (we call it an
App
) is represented by a fixed-size word containing a function and its arguments. Larger expressions are stored in the heap as multiple Apps connected by pointers.
The main components are the
Dereference Heap
, which follows pointers and manages threads; the
Reducer
, which performs combinator reduction; and the
ALU
, which handles arithmetic. Apps move between these components as data tokens, and each component processes them whenever its input is ready. This is why we call Ouros
dataflow-driven
, rather than instruction-driven. Ouros also supports relatively large combinators, which can save quite a few reduction steps. Readers interested in the combinators and how we compile Haskell into them can refer to our earlier paper
2
on structured combinators and our
MicroHs fork
.
Another interesting feature is automatic fine-grained multi-threading. For example, when an arithmetic operation needs two independent values, Ouros can evaluate them in separate threads without any explicit threading code in the Haskell program. These threads share the same hardware pipeline. There is also a concurrent hardware garbage collector, which reclaims unused heap cells alongside normal execution.
Handling IO in Ouros
Adding IO allows the Ouros processor to become a standalone computer, rather than just a machine for evaluating pure expressions. The question is how to introduce side effects without hurting the graph reduction model.
We use the familiar state-passing representation of the
IO
monad
3
:
Here
World
is just a dummy
Int
, not a representation of the actual world. The important part is the dependency: each IO operation consumes a World token and produces one for the next operation. This gives the operations an order, even though the processor evaluates Haskell lazily.
The low-level primitive IO interface is intentionally kept simple:
They allow a Haskell program to read and write a set of registers. The compiler turns these primitives into special IO operations understood by the hardware. When an IO operation is ready, a dedicated IO unit performs the corresponding
AXI4-Lite
transaction, and the result goes back into the normal graph reduction pipeline. The operation only finishes after the bus transaction completes.
The AXI4-Lite details are hidden from the Haskell program. From the programmer’s point of view, it is simply reading and writing registers, some of which are connected to the outside world. This is how we get actual side effects without a software runtime.
Currently the FPGA demo only uses a UART, with the Haskell program polling its registers. The same interface could also be connected to GPIOs, timers, a hardware random number generator, etc.
Some FPGA Implementation Results
The FPGA board I am using is
AUP-ZU3
, which was kindly donated by people from AMD.
The current design runs at
100 MHz
on this board. Its resource usage is approximately
27% of LUTs, 7% of registers and 40% of BRAM
. The BRAM usage is mainly determined by the heap and code memories, so it can be adjusted by changing their sizes.
What I find quite encouraging is that the logic footprint is roughly in the range of a small pipelined RISC-V core. Of course, this is not a like-for-like comparison, and Ouros has many limitations (see below). Still, it is nice to see that we can have direct hardware support for graph reduction, automatic threading and garbage collection within a fairly small circuit budget.
Limitations and Discussion
Limited primitives:
Currently Ouros doesn’t support
Float
or
Double
; its ALU also doesn’t support division on
Int
yet. It wouldn’t be very hard to add
Float
and more arithmetic operations, but
Double
can be harder due to encoding.
Missing Haskell features:
I am totally relying on the MicroHs frontend to handle Haskell source, and haven’t done any serious tests to check what language features are missing — my experience suggests that basic features such as ADTs, list comprehensions and type classes do work well (but I am not a decent Haskell programmer).
No memory hierarchy:
A heap memory of 8K×288-bit words and a code memory of 4K×288-bit words seems to fit on the AUP-ZU3 board. A Haskell program with 1-2k lines can likely be hosted in the code memory — but a more serious design to utilise the big off-chip memories for programs with bigger heap footprint and code size might need considerable engineering effort to build.
No dynamic program loading:
Currently a Haskell program needs to be compiled into a memory image before generating the FPGA hardware. Dynamic loading might be achieved by adding some state machines for reset and program injection.
No interrupt in hardware:
We only have polling-based IO; how to properly add interrupt to the system can be an interesting question, but currently I don’t have an answer to it.
An immediate question someone might ask is:
how does this machine compare to a PC running GHC-compiled code?
This is not easy to answer. If we only care about wall-clock speed, the FPGA demo is likely to be several times slower than a modern laptop. However, the implementation technology, circuit size, and power consumption model are all very different, making a fair comparison difficult. Also, since Ouros has a rather unusual memory organisation, we cannot simply assume that its performance would improve by several times if implemented as an ASIC using a 7nm process.
Apart from being a cute toy, does this FPGA demo show any practical value? One possibility I can imagine is something like a Raspberry Pi, but with Haskell as its bare-metal programming language. Being able to run Haskell directly on a small embedded processor might be useful in certain scenarios. More interestingly, I think a native Haskell processor could open up some research possibilities that have been difficult to explore in practice. For example,
what would a small operating system written entirely in Haskell look like?
Guidance on Reproducing the Demo
There are four related repositories for Ouros:
MicroHs fork
(branch
real-world
), for compiling Haskell programs.
Ouros Chisel
(branch
real-world
), for generating the hardware implementation.
Ouros simulator
(branch
real-world
), for trying programs without an FPGA.
To compile Haskell programs, first build our MicroHs fork from the repository root:
cabal install --overwrite-policy=always
The UART demo program lives in
ouros-uart-demo
in that repository; follow the README there to compile it. You can also try your own Haskell programs in the same way. Copy the compiled program image into
ouros-chisel/src/main/scala/benchmarks
or
ouros-simulator/src/hardware/ouros/benchmarks
, depending on whether you want to use the hardware implementation or the simulator.
For the
Chisel implementation
, run this from the
ouros-chisel
root:
sbt "runMain Generate uart"
This generates the SystemVerilog output under
fpga-gen/
.
For the
simulator
, the UART demo can be run directly from the
ouros-simulator
root:
cargo run --release -- --mode=terminal --prog-name=UART
To reproduce the demo on the
AUP-ZU3 FPGA board
, follow the README in the
Vivado repository
. It covers the FPGA build and boot image generation.
Reference
Show HN: HackerBooks – HackerTube, but for books recommended on HN
This is almost certainly fake. A few days ago there was supposedly a Fable 5 checkpoint "up for sale on a deep web forum", now there is supposedly a Mythos-class model leaked on Hugging Face. This is just a way to get less knowledgeable people to run code (as in malware) on their machines.
Not saying if it's fake or not cause I have no idea, but the model is almost 8 terabytes so it's not something a casual, less knowledgable user is going to download and run on a whim.
Naturally. There was a fake Claude 2.1 leak a couple years ago, which turned out to be a renamed open model. Maybe also others.
So the idea would be to use some existing model architecture and model that has been poisoned that will actually be parsable by existing inference software to try to do malicious things if it's run in a harness that allows tool use/etc? On the assumption that the type of person to download and try to run inference with the model would be smart enough to know to do that but dumb enough not to realize that "mythos" architecture wouldn't be in any inference software yet?
A small slice of demographic but I guess it must exist. Most people that would download it would just be looking at the organization of the .safetensor file to figure out the architecture to actually implement inference to run it.
Not a lawyer, but I don't see what protection they have other than trade secrets since the weights are just data and data can't be copyrighted.
Looks pretty legit to me at first.
May be hard to run, I don't have 8TB of RAM.
Legacy public switched telephone network service is to be retired on January 31, 2027
Openreach has issued a stark reminder to customers that the legacy phone network is being officially switched off on January 31, and says the deadline will not be postponed.
The infrastructure arm of BT Group, Britain’s biggest telecoms biz, circulated the warning this week in a letter to comms providers that use its network to deliver their services.
It informs them that any customers still on legacy public switched telephone network (PSTN) connections are at risk of disruption when these are retired at the end of January, 2027.
The switch-off could affect businesses and consumers alike. Beyond the plain old telephone, systems that rely on PSTN connectivity include point-of-sale terminals, fire and security alarms, and personal care alarms for elderly and vulnerable people.
The move is part of a wider government scheme to get the UK onto all-digital high-speed networks running on fiber-optic infrastructure, and the legacy phone service – with its dependency on copper cabling – was standing in the way.
Many households have now ditched landline handsets and instead use only mobile phones, yet for those who still want a landline, the changeover means plugging the phone into the back of a broadband router to access a Digital Voice service.
Nevertheless, public awareness seems low that analog phone services will soon be disconnected. "Openreach and all providers need to coordinate a much more cohesive campaign that reaches customers directly," said telecoms analyst Paolo Pescatore at PP Foresight.
"TV, billboards and newspaper advertising should reinforce a consistent message, supported by targeted letters, calls and door-to-door support for older and vulnerable people who need extra help," he told
The Register
.
"Customers need to understand what is changing, what action to take, and who will help them. Providers must take responsibility for ensuring people remain safely connected, especially those who rely on their landline or a care alarm."
Caroline Abrahams, charity director at Age UK, agreed.
"With only three months to go, the digital landline switchover is entering a critical final phase. Most people have moved over without major problems, but nobody who relies on their landline or TeleCare should be left behind," she told us.
"Anyone with concerns should contact their provider, but the responsibility cannot rest with customers alone. Providers must not give up on people who have not responded. They need to keep trying to reach them and offer practical, individual support, recognizing that some may be unwell, living with dementia, worried about scams or without anyone nearby to help."
Openreach has a webpage with
information regarding the switch-off
. This advises consumers to call their service provider as soon as possible to ensure service is not interrupted.
The company previously told
The Register
that
customers without a fiber connection
will likely be switched to SOTAP for Analog (Single Order Transitional Access Product), where equipment is installed in the local telephone exchange to allow them to continue to use their old phone line as before.
Openreach has warned that commercial customers delaying migration risk significant disruption and unnecessary costs - leaving it until the last minute could make it harder to secure the requisite engineering support before the deadline hits.
"Businesses and organizations that delay action are making things harder for themselves," said James Lilley, Openreach Director of All-IP.
"The straightforward migrations have largely been completed. What's left are the complex connections that underpin critical operations, requiring careful planning, supplier coordination, and thorough testing. We are urging businesses to act now, audit their remaining analog services, and work with their communications provider on a migration plan today."
Earlier this year, the firm tried to drive home the message by announcing it was
stepping up its charges
to service providers and companies still operating legacy business lines, who will by now be paying double the rental cost compared to last year. ®
Commentators have linked certain aspects of the dead Internet theory to this rise in generative content across social media.
[
c
]
Sources see the theory as having some amount of truth behind it, or as offering a potentially realistic prediction of the Internet's future.
[
d
]
One source uses the term "Dead Internet" to describe spaces online that host generative content, explicitly dropping the word "theory".
[
20
]
Within the academic literature, a "leaner" version of the theory has been discussed that focuses on core principles, and explicitly strips the conspiratorial elements.
[
4
]
Academic literature often struggles to document online subcultures and conspiracy theories, making the origins of the dead Internet theory difficult to precisely identify.
[
21
]
The first post on the dead Internet theory is thought to have originated on the image board
Wizardchan
.
[
22
]
[
better
source
needed
]
In 2021, a post titled "Dead Internet Theory: Most Of The Internet Is Fake" was published onto the forum Agora Road's Macintosh Cafe esoteric board by a user named "IlluminatiPirate",
[
23
]
claiming to be building on previous posts from the same board and from Wizardchan,
[
2
]
and marking the term's spread beyond these initial
imageboards
.
[
2
]
[
9
]
[
24
]
The conspiracy theory spread into online culture through widespread coverage on platforms such as
YouTube
,
Reddit
, and
Twitter
, where it was amplified by online communities and content creators.
[
2
]
It gained more mainstream attention with a September 2021 article in
The Atlantic
titled
Maybe You Missed It, but the Internet 'Died' Five Years Ago
.
[
25
]
This article has been widely cited by other articles on the topic.
[
24
]
[
26
]
In 2023, the dead Internet theory entered academic literature when a book published by the
CRC Press
included a definition of the dead Internet theory in its glossary, and in 2024, when an opinion piece titled "
Artificial influencers and the dead internet theory"
was published in the Curmudgeon Corner of
AI & Society
.
[
3
]
[
16
]
The glossary definition discussed the full theory, while the opinion piece did not, focusing instead on AI-generated content and AI-driven Interactions. These two sources have been cited by other academic articles that discuss the topic.
[
17
]
[
18
]
[
25
]
[
27
]
In 2026, a publication in
Computer
built upon the
AI & Society
article by distinguishing between a "Leaner" version of the dead Internet theory, centered on the core evidence, and the "conspiracy-laden" full version.
[
4
]
The recent growth of interest in the dead Internet theory has been linked to increasing public awareness of bots, algorithmic content distribution, and advances in artificial intelligence.
[
28
]
With the increase in public awareness, a 2026 article identified that the dead Internet theory has been used colloquially to refer to the assumption that bots are creating the majority of online content.
[
5
]
The effect of increasing AI content on the internet, caused by bot activity, has been likened to the increase in
entropy
as driven by the
Second Law of Thermodynamics
.
[
29
]
A broken link usually leads to an error message. These broken links over time contribute to
link rot
.
The dead Internet theory has two main components: that organic human activity on the web has been displaced by bots and algorithmically curated search results, and that state actors are doing this in a coordinated effort to
manipulate the human population
.
[
3
]
[
30
]
[
31
]
The first part of the theory is described as the main argument, and the second part is where the conspiracy portion begins.
[
31
]
This first part, that bots create much of the content on the Internet and perhaps contribute more than organic human content, has been a concern for a while, with the original post by "IlluminatiPirate" citing the 2018 article "How Much of the Internet Is Fake? Turns Out, a Lot of It, Actually" in
New York
magazine.
[
2
]
[
30
]
[
32
]
The dead Internet theory goes on to include that search engines are censoring the Web by filtering content that is not desirable by limiting what is indexed and presented in search results.
[
3
]
This problem is exacerbated by the phenomenon known as
link rot
, which is caused when content at a website becomes unavailable, and all links to it on other sites break.
[
3
]
The second half of the dead Internet theory builds on this observable phenomenon by suggesting that this is part of a conspiracy—that the U.S. government, corporations, or other actors are intentionally limiting users to curated, and potentially artificial, AI-generated content to manipulate the human population for a variety of reasons.
[
2
]
[
3
]
[
30
]
[
31
]
In the original post, the idea that bots have displaced human content is described as the "setup", with the "thesis" of the theory itself focusing on the United States government being responsible for this, stating:
The U.S. government is engaging in an artificial intelligence-powered
gaslighting
of the entire world population.
[
2
]
[
6
]
A 2025 chapter in the book
Market-Oriented Disinformation Research
described the theory as having a "weak" and "strong" version.
[
25
]
The "weak" version of the theory asserts that there is a group of elites using bots to shape public discourse, while the "strong" version of the theory asserts that society itself collapsed because of some catastrophic event, and some entity (perhaps aliens or highly advanced
artificial intelligence
[
33
]
) is keeping people connected to the internet to disguise this reality.
[
25
]
In a 2026 article, Hal Berghel discussed what he called a "leaner" dead Internet theory, "stripped of paranoia, prejudice, politics and polemic."
[
4
]
In this discussion, Berghel points to the 2024 publication in
AI & Society
by Yoshija Walter, and lists algorithmically generated content, generative AI byproducts, the difficulty for some people to distinguish between those and human-generated content, and the resulting mistrust and misinformation as the core of the dead Internet theory. Berghel laments that conspiracy theorists take these phenomena and make implausible claims, while arguing that its core criticisms should not be dismissed.
[
4
]
In 2016, the security firm
Imperva
released a report on bot traffic after examining over 16.7 billion visits to 100,000 randomly selected domains, and found that automated programs were responsible for 52% of web traffic.
[
34
]
[
35
]
This report has been used as evidence in reports on the dead Internet theory.
[
2
]
Imperva's report for 2023 found that 49.6% of Internet traffic was automated, a 2% rise from 2022, which was partly attributed to artificial intelligence models
scraping the web
for training content.
[
36
]
A 2023 policy paper from the Institutul Diplomatic Român cited this increase in bot traffic as the basis for the dead Internet theory.
[
37
]
Fortune
reported that as of 2026 more than half of all website requests were generated by bots.
[
38
]
The late 2022 release of
ChatGPT
, an AI
chatbot
, by
OpenAI
to the general public led journalists to call the dead Internet theory potentially more realistic than before.
[
9
]
[
43
]
[
44
]
Before ChatGPT's release, the dead Internet theory mostly emphasized government organizations, corporations, and tech-literate individuals.
[
9
]
ChatGPT gives the average Internet user access to large language models.
[
43
]
[
44
]
This technology caused concern that the Internet would become filled with content created through the use of AI that would drown out organic human content.
[
45
]
[
43
]
[
44
]
[
46
]
[
22
]
In September 2025, OpenAI CEO
Sam Altman
posted on Twitter (by then called
X
), bringing attention to the dead Internet theory.
[
47
]
He stated that:
i never took the dead internet theory that seriously but it seems like there are really a lot of LLM-run twitter accounts now.
[
48
]
This post went viral and led to discussion about the impact of generative AI on society at large, including online experiences, human language, and education.
[
47
]
Proponents of the dead Internet theory argue that
Google
and other search engines are censoring the Web by filtering content that is not desirable by limiting what is indexed and presented in search results.
[
3
]
The belief is that while Google may suggest that there are millions of search results for a query, the results available to a user do not reflect that.
[
3
]
This problem is exacerbated by the phenomenon known as
link rot
, which is caused when content at a website becomes unavailable. All links to it on other sites break.
[
3
]
This has led to the theory that Google is a
Potemkin village
, and the searchable Web is much smaller than we are led to believe.
[
3
]
Proponents expand this to suggest it is an intentional effort to limit the content available to users.
In 2024, Google reported that its search results were being inundated with websites that "feel like they were created for search engines instead of people".
[
49
]
In correspondence with
Gizmodo
, a Google spokesperson acknowledged the role of
generative AI
in the rapid proliferation of such content and that it could displace more valuable human-made alternatives.
[
50
]
Bots using LLMs are anticipated to increase the amount of spam, and run the risk of creating a situation where bots interacting with each other create "self-replicating prompts" that result in loops only human users could disrupt.
[
45
]
In an article in
AI & Society
, Henrique Marcos discusses the possibility of LLMs impacting linguistic communities as they become more widespread in a scenario like the dead Internet theory.
[
51
]
In 2020, several
Twitter
accounts started posting tweets starting with the phrase "I hate texting" followed by an alternative activity, such as "i hate texting i just want to hold ur hand", or "i hate texting just come live with me".
[
2
]
These posts received tens of thousands of likes, many of which are suspected to be from
bot accounts
. Proponents of the dead Internet theory have used these accounts as an example.
[
2
]
[
24
]
An image posted on many subreddits as protest during the blackout
[
52
]
In the past, the
Reddit
website allowed free access to its
API
and data, which allowed users to employ third-party moderation apps and train AI in human interaction.
[
46
]
In 2023, the company
moved to charge for access to its user dataset
. Companies training AI are expected to continue to use this data to train future AI.
[
53
]
As LLMs such as ChatGPT become available to the general public, they are increasingly being employed on Reddit by users and bot accounts.
[
13
]
[
46
]
Professor
Toby Walsh
, a computer scientist at the University of New South Wales, said in an interview with
Business Insider
that training the next generation of AI on content created by previous generations could cause the content to suffer.
[
46
]
University of South Florida professor John Licato compared this situation of AI-generated web content flooding Reddit to the dead Internet theory.
[
46
]
In 2024, AI-generated images on
Facebook
, referred to as "
AI slop
", began going viral.
[
57
]
[
58
]
Subjects of these AI-generated images included flight attendants, black children next to artwork they supposedly created, and various iterations of "Shrimp Jesus", depictions of
Christ
"meshed in various forms" with shrimp. Many of these posts had hundreds or even thousands of comments saying "Amen".
[
8
]
[
59
]
The images were cited as an example of the Internet of the time having begun to feel "dead".
[
13
]
[
60
]
Sommerer discussed Shrimp Jesus in detail within his article as a symbol to represent the shift in the Internet, specifically stating:
Just as Jesus was supposedly the messenger for God, Shrimp Jesus is the messenger for the fatal system [we've] maneuvered ourselves into. Decoupled, proliferated, and in a state of exponential metastasis.
[
17
]
—
Thomas Sommerer
Facebook includes an option to provide AI-generated responses to group posts. Such responses appear if a user explicitly tags @MetaAI in a post, or if the post includes a question and no other users have responded to it within an hour.
[
61
]
In January 2025, interest renewed in the theory following statements from Meta on their plans to introduce new
AI-powered autonomous accounts
.
[
9
]
[
62
]
Connor Hayes, vice-president of product for generative AI at Meta stated, "We expect these AIs to actually, over time, exist on our platforms, kind of in the same way that accounts do
... They'll have bios and profile pictures and be able to generate and share content powered by AI on the platform."
[
63
]
These accounts were quickly removed.
[
according to whom?
]
SocialAI, an app created on September 18, 2024, by
Michael Sayman
, was designed to chat with only AI bots without human interaction. An article on the
Ars Technica
website linked SocialAI to the dead Internet theory.
[
65
]
On June 23, 2025, Alexis Ohanian, one of the
Reddit
co-founders, said he thought he "long subscribed to the dead Internet theory" ever since AI has started being able to pass the
Turing test
,
[
66
]
and on October 29, 2025 at TechCrunch Disrupt, Alexis reportedly told Kevin Rose, one of the original founders of
Digg
(a social media website originally created in 2004), "the dead internet theory is real", whilst Kevin said that he wanted to use
zero-knowledge proofs
to make a platform full of trusted users.
[
67
]
On January 14, 2026, Digg was relaunched in open beta by Alexis Ohanian and Kevin Rose, but was closed 2 months later on March 14 due to an "unprecedented bot problem" among other issues,
[
68
]
and on May 11, it was rebooted again as an AI news aggregator taking from
X
(
formerly called Twitter
).
[
69
]
Digg said on their homepage, that they are doing this because of the very bot problem that shut them down in the first place.
[
70
]
YouTube
is susceptible to fake views generated by computers, not human users,
[
72
]
and fake views were so prevalent that some engineers were concerned YouTube's algorithm for detecting them would begin to treat the fake views as default and start misclassifying real ones.
[
2
]
[
72
]
YouTube engineers coined the term "the Inversion" to describe this phenomenon.
[
32
]
[
22
]
[
72
]
YouTube bots and the fear of "the Inversion" were cited as support for the dead Internet theory in a thread on the Internet forum Melonland.
[
2
]
In 2024,
TikTok
began discussing offering virtual influencers to advertising agencies.
[
31
]
In a 2024 article in
Fast Company
, journalist
Michael Grothaus
linked this and other AI-generated content on social media to the dead Internet theory.
[
31
]
In this article, he referred to the content as "
AI slime
".
[
31
]
Research has shown that automated bot accounts make up a significant portion of internet traffic, though experts emphasize that this does not prove a coordinated effort to replace human activity online.
[
28
]
Critics argue that the theory is based largely on unverified observation, such as perceived declines in content quality or repetitive online interactions, rather than verifiable data. Researchers also note that many forms of bot activity, such as spam filtering,
Denial-of-service attacks
, and web scraping, serve specific functions, sometimes maliciously, but do not indicate in any way that human participation on the internet is being replaced.
[
73
]
Researchers have also compared the dead Internet theory to other internet-related
conspiracy theories
and note that these theories often stretch current trends, like artificial intelligence and increased bot activity, into speculative claims without using concrete evidence.
[
74
]
Research discussing the dead Internet theory has focused on several issues identified in the claims of the theory without discussing the speculative aspects, and colloquially researchers and the public have used it to refer to these issues without the conspiracy aspects.
[
4
]
[
5
]
Caroline Busta, founder of the media platform
New Models
, was quoted in a 2021 article in
The Atlantic
calling the conspiratorial aspect of the dead Internet theory a "paranoid fantasy", despite acknowledging that there are legitimate criticisms involving bot traffic and the integrity of the Internet, and agreeing with the "overarching idea".
[
2
]
A 2021
Ouest-France
article, which heavily referenced the 2021 article in
The Atlantic,
went on to compare the dead Internet theory to
flat Earth
and
COVID-19
conspiracy theories.
[
75
]
The article stated that even though bots do produce online content, the dead Internet theory is still not realistic.
[
75
]
In 2023 in
The New Atlantis
, Robert Mariani called the theory a mix between a genuine conspiracy theory and a
creepypasta
.
[
6
]
A 2023 book published by
CRC Press
discussed the dead Internet theory, specifically mentioning
Google
censoring the web.
[
3
]
The book included an entry for the term in its glossary defining it:
The Dead Internet Theory is a conspiracy theory that suggests the Internet has died and that much of the content we see online is now artificially generated by AI to manipulate the world population. The theory raises concerns about the impact of AI on propaganda, art, and journalism.
[
3
]
Like all good conspiracy theories, the Dead Internet Theory takes a kernel of truth or agreed sentiment (that the internet is getting worse, and that bot activity is increasing) and twists it into something it isn't.
In 2024, "dead Internet theory" was sometimes used to refer to the observable increase in content generated via
large language models
(LLMs) such as
ChatGPT
appearing in popular Internet spaces without mention of the full theory.
[
11
]
[
16
]
[
17
]
[
18
]
[
36
]
In a 2024 opinion column in
AI & Society
'
s "Curmudgeon Corner", Yoshija Walter stated that the once speculative theory is now observable with the introduction of AI generated content.
[
16
]
In a 2025 article by Thomas Sommerer, this portion of the dead Internet theory is explored, with Sommerer citing Walter and calling the displacement of human generated content with artificial content "an inevitable event".
[
17
]
Sommerer states the dead Internet theory is not scientific in nature, but reflects the public perception of the Internet.
[
17
]
Another article in the
Journal of Cancer Education
discussed the impact of the perception of the dead Internet theory in online cancer support forums, specifically focusing on the psychological impact on patients who find that support is coming from an LLM rather than a genuine human.
[
18
]
The article cited both Walter and the CRC Press book when defining the dead internet theory, but did not mention the conspiracy aspect. The article also discussed potential problems in training data for LLMs that could emerge from using AI-generated content to train the LLMs.
[
18
]
In a 2025 paper, Roland Leikauf described the dead Internet theory as "pseudoscientific" while questioning if new AI tools would justify our fear that the theory might become reality.
[
27
]
Leikauf cites Walter's 2024 publication for his definition of the dead Internet theory.
[
27
]
In a chapter of the 2025 book
Market-Oriented Disinformation Research,
it states:
What makes the Dead Internet a nameworthy conspiracy is that even though it is rooted in selective truths that are exaggerated or even taken to their logical extremes, it also draws attention to a legitimate problem.
In a 2025 interview with
Time
, linguist
Adam Aleksic
stated that the dead Internet theory "used to be a lunatic fringe conspiracy theory, but it's looking a lot more real".
[
10
]
In a 2026 paper published in
Computer
,
Hal Berghel discusses a "leaner" version of the theory, without the conspiratorial elements, focusing on the core claims, specifically referencing the 2024 paper by Yoshija Walter.
[
4
]
In this paper, he states:
We must admit that some of the core principles of the DIT are convergent with our technical and historical experience. Unfortunately, conspiracy theorists augment these very plausible observations with their own mix of biases and agendas, leading to implausibility and absurdity, which in turn leads to rejection. But it is a mistake of the first order to dismiss the core criticisms unequivocally.
The dead Internet theory has been discussed among users of the social media platform
X
(formerly known as
Twitter
).
[
47
]
Users have noted that bot activity has affected their experience.
[
2
]
Numerous YouTube channels and online communities, including the
Linus Tech Tips
forums and the
Joe Rogan
subreddit, have covered the dead Internet theory, bringing the idea into mainstream discourse.
[
2
]
The DIT is often used colloquially to refer to the issue surrounding bots online in popular culture, without reference to the broader conspiracy theory.
[
5
]
1
2
3
4
5
Lapowsky, Issie (May 8, 2025).
"Maybe Al Slop Is Killing the Internet, After All"
.
Bloomberg News
. Retrieved
August 26,
2025
.
The dead internet theory became popular in 2021, following a post from a user named IlluminatiPirate on an obscure online forum. IlluminatiPirate argued that the internet
[
sic
]
had become a vast, inhuman wasteland, filled with algorithmically optimized copycat posts. The theory blamed the entire thing on a covert government conspiracy, which made it easy to dismiss. But the arrival of tools such as ChatGPT and Midjourney has made it look downright prophetic. Social media feels weirder. Search feels worse. Entire AI-generated news networks have sprung up overnight. Meta Platforms Inc. envisions a future in which AI is involved in creating a substantial share of posts on Facebook and Instagram. Sites such as Wikipedia are straining under the weight of AI crawlers that root around their pages, searching for fresh information to feed their models. All of this is creating a feedback loop, where AI-generated content is being produced to please AI-powered recommendation systems, threatening to turn humans into bystanders.
1
2
Knibbs, Kate (October 28, 2024).
"AI Slop Is Flooding Medium"
.
Wired
. Retrieved
February 14,
2026
.
It also suggests a future in which the Dead Internet theory comes to fruition. The theory, once the domain of
extremely online
conspiratorial thinkers, argues that the vast majority of the internet is devoid of real people and human-created posts, instead clogged with AI-generated slop and bots. As generative AI tools grow more commonplace, platforms that give up on trying to blot out bots will incubate an online world in which work created by humans becomes increasingly harder to find on platforms swamped by AI.
1
2
3
4
5
Ball, James (September 14, 2024).
"Is anyone out there? — Dead Internet Theory says that you're the only human left online. It started out as a conspiratorial joke, but it is edging ever closer to reality"
.
Prospect
. Retrieved
February 14,
2026
.
This is the idea at the core of what became known as Dead Internet Theory, a joke-cum-conspiracy that says if you're reading these words online, you're the last person on the internet. Everyone else is a bot. [...] The idea began to gain traction almost a decade ago, with the "time of death" of the internet typically given as being around 2015 or 2016—but in the years since, reality has begun to mirror this once unserious conspiracy. The complaint of the modern internet is that it is filled with 'slop' content, the spiritual successor to email spam.
↑
Vladisavljević, Radovan; Stojković, Predrag; Marković, Svetlana; Krstić, Tamara (2023). "New challenges of formulating a company's marketing strategy based on social network analysis". In Premović, Jelena (ed.).
Challenges of modern economy and society through the prism of green economy and sustainable development
. Educational and business center for development of human resources, management, and sustainable development. pp.
374–
380.
ISBN
978-86-81506-23-3
.
↑
"Digg - AI news, before it trends"
.
Digg
. May 11, 2026
. Retrieved
May 18,
2026
.
Why this approach? Bots are everywhere now - the real question is who's human, and which humans to trust. So we mapped 9 million connections, found the 2,000 voices that move first, and surface what they touch before it spreads. Kick the tires, AI news first, more topics coming soon.
Recently, Anthropic wrote a blog post about their new
centralized version control system Scion
. As someone working on the Source Control team at Databricks, where we have a huge monorepo and are exploring different options, I figured it might be useful to share my perspective here.
(Obligatory disclaimer: This post represents my own views, not my employer’s.)
Before we get into the weeds, it might be helpful to you to know where I’m coming from.
If you’re not interested in this, you can jump to the
Technical Commentary
section.
I have never worked at Facebook, but one of my all-time favorite tech talks is Durham Goode’s
Scaling Mercurial at Facebook
from Git Merge 2017.
Goode was the longest-serving and most senior member on Facebook’s Source Control team. (
source
)
The talk covers various interesting aspects for both the server side and the client side. For the server side, some of the topics include solving push contention with server-side rebasing (pushrebase), using scalable object storage for commits, moving various source control queries to the server, etc. For the client, there’s mention of the smartlog and other CLI commands for easier manipulation of history (compared to rebasing in Git).
However, more so than the technical aspects of the talk (which are interesting in their own right!), the main thing that has stayed with me throughout these years is the
deep care for users
that is reflected in Goode’s tone and words. It’s a bit difficult to convey this without the power of voice,
You should watch the presentation!
but here are some quotes which reflect this:
In the context of
hg smartlog
:
hg smartlog
gives you a focused view of
your
branches and how they relate to the development trunk.
I know Git can produce this kind of UI if you know the incantation, but the critical thing here is not that it can be done; it’s that it becomes the default view of their source-control environment. That is where the power comes from: making it really intuitive for users to use source control and understand the concepts without reading a man page.
In the context of
hg absorb
:
hg absorb
auto-magically identifies the right commits in a stack to squash fixes into and rewrites history accordingly. This makes it easier to work with stacked changes.
This is the type of UI that we need to be inventing for our users: ones that match their intent and do it to the best of their ability, without making the user understand everything they have to do. It’s a little bit magic, but it’s magic in a very clean way. I think a lot of user experiences could benefit from this approach.
On recovering from mistakes:
If you want to undo an amend or undo parts of a commit in Mercurial, you just run unamend and uncommit. You don’t have to understand a reflog; you don’t have to get a PhD to understand reset anymore. It just works, and it gives users confidence that they can recover from mistakes. That’s one of the most important things we can do: provide user experiences that give them confidence in what they’re doing.
Near the end, summing things up:
On the user experience side, we as a source control community need to do a better job of building UIs that are accessible, intuitive, and that enable novice users to be power users. Everything I just talked about is how we are trying to do that for our users at Facebook. If the Git community could do that, the massive number of Git users taking advantage of it has the potential to redefine productivity for an entire generation of users.
Over the years, the possibility of these workflows lingered in my mind as I continued working with Git. I tried building my own wrappers on top of the
git
CLI to do some of these things, but I couldn’t get the performance to be reasonable since everything shelled out to
git
all the time, and I didn’t understand Git’s internals well enough to do something better.
I ended up sticking with vanilla Git until Waleed Khan
Waleed worked at Facebook on dev tools, and later did work at Twitter related to version control.
created
git-branchless
, with support for a smartlog, in-memory rebases, undo and more. I eagerly adopted it. I also used Stephen Jung’s
git-absorb
quite a bit. Even today, the initial bit of
git absorb
’s README cites the mention of
hg absorb
on a Mozilla mailing list as the key inspiration.
Finally, earlier this year (2026), I mostly switched to using
jj
, after having bounced off of it a couple of times. jj was created by Martin von Zweigbergk, who previously contributed to Git and later joined Google and worked on their internal Mercurial client for the monorepo.
Source:
Mercurial at Google (2023)
.
So even though I have almost exclusively worked with Git throughout my career, I am in a way indebted to the work on Mercurial by Goode and numerous other contributors.
I’ve talked a bit about my personal usage of version control tooling. Let me briefly describe work I’ve done related to version control.
From late 2021 to late 2025, I worked at Sourcegraph on code indexing and navigation. For the last two of those years, I worked more on the backend written in Go, where the code navigation was coupled to version control.
This is because code navigation data could be uploaded for ~any commit.
During this time, one of the recurring sources of incidents was the (non-)incremental updates for the commit graph structure that was mirrored into Postgres, in order to be able to efficiently compute what code navigation data (stored in Postgres) was “visible” at which commit.
I recall being frustrated with that, and half-jokingly proposing that it would be good to move to a more custom storage solution or have custom indexes next to Git’s storage itself. That proposal didn’t particularly go anywhere (not that I expected it to), but the frustration with Git as a hard-to-operate database stayed with me.
Since around March 2026, I’ve been working on Source Control at Databricks. Some of the things I’ve worked on include:
Enforcing consistent Git configurations across our devboxes.
Creating various safety nets to prevent agents from messing up Git repos in ways that trigger pathological performance.
Root-causing and fixing various client-side Git performance issues.
So in a way, I’ve been ~responsible for monitoring and maintaining the health of thousands of git checkouts (of our huge internal monorepo
Depending on the specifics of compression, the history of the
master
branch ends up in the range of 20-50GB.
) for the past few months. (Life lesson: don’t get upset at a database for being hard-to-operate lest the monkey’s paw curls? 😄)
Apart from my main work, I have spent dozens of hours studying jj,
Mononoke
(the open-source server for Facebook’s VCS),
TigerBeetle
and, to a lesser extent, RocksDB and papers on LSM trees. I also prototyped a minimal code storage server in
Tiger Style
.
Over this time, I was also fortunate enough to attend Git Merge 2026.
Interestingly enough, the sponsors included OpenAI, Google, GitHub, Cursor, etc., and the attendees included a bunch of employees from Meta who work on Sapling/Mononoke, but no Anthropic employees were present AFAICT.
Whew, that was a lot of words! At least, I hope that this helps you understand where I’m coming from when I write the points below.
This section covers what I think of the overall design and development processes that have been described in the post on Scion.
The familiar bits
Having studied Sapling/Mononoke, and having spent some time thinking about how to build a version control server, the following bits felt obvious
I understand that the post is written for a broader audience, so it makes sense to repeat things which may be “obvious” to people who work in the area. That said, I think there’s value in separating out/clarifying to readers what is common knowledge within a given area vs what is genuinely novel. For example, if you were writing an academic paper, common knowledge would go into a Background section, not into the later sections.
(and hence not worth emphasizing) and/or copied (without clear attribution):
Representing commits with changesets instead of
Git-style trees
: Copied from Mononoke.
Attribution for this was added in a later update to the post after Rain’s blog hit the front page of HN.
Client-server asymmetry: No huge monorepo that I know of actually has symmetry here. Even with Git, there are certain configuration options which are only meant for the client or the server.
APIs for version control: This is a natural consequence of client-server asymmetry.
Horizontal scalability: For a VCS server, horizontal scalability largely comes for free because it’s borrowed from the storage for the objects.
Specifically, one of the things which makes caching tricky is handling cache invalidation, but that doesn’t apply to a content-addressed store.
“Commits must be cheap, incremental, and lock-free”: Once you’ve decided “commits are immutable content-addressed objects,” this falls out naturally.
“The working copy is optional, and sparse by design”: At this scale, nobody materializes millions of files on disk all the time everywhere.
Per-directory ACLs: These are supported by Perforce, Piper and Mononoke.
No merge queue with full CI:
I’m reading between the lines here – the wording in the post “What does CI even mean when hundreds or even thousands of changes can merge while tests are running? We haven’t been able to effectively replicate the industry standard of a global repository lock on the merge queue” doesn’t make it clear what exactly it is referring to.
Mononoke does the same. At a smaller scale, Uber’s
SubmitQueue
also does not run full CI.
The surprising/novel bits
The speed of migration
In order to support this growth, we built Scion, an in-house version control system, and fully migrated our monorepo from git in five months. We also built and shipped a code review surface (called Antique), code search, a rich source control IAM system, novel ways of managing and landing large amounts of reviewed code, and more, all in that same time period.
This was done by a team of 7 people and an undisclosed number of Claude tokens.
On reading this, my first thought was, “That’s still quite a lot of work.” With Git, it’s very easy for people (or agents) to come up with complex operations to do specific things, which can be difficult to easily map to other VCSes as-is. So it’s impressive that they managed to rip out Git fully regardless.
It’s not 100% clear in the post if they’ve done so, but I’m assuming this based on the wording in the post.
I’m guessing that one or more of the following are true: (1) their monorepo lacks a file-based OWNERS system (2) the OWNERS system largely goes through with automated approvals from Claude (and hence does not bottleneck on human approvals) or (3) they used some targeted bypasses to go in and change people’s code to use their custom
scn
client instead of
git
.
“Superfan” Claudes
we created Slack channels where we tasked Claude with being obsessively focused on the health of a single component. We call these Claudes “superfans.” We have a scion-daemon superfan, a scion-cache superfan, an Antique merge queue superfan [..] They have access to our telemetry and logs, and act like an experienced engineer who has nothing better to do than to refresh charts and look for anything strange. We also have traditional alerts for all of our systems, but our superfans can detect subtle and novel issues that make these alerts seem one-dimensional by comparison.
This is a pattern which I (personally) have not heard of being deployed elsewhere, presumably in large part due to the cost of frontier-model tokens. I’m guessing this will become more widespread as token prices fall.
It would’ve been nice if they’d given some examples of what “subtle and novel issues” these superfans found.
The odd bits
Discouraging refs in the dev loop
The development lifecycle in Scion discourages refs (pointers to commits, such as branches and tags), except for the main ref or release branches, because global state that every server has to agree on is a looming scaling problem.
This point didn’t quite add up for me. The amount of data associated with a ref is very small – it’s the ref’s name and the ID of the object it points to. If you separate out temporary refs (for landing/organizing work) and long-lived refs (for release branches), then temporary refs can be automatically cleaned up and would be proportional to the number of changes in flight. The long-lived refs would keep adding up anyway (and potentially contribute to whatever scaling problem is anticipated).
So how does not using refs in the dev loop help?
I can see a downstream
correctness
argument for avoiding refs, where it’s easy to accidentally write a job which (if it works with refs) accidentally pulls in two different commits (as the ref moves) instead of detecting ref movement and triggering failure/retry. However, it’s also easy to make a different kind of mistake just working with commit IDs – you could keep running the job for a commit that has become unreachable due to a push.
Inefficiencies in resource utilization
There are several points in the article related to resource utilization which didn’t add up in my head.
Beyond these, there are standard workarounds for large monorepos, like internal git mirrors and repository snapshots (seeds) kept in blob storage. We’ve used all of them—at our peak, we operated several hundred internal git mirrors to deal with the load.
At Databricks, we do something similar with reference repos which are updated every ~2 hours, and we have fewer than 10 git mirrors to deal with CI load, despite the repo size being within the ~same order of magnitude. Even with this many mirrors, the CPU load is well under control, and increasing the frequency of snapshots from 2h to, say, 1h or even 30min – which would reduce CPU load on the mirrors – would be doable with a bit of work.
So it’s difficult for me to think of what kind of workload would necessitate “several hundred” git mirrors. Maybe very wide fan-outs for tests in CI?
Similarly, there’s another point about maintaining the mirrors later on:
At our peak, we ran several hundred git mirrors, which significantly strained capacity and no longer scaled once the repo outgrew the mirrors’ disks or memory limits
It’s unclear how a repo with 100GB of history on the main branch could outgrow the disks on modern cloud instances, which can have several TB of storage. Yes, it requires some level of care wrt. maintenance, but disk limits and memory limits should generally not be the major problems here. Memory in particular is even more odd, because most accesses will likely be for the most recent data, which can easily stay resident in a few GB of RAM at most.
Performance numbers
Many of the pre-migration numbers provided on the chart (which make the Scion numbers look impressive) seem way too high, and some of the ones post-migration also seem unusually high. As one example, the charts show that the p50 for ‘pull latest’ fell from 87s → 3.5s. Each of these feels off for different reasons.
This is one of the places where the post – by going wide instead of deep – makes it difficult to understand what’s actually being said. For example, percentiles can easily be skewed downwards due to automated background loops (esp. those running at high frequency) if most operations short-circuit to no-ops.
Pre-migration: At our scale, fetching the changes at the start of a day can take over a minute, but after that, fetches are much faster. So even the p90 for fetch operations ends up being under 1 minute. Granted, “fetch” and “pull latest for the trunk” are not equivalent operations. However, I’m not sure what it would take for things to be so bad for the p50 to be at 87s if you do background pre-fetching once a devbox has booted up.
Post-migration: 3.5s seems surprisingly high, given everything else. With sparse checkouts, presumably this number should be proportional to the size of the diff within that subset between pull operations. For example, naively, I’d expect network to be the limiting factor: even at 2Gbps
GCP doesn’t appear to have
ingress bandwidth limits
for instances within a VPC, so I’m using the “2Gbps per vCPU” (for egress) as a rough guide/underestimate of available bandwidth.
you could transfer 875MB in 3.5s. I can’t imagine the amount of data being transferred for a p50 “pull latest” being even close to that, so there must be something else that’s the bottleneck. Maybe the query for determining what data should be pulled is the slowest bit? Too many RPCs? Serialization/deserialization overhead? 🤷
Squatting on the shoulders of giants
Before we explore what we built, we should acknowledge how unusual our workload is. Unsurprisingly, Anthropic uses Claude to write software far more than almost any other company, so we’re often the first to see where tools built for human scale start to strain. Source control is just one example of this. While we don’t expect many organizations to have these problems in the very near term, we suspect it won’t be too long before others who use AI coding tools experience these same pressure points, which is why we wanted to share what we built and what we learned.
The vibe of this paragraph captures the vibe of the rest of the article. It is simultaneously:
Factually inaccurate: Even pre-LLMs, Google, Facebook and others have run into scaling issues with source control and solved them in different ways.
Self-congratulatory: This one needs no explanation.
Special shout-out to “A git server doesn’t have to work the same way as a git client does; novel server architectures exist that provide significant scaling improvements.” – Erm, what exactly does Anthropic think that GitHub, GitLab, Cursor, Entire, ERSC etc. have been working on since the start of 2026?
Overstating the contribution: “share what we built and what we learned”, except that there’s very little of substance that’s not a copy of something that’s already available. And for Mononoke, the entire source code is available to study, not just some blog posts.
It is always good to situate your work in the larger context. We all stand on the shoulders of giants! There is nothing wrong with borrowing and adapting existing work to new contexts — that is the vast majority of what I do as an engineer. When you position your work as derivative, you are acknowledging those that came before you, as you hope to be acknowledged by those who come after you.
Given how they’ve framed their work and how they’ve recognized prior work only with cursory nods (this changed somewhat after they got called out for it!), I’m now much less surprised by their absence at Git Merge 2026. Historically, companies and developers have joined communities to learn from other people’s expertise, and to be able to recruit people interested in the same areas.
Now that companies can increasingly point LLMs to clone your codebase, it is understandable that some of them no longer feel the need to engage with the corresponding communities in the same manner.
Words from nowhere for nobody
One of the major problems with the blog post is that it is unclear as to what primary purpose it is meant to serve and who the target audience is.
Is it meant as a doc for organizations to encourage them to build more ambitious projects (like a version control system) in-house with Claude?
If so, it seems a bit too modest, and has too many technical details.
Is it meant as a recruiting pitch? (Come join us, we’ll give you a ton of free tokens and you get to work with smart people?)
I guess it technically got to the front page of Hacker News, so it served that purpose to some extent.
Is it meant to convey some cool factor/novelty/innovation in Source Control?
If so, it fails to place the work in the larger context of source control for monorepos as it has evolved over the years.
Perhaps because of the lack of a clear audience in mind, even though the post was (thankfully!) not LLM-written, many parts of it suffer from LLM-isms like:
Stating the obvious: See the many points in earlier sections.
Meaningless comparisons: There are many throughout the post, but here’s one example. “Fetched naively, that would be over 600 PB of git traffic per day.” – here, the 600 PB number is based on 25 million checkouts × 25 GB per compressed copy of the checkout. Given that most clients already have a copy of the repo, this is not a meaningful baseline to compare against unless the daily churn and growth rates add up to close to 100%.
Going broad instead of deep: Many charts and numbers are provided without sufficient context, used as window dressing, and discussed briefly; then the post moves on to the next thing.
As a concrete example, there are multiple mentions of the growth in PR counts, but there’s no separation between automation PRs and human-in-the-loop PRs. One can try inferring this from the charts (e.g. the Sept data shows about 10K PRs/day landing on weekends), but that still leaves a lot of open questions. For example, given the jump from 15K PRs on Sept 7 → 25K PRs on Sept 8 → 30K PRs on Sept 9, how many of the 30K PRs on Sept 9 are from runaway automations?
What about the user experience?
Historically, using version control has been a fairly human-centric activity. While you might have a sprinkling of deterministic automations, such as for bumping dependencies, reverting PRs which introduced regressions, etc., the main user was the developer.
As increasingly more companies use LLM agents for increasingly more parts of the development lifecycle, it is understandable that LLMs become the primary “user” for a VCS, and the developer is abstracted away from the VCS. In such a world, if you’re focused on the LLM as the “user”, you could ask questions about metrics like error rates/undo rates/latency for tool calls.
If you take a broader view and are interested in the end-to-end developer experience, you might still care about questions such as “what are the problems people face when trying to use the VCS via LLMs”, as well as “which operations are slow but frequent, and could be replaced with fast and deterministic tooling instead.”
But the post really does not touch upon either of those points, and overall ends up having no clear narrative arc as a result.
The post starts out with a relatively vague motivation:
the latent demand for merging code is practically infinite, and every bottleneck we remove immediately leads to a further increase in the amount of code we produce. In order to support this growth, we built Scion [..]
However, near the end of the post, the wins are stated in terms of reduced CPU utilization (which translates to increased capacity for other things) and simplified systems (I understand why this is a win, but it seems like this was not the original motivation for pursuing this path).
I understand that the new VCS is able to support the current growth, but the implied thesis that the VCS was the bottleneck and removing that bottleneck actually unlocked the growth lacks compelling evidence in the post – one is left to wonder how much of the growth is attributable to increased background automations vs. model changes vs. other factors.
For example, were people frequently blocked from landing PRs
because
CI was at capacity and runs remained queued for a long time? If so, did freeing up capacity help reduce the time from PR creation to merging, which in turn led to more PRs being created? Maybe! But the reader is basically left to just guess all of the intermediate details without a clear connecting thread.
Closing thoughts
When I first started reading the post, I went in with high expectations. After all, it has historically been rare for companies to build their own version control systems, and given how much money and talent Anthropic have at their disposal, I thought the post would’ve been a lot more interesting than it actually turned out to be.
Overall, had Anthropic clearly stated up front what parts of the design were copied from other systems with proper attribution, and focused the post on how they managed the migration, I think the post would’ve been more honest, more streamlined and also more useful to people outside Anthropic. Specifically:
For the migration: sharing things like how they handled dependencies between various bits of work, making changes to non-centrally-owned tools, decoupling things, client libraries, project management, etc. – that could’ve potentially been useful.
For the design:
Publishing RPC interfaces, database schemas and/or pseudocode for any of the interesting algorithms.
Discussing alternative designs that were considered and rejected along with reasons.
The post does none of that,
It’s understandable that they wouldn’t want to open-source the code itself, as untangling it from the rest of the monorepo and having development in multiple places would add friction for themselves for unclear benefit.
presents several previously established ideas as seemingly original without a clear separation between the two, and frames the high-level description as a “gift” to the developer ecosystem/Source Control community.
.
Thank you Polyend!!
Subscribe to comments on this entry
I purchased a memory card and flashed Raspberry Pi OS
lite onto it using the raspberry pi imager tool.
Or
take the example of newspapers. The papers were revolutionized by
productivity gains starting with the arrival of digital layout tools.
The industry employed
:quality(85)/https://andros.dev/media/blog/2026/10/tyrtulia-tui-thread.png)
:quality(85)/https://andros.dev/media/blog/2026/10/tyrtulia-el-thread.png)
:quality(85)/https://andros.dev/media/blog/2026/10/tyrtulia-web-thread.png)
:quality(85)/https://andros.dev/media/blog/2026/10/tyrtulia-el-forums.png)
was. First, there was the childhood fantasy of finding a formula that Euler missed. But soon one learns more: how Grothendieck’s period conjecture, together with a sufficiently fine understanding of the geometry of mixed Tate motives (following work of Don Zagier, Sasha Goncharov, Francis Brown, and others) gives a completely satisfying structural, conjectural explanation of what’s going on. The irrationality of
years not because of its conclusion, but for how it reveals the tiniest hints of some still secret story relating algebraic
-functions to arithmetic.
, there are many commensurability classes of compact arithmetic hyperbolic three-manifolds indexed by non-split quaternion algebras
. Manifolds
in each of these commensurability classes give rise to elements in
via the decomposition of
. I have always had this idea that one could conceive of elements in these different commensurability classes as forming some type of geometric Kolyvagin system (“Heegner points”) associated to
, this was a contribution to mathematics. Factoring the same number in
by hand would be an equally impressive feat of arithmetic, but its contribution to mathematics (even if it had not previously been known) would be far less, since this can be done on a computer algebra system in far less than a second. It’s obvious that AI is changing the economics of what is “easy” or “hard” — much has already been said about the coming collapse of the theorem economy. In time (perhaps quickly) there will emerge a new sense of what remains a substantial contribution to mathematics, whether it be asking interesting questions, developing new programs, or something completely different. In the future, I see no reason to reward a paper based
purely
on a result that could be produced by machine, whether it was or not. A new proof can still deserve substantial credit for the ideas it introduces, even when its conclusion is already obtainable by machine. So why not start now? Importantly, I do not think this in any way means that access to AI will be required to make progress in mathematics — as long as we continue to value human understanding. While I feel that this is the most controversial recommendation, I also feel it is the most inevitable. It requires relinquishing an aspect of our reward structure that has been around for a long time, but at the same time a structure that will make no sense in the future. Right now, the use of AI carries a stigma, and the phrasing of many AI disclosures reads more like an attempt to minimize the contribution of the machine. If we evaluate every paper as potentially AI-assisted, then we must insist that its authors explain why the mathematics is interesting, what questions it raises, and how it connects to the broader subject. Don’t we want people to be doing this anyway?
_(3x4_cropped_on_Altman).jpg)
_(cropped).png)
Comments (0) Trackbacks (0) Leave a comment Trackback