The Trump administration, led by a President who was previously banned from major social networks for inciting violence and spreading disinformation after the 2020 US election, poses a particular challenge for the upstart platform Bluesky.
As Erin Kissane noted
in a recent article in Tech Policy Press, Bluesky was designed for openness and interoperability, yet it now finds itself as a single point of pressure. If it enforces
its rules against harassment and incitement
against official Trump administration accounts for some future infraction, it risks
political retaliation.
If it weakens its rules or shies away from enforcement, it may lose the trust of the communities who turned to the network for protection from coordinated abuse.
Composable moderation, which decentralizes rule-setting by letting users pick the moderation services that best reflect their needs and values, mitigates this problem. It shifts enforcement away from a single platform and into a distributed ecosystem of user-selected moderation services. With no central referee to target, political actors and influencers lose the ability to “work the refs” and pressure a singular trust and safety team into making decisions that favor their side.
Spreading the burden of moderation
“Bluesky the app” is the company’s shorthand for distinguishing its consumer-facing social app from the AT Protocol, the decentralized social networking protocol it is building. The app is just one client in what is intended to become a broader ecosystem of services built on the protocol. For now, however, Bluesky the company still carries the full responsibility for moderation and governance across the AT Protocol.
Centralized governance of a decentralized protocol cannot withstand sustained political or social pressure. When one company sets moderation rules for a network that is meant to be open and distributed, it becomes a single point of influence that governments, interest groups and powerful users can target. As AT Protocol’s
Ethos
statement
makes clear, its long-term vision sits at the intersection of three movements: the early web’s open publishing model, the peer-to-peer push for self-certifying and decentralized data, and the large-scale distributed systems that underpin modern internet services.
Bluesky’s goal is for AT Protocol to embody the openness of the web, the user-control of peer-to-peer networks, and the performance of modern platforms. In the future, we could see photo-sharing apps, community forums, research tools and more all using the same identities and social graph. Bluesky is only one expression of the protocol, not the limit of it.
Composable moderation is the feature that will make that possible. Rather than treating moderation as a network-wide ban, it uses labels to describe issues with content or accounts, leaving individual apps to decide how to act on them. Following
a letter
from Daphne Keller, Martin Husovec, and my colleague Mallory Knodel,
Bluesky has committed
to this approach.
Instead of blocking someone in a way that removes them from every app built on the protocol, Bluesky will mark a suspended account with a label that only affects how that account appears inside Bluesky. Other apps can choose to hide the account, show it with a warning, or ignore the label entirely. This also keeps the user’s underlying account intact, because it’s stored on their personal data server or PDS, the place where their identity and posts live, which should only cut someone off for serious issues like illegal content. The result is a more flexible, decentralized system where no single app controls whether someone exists on the network.
Why this approach solves the potential Trump problem
The closest analogy to existing social media is to how Reddit operates: the platform sets a baseline of what is acceptable, but thousands of subreddit communities apply their own rules, filters, and enforcement styles on top. For example, r/AskHistorians expects in-depth, well-sourced answers that reflect current academic research, and moderators routinely remove unsourced or speculative replies that don’t meet that standard. Composable moderation takes that layered, community-defined model and implements it at the protocol level, so many different apps and services can choose the moderation approaches that fit their values.
And because moderation could be provided by many different apps and services, not just Bluesky, it would reduce the political vulnerability that comes from having a single company responsible for every enforcement call. Communities can also choose moderation services that reflect their own context and needs, giving vulnerable groups more control over the protections they rely on. And if one app or operator fails or comes under political pressure, others can continue enforcing their own standards without breaking the network.
Taken together, this shift could help Bluesky, and future AT Protocol services, navigate the pressures Kissane highlights, distributing power across the network rather than concentrating it in a single company.
Support the
Exchange Point
— Your Donation Helps Us Build a Feminist Future
If you love our work and want to power more of it, here are the ways to support EXP and
our projects
:
🔐 Protect digital rights with a tax-deductible donation.
Give directly through
PayPal
(tax deductible in the U.S.). ➡️
https://exchangepoint.tech/donations
🌱 Double your impact with employer matching.
Most tech companies match donations through
Benevity
— search “Exchange Point.” Here are three projects to support with matching donations: ➡️ Social Web ➡️ Human Rights and Standards ➡️ Protect E2EE
📅 Sustain the movement with a large or recurring monthly gift.
Become a monthly supporter and help us plan long-term. ➡️ Please email
grants@exchangepoint.tech
.
🛠️ Fund the work through contracts or sponsored projects.
Partner with us on research, workshops, audits, or ecosystem strategy. ➡️
Work with us
!
🩵 Support our sister effort on the Free Our Feeds campaign.
Chip in through the FOF community GoFundMe. ➡️
https://gofund.me/1ef4d5d5d
Thank you for helping us build an open, joyful, people-powered internet.
The Internet still depends on fragile paperwork to prove who controls an IP address. The Internet Society explores how a new tool, the RPKI Signed Checklist, can use cryptographic proof to replace these weak systems and make routing more secure.
https://www.arin.net/blog/2025/11/19/2024-grant-report-isoc
Apple and Google’s Play stores shape what apps are available to most people as they use the Internet. When those app stores block or limit apps based on government requests, they are shaping what people can do, say, communicate, and experience, says ACLU’s Daniel Kahn Gillmor.
https://www.aclu.org/news/free-speech/app-store-oligopoly
Podcast: American Prestige. Silicon Valley and the Israeli Occupation, featuring Omar Zahzah, Assistant Professor of Arab Muslim Ethnicities and Diasporas Studies at San Francisco State University, discussing his book Terms of Servitude: Zionism, Silicon Valley, and Digital Settler Colonialism.
https://americanprestigepod.com/episodes/4215556855
Airbnb.org is offering emergency housing to families displaced by disasters, and any donation made before 31 December will be matched to double the number of nights provided.
https://www.airbnb.org/giftanight
The Arab Center for Social Media Development (ACSD) is pleased to announce the opening of proposals for workshops and sessions at the Palestine Digital Activism Forum 2026, Forms are due
December 15.
https://pdaf.net/invitation
The
Dolt Workbench
is an open-source SQL workbench supporting MySQL, Postgres,
Dolt
, and
Doltgres
databases. We built the workbench using
Electron
, which is a popular framework that allows you to convert web apps built with traditional web technologies like HTML, CSS, and Javascript to desktop applications. Since the workbench shares much in common with
DoltHub
and
Hosted Dolt
, the architecture is very similar to those products. That is, the workbench uses
Next.js
for the frontend with an additional GraphQL layer that handles database interactions. For this reason, it made a lot of sense to use Electron to get the desktop version of our application up and running.
That said, Electron comes with a few rather significant drawbacks, and those drawbacks have started to become more apparent as the workbench has matured. Because of this, I spent some time exploring
Tauri
, a newer framework that supports the same web-to-desktop use case as Electron. In this article, we’ll discuss how well Electron and Tauri integrate with the workbench, and weigh some pros and cons between the two frameworks.
Next.js doesn’t translate very cleanly to a desktop application context. This is primarily due to the framework’s architecture around server-side rendering and API routing features. In a desktop app, there’s no application server interacting with a client; we just need to render HTML, CSS, and JavaScript in a window. For these reasons, Electron only loosely supports Next.js applications. That’s not to say you can’t build an Electron app with Next.js, but it requires some workarounds to make it function properly. One of the more popular workarounds is a project called
Nextron
, which aims to wire Next.js applications to the Electron framework and streamline the build process. This is the project we use for the workbench. The issue is that, at the time of writing, it appears that Nextron is no longer being maintained, and we started hitting a few bugs with it.
Tauri is largely frontend-framework agnostic. For Next, specifically, you still can’t use the server-side features, but Tauri makes the integration process much simpler by relying on Next’s static-site generation capabilities. To make a Next app work with Tauri, you just need to set
output: 'export'
in your Next configuration file, and Tauri handles the rest.
The biggest difference between Electron and Tauri comes from how they render the UI. The Electron framework comes with a full Chromium browser engine bundled in your application, which is the same engine that backs Google Chrome. This is useful because it means you don’t have to worry about browser compatibility issues. Regardless of the end user’s machine or architecture, the same Chromium instance renders your application UI. This results in a very standardized experience that ensures your app will look the same regardless of where it’s running. However, this also results in a fair amount of bloat. For the vast majority of desktop apps, a full Chromium browser engine is overkill. Even the simplest “Hello World” applications using Electron can run you up to 150 megabytes of disk space.
Tauri solves this problem by leveraging the system’s native webview. Instead of bundling a full browser engine, Tauri uses a library called
WRY
, which provides a cross-platform interface to the appropriate webview for the operating system. As you’d expect, this makes Tauri apps far more lightweight. The downside here is that you no longer have a hard guarantee on compatibility. From what I can tell, however, this mostly seems to be a non-issue. Compatibility issues across system webviews are exceedingly rare, especially for the major operating systems.
Another major difference between the two frameworks is how they handle the “main” process. This refers to the backend process that orchestrates the application windows, menus, and other components of a desktop app that require interaction with system APIs. In Electron, the main process runs in a Node.js environment. This means you get access to all the typical Node APIs, you can import things like normal, and, perhaps most importantly, you can write your Electron-specific code in pure JavaScript. This is a huge bonus for Electron’s target audience: web developers.
Tauri, on the other hand, uses Rust. All the framework code and the main process entrypoint are written in Rust. Obviously, this makes it a bit less accessible to the average web developer. That said, Tauri provides a fairly robust set of JavaScript APIs to interact with the Rust layer. For most applications, these APIs will be sufficient to do what you need to do. In the case of the workbench, I was able to fully replicate the functionality of the Electron version using the JavaScript APIs and some minimal Rust code.
In my experience, I found the Tauri APIs to fit more naturally in our application code. With Electron, if you need the main process to do something, you must always use inter-process communication, even for the simplest of tasks. If you want to write to a file on the host machine, for instance, your frontend needs to send a signal to the Electron main process, which will then spawn a new process and run the function you wrote that performs the write. With Tauri, you can just use Tauri’s filesystem API directly in your application code. Under the hood, the same sort of IPC pattern is happening, but I think the Tauri abstraction is a bit nicer.
Since Electron runs on Node.js, it also bundles a full Node.js runtime with your application. This comes with some pros and cons. For the workbench, specifically, this is beneficial because the GraphQL layer is itself a separate Node.js application that needs to run alongside the frontend. Since Electron ships with Node.js, this means we can directly spin up the GraphQL server from the Electron main process using the Node runtime. This eliminates a lot of the headache associated with bundling and running a typical sidecar process. For instance, our app also ships with a copy of Dolt, which allows users to start up local Dolt servers directly from the workbench. To make this work, we have to bundle the appropriate Dolt binary with each workbench release that corresponds to the correct architecture. Without the Node runtime, we’d have to do something similar for the GraphQL layer.
With Tauri, this is exactly the problem we run into. To get around it, we need to compile the GraphQL server into a binary using a tool like
pkg
, then run it as a sidecar the same way we run Dolt. Thankfully, this seems to be a fairly common use case for Tauri applications, and they have a useful guide on
how to run Node.js apps as a sidecar
.
It’s also worth mentioning that the full Node.js runtime is quite heavy, which also contributes to bloated Electron app sizes. After building the workbench using both Electron and Tauri, the difference in size was substantial. The left is the Electron version and the right is Tauri:
After replicating the workbench’s functionality in Tauri, we’re holding off on making the full transition for a couple reasons:
Lack of support for .appx and .msix bundles on Windows
- Currently, Tauri only support .exe and .msi bundles on Windows. This means your Microsoft Store entry will only link to the unpacked application. The workbench is currently bundled and published using the .appx format. To address this, we would need to take down the workbench entirely from the Microsoft store and create a new application that uses the .exe format.
Issues with MacOS universal binaries
- This is more an annoyance than a bug, but I ran into a few issues related to codesigning universal binaries for MacOS. Namely, Tauri doesn’t seem to be able to create Mac universal binaries from their arm64 and x64 subcomponents. It also seems to be codesigning the Mac builds twice.
Neither of these are hard blockers, but they’re annoying enough that I’m holding off on migrating until they’re resolved or our issues with Nextron become more problematic. For now, I’m leaving
my branch with the migration
open and hope to revisit soon. If you’re on the Tauri team, let us know if you have solutions!
Overall, I’m impressed with Tauri. It eliminates much of the classic Electron bloat and integrates naturally with our existing codebase. If you’re curious about Tauri or the Dolt Workbench, let us know on
Discord
.
Obelisk and
DBOS
are both open-source
durable workflow engines.
Let's see how they compare in terms of ease of use, nondeterminism prevention and performance.
I will go through the
Learn DBOS Java
tutorial
and compare it with Rust version of the same code written for Obelisk.
I chose Java because of familiarity, however the library has just been released so the code is still quite young.
On the Obelisk side, Rust is the obvious choice as it has the best
performance
and
tooling
.
Setting up the environment
DBOS-Java needs a JDK, Gradle and a PosgreSQL database.
For building WASM Components we need Rust and Cargo.
Intro into deterministic workflow engines
As both
Obelisk
and
DBOS
emphasize, workflows must be deterministic and activities / steps must be idempotent.
This ensures that long running workflows
can continue after a server crash or when they are migrated from one machine to another.
Activities can be retried automatically on a failure, but even a successful activity might
be retried if the server crashes just before persisting the result.
Our test bed will be:
An idempotent activity (step in DBOS lingo) that interacts with the world: sleeps and creates a file
A
serial
and
parallel
workflows that run the activity with persistent sleep in a loop
An HTTP endpoint (webhook endpoint in Obelisk lingo) that triggers these workflows.
package tutorial:activity;
interface activity-sleepy {
step: func(idx: u64, sleep-millis: u64) -> result<u64>;
}
world any {
export tutorial:activity/activity-sleepy;
}
use exports::tutorial::activity::activity_sleepy::Guest;
use std::time::Duration;
use wit_bindgen::generate;
generate!({ generate_all });
struct Component;
export!(Component);
impl Guest for Component {
fn step(idx: u64, sleep_millis: u64) -> Result<u64, ()> {
println!("Step {idx} started");
std::thread::sleep(Duration::from_millis(sleep_millis));
println!("Step {idx} creating file");
let path = format!("file-{idx}.txt");
std::fs::File::create(path)
.inspect_err(|err| eprintln!("{err:?}"))
.map_err(|_| ())?;
println!("Step {idx} completed");
Ok(idx)
}
}
Workflow
package tutorial:workflow;
interface workflow {
serial: func() -> result<u64>;
parallel: func() -> result<u64>;
}
world any {
export tutorial:workflow/workflow;
// Import of the activity.
import tutorial:activity/activity-sleepy;
// Generated extensions for `parallel` workflow.
import tutorial:activity-obelisk-ext/activity-sleepy;
// Obelisk SDK
import obelisk:types/execution@3.0.0;
import obelisk:workflow/workflow-support@3.0.0;
import obelisk:log/log@1.0.0;
}
use exports::tutorial::workflow::workflow::Guest;
use obelisk::{
log::log,
types::time::{Duration, ScheduleAt},
workflow::workflow_support::{self, ClosingStrategy, new_join_set_generated},
};
use std::collections::HashSet;
use tutorial::{
activity::activity_sleepy::step,
activity_obelisk_ext::activity_sleepy::{step_await_next, step_submit},
};
use wit_bindgen::generate;
mod util;
generate!({ generate_all });
struct Component;
export!(Component);
impl Guest for Component {
fn serial() -> Result<u64, ()> {
log::info("serial started");
let mut acc = 0;
for i in 0..10 {
log::info("Persistent sleep started");
workflow_support::sleep(ScheduleAt::In(Duration::Seconds(1)));
log::info("Persistent sleep finished");
let result = step(i, i * 200).inspect_err(|_| log::error("step timed out"))?;
acc += result;
log::info(&format!("step({i})={result}"));
}
log::info("serial completed");
Ok(acc)
}
#[allow(clippy::mutable_key_type)]
fn parallel() -> Result<u64, ()> {
log::info("parallel started");
let max_iterations = 10;
let mut handles = HashSet::new();
for i in 0..max_iterations {
let join_set = new_join_set_generated(ClosingStrategy::Complete);
step_submit(&join_set, i, i * 200);
handles.insert((i, join_set));
}
log::info("parallel submitted all child executions");
let mut acc = 0;
for (i, join_set) in handles {
let (_execution_id, result) =
step_await_next(&join_set).expect("every join set has 1 execution");
let result = result.inspect_err(|_| log::error("step timed out"))?;
acc = 10 * acc + result; // order-sensitive
log::info(&format!("step({i})={result}, acc={acc}"));
workflow_support::sleep(ScheduleAt::In(Duration::Milliseconds(300)));
}
log::info(&format!("parallel completed: {acc}"));
Ok(acc)
}
}
Webhook endpoint
package any:any;
world any {
import tutorial:workflow/workflow;
}
use crate::tutorial::workflow::workflow;
use anyhow::Result;
use wit_bindgen::generate;
use wstd::http::body::Body;
use wstd::http::{Error, Request, Response, StatusCode};
generate!({ generate_all });
#[wstd::http_server]
async fn main(request: Request<Body>) -> Result<Response<Body>, Error> {
let path = request.uri().path_and_query().unwrap().as_str();
let response = match path {
"/serial" => {
let acc = workflow::serial().unwrap();
Response::builder().body(Body::from(format!("serial workflow completed: {acc}")))
}
"/parallel" => {
let acc = workflow::parallel().unwrap();
Response::builder().body(Body::from(format!("parallel workflow completed: {acc}")))
}
_ => Response::builder()
.status(StatusCode::NOT_FOUND)
.body(Body::from("not found")),
}
.unwrap();
Ok(response)
}
Ergonomics
DBOS uses a callback approach when submitting child executions:
int result = DBOS.runStep(() -> step(i, 200 * i), "step " + i);
The drawback here is lower readability when orchestrating a large number of steps.
DBOS could use Proxy pattern instead,
but it would make more difficult to attach metadata such as name, retry configuration
etc.
Obelisk workflows can call a child execution directly without a callback:
let result = step(i, i * 200).unwrap();
or using
Extension Functions
,
automatically generated from the activity's WIT file:
step_submit(&join_set, i, i * 200);
let (_execution_id, result) = step_await_next(&join_set).unwrap();
The great thing about DBOS is that the whole thing fits into a single Java file.
However, schema-first approach has several advantages:
Strong, explicit interface contracts
No way of forgetting to use the callback style or to construct a proxy object.
This would lead to mixing side effects into workflows.
Cross-language interoperability: The WASM Component Model supports many languages such as
Rust, JavaScript, Python, Go and more.
Versioning and backward-compatibility: Activities can export several versions
of the same interface, enabling workflows to catch up on their pace.
Codegen allows creating multiple variants of the same exported function, taking
advangates of both proxy and callback patterns without the drawbacks.
Experiments
Experiment A: Simulating a server crash
The task is to start the workflow, kill the server and start it again to see if the workflow finishes.
Submitting the parallel workflow in DBOS and then killing the server
revealed an unplesent surprise: After restart the workflow ended in
ERROR
state, with an error and a huge stack trace:
However, after I
reported
the bug, the PR was merged within 24 hours. Kudos to the DBOS team.
Other than that, the only notable issue I found was that DBOS recovery started after around a one minute delay.
I was
instructed
to disable Conductor, a proprietary distributed orchestrator, which resolved the problem.
No problems with Obelisk were found.
Experiment B: Breaking determinism with code changes
Changing code of a running workflow is a delicate process. It can only work when the change does not affect
the execution log, or when the running execution did not reach the changed line yet.
One clever trick that DBOS does is that it detects code changes by
hashing the bytecode
of workflow classes. Obviously this is a best-effort solution, as the change can come from a dependency, or from
using a nondeterministic construct, as discussed later.
Instead of extracting the workflow logic into another file, I have disabled the hashing by setting a constant application version.
Obelisk does not currently perform a hash digest of the workflow's WASM executable. In the future, it will
store the WASM hash
when an execution is created. Contrary to DBOS, this approach reliably reflects whether the code (including dependencies) changed or not.
DBOS
should
throw an error when events produced on replay do not match the original execution log.
Let's start with an obvious change in the
serial
workflow:
public int serial() throws Exception {
...
- for (int i = 0; i < 10; i++) {
+ for (int i = 9; i >= 0; i--) {
// Note that the step name changed as well:
int result = DBOS.runStep(() -> step(i2, 200 * i2), "step " + i);
fn serial() -> Result<(), ()> {
...
- for i in 0..10 {
+ for i in (0..10).rev() {
// Call the activity with different parameters
let result = step(i, i * 200).unwrap();
An in-progresss execution that is replayed after restart should detect it immediately as the
i
variable
is used as a parameter to
step
invocation.
Both engines detect the failure:
Full error:
key does not match event stored at index 4:
key: ChildExecutionRequest(E_01KAJVHDX4VAPCPAXXZ2QCVWCZ.o:1_1.o:2-step_1, tutorial:activity/activity-sleepy.step, params: [9, 1800]),
event: JoinSetRequest(ChildExecutionRequest(E_01KAJVHDX4VAPCPAXXZ2QCVWCZ.o:1_1.o:2-step_1, tutorial:activity/activity-sleepy.step, params: [0, 0]))
Notice that the DBOS error only contains the naming differences,
step 0
vs
step 9
whereas Obelisk contains the actual parameters.
In fact, if we are a bit lazy, and do not serialize parameters into the name properly:
- int result = DBOS.runStep(() -> step(i2, 200 * i2), "step " + i);
+ int result = DBOS.runStep(() -> step(i2, 200 * i2), "step");
the workflow will finish happily with result 0+1+2+6+5+4+3+2+1+0=24!
Experiment C: Trimming execution events
Another interesting test is to lower the number of iterations.
Let's start a
serial
workflow, wait until around
step 8
, kill the server and change:
public void serial() throws Exception {
...
- for (int i = 0; i < 10; i++) {
+ for (int i = 0; i < 3; i++) {
...
int result = DBOS.runStep(() -> step(i2, 200 * i2), "step " + i);
This kind of error can lead to resources like temporary VMs running without a cleanup and should be reported by the engine.
The latest version of Obelisk detects it correctly:
found unprocessed event stored at index 18: event: JoinSetCreate(o:7-sleep)
DBOS marks the trimmed execution as successful, returning 0+1+2=3.
Full disclosure: Obelisk 0.26.2 did not detect these changes correctly, fixes landed for 0.27.0 .
Experiment D: Breaking determinism using nondeterministic code
Instead of changing the code, determinism can be broken by using nondeterministic constructs.
DBOS documentation
warns
:
Java's threading and concurrency APIs are non-deterministic. You should use them only inside steps.
Using any source of nondeterminism, including current date, IO, environment variables, RNG or something more subtle like hash maps will break the replay as well.
For posterity, let's replace the
ArrayList
/
Vec
with a hash set:
@Workflow(name = "parallel-parent")
public long parallelParent() throws Exception {
System.out.println("parallel-parent started");
- ArrayList<Map.Entry<Integer, WorkflowHandle<Integer, Exception>>> handles = new ArrayList<>();
+ HashSet<Map.Entry<Integer, WorkflowHandle<Integer, Exception>>> handles = new HashSet<>();
for (int i = 0; i < 10; i++) {
final int index = i;
var handle = DBOS.startWorkflow(
() -> this.proxy.childWorkflow(index),
new StartWorkflowOptions().withQueue(this.queue));
handles.add(new AbstractMap.SimpleEntry<>(i, handle)); // Tuple (i, handle)
}
System.out.println("parallel-parent submitted all parallel-child workflows");
long acc = 0;
for (var entry : handles) {
int result = entry.getValue().getResult();
acc = 10 * acc + result; // Order-sensitive
int i = entry.getKey();
System.out.printf("parallel-child(%d)=%d, acc:%d%n", i, result, acc);
DBOS.sleep(Duration.ofMillis(300));
}
System.out.println("parallel-parent completed");
return acc;
}
Crasing an in-progress workflow and replaying it in a new process:
parallel-parent started
parallel-parent submitted all parallel-child workflows
parallel-child(2)=2, acc:2
parallel-child(8)=6, acc:26
parallel-child(5)=8, acc:268
parallel-child(3)=5, acc:2685
parallel-child(4)=4, acc:26854
parallel-child(7)=9, acc:268549
parallel-child(1)=7, acc:2685497
parallel-child(9)=0, acc:26854970
# replayed up to this point
parallel-child(0)=0, acc:268549700
parallel-child(6)=6, acc:2685497006
parallel-parent completed: 2685497006
Noticed something is off?
parallel-child(n)
should always return
n
, and the final
acc
should contain every digit exactly once.
However the ordering of
HashSet
iteration depends on each object’s memory address or a per-run JVM-specific identity value.
This is an obvious source of nondeterminism, and in this case leads to replaying wrong return values, exactly as in Experiment B.
The best DBOS could do here is to throw a nondeterminism detected error.
Applying the same change in Obelisk:
fn parallel() -> Result<u64, ()> {
log::info("parallel started");
let max_iterations = 10;
- let mut handles = Vec::new();
+ let mut handles = HashSet::new();
for i in 0..max_iterations {
let join_set = new_join_set_generated(ClosingStrategy::Complete);
step_submit(&join_set, i, i * 200);
handles.insert((i, join_set));
}
log::info("parallel submitted all child executions");
let mut acc = 0;
for (i, join_set) in handles {
let (_execution_id, result) =
step_await_next(&join_set).expect("every join set has 1 execution");
let result = result.inspect_err(|_| log::error("step timed out"))?;
acc = 10 * acc + result; // order-sensitive
log::info(&format!("step({i})={result}, acc={acc}"));
workflow_support::sleep(ScheduleAt::In(Duration::Milliseconds(300)));
}
log::info(&format!("parallel completed: {acc}"));
Ok(acc)
}
I have crashed an execution twice, but its output is stable:
I am not aware of any way how to inject nondeterminism into the code:
Accessing IO, source of randomness, spawning a thread etc, are all forbidden
and lead to WASM trap (a non-recoverable runtime fault).
Even if there was a way to bypass the WASM sandbox, it would only lead to the
same result as we saw earlier with code changes - a nondeterminism detected error.
This experiment shows that without a properly isolated environment code that
works fine on the first run can be utterly broken on replay.
Experiment E: Resource usage with 10k or 100k of workflows
One of main selling points of workflow engines is the fact that the
workflows can durably sleep for weeks or more.
Let's model the situation where a main workflow spawns
one child workflow for every customer. This child workflow will just sleep for 1 day in order
to drive some business logic.
@Workflow
public void sleepyWorkflow(int idx) {
System.out.printf("%d%n", idx);
DBOS.sleep(Duration.ofDays(1));
// do some logic here
}
// Test submitting many customer workflows
@Workflow
public void sleepyParent(int max) {
var handles = new ArrayList<WorkflowHandle<Void, RuntimeException>>();
for (int i = 0; i < max; i++) {
final int index = i;
System.out.printf("Submitting child workflow %d%n", i);
var handle = DBOS.startWorkflow(
() -> this.proxy.sleepyWorkflow(index),
new StartWorkflowOptions().withQueue(this.queue));
handles.add(handle);
}
System.out.printf("Created %s child workflows%n", max);
int counter = 0;
for (var handle : handles) {
handle.getResult();
counter++;
System.out.printf("Collected %d child workflows%n", counter);
}
System.out.printf("Done waiting for %d child workflows%n", max);
}
When executing the parent workflow submitting 10k child workflows (on a machine wih 16GB of RAM), the JVM process was slowing down until it reached
a breaking point. After a few minutes, with parent workflow reporting the index 8602,
the process RSS grew from ~130MB to 1.3GB,
reported an OOM error and did not make any further progress.
Even after restart the same thing happened:
[172,701s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[172,701s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-1-thread-18613"
Exception in thread "QueuesPollThread" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
This could be mitigated by changing the limits on the OS level, but using one thread for each inactive workflow execution seems excessive.
fn sleepy_parent(max: u64) -> Result<(), ()> {
let join_set = &new_join_set_generated(ClosingStrategy::Complete);
for idx in 0..max {
log::info(&format!("Submitting child workflow {idx}"));
sleepy_workflow_submit(join_set, idx);
}
log::info(&format!("Created {max} child workflows"));
for counter in 0..max {
let (_execution_id, _result) = sleepy_workflow_await_next(join_set).unwrap();
log::debug(&format!("Collected {counter} child workflows"));
}
log::info(&format!("Done waiting for {max} child workflows"));
Ok(())
}
To make it a bit more challenging let's use
100k
child workflows.
Although Obelisk is yet not hardened for such scale, it managed to start all child workflows with RSS under 500MB.
Part of the reason is that instead of using native threads, each execution lives inside a light-weight
Tokio
task.
Inactive workflows are automatically unloaded from memory after a configurable time, which keeps the
memory footprint low.
Conclusion
Differences between DBOS and Obelisk can be summarized in a table.
Feature / Category
DBOS
Obelisk
Compared Language
Java
(Other languages: Python,TypeScript,Go are supported through their respective SDKs).
Rust
(Other languages supporting WASM Components: JavaScript, Python, Go)
Workflow Definition
Code-first:
Workflows and steps can reside in a single Java class.
Schema-first:
Uses WIT (WebAssembly Interface Type) IDL to define contracts between components.
Ergonomics
Uses a
callback approach
for steps (e.g.,
DBOS.runStep(...)
), which can lower readability.
Supports
direct function calls
or generated extension functions, offering cleaner syntax (e.g.,
step(...)
).
Determinism Enforcement
Manual/Cooperative:
Developers must avoid Java's non-deterministic APIs (threading, IO). Constructs like
HashSet
can break replay.
Strict/Sandboxed:
WASM sandbox prevents access to non-deterministic resources (IO, RNG).
HashSet
iteration is stable.
Code Change Detection
Best-effort:
Hashes bytecode of a class, but not the entire codebase. Failed to detect parameter changes if step names did not encode them; marked trimmed workflows as successful.
Strict:
reliably detects parameter mismatches and unprocessed events (e.g., trimmed events) on replay. Currently does not hash WASM files.
Execution Model
Thread-based:
Uses native threads for workflow executions.
Async/Task-based:
Uses lightweight Tokio tasks.
Scalability & Resource Usage
Lower Scalability:
Failed at
10k
concurrent workflows (OOM, 1.3GB RSS) due to thread exhaustion.
High Scalability:
Successfully handled
100k
concurrent workflows (<500MB RSS) by unloading inactive workflows.
Infrastructure Dependencies
Requires
PostgreSQL
.
Embeds
SQLite
.
DBOS is an interesting project that is very easy to start with, as it encourages developers to include it as a library
in their main application. This architectural decision shapes its features.
Since there is no isolation, determinism is managed by developer's discipline.
This may make it harder to review changes, as seemingly unrelated code changes can lead to (silent) determinism breakage.
Its code change detection can be improved, however it is simply impossible to prevent
accidental nondeterminisic constructs unless workflows are completely isolated
in a runtime that is built around deterministic execution.
On the plus side, DBOS requires no configuration, as workflows, steps and HTTP endpoints are just code.
Obelisk's main strenth is its WASM runtime:
no footguns (threads, IO, hash maps etc.) in workflows
WASM contains the entire code, so associating executions with is original codebase is trivial (although not implemented yet).
Able to unloaded running executions from memory transparently.
Separation of components
Coordination through WIT schemas
Easy to deploy on lightweight VM
The Input Stack on Linux An End-To-End Architecture Overview
Let’s explore and deobfuscate the input stack on Linux. Our aim is to
understand its components and what each does. Input handling can be
divided into two parts, separated by a common layer:
Kernel-level handling: It deals with what happens in the kernel and how events are exposed to user-space
The actual hardware connected to the machine, along with the different buses and I/O/transport subsystems
The input core subsystem, and the specific device drivers that register on it
Exposed layer (middle)
The event abstraction subsystem (evdev)
devtmpfs for device nodes
sysfs for kernel objects and device attributes
procfs for an introspection interface of the input core
User-space handling:
The user-space device manager (udev) and hardware database (hwdb) for device management and setup
The libinput library for general input, and other libraries such as
XKB for keyboards, to interpret the events and make them manageable
The Widgets, X Server, X11 window managers, and Wayland compositors,
which rely on everything else
We’ll try to make sense of all this, one thing at a time, with a logical
and coherent approach.
NB: This article compiles my understand, for any correction please
contact me.
The Kernel’s Input Core
How are input devices and their events handled in the kernel? You might
think it is useless to know, but understanding some of the kernel logic
is what makes things click.
The input core is the central piece of the kernel responsible for handling
input devices and their events. Most input devices go through it, although
some bypass it entirely but these are special use-cases. It provides
common abstract components that sit between the low-level hardware,
and the more useful features for user-space, along with a sort of
publish-subscribe system.
To follow along you can either download the kernel
source, or view it in any browser explorer (such as
this
,
this
,
or
this
).
Practically, the input core is found in the kernel under
drivers/input/input.c
, it defines the basic functionalities related
to the lifecycle of an input device, defined as a
struct input_dev
(
input.h
). Namely:
Allocating the input device structure (
input_allocate_device
that
returns a
struct input_dev
)
Registering and unregistering the input device in the system
along with setting sane default values
(
input_register_device
adds to
input_dev_list
). This also integrates with devtmpfs,
exposing the device, and with procfs, exposing debugging information
(
/proc/bus/input/
).
Drivers push events to the input core using
input_event
. The core then
forwards the events to the registered handlers in
a fan-out fashion (
input_register_handler
adds an
input_handler
to
input_handler_list
). Then handlers forward
them to all clients in user-space (called
input_handle
) listening
for events on that handler. The clients are registered on
the handler with
input_register_handle
(similar confusing names).
The user-space client/handle can also grab the handler with exclusivity
through
input_grab_device
(ex:
EVIOCGRAB
in evdev).
By default the evdev (event device) is attached as the default input
handler and exposes these events to user-space in a standardized way
via an evdev created character stream in devtmpfs (
/dev/input/eventX
).
An input handler is an implementation of an abstract interface
(
include/linux/input.h
), which the input core will call. Particularly,
the
input_event
function in input core will invoke the implementation
of the input handler’s
events
function. Here’s the interface an input
handler should fulfil:
Each actual specific input device driver builds on top of the functions of the
input core in their internal code, adding their own specificities and
advertising what capabilities and features the device can generate. This
creates a polymorphic-like abstraction where common input core logic is
reused, and where input event handlers are abstracted away. In general,
the main role of input drivers is to translate the device specific
protocol to a more standardized protocol, such as evdev, so that it can
be useful in user-space. And additionally, as with most drivers way of
communicating with the rest of the system, they can possibly have extra
configuration through an ioctl interface.
Along with all this, the kernel has a mechanism called sysfs that is
used to expose its internal objects (kobject) to user-space. Anytime
a device is created, it is exposed in
/sys/
(usually mounted there)
with its properties (
/sys/devices/
). For the input core part, we can
find it in
/sys/class/input/inputN
, and within each sub-directories
we have the properties of the object.
Furthermore, when a device is plugged or unplugged (
device_add
,
device_remove
in
drivers/base/core.c
), the kernel also emits events,
called uevent, via netlink (
PF_NETLINK, NETLINK_KOBJECT_UEVENT
) which
can then be caught in user-space and acted upon. We’ll see later how
these are handled by udev.
This is a general overview of our understanding of the input core so far:
The Logical Input Device Topological Path
We may commonly say that devices are connected to a machine and magically
handled from there on. Yet, we know that it’s an abstraction and that
there’s more to it. What happens in reality is that the electrical
connection first passes over a bus/host controller, which then let’s the
data be transported. This data is formatted in a specific input protocol
that should be handled by a driver that speaks it and that subsequently
creates a related input device. In most input device cases, the driver
then translates the protocol into evdev
“common speech”
.
That’s a whole layer of things before reaching the input core. Just like
in the world of networking, one thing wraps another. In this particular
case, devices have parents, a hierarchy, a stack of devices, drivers,
and helpers.
Here’s what that stack is like in theory, with in reality some lines
blurred together:
In this section, let’s try to understand how the kernel uses
plug’n’play/hotplug to pick the right drivers in this stack, and how we
pass from electrical signal to evdev. To do that we’ll first look at
how the kernel pictures its internal objects, and how these together
somehow create the above hierarchy. Finally, we’ll see some concrete
examples of that, along with some command line tools that can clearly
display this encapsulating behavior.
As we said there’s a hierarchy of kobjects in the kernel from the bus
to its connected devices. These are stored in-memory as a linked list
hierarchy, which is also represented under sysfs as a file system tree.
Specifically, in
drivers/base/core.c
this is what is used to create
the parent-child relationship:
/devices/...
— root of the kernel’s sysfs device tree, showing all devices known to the kernel.
pci0000:00/0000:00:14.0
— PCI bus and controller (the USB host controller here).
usb1/1-1/1-1:1.0
— USB bus and port hierarchy (device 1-1, interface 1.0).
0003:046D:C31C.0003
— HID device node (bus
0003
= USB HID, vendor
046D
= Logitech, product
C31C
= specific keyboard).
input/input6
— input subsystem device registered under
/sys/class/input/input6
.
event3
— the evdev interface, the character device exposed in
/dev/input/event3
.
How did we end up with this long list, how did it get created? Let’s see
how the kernel stores this info, and what happens from its perpective.
As far as its concerned, the device-related things it knows is summed
up in these types of objects:
bus - a device to which other devices can be attached
device - a physical/logical device that is attached to a bus
driver - a software entity that can be associated with a device and
performs operations with it
class - a type of device that has similar behavior; There is a class
for disks, partitions, serial ports, input, etc.
subsystem - a view on the structure of the system; Kernel subsystems
include devices (hierarchical view of all devices in the system), buses
(bus view of devices according to how they are attached to buses),
classes, input, etc. We care about the input subsystem.
For example, there are different views of the same device. You’ll find
the physical USB device under
/sys/bus/usb/devices/
and the logical
device of the input class under
/sys/class/input/.
Let’s go over these objects, tracing the path, starting with buses.
A hardware bus/host controler is a communication channel between the
processor and input/output device. But a kernel bus object is more
generic than this, it’s a logical function which role is to be a point
of connection of devices. All devices are connected to a kernel bus,
even if it needs to be a virtual one. So kernel buses are the root of
the hierarchy.
The main buses are things such as PCI, USB, IDE, SCSI, platform,
ACPI, etc.
Kernel buses are the connective tissue of everything, the base of the
infrastructure. As you can see from the structure it’s responsible for
probing the device to get info about it, handling connected/disconnected
events, creating a new node for it, and sending uevent to notify
user-space and triggering a chain reaction.
Yet, one of their most important role is to start the match between
devices and registered device drivers, as can be noted from the
match
function. Keep in mind that the matched driver can be another bus, so
this initiates a cascade of different handlers, bubbling up the hierarchy.
A concrete example of this recursion:
A PCI bus controller (the host bridge) is a device on the platform bus.
The USB bus (usbcore) is a device on the PCI bus (via xHCI controller).
The HID bus is a device on the USB bus (via usbhid).
The specific HID protocol driver is a device on the HID bus
The low level kernel buses such as the hardware bus/host controlers
generally don’t handle input data directly, though there are some bus/host
controller drivers that do register input devices to the input core,
bypassing everything else in the stack and acting as event sources. These
exceptions are usually for brightness control hotkeys, lid sensors,
built-in special functions keys, etc.. We have for example the drivers
acpi_video
,
thinkpad_acpi
,
asus_wmi
, etc..
To know how to handle the devices and whether a driver needs to be loaded
from a module, all devices and buses have specially formatted IDs, to
tell us what kind of devices they are. The ID, which we call MODALIAS,
consists of vendor and product ID with some other subsystem-specific
values. Each bus and device has its own scheme for these IDs.
For a USB mouse, it looks something like this:
This is needed in case the driver isn’t built-in the kernel and instead
was an external module (
*.ko
). As a reminder, a driver is some piece
of code responsible for handling a type of device, and a module is a
piece of external kernel code that can be dynamically loaded at runtime
when needed. Depending on the distro choices, some drivers are set as
external modules that need to be loaded at runtime.
To achieve this, the kernel, after composing the MODALIAS string, sends
it within the uevent towards user-space. To complete this information,
each external kernel module comes with a list of known MODALIASes it can
handle, so that they can be loaded as needed. These lists are compiled
by programs such as
depmod
that creates files like
modules.alias
in the kernel’s
/lib/modules
directory for all currently available
modules that aren’t built-in (
/lib/modules/VERSION
), and the built-in
ones (
modules.builtin
).
In theory that’s fine, this infrastructure model makes it easy to
dynamically load modules that are not already built-in, but we need
a piece of software in user-space to catch the events and perform the
actual loading. This is a role that udev embodies by calling
modprobe
for every event that has a MODALIAS key, regardless of whether a module
needs loading or not. We’ll see more of udev but for now keep in mind
that its doing this hotplug mechanism.
If you’re curious, you can try this udev command to monitor the MODALIAS.
devadm monitor --property
Yet, this doesn’t solve what happens to devices that were present at
boot and which need modules. The solution: there’s a file in the device
directory in sysfs with all the uevent generated at boot for every
devices in sysfs file system, appropriately named “uevent”. If you write
“add” to that file the kernel resends the same events as the one lost
during boot. So a simple loop over all uevent files in
/sys
triggers
all events again.
The MODALIAS value is also stored in sysfs along with the device
properties, here are a few commands to gather information on this:
> cat /sys/devices/pci0000:00/0000:00:10.0/modalias
pci:v00001022d00007812sv00001025sd00000756bc0Csc03i30
> modprobe --resolve-alias $(cat /sys/devices/\
pci0000:00/0000:00:13.2/usb1/1-0:1.0/usb1-port3/modalias)
Not everything has an associated module
> ls -l /sys/devices/pci0000:00/0000:00:10.0/driver
lrwxrwxrwx 1 root root 0 Oct 25 11:37 driver \
-> ../../../bus/pci/drivers/xhci_hcd
If the driver link exists, check which module implements it:
> modprobe -R xhci_hcd
xhci_hcd
> modinfo xhci_hcd
name: xhci_hcd
filename: (builtin)
license: GPL
file: drivers/usb/host/xhci-hcd
author: Sarah Sharp
description: 'eXtensible' Host Controller (xHC) Driver
license: GPL
file: drivers/usb/host/xhci-hcd
description: xHCI sideband driver for secondary interrupter management
parm: link_quirk:Don't clear the chain bit on a link TRB (int)
parm: quirks:Bit flags for quirks to be enabled as default (ullong)
For example that xhci_hcd module is builtin
So far we’ve learned two things: buses which devices are connected to,
and the MODALIAS mechanism to match modules and dynamically load drivers
that aren’t built-in. Let’s see the devices attached to buses as they
appear as kernel objects.
structdevice{// …structdevice*parent;structdevice_private*p;structkobjectkobj;constchar*init_name;/* initial name of the device */// …structbus_type*bus;/* type of bus device is on */structdevice_driver*driver;/* which driver has allocated this
device */// …conststructclass*class;// …void(*release)(structdevice*dev);};
Along with the related driver:
structdevice_driver{constchar*name;structbus_type*bus;structdriver_private*p;structmodule*owner;constchar*mod_name;/* used for built-in modules */int(*probe)(structdevice*dev);int(*remove)(structdevice*dev);void(*shutdown)(structdevice*dev);int(*suspend)(structdevice*dev,pm_message_tstate);int(*resume)(structdevice*dev);};
As you can notice, they also have a probing and lifecycle functions to
be implemented. We also have the registration/unregistration functions
(
input_register_device
and
input_unregister_device
in our case)
which will announce that the device is now available in the system (plus
a uevent and other user-space stuff). Each of the registered devices
have an entry in sysfs
/sys/devices
, along with the information about
its driver, and similar info in
/sys/class
and
/sys/bus
. The
device also creates files in devtmpfs that represent its interfaces. Let’s
note that devtmpfs is usually mounted by default to user-space as a virtual
filesystem on most distros.
To check whether devtmpfs is enabled, which is almost always the case today:
> zcat /proc/config.gz | grep DEVTMPFS
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_DEVTMPFS_SAFE=y
> mount | grep devtmpfs
dev on /dev type devtmpfs (rw,nosuid,relatime,size=2720672k,\
nr_inodes=680168,mode=755,inode64)
> mount | grep sysfs
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
Devices are associated to classes and subsystems that handle them.
The subsystem we care about here is what we’ve seen in the earlier
section: the input core, the input device subsystem.
subsys_initcall(input_init);
As for the concept of a class, it’s a high-level view of the device
model, abstracting implementation details. For example there are drivers
for SCSI and ATA but both are in the disks class. Similarly, all input
devices are in the input class, which is what we care about. This is
a grouping mechanism, unrelated to how the devices are connected. They
can be found in sysfs
/sys/class/
.
The
struct class
is instantiated in the
struct device
through
class_register
and
class_unregister
. This will in turn also help
udev, as we’ll see later, better manage the devices in devtmpfs user-space
mapping
/dev/
, adding a filter for rules.
This completes our overview of the way the kernel perceives the different
types of objects it manages. However, that didn’t clarify how we ended
up with the example path above other than somehow having a kernel bus and
device hierarchy.
We talked about hardware buses and host controllers drivers that aren’t
handling data and that they delegate this to an upper layer. In theory
this upper layer is split between a kernel bus&device for the transport
layer, aka IO layer, and a kernel bus&device for the protocol layer,
but in reality those might get mixed up (bus&device because it’s both).
The IO layer is responsible for handling the physical electrical
communication with the device, it’s setup, and management. At this level
we have USB, Bluetooth, I2C, SPI, etc.. In drivers that means:
usbhid
for HID devices over USB,
btusb
and
hidp
for HID over Bluetooth,
i2c-hid
for touchpads and keyboards that are wired to the motherboard’s
I2C,
psmouse
and
serio
for PS2 mouse, etc..
The protocol or function specific layer then takes over and has as role
to integrate with the input core and translate the raw transport data
into a common format, usually evdev. The evdev format is favored
as it provides a uniform API to represent input devices (via
/dev/input/eventX
).
A few examples:
There’s a mouse communication protocol usin 9 pins DE-9 over the RS-232
standard for communication with UART
The PS/2 mouse which uses a serial transport protocol with 6 pins (
serio
)
The atkbd keyboard also over serial transport
A gamepad that uses HID but the particular case of a sony joystick over USB
There’s another component of complexity to add: we don’t have a
single protocol for a single transport over a single hardware bus/host
controller. Sometimes there’s a generic protocol layer which is reused
with different transport mechanisms. There can also be a delegation
mechanism for the more specific sub-protocol handlers for specific
devices or modes.
For example, you can have an HID protocol over USB (
usbhid
), and the
particular part of the HID protocol used for input devices, and the more
specific sub-HID protocol of a type of device (
hid-generic
and others).
We’ll see an example of this by diving into the HID subsystem which is
the most popular input protocol these days, but first let’s check some
tools that can help us see all that we’ve learned thus far and make
sense of the hierarchy:
lspci -vn
list info about devices connected via PCI buses
lsusb -v
or
usb-devices
list usb devices information in a more human readable form
dmesg
the sys logs
hwinfo --short
to probe hardware
Yet the best way to get a lot of info about the bus and device hierarchy
is to rely on
udevadm
, a user-space tool that comes with udev. Here’s
how it looks for an input device:
> udevadm info -a-p$(udevadm info -q path -n /dev/input/event9)
looking at device '/devices/.../input/input6/event9':
KERNEL=="event9"SUBSYSTEM=="input"
ATTR{name}=="Logitech USB Keyboard"
...
looking at parent device '/devices/.../input/input6':
KERNEL=="input6"SUBSYSTEM=="input"
...
looking at parent device '/devices/.../0003:046D:C31C.0003':
KERNELS=="0003:046D:C31C.0003"DRIVERS=="hid-generic"SUBSYSTEMS=="hid"
...
looking at parent device '/devices/.../1-1:1.0':
KERNELS=="1-1:1.0"SUBSYSTEMS=="usb"DRIVERS=="usbhid"
...
looking at parent device '/devices/.../1-1':
KERNELS=="1-1"SUBSYSTEMS=="usb"DRIVERS=="usb"
...
looking at parent device '/devices/.../0000:00:14.0':
KERNELS=="0000:00:14.0"SUBSYSTEMS=="pci"DRIVERS=="ohci-pci"
...
NB
: It is also a bit more clearer, though for the moment confusing,
to also look at
udevadm info --tree
. Similarly, the
loginctl
seat-status
also clearly shows the hierarchy of devices in the current
session. We’ll talk more about the concept of seats later on.
We see the “looking at parent device” block that corresponds to one
struct device
in the kernel kobject mapped in sysfs, along with the
driver, when it’s present, and other info it gathers at every step,
walking down the bus hierarchy. Let’s note that not everything has
an associated driver since the hardware topology might not match the
driver topology. That often means one kernel component handles multiple
parts of the stack. In the above trace,
hid-generic
handles the input
registering.
This example in particular shows:
PCI → USB controller → USB device → HID interface → input device → evdev node
Another source of information that we briefly mentioned is the procfs
introspection interface (
/proc/bus/input/
), it can also help see the
handling of input devices more clearly as it’s a text-based view of what
the kernel input subsystem knows. It is more or less analogous to the
sysfs view but is meant for human-readable diagnostics. In conjunction
with what we’ve learned in the previous input core section, it should
clarify some of our understanding. It has two files underneath:
devices
and
handlers
.
The
devices
file contain all the current input devices and has entries
with these fields:
I
: basic info (bus type, vendor/product/version)
N
: name
P
: physical path (e.g.,
isa0060/serio0/input0
)
S
: sysfs path
U
: unique identifier (if provided)
H
: list of event handler interfaces bound (like
event3
,
js0
, etc.)
B
: capability bitmaps (
EV
,
KEY
,
REL
,
ABS
, etc.) we’ll explore what
this means when looking at evdev
Here you can see that a single physical device can possibly present
itself as multiple input devices with different handlers attached for
separate functions (here the keys of the System Control handler are
fewer). Here,
kbd
is console handler, and
eventN
is the evdev
user-space handler. Libinput, which we’ll cover later, uses groups
LIBINPUT_DEVICE_GROUP
to logically combine the different devices that
are actually on the same hardware.
The handlers file is about instances of the
input_handler
that will
be called from input core’s
input_event
we mentioned before. As we
said most of it is handled by evdev, but there are exceptions such as:
We’ll talk about joydev later on. As for mousedev, it is there only for
legacy compatibility of old
/dev/psaux
-style mouse interface.
Let’s now see the example of a dummy input driver, to get the idea across.
// SPDX-License-Identifier: GPL-2.0#include<linux/module.h>
#include<linux/init.h>
#include<linux/input.h>
#include<linux/timer.h>staticstructinput_dev*dummy_input_dev;staticstructtimer_listdummy_timer;staticvoiddummy_timer_func(structtimer_list*t){staticboolkey_down=false;/* Simulate key press/release of KEY_A */key_down=!key_down;input_event(dummy_input_dev,EV_KEY,KEY_A,key_down);input_event(dummy_input_dev,EV_SYN,SYN_REPORT,0);/* Reschedule timer */mod_timer(&dummy_timer,jiffies+msecs_to_jiffies(2000));}staticint__initdummy_input_init(void){interr;dummy_input_dev=input_allocate_device();if(!dummy_input_dev)return-ENOMEM;dummy_input_dev->name="Dummy Input Device";dummy_input_dev->phys="dummy/input0";dummy_input_dev->id.bustype=BUS_VIRTUAL;dummy_input_dev->id.vendor=0x0001;dummy_input_dev->id.product=0x0001;dummy_input_dev->id.version=0x0100;/* Declare we can emit key events */__set_bit(EV_KEY,dummy_input_dev->evbit);__set_bit(KEY_A,dummy_input_dev->keybit);err=input_register_device(dummy_input_dev);if(err){input_free_device(dummy_input_dev);returnerr;}/* Setup a timer to inject key events periodically */timer_setup(&dummy_timer,dummy_timer_func,0);mod_timer(&dummy_timer,jiffies+msecs_to_jiffies(2000));pr_info("dummy_input: registered fake input device\n");return0;}staticvoid__exitdummy_input_exit(void){del_timer_sync(&dummy_timer);input_unregister_device(dummy_input_dev);pr_info("dummy_input: unregistered\n");}module_init(dummy_input_init);module_exit(dummy_input_exit);MODULE_AUTHOR("Example Author");MODULE_DESCRIPTION("Minimal Dummy Input Device");MODULE_LICENSE("GPL");
That’s it, you should now somewhat have an idea of how we pass from
hardware events, to kernel objects, and end up within the input core
subsystem, which should prepare events for user-space. Let’s now dig on
and explore a few of the topics we’ve grazed in the past two sections.
sysfs
We already covered a lot of ground in understanding sysfs, so let’s
continue and summarize everything we know and complete the full picture.
As we briefly said before, sysfs is a virtual file system representation in
user-space of the kernel objects and their attributes, it’s how the kernel
views the current state of the system, and also how the user can interface
with the parameters of the kernel in a centralized manner. It’s all done
in a very Unixy way by manipulating simple files.
The file mapping happens as such: kernel objects are directories, their
attributes are regular files, and the relationship between objects is
represented as sub-directories and symbolic links.
The object information is categorized as one of the following. Each of
these is a sub-directory under
/sys/
.
block - all block devices available in the system (disks, partitions)
bus - types of bus to which physical devices are connected (pci, ide, usb)
class - drivers classes that are available in the system (net, sound, usb)
devices - the hierarchical structure of devices connected to the system
dev - Major and minor device identifier. It can be used to automatically
create entries in the
/dev
directory. It’s another categorization of
the devices directory
firmware - information from system firmware (ACPI)
fs - information about mounted file systems
kernel - kernel status information (logged-in users, hotplug)
module - the list of modules currently loaded
power - information related to the power management subsystem
information is found in standard files that contain an attribute
device (optionally) - a symbolic link to the directory containing devices; It can be used to discover the hardware devices that provide a particular service (for example, the ethi PCI card)
driver (optionally) - a symbolic link to the driver directory (located in
/sys/bus/*/drivers
)
As far as we’re concerned, when it comes to input devices, the
/sys/devices/
directory is probably one of the most important. It’s
the representation of the hierarchy of devices we’ve talked about in
the previous section.
Pasting the tree here would be cumbersome, but try
tree -L 5 | less
within
/sys/devices
and you’ll clearly see how things fit together,
a direct hierarchical mapping of how devices are connected to each others.
Within this directory we can find interesting information associated to
the device and its type. For usb devices, for example, we have info such
as the bus number, port number, the vendor and product id, manufacturer,
speed, and others.
Furthermore, the
/sys/bus
directory organizes devices by the type of
bus they are connected to. You can imagine that this isn’t a linear
view since buses can have buses as devices (
usb
and
hid
each have
their directory even though
hid
is probably under
usb
), but it it
helpful to perceive what is happening, an easy shortcut. Within each
bus directory there are two subdirectories: drivers, that contains the
driver registered for the bus, and devices, that contains symbolic links
to the devices connected to that bus in
/sys/devices
.
Similarly, the
/sys/class
directory has another view of the system
from a more functional/type perspective. It’s about what devices do and
not how they’re connected. As far as we’re concerned, the subdirectory
/sys/class/input/
is where we’ll find symbolic links to all devices
that have the input class in
/sys/devices
.
This directory contains both symlinks to input devices and evdev devices,
the latter are usually sub-directories of the former. A notable file in
the input directory is the “capabilities” file, which lists everything
that the device is capable, as far as input is concerned. We’ll revisit
this in the evdev section.
Finally, the last directory that is of interest to us in sysfs is
/sys/module/
which provides information and settings for all loaded
kernel modules (the ones that show with
lsmod
), their dependencies
and parameters.
Lastly, and it might not need to be mentioned, but sysfs needs to be
enabled in the kernel confs. It always is these days since it’s expected
by many software.
HID — Human Interface Device
HID, or Human Interface Device, has been mentioned and sprinkled all
over the place in the last sections, we said it’s a device protocol but
what is it exactly?
HID is probably the most important input/output standard device protocol
these days, it’s literally everywhere and most new devices, from mice
to microphones, speak it over all types of transports such as USB, i2c,
Bluetooth, BLE, etc… It’s popular because it’s a universal way to let the
device first describe its capabilities (buttons, keys, axis, etc..),
what it can send/receive (Report Descriptor), and then send/receive them in
the expected way (Input/Output/Feature Reports).
In sum, in the ideal case it would mean avoiding having a specific driver
for every new device out there and instead have a centralized generic way
to handle all categories of devices, all working out-of-the-box. Indeed, in
practice it has worked great and there have only been minor vendor or
hardware quirks fixes (
drivers/hid/hid-quirks.c
and others).
For example, a HID Report Descriptor may specify that “in a report
with ID 3 the bits from 8 to 15 is the delta x coordinate of a mouse”.
The HID report itself then merely carries the actual data values without
any extra meta information.
The current list of HID devices can be found under the HID
bus in syfs,
/sys/bus/hid/devices/
. For each device, say
/sys/bus/hid/devices/0003:1B3F:2008.003E/
, one can read the
corresponding report descriptor:
The raw HID reports can also be read from the
hidraw
file created by
hid core in devtmpfs
/dev/hidrawN
.
What does an input device HID Report and Report Descriptor look like?
We won’t go into too much details since the HID specifications are huge
but we’ll only do a tour to get an idea and be productive with what
we know. If you want to dive deeper, check the specifications
here
, it’s divided into a basic structure doc
“HID USB Device Class Definition”, and the HUT, “HID Usage Tables”, which
defines constants to be used by applications.
So as we said, the main logic of the protocol is that HID messages are
called Reports and that to parse them we need a Report Descriptor. The
Report Descriptor is a kind of hashmap stream, it contains Items, which
are 1B header followed by an optional payload of up-to 4B. The Items
don’t make sense by themselves, but do make sense together as a whole
when read as a full stream since each Item has a different meaning. Some
meaning apply locally and others globally.
The encapsulating and/or categorizing Items are the Usage Page, which
is a generic category of thing we’re describing, with its subset of
Usage, which is the specific thing we control within that Page. These
are defined in the “HID Usage Tables” doc. It’s things such as:
It’s a couple of info to know how to better handle the HID internal data,
it tells you what is actually being handled.
Another grouping mechanism is the Collection, a broader category
to put together all that the device handles. Let’s say a mouse can
have both buttons, a scroll wheel, and axis it moves on, all within
a Collection. There are 3 types of collections that encapsulate each
others: Application (mandatory) the device-level group, Logical (optional)
sub-grouping for related controls, and Physical (optional) sub-grouping
for physical sensors.
Reports within Collections can also be grouped by IDs to facilitate
parsing.
Within all these, within the inner Collections, we finally have the
definition of what the Reports will actually look like. Here’s a subset
of what a Report Descriptor can look like:
This is all a single report with ID
0x01
, and we see first that within
the Button page we have values ranging from 1 to 5, a count of fields
in the current report size of 5, for 5 buttons each having one bit. The
Input
Item tells us to start processing the Report as input data (there’s
also
Output
and
Feature
). It also indicates that buttons have absolute
values, unlike the X/Y axis which are relative.
The
Cnst
of the following data in the stream stands for constant,
and it’s basically ignored, it’s padding.
And so on, we parse the data afterward, the X/Y relative movements.
One thing to note, is the scope of the meaning of the Items. Some apply
globally, such as the Usage Page, Logical Min/Max, Report Size, Report
Count, etc.. Meanwhile, Usage only apply locally and needs to be set
again. Other Items have special meaning such as Input, Output, Feature,
Collection and End Collection, and are about defining the structure of
data and when to process it.
Here’s a full real example with the Collection grouping mechanism:
As you can see, lots of it may seem redundant within the Logical and
Physical optional sub-collections but they’re often there by default
for hierarchical grouping. They’re not mandatory but common.
Let’s also note that from hid-input’s perspective, one device is created
per top-level Application Collection, so in theory a device can have
many sub-devices.
From the kernel’s perspective, the transport bus notices that a device
is advertised as an HID class and then the data gets routed to the hid
core bus.
For example, this is what the USB transport might notice:
The HID core subsystem is in charge of managing the lifecycle
(connect/disconnect/open/close), parsing the HID report descriptors
to understand the device capabilities. Once parsed, it dispatches
Reports to the HID drivers registered on the HID bus, each driver can
inspect the Usage Page and Usage to decide how and whether to handle
them. This is like a publish-subscribe mechanism. The most specific
registered driver (vendor specific) will match and handle Reports in
whatever way they see fit, otherwise the hid-generic driver is the
fallback.
Several
*_connect
hooks in the HID core subsystem allow attaching
handlers for different behavior that HID device provide. The most
important for us is the
hidinput_connect
for the
HID_CONNECT_HIDINPUT
,
to handle HID input devices. It’s default implementation lives in
hid-input
(internally
hidinput_report_event
). Device specific drivers
can override this behavior if needed. The hid-input role is to bridge
with the input core, allocating and registering the input device via
input_register_device
, which will in turn expose
/dev/input/eventN
,
as we’ve seen before, and translate HID Reports to evdev.
Similarly, in this pub-sub fan-out fashion, another handler is the
default one registered for
HID_CONNECT_HIDRAW
, from
hidraw.c
(
hidraw_report_event
). This driver will create a raw interface on
devtmpfs (
/dev/hidrawN
) to interface with raw HID events that aren’t
necessarily input-related.
This looks somewhat like this:
This is all neat, let’s list a couple of tools that can help us debug
HID and inspect HID Reports Descriptors and Reports.
usbhid-dump
- will dump USB HID device report descriptors and streams
hidrdd
- verbose description of hid report descriptors
hid-tools
- has many sub-tools such as replay, decode, and recording
The simplest one in my opinion is hid-tools, here’s an example of a
keyboard with consumer control and system control, the same one we’ve
seen in the procfs introspection interface earlier (
/proc/bus/input/
):
You can see it has two Application Collections, so that’s why we had
two entries for the keyboard.
In some cases, the HID Device Descriptor is wrong
and needs some patching, which can either be done in a
special driver, or on a live system dynamically by relying on
udev-hid-bpf
which will be invoked before the kernel handles HID.
evdev — Event Device
Let’s tackle the last piece of the exposed middle-layer that we didn’t
explain yet: The Event Device common protocol, the evdev layer.
From what we’ve seen, we know that evdev is a standardization interface,
it decouples and abstracts the underlying devices. It could be a USB
keyboard, a Bluetooth pointer, or PS/2 device, and all the user needs
is to read from the evdev interface, without worrying about their
differences.
It works because evdev registers itself as the default input handler in
the input core, and the main job of most input driver is to translate
to it:
When its “connect” event is fired, it creates the corresponding evdev
node in
/dev/input/eventN
. Furthermore, the info is also reflected
in sysfs within the
/sys/class/input/eventN
directory along with its
related
/sys/class/input/inputN
device created by the input core, which
it is the children of (
eventN
within
inputN
).
The evdev driver also supports certain ioctl to query
its internal state, let a client handle exclusively grab a
device (
EVIOCGRAB
), or change certain values. The list of ioctl can be found
here
within libevdev, though libevdev doesn’t support all of them (the list
can also be found in
include/linux/input.h
).
Let’s see what the evdev format is about, and how the input core
translates to it and generates the events.
The evdev protocol is stateful, it doesn’t forward everything to
user-space but only does when it notices a change. To inquire about
its current state one can rely on ioctl instead.
The format of evdev is composed of a series of
input_event
(from
include/linux/input.h
) which look like the structure here under,
grouped in what’s called a sequence or a frame:
Basically a timestamp along with a type-code couple and an associated
value. The type is the general category to which this event is part of,
and the code the sub-category. For example it could be a relative movement
(type), on the x-axis (code), of 1 unit (value). The available types of
events and codes can be found under
include/linux/input-event-codes.h
.
Each frame ends whenever a synchronization event comes up, the most
common is of type.code(value)
EV_SYN.SYN_REPORT(0)
. It’s the marker
that it’s time to make sense of the stream, the whole frame.
An example snapshot of a frame of an “absolute touchpad” would look
like this:
As we’ve said, it’s stateful, so the events are only sent when there is a
state change, even when the hardware keeps resending the same event. So
for example, if a key is kept pressed, it won’t resend the event until
it’s released.
These events might seem simple on their own but are in fact absolutely
complex to handle, especially touchpads. There are many features such as
pressure, multi-touch, and the tracking of different fingers, which needs an
upper layer to make sense of all this. This is where libinput shines, and
we’ll see that later on. For now just keep in mind it’s a series of event.
So how do drivers use evdev to send events, we’ve talked about
input_event
before, but how does it work.
Well, first of before sending any event, the input driver needs at the
registration phase to advertise to the system what it’s capable of,
to say what kind of events it can generate. These event “capabilities”,
as they’re called, are a couple of different bits in sets that are also
inspectable in sysfs
/sys/class/input/inputN/capabilities/
.
You’ll find the following types of capabilities:
ev
, set in
input_dev->evbit
, Which event types the device can generate (
EV_KEY
,
EV_REL
, etc.)
key
, set in
input_dev->keybit
, Which key/button codes it supports
rel
, set in
input_dev->relbit
, Which relative axes (e.g., REL_X, REL_WHEEL)
abs
, set in
input_dev->absbit
, Which absolute axes (e.g., ABS_X, ABS_Y)
led
, set in
input_dev->ledbit
, LED indicators (e.g., keyboard LEDs)
sw
, set in
input_dev->swbit
, Switch states (e.g., lid switch)
ff
, set in
input_dev->ffbit
, Force feedback capabilities
msc
, set in
input_dev->mscbit
, Miscellaneous events
snd
, set in
input_dev->sndbit
, Sound events
As you can see, it’s somewhat related the HID capabilities in a sense,
but applies to all devices.
We’ve also seen these capabilities bits during our inspection of the
input core procfs interface
/proc/bus/input/
in the
B
field:
However, parsing the bits manually in procfs or sysfs would be cumbersome,
it’s better to rely on tools such as
libinput record
, check the
“Supported Events”
section:
As a note, the Properties can let us know whether we’re dealing with
a touchscreen
INPUT_PROP_DIRECT
, or a touchpad
INPUT_PROP_POINTER
,
and
INPUT_PROP_BUTTONPAD
also tells us that it’s a so-called clickpad
(no separate physical buttons but the whole touchpad clicks). These are
hints for libinput to properly handle different kinds of devices.
So after registering its capabilities, the input driver simply reports
its events by relying on the
input_event
function, or one of it’s
many wrappers:
That’s it mostly to understand evdev! There are multiple tools to help
debug evdev-related issues. We’ve seen
libinput record
. Similarly,
there’s the
evemu
suite with its record, device, play functions to
simulate and test devices, and
evtest
.
There’s also
evsieve
, a tool to
intercept and modify evdev events on the fly.
Along with these, the library libevdev, in C and python, is the most
used to integrate with evdev-related things.
udev & hwdb
After going through the kernel and exposed layers, we’re finally in
user-space!
The first component we’ll see is udev, since we mentioned its role
countless times in the previous sections.
Udev, or the dynamic user-space device manager, implemented as the
udev daemon
systemd-udevd
, has as role to take actions whenever a
uevent (
PF_NETLINK, NETLINK_KOBJECT_UEVENT
) is sent from the kernel
to user-space. We’ve seen a few of the possible actions it performs,
here’s a summary of the kind of things it does:
Load kernel modules based on the uevent MODALIAS
Set access rights on device nodes
Attach properties to devices on detection
Create symlinks so that devices have more predictable names
Keep track internally of device info in its internal db
Use its rule system to take any kind of action on plug/unplug of a device
The most important part is the last point: udev has a set of rules against
which it can match devices and their attributes and take all sorts of
actions based on that. The fields it has access to not only come from the
uevents but also from all related info on the system.
These rules, as is the convention for pretty much all big daemons these
days, are read from system locations such as
/usr/lib/udev/rules.d
,
/usr/local/lib/udev/rules.d
, and the volatile runtime in
/run/udev/rules.d
, and from the local admin directory of
/etc/udev/rules.d
which takes precedence over the other locations. The
directories contains files with a
.rules
extension and are processed and
ordered lexically (
01-example.rules
comes before
05-example.rules
).
Now the syntax of udev rules, which are mainly composed of matching
patterns and actions to perform or properties to set upon match, is
dense and complex (it even has branching). Only a deep study of
udev(7)
man page will help. Yet, we can still learn the very basics of it to be
able to understand what’s happening.
Our approach will consist of, first checking two examples, then have a
general overview of the possible components of the syntax, and finally
talking about the particularities of that system.
The first example is quite simple, it will run a script when a specific
keyboard is plugged/unplugged.
The rule is pretty clear about what it does, on “add” or “remove” action
for specific match it’ll execute a script. But you’ll also notice that the
match components such as SUBSYSTEM and ATTRS are things we’ve seen before
in previous traces of
udevadm info
, which is exactly the point.
udevadm
info
will show us certain components we can used to match.
The second example is a tad bit more complex, we will parse
/usr/lib/udev/rules.d/60-persistent-input.rules
. That file creates
a more persistent naming scheme for input devices in devtmpfs under
/dev/input/by-id
and
/dev/input/by-path/
. Here’s a simplified version
of it.
ACTION=="remove", GOTO="persistent_input_end"
SUBSYSTEM!="input", GOTO="persistent_input_end"# …# determine class name for persistent symlinks
ENV{ID_INPUT_KEYBOARD}=="?*", ENV{.INPUT_CLASS}="kbd"
ENV{ID_INPUT_MOUSE}=="?*", ENV{.INPUT_CLASS}="mouse"# …# by-id linksKERNEL=="event*", ENV{ID_BUS}=="?*", ENV{.INPUT_CLASS}=="?*",
ATTRS{bInterfaceNumber}=="|00",
SYMLINK+="input/by-id/$env{ID_BUS}-$env{ID_SERIAL}-event-$env{.INPUT_CLASS}"# by-path
ENV{.INPUT_CLASS}=="?*", KERNEL=="event*",
ENV{ID_PATH}=="?*",
SYMLINK+="input/by-path/$env{ID_PATH}-event-$env{.INPUT_CLASS}"# …LABEL="persistent_input_end"
We can see multiple things from this short example. First of all,
the branching mechanism with its use of
GOTO
whenever certain matches
don’t fit the specific use-case. We can also see the standard comparison
operators such as
==
and
!=
.
Then we see different variables/values that are either compared against
such as
SUBSYSTEM
,
ACTION
,
KERNEL
,
ATTRS{…}
,
ENV{}
, or
assigned such as
ENV{…}
,
GOTO
, or
SYMLINK
. The assignment seems
to either use
=
or
+=
.
Furthermore, from this example we can also see some regex-like pattern
matching, and string substitution within assignment.
Yet, overall the idea makes sense. We create some string variable based
on what type of input device we’re dealing with (prepended with
.
means it’s only temporary), which we found in
ENV{…}
, the device
properties. Then for event devices we create two symlink files in
different directories “by-id” and “by-path”. For the by-id it’s composed
of the bus name, followed by the device name, “-event-“, and the input
class we’ve stored in the temporary variable.
The lines starting with
E:
are device properties that are in
ENV{…}
,
the meaning can be found in
udevadm(8)
manpage, which we’ll see more
of in other examples.
So from this, the device should be symlinked as
/dev/input/by-id/usb-SEMICO_USB_Keyboard-event-kbd
, which it indeed is.
That’s a neat example, it gives us a generic idea of udev. Let’s continue
and try to get a more general idea of the udev syntax.
So far we’ve seen that the rules files contain key-value pairs, or
comments starting with
#
as is standard in most conf files, and
has operators that are either for comparison,
==
and
!=
, or for
assignment, we’ve seen
=
and
+=
.
The difference between these two assignment operators is that some
variables/keys are lists, and the
+=
appends to that list, while the
=
operator would basically empty the list and set only the single
value in it. Additionally, there are two other assignment operators
we haven’t seen: the
-=
to remove a value from a list, and the
:=
which sets a constant and disallow future change.
How to know if something is a list or a scalar value, and if the key can
be used in comparison or assignment. Well, it depends on the key itself,
which are listed in the man page
udev(7)
, we’ll see the most common
but first let’s talk about the values.
The values assigned are always strings within double quotes, and use
the usual same escape mechanism that C and other languages use. It also
allows case-insensitive comparison by having the string preceded by “i”,
such as
i"casedoesn't matter"
.
The strings also allow internal substitution with variables/keys, some
that can be set on the fly, from the match, or from a set of global
ones. It’s similar to a lot of languages:
"hello $kernel $env{ID_PATH}"
.
This is what we’ve seen in one of our examples.
Furthermore, if a string is used during matching, it can include
glob patterns, also the usual ones, such as
*
to match zero or more
characters,
?
to match a single characters,
|
for the or separator,
and
[]
to match a set of characters. Obviously, these special characters
will need to be escaped if used as-is.
Now, as we said there are keys used to do matching/searching, and keys
that allow assigning values (list or not), yet what’s confusing is that
lots of keys can be used for both, but not all of them. A quick look at
udev(7)
to be sure doesn’t hurt.
Here are some common matching keys:
KERNEL
: kernel name
SUBSYSTEM
: the kernel subsystem the device is associated to
DRIVER
: the driver currently handling the device
ACTION
: Represents what’s happening on a device. Either
add/remove
when the device is created or removed,
bind/unbind
for the driver,
change
when something happens on a device such as a state change (ex:
eject, power plug, brightness),
offline/online
for memory and cpu,
move
when a device is renamed.
ATTR{attributename}
: match any sysfs attribute of the device
TAG
: arbitrary tags, mostly used for user-space special behavior
ENV{property_name}
: Context info, device properties, added by the
kernel or other udev rules associated to device. They are not
environment variables, but do get passed as
env
to
RUN+=
commands.
PROGRAM
and
RESULT
: The first executes an external program and
if it’s successful then the match is ok, the second checks the string
result of the last program and uses it as a comparator.
Still, there are variants of some of the above to allow a match with any
of the parents of the devices in the topological hierarchy, these include
KERNELS
,
SUBSYSTEMS
,
DRIVERS
, and
ATTRS
.
Now, we’ve dealt with the keys used for comparison, let’s see the common
assignment keys:
SYMLINK
: A list of symlinks to be created
ATTR{attributename}
: Value that should be set in sysfs
TAG
: A list of special attributes for user-space to act
upon. For example, systemd acts on
TAG+="systemd"
and will read
ENV{SYSTEMD_WANTS}
and interpret it as a unit dependency for the
device. It can be used to automatically start services.
ENV{property_name}
: Context info, device properties, of the device. If
the property name is prepended with a dot
.
, then it will only
temporarily be set.
OWNER
,
GROUP
,
MODE
: Set permissions on the device
RUN{type}
: A list of external programs to run. The type is optional
and defaults to “program”, but it can be “builtin”, which are
plugins. Beware that
RUN
will timeout, and so it’s always better to
dispatch long running process to starter scripts instead that will exit
directly.
systemd-run --user
is often used here to execute things
in a normal graphical session such as notifications.
IMPORT{type}
: Similar to
RUN
but used to import a set of variables (
ENV
)
depending on the type, can be “program”, “builtin”, “file”, “db”,
“parent”, “cmdline”.
LABEL
,
GOTO
: A label and goto to jump to it, creating branching.
The
RUN{builtin}
is a bit of an edge-case within udev since there are
many builtin modules and most of them are blackboxes that are hardly
documented. We know from
udevadm test-builtin --help
that these exist:
blkid Filesystem and partition probing
btrfs btrfs volume management
dissect_image Dissect Disk Images
factory_reset Factory Reset Mode
hwdb Hardware database
input_id Input device properties
keyboard Keyboard scancode mapping and touchpad/pointingstick characteristics
kmod Kernel module loader
net_driver Set driver for network device
net_id Network device properties
net_setup_link Configure network link
path_id Compose persistent device path
uaccess Manage device node user ACL
usb_id USB device properties
Unfortunately, what they do isn’t clear unless you step in the code of
udev-builtin
.
For example,
input_id
will set a series of
ENV
info on the device depending on what it thinks
it is. Here’s some relevant code snippet:
And, that’s the tip of the iceberg to understand udev rules. Yet, the ones on
a real system are a monstrously big patchup. The only way to visualize
all of them on your system, in the way they’ll be processed, is with
systemd-analyze cat-config udev/rules.d
.
Before getting on with actual examples and tools, let’s take some time
to talk about one of the most important builtin module to udev:
hwdb
,
the harware db, or
systemd-hwdb
. Which is an extra mechanism to write
rules for udev to add device properties (
ENV{}
).
The hardware db is a lookup table that lives in files with the
.hwdb
extension under the udev directory in the
hwdb.d
directory. These
key-values at
systemd-hwdb
start are compiled in a
hwdb.bin
file for
quick retrieval. They consist of matches of modalias-like keys and then
a series of assignment for properties. Something like:
The format is a simple series of match strings, one or multiple, and then
assignment values following it on lines that start with a space. Match
strings can use glob for the match, they’re not really following any
specific format other than
prefix:search criteria
. Yet, the question is:
how are these modalias-like strings used. And the answer is obviously:
it’s used by udev via its
IMPORT
of the builtin hwdb to set certain
device properties based on the lookup. For example:
So udev passes a set of parameters to hwdb, along with the device, and it
will return
ENV
properties to set. hwdb also has an accompanying command
line tool that works in a similar way and allows querying it. However, it
has no man page, as far as I can see, but the following args are allowed:
--filter or -f:
--device or -d:
--subsystem or -s:
--lookup-prefix-p:
So for example when passing
--subsystem=usb
and a device, hwdb will get
the actual
MODALIAS
of the device, or construct one from the
idVendor
,
idProduct
, and
product
, then try to match it in its lookup table.
Anyhow, we won’t spend time breaking down the source code. Let’s just
add that since the
hwdb
lookup table is compiled at the start, then
when entries are added or modified
systemd-hwdb
needs to be updated
or notified via:
systemd-hwdb update # compile the hwdb
Similarly, the same is also true of udev. However, udev has more granular
reload mechanism, either to reload rules or to re-emit events so that
they can be processed by the new rules:
udevadm trigger # re-emits all the uevents
udevadm trigger /sys/class/input/eventXYZ # only re-emit this device events
udevadm control --reload# reload all rules but will only apply to new events
Let’s see more examples of
udevadm
, which is the main way to interface
with udev.
udevadm info
is used to gather information about devices, we’ve seen
it earlier in previous sections. It’s handy to write udev rules. You can
pass it either a devtmpfs path, a sysfs path, a device ID, or a systemd
unit name of
.device
type (these are the
TAG+="systemd"
devices to
automatically load other units).
For example, we can walk and find the attribute hierarchy of a certain
device.
Another option is to rely on
udevadm monitor
, which is a live trace
of all the uevent being sent.
Yet another option is
udevadm test
to print the rules that will get
triggered on a certain device uevent. This is useful to check whether
the rules make sense and will get executed.
A last tip to remember when writing udev rules is that
ATTR{}
is
anything in the files of sysfs. So we can simply match like this:
> cat /sys/class/input/event5/device/name
SEMICO USB Keyboard
And the rule would be
ATTR{name}=="SEMICO USB Keyboard"
.
Finally, let’s have a honorable mention to the mdev and eudev projects,
which are udev-like projects but more compatible with other init systems.
libinput
Libinput is a wrapper over udev and evdev. It provides a centralized way
to perform device detection, device event handling, input processing,
along with abstractions and common set of facilities to make the
practical, and user-expected, input handling easier. Today, libinput is
the major input library used by all graphical environments and toolkits,
it’s used by Xorg (through a driver) and Wayland compositors, so we’re
all probably using it indirectly.
Its basic mechanism works as you’d expect.
As far as udev is concerned, it relies on
libudev/sd-device
to
enumerate devices and listen to kernel’s uevent. In particular, it
analyzes properties added by udev that helps categorize devices and
override settings (
ID_INPUT
,
ID_INPUT_*
,
LIBINPUT_*
), and filters
which devices it is allowed to handle by looking at which “seat” they’re
associated with. The whole udev part can be skipped by manually passing
events with
libinput_path_add_device
, but that’s a fallback scenario.
And when it comes to evdev, it gets the handle to the corresponding
input stream devices then continuously read events and processes
them. This processing includes a lot of things such as scaling
touch coordinate, calculating pointer acceleration, debouncing
keys, etc.. Then finally, libinput returns these events in a unified
API as
LIBINPUT_EVENT_POINTER_BUTTON
,
LIBINPUT_EVENT_POINTER_MOTION
,
and
LIBINPUT_EVENT_POINTER_MOTION_ABSOLUTE
.
That also means it handles only the usual input devices such as mice,
keyboards, touchpads/clickpads, switches, trackpoints/pointing sticks,
touchscreens, and graphic tablets. It doesn’t handle joysticks, for
example, since these aren’t used for desktop environment but for games.
Scrolling, Three-finger drag, and Tap-to-click behaviour
Gestures
We’ll see what these means, but first, why is libinput needed, can’t
udev and evdev be handled directly? Why have another layer of indirection?
The answer is twofold: to avoid having additional separate modules in the
upper stack such as in the X server, and because handling input devices
is messy and not as simple as taking evdev events as-is, they need a
bit more interpretation and cleanup.
Previously, before Wayland got traction, the X11 stack had specific
custom drivers, the xf86 input driver API, for each type of hardware and
use-case. Yet, these xf86 drivers could also have common functionalities
such as two-finger scrolling, which adds confusion. This was mostly a
hack for days before evdev existed, and there was a need for a library
independent of X11 that would centralize this responsibility, instead
of having it dispersed in different places. This makes it easier to
test each options, and have the features interact with one another,
cross-device communication.
Now why not handle it all directly, well because it’s messy. Multiple
devices have bad firmware and might send wrong capabilities and info
in their HID Report Descriptors, which will then be forwarded as-is
with evdev. Plus, having handling these in the driver would be even
more messy. For example, it could say that the size or resolution of
the touchpad is something while it’s something else. Or that the range
of valid inputs is 0 to 10 but that it’s 5-10. That’s why libinput
includes vendor-specific quirks handling in
/usr/share/libinput/
along with the help of hwdb, which we’ve seen earlier, that has
/usr/lib/udev/hwdb.d/60-evdev.hwdb
.
And this says that when a keyboard’s bus is over Bluetooth, it should
add the libinput attribute to say it’s an external keyboard.
The
60-evdev.hwdb
is mostly for touchpad’s axis, the device properties
set will look like this:
EVDEV_ABS_<axis>=<min>:<max>:<res>:<fuzz>:<flat>
# where <axis> is the hexadecimal EV_ABS code as listed in linux/input.h and# min, max, res, fuzz, flat are the decimal values to the respective fields of# the struct input_absinfo as listed in linux/input.h. If a field is missing# the field will be left as-is. Not all fields need to be present. e.g. ::45# sets the resolution to 45 units/mm.# resolution: it is given in units per millimeter and thus tells us the# size of the device. in the above case: (5112 - 1024)/42 means the device# is 97mm wide. The resolution is quite commonly wrong, a lot of axis# overrides need the resolution changed to the correct value.
Furthermore, apart from quirks, there are hardware physical issues,
such as the fact that some touchpads send out events before the finger
even touches them, or how to handle the difference in pressure on them,
or what to do to track different fingers on multitouch (MT) hardware
which requires handling evdev tracking ID and slots.
Here’s a two-fingers scroll example, see how complex that is:
Additionally, you also have misbehaving keyboards, with bad
firmware, buttons that are old, that get stuck, or send the same
events multiple time (so-called contact bouncing or chatter). We
need a mechanism to decide whether the event is valid or not, that’s
called hardware debouncing, and libinput does it out-of-the-box for us
(
see
),
which is truly impressive. This feature, with the help of the upper stack,
may also help people that have certain disabilities with involuntary
muscle movement.
So, for many reasons, libinput is indispensable!
We’ve already covered some of its features, let’s see more.
One of the interesting part of libinput is that it’s minimal in how
it decides to access external things. As we said, you can either opt
for events coming from udev, or manually pass them by path, both will
create libinput internal objects (pointer, keyboard, etc..). Furthermore,
libinput has no configuration files, it’s up to the caller to decide
how to configure each device, as we’ll see Wayland compositors and X11
have different ways. Similarly, it leaves the opening of evdev character
devices up to the caller implementation, usually either manually opening
it, which requires root privileges, or via
systemd-logind
or
seatd
, dbus
services which will automatically pass back the file descriptors of evdev
devices associated with the current “seat”.
A seat is a collection of input devices associated with a user
session. That seems redundant, since most machines have only one seat,
yet it only truly makes sense in multi-seat machines: one machine,
multiple input devices, with multiple users. Still, it takes this
particular use-case in consideration.
> libinput list-devices.
…
Device: SEMICO USB Keyboard
Kernel: /dev/input/event5
Id: usb:1a2c:6004
Group: 6
Seat: seat0, default
Capabilities: keyboard
…
> loginctl seat status # will list all input in the hierarchy
As you would’ve guessed, the safest and most favored way to get access
to evdev event file descriptors is through the delegation that
systemd-logind
provides. This is done in the code by implementing
open_restricted
to call the dbus service.
The seat is assigned with the
ENV{ID_SEAT}
udev property, which can be
controlled with the
loginctl
command. To permanently attach a device
to a seat.
There are alternatives to
logind
such as
elogind
and
seatd
that don’t depend on
systemd.
Another detail is that we’ve seen that the same physical device can
appear as multiple input devices on the system. With the help of udev,
libinput gets the device property
LIBINPUT_DEVICE_GROUP
to group them,
like that we can have the whole group under a single seat, which is more
logical than giving access to only part of a physical hardware.
And from
libinput list-devices
, look at the
Group
part:
Device: SEMICO USB Keyboard
Kernel: /dev/input/event5
Id: usb:1a2c:6004
Group: 6
Seat: seat0, default
Capabilities: keyboard
…
Device: SEMICO USB Keyboard Consumer Control
Kernel: /dev/input/event6
Id: usb:1a2c:6004
Group: 6
Seat: seat0, default
Capabilities: keyboard pointer
…
Device: SEMICO USB Keyboard System Control
Kernel: /dev/input/event7
Id: usb:1a2c:6004
Group: 6
Seat: seat0, default
Capabilities: keyboard
You can get more info on this by checking the related udev rule in
80-libinput-device-groups.rules
, which calls the built-in program
libinput-device-group
with the sysfs mount point. The
IMPORT{program}
basically uses a program right within
/usr/lib/udev/
directory.
As far as the technical features are concerned, there are the ones which
we listed earlier, so let’s explain the rest of them.
It offers full clickpad management. A clickpad (
INPUT_PROP_BUTTONPAD
)
is basically a touchpad with a single button, which we might not notice at
first because depending on where we press in the “software button area”
at the bottom, we have different behavior. That’s exactly the behavior
that libinput facilitates. It also handles what happens when a finger
enters or exits that area, these sort of edge cases.
Furthermore, libinput handles tap to click, be it one-finger tap for
left click, two-fingers for right click, and three-fingers tap for
middle click. While that seems simple in theory, libinput has to draw
the line between what is considered a tap and what is considered a finger
drag/move; indeed, our fingers aren’t very stable in the real world.
Unfortunately, by default libinput disables tapping when there are other
methods to trigger button clicks, but it can always be enabled again.
When talking about multiple fingers, the hardware needs to support
it obviously, but also libinput needs to track each one individually,
which is done via evdev tracking ID and slots, what we call multi-touch
handling or MT.
Within multi-touch we have the concept of “gestures” and libinput supports
two standard ones: swiping, fingers going in the same direction, and
pinching when fingers move apart or towards each others.
Similarly, there’s also different scrolling use-cases that are supported
by libinput: two-fingers scrolling, similar to a swipe, edge scrolling,
when there’s a specific area on the trackpad used for scrolling, and
on-button scrolling, which scrolls while having a button pressed just by
moving the finger.
The scrolling can either be horizontal or vertical. The user also has
a choice between natural scrolling an traditional scrolling; natural
scrolling matches the motion of the scroll like a phone, and traditional
scrolling matches the scroll bar directin so going downward will move
the page downward.
One thing libinput doesn’t provide when it comes to scrolling is kinetic
scrolling. Basically, scrolling that is faster or slower depending on
the speed. However, it allows widget libraries to implement it by relying
on the
libinput_event_pointer_get_axis_source()
function.
With all these, libinput offers palm and thumb detection to disable
the clickpad/touchpad when typing, or ignore a thumb in the corner or
accidental touches while other fingers are moving. It achieves this by
detecting the different pressure, speed, or touch sizes reported by evdev,
along with where they are happening (exclusion zones).
It’s also possible to automatically disable the touchpad when typing,
or when the lid is closed.
Lastly, libinput has lua plugins in
/usr/lib/libinput/plugins/
and
/etc/libinput/plugins
. As with other quirk fixing mechanisms in
udev and the quirk directory, the plugins are there for the last few
unfixable issues. They can be used to override evdev events.
libinput:register(1)-- register plugin version 1libinput:connect("new-evdev-device",function(_,device)ifdevice:vid()==0x046Danddevice:pid()==0xC548thendevice:connect("evdev-frame",function(_,frame)for_,eventinipairs(frame.events)doifevent.type==evdev.EV_RELand(event.code==evdev.REL_HWHEELorevent.code==evdev.REL_HWHEEL_HI_RES)thenevent.value=-event.valueendendreturnframeend)endend)
For example, the above script will reverse the horizontal scroll wheel
(
EV_REL.REL_HWHEEL
) event value for a certain device vendor and
product ID.
We’ve covered most of the libinput features, now let’s see how to debug
and interface with it.
The main command line interface is
libinput
, as we’ve seen it can allow
to
list-devices
, which is a quick summary of the devices it knows about
and on which seat they are connected. Yet most other commands are there
for debugging and testing.
libinput debug-gui
: is a graphical tool mostly to debug touchpad
libinput debug-events
: is a cli tool to debug all events as they are
interpreted by libinput, if you want it’s similar to
evtest
or
xev
in Xorg
libinput record
and
libinput replay
: Used to save and then simulate
again devices. This is amazing if you have a bug and want others to be
able to replicate it on their machines. This is similar to how
hid-tools
work.
libinput measure
: mostly used for touchpad, to measure things such
as pressure, touch size, tap to click time, etc..
The other way to interface with libinput is programmatically. Here’s
the most simple complete example I could come up with:
#include<libinput.h>
#include<libudev.h>
#include<fcntl.h>
#include<unistd.h>
#include<stdio.h>
#include<errno.h>
#include<string.h>staticintopen_restricted(constchar*path,intflags,void*user_data){intfd=open(path,flags);if(fd<0)fprintf(stderr,"Failed to open %s (%s)\n",path,strerror(errno));returnfd;}staticvoidclose_restricted(intfd,void*user_data){close(fd);}staticconststructlibinput_interfaceinterface={.open_restricted=open_restricted,.close_restricted=close_restricted,};intmain(void){structudev*udev=udev_new();if(!udev){fprintf(stderr,"Failed to create udev\n");return1;}structlibinput*li=libinput_udev_create_context(&interface,NULL,udev);if(!li){fprintf(stderr,"Failed to create libinput context\n");return1;}if(libinput_udev_assign_seat(li,"seat0")!=0){fprintf(stderr,"Failed to assign seat\n");return1;}structlibinput_event*event;while(1){libinput_dispatch(li);event=libinput_get_event(li);if(!event){usleep(10000);continue;}if(libinput_event_get_type(event)==LIBINPUT_EVENT_KEYBOARD_KEY){structlibinput_event_keyboard*k=libinput_event_get_keyboard_event(event);uint32_tkey=libinput_event_keyboard_get_key(k);enumlibinput_key_statestate=libinput_event_keyboard_get_key_state(k);printf("Key %u is %s\n",key,state==LIBINPUT_KEY_STATE_PRESSED?"PRESSED":"RELEASED");}libinput_event_destroy(event);}libinput_unref(li);udev_unref(udev);return0;}
But what about configuring devices, setting up things that we want to
setup per device. Well, as we’ve said this is done in the upper stack
since libinput has no configuration files, and we’ll cover this later. For
now let’s just list a few of the things that can actually be configured.
tap-to-click related, such as how many fingers are supported
three-finger drag
pointer acceleration profiles
scrolling method natural vs traditional
left-hand mode
middle button emulation
click method
disable while typing (DWT)
disable while trackpointing (DWTP)
direct-input device calibration
rotation confs (if touchpad is sideways)
area confs
You can glimpse at these on X11 with the command
xinput --list-props <device_id>
or at
libinput list-devices
which we’ve seen earlier that
should show the conf per-device:
That’s about it when it comes to libinput. Now we can move to more
specific things in the upper stack.
Keyboard Specifics
We’re pretty much done with the lower part of the user-space stack, but
before moving on to the graphical library widgets and desktop environments,
let’s take some time to see some of the specific device handling that
are good to know about, namely keyboards, mice, and gamepads.
In this section we’ll see three important concepts related to keyboards:
scancodes to keycodes, console keyboard handling, and XKB.
Scancodes to Keycodes
Like other input drivers, the role of keyboard drivers is to translate
from raw hardware keys to events that can be normalized and interpreted by
user-space. We call the raw keys scancodes, and the events ones keycodes
(
/usr/include/linux/input-event-codes.h
). Keycodes are also mapped
to key symbols in user-space unrelated to their actual keycodes, which we
call keysyms.
For example, a scancode can look like a random hex
0x1E
, a kernel-mapped
event as
KEY_A
, and a keysym will look like a symbol such as “a” or “A”.
We’ll talk more about keysyms mapping when we see XKB. But let’s focus
on the scancodes to keycode translation for now.
When a keyboard input device registers itself in the input core
(
input_register_device
) it has to report which keycodes it supports
in its capabilities (
keybit
capability). In general it has to set its
keycode
,
keycodemax
, and
keycodesize
fields, which are a map of
the translation of scancodes to keycodes.
These keymaps can either be full fledge dense keymap or sparse keymap,
which means they’re smaller and use less memory. The sparse keys are
mostly used when registering a few entries such as special keys that
don’t need huge arrays.
If a scancode isn’t found in these translation arrays, they’re often
either completely ignored, or the driver returns that it’s an unknown key.
Keyboard input devices can also optionally implement two important
functions:
getkeycode
and
setkeycode
, which will by default retrieve
the current keymap and alter the current keymap respectively. Most drivers
fallback to the default mechanism, so this can be taken for granted.
Importantly, the
evdev
and
kbd
(console) handlers offer ways to call
these via ioctl interfaces, which will be propagated to the devices
they’re currently handling. For
evdev
it’s through
EVIOCGKEYCODE
and
EVIOCSKEYCODE
, to get and set keycodes respectively. For the console
handler it’s through
KDGETKEYCODE
and
KDSETKEYCODE
. The exception
is that the console driver will propagate it to all handlers, and thus
indirectly to all devices on the platform.
You can also do the runtime patching of scancode to keycode
mapping through udev and hwdb by setting a device property in
ENV{KEYBOARD_KEY_<hex scan code>}=<key code identifier>
which will in
turn be caught by
systemd/src/udev/udev-builtin-keyboard.c
and also
call the same ioctl interfaces.
For example:
ENV{KEYBOARD_KEY_b4}=dollar
To find out the actual scancodes the device is generating the
showkey(1)
tool from the Linux Keyboard tools project, with the
--scancodes
flag,
will attach to the console handler and display them in raw mode. And the
setkeycodes(8)
command from the same project will propagate it to the
driver via the console input handler.
There are multiple other tools used to do the keycode remapping such
as
evmapd
,
evremap
,
evdevremapkeys
, but these work at the evdev
layer and don’t know about scancodes. So for now, the simplest one to
do scancode to keycode mapping is obviously the built-in one: hwdb.
This mechanism for runtime modifications might save us time instead of
getting our hands dirty and having to modify kernel drivers.
Console Keyboard
We’ve discussed the
evdev
handler extensively, however in the console
it’s the
kbd
input event handler (
drivers/tty/vt/keyboard.c
) that is
used, and it’s working in sync with the tty and line discipline mechanism.
This particular handler exposes its devices as TTYs in the infamous
/dev/ttyN
and
/dev/console
(system) and handles all the messiness
of console text-mode input.
The input handlers coexist. When switching from graphical environment
to console, the VT
kbd
handler takes over.
Obviously, as a console input handler, the
kbd
handler has much more
work to do, and it has a lot of special handling via ioctl too. From
bell and tone, leds, setting console key rate, modifiers, interpreting
special keys that have meanings for the TTY, switching to other modes
such as raw input mode or graphic (X11, Wayland session), all the push
towards line discipline, etc.. It’s handling things that are often
handled in user-space (key rate is handled in the graphical stack too
as we’ll see). The reason: historical entangling, the console existed
before the graphical stack.
For instance,
showkey(1)
which we’ve just seen, relies on changing the
mode of the terminal via ioctl
KDSKBMODE
to
K_RAW
.
There are a bunch of commands to interface with the ioctl such as
kbdrate(8)
to set the keyboard rate,
kbdinfo(1)
to get more info
about the kbd driver, and
kbd_mode(1)
to get the current keyboard mode
(raw, xlate, etc..)
Furthermore, since it’s taking a bigger role on handling scancode to
keycode, it also somewhat does keycode interpretation via its internal
keymap. That means, the
kbd
handler can be responsible of handling the
difference in regional keyboard layouts and special keys. This is
something which usually happens in XKB, in user-space, which we’ll see
in the next section.
Thus it has two sets of ioctl:
KDSETKEYCODE
and
KDGETKEYCODE
for
low-level scancodes to keycodes, and
KDGKBENT
and
KDSKBENT
for the
keycode to symbol/action mapping (internally also confusingly called
key_maps
, as you’ll see everyone uses the word “keymap”).
The format of the keymaps translating keycode to symbol (
keymaps(5)
)
is managed by the kernel for each console, but usually more
easily set with user-space tools also from the
Linux keyboard
tools
project. For example
loadkeys(1)
and
dumpkeys(1)
. These can rely on files in
/usr/share/kbd/keymaps/
for a predefined set of keymaps. Let’s also mention that the default
one is found in
/usr/src/linux/drivers/tty/vt/defkeymap.map
.
Before we end, let’s mention
systemd-localed.service(8)
and its
localectl(1)
command. It is used to set the keyboard map for
both the console and the graphical environment (XKB in X11 as we’ll see)
based on the current locale. For example, it sets the keymap,
font, and others, of the console and X11 XKB to the value found in
/etc/vconsole.conf
(see
vconsole.conf(5)
) through its service called
systemd-vconsole-setup(8)
, which is also called when the console is
initialized with udev. It can also help in setting the same values in
both the console and graphical stack.
Here’s
vconsole.conf
:
KEYMAP=us
XKBLAYOUT=us
> localectl
System Locale: LANG=en_US.UTF-8
VC Keymap: us
X11 Layout: us
NB: Terminal emulators don’t rely on the console input handler at all,
they use pseudo-terminals instead (PTYs). These don’t have VGA console,
nor plug to the kbd handler, nor screen, etc.. They are fed entirely by
user-space programs.
Example:
Line discipline <-> TTY driver (PTY slave side) <-> user process
`-> PTY master side <-> xterm process
Now let’s see how the keycode to keysym is done in user-space in the
graphical stack with XKB.
XKB
XKB, or X keyboard, is a common library (xkbcommon, xkbregistry,
xkbcompose) with a set of tools, an X11 protocol extension (X
Keyboard Extension), and a database collection of descriptions
(xkeyboard-config). Its role is to handle the keycode to keysym
translation in user-space.
While the name includes “X”, the common library and database are not only
used by Xorg but by most software, including graphical widgets such as
GTK and Qt, and Wayland compositors. We won’t cover the older X protocol
extension here, yet the reason why there’s an “X” in the name is that
it started as an extension and then got separated into a common library.
The two things we’ll focus on are xkbcommon, the xkb core engine that
parse and executes XKB definitions, and the xkeyboard-config, which
is a project compiling a database of keyboard info, layouts, variants,
symbols, and rules. They work together.
As a word of notice, XKB is one of the most complex piece of software I’ve
encountered and its documentation is fiercely lacking and dispersed. It
has its own language, compiler, and the format is extremely convoluted
and inconsistent, often mixing camel case and snake case for no apparent
reasons.
Even in the XKB documentation we find such comments:
Todo
Explain how to configure XKB, with examples
Due to the complexity of the format, this document is still is
construction.
And internally Xorg devs called it
“X Kitten Butcher”
.
We’ll try to make it approachable, and break the bad
spell. However, if you ever want more info check
the official
format
.
In order to perform the translation from keycodes coming from event
handlers to actual symbols, XKB relies on something called an XKB keymap
(yes everything is called a keymap). This XKB keymap is a compilation
of different components coming from the xkeyboard-config database that
are chosen based on the abstract, and more coherent, concept of layout,
variants, models, and options the user pick:
“RMLVO”
.
After this is picked, the XKB client software just has to keep track of
what’s called a state, and then send it along with the received keycode
to receive back the keysym.
A very basic example looks like this:
// create a complete keymap from the xkeyboard-config dbstructxkb_keymap*keymap;// .. and a state object to keep track of what special state we're in// that could affect the keysym outputstructxkb_state*state;…state=xkb_state_new(keymap);xkb_state_update_key(state,keycode,pressed?XKB_KEY_DOWN:XKB_KEY_UP);xkb_keysym_tsym=xkb_state_key_get_one_sym(state,keycode);
The XKB state object tracks what affects the output of keycode to keysym,
things like modifiers and groups. This example doesn’t mention the idea
of key composing, but we’ll come back to it.
This is important to understand, since you can either have XKB handle
what happens in a specific state when a key is pressed, or do it from
the client side. For example, a client can choose to catch all Ctrl keys
and interpret Ctrl+h as backspace, or leave it up to XKB with a custom
mechanism to know what Ctrl+h means, and the client will receive back
the keysym for backspace directly, with no special handling from its side.
Yet, the downside is that this key combination will apply to everyone
that relies on this XKB keymap.
Before moving forward, we need a little baggage of definitions, and
understanding, otherwise nothing will make sense.
evdev keycodes: the events coming from evdev, the ones listed in
/usr/include/linux/input-event-codes.h
XKB keysyms: Actual symbols (or dead key), actions, and special keys
that XKB will return, they exist in
/usr/include/xkbcommon/xkbcommon-keysyms.h
Modifier Keys: Special keys that can affect other keys such as shift,
alt, ctrl, “win”, etc.. Modifiers are also keysyms.
Geometry: The physical layout of a keyboard, what it looks like and
where the keys are
Levels and Groups: Levels is another state a key could be in when you
press a modifier. For example, it’s expected that pressing shift with
“a” will output “A”, upper case “A” is the level 2 of what happens when
pressing the key. A Group is similar but it completely switches the
whole keyboard to another key mapping, as if you switched variants.
As you can imagine, there’s a lot at play with levels, groups, modifiers,
and actions that can happen, and that’s apart from the basic idea of
keycodes to keysym.
Even when it comes to keysym, the translation isn’t straight away. XKB
relies on intermediary objects.
XKB keycodes are not straight up evdev keycodes, but
evdev keycodes + 8
. Why 8, someone might ask. Well, the only answer
is backward compatibility from before evdev was a thing, and it’s
still there.
Furthermore, XKB converts these keycodes into physical key
positions values that are compatible with ISO/IEC 9995-1. So we
move from evdev keycodes, to XKB keycodes, to physical abstract
position on a keyboard layout. This is what happens in the keycode
component files under
/usr/share/xkeyboard-config-2/keycodes/
. Keycodes
have this form within
<...>
tags. For example:
Or basically the first row from ISO/IEC 9995-1 on a keyboard.
To make it easier for users to pick an XKB keymap, without
having to know much details, the idea of picking only RMLVO,
Rules-Model-Layout-Variant-Options, was invented. This is an abstraction
on top to pick the components that make up a keymap, and thus come up with
the right keyboard behavior expected by the user. This is managed by the
XKB registry, which graphical environments interact with, this is what
is shown to the user when they’re asked about picking their keyboard
layout, the list of possible layouts and variants on those layouts,
along with special options.
Model – the name of the model of your keyboard
Layout – the layout(s) you intend to use (usually refer to country code)
Variant – the variant(s) of the layout(s) you intend to use (minor
and national variants)
Options – extra XKB configuration options to customize the standard
layout. For example to change modifier keys.
To know what’s actually picked as the final keymap, what’s called KcCGST,
we can run
xkbcli
. For example, for a dvorak keyboard, or a normal
qwerty keyboard:
> xkbcli compile-keymap --kccgst\--layout us \--variant dvorak \--options terminate:ctrl_alt_bksp
xkb_keymap {
xkb_keycodes { include "evdev+aliases(qwerty)"};
xkb_types { include "complete"};
xkb_compat { include "complete"};
xkb_symbols { include "pc+us(dvorak)+inet(evdev)+terminate(ctrl_alt_bksp)"};
xkb_geometry { include "pc(pc105)"};};> xkbcli compile-keymap --kccgst\--layout us \--variant qwerty \--options terminate:ctrl_alt_bksp
xkb_keymap {
xkb_keycodes { include "evdev+aliases(qwerty)"};
xkb_types { include "complete"};
xkb_compat { include "complete"};
xkb_symbols { include "pc+us(qwerty)+inet(evdev)+terminate(ctrl_alt_bksp)"};
xkb_geometry { include "pc(pc105)"};};
We’ll revisit the RMLVO, let’s just say it’s all about what the “rule”
part refers to: a lookup table with rules mapping the abstract names to
the components of the keymaps which are called KcCGST.
KcCGST, or the Keycodes, Compat, Geometry, Symbols, Types, are the
component parts of an XKB keymap. This is the actual functional XKB
configuration that is used behind the RMLVO easy facade. In general, XKB
considers it an implementation detail and pushes for users to favor
configuring XKB through RMLVO. Yet, it’s the core of XKB!
The resolution of the RMLVO will create a complete keymap, a self-contain
object that has all the related KcCGST components assembled together. This
complete XKB keymap is what is used by the clients.
To get a quick glimpse at what a full resolved keymap looks like, try
this command:
> xkbcli compile-keymap --layout us --rules evdev
Or for a more compact one, look again at the command such as the one we
just did before:
> xkbcli compile-keymap --kccgst--layout us --options terminate:ctrl_alt_bksp
xkb_keymap {
xkb_keycodes { include "evdev+aliases(qwerty)"};
xkb_types { include "complete"};
xkb_compat { include "complete"};
xkb_symbols { include "pc+us+inet(evdev)+terminate(ctrl_alt_bksp)"};
xkb_geometry { include "pc(pc105)"};};
Let’s go over these components and explain them.
First of, the KcCGST configurations that come from the keyboard-config
project are often found in the following places in reverse order of
precedence, with the component bundled underneath:
Most of the components have a useful utility. That’s apart from the
geometry, which is a complex file used to describe what a keyboard
physical layout looks like. It’s not used in the latest xkbcommon
mechanism though, so we’ll skip explaining it.
The XKB configuration format has types: string, numbers, key positions,
and keysym:
"hello"
,
"%S/pc"
42
,
134
<AE12>
,
<BKSP>
percent
,
a
,
A
It also has many special keywords, and some structure format. The main
structural format is called a component, basically the components of
the KcCGST. Each XKB conf file is an aggregation of multiple of these
components. They have the form:
default
: One of these “variant” per component file, the default
values to be used
partial
: To be used in another conf
hidden
: Only used internally within the file’s scope
And the symbols flags can be one or many of these:
alphanumeric_keys
modifier_keys
keypad_keys
function_keys
alternate_group
The symbols flags are mostly metadata and don’t affect the XKB processing.
They’re indicators of what the component configuration covers, and if
none are present it’s assumed it covers a complete keyboard.
Let’s start with the most important keywords, the ones used to import
and merge files together, we’ve seen the
include
. It works by finding
the file of the same component with the specified name, if it exists
in any of the valid conf paths (or if explicitly mentioned with string
substitution shorthands), and then look for the variants inside or the
default value if none are passed:
include "file(variant)"
.
The
include
will override any information that already exists:
that is if new values are undefined it will keep the old one, but new
defined values will always override old ones. To avoid this, the
augment
"file(variant)"
should be used instead, it will update the properties
that are undefined, but keep the defined ones (it’s the reverse). Another
option is the
replace "file(variant)"
which will, as the name implies,
completely replace the full properties, regardless if some elements are
defined or not.
This “merge resolution” mechanism also applies to values within the
components objects, which can be tagged with
augment
,
override
,
replace
, too.
As for files, a shorthand exists to have a single statement with multiple
includes concatenated. In this case the following merge mode prefixes
are used:
+
selects the override merge mode (default).
|
selects the augment merge mode.
^
selects the replace merge mode.
So you can now understand why the following line we’ve seen works, and
how it creates an inheritance mechanism, plugging multiple files together:
Let’s now explain what each component does, and wrap up with how the
rules mechanism of the RMLVO then resolves them into an XKB full keymap.
The keycodes file is the most obvious one and the first entry-point
for XKB logic, it translates from XKB keycodes to the physical codes
ISO/IEC 9995-1. The syntax of the components looks something like this:
defaultxkb_keycodes"mykeycode"{// defining the rangeminimum=8;maximum=255;// mapping of keycodes to layout keys<TAB>=23;<AD01>=24;<AD02>=25;<AD03>=26;<AD04>=27;<AD05>=28;<AD06>=29;<BKSL>=51;<RTRN>=36;// making one physical key name equivalent to anotheralias<LatQ>=<AD01>;alias<LatW>=<AD02>;alias<LatE>=<AD03>;alias<LatR>=<AD04>;alias<LatT>=<AD05>;alias<LatY>=<AD06>;// these are for LEDs, not always used by clientsindicator1="Caps Lock";indicator2="Num Lock";indicator3="Scroll Lock";};
The syntax is straight forward, it’s a couple of assignment, with the
possibility to have aliases, and giving names to LEDs, indicators, which
aren’t really leds afaik but keys that lock or latch. By convention it
explicitly names special keys, but other keys as their ISO positions.
Here’s a standard keyboard with its key positions:
Let’s move to the types component. This is where the information about
levels, and how to switch between them is defined.
virtual_modifiersNumLock;type"ONE_LEVEL"{modifiers=None;map[None]=Level1;level_name[Level1]="Any";};type"TWO_LEVEL"{modifiers=Shift;map[Shift]=Level2;level_name[Level1]="Base";level_name[Level2]="Shift";};type"ALPHABETIC"{modifiers=Shift+Lock;map[Shift]=Level2;map[Lock]=Level2;level_name[Level1]="Base";level_name[Level2]="Caps";};// override ALPHABETIC Shift will cancel capslockoverridetype"ALPHABETIC"{modifiers=Shift+Lock;map[Shift]=Level2;preserve[Lock]=Lock;level_name[Level1]="Base";level_name[Level2]="Caps";};// override ALPHABETIC, Shift will ignore capslockoverridetype"ALPHABETIC"{modifiers=Shift;map[Shift]=Level2;level_name[Level1]="Base";level_name[Level2]="Caps";};// CapsLock acts as Shift with locking, Shift does not cancel CapsLock.type"ALPHABETIC"{modifiers=Shift+Lock;map[Shift]=Level2;map[Lock]=Level2;map[Shift+Lock]=Level2;level_name[Level1]="Base";level_name[Level2]="Caps";};
The syntax here is more cumbersome. Firstly, there are some definition
lines. In each
type
entry (which can be prepended with merge syntax
like anything else in this syntax really) of the form
type "name"
,
we have to define the modifiers that will be used as such:
modifiers=Shift+Lock;
The
+
, is just a separator here.
If the modifiers are not real keysym but virtual ones, then those virtual
modifiers also need to be defined earlier in the scope:
virtual_modifiersNumLock;
After defining the modifiers that are used for that type, we have a series
of mapping to define the combination and what levels these will achieve.
This has to do with how XKB consumes modifiers as it processes types
and outputs keysyms, its internal list of effective modifiers. Simply
said, without the
preserve
when the keysym is sent back to the client
(
xkb_state_key_get_one_sym
) the state object doesn’t consume the
modifier, so the client can inspect it for further special handling.
The logic within XKB clients looks something like this:
That’s useful for layout where you have, let’s say Greek letters for
Level1 and Level2, and at Level3 and Level4 there are the usual Latin
letters. So you’d want to preserve
Ctrl
and
Shift
, so that the
application can catch
Ctrl+c
for example, which would be in Level3
(Latin lower-case).
I’ve added different versions of the
ALPHABETIC
type in the example,
and how the capslock and shift combinations can affect letters.
Later on we’ll see how we assign the levels logic to symbols and
compatibility logic, but let’s just say that XKB will categorize keys
with a heuristic and assign them to default types if no other types were
explicitly chosen. These are:
"ONE_LEVEL"
: When there are only one level change for the keysym
"TWO_LEVEL"
: When there are exacly two levels change for the keysym
"ALPHABETIC"
: When the keysym is alphabetic and has two levels
"KEYPAD"
: For keypad keys of any level (two usually)
"FOUR_LEVEL_ALPHABETIC"
,
"FOUR_LEVEL_SEMIALPHABETIC"
, 3 to 4 keysym
"FOUR_LEVEL"
: When nothing else matches
The next component is the XKB compatibility, which is used to translate
key combinations into action statements. Actions can also be attached
directly in the XKB symbols component for each key, however it’s done in
the compatibility layer because it has a mechanism for generic pattern
matching of keysym combinations, so we don’t have to repeat the same
things in different places.
The actions that can be done in the XKB compatibility are varied
from latching/unlatching/locking/unlocking modifiers, changing level,
switching group, etc.. Many of these actions, however, only make sense in
combination with the XKB symbols component, so keep that in mind for now.
A compatibility map looks something like:
defaultxkb_compatibility"basic"{virtual_modifiersNumLock,AltGr;...interpret.repeat=False;setMods.clearLocks=True;...interpretShift_Lock+AnyOf(Shift+Lock){action=LockMods(modifiers=Shift);};...group2=AltGr;...indicator.allowExplicit=False;...indicator"Caps Lock"{whichModState=Locked;modifiers=Lock;};...};defaultpartialxkb_compatibility"pc"{// Sets the "Alt" virtual modifier.virtual_modifiersAlt;setMods.clearLocks=True;interpretAlt_L+Any{virtualModifier=Alt;action=SetMods(modifiers=modMapMods);};interpretAlt_R+Any{virtualModifier=Alt;action=SetMods(modifiers=modMapMods);};};
This has many components, the
interpret
sections to map keys to actions,
the virtual modifier definitions, indicators, repeat behavior of keys,
and more. The important part is the
interpret
section which matches
keysym along with a modifier (
AnyOfOrNone
,
AnyOf
,
Any
,
NoneOf
,
AllOf
,
Exactly
). The body of the interpret can also be more specific
by setting values of
useModMapMods
to match a certain level.
Default values to params can be set globally such as
setMods.clearLocks
,
which affects how
SetMods
and other mods actions behave.
The list of possibilities and actions within the compatibility is
too long to explain here, the list is extensive and can be found
here
.
Let’s move to the keysym or symbol component, which as you would
have guessed, finally maps physical keys in ISO location format to
symbols. These files are often named after countries or languages or
specific features,
us
,
jp
,
group
.
It first has a metadata name in the
name[GroupX] = "Symbols Name"
property, which can also be used to find which groups the symbols
belong to.
This is also where virtual modifiers are mapped to actual keys with the
modifier_map VIRTUAL_MOD { Symbol1, Symbol2}
.
And obviously, that’s where the
key <VAL>
are mapped to list of groups
within
{}
, and levels within
[]
.
key<TLDE>{[quoteleft,asciitilde]};
This means the physical key
<TLDE>
, in level1 will output a left quote
(backtick), and in level2 will output the tilde character.
Additionally, we can also specify within the curly brackets whether a
specific type should be used instead of the default matching one:
Similarly, the actions can be assigned here instead of in the
compatibility component, and the groups can also be explicitly expressed with
the syntax:
That all should cover the KcCGST component syntax. It’s very long already,
I know, yet it barely covers the basics. Let’s see a few examples to
grasp the concepts.
In
symbol/group
we have:
// The left Alt key (while pressed) chooses the next group.partialmodifier_keysxkb_symbols"lswitch"{key<LALT>{[Mode_switch,Multi_key]};};
And in
compat/basic
we have these
interpret
:
interpretMode_switch{action=SetGroup(group=+1);};
The
Multi_key
maps to a compose key in
compat/ledcompose
:
Here’s another example swapping the top row numbers on shift:
defaultpartialalphanumeric_keysxkb_symbols"basic"{include"us(basic)"name[Group1]="Banana (US)";key<AE01>{[exclam,1]};key<AE02>{[at,2]};key<AE03>{[numbersign,3]};key<AE04>{[dollar,4]};key<AE05>{[percent,5]};key<AE06>{[asciicircum,6]};key<AE07>{[ampersand,7]};key<AE08>{[asterisk,8]};key<AE09>{[parenleft,9]};key<AE10>{[parenright,0]};key<AE11>{[underscore,minus]};key<AE12>{[plus,equal]};};// Same as banana but map the euro sign to the 5 keypartialalphanumeric_keysxkb_symbols"orange"{include"banana(basic)"name[Group1]="Banana (Eurosign on 5)";include"eurosign(5)"};
Here’s a symbol component which replaces key “B” to have a third level
activated with the right alt to display a broccoli.
NB
: XKB has keysym to allow controlling the mouse pointer from the
keyboard, this can be useful if clients actually understand these keysym
and act on them.
It’s fine and all but we need the RMLVO so that the users can actually
use the keymap properly without bothering with all that we’ve seen.
The rules are in the
rules
directory as simple files without extensions,
and are accompanied with two listing files for GUI selectors:
*.lst
and
*.xml
that follow the
xkb.dtd
in the same directory. The listing
files are simply listing all the models, variants, layouts, and options
available, nothing more, and are used by the XKB registry library. That’s
in turn used by GUI selectors.
The logic exists within the rules files, that have this sort syntax:
! include %S/evdev
! option = symbols
custom:foo = +custom(bar)
custom:baz = +other(baz)
// One may use multiple MLVO components on the LHS
! layout option = symbols
be caps:digits_row = +capslock(digits_row)
fr caps:digits_row = +capslock(digits_row)
We won’t go into details, but basically it has lines starting with
!
that set certain MLVO values and then map them to KccgstValue specific
component values. There are also variable names that can be defined
as shorthand for multiple values with
$var = val1 val2
, and there
are string substitutions starting with
%
. More info can be found
here
.
So we’ve got the full scope now of RMLVO to KcCGST, the big picture!
We didn’t discuss another sub-feature of XKB called composing, or the
compose key processor. We didn’t mention it because the configuration
doesn’t come with the xkeyboard-config project. It’s loaded independently
by clients that want to perform composition.
For X11 the configuration is found under
/usr/share/X11/local/*/Compose
and
compose.dir
, and the home directory in
~/.XCompose
. The
content of this directory is mostly deprecated apart from the compose
definitions, which follows the
XKB_COMPOSE_FORMAT_TEXT_V1
format (see
Compose(5)
). It’s a simple format that looks like this:
As you can see, this is the
<Multi_key>
keysym we’ve talked about in
an earlier example, this is where it’s interpreted.
After editing any of the files, the syntax can be validated with
xkbcli
compile-compose
.
The way the file is used is that clients will pass it to the XKB compose
parser to get an in-memory table of it. Then the client keeps the compose
state, just like the modifier state, and plug it in the main interaction
with XKB we’ve seen earlier. Like this:
// 1. Load compose table (locale-dependent)structxkb_compose_table*table=xkb_compose_table_new_from_locale(ctx,getenv("LANG"),XKB_COMPOSE_COMPILE_NO_FLAGS);// 2. Create a compose statestructxkb_compose_state*compose=xkb_compose_state_new(table,XKB_COMPOSE_STATE_NO_FLAGS);// 3. For each key press:xkb_keysym_tsym=xkb_state_key_get_one_sym(state,keycode);xkb_compose_feed_resultres=xkb_compose_state_feed(compose,sym);// Feed all keysyms into the compose engine:xkb_compose_state_feed(compose_state,sym);// 4. Check compose statusswitch(xkb_compose_state_get_status(compose_state)){caseXKB_COMPOSE_COMPOSED:composed_sym=xkb_compose_state_get_one_sym(compose_state);// Use composed_sym; DO NOT use 'sym'// char buf[64];// xkb_compose_state_get_utf8(compose_state, buf, sizeof(buf));// printf("→ composed result: %s\n", buf);break;caseXKB_COMPOSE_CANCELLED:// Typically fall back to original symbreak;caseXKB_COMPOSE_COMPOSING:// Wait for next keybreak;caseXKB_COMPOSE_NOTHING:// No composition; use raw 'sym'break;}// otherwise// xkb_state_key_get_utf8
So, to make key composing work, it’s all dependent on the client, be
it in X11 or Wayland. In general widget/toolkit libraries, and Xlib,
does it out-of-the-box and/or easily for us.
Finally, let’s review how to interface with XKB from the command line.
There are a couple of X11 bound, and deprecated legacy, commands such as:
xmodmap
(pre-XKB even)
setxkbmap
xkbcomp
xev
xkbprint
xkbevd
They will not work on Wayland since they rely on the XKB X11 specific
proto (XKM binary format and others), but are still good to debug certain
behavior on X11, and to directly interface with X11 to configure XKB
interpretation on the fly, since obviously it’s these software that rely
on the library and load the appropriate configurations.
The main interaction these days should all pass through
xkbcli
and
its subcommands. It comes with a few handy man pages:
xkbcli
xkbcli-list
xkbcli-dump-keymap-x11
xkbcli-dump-keymap-wayland
xkbcli-interactive-x11
xkbcli-interactive-wayland
xkbcli-compile-compose
xkbcli-how-to-type
xkbcli-compile-keymap
xkbcli-interactive-evdev
> xkbcli how-to-type 'P'
keysym: P (0x0050)
KEYCODE KEY NAME LAYOUT LAYOUT NAME LEVEL# MODIFIERS
33 AD10 1 English (US) 2 [ Shift ]
33 AD10 1 English (US) 2 [ Lock ]
> xkbcli compile-keymap --kccgst--layout us --options terminate:ctrl_alt_bksp
xkb_keymap {
xkb_keycodes { include "evdev+aliases(qwerty)"};
xkb_types { include "complete"};
xkb_compat { include "complete"};
xkb_symbols { include "pc+us+inet(evdev)+terminate(ctrl_alt_bksp)"};
xkb_geometry { include "pc(pc105)"};};
To list the whole RMLVO possible values from the registry:
Another thing that is interesting to know is that the XKB
keymap can be converted to Console keymap with scripts such as the
setupcon(1)
which relies on
ckbcomp
and others, and will read confs from
/etc/default/keyboard
.
Obviously, let’s not forget to mention
localectl(1)
to interface with
systemd-localed.service(8)
that is the newer version of
setupcon(1)
. It’s sort of a big wrapper over other tools and behavior
to automate things.
> localectl
System Locale: LANG=en_US.UTF-8
VC Keymap: us
X11 Layout: us
We’ll see how it sets it in X11, but let’s just say it can be used to
list keymaps:
There are also the options
list-x11-keymap-models
,
list-x11-keymap-layouts
,
list-x11-keymap-variants [LAYOUT]
,
list-x11-keymap-options
.
And to set it with
set-x11-keymap
. However it always tries to convert
the XKB keymap to console keymap whenever it can, if you don’t want that
behavior, you should add this option:
> localectl set-x11-keymap --no-convert keymap
Let’s end on a funny note to wrap things up about XKB. Yubikeys work by
simulating keyboards, and thus they have to anticipate a very specific
layout and variant, otherwise inserting a Yubikey would output the
wrong values. To skip this, there are udev device properties (
ENV{}
set from hwdb) called
XKB_FIXED_LAYOUT
and
XKB_FIXED_VARIANT
that
need to be set and respected by the clients of libxkbcommon.
From
60-keyboard.hwdb
:
# Yubico Yubico Yubikey II
evdev:input:b0003v1050p0010*
# Yubico Yubikey NEO OTP+CCID
evdev:input:b0003v1050p0111*
# Yubico Yubikey NEO OTP+U2F+CCID
evdev:input:b0003v1050p0116*
# OKE Electron Company USB barcode reader
evdev:input:b0003v05FEp1010*
XKB_FIXED_LAYOUT=us
XKB_FIXED_VARIANT=
Here’s a summary of what was discussed in the XKB stack:
Pointer Specifics
We’ve seen a lot of complex keyboard specific input behavior, let’s
dabble a bit with pointer devices now, from mice to touchpads.
Types of Touchpads
Let’s mention a few definitions.
In general we call a pointer the representation of the input device,
and the cursor the drawn icon representation.
We have clickpads, a touchpad that has no separate buttons, but
that is all clickable. The behavior then depends on where the click
happens. Meanwhile, forcepads are like clickpads but they don’t have
any buttons and instead will vibrate when pressed. Lastly, trackpoints
are the little balls/nudge in the middle of the keyboard of Thinkpads,
they’re tagged in udev/hwdb with
ID_INPUT_POINTINGSTICK
property.
Device: TPPS/2 Elan TrackPoint
trackpoint: the nudge of thinkpads
# Name: TPPS/2 Elan TrackPoint# ID: bus 0x0011 vendor 0x0002 product 0x000a version 0x0063# Supported Events:# Event type 0 (EV_SYN)# Event type 1 (EV_KEY)# Event code 272 (BTN_LEFT)# Event code 273 (BTN_RIGHT)# Event code 274 (BTN_MIDDLE)# Event type 2 (EV_REL)# Event code 0 (REL_X)# Event code 1 (REL_Y)# Properties:# Property 0 (INPUT_PROP_POINTER)# Property 5 (INPUT_PROP_POINTING_STICK)
properties:
- ID_INPUT=1
- ID_INPUT_MOUSE=1
- ID_INPUT_POINTINGSTICK=1
driver:psmouse
As you can see from the above, the trackpoint also has attached to it
some physical buttons, they’re the ones above the Thinkpad touchpad. It’s
in between a mouse and a touchpad.
There are internal touchpads and external touchpads. The external
touchpads don’t get turned off when the lid is closed, nor disabled
while typing. A graphic tablet such as a wacom device is effectively an
external touchpad.
This information can be embedded in a udev device property called
ENV{ID_INPUT_TOUCHPAD_INTEGRATION}
, and set to either “external” or
“internal”. This is part of the hwdb, out-of-the-box:
Last interesting fact is that some touchpad can have capacitive touch,
that means they can detect the finger in a range above the touchpad,
hovering in proximity. This is the
BTN_TOOL_FINGER
in contrast to
BTN_TOUCH
, but they often come together and so you have to discern if
it’s a real touchdown or not. For MT there’s also
ABS_MT_PRESSURE
and
ABS_MT_DISTANCE
that can be used for this. That’s another job that
libinput is good at.
MT — MultiTouch
We quickly went over the concept of MT, or multitouch before, let’s add
a bit more info to that.
Multitouch are touchpads that support tracking more than one finger. They
speak evdev multitouch to user-space (type B), and most often are handled
by the hid-multitouch driver from the kernel side.
The capabilities of an MT touchpad should have something similar to this
(
libinput record
output or others):
key: BTN_LEFT, BTN_TOOL_FINGER, BTN_TOOL_DOUBLETAP, BTN_TOUCH
(BTN_TOOL_DOUBLETAP up to BTN_TOOL_QUINTTAP)
abs: ABS_X, ABS_Y, ABS_MT_SLOT, ABS_MT_POSITION_X, ABS_MT_POSITION_Y,
ABS_MT_TOOL_TYPE, ABS_MT_TRACKING_ID
There can also be
ABS_MT_TOUCH_MAJOR
,
ABS_MT_TOUCH_MINOR
,
ABS_MT_WIDTH_MINOR
, and
ABS_MT_WIDTH_MAJOR
, that are used to provide
the size of the contact area in surface or absolute units. There’s
also
ABS_MT_ORIENTATION
, for the orientation of the touching ellipse
(finger).
For MT, the key events are simple, they tell us how many fingers are
tapping.
Then, fingers are tracked in what’s called “slots” along with a new
unique tracking id each time a finger touchdown again, and like all
evdev it’s a stateful protocol.
So for example, slot 0 gets assigned tracking id 1 when the first finger
is down, then slot 1 gets assigned tracking id 2 when the second finger
is down, then the first finger is lifted and put back down again, and
slot 0 gets assigned tracking id 3.
That can sound complex to track, and again that’s where libinput
shines. Here’s what it looks like in a simplified evdev trace:
Once upon a time everyone was bragging about their synaptics touchpad
confs, yet this is now deprecated in favor of libinput. What was that
all about?
Synaptics, unrelated to synaptics inc, was a complex X11 driver with
so many configurations. It was buggy and had lots of internal magic,
especially its acceleration profiles, which had logic split between the
X11 server and the driver.
Synaptics was configured through the command line
synclient
. They
talked through a special interfaced with a custom protocol (shared memory
segment). That is before X11 had any standard way to dynamically be
configured (with
xinput
), and before evdev was a thing. This was hacky.
These days X11 and Wayland rely on libinput so this should be used
instead.
The only feature missing from libinput, which is implemented in user-space
by the widget libraries and DE, is non-linear acceleration and speed,
kinetic scrolling. That’s mostly a non-issue.
Acceleration Profile
Simply said, pointer acceleration is the function that multiplies the
movement deltas with a given factor:
One of the main role of libinput is to make pointer movement as precise
as possible on all devices. If the user intends and performs action,
the feedback should be that it’s what they expected to do.
An acceleration profile defines a series of points of the form
(x, f(x))
,
input to output speed, that are linearly interpolated (a curve is drawn
between them for deduction). For example, flat acceleration is
[(0.0, 0.0), (1.0, 1.0)]
.
The default acceleration, adaptive, is pretty smart, and differs per
device type and resolution, it already has these configured for touchpads
for example:
super-slow: deceleration
slow: deceleration
medium: adaptive+deceleration
fast: adaptive+fast
flick: fast
In general, libinput allows to configure this behavior. We can pick
between 3 pointer acceleration profiles: adaptive (default), flat the
45° one we’ve seen, and custom profiles. Along with different types
of motions the profiles can apply to: motion, scroll, fallback. We can
configure points and steps for each one: the points are the x and y
creating the curve of the acceleration profile we talked about, and the
steps is how the interpolation granularity happens between the points
(a value of 0 will use the default).
In most cases, not touching the acceleration profile provides better
results.
In
libinput list-devices
for a touchpad:
Accel profiles: flat *adaptive custom
Gestures
We’ve seen that libinput offers two types of gestures out-of-the-box:
swiping and pinching. For anything else, one has to rely on third party
libraries. Here are a few:
Let’s close this section with a few random details that don’t need
much discussion.
High-end gaming mice are finicky and often normal basic drivers are not
enough to configure their high precision, nor is libinput. That’s why the
libratbag
project exists.
The libwacom (not only wacom) and tools such as Tuhi are used to manage
information needed by libinput to handle drawing tablets. These tablets
come with a tool such as a pen/stylus, it’s specificities are handled too.
For example, pressing certain button to reverse the behavior and start
erasing. There are X11 tools such as
xsetwacom
that also help.
An interesting software is
gpm(8)
which is a mouse in
the console that relies on reading directly the mouse stream
character device and interfacing/translating them to
TIOCLINUX
TIOCL_SELMOUSEREPORT
, terminal ioctl, to draw it. The terminal
will then output specific mouse reporting escape codes (more info
here
).
Finally, here’s a few pointer specific debug tools:
Gamepads aren’t handled by libinput in user-space, nor do they rely on
the evdev handler in the kernel. Instead they rely on the joydev handler.
The gamepads get associated to their specific drivers, which
will consume all these events. The joydev handler then normalizes
and sends them to user-space in a format called
js_event
from
include/uapi/linux/joystick.h
.
The handler will listen to all devices that support
EV_KEY
BTN_JOYSTICK
or
BTN_GAMEPAD
and similar events, and create a stream
device in devtmpfs for it
/dev/input/jsN
.
The handler character device supports a bunch of standard ioctl calls
to get/set info:
JSIOCGVERSION
: get driver version
JSIOCGAXES
: get number of axes
JSIOCGBUTTONS
: get number of buttons
JSIOCGNAME(len)
: get identifier string
JSIOCSCORR
: set correction values
JSIOCGCORR
: get correction values
JSIOCSAXMAP
: set axis mapping
JSIOCGAXMAP
: get axis mapping
JSIOCSBTNMAP
: set button mapping
JSIOCGBTNMAP
: get button mapping
Obviously, it’s better to do this via tools such as:
jstest
and
jstest-gtk
jscal
joyful
Upper Stack: X11 & Wayland
We’ve reached the graphical environment with desktop widget libraries such
as GNOME and Qt, and the XServer and Wayland Compositors. They’re
the ones that rely on all types of input events for concrete behavior,
from clicking buttons on the appropriate window, drawing a cursor
on screen, scrolling, and literally all interactions a user has with
a computer.
This upper stack relies on libinput and XKB to make everything happen. As
far as these two are concerned, the role of the upper stack is to
initialize them with the right configurations, and then create the
handling for whatever they’re meant to do.
The big difference between the X11 stack and Wayland stack is related to
the protocol and where these libraries are included. There are no window
managers in Wayland, but compositors that fully implement the standard
protocol of both a display server and window manager at the same time. So
it’s not a two-process equation, the compositor is the one handling
libinput and implementing the desktop interface. Meanwhile, in X11, the
Xserver, which is quite old, has the abstract concept of input drivers,
of which the currently only useful one is
xf86-input-libinput
. The
X11 input are interfaced with through the X11 protocol with XInput
events shared to the WM and other clients so that they can use them,
and configure the server’s input devices. Similarly, in X11 all the
configurations happen over the X protocol and its extensions, meanwhile
for compositors there’s no agreed way to configure things, so each
compositor can implement their own thing.
Here’s a general picture of the stack (
courtesy of
who-t, Peter Hutterer
):
Obviously, each have their own internal representation and ways of managing
the information they get from libinput, XKB, and others, but this is
outside the scope of this article (
wl_pointer
and
wl_keyboard
on
Wayland for example). Let’s focus more on how they configure the input
stack we’ve seen.
The X server has an internal store of information about input devices,
and their drivers, and will apply the default settings for each. To
apply specific configurations for certain devices, we can add snippets
in the X11 config directory, usually
/usr/share/X11/xorg.conf.d/
.
The
libinput(4)
driver settings can be passed there for a matching
device.
The “Identifier” is just a human-readable string for logging, meanwhile
the series of “Match” statements can be found in
xorg.conf(5)
, there’s
quite a few of them and they remind us of udev rules. The “Option”
part is what interests us, these are the settings to pass to libinput
and that can be found in
libinput(4)
. For example:
On the X11 stack, the server will initially set these values to override
the default ones, but afterward, during runtime, any caller can rely on
the X protocol to update them. The
xinput(1)
command can be used to
debug and test setting X input devices.
To list input devices that the X server is aware of:
> xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ ETPS/2 Elantech Touchpad id=15 [slave pointer (2)]
⎜ ↳ SEMICO USB Keyboard Consumer Control id=10 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Power Button id=8 [slave keyboard (3)]
↳ Sleep Button id=9 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=14 [slave keyboard (3)]
↳ Acer WMI hotkeys id=16 [slave keyboard (3)]
↳ GeneralPlus USB Audio Device id=17 [slave keyboard (3)]
↳ SEMICO USB Keyboard Consumer Control id=11 [slave keyboard (3)]
↳ SEMICO USB Keyboard System Control id=12 [slave keyboard (3)]
↳ SEMICO USB Keyboard id=13 [slave keyboard (3)]
NB
: Keep in mind the XTEST virtual devices, which only exist within
X11 internally and don’t appear in
libinput list-devices
, we’ll get
back to these in the next section.
Or list the properties of a particular device entry:
What happens here is that the client (
xinput
) talks to the X server
over the X protocol, then the X server talks to its libinput driver
xf86-input-libinput
which in turn talks to libinput and updates its
configurations, and the X server keeps track of all this.
These all look somewhat redundant, as you can see, it’s like having an
intermediate layer. That’s why on Wayland there’s no intermediary, if
a client tells it, through whatever configuration means it exposes, to
set certain settings on an input device, it does it directly via libinput.
Yet, the list of input devices is internal to Wayland, and not exposed
directly in the protocol, that’s why it differs in each compositor
implementation.
For instance, if we’re toggling a setting in GNOME, KDE, MATE, or others,
the behavior will be more direct. In GNOME, things happen through
gsettings
:
So that’s how you’d configure input devices on GNOME Wayland compositor
Mutter. Yet that’s annoying, isn’t there a common way to do this on
Wayland?
There are workarounds such as
libinput-config
but it’s not
very well maintained.
So, clients in graphical environments need to get input events to
them. On X11 these are called X events, and they can be spied on with the
xev(1)
tool, which can help debug issues. It shows events sent to the
particular window chosen.
In theory on X11 one could catch all events on the “root window” is
subscribed to (
xev -root
does that) or of any other window. Events
conceptually travel down the window hierarchy, and clients only
receive the events for which they have selected an appropriate event
mask. However, the root window always sits at the top of this hierarchy
and can optionally subscribe to essentially all events before they
propagate to child windows, while grabs and higher-priority selections
(such as by the window manager) can intercept or redirect them. That’s
how WMs work, they’re the parent window and have an “event mask” to
catch certain events and input for itself, and is exclusively allowed
to do redirect of certain events such as mapping/moving/configuring
windows.
Meanwhile, a sort of equivalent, but more simple, tool on Wayland is
called
wev
, we’ll do the comparison
in a bit to help us understand the differences. Here’s a trace of
xev
> xev -event keyboard
KeyRelease event, serial 28, synthetic NO, window 0x2e00001,
root 0x3fa, subw 0x0, time 465318306, (81,81), root:(893,376),
state 0x10, keycode 108 (keysym 0xff20, Multi_key), same_screen YES,
XLookupString gives 0 bytes:
XFilterEvent returns: False
…
KeyRelease event, serial 28, synthetic NO, window 0x2e00001,
root 0x3fa, subw 0x0, time 465318602, (81,81), root:(893,376),
state 0x10, keycode 48 (keysym 0x27, apostrophe), same_screen YES,
XLookupString gives 1 bytes: (27)"'"
XFilterEvent returns: False
KeyPress event, serial 28, synthetic NO, window 0x2e00001,
root 0x3fa, subw 0x0, time 465318866, (81,81), root:(893,376),
state 0x10, keycode 26 (keysym 0x65, e), same_screen YES,
XLookupString gives 1 bytes: (65)"e"
XmbLookupString gives 1 bytes: (65)"e"
XFilterEvent returns: True
KeyPress event, serial 28, synthetic NO, window 0x2e00001,
root 0x3fa, subw 0x0, time 465318866, (81,81), root:(893,376),
state 0x10, keycode 0 (keysym 0xe9, eacute), same_screen YES,
XLookupString gives 0 bytes:
XmbLookupString gives 2 bytes: (c3 a9)"é"
XFilterEvent returns: False
As you can observe here, The Xlib client does a lookup for keycode to
keysym translation by relying on functions such as
XLookupString
and
XmbLookupString
. These particular functions use a keymap logic that
dates back to pre-XKB time, we’ll talk more about them in a bit. Yet,
internally now, the X server does rely on XKB in the backend, just like
for input device info, it keeps a keymap table internally, and it’s
shared over the X protocol with clients (they ask for it at connection,
or lazily when calling functions, and cache it) so that they perform
the translation with Xlib or XCB.
There are two main formats for the shared X server keymap the clients
can rely on: the old “X core keymap”, and an XKB keymap. We’ll discuss
that old core keymap in a bit.
In XCB, the old keymap translation is done via:
xcb_key_symbols_get_keycode
xcb_key_symbols_get_keysym
And in Xlib with functions such as:
XLookupString
Xutf8LookupString
XLookupKeysym
XkbTranslateKeyCode
XkbTranslateKeySym
XStringToKeysym
XKeysymToKeycode
Meanwhile, with the newer XKB keymap it’s done via:
XkbTranslateKeyCode
Or in XCB with the
xcb_xkb_*
functions (you have to do it manually).
In all cases, since XKB is the tech in the backend of the X server
that stores the keymap truth, it’s what needs to be configured. The XKB
configuration can be set statically, along with the usual input confs
we’ve seen earlier, with the Xkb options:
There are also two special options that get interpreted when certain
special keysym are generated, the
DontVTSwitch
which is there to disable
the
ctrl+alt+fn
sequence to switch virtual terminal, and the
DontZap
which catches the
Terminate_Server
keysym of XKB and will kill the Xorg
server. Both are enabled by default and these options would turn them off.
To change the XKB options on a running X server on-the-fly, we need to
rely on two tools:
xkbcomp(1)
and
setxkbmap(1)
. The first one is
used to compile new KcCGST and upload it to the server as a full keymap
in XKM compiled format that the server understands, and the second one
to change the current value of the RMLVO.
> setxkbmap -print-verbose 10
Setting verbose level to 10
locale is C
Trying to load rules file ./rules/evdev...
Trying to load rules file /usr/share/X11/xkb/rules/evdev...
Success.
Applied rules from evdev:
rules: evdev
model: pc105
layout: us
options: compose:ralt
Trying to build keymap using the following components:
keycodes: evdev+aliases(qwerty)
types: complete
compat: complete
symbols: pc+us+inet(evdev)+compose(ralt)
geometry: pc(pc105)
xkb_keymap {
xkb_keycodes { include "evdev+aliases(qwerty)"};
xkb_types { include "complete"};
xkb_compat { include "complete"};
xkb_symbols { include "pc+us+inet(evdev)+compose(ralt)"};
xkb_geometry { include "pc(pc105)"};};
Now let’s talk about that pre-XKB logic with functions such as
XLookupKeysym(3)
we’ve seen in the
xev
trace earlier. It’s currently
basically a wrapper over XKB, but that can also bypass it entirely. It
relies on the old “X core keymap table” in the X server, a facade on
the authoritative keymap that is XKB backed. The client asks for it
via a request, cache it, and use it for the mapping of X11 keycode
to X11 keysym. It’s own X11 keycodes are implementation dependent,
but nowadays it’s mostly
evdev + 8
, and its keysyms are found in
/usr/include/X11/keysymdef.h
, which the newer XKB stack also relies
on in X11. So that old keymap is indeed initially filled with the XKB
keymap. The tool
xmodmap(1)
will help us explore and show some of the
things it handles.
Yes,
xmodmap
has its own configuration in
~/.Xmodmap
and expression
grammar that looks something like a simplified version of XKB:
! remove Caps Lock functionality
remove Lock = Caps_Lock
! make CapsLock (keycode 66) act as Tab
keycode 66 = Tab
! set Menu key (keycode 134) properly
keycode 134 = Menu
! Set Right Alt as Compose (Multi_key)
! Use keysym form so you don't need to know the numeric keycode:
keycode 108 = Multi_key
! ensure Right Alt is not still treated as an Alt modifier
remove Mod1 = Alt_R
There’s even the
xkeycaps
GUI around it, and wrappers like
xcape
.
Yet, GNOME and certain other toolkits and desktop environments
have stopped relying on the old core keymap a long time ago,
deprecating it in favor of the XKB related functions. Still, the
X server will internally reflect these changes in its XKB cache,
making them internally compatible, notifying X clients of teh
change, and it’ll work but temporarily
(mainly with
XChangeKeyboardMapping
which calls
XkbApplyMappingChange
in the X Server). It’s fragile and legacy. Also, changing the keymap with
xmodmap
is flimsy since any time the XKB keymap is reloading the changes
to the old in-memory X keymap compatibility is lost. Those combined
together means that it isn’t reliable to use the old X11 core keymap.
As you can see yet again, this is quite confusing and redundant, and
obviously Wayland doesn’t have these old layers of indirection and
relies on XKB directly. It also doesn’t need a compiled forms like XKM
to upload keymaps to the server, but it doesn’t even include that upload
part in the protocol anyhow. The keycode to keysym translation is also
done in the client (with calls such as
xkb_state_key_get_one_sym
)
but the keymap is directly shared along the
wl_keyboard
object that
it gets accessed to when it wants input access on the seat, so there’s
no need for another round-trip.
Yet, again the configuration of XKB-related stuff on Wayland depends on
the compositor implementation.
For example
wlroots
relies on environment variables to set the RMLVO.
GNOME on
gsettings
with
gsettings set org.gnome.desktop.input-sources sources "[('xkb', 'us'), ('xkb', 'fr')]"
Let’s go back to the
wev
tool,
which displays input events on Wayland, it’ll help us understand a
huge difference in input handling on Wayland compared to X11. Unlike X
severs, a Wayland compositor doesn’t propagate and broadcast the events
globally to anyone listening. Instead, clients must explicitly register
a listener for the objects they care about. These are announced via the
global Wayland registry, which it has to register to (
wl_registry
).
Afterward, a client has to bind and listen to the given seat (
wl_seat
)
of the given name by the registry (this is where the
ENV{ID_SEAT}
and
loginctl
can help since they often map 1-to-1), and advertise the set of
“seat capabilities” it requires and wants to bind to, such as pointer,
keyboard, or touch. Once bound, the client can now fetch a handle to the
wl_<pointer/keyboard/touch>
objects, and register listener handlers
for their events. Let’s note that a
wl_keyboard
is an abstraction
of all logical keyboard events. So clients aren’t aware of underlying
devices, it’s abstracted and aggregated in the compositor internally,
by its own logic. Plus, for extra security,
wl_<pointer/keyboard/touch>
events are only forwarded to the currently focused client. All and all,
it’s very choreographed, unlike in X11.
Beyond the core protocol, there are more “unstable”
or “non-standard” extensions that allow clients to do
more things related to input. Here’s a non-exhaustive
list
:
Repositioning the pointer (
wp_pointer_warp_v1
)
Subscribing to high-def keyboard timestamps (
zwp_input_timestamps_manager_v1
)
Ignoring keyboard shortcuts from a client (
zwp_keyboard_shortcuts_inhibit_manager_v1
)
Adding constraints to pointer motion (
zwp_pointer_constraints_v1
)
Register to handle gestures, swipe, pinch, and hold (
zwp_pointer_gestures_v1
)
Specific XWayland grabbing of input, monopolizing it (
zwp_xwayland_keyboard_grab_manager_v1
)
Grab hotkeys, usually not needed since the compositor do this (
hyprland_global_shortcuts_manager_v1
)
Grab/Inhibit an input to a single surface such as lock screen (
zwlr_input_inhibit_manager_v1
,
hyprland_focus_grab_manager_v1
)
Notice too that nowhere in the protocol is there any interface
to list the compositor’s internal input devices in its registry,
it’s intentionally abstracted away. It’s up to each compositor
to choose if it wants to expose this info. To my knowledge,
there’s only Sway that offers an interface for this through
swaymsg
, it’s kind of
similar to
gsettings
.
The closest compositor-agnostic tools are external utilities such
as
libinput list-devices
or
loginctl seat-status
. However, these
enumerate kernel devices, not the compositor’s internal virtual devices,
so you will not see compositor-created synthetic devices there.
In short, which compositor implements which part of the “non-standard”
protocol varies a lot. GNOME uses almost none of the wlroots/WLR
extensions. KDE uses KDE-specific extensions. wlroots-based
compositors share WLR extensions. It’s a mix really, check
this
for support and more info.
We mentioned before
localectl
too for setting keyboard keymap
setups that works across environments. Let’s add that when using
the
set-x11-keymap
option it will modify X11 configurations in
/etc/X11/xorg.conf.d/00-keyboard.conf
and pre-fill them for you so
you won’t have to worry about editing anything with the options we’ve
listed. It doesn’t have this option for Wayland though.
Yet, what if someone on Wayland wants to remap just a specific
key without passing by the static XKB and its mess, just a
quick runtime change. There’s no real solution to that other
than what we’ve already mentioned in the scancode to keycode
section, namely tools that rely on evdev interception to remap events
such as
evmapd
,
evremap
,
evdevremapkeys
,
evsieve
,
keyd
,
kbct
,
makima
,
input-remapper
,
etc.. A true panoply of tools that are hacks. Most, if not all, of
these work by intercepting evdev events, creating a new virtual device
(we’ll see how
uinput
works in the next section), and modifying the
events on-the-fly to write them to the virtual device. This adds a new
unnecessary layer of indirection, which you should obviously avoid if
you are doing anything speed sensitive with the keyboard. Furthermore,
some of these re-include the key composition and a semblance of XKB
logic within them, which creates a total mess.
Let’s continue…
Contrary to everything else in the GUI stack, XKB composition is a
bit less cumbersome. Both Wayland clients, through their toolkits
(GTK, Qt, etc..) and X11, through Xlib with the functions we’ve seen
earlier that do it out-of-the-box (
XLookupString
), rely on the
same configuration files we’ve discussed in the XKB section. Namely,
/usr/share/X11/locale/<locale>/Compose
and the home
~/.XCompose
. It
follows the simple format described in
Compose(5)
.
And lastly, one thing that isn’t handled neither in libinput nor XKB is
key repeat: how long when pressing a key will the client wait to print
it again.
In X11 this is configured in the X Server, either as a startup option
-ardelay
and
-arinterval
, or dynamically via
xset(1)
. There’s the
option to set the delay and interval for a specific key too.
> xset r rate delay [rate]
> xset r rate 210 50
If you inspect
xev
you’ll see that the server resends keys to the
client continuously.
Meanwhile, as with everything else on Wayland, it depends on the
compositor. The compositor sends to the clients the repeat parameters
wl_keyboard.repeat_info(rate, delay)
and it’s up to them to respect
it. So, the compositor doesn’t keep forwarding the key to the client but
instead this is handled directly in the client.
And similarly, these are configured in
gsettings
and other
compositor-specific configurations.
The repeat key rate and delay being delegated to
clients on Wayland has had its share of issues it created though
(
see
)
and some people want to have it
back in the
compositor
.
That’s it, we’ve covered most of the things we wanted in the upper
graphical stack.
Virtual Input, Automation, Emulation, and Remote Desktop
We’ve grazed the topic of virtual inputs before, in this section we’ll
see what types exist and where they’re used, from automation, emulation,
and remote desktop.
The first layer where we can create virtual input devices is at the
kernel layer. It provides two modules that can be used for this:
UHID
, User-space I/O driver support for HID subsystem, and uinput,
the User-space input emulation module.
The
uhid module
,
as the name implies, allows simulating HID events from user-space
by reading/writing to a special character device in devtmpfs
/dev/uhid
. The interface is quite simple as is shown in
this
example
.
However, this is only used for emulating devices and debugging, not for
the average user’s virtual input. This is the underlying mechanism behind
hid-record
and
hid-replay
, which can easily allow debugging hid
issues by reproducing the exact sequences of events on anyone’s machine.
While uhid acts in the HID layer, the uinput module
(
drivers/input/misc/uinput.c
) acts at the input core layer, which
makes it more approachable for basic input event virtualisation.
It is also a character device in devtmpfs
/dev/uinput
that exposes
particular ioctl to create, manage, and configure capabilities of a
virtual device, and then allow writing to
/dev/uinput
file descriptor
to simulate the events of said device. The device will appear, like any
other input device, in devtmpfs and sysfs, since it’ll pass by the same
pipeline with
struct input_dev
and the default evdev event handler.
There are two main ways to use uinput in the code, via
<linux/uinput.h>
or via
<libevdev/libevdev-uinput.h>
. The libevdev mechanism is simpler
and recommended.
Example 1:
#include<unistd.h>
#include<linux/uinput.h>
#include<fcntl.h>
#include<string.h>
#include<stdio.h>
#include<errno.h>
#include<linux/uinput.h>voidemit(intfd,inttype,intcode,intval){structinput_eventie;ie.type=type;ie.code=code;ie.value=val;/* timestamp values below are ignored */ie.time.tv_sec=0;ie.time.tv_usec=0;write(fd,&ie,sizeof(ie));}intmain(void){structuinput_setupusetup;intfd=open("/dev/uinput",O_WRONLY|O_NONBLOCK);/*
* The ioctls below will enable the device that is about to be
* created, to pass key events, in this case the space key.
*/ioctl(fd,UI_SET_EVBIT,EV_KEY);ioctl(fd,UI_SET_KEYBIT,KEY_SPACE);memset(&usetup,0,sizeof(usetup));usetup.id.bustype=BUS_USB;usetup.id.vendor=0x1234;/* sample vendor */usetup.id.product=0x5678;/* sample product */strcpy(usetup.name,"Example device");ioctl(fd,UI_DEV_SETUP,&usetup);ioctl(fd,UI_DEV_CREATE);/*
* On UI_DEV_CREATE the kernel will create the device node for this
* device. We are inserting a pause here so that user-space has time
* to detect, initialize the new device, and can start listening to
* the event, otherwise it will not notice the event we are about
* to send. This pause is only needed in our example code!
*/sleep(60);/* Key press, report the event, send key release, and report again */emit(fd,EV_KEY,KEY_SPACE,1);emit(fd,EV_SYN,SYN_REPORT,0);emit(fd,EV_KEY,KEY_SPACE,0);emit(fd,EV_SYN,SYN_REPORT,0);/*
* Give user-space some time to read the events before we destroy the
* device with UI_DEV_DESTROY.
*/sleep(100);ioctl(fd,UI_DEV_DESTROY);close(fd);return0;}
Compile with
gcc -o uinput_test uinput_test.c -Wall -Wextra
And example 2 with libevdev:
#include<libevdev/libevdev.h>
#include<libevdev/libevdev-uinput.h>
#include<unistd.h>
#include<stdio.h>
#include<string.h>
#include<errno.h>intmain(void){structlibevdev*dev=NULL;structlibevdev_uinput*uidev=NULL;interr;/* Allocate and configure the virtual device */dev=libevdev_new();if(!dev){fprintf(stderr,"Failed to allocate libevdev device\n");return1;}libevdev_set_name(dev,"Example device (libevdev uinput)");libevdev_set_id_bustype(dev,BUS_USB);libevdev_set_id_vendor(dev,0x1234);libevdev_set_id_product(dev,0x5678);/* Enable only one key: KEY_SPACE */libevdev_enable_event_type(dev,EV_KEY);libevdev_enable_event_code(dev,EV_KEY,KEY_SPACE,NULL);/* Create the uinput device */err=libevdev_uinput_create_from_device(dev,LIBEVDEV_UINPUT_OPEN_MANAGED,&uidev);if(err!=0){fprintf(stderr,"Failed to create uinput device: %s\n",strerror(-err));return1;}/* A pause to allow the system (udev etc.) to register the device */sleep(100);/* Emit a space key press */libevdev_uinput_write_event(uidev,EV_KEY,KEY_SPACE,1);libevdev_uinput_write_event(uidev,EV_SYN,SYN_REPORT,0);/* Emit the key release */libevdev_uinput_write_event(uidev,EV_KEY,KEY_SPACE,0);libevdev_uinput_write_event(uidev,EV_SYN,SYN_REPORT,0);/* Let user-space read the events before destruction (optional) */sleep(200);libevdev_uinput_destroy(uidev);libevdev_free(dev);return0;}
The disadvantage of uhid and uinput is that, since they interface with
the kernel, they require root privilege and relying on HID or evdev might
not be practical for the average day-to-day usage. For example, if we
want to output a symbol, let’s say ‘p’, we have to know its keycode, and
for that we need to know the keymapping, which in turn requires XKB or
others. Thus, we’re back to square one and re-creating the upper input
stack from scratch.
What if we could directly say “send this keycode or keysym, it’s from this
virtual device”, without even needing extra permission if we’re already in
a desktop environment. Well, that’s exactly what the X11 XTEST extension
does, and what some Wayland extensions and mechanisms achieve too.
Remember when we used xinput to list some devices and some virtual ones
were listed:
These were created by the XTest extension which was written to support
automated testing of X server. These days this can be used for remote
desktop, task automation, password managers (autofill), and others. When
clients interface through this extension they directly inject keyboard
and mouse events into the X server, bypassing the whole input stack,
and these events are propagated afterward to the X clients.
Let’s see a simple programming example relying on
XTEST(3)
that will
send the keysym “a”:
That’s clean and easy, now on Wayland the picture is a bit more complex
since the protocol doesn’t allow clients to randomly generate input
events. It was designed this way for security reasons.
As with anything Wayland, there are a few unstable extensions,
though deprecated now, such as
zwlr_virtual_pointer_v1
and
zwp_virtual_keyboard_manager_v1
, mostly wlroots Wayland extensions.
An example of the
zwp_virtual_keyboard_manager_v1
extension would look
somewhat like this:
#define _POSIX_C_SOURCE 200809L
#include<wayland-client.h>
#include"virtual-keyboard-unstable-v1-client-protocol.h"staticstructzwp_virtual_keyboard_v1*vk;staticvoidglobal_add(void*data,structwl_registry*reg,uint32_tname,constchar*iface,uint32_tver){if(strcmp(iface,zwp_virtual_keyboard_manager_v1_interface.name)==0){auto*mgr=wl_registry_bind(reg,name,&zwp_virtual_keyboard_manager_v1_interface,1);// NULL seat → compositor chooses default seatvk=zwp_virtual_keyboard_manager_v1_create_virtual_keyboard(mgr,NULL);}}staticconststructwl_registry_listenerreg_listener={.global=global_add,.global_remove=NULL};intmain(){structwl_display*d=wl_display_connect(NULL);structwl_registry*reg=wl_display_get_registry(d);wl_registry_add_listener(reg,®_listener,NULL);wl_display_roundtrip(d);// wait until vk is readyuint32_tkeycode=30;// Linux evdev (KEY_A)uint32_tstate_pressed=1;uint32_tstate_released=0;zwp_virtual_keyboard_v1_key(vk,0,keycode,state_pressed);zwp_virtual_keyboard_v1_key(vk,0,keycode,state_released);wl_display_flush(d);return0;}
But for a better example check the source of
wlrctl
that also relies on the
zwp_virtual_keyboard_manager_v1
extension.
Yet, these days, this isn’t the path that Wayland has taken, and none of
the compositors agree on these extensions, instead they rely on libei,
a library to consolidate Emulated Input. This is its architecture:
It has two pieces: a client side that creates virtual devices and
generates evdev events, and the server side called EIS that lives within
the compositor (but that isn’t limited to Wayland) and is responsible for
giving a file descriptor to the client to interface with, and dispatching
received events to where they need to go. The dispatching could be through
uinput devices, that’s an implementation detail, yet most compositors
just store it as an internal virtual device.
This allows compositor to be aware of who is currently emulating input,
which capabilities they require (keyboard, touch, pointer), and to
restrict and/or suspend devices at any time.
Optionally, the compositor may delegate the file descriptor mechanism to
a xdg-desktop-portal dbus service implemented by the desktop environment
so that it can check with polkit and others the allowed permissions
(
see
). So it would
look like this:
An example implementation of a client can be found
here
.
Or mixed with an XKB mess to translate from keysym to keycode, for the
pleasure of your eyes:
#include<ei.h>
#include<xkbcommon/xkbcommon.h>
#include<string.h>intmain(){// ----------------------------// 1. Create an XKB context// ----------------------------structxkb_context*ctx=xkb_context_new(XKB_CONTEXT_NO_FLAGS);// load the default system keymap (XKB rules, model, layout, variant, options)structxkb_keymap*keymap=xkb_keymap_new_from_names(ctx,NULL,XKB_KEYMAP_COMPILE_NO_FLAGS);if(!keymap){fprintf(stderr,"failed to load xkb keymap\n");return1;}// ----------------------------// 2. Convert keysym → evdev code but only for group 1 and level 1// ----------------------------xkb_keysym_tsym=xkb_keysym_from_name("a",XKB_KEYSYM_NO_FLAGS);// if we know the key by name it would be much easier// xkb_keycode_t code = xkb_keymap_key_by_name(keymap, "AD01"); // But better: find the keycode for the keysymxkb_keycode_tkey=0;// Iterate keycodes until we find the one producing this keysym// That's because keycodes->keysym is many-to-onexkb_keycode_tmin=xkb_keymap_min_keycode(keymap);xkb_keycode_tmax=xkb_keymap_max_keycode(keymap);for(xkb_keycode_tk=min;k<=max;k++){if(xkb_keymap_key_get_level(keymap,k,0,0)>=0){intnsyms;constxkb_keysym_t*syms=xkb_keymap_key_get_syms_by_level(keymap,k,0,0,&nsyms);for(inti=0;i<nsyms;i++){if(syms[i]==sym){key=k;break;}}}}if(!key){fprintf(stderr,"could not map keysym\n");return1;}// IMPORTANT: xkbcommon keycodes are **+8** relative to evdevintevdev_code=key-8;structei*ei=ei_new();// compositor socket, we'd need to get it through portal in real scenarioei_connect(ei,"unix:path=/run/user/1000/ei_socket");structei_client*client=ei_get_client(ei);ei_client_set_name(client,"xkb-sender");structei_device*dev=ei_device_new(client,"xkd-virt-keyboard0");ei_device_add_capability(dev,EI_DEVICE_CAP_KEYBOARD);ei_device_start_emulating(dev);// press and releaseei_key(dev,evdev_code,true);ei_key(dev,evdev_code,false);ei_flush(ei);ei_disconnect(ei);ei_free(ei);return0;}
Then obviously the EIS side has to catch these events and handle
them. There’s also an example that creates uinput devices found
here
.
The main logic of an EIS is quite straight forward (from the official docs):
create a context with
eis_new()
set up a backend with
eis_setup_backend_fd()
or
eis_setup_backend_socket()
register the
eis_get_fd()
with its own event loop
call
eis_dispatch()
whenever the fd triggers
call
eis_get_event()
and process incoming events
And whenever a new client connects:
accept new clients with
eis_client_connect()
create one or more seats for the client with
eis_client_new_seat()
wait for
EIS_EVENT_SEAT_BIND
and then
create one or more devices with the bound capabilities, see
eis_seat_new_device()
That’s kind of like network programming.
So far, most Wayland compositors implement this mechanism
along with portals. You can see the list of support
here
, from
GNOME, KDE, XWayland, and more.
On that note, XWayland is both an X server, and a Wayland client. So it
understands XTest requests. Yet what happens when it receives them is
that internally it relies on libei client side to handle virtual device
events. That means xdotool can work on XWayland with libei context.
An X11 client sends a key event using XTEST (normal)
XWayland receives it and initiates Remote Desktop XDG Portal session to … your own system (???)
XDG Portal uses DBus in an odd way, with many method calls receiving responses via signals because DBus isn’t designed for long asynchronous methods.
Once the Remote Desktop portal session is setup, Xwayland asks for a file descriptor to talk an libei server (emulated input server).
After that, libei is used to send events, query the keyboard map, etc.
You can ask libei for the keyboard mapping (keycodes to keysyms, etc), you get another file descriptor and process that with yet another library, libxkbcommon.
The main issue is that if the libei client gets its file descriptor
via dbus portal, then every time it asks for it then the user will get
prompted to “Allow remote interaction?”. And most portal software don’t
have config or whitelist rule mechanisms to skip that (as far as I know),
which would make sense while keeping the same security level.
When it comes to remote desktop on Wayland, it’s quite similar, it
relies on the same libei mechanism. Yet, we need to add to the equation,
as far as input goes, a listener that captures input regardless of the
focused window.
The remote desktop is also achieved with libei and a dbus xdg-desktop-portal
either
org.freedesktop.portal.RemoteDesktop
or
.InputCapture
, which will give back to the client
a special file descriptor for listening to the input stream.
And similarly, it is always explicit about asking for
permission to share input or share the screen (or specific
window/surface), and there doesn’t seem to be a general
configuration to turn it off or whitelist certain clients (
see
discussion
).
Let’s note that in the case of Wayland it is the compositor that usually
provides VNC/RDP servers, for example KWin and GNOME Mutter (apart from
wayvnc
for wlroots compositors).
Meanwhile, on X11 the remote desktop protocol was part of the X11 protocol
itself from the start, with full access, the whole events. The X server
can be on another machine and clients can communicate with it over the X11
protocol, or the events could be forwarded over ssh and others. VNC and
other remote desktop protocol can rely on how open it is too. Plus, XTEST is
there for injecting events too. There’s no limitation for apps to read the
screen framebuffer either, send it, and draw it in another X session,
but it’s often re-rendered when doing remote desktop. (x11vnc, TigerVNC,
etc..). There have been extensions over the years for security (XACE)
but nobody is relying on them.
There’s also xrdp, but this creates a whole new virtual Xorg session,
so it’s another story.
Let’s now review a couple of tools used for automation.
We’ve already seen quite a lot of the ones that rely on evdev and uinput,
but now they will make more sense with our current context:
evmux
and
inputattach
- multiplex multiple evdev devices into a
single virtual stream
The most popular tool that relies on XTEST (plus EWMH and others) is
xdotool
.
NB
: the “toplevel-management” Wayland “unstable” extension somewhat
replaces some of the EWMH, but it’s not implemented by most compositor
for security reasons.
Similar tools to
xdotool
but that relies on uinput are
ydotool
and
dotool
.
We’ve seen
wlrctl
that relies on the unstable wayland protocol for
wlroots-based compositors. There’s also
wtype
that also relies on the
unstable virtual keyboard protocol.
We can also possibly perform automation via very specific desktop
environment mechanisms. That means using something such as GNOME shell
extensions for example, which has a javascript API. KDE has that
concept and the utility
kdotool
relies on this.
As you’ve observed, the situation is a bit fragmented on Wayland when
it comes to automations, both in utilities and extensions.
Input Method
In this last section we’ll explore the concept of input method (IMF &
IME), a mechanism to input keysym/characters that are not natively
available on the user’s input device. This is necessary for languages
that have more graphemes than there are keys on the keyboard.
There are two sides to the equation: the IMF, the input method framework,
and the IME, the input method engine which works within the framework. An
input method framework’s role is to pick the most appropriate way
to enter the text, shape it, and return it to the widget. The IME is
basically the place where input are interpreted in any way shape, form,
or logic, to produce the text that the IMF asked for.
The IMF can also act as wrapper over XKB configs, to allow easily swapping
between keyboard layouts, it coexist with the idea of switching between
different IMEs.
Simply said, the IME is a middleman between the keyboard and the actual
output text when relying on the toolkit/widget.
The way the input method plugs into the whole input stack is at the window
client side, within the widget/toolkit library framework in the input
handling event loop. After the client performs the keycode to keysym
translation and composing, it calls the toolkit specifically configured
input method, which will reroute it to the IM pipeline. Within the
pipeline, the IMF implementation will talk over its protocol to have
the IME interpret the input, and return preedit and committed text. This
will in turn be pushed back to the toolkit to display.
That means that simply by relying on input widgets from a framework
such as GTK or Qt, it will automatically handle the integration with the
Input Method.
Some of these input frameworks are swappable, either because they talk
over the same protocol, be it the old deprecated XIM protocol for legacy
purpose (X Input Method over X protocol extension), or because they
plug straight as a module into the widget framework, which is mostly that
case today.
There are a lot of IMFs and IMEs implementations, and interoperability,
see this list
.
These days the two major IMFs are IBus (Intelligent Input Bus, GTK-based like GNOME), and Fcitx5
(Qt-based like KDE).
To swap between them, if they are compatible with the toolkit, one can
set certain environment variables related to their toolkit:
For example the path that the text will take with Ibus looks like this:
Application → GTK/Qt IM module → D-Bus → IBus/Fcitx
→ IME → D-Bus → GTK/Qt widget
As you can see, this bypasses all graphic servers, be it the X servers
or Wayland compositors.
Yet for it to work across the Wayland ecosystem, and not only on some
widgets like GTK and Qt (games, electron apps, java apps, sandboxed apps,
etc..), the IMF/IME stack needs to be able to listen to key events from
any application, provided it is focused, get the surrounding context,
take field focus, and inject text into clients. This is why some
“unstable” extensions were created, mostly “text-input-unstable-v3”
(
zwp_text_input_v3
) and “input-method-v2” (
zwp_input_methd_v2
)
protocol. With this, there’ll be consistent IM behavior across all
applications without compromising security.
On a side note, this same extension protocol for injecting text can be
used for the speech-to-text accessibility framework. In practice this
can either be done via a virtual input device, or a specific desktop
service mechanism integrated in the toolkits. We have a desktop service
catching voice input, a standalone voice recognizer to convert it to
text, and a virtual keyboard or feature to inject events. For example,
GNOME VoiceInput, Plasma Whisper Integration, QtSpeech, SpeechDispatcher,
Caribou, Onboard, or GNOME Accessibility Services (AT-SPI). We won’t
go into details on that, nor mention text-to-speech, since it’s outside
our scope.
One issue remains though, and it’s related to the key repeat rate
and delay, which on Wayland is implemented client-side. It’s
not implemented by IMs, and tough to handle apparently (
see
).
And that it!
Conclusion
Congratulations for making it this far into the article!
We’ve covered a lot of ground, literally from hardware to the very
abstract components of the graphical input stack.
I have some hope that in the future there’s going to be a more common
way to configure the Wayland input stack across compositors and have
fewer discrepancies and fragmentation. I also wish the XKB stack would
one day get cleanup up, but on this one my hopes are pretty low. It’s
fallen victim to entropy and chaos.
A huge gigantic thanks to “who-t” aka Peter Hutterer, whose
blog
has
been my trusty companion for the past months.
We need more articles like this in the age of AI overlords, so please
share it if you’ve enjoyed it!
Thanks for reading, have a wonderful end of day!
NB: This article compiles my understand, for any correction please
contact me.
If you want to have a more in depth discussion I'm always available by
email or irc
.
We can discuss and argue about what you like and dislike, about new ideas to consider, opinions, etc..
If you don't feel like "having a discussion" or are intimidated by emails
then you can simply say something small in the comment sections below
and/or share it with your friends.
Quoting Qwen3-VL Technical Report
Simon Willison
simonwillison.net
2025-11-27 17:01:11
To evaluate the model’s capability in processing long-context inputs, we construct a video “Needle-in-
a-Haystack” evaluation on Qwen3-VL-235B-A22B-Instruct. In this task, a semantically salient “needle”
frame—containing critical visual evidence—is inserted at varying temporal positions within a lon...
To evaluate the model’s capability in processing long-context inputs, we construct a video “Needle-in-
a-Haystack” evaluation on Qwen3-VL-235B-A22B-Instruct. In this task, a semantically salient “needle”
frame—containing critical visual evidence—is inserted at varying temporal positions within a long video.
The model is then tasked with accurately locating the target frame from the long video and answering the
corresponding question. [...]
As shown in Figure 3, the model achieves a perfect 100% accuracy on videos up to 30 minutes in
duration—corresponding to a context length of 256K tokens. Remarkably, even when extrapolating to
sequences of up to 1M tokens (approximately 2 hours of video) via YaRN-based positional extension,
the model retains a high accuracy of 99.5%.
Hard drives remain a vital component in building high-capacity storage solutions, especially in the data center. IT Home
reports
that
Seagate
is continuing to break barriers on how many TBs can be stored on a single hard drive and has achieved a whopping 6.9TB per platter in its laboratory, making 55TB to 69TB hard drives a possibility for the first time.
Seagate's experimental 6.9TB platter boasts more than double the capacity of platters it uses in official products right now. Outgoing models such as Seagate's 30TB HAMR HDDs use 10 3TB platters to reach maximum capacity. With 6.9TB platters, Seagate will be able to build drives with more than double the capacity of its outgoing drives in the same form factor.
(Image credit: Seagate)
Seagate is leveraging its heat-assisted magnetic recording (HAMR) technology to deliver its 6.9TB platter. If you want to check out how Seagate's HAMR technology works, check out our
previous coverage
. In a nutshell, HAMR uses heat-induced magnetic coercivity to write to a hard drive platter. In Seagate's outgoing drives, this tech is combined with Mozaic 3+ to reduce the media grain size compared to typical HDD platters.
However, these 6.9TB platters are still in development and are not planned to be used for another 5 years. Seagate's roadmap reveals that 6.9TB platters won't be used in official products until 2030. In the meantime, Seagate is working on developing 4TB, 5TB, and 6TB platters that will enter production in 2027, 2028, and 2029, respectively. But the company won't be stopping there; it projects that it will have 7TB to 15TB platters available from 2031 onward. Assuming nothing changes, Seagate could likely have petabyte-sized hard drives before 2040.
Seagate's continuous improvements in hard drive capacity will be vital to keeping up with the increasing demand for hard drives. Despite the rise in SSD maturity, hard drives are still the backbone of the world's long-term storage, thanks to better
reliability
and far superior storage capacity per drive (for the most part) and storage capacity per dollar. Hard drive reliability has only improved as the AI boom gobbles up hard drive orders, leaving HDD manufacturers with
2-year backorders on hard drives
alone.
Luckily, this problem has mostly been regulated to datacenter drives (for now). If you are looking to pick up a new hard drive right now, be sure to check out our best
Black Friday HDD deals for 2025
, which include a 24TB Seagate BarraCuda discounted to just $239.99 (at the time of writing).
Fabricated by teenage brothers in 1911, this unique homebuilt is once again airworthy.
Despite its shortcomings, the Blériot XI was one of the great designs of aviation’s early years. The successful fruit of numerous prior attempts—and failures—by French pioneer aviator Louis Blériot, it was tricky and even dangerous to fly, largely because its horizontal stabilizer had an airfoil like the wing, which could cause the nose to suddenly pitch down during high-speed dives. When Blériot piloted the shoulder-winged monoplane on a historic 23½-mile hop across the English Channel in July 1909, however, he won his design a worldwide stamp of approval beyond its inherent merits. From then on, aviators out to score more firsts in distance, speed, altitude or endurance, or simply out to experience the thrill of early flight for its own sake, wanted a Blériot XI. Besides the examples Blériot produced, a number of other companies on either side of the Atlantic manufactured it under license, while other budding fliers built their own planes based on its basic layout. It was in that last category that the VanDersarl brothers staked their modest claim to fame.
Little is now known about Jules “J.J.” VanDersarl and his younger brother, Frank, except that they lived just outside Denver, Colo.; their mother worked as a housekeeper; and they barely made it through grade school. But both brothers proved to have innate mechanical talents that made them proficient at machining, carpentry and other skills. Given that, it’s not surprising these young men, like a good many others at the time, became enthralled with aviation. J.J. experimented with gliders at age 12, and later, a few months after Blériot’s 1909 Channel flight, he and Frank got more ambitious. Obtaining all the publications and photographs they could, they used those references to build their own Blériot XI in 1911…then learned to fly it.
According to Javier Arango, director of The Aeroplane Collection in Paso Robles, Calif., who now owns the VanDersarl Blériot, the brothers “must have had some guidance and lots of information,” because the dimensions of their airplane are close to those of the original. Their homebuilt differs from the standard Blériot XI in three respects, however. First and foremost, instead of the 25-hp Anzani 3-cylinder radial or Gnome rotary engine that normally powered Blériots, the VanDersarls, using their general knowledge and machining skills, adapted a 4-cylinder inline air-cooled automobile engine with a reworked oil system to aerial use. Just what that engine was remains uncertain, though Arango said it was “close in dimensions” to the power plant used in the Metz, a car equipped with a liquid-cooled engine that the company had planned to adapt to aviation but which never quite materialized.
A second difference, Arango noted, was that “some of the structure around the empennage is placed slightly differently than in most Blériots.” Finally, he said, “The French Blériots were built to a high quality, but our plane was built by teen agers in Colorado who just wanted to go fly—it’s a little rougher than pristine Blériots.”
Even so, the handmade airplane worked remarkably well. “There is a photo of the first flight, which ended up in a landing that broke the landing gear,” Arango reported. “But it was repaired and flew again. Both brothers flew it.”
The VanDersarls went on to fly Curtiss JN-4 Jennys and Standards, and in the 1920s, Arango said, “Frank started an airport and barnstorming operation.” The most remarkable thing, though, is that Frank kept the homebuilt in which he and J.J. had first learned how to fly. “I’m so glad they kept it,” Arango remarked. “This breed of airplane is quite rare. The Smithsonian Institution has only one such aircraft.”
In the 1960s Frank VanDersarl tried to restore the Blériot, but he died before completing the project. After J.J. VanDersarl died in Raton, N.M., in November 1977, the monoplane was exhibited at the Museum of New Mexico. In 1994 it was bought by Joseph Gertler, who loaned it to Dowling College in Bayport, N.Y. There it was further restored by John Zale, Frankie Mineo, Russ Moore and the Bayport Aerodrome Society. Then in 2009 Arango’s C.C. Air Corporation purchased it and added it to The Aeroplane Collection, with the ultimate goal of making it airworthy for the first time in a century.
“When we got it the plane was minimally restored,” Arango explained. “It was extremely authentic.” That meant it served as a useful reference toward the inevitable replacement of deteriorated material and components. “Chuck Wentworth from Antique Aero, who is really the main character in the restoration project, inspected it and went through everything,” he said. “The entire fuselage was in good shape. There were busted wires and turnbuckles that had to be reproduced and replaced to get it back to original condition. Chuck had to find parts of 1911 vintage to get the correct look, based on plans and photos. For example, they’d stuck a fake control wheel in the cockpit for display. We took all of that out.
“The wings were difficult—they were not the same age as the fuselage. They were probably damaged and were repaired or rebuilt by the VanDersarls. It took a lot of work with the wings to make them airworthy. The cotton covering was difficult to work with, and we even had to find the ‘honey-colored coating’ the VanDersarls described. We used a varnish that was tinted to get the correct honey color.”
Though he considered obtaining an Anzani engine, Arango decided to heed the advice of the National Air and Space Museum and “keep it as it was” by reconstructing the original engine. Fortunately for the restoration team, the VanDersarls “left good data on the cylinders, the copper cooling fins—all the specifications we needed to build the engine from scratch. The engine was put together with help from period publications and photos of the original.” The most difficult part was getting period components, but they managed to obtain a 1905 Bosch magneto, a brass carburetor of 1909 vintage, a tachometer, a magneto switch and a 1910 automobile oil gauge. In 2011 Wentworth unveiled the Blériot at the National Aviation Heritage Invitational in Reno, Nev. There on September 18 it won the event’s top award, the RollsRoyce Aviation Heritage Trophy.
Once the four-year project was completed, Arango and his team went through a systematic process toward getting it approved for flight by the Federal Aviation Administration. This presented some challenges, Arango said, since the Blériot XI predated the Civil Aeronautics Administration, let alone the FAA, and “there is no certificate and no paperwork of the age to make it current.” After the FAA inspected the aircraft, however, it finally registered the VanDersarl Blériot as an experimental airplane on August 16, 2012. This meant it could be flown under certain restrictions, such as not carrying a passenger for hire and with limits on the number of flights and travel radius around the airfield. “That was fine by us,” said Arango, “because we were happy to just fly, more or less in a straight line.”
Even with FAA approval, the VanDersarl Blériot underwent testing, reinspection and taxiing trials before it finally got airborne for the first time in more than a century on November 3, 2012. Since then, Arango keeps its flight itinerary at Paso Robles under tight self-imposed restrictions. “It’s a marginal airplane,” he explained, “with a 50-hp engine and very cambered wings that cause a lot of drag. It’s a good-flying airplane, but I’m not going to risk the airframe. It’s one of a kind, touched twice by its creators, and once by Chuck. I wanted it authentic to its own type.”
Originally published in the March 2014 issue of
Aviation History
. To subscribe, click
here
.
The best Black Friday 2025 deals in the UK on the products we love, from electric blankets to sunrise alarms
Guardian
www.theguardian.com
2025-11-27 16:23:15
We’ve cut through the noise to find genuinely good Black Friday discounts on Filter tried-and-tested products across home, tech, beauty and toys • How to shop smart this Black Friday• The best Black Friday beauty deals Like Christmas Day, Black Friday has long since ceased to be a mere “day”. Yuleti...
L
ike Christmas Day, Black Friday has long since ceased to be a mere “day”. Yuletide now seems to start roughly whezn Strictly does, and Black Friday seemed to kick off around Halloween. But now, at last, we’ve reached the day that puts the “Friday” into Black Friday.
Black Friday is a devil worth dancing with if you want to save money on products you’ve had your eye on. Some of the Filter’s favourite items spent most of November floating around at prices clearly designed to make them sell out fast. Other deals have been kept back until now, and some won’t even land until the daftly named Cyber Monday (1 December).
As ever, we’d encourage you not to buy anything unless you really need it and have the budget to do so – read our advice on
how to shop smartly
.
Here’s our fully updated guide to the best genuine Black Friday bargains on the Filter’s favourites, from Anker battery packs to KidiZoom cameras via the espresso machine you loved more than any other product this year.
The key to
shopping smart
on Black Friday, Cyber Monday or any discount event is to know what you want – and we’re here to help you target the good stuff. We’ve tested thousands of products at the Filter in 2025 and warmly recommended hundreds of them, including many that have genuinely good Black Friday discounts.
Instead of listing price cuts on all the products we’ve featured, we’ve focused on the things you’ve liked the most this year, and looked for deals that undercut their long-term average prices by a significant amount. Ideally, their Black Friday price will be their lowest of the year.
We don’t take retailers at their word on discount size, either. Amazon may say it’s “70%” off the RRP, but we study the price history of every item using independent tools such as
the Camelizer
to find out how generous a discount really is. If an item’s price has been all over the place in 2025, we’ll give the average price below instead of a “was …” price, so you can judge how good a deal it is.
Q&A
How is the Filter covering Black Friday?
Show
At the Filter, we believe in buying sustainably, and the excessive consumerism encouraged by Black Friday doesn’t sit easily with us. However, we also believe in shopping smarter, and there’s no denying that it’s often the best time of year to buy big-ticket items that you genuinely need and have planned to buy in advance, or stock up on regular buys such as skincare and cleaning products.
Retailers often push offers that are not as good as they seem, with the intention of clearing out old stock, so we only recommend genuine deals. We assess the price history of every product where it’s available, and we won’t feature anything unless it is genuinely lower than its average price – and we will always specify this in our articles.
We only recommend deals on products that we’ve tested or have been recommended by product experts. What we choose to feature is based on the best products at the best prices chosen by our editorially independent team, free of commercial influence.
The best
Black Friday deals on the Filter’s favourite products
The best home and mattress deals
The best artificial Christmas tree
Habitat 6ft mixed top upswept Christmas tree, £84 (was £120)
Habitat’s gloriously lifelike and easy-to-assemble tree topped our test to find the
best artificial Christmas trees
, and as December rattles towards us it’s inevitably discounted for Black Friday. The code XMAS30 will get you 30% (£36) off.
Heated fleece throw
Silentnight luxury heated throw, from £36 (was £45)
One of Amazon’s best sellers this Black Friday but 25p cheaper at Boots, Silentnight’s toasty fleece blanket was one of the lighter and thinner options in our
best heated throws
roundup. That makes this 120 x 160cm throw ideal for wrapping around yourself (and no-one else) on the sofa as the evenings grow ever colder.
Owlet’s feature-packed smartphone-compatible baby monitor was one of the favourite
baby products
when we spoke to parents last year. If you’d rather not give your £199 to Amazon, John Lewis is only 99p more.
The best combination steam cleaner
Vax Steam Fresh Total Home mop, from £84 (was £160)
Emerging from Stuart Andrews’
best steam cleaners
test as the “best combination cleaner”, Vax’s versatile mop proved easy and effective to use on multiple surfaces and tight corners. The handheld bit detaches easily from the body then slots back in when needed, and you get an array of brushes, scrapers, pads and nozzles. This dirt-blitzing package has dropped more than 40% at Currys and Amazon.
Smart wake-up and reading light
Philips SmartSleep sleep and wake-up light, £139.99 (avg £179.61)
When testing products for his guide to the
best sunrise alarm clocks
, our writer Pete Wise was struck by how well this one worked as a reading light. “Even when a bright setting is selected, the light seems relatively mellow and restful,” wrote Pete, who also liked the range of alarm sounds and audio input option. He found it a little too expensive, however – and it’s still north of £100, but somewhat less so.
A heated airer dries your clothes fast enough to avoid the dreaded stink of slow-dried laundry, and without the cost or noise of a tumble dryer. Lakeland’s three-tier heated airer – the top performer in our
heated airers
test – has proved enduringly popular with the Filter’s readers, and is now at its lowest price ever. Lakeland has also dropped the price of the
airer with cover
to £195.98 for Black Friday.
The best hybrid mattress
Photograph: Jane Hoskyn/The Guardian
Otty Original Hybrid double, £
533.58with code THEFILTER7 (was £647.99)
The most comfortable and supportive foam-and-springs hybrid of all the
mattresses
we’ve tested, the Otty already came at an impressive price of £647.99 for a double, but the Filter’s exclusive code gives you a small but perfectly welcome additional 7% off for Black Friday. For a deeper dive into this cosy mattress, read our
Otty Original Hybrid review
(spoiler: it gets five stars).
One of your favourite Filter recommendations of the year, this gentle
sunrise alarm clock
will wake you up with kittens purring, birdsong, gently brightening light – or a plain old alarm sound if you prefer. It’s been around for a few years and saw a price hike in 2022 (cost-of-waking-up crisis?) before settling at just under £50 from most retailers, so this is a deal worth grabbing.
Mattress “discounts” may seem to be a 24/7/365 thing, but UK watchdogs have given companies short shrift over money-off claims that aren’t all they seem. We’ve certainly noticed Simba playing by the rules lately, and its current 30%-off sale is the first we’ve seen in months. The excellent
Simba Hybrid Pro
, another of our
best mattresses
, is now hundreds of pounds cheaper in all sizes, from single (now £599.25) to super king (now £1,091.22).
Wool mattress topper
Woolroom Deluxe wool topper (double), from £
148.74 (was £174.99)
The sustainably sourced wool in Woolroom’s bedding is a hypoallergenic temperature regulator, helping to keep you warm in winter and cool on hotter nights. The company’s deluxe
mattress topper
adds a touch of softness to a too-hard mattress, and is one of the easiest toppers we tested to move and store. Woolroom’s 35% isn’t quite as big a discount as Amazon’s, but it applies to everything on its site, including duvets, mattresses and linens.
Powerful pressure washer
Bosch UniversalAquatak 135 high pressure washer, £135 (was £209)
Blitz the gunk from your patio, decking, gutters and any flat surface you find yourself unable to resist pointing the nozzle at. Our writer Andy Shaw found the UniversalAquatak to be the most powerful of all the
pressure washers
he tested, and he thought its price was reasonable too. It’s now even cheaper for Black Friday, although not quite its lowest price of 2025 – it was briefly (very briefly) under £120 for Prime Day.
A vacuum cleaner that empties itself? Yes please, said our writer Andy Shaw in his roundup of the
best cordless vacuum cleaners
– and you agreed, making Shark’s ingenious and powerful cordless cleaner one of your favourite products of the year. Vacuums that look after themselves don’t come cheap, and it’s great to see this one heavily discounted at Shark’s own website as well as at Amazon.
You wait a lifetime for a self-emptying vacuum cleaner, then Black Friday brings you two at once. The Eufy X10 was named “best overall” by Stuart Andrews in his guide to the
best robot vacuums
, and it’s already one of the fastest-selling items in Amazon’s Black Friday sale. Its price cut isn’t quite the 38% Amazon suggests, because it cost £579 throughout 2025, but this is still a legitimately good deal.
Damp-destroying dehumidifier
ProBreeze dehumidifier, from £151.99 (was £189.99)
This “workhorse”, which “extracted moisture powerfully” in our
best dehumidifiers
test, has tumbled to its lowest price of the year (except for a few days in May, because no one buys dehumidifiers in May). If the recent cold snap gave you the condensation blues, here’s your chance to snap up the ProBreeze for a chunk below its average Amazon price of just over £180.
Microchip cat flap
SureFlap microchip cat flap, from £55.99 (was £61.99)
Let your cat (and
only
your cat) come and go without the risk of the neighbourhood Billy Six Dinners sneaking in through the flap. One of our top
cat essentials
, the SureFlap hasn’t been this cheap at Amazon since 2023, and Currys is only a penny behind. The moggie-tracking
Connect version
is also discounted at £119.99 – its lowest price since 2022.
Beurer’s “soft and sumptuous” fleece blanket was crowned “best throw overall” in our guide to the
best electric blankets
thanks to its ability to get toasty fast without using much energy. A fiver off is not a massive discount, but this is its cheapest recent price on Amazon, where it normally costs £84.99. Beurer has now matched Amazon’s Black Friday price, dropping from £94.99 to £79.99.
Sort the cold-callers from the welcome visitors when they’re still metres away from your front door, with this outstanding battery-powered doorbell that crashes to its lowest price since Black Friday 2023. Andy Shaw named it the
best video doorbell
overall, but lamented that you also have to fork out for a
Nest Aware subscription
at £80 a year to save recordings.
Subscription-free video doorbell
Tapo D235 video doorbell camera, from £79.99 (avg £101)
The Tapo doorbell camera from router giant TP-link emerged as a fine mid-range choice in Andy Shaw’s test to find the
best video doorbell
, thanks to its good picture quality and ability to record video locally on a microSD card. With more than £30 shaved off Amazon’s average price for the camera, it’s now an even more reasonable buy.
Budget electric blanket
Slumberdown Sleepy Nights electric blanket, king size, from £30.59
(was £45.99)
This Slumberdown Sleepy Nights performed admirably in Emily Peck’s test of
the best electric blankets
, heating quickly to a temperature that was comfortable to keep our reviewer warm through the night. It also has elasticated fitted straps to make fitment easy, and comes in a variety of sizes to suit your bed size. It’s the king-size one that’s been discounted.
Lots of video doorbells and home surveillance systems come with a recurring subscription to access some of their features, which you may wish to avoid. If so, then the Eufy Video Doorbell E340 was Andy Shaw’s pick in his testing of the
best video doorbells
out there. He liked the E340 precisely because of its dual camera setup to make keeping an eye on parcels a breeze, plus the onboard storage to stick it to cloud storage. Reliability of movement detection needed some work, though. At £74.99 from Amazon, it’s also at its lowest price ever this Black Friday from the big online retailer.
Block out the world and drift off to whatever music, podcast or white noise you choose with this comfy silk sleep mask that incorporates flat Bluetooth speakers for pairing with your phone. It impressed our writer Jane Hoskyn in her mission to find
sleep aids
that actually work, but she found it a little pricey – so this discount is very welcome, and makes the Snoozeband an even better
Christmas gift
idea.
Running watch with Spotify
Garmin Forerunner 165 Music smartwatch, £208.05 (was £289)
One of our favourite
fitness tech
gadgets, Garmin’s GPS smartwatch can’t run a
marathon
for you, but it sure can help ease the pain with its pace-tracking tools, offline Spotify support and 19-hour battery life. John Lewis and Amazon are both offering this deal on the aqua green edition of the watch, now at its lowest price ever.
Professional DJ headphones
AiAiAi Audio TMA-2 DJ headphones, £124.94 (was £159)
Many headphones claim to be pro or DJ-level, but this modular set is a favourite with
actual DJs
. DJ and producer
Sophie Lloyd
told the Filter’s Kate Hutchinson that she loves the sound quality, size and durability of these phones, adding that their modular design means “you can buy a new lead or earpieces separately, which is essential when you’re using them all the time”. This Black Friday deal takes them to their lowest price of 2025.
This fab portable speaker boasts 12-hour battery life, durability and a range of swish colours, making it a must-have for
university life
and beyond. It’s a superb piece of kit for the price, with excellent sound quality, nine-metre Bluetooth connectivity and smart TV support.
The best kitchen deals
Affordable Morphy Richards slow cooker
Morphy Richards 3.5
l slow cooker, from £24 (was £34.99)
Our writer Joanne Gould chose Morphy Richards’ ceramic 3.5l model as one of her top budget-friendly
slow cookers
, and this Black Friday deal makes it an even more pocket-friendly purchase. It’s also pocket-sized compared with some of the 7l and 8l beasts you can buy, but it happily accommodated 500g of potatoes, half a lamb shoulder and various vegetables in Joanne’s test.
Having cornered the market in air fryers, Ninja now has its eye on all your kitchen needs, starting with your morning coffee – however you take it, from cold brew to latte. The “sublime espresso”, “ingenious milk frother” and Barista Assist feature of the Ninja Luxe impressed our writer Sasha Muller enough to win it a place in the
best espresso machines
and
best coffee machines
, where Sasha noted that “you get a lot for your money” even at full price.
The best budget kettle in Rachel’s
best kettles
test, the handsome Kenwood looks more expensive than even its RRP suggests, and impresses with a wide pouring spout, single-cup boil and two water windows. Currys has the best Black Friday deal so far, with the white edition dropping to a bargain £27. At John Lewis it’s £28 for white or eggshell blue, while the Amazon deal is for midnight black.
This curious-looking device is a widget on steroids. It brings the nitro beer effect to your Guinness at home, enabling you to pour the black stuff in two-part draught style, just like any good bartender. It’s a brilliant
Christmas gift
idea, now with a third wiped off its price … so, sincere apologies if you bought it last week when we first recommended it. Note you’ll need to buy special
Nitrosurge Guinness
too, but that’s also in the Black Friday sale, at £16.50 for a pack of 10 one-pint cans.
The promise of “ludicrously tasty” espresso and “perfect microfoam for silky cappuccinos and flat whites” proved so irresistible that this was one of the Filter recommendations you loved most in 2025. Our writer Sasha Muller was already wowed by its affordability in his
espresso machines
test, and it’s rarely discounted at all, so we’re not too sad to see it drop just a few pounds for Black Friday.
Capsule coffee machine
Philips L’or Barista Sublime, from £45 (avg £69.40)
The price of this sleek machine has bounced between £105 and about £60 since 2023, only ever dipping to £45 for Black Friday each year. Its compatibility, compactness and coffee impressed the Filter’s cuppa connoisseur, Sasha Muller, enough to be named “best capsule machine” in his bid to find the
best coffee machines
.
If you’re still holding out on buying an air fryer, here’s a rare chance to grab a big-name, big-capacity Ninja without the big price tag. Not quite so big, anyway. Rachel Ogden named the Double Stack XL “best compact air fryer” in her guide to the
best air fryers
, but with its 9.5lL capacity and four cooking levels, this thing can cook a
lot
. Still not cheap, but far below its average price of £229.
You can spend about £500 on a premium blender, but this superb model from Braun costs below £200 even at full price – something our
best blenders
tester, Rachel Ogden, could hardly believe when she named it “best overall”. Hold on to your smoothie, Rachel, because it’s now less than £150, and not just at Amazon.
Tefal is known mostly for its ActiFry tech, so when Rachel Ogden crowned the Tefal Easy Fry Dual XXL as the
best air fryer
, it made sense. She found it to be a sublime all-rounder in her testing, handling both chips and frozen food very well. With an 11-litre capacity, it’s also Tefal’s largest dual zone air fryer, making it handy for cooking a lot of food for larger families when you need to.
Crowned overall winner in Rachel Ogden’s missions to find the
best kettles
, this Bosch beauty now comes at a price offer you can’t refuse – and not just from Amazon. “A brilliant blend of robust form and function” wrote Rachel of this fashionably industrial-looking kettle, whose features include a low minimum boil (300ml), keep-warm setting and touch controls. Now its lowest price ever, in white or black.
December might be just around the corner, but there’s still time to buy a beauty advent calendar. The Boots Beauty Advent calendar is our current top pick (after Cult Beauty and SpaceNK sold out) since it has a brilliant range of full size products, including the bestselling Drunk Elephant Protini Polypeptide cream, a mini MAC Velvet Teddy lip stick and a full-size Sol De Janeiro body spray. Surprisingly, it’s already discounted before the end of the month.
LED face masks are this year’s most coveted beauty purchase – we tested 10 of the most popular
light therapy masks
this year and found the popular Shark Cryoglow lived up to the hype. It’s got daily targeted treatments for ‘better ageing’, ‘blemish repair’ and ‘skin sustain’, so there’s something to suit all ages and skin concerns. Sarah first tested this mask a year ago, and she has been using the mask religiously to help calm her breakouts. For £50 off, it’s an easy recommendation.
Upgrading your
hair dryer
is one of Sarah Matthews’ biggest beauty tips, and if you don’t want to spend hundreds, this is her recommendation. It’s not the fastest money can buy but it has varied heat and speed settings, a precise concentrator nozzle and a diffuser for drying natural curls. It’s easily the best budget-friendly hair dryer, and it’s now the cheapest it’s ever been.
The Dyson Airwrap v Shark FlexStyle debate has been raging on for years now, with both manufacturers recently launching upgraded versions. The original Shark FlexStyle still holds up, with brilliant smoothing brushes, a versatile twisting design and good curling power if you’re willing to spend more time styling. Now’s a brilliant time to buy it for £159 – its lowest ever price.
Water flosser
Waterpik Ultra Professional, from £59.99 (was £91)
Blast the gunk from your gums without having to grapple with floss. The Waterpik Ultra is a countertop model so it takes up more space than the cordless type, but this gives it more versatility and saw it score top marks with our
water flosser
tester Alan Martin. If you’d rather avoid Amazon, you can find it discounted by other retailers, albeit not by as much.
The best IPL device
Philips Lumea 9900 BRI951/01, from £
3
36 (avg £501.33)
IPL (intense pulsed light) hair remover devices promise to banish stubbly regrowth without the pain of waxing and epilation – at a price. The Philips Lumea 9900, Lise Smith’s pick for
best IPL device
overall, has cost as much as £599.99 for much of the year, and occasional discounts rarely go below £450. Amazon’s current price shaves more than £40 off any other Black Friday deal we’ve found for this version, which comes with four attachments.
A bargain beauty Advent calendar
W7 Beauty Blast Advent calendar, £16.95 (was £19.95)
Advent calendars are a Christmas staple, and we’ve seen lots of brands try to put a different spin on them in the past – beauty Advent calendars are some of the most prominent. This W7 Beauty Blast calendar provides excellent value for money at a deal-busting £16.95 from Amazon, especially as it provides genuinely useful products for most folks. The likes of the eyeshadows, primers, lip balms and such are travel-size, but apart from that, Sarah Matthews had little cause for complaint in her ranking of the
best beauty Advent calendars
.
Best toys and games deals
Classic dart board
Winmau Diamond Plus professional bristle dartboard, from £23.76 (avg £35.96)
Get in touch with your inner
Luke Littler
using this classic, professional and surprisingly affordable dart board, which featured in the Filter’s
gift guide
and is now available for under £25 in the Black Friday sale.
Mattel’s strategy game for two to four players was recommended in our guide to
keeping kids entertained in the summer holidays
, and it’s even more useful now that it’s cold and dark. The game is like Tetris with a twist: players compete to fit together blocky coloured pieces on the board, while strategically blocking opponents. Amazon is determined to outdo other retailers on this one, cutting the price to its lowest since 2022.
This blackjack variant is one of those high-stakes card games that’s super easy to learn but fiendishly strategic and addictive, so it slotted perfectly into our
gift guide
and will help keep boredom at bay over Christmas. Zatu offers an additional 5% discount to students and healthcare workers.
Uno fans have more than 700 editions of the game to choose from, but the one that inspired our food columnist Yotam Ottolenghi to get out of the kitchen and recommend a non-edible
Christmas present
was this new version, which dials the ruthlessness up to 11 and will currently set you back just £5.99.
Family board game that isn’t Monopoly
Azul tile laying game, £2
5.49 (avg £31.42)
The Filter team recommended this pattern-building game as an “addictive”
Father’s Day gift
“guaranteed to be a hit”, but it’s far too good to leave to just the dads. It’s mercifully quick to learn and suitable for tweens and up, so you and your Christmas visitors can have a bout underway faster than you can say “read the instructions”. This is the first time its price has dropped much below £30 since 2023.
Family card game
Dobble original, £6.99 (avg £9.16)
Race to find the matching images in this popular observation game – one of our top tips for
keeping kids entertained on long train journeys
. You can mix things up with games-within-games such as “hot potato” and “catch them all”, and it’s versatile enough to suit any number of players from two to eight. This deal isn’t quite the 50% off that Amazon claims (its average price on the site is under £10), but this is its lowest price of 2025.
EA Sports FC 26
EA Sports FC 26 for PS5, from £34.99 (was £69.99)
EA’s FC 26 was released to great fanfare in September, and it’s proved to be one of Amazon’s best Black Friday sellers so far. As Ben Wilson explains in his four-star
review
, this versatile game is a sim offline and a whole other beast online, where it’s purely an esport with shots and goals prioritised over defending. Unusually, Amazon is beaten to the lowest price on this one – by the PlayStation Store, no less.
“The right headgear can save a run when the wind is blowing hard,” wrote LIsa Buckingham in her guide to the
best winter running gear
, noting that Buff’s Thermonet range of beanies are a great choice for men and women because they’re breathable and quick drying as well as reliably warm. It comes in various colours, with the black appropriately enough getting the biggest reductions for Black Friday.
Fruity electrolyte fizzers
Science in Sport hydro electrolyte tablets, from £4.49 (was £7.50)
Electrolyte supplements are a fitness essential because they replace the salts your body loses when you sweat. They can help rehydrate you in hot weather, too, so Lily Smith was wise to include them on her
ultimate festival packing list
. SiS’s tasty electrolyte fizzers can get pricey, so take this chance to stock up.
Blackout tent for two
Coleman Darwin 2 plus blackout tent, from £64.74 (was £99.99)
The classic Coleman Darwin tent has a porch canopy that keeps the rain off your boots and other muddy stuff you’d rather leave outside, writes Tom Bruce in his
camping essentials
guide. It also provides a lovely link between indoors and outdoors. The tent comes in various sizes, but the best deal is on the two-plus blackout version, which claims to “block up to 99% of daylight” to stop you waking up at the crack of dawn.
Same-day upstream Linux support for Snapdragon 8 Elite Gen 5
deepseek-ai/DeepSeek-Math-V2
New on Hugging Face, a specialist mathematical reasoning LLM from DeepSeek. This is their entry in the space previously dominated by proprietary models from OpenAI and Google DeepMind, both of which achieved gold medal scores on the International Mathematical Olympiad ea...
deepseek-ai/DeepSeek-Math-V2
. New on Hugging Face, a specialist mathematical reasoning LLM from DeepSeek. This is their entry in the space previously dominated by proprietary models from OpenAI and Google DeepMind, both of which
achieved gold medal scores
on the International Mathematical Olympiad earlier this year.
We now have an open weights (Apache 2 licensed) 685B, 689GB model that can achieve the same. From the
accompanying paper
:
DeepSeekMath-V2 demonstrates strong performance on competition mathematics. With scaled test-time compute, it achieved gold-medal scores in high-school competitions including IMO 2025 and CMO 2024, and a near-perfect score on the undergraduate Putnam 2024 competition.
Security updates for Thursday
Linux Weekly News
lwn.net
2025-11-27 14:36:11
Security updates have been issued by Debian (kdeconnect, libssh, and samba), Fedora (7zip, docker-buildkit, and docker-buildx), Oracle (bind, buildah, cups, delve and golang, expat, firefox, gimp, go-rpm-macros, haproxy, kernel, lasso, libsoup, libtiff, mingw-expat, openssl, podman, python-kdcproxy,...
Someone Is Trying to ‘Hack’ People Through Apple Podcasts
403 Media
www.404media.co
2025-11-27 14:00:21
For months Apple Podcasts has been randomly opening spirituality and religion podcasts by itself, and one case directing listeners to a potentially malicious website....
Something very strange is happening to the Apple Podcasts app. Over the last several months, I’ve found both the iOS and Mac versions of the Podcasts app will open religion, spirituality, and education podcasts with no apparent rhyme or reason. Sometimes, I unlock my machine and the podcast app has launched itself and presented one of the bizarre podcasts to me. On top of that, at least one of the podcast pages in the app includes a link to a potentially malicious website. Here are the titles of some of the very odd podcasts I’ve had thrust upon me recently (I’ve trimmed some and defanged some links so you don’t accidentally click one):
“5../XEWE2'""""onclic…”
“free will, free willhttp://www[.]sermonaudio[.]com/rss_search.asp?keyword=free%will on SermonAudio”
There was another with a title in Arabic that loosely translates to “Words of Life” and includes someone’s Gmail address. Sometimes the podcasts do have actual audio (one was a religious sermon); others are completely silent. The podcasts are often years old, but for some reason are being shown to me now.
I’ll be honest: I don’t really know what exactly is going on here. And neither did an expert I spoke to. But it’s clear someone, somewhere, is trying to mess with Apple Podcasts and its users.
“The most concerning behavior is that the app can be launched automatically with a podcast of an attacker’s choosing,” Patrick Wardle, a macOS security expert and the creator of Mac-focused
cybersecurity organization Objective-See
, said. “I have replicated similar behavior, albeit via a website: simply visiting a website is enough to trigger Podcasts to open (and a load a podcast of the attacker’s choosing), and unlike other external app launches on macOS (e.g. Zoom), no prompt or user approval is required.”
💡
Do you know anything else about these weird podcasts? I would love to hear from you. Using a non-work device, you can message me securely on Signal at joseph.404 or send me an email at joseph@404media.co.
To caveat straight away: this isn’t
that
alarming. This is not the biggest hack or issue in the world. But it’s still very weird behavior and Apple has not responded to any of my requests for comment for months. “Of course, very much worth stressing, on its own this is not an attack,” Wardle continued. “But it does create a very effective delivery mechanism if (and yes, big if) a vulnerability exists in the Podcasts app.
That said, someone has tried to deliver something a bit more malicious through the Podcasts app. It’s the first podcast I mentioned, with the title “5../XEWE2'""""onclic…”. Maybe some readers have already picked up on this, but the podcast is trying to direct listeners to a site that attempts to perform a cross-site scripting, or XSS, attack. XSS is basically when a hacker injects their own
malicious code into a website that otherwise looks legit
. It’s definitely a low-hanging fruit kind of attack, at least today. I remember it being way, way more common 10 years ago, and it was ultimately what led
to the infamous MySpace worm
.
The weird link is included in the “Show Website” section of the podcast’s page. Visiting that redirects to another site, “test[.]ddv[.]in[.]ua.” A pop-up then says “XSS. Domain: test[.]ddv[.]in[.]ua.”
I’m seemingly not the only one who has seen this. A review left in the Podcasts app just a few weeks ago says “Scam. How does Apple allow this attempted XSS attack?” The person gave the podcast one star. That podcast itself dates from around 2019.
“Whether any of those attempts have worked remains unclear, but the level of probing shows that adversaries are actively evaluating the Podcasts app as a potential target,” Wardle said.
Overall, the whole thing gives a similar vibe to Google Calendar spam, where someone will sneakily add an event to your calendar and include whatever info or link they’re trying to spread around. I remember that being a
pretty big issue a few years ago
.
Apple did not acknowledge or respond to five emails requesting comment. The company did respond to other emails for different articles I was working on across that time.
About the author
Joseph is an award-winning investigative journalist focused on generating impact. His work has triggered hundreds of millions of dollars worth of fines, shut down tech companies, and much more.
My Father Is a Warrior & My Hero: An Interview with Leonard Peltier's Daughter Marquetta
Democracy Now!
www.democracynow.org
2025-11-27 13:50:26
Marquetta Shields-Peltier was just a toddler when her father, Leonard Peltier, was jailed in 1976. During our recent trip to Turtle Mountain Reservation in North Dakota, we spoke to Marquetta about the campaign to free her father and what it meant to see him released in February....
This is a rush transcript. Copy may not be in its final form.
AMY
GOODMAN
:
While we were there, I also had a chance to talk to one of Leonard Peltier’s daughters, who was just a toddler when Leonard Peltier was jailed in 1976.
MARQUETTA
SHIELDS
-
PELTIER
:
My name is Marquetta Shields-Peltier. I’m the daughter of Leonard Peltier. I’m 52 years old. And I’m standing outside my dad’s house. After 49 years of imprisonment, my dad is finally free, and I’m home with him.
AMY
GOODMAN
:
So, you live in Lawrence. You have for decades. Lawrence is not that far from Leavenworth, where he was held for many, many years before moving on to Coleman in Florida. What are your earliest memories of your dad?
MARQUETTA
SHIELDS
-
PELTIER
:
My earliest memory of me and my dad was actually in Marion, Illinois, when he was there, before it was supermax. I was there with my grandmother, Hazel, and my little brother, Waha. And I remember sitting beside my grandma, waiting for my dad to come out those doors for the first time ever. And I remember asking my grandma, I said, “Do you think my dad’s gonna like me?” And she’s like, “Yeah, he’s gonna love you.” And I said, “Is it OK if I hug my dad?” You know? And she was like, “Yeah, of course. Go hug him.” And she kind of pushed me in that door, towards the door. And when it opened and he came out, he just smiled, and I ran to him. And I just remember him hugging me, and I was like, “That’s my dad. That’s my dad,” you know, because I hadn’t — I don’t have any memories prior to that. They said we lived together in Oregon, but I don’t remember that. They said we saw him in Canada when he — before they extradited him, but I don’t remember that.
AMY
GOODMAN
:
How old were you when you saw him in Marion?
MARQUETTA
SHIELDS
-
PELTIER
:
I think I was around 7 or 8 years old, yeah.
AMY
GOODMAN
:
So, what was it like through those years? I mean, it’s the only thing you knew, but to visit him only behind bars?
MARQUETTA
SHIELDS
-
PELTIER
:
When I was young, it was really confusing. It was hard to understand why, you know, because he kind of protected me, probably because I was so young and I didn’t understand what was going on. But it was bittersweet, because I loved going there. And, like, for me, that was normal, you know, to see my dad there. Even though there’s dads around me with my other friends and stuff, I just — for me, that was normal.
But as I got older and started to understand what was going on, why he was there, I started to resent it, you know, because every time 3:00 came around, I hated it, because I knew I was going to have to leave my dad, and I wouldn’t see him again, especially that last day of the visit. We would usually spend about a week. And that last day of visits, I would just, like, “Man, I’m not going to see my dad again for six, seven months,” you know? And then, as I got older, it was just like, “Oh my god, what if I don’t get to see him again next time?”
You know, so, eventually I packed up my kids and moved to Kansas so I could be closer to him. And then that worked out for about three years. And then, from there, they moved him again across the country. So, yeah, we didn’t get to have the relationship I thought we would during that time, just because he was way over in Pennsylvania, then Florida, and I was stuck in Kansas.
AMY
GOODMAN
:
And what was your understanding from early on, and did it change, of why he was behind bars?
MARQUETTA
SHIELDS
-
PELTIER
:
Yeah, I just — like, I knew things had happened that were, you know, not — I’ve always been told and taught that my dad was there because he wanted better for me as his daughter, as a Native person. He wanted people to respect me, not only as his daughter, but just as a Native person, you know, to understand that we are not property or we are not animals or savages, that we are human beings, just like everybody else.
And as I got older and started understanding his case and what was, you know, the details of it, then it went from, you know, resenting — I never resented my dad, but I resented the government, and I resented — you know, but it went from not knowing the extent of it to knowing the full extent of it and just being proud of — like, even though I prayed for him to get out all the time, I knew what he stood for, and I was proud. And I had to, you know, keep fighting for him, because I knew that someday, someday, he would get out, you know? And he did. He did, which is unbelievable to me still.
AMY
GOODMAN
:
When did you hear that Biden had commuted his sentence and sentenced him to home confinement?
MARQUETTA
SHIELDS
-
PELTIER
:
I don’t remember the date exactly, but it was sometime at the end of January. It was just crazy, because I was planning on leaving. I was going to leave the country and just disappear, because I — after his parole was denied in June of ’24 — I think it was ’24 — I basically thought my dad was going to die there. So I had given up on everything, and I was getting ready to disappear into Canada and just disappear.
But I was sleeping on my mom’s couch that morning, and I heard the phone ring. And then I heard my mom, and she said, “What?” And I thought the worst, of course, that they called to tell me my dad was dead. And then my cousin was crying, and she said, “Marquetta, I’m so happy.” So I was like, “What are you talking about?” She’s like, “Your dad’s getting out of prison.” I was like, you know, like — I cussed in my mom’s house. And I usually — unless I’m joking with her. I just was like, “You’re lying. You’re lying to me. You’re lying to me. My dad’s” — She’s like, “They’re — Joe Biden” — and, you know, this, that and the other. And I just — I couldn’t — I didn’t know what to do. I just — I froze. And I still can’t remember if I called my nephew or if I called my brother, who I called, but I called somebody, and I asked them. I was like, “Is it true? My dad’s getting out of prison?” And they’re like, “Yeah.”
I’m so thankful to millions of people for the last 49 years of my life that helped pray for him, that helped write letters, that helped make phone calls, that sent signed petitions. You know, that’s all those people in it and to help bring my dad home.
But the thing of it is, is people don’t understand that, you know, when my dad went to prison, so did we. You know, we were out here free, but we weren’t free. We were out here struggling when he was in there struggling. And I was blessed enough to have people like my grandmother and my mom to show me that, you know what, it’s going to be OK. It’s going to be OK.
AMY
GOODMAN
:
Leonard Peltier means so much to so many people around the world. Talk about what he means, not just to you as his daughter, but to you as an Indigenous woman.
MARQUETTA
SHIELDS
-
PELTIER
:
Oh, man, my dad, I told him this before. He’s my hero. He’s the definition of what a warrior should be, you know, to be able to stand strong and still come out of that prison smiling, to be able to set an example. And, like, I look up to my dad, because I don’t know very many people that could go through the stuff he’s been through and still have a smile on his face. And it makes me proud to call him my dad. You know, that’s my dad.
AMY
GOODMAN
:
Marquetta Shields-Peltier, the daughter of Leonard Peltier, speaking in September at Turtle Mountain Reservation in North Dakota, where Leonard has been living since being released from prison in February.
And that does it for today’s show. To see all of
Democracy Now!
's
coverage
of Leonard Peltier and our
interview
with him over the years behind bars, you can go to democracynow.org. Special thanks to Denis Moynihan, Charina Nadura, Sam Alcoff and Zazu, the newshound.
Democracy Now!
is produced with Mike Burke, Renée Feltz, Deena Guzder, Messiah Rhodes, Nermeen Shaikh, María Taracena, Nicole Salazar, Sara Nasser, Charina Nadura, Sam Alcoff, Tey-Marie Astudillo, John Hamilton, Robby Karran, Hany Massoud and Safwat Nazzal. Our executive director is Julie Crosby. Special thanks to Becca Staley, Jon Randolph, Paul Powell, Mike Di Filippo, Miguel Nogueira, Hugh Gran, Carl Marxer, David Prude, Dennis McCormick, Matt Ealy, Anna Özbek, Emily Andersen, Dante Torrieri and Buffy Saint Marie Hernandez. I'm Amy Goodman. Thanks so much for joining us.
The original content of this program is licensed under a
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
. Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.
Have you ever noticed how
just
before sharing your work, you suddenly spot all its flaws? This isn't just a coincidence or bad luck. It's your social brain kicking into gear.
When making something with others in mind, you activate the parts of your brain wired for social interaction. This social brain is disproportionately powerful – it's the same neural machinery that makes us instinctively notice faces in a crowd or prioritize human voices over background noise.
Think of it like lifting with your legs instead of your back. Your legs are designed for carrying heavy loads safely and efficiently, while your back is more vulnerable and prone to injury when misused. Similarly, your social brain is optimized for clear communication and spotting problems, while your solitary thinking is more susceptible to blind spots and lazy reasoning.
Here are some ways you can deliberately trigger this superpower:
Imagine a specific person.
What questions would they have? What background knowledge are they missing? What would they disagree with?
Speak your ideas aloud.
The act of verbalization forces clarity in a way silent thinking doesn't. It can also help you spot overly complicated ways of saying something.
Share drafts with friends.
Even just the
anticipation
of real feedback sharpens your thinking.
Turning on your social brain can also help with Writer's Block. When I'm staring at a blank page, I try to imagine explaining my half-formed idea to a curious friend. Suddenly, I "hear" their natural follow-up questions, guiding my next steps.
Important caveat: Relying too heavily on your social brain can lead to people-pleasing and safe, boring work. You might avoid real creative risks if you rely on it too much. The sweet spot is using your social brain as a clarity tool, not as the ultimate judge of what's worth creating. Use it to sharpen the ideas you care about, not to replace them with what you think others want to hear.
Next time you're struggling with a project—whether it's writing, designing, coding, or something else entirely—try shifting your perspective. Instead of wrestling with your ideas alone, lean on the strength of your social brain. Your thoughts become sharper, logical gaps become obvious, and unnecessary complications fall away. It's not a panacea, but it can be a helpful tool.
Your social brain isn't just for socializing; it can also be a powerful intellectual tool. All you need to do is remember to switch it on!
Face transplants promised hope. Patients were put through the unthinkable
I
n the early hours of 28 May 2005, Isabelle Dinoire woke up in a pool of blood. After fighting with her family the night before, she turned to alcohol and sleeping tablets “to forget”, she later said.
Reaching for a cigarette out of habit, she realized she couldn’t hold it between her lips. She understood something was wrong.
Isabelle crawled to the bedroom mirror. In shock, she stared at her reflection: her nose, lips, and parts of her cheeks were gone, replaced by a raw, mangled wound.
While Isabelle was unconscious, her beloved dog Tania, a cross between a Labrador and a Beauceron, had chewed away her features.
“I could see a pool of blood next to me,” Isabelle
told
the BBC. “And the dog was licking the blood. But I couldn’t imagine that it was my blood or my face.”
On 27 November 2005, Isabelle received the world’s first face transplant at University Hospital, CHU Amiens-Picardie, in northern France. The surgery was part of an emerging field called
vascularized composite allotransplantation
(VCA), that transplants parts of the body as a unit: skin, muscle, bone and nerves.
Two teams, overseen by Bernard Devauchelle, Sylvie Testelin and Jean-Michel Dubernard, grafted a donor’s nose, lips and chin onto Isabelle’s skull. The donor, a 46-year-old woman, had died by suicide. The donor graft was painstakingly attached: sensory nerves to restore feeling, motor nerve fibres for movement, arteries and veins to establish blood flow. The operation involved 50 people and took more than 15 hours.
The results were
presented
to the press the following February, when Isabelle amazed the world by speaking through her new mouth and drinking water from a cup. “I now have a face like everyone else,” she said. “A door to the future is opening.”
The case for face transplants seemingly made, several teams scrambled to perform their nation’s first. The US saw the first partial face transplant (2008), then the first full one (2011); the first African American recipient (2019); the first face and double hand transplant combined (2020); the first to include an eye (2023). There have been about 50 face transplants to date, and each milestone brought new grants, donations and prestige for the doctors and institutions involved.
The patients, meanwhile, continue on living as they can. Some of them, like Isabelle, have suffered greatly. Others, like Joe DiMeo, who received the world’s first double hand and face transplant at NYU Langone in 2020, find new ways to forge a career by selling their stories online. But he and his wife Jessica, a nurse, are constantly trolled, and the spectre of rejection never leaves.
Jessica and Joe DiMeo (the first double hand and face transplant recipient).
Photograph: April Kirby
For the past six years, I have been researching the history of face transplants, interviewing surgeons and patients in the US, France, China, Spain, Italy, Mexico and Canada. I have contributed to surgical articles and conferences, brought patient voices to the table, and advised on a critical Department of Defense-funded study to regulate all kinds of VCA.
What I’ve found is alarming: it’s a field where negative data is often buried, driven by funding battles and institutional rivalry. In places where publicity functions as marketing, some clinics expose patients to intrusive media attention. Support networks are uneven, and few patients are prepared for the toll of lifelong immunosuppressants. Add to this picture a set of ethical challenges: face transplants take otherwise healthy people with disfigured faces and turn them into lifetime patients.
People tend to remember a dramatic “
before and after
”. The reality is different.
Dallas Wiens
thought he’d won the medical lottery when he became America’s first full face transplant recipient in 2011. The 25-year-old electrician was electrocuted while painting a church; this destroyed his face and his sight. Dallas worried that his daughter Scarlette would be bullied for how he looked. He wanted to give “something back” to veterans. He wanted to be able to hail a cab.
Like Isabelle, Dallas was grateful to his donor and surgeons. He attended medical conferences so surgeons could see the results. He met prospective patients and was courted by global media as proof that face transplants worked.
For several years, the narrative held; then reality intruded.
The anti-rejection drugs that kept his new face alive destroyed his kidneys. Dallas had repeated episodes of rejection, each requiring stronger immunosuppression. He lived in Texas, in poverty, with his beloved wife Annalyn, who was also blind. Dallas’ primary medication alone cost $120 per month, a significant expense on disability benefits.
“It’s one thing to be told about risks,” Dallas told me when his kidneys were failing. “It’s another thing to experience them.”
In the US, now the world’s leader in face transplants, the Department of Defense has bankrolled most operations, treating them as a frontier for wounded veterans while private insurers refuse to cover the costs.
With insurance unwilling to pay until the field proves its worth, surgeons have been eager to show results. A 2024
JAMA Surgery study
reported five-year graft survival of 85% and 10-year survival of 74%, concluding that these outcomes make face transplantation “an effective reconstructive option for patients with severe facial defects”.
Yet patients like Dallas tell a different story. The study measures survival, but not other outcomes such as psychological wellbeing, impact on intimacy, social life and family functioning, or even comparisons with reconstruction.
Most surgeons care about their patients, though they will have their own personal ambitions. Globally, there are perhaps 20 (mostly male) specialized surgeons capable of face transplants; nobody could become part of that elite group without ambition, for themselves, and for the field. And what can they do, surgeons say, if the system doesn’t provide?
It’s a double-bind. Without proof of success, face transplants are experimental. And because the procedures are experimental, patients’ long-term needs aren’t covered by grants, leaving patients to carry the burden.
Dallas and Annalyn Wiens. Dallas died in 2024 of kidney failure.
Photograph: April Kirby
“I don’t have $100 for Ubers to and from hospital,” Dallas said, explaining how public transport led to infections, given his weakened immune system, and infections could make his face reject. “But if I don’t attend, it can be seen as non-compliance. Is that fair?”
On 27 September 2024, Dallas died suddenly at his home in Fort Worth. His death certificate lists complications due to electrocution, his original 2008 accident. His wife Annalyn still doesn’t know what happened. “His body gave up,” she said. “He was constantly tested and made to feel like a guinea pig. I wanted his body to be left alone.”
Annalyn had Dallas’ body cremated quickly, fearful that the DoD or Yale would want it for research. Neither did, but it says something about the gap between surgical intentions and patient experiences that this was her fear.
That fear was also expressed privately to me by a member of Isabelle’s immediate family, who wants to remain anonymous. From their perspective, Isabelle’s face transplant was not a success, despite launching an entire field.
In fact, nobody expected France to do the first face transplant. Those in the know presumed it would be Cleveland Clinic, where Maria Siemionow had spent years refining the method and the ethics.
In contrast, Devauchelle’s first application for ethical approval was rejected. In the early 2000s, French ethicists were, like those in the UK, concerned about immunosuppressant risks – and psychological ones. How could anyone cope with seeing another person’s face in the mirror?
For his next, successful bid, Devauchelle teamed up with Dubernard, who was not only an influential member of the French National Assembly, but also the surgeon who had made history in 1998 with the
world’s first hand transplant
. And making history has always brought glory, especially for transplant surgeons.
What of Isabelle? Three months before her operation, she signed a contract with British documentary maker Michael Hughes, agreeing to let cameras document her transformation in exchange for payment. The Times of London
revealed
this deal, showing how a vulnerable, suicidal woman with no face had been effectively “sold” even before surgery. Isabelle was swayed by the promise of a bright future, though that never transpired.
Describing how he watched the blood flow into Isabelle’s lips in surgery, Dubernard compared himself to the prince who awakened Sleeping Beauty, adding: “I still see her image among the stars in my dreams”.
Isabelle felt less like a princess than a
circus animal.
After the transplant, she spoke of being tormented: “Everyone would say: Have you seen her? It’s her. It’s her … And so I stopped going out completely.”
Living with a stranger’s face was as psychologically difficult as ethicists feared. Two years after the transplant she
spoke
to the strangeness of having “someone else’s” mouth. “It was odd to touch it with my tongue. It was soft. It was horrible.”
And then one day she found a new hair on her chin – “It was odd. I’d never had one. I thought, ‘It’s me that has given it life, but the hair is hers.’”
Surgeons and ethicists observed that Isabelle wasn’t given proper alternatives, and she wasn’t in a good state of mind; the most the French team has conceded is that she wasn’t an “ideal patient”.
Isabelle might have fared better in a country like Finland, where transplants are anonymous. Patients and families are not harassed by journalists – as Isabelle and her family were – and clinics don’t use patients as media opportunities.
Instead, Isabelle never resumed a normal life, never returned to work or good mental health, and from 2013 experienced regular episodes of rejection. In 2010 she contracted cervical cancer, followed by lung cancer. She died in 2016, though her surgeons deny this was connected to immunosuppressant use.
In fact, Isabelle’s face died before she did; after it became necrotic, it was removed and replaced with a graft from her thigh. As she told her family, she “didn’t want to die without a face”.
I also learned from Isabelle’s immediate family member that her wellbeing declined dramatically after her transplant, and that she was in “psychological distress” when consenting for the procedure. “They took her away from us, so we didn’t have the power to dissuade or counsel her.” And after each psychiatric appointment, she would come home “at the lowest, full of guilt and suicidal desires”. More than once, according to her, she attempted suicide after her transplant; this story isn’t part of the record.
Robert Chelsea, the first African American to receive a new face, wanted to kiss his daughter’s cheek. Now he can, but his daughter can’t look at him the same way.
“Only when he opens his mouth, I know it’s him”, she says, otherwise he’s a stranger. Today, Robert is in and out of hospital and unable to find income.
Robert knows race makes a difference – the sinister history of medical
experimentation on Black bodies
means African Americans are less likely to donate organs of any kind. And scientific medicine privileges whiteness; until Robert’s surgery, the hospital hadn’t considered that donors with a wide range of skin colours were needed.
Robert Chelsea, the first African American face transplant recipient.
Photograph: Lol Crawley
Once a successful businessman, Robert is now reliant on GoFundMe campaigns; his car has been repossessed, and he can’t get to church. He suffers through rejections and infections, and he cannot afford caregivers. Sometimes, he gets so weak he can’t even call an ambulance. And if he did, that would be an extra cost he can’t afford either. Aftercare is the biggest issue for US face transplant recipients. Yet the JAMA study only measured outcomes by graft survival; not whether patients could work, afford medications, maintain relationships. It did not track financial ruin, mental health or quality of life. It counted 10 deaths but not how people died or what their final years were like.
No one tracked Dallas’s failing kidneys or Robert’s repossessed car.
These patients are pioneers. During the second world war, plastic surgeon Archibald McIndoe treated severely burned pilots. His patients formed the
Guinea Pig Club,
a brotherhood that acknowledged their experimental status openly. They received lifelong care, peer support, and recognition for their contribution to advancing surgery. We can’t say the same for face transplant recipients.
One question remains: how can science and medicine ethically innovate, without knowing what has gone before?
Most innovation follows the same trend: the possibility is raised, there’s ethical debates, someone breaks cover and there’s a race to join in.
These innovations usually end one of three ways: they fade quietly into history, they implode in scandal, or they mature into a stable, standardized practice.
Reality is now forcing that question for face transplants. Roughly 20% of patients have died – from rejection, kidney failure, heart failure. That’s an unacceptably high toll for an elective, supposedly “life-enhancing” procedure, especially when we still can’t agree on who is an ideal candidate, how to measure success, or what long-term support actually looks like.
We have seen this before – in lobotomy, a field that faded out. The Portuguese physician Egas Moniz won the Nobel Prize for developing lobotomy in 1949 and 3,500 brutal procedures were performed.
The same arc unfolded for vaginal meshes in the 1990s. Introduced with great fanfare, they caused chronic pain and organ damage, led to millions in lawsuits, and became synonymous with prioritizing profits over patient safety. Unlike face transplants, vaginal mesh victims found strength in numbers – 100,000 alone in the US took legal action.
A more successful innovation story is IVF, which moved from controversial “test-tube baby” experiments into mainstream medicine by rigorous patient selection, improved safety standards, and proper regulation – all of which were led by surgeons.
Which path will face transplants take? The numbers are already slipping – fewer procedures since the 2010s as outcomes falter and budgets shrink. And unless the field raises its standards, enforces rigorous follow-up, and commits to transparent, systematic data sharing that actually includes patients and their families, there’s no way to demonstrate real success. Without that, face transplants aren’t headed for evolution or stability; they’re headed straight for the dustbin of medical history.
Isabelle’s loved one is watching closely. It is not easy for her to speak out, even now, for fear that the family will be harassed by journalists. But, she says, “I must find the strength.”
“Isabelle did not want the transplants to continue. She had collected documents to prove the various dysfunctions and told me that after her death I could talk about everything because during her lifetime, she feared the medical team, the pressure was too strong.”
She felt obliged to be upbeat, to represent innovation success, whatever the cost – an insurmountable amount of pressure for someone who is already vulnerable
“You only hear from the complainers,” one delegate told me earlier this year, after I spoke at a talk to face transplant surgeons at an international conference in Helsinki. “The happy patients are quietly living their lives.”
Such claims are meaningless without accurate data. And patients are often scared to tell surgeons the truth; even without the power imbalances of healthcare, people feel ungrateful or worry about being a “bad patient”.
Yet they are the only ones who know if face transplants are a success. They are the ones who live with the reality of a face transplant after the news has broken and the cameras move on.
Are things changing? Not fast enough. The same pattern repeats with each innovation: surgeons pioneer, patients sacrifice, papers get published, the field moves on.
I saw Robert recently in an online meeting to discuss a Department of Defense grant aimed at improving standards. He had recently left the hospital after another round of rejection, and was one of three patients sharing experiences.
He looked tired and unimpressed.
“Everybody here is getting paid,” he said. “Except us. Who is feeding our children, while we are making history?”
Fay Bound-Alberti is professor of Modern History at King’s College London. Her new book, The Face: A Cultural History will be published by Penguin in February 2026
LowType introduces the concept of "type expressions" in method arguments. When an argument's default value resolves to a type instead of a value then it's treated as a type expression. Now you can have types in Ruby in the simplest syntax possible:
classMyClassincludeLowTypedefsay_hello(greeting:String)# Raises exception at runtime if greeting is not a String.endend
Default values
Place
|
after the type definition to provide a default value when the argument is
nil
:
If you need a multi-line return type/value then I'll even let you put the
-> { T }
on multiple lines, okay? I won't judge. You are a unique flower
🌸
with your own style, your own needs. You have purpose in this world and though you may never find it, your loved ones will cherish knowing you and wish you were never gone:
defsay_farewell_with_a_long_method_name(farewell:String)->{::Long::Name::Space::CustomClassOne|::Long::Name::Space::CustomClassTwo|::Long::Name::Space::CustomClassThree}# Code that returns an instance of one of the above types.end
Instance variables
To define typed
@instance
variables use the
type_[reader, writer, accessor]
methods.
These replicate
attr_[reader, writer, accessor]
methods but also allow you to define and check types.
Type Reader
type_readername:String# Creates a public method called `name` that gets the value of @namename# Get the value with type checkingtype_readername:String|'Cher'# Gets the value of @name with a default value if it's `nil`name# Get the value with type checking and return 'Cher' if the value is `nil`
Type Writer
type_writername:String# Creates a public method called `name=(arg)` that sets the value of @namename='Tim'# Set the value with type checking
Type Accessor
type_accessorname:String# Creates public methods to get or set the value of @namename# Get the value with type checkingname='Tim'# Set the value with type checkingtype_accessorname:String|'Cher'# Get/set the value of @name with a default value if it's `nil`name# Get the value with type checking and return 'Cher' if the value is `nil`name='Tim'# Set the value with type checking
ℹ️
Multiple Arguments
You can define multiple typed accessor methods just like you would with
attr_[reader, writer, accessor]
:
type_accessorname:String|nil,occupation:'Doctor',age:Integer|33name# => niloccupation# => Doctor (not type checked)age='old'# => Raises ArgumentTypeErrorage# => 33
Local variables
type()
alias:
low_type()
To define typed
local
variables at runtime use the
type()
method:
my_var=typeMyType|fetch_my_object(id:123)
my_var
is now type checked to be of type
MyType
when assigned to.
Don't forget that these are just Ruby expressions and you can do more conditional logic as long as the last expression evaluates to a value:
my_var=typeString|(say_goodbye||'Hello Again')
ℹ️
Enumerables
To use the
Array[]
/
Hash[]
enumerable syntax with
type()
you must add
using LowType::Syntax
when including LowType:
includeLowTypeusingLowType::Syntax
Syntax
[T]
Enumerables
Array[T]
and
Hash[T]
class methods represent enumerables in the context of type expressions. If you need to create a new
Array
/
Hash
then use
Array.new()
/
Hash.new()
or Array and Hash literals
[]
and
{}
. This is the same syntax that
RBS
uses and we need to get use to these class methods returning type expressions if we're ever going to have inline types in Ruby.
RuboCop
also suggests
{}
over
Hash[]
syntax for creating hashes.
|
Union Types / Default Value
The pipe symbol (
|
) is used in the context of type expressions to define multiple types as well as provide the default value:
To allow multiple types separate them between pipes:
my_var = TypeOne | TypeTwo
The last
value
/
nil
defined becomes the default value:
my_var = TypeOne | TypeTwo | nil
If no default value is defined then the argument will be required.
-> { T }
Return Type
The
-> { T }
syntax is a lambda without an assignment to a local variable. This is valid Ruby that can be placed immediately after a method definition and on the same line as the method definition, to visually look like the output of that method. It's inert and doesn't run when the method is called, similar to how default values are never called if the argument is managed by LowType. Pretty cool stuff yeah? Your type expressions won't keep re-evaluating in the wild
🐴
, only on class load.
ℹ️
Note:
A method that takes no arguments must include empty parameters
()
for the
-> { T }
syntax to be valid;
def method() -> { T }
.
value(T)
Value Expression
alias:
low_value()
To treat a type as if it were a value, pass it through
value()
first:
defmy_method(my_arg:String|MyType|value(MyType))# => MyType is the default value
Performance
LowType evaluates type expressions on
class load
(just once) to be efficient and thread-safe. Then the defined types are checked per method call.
However,
type()
type expressions are evaluated when they are called at
runtime
on an instance, and this may impact performance.
Evaluation
Validation
ℹ️
Example
Method param types
🟢
Class load
🟠
Runtime
def method(name: T)
Method return types
🟢
Class load
🟠
Runtime
def method() -> { T }
Instance types
🟢
Class load
🟠
Runtime
type_accessor(name: T)
Local types
🟠
Runtime
🟠
Runtime
type(T)
Architecture
LowType only affects the class that it's
include
d into. Class methods
Array[]
/
Hash[]
are modified for the type expression enumerable syntax (
[]
) to work, but only for LowType's internals (using refinements) and not the
include
d class. The
type()
method requires
using LowType::Syntax
if you want to use the enumerable syntax but will still only affect the
Array[]
/
Hash[]
class methods of the
include
d class.
Config
Copy and paste the following and change the defaults to configure LowType:
LowType.configuredo|config|# Set to :log or :none to disable the raising of an exception when types are invalid. [UNRELEASED]config.error_mode=:error# Set to :value to show a concatenated inspect of the invalid param when an error is raised. Or :none to redact.# Great for debugging, bad for security, and makes tests harder to write when the error messages are so dynamic.config.output_mode=:typeconfig.output_size=100# Set to true to type check all elements of an Array/Hash (not just the first)config.deep_type_check=false# The "|" pipe syntax requires a monkey-patch but can be disabled if you don't need union types with default values.# This is the only monkey-patch in the entire library and is a relatively harmless one, see "syntax/union_types.rb".# Set to false and typed params will always be required, as there's no "| nil" syntax (remove type to make optional)config.union_type_expressions=trueend
Types
Basic types
String
Integer
Float
Array
Hash
nil
represents an optional value
ℹ️
Any class/type that's available to Ruby is available to LowType, you just might need to
require
it.
Complex types
Boolean
(accepts
true
/
false
) [UNRELEASED]
Tuple
(subclass of
Array
)
Status
(subclass of
Integer
) - TODO: Check integer to be a valid HTTP status code
Headers
(subclass of
Hash
)
HTML
(subclass of
String
) - TODO: Check that string is HTML
JSON
(subclass of
String
) - TODO: Check that string is JSON
XML
(subclass of
String
) - TODO: Check that string is XML
Integrations
Because LowType is low-level it should work with method definitions in any framework out of the box. With that in mind we go a little further here at free-software-by-shadowy-figure-co to give you that extra framework-specific-special-feeling:
Sinatra
include LowType
in your modular
Sinatra::Base
subclass to get Sinatra specific return types.
LowType will automatically add the necessary
content_type
[UNRELEASED] and type check the return value:
require'sinatra/base'require'low_type'classMyApp<Sinatra::BaseincludeLowType# A simple string response type.get'/'do->{String}'body'end# Standard types Sinatra uses.get'/'do->{Array[Integer,Hash,String]}[200,{},'<h1>Hello!</h1>']end# Types specifically for Sinatra.get'/'do->{Tuple[Status,Headers,HTML]}[200,{},'<h1>Hello!</h1>']endend
Rubocop
Because we're living in the future, Rubocop isn't ready for us. Put the following in your
.rubocop.yml
:
# Support LowType return value "-> { T }" syntax.Style/TrailingBodyOnMethodDefinition:Enabled:falseLayout/IndentationConsistency:Enabled:falseLayout/MultilineBlockLayout:Enabled:falseStyle/DefWithParentheses:Enabled:falseLint/Void:Enabled:false# Support Array[]/Hash[] syntax.Style/RedundantArrayConstructor:Enabled:false
Installation
Add
gem 'low_type'
to your Gemfile then:
bundle install
Philosophy
🦆
Duck typing is beautiful.
Ruby is an amazing language
BECAUSE
it's not typed. I don't believe Ruby should ever be fully typed, but you should be able to sprinkle in types into some areas of your codebase where you'd like self-documentation and a little reassurance that the right values are coming in/out.
🌀
Less DSL. More types.
As much as possible LowType looks just like Ruby if it had types. There's no special method calls for the base functionality, and defining types at runtime simply uses a
type()
method which almost looks like a
type
keyword, had Ruby implemented types.
🤖
AI makes you dumb.
AI is theoretically a cool concept but in practice capitalism just uses it to steal wealth.
Social media has become a reminder of something precious we are losing in the age of LLMs:
unique voices
.
Over time, it has become obvious just how many posts are being generated by an LLM. The tell is the voice. Every post sounds like it was posted by the same social media manager.
If you rely on an LLM to write all your posts, you are making a mistake.
Your voice is an asset. Not just what you want to say, but how you say it.
Your voice is unique. It is formed from your lifetime of lived experiences. No one's voice will be exactly like yours.
Your voice becomes recognizable. Over many posts it becomes something people subconsciously connect with, recognize, trust, and look forward to.
Your voice provides the framework for the impression you leave in a job interview, while networking at a meet-up, or with a co-worker.
Years ago I got a job thanks to my blog posts. A manager wanted my voice influencing their organization. Your voice is an asset.
Your voice matures and becomes even more unique with time and practice.
LLMs can rob you of that voice, and the rest of us lose something precious in the process.
Having an LLM write "in your voice" is not the same. Your voice is not static. It changes with the tides of your life and state of mind. Your most impactful message may come because it was the right moment and you were in the right frame of mind.
Let your voice grow with use. Let it be unique.
Do not let one of your greatest assets fade into atrophy, wilted by cognitive laziness.
Write in
your
voice.
I do not care what the linguistic remix machine juggles into being.
I care what you have to say.
Don't be a scary old guy: My 40s survival strategy with charm
Hi, it’s
Takuya
.
Last week I had my birthday and turned 41 (November 19th).
When I was younger, I could never really picture what life in my 40s would look like. It’s this vague age where you don’t have a clear image of how you’re supposed to live, right? Even if I try to look back at my dad at this age, he was always at work during the day, so he’s not much of a reference.
I make a living as an indie developer
, and thanks to what I built up through my 20s and 30s, I can live the way I do now. Compared to a typical Japanese salaryman, I can join childcare much more flexibly, and I get to spend a lot of time with my kid. I’ve even made some “mom friends (mama-tomo)” at kindergarten.
In this post, I'd like to share my survival strategy for the 40s. As the title says, the conclusion is:
“charm”
— being warm and approachable.
Let me explain why I think this kind of charm matters so much for middle-aged men.
TL;DR
“You’ve got presence” just means “You look older now.”
Make a smile. A grumpy middle-aged guy is just scary
Be humble. The more achievements you stack, the more people shrink back
Use the charm of contrast
“You’ve got presence” just means “You look older now.”
For students, guys in their 40s are full-on old dudes. At least that’s how I saw them. It’s basically the age of school teachers.
When I hit my late 30s, people around me started to say things like:
“You’ve got
kanroku
now.”
In Japanese,
kanroku
means something like “gravitas” or “presence.”
And no, they didn’t mean my belly was growing.
At first, I secretly thought:
“Finally, my life experience is starting to radiate as an aura!”
…but over time I realized that wasn’t it at all. It simply meant: I got older.
In other words, “You’ve aged,” “You look older now,” wrapped in the most positive wording possible.
I mean, think about it. What even is “aura,” really? lol
If I’ve really built up so much life experience, why am I still getting scolded by kindergarten teachers for being late to the bus pick-up? It doesn’t feel like I’m walking around radiating some wise, dignified aura.
Make a smile. A grumpy middle-aged guy is just scary
Having gravitas doesn’t actually help you that much. If a middle-aged guy is frowning, shoulders slumped, walking around with a dark cloud over him, you just want to keep your distance, right?
If you wrap yourself in charm instead, you can cancel out like half of that rough “old guy presence.”
I used to work part-time at a café. When I asked the manager why he decided to hire me, he said:
“Because your smile was good.”
On YouTube as well, I try to make a smile in my videos. A smile is a key ingredient of charm.
To cancel out this heavy “presence,” I want to be even more intentional about smiling and staying approachable in my daily life.
Be humble. The more achievements you stack, the more people shrink back
If you just keep doing something for a long time, your achievements naturally pile up. And if you’re lucky, some of them end up being work that lots of people praise you for.
But then one day you realize: The friends who used to argue with you freely and push back hard are suddenly keeping their distance.
Indie dev is already lonely enough. But the more “achievements” you stack, the more your potential conversation partners quietly disappear.
I read somewhere that Hirohiko Araki, the manga artist behind
JoJo’s Bizarre Adventure
, once said that he’s become so successful and revered that people are now scared of him, and no one gives him advice anymore.
It makes sense. If you imagine giving feedback to a famous author or legendary director, it feels terrifying, right?
That’s why Araki-sensei apparently gets really happy when someone ignores that aura, doesn’t shrink back, and just casually says what they think.
From what I’ve seen of him on TV and such, he seems full of charm. He smiles, teaches kids, and comes across as very gentle and kind. He’s a great example. If even someone like him still gets put on a pedestal and loses people to bounce ideas off, I absolutely have no business acting all high and mighty.
Use the charm of contrast
The more serious and stern someone looks, the more powerful their smile becomes. That contrast is what makes it hit. In Japanese, we even have a word for this:
gap moe
(ギャップ萌え) — the charm that comes from an unexpected contrast in someone’s personality or appearance.
Take guitarist Eddie Van Halen, for example:
When I picture an amazing guitarist, I tend to imagine someone completely lost in their own world, making intense faces while they play.
But Eddie often turns to the crowd and smiles, clearly trying to entertain and enjoy it
with
them. That attitude is incredibly likeable.
Programmers are a good example of a job that’s hard for people to picture. When mom friends ask what I do and I say:
“I’m a programmer.”
I often get:
“Ah, I don’t really know much about computers…”
It’s not that they’re rejecting it; they just can’t imagine what I actually do, so they don’t know how to respond. The fewer shared reference points you have with someone, the more important it is to approach them with a soft, relaxed attitude. You don’t have to explain everything in detail. If they can at least feel that “he seems to enjoy his work and looks like he’s having fun,” that’s more than enough.
So that’s what I want to value in my 40s.
Lately, I’ve been feeling like younger people show me more respect than before. Precisely because of that, this is the time to
not
act superior, but instead live humbly and gently. I want to keep learning from younger generations and be inspired by them. I want to stay in touch with new values and cultures all the time. To do that, I have to break through the “gravitas” barrier myself. And I think charm is essential for that.
If you’re around my age, what do
you
want to value in your life?
I’d love to hear.
Here’s to good 40s for all of us!
Thanks for reading. Inkdrop is a Markdown-focused note-taking app for developers. It’s not about having tons of features — its strengths are the clean design and simplicity. If you’re looking for a clean and simple notes app, check it out:
SyncKit is a
production-ready sync engine
that makes building local-first applications trivial.
"Add
sync.document()
to your app, get real-time sync automatically."
The problem:
Building sync from scratch takes months. Existing solutions are complex (Yjs), expensive (Firebase), or don't work offline (Supabase).
The solution:
SyncKit gives you production-ready sync in 3 lines of code.
constsync=newSyncKit()awaitsync.init()constdoc=sync.document<Todo>('todo-123')awaitdoc.update({completed: true})// ✨ Works offline, syncs automatically, resolves conflicts
🎬 See It In Action
Real-time collaboration with offline resilience:
Watch tasks sync instantly across tabs—even while offline. The example app demonstrates SyncKit's offline-first capabilities combined with smart browser storage to create a seamless collaborative experience.
✨ Why SyncKit?
🚀
Works When Internet Doesn't
True offline-first architecture—not just caching. Your app works perfectly on planes, trains, tunnels, and coffee shops with spotty WiFi.
import{SyncKit}from'@synckit-js/sdk'import{SyncProvider,useSyncDocument}from'@synckit-js/sdk/react'// Initialize (works offline-only, no server needed!)constsync=newSyncKit()awaitsync.init()functionApp(){return(<SyncProvidersynckit={sync}><TodoApp/></SyncProvider>)}functionTodoApp(){const[todo,{ update }]=useSyncDocument<Todo>('todo-1')if(!todo||!todo.text)return<div>Loading...</div>return(<div><inputtype="checkbox"checked={todo.completed}onChange={(e)=>update({completed: e.target.checked})}/><span>{todo.text}</span></div>)}
graph TD
A[Your Application<br/>React/Vue/Svelte] --> B[SyncKit SDK<br/>TypeScript]
B -->|Simple API| B1[document, text, counter]
B -->|Framework adapters| B2[React/Vue/Svelte hooks]
B -->|Offline queue| B3[Storage adapters]
B --> C[Rust Core Engine<br/>WASM + Native]
C -->|80% of use cases| C1[LWW Sync]
C -->|Collaborative editing| C2[Text CRDTs]
C -->|Advanced features| C3[Custom CRDTs<br/>counters, sets]
C --> D[IndexedDB Storage<br/>Your local source of truth]
D -.->|Optional| E[SyncKit Server<br/>TypeScript/Python/Go/Rust]
E -->|Real-time sync| E1[WebSocket]
E -->|Persistence| E2[PostgreSQL/MongoDB]
E -->|Security| E3[JWT auth + RBAC]
style A fill:#e1f5ff,stroke:#333,stroke-width:2px,color:#1a1a1a
style B fill:#fff4e1,stroke:#333,stroke-width:2px,color:#1a1a1a
style C fill:#ffe1e1,stroke:#333,stroke-width:2px,color:#1a1a1a
style D fill:#e1ffe1,stroke:#333,stroke-width:2px,color:#1a1a1a
style E fill:#f0e1ff,stroke:#333,stroke-width:2px,color:#1a1a1a
// Note: Text CRDT API is planned for v0.2.0consttext=sync.text('document-456')awaittext.insert(0,'Hello ')text.subscribe(content=>editor.setValue(content))// Character-level sync, conflict-free convergence
// Note: Counter API is planned for v0.2.0constcounter=sync.counter('likes-789')awaitcounter.increment()// Conflict-free counter (additions never conflict)
In addition to the following, see the
tests folder
for more example
.prompt
files.
Basic prompt with stdin
---model: anthropic/claude-sonnet-4-20250514---
Summarize this text: {{STDIN}}
cat article.txt | ./runprompt summarize.prompt
The special
{{STDIN}}
variable always contains the raw stdin as a string.
Structured JSON output
Extract structured data using an output schema:
---model: anthropic/claude-sonnet-4-20250514input:schema:text: stringoutput:format: jsonschema:name?: string, the person's nameage?: number, the person's ageoccupation?: string, the person's job---
Extract info from: {{text}}
echo"John is a 30 year old teacher"| ./runprompt extract.prompt
# {"name": "John", "age": 30, "occupation": "teacher"}
Fields ending with
?
are optional. The format is
field: type, description
.
Chaining prompts
Pipe structured output between prompts:
echo"John is 30"| ./runprompt extract.prompt | ./runprompt generate-bio.prompt
The JSON output from the first prompt becomes template variables in the second.
CLI overrides
Override any frontmatter value from the command line:
The team of people working on editors and editor support at Tarides is excited to announce the release of
ocaml-eglot
! The project is part of Tarides’ efforts to improve the OCaml developer experience across different platforms and workflows, a high-priority goal continuously evolving with community feedback.
Bringing Emacs integration to OCaml’s LSP server benefits both the user and the maintainer. If you use Emacs and want to start using OCaml, or switch to a more simplified setup, check out the
ocaml-eglot
repository
on GitHub to try the new Emacs minor mode.
This post will give you some background to the development of the new tool, as well as the benefits and limitations of LSP, and the features of
ocaml-eglot
. Let’s dive in!
The Problem: ‘Editor Burnout’
The goal of the
ocaml-eglot
project was to address a problem the engineers had dubbed
editor burnout
. Developers rely on editors to simplify their coding workflow, and over the years, the creation of more and more editor features has transformed editors into sophisticated, feature-rich development environments. However, all these features need to be added and maintained in every editor. Maintaining support for so many different features across different editors, including updating the support every time something changes on the language server's end, can quickly become untenable. ‘Editor burnout’ refers to the pressure this puts on maintainers.
In OCaml, the editor-agnostic server
Merlin
is used to provide IDE-like services. By providing contextual information about the code, Merlin lets developers use simple text editors to write OCaml and benefit from features that typically come from a fully integrated IDE. However, Merlin also had a high maintenance cost due to each code editor needing its own integration layer.
So, now that we understand the problem, what is the solution?
LSP and OCaml
LSP, or the
Language Server Protocol
, is a widely documented open protocol that standardises the interactions between an editor and a server providing IDE services. LSP defines a collection of standard features across programming languages, which has contributed to its widespread adoption. This adoption has made LSP a standard protocol across editors, including
Visual Studio Code
,
Vim
,
Emacs
, and many more.
The language server implementation for LSP in OCaml is
ocaml-lsp
. It uses Merlin as a library. It was originally designed to integrate with
Visual Studio Code
when paired with the
vscode-ocaml-platform
plugin. We can significantly reduce the maintenance burden by relying on LSP's defaults for editor compatibility and only providing support for OCaml-specific features. This benefits not only the maintainers, but also the user by ensuring the plugins remain performant, compatible, maintainable, and up-to-date.
LSP aims to be compatible with as many languages as possible, making some assumptions about how those languages are structured and function. Inevitably, these assumptions cannot cover all the features of every language. This is true of OCaml, where the editing experience relies on custom features outside the scope of the LSP.
The solution to this incompatibility is to create a
client-side extension
that covers what the editor’s standard LSP support does not. That way, we have both the basic LSP compatibility and an extension that adds support for OCaml-specific features. As we’ve hinted above, this has the added benefit of keeping the maintenance workload on the editor side down by delegating the standard LSP handling to the generic LSP plug-ins.
As an editor popular with the OCaml community, let’s take a brief look at how Emacs and OCaml work together. In Emacs, developers can attach a "buffer"/file to a major mode to handle a feature of a language like OCaml: features like syntax highlighting, for example. One file is always attached to just one major mode.
OCaml has four major modes:
caml-mode
: the original,
tuareg
: a full reimplementation of
caml-mode
and the most common choice by users,
ocaml-ts-mode
: an experimental version of
caml-mode
based on tree-sitter grammar,
neocaml
: an experimental full reimplementation of
tuareg
based on tree-sitter grammar.
Now, we can also attach one or multiple
minor-mode
s to a file, and this is where
ocaml-eglot
comes into play. For example, we can use a major mode (we generally recommend Tuareg) and link
ocaml-eglot
to it as a minor mode, thereby attaching LSP features to all files in which Tuareg is active.
Eglot is the default LSP client bundled with Emacs, and
ocaml-eglot
provides full OCaml language support in Emacs as an alternative to Merlin integration. (By the way, thanks to the
ocaml-eglot
client using LSP’s defaults, its code size is a lot smaller than the traditional OCaml
Emacs
mode, which also makes it easier to maintain!).
The ideal user of
ocaml-eglot
is someone who is already an active Emacs user and wants to start using OCaml with minimal start-up hassle. The simplified configuration, automated setup, and consistency across different editors and languages are helpful both to people new to OCaml and to seasoned users with multiple editors, since they improve the workflow. The plugin supports all the features of the integration of Merlin into Emacs,
merlin.el
, meaning that users don’t lose any functionality with the new system. The
ocaml-eglot
project is also actively maintained, and users can expect regular future updates and a tool that evolves with the times.
Creating OCaml-Eglot
Let's peek behind the curtain at the development of
ocaml-eglot
. There are two common approaches that developers who implement server languages tend to use to add features outside of the LSP. These are
Code Actions
and Custom Requests:
Code Action: A contextual action that can be triggered from a document perspective, can perform a file modification, and potentially broadcast a command that can be interpreted by the client. Code Actions are more
‘integrated’, which means that they sometimes even work ‘out of the box’ with the client. However, they are limited in terms of interactions and the command lifecycle.
Custom Request: Not formally part of the protocol, but since LSP is a protocol layered on top of a regular server that can handle JSON RPC messages and responses, developers can still use arbitrary requests to provide extra features. Custom Requests give developers more power to add interactions and experiences, but always need specific editor integration.
The design process behind OCaml-eglot essentially boiled down to identifying all the features offered by
merlin.el
that were not covered by the LSP, and then adding them using Code Actions or Custom Requests. During this process, the developers asked themselves two questions to help them decide which approach to use:
Should the feature be configured by arguments that are
independent of the context:
If the answer is yes, they used a Custom Request; if no, they used a Code Action.
Does the feature require
additional interaction
such as choosing one option from a set of possible results?: If yes, they used a Custom Request; if no, they used a Code Action.
Of course, things were a little more complicated than this in reality, but it still gives you a good idea of the types of technical decisions the team made during development.
Try it Out!
Install
ocaml-eglot
by checking out its
GitHub repository
and following the instructions. When you have had a chance to test it out in your projects, please share your experience on
OCaml Discuss
to give other users an idea of what to expect and the maintainers an idea on what to improve!
Installing
ocaml-eglot
is just like installing a regular Emacs package. It is available on
Melpa
and can be installed in many different ways, for example with GNU’s
use package
. More detailed instructions are available
in the repo’s readme
, including instructions on recommended configurations for
ocaml-eglot
.
Features
Some of the features that
ocaml-eglot
comes with are:
Error navigation: Quickly jump to the next or previous error(s).
Type information: Display types under cursor with adjustable verbosity and navigate enclosing expressions.
Code generation: Pattern match construction, case completion, and wildcard refinement via the ‘destruct’ feature.
Navigation: Jump between language constructs like let, module, function, match, and navigate phrases and pattern cases.
Search: Find definitions, declarations, and references. The team also recently introduced a new Xref Backend inspired by one used by Jane Street for years.
Check out the project's readme to discover the full list of commands offered by
ocaml-eglot
. The new mode is ‘agile’, meaning that the team can also incubate new features quickly, like the
refactor extract at toplevel
.
You can connect with us on
Bluesky
,
Mastodon
,
Threads
, and
LinkedIn
or sign up to our mailing list to stay updated on our latest projects. We look forward to hearing from you!
We made a promise to never brick your device. Here's a progress report:
1. July 2024 - open sourced
our firmware
2. December 2024 - built or commissioned server clients in
Ruby
,
Elixir
, and
Python
3. January 2025 - began selling
BYOD licenses
-
DIY build docs coming soon
4. February 2025 -
launched Framework
, a beautiful and free e-ink UI kit
5. February 2025 - onboarded a senior engineer to focus on OSS (hi
Brooke
!)
But there's still 1 concern.
As more TRMNL devices opt into our lifetime-access hosted platform,
how will we handle growing server costs?
Introducing the Unbrickable Pledge.
As of February 18, 2025, the entire TRMNL team, operating under TRMNL Holdings LLC, hereby affirms our intent to release the core web application source code if and when we ever become insolvent as a company.
Sharing source code is the right thing to do for our customers. Preventing e-waste is the right thing to do for our planet. And sharing How is the right thing to do for TRMNL.
We hope this alleviates concerns by those who are much better at math than us and wondered:
how is this business model possible
?
It's possible because of you.
To staying focused,
Ryan and the TRMNL team
TPUs vs. GPUs and why Google is positioned to win AI race in the long term
As I find the topic of Google TPUs extremely important, I am publishing a comprehensive deep dive, not just a technical overview, but also strategic and financial coverage of the Google TPU.
Topics covered:
The history of the TPU and why it all even started?
The difference between a TPU and a GPU?
Performance numbers TPU vs GPU?
Where are the problems for the wider adoption of TPUs
Google’s TPU is the biggest competitive advantage of its cloud business for the next 10 years
How many TPUs does Google produce today, and how big can that get?
Gemini 3 and the aftermath of Gemini 3 on the whole chip industry
Let’s dive into it.
The history of the TPU and why it all even started?
The story of the Google Tensor Processing Unit (TPU) begins not with a breakthrough in chip manufacturing, but with a realization about math and logistics. Around 2013, Google’s leadership—specifically Jeff Dean, Jonathan Ross (the CEO of Groq), and the Google Brain team—ran a projection that alarmed them. They calculated that if every Android user utilized Google’s new voice search feature for just three minutes a day, the company would need to double its global data center capacity just to handle the compute load.
At the time, Google was relying on standard CPUs and GPUs for these tasks. While powerful, these general-purpose chips were inefficient for the specific heavy lifting required by Deep Learning: massive matrix multiplications. Scaling up with existing hardware would have been a financial and logistical nightmare.
This sparked a new project. Google decided to do something rare for a software company: build its own custom silicon. The goal was to create an
ASIC (Application-Specific Integrated Circuit)
designed for one job only: running TensorFlow neural networks.
Key Historical Milestones:
2013-2014:
The project moved really fast as Google both hired a very capable team and, to be honest, had some luck in their first steps. The team went from design concept to deploying silicon in data centers in just 15 months—a very short cycle for hardware engineering.
2015:
Before the world knew they existed, TPUs were already powering Google’s most popular products. They were silently accelerating Google Maps navigation, Google Photos, and Google Translate.
2016
:
Google officially unveiled the TPU at Google I/O 2016.
This urgency to solve the “data center doubling” problem is why the TPU exists. It wasn’t built to sell to gamers or render video; it was built to save Google from its own AI success. With that in mind, Google has been thinking about the »costly« AI inference problems for over a decade now. This is also one of the main reasons why the TPU is so good today compared to other ASIC projects.
The difference between a TPU and a GPU?
To understand the difference, it helps to look at what each chip was originally built to do. A GPU is a “general-purpose” parallel processor, while a TPU is a “domain-specific” architecture.
The GPUs were designed for graphics. They excel at parallel processing (doing many things at once), which is great for AI. However, because they are designed to handle everything from video game textures to scientific simulations, they carry “architectural baggage.” They spend significant energy and chip area on complex tasks like caching, branch prediction, and managing independent threads.
A TPU, on the other hand, strips away all that baggage. It has no hardware for rasterization or texture mapping. Instead, it uses a unique architecture called a Systolic Array.
The “Systolic Array” is the key differentiator. In a standard CPU or GPU, the chip moves data back and forth between the memory and the computing units for every calculation. This constant shuffling creates a bottleneck (the Von Neumann bottleneck).
In a TPU’s systolic array, data flows through the chip like blood through a heart (hence “systolic”).
It loads data (weights) once.
It passes inputs through a massive grid of multipliers.
The data is passed directly to the next unit in the array without writing back to memory.
What this means, in essence, is that a TPU, because of its systolic array, drastically reduces the number of memory reads and writes required from HBM. As a result, the TPU can spend its cycles computing rather than waiting for data.
Google’s new TPU design, also called Ironwood also addressed some of the key areas where a TPU was lacking:
They enhanced the SparseCore for efficiently handling large embeddings (good for recommendation systems and LLMs)
It increased HBM capacity and bandwidth (up to 192 GB per chip). For a better understanding, Nvidia’s Blackwell B200 has 192GB per chip, while Blackwell Ultra, also known as the B300, has 288 GB per chip.
Improved the Inter-Chip Interconnect (ICI) for linking thousands of chips into massive clusters, also called TPU Pods (needed for AI training as well as some time test compute inference workloads). When it comes to ICI, it is important to note that it is very performant with a Peak Bandwidth of 1.2 TB/s vs Blackwell NVLink 5 at 1.8 TB/s. But Google’s ICI, together with its specialized compiler and software stack, still delivers superior performance on some specific AI tasks.
The key thing to understand is that because the TPU doesn’t need to decode complex instructions or constantly access memory, it can deliver significantly higher Operations Per Joule.
For scale-out, Google uses Optical Circuit Switch (OCS) and its 3D torus network, which compete with Nvidia’s InfiniBand and Spectrum-X Ethernet. The main difference is that OCS is extremely cost-effective and power-efficient as it eliminates electrical switches and O-E-O conversions, but because of this, it is not as flexible as the other two. So again, the Google stack is extremely specialized for the task at hand and doesn’t offer the flexibility that GPUs do.
Performance numbers TPU vs GPU?
As we defined the differences, let’s look at real numbers showing how the TPU performs compared to the GPU. Since Google isn’t revealing these numbers, it is really hard to get details on performance. I studied many articles and alternative data sources, including interviews with industry insiders, and here are some of the key takeaways.
The first important thing is that there is very limited information on Google’s newest TPUv7 (Ironwood), as Google introduced it in April 2025 and is just now starting to become available to external clients (internally, it is said that Google has already been using Ironwood since April, possibly even for Gemini 3.0.). And why is this important if we, for example, compare TPUv7 with an older but still widely used version of TPUv5p based on Semianalysis data:
TPUv7 produces 4,614 TFLOPS(BF16) vs 459 TFLOPS for TPUv5p
TPUv7 has 192GB of memory capacity vs TPUv5p 96GB
TPUv7 memory Bandwidth is 7,370 GB/s vs 2,765 for v5p
We can see that the performance leaps between v5 and v7 are very significant. To put that in context, most of the comments that we will look at are more focused on TPUv6 or TPUv5 than v7.
Based on analyzing a ton of interviews with Former Google employees, customers, and competitors (people from AMD, NVDA & others), the summary of the results is as follows.
Most agree that TPUs are more cost-effective compared to Nvidia GPUs, and most agree that the performance per watt for TPUs is better. This view is not applicable across all use cases tho.
A Former Google Cloud employee:
»If it is the right application, then they can deliver much better performance per dollar compared to GPUs. They also require much lesser energy and produces less heat compared to GPUs. They’re also more energy efficient and have a smaller environmental footprint, which is what makes them a desired outcome.
The use cases are slightly limited to a GPU, they’re not as generic, but for a specific application, they can offer as much as 1.4X better performance per dollar, which is pretty significant saving for a customer that might be trying to use GPU versus TPUs.«
Similarly, a very insightful comment from a Former Unit Head at Google around TPUs materially lowering AI-search cost per query vs GPUs:
»TPU v6 is 60-65% more efficient than GPUs, prior generations 40-45%«
This interview was in November 2024, so the expert is probably comparing the v6 TPU with the Nvidia Hopper. Today, we already have Blackwell vs V7.
Many experts also mention the speed benefit that TPUs offer, with a Former Google Head saying that TPUs are 5x faster than GPUs for training dynamic models (like search-like workloads).
There was also a very eye-opening interview with a client who used both Nvidia GPUs and Google TPUs as he describes the economics in great detail:
»If I were to use eight H100s versus using one v5e pod, I would spend a lot less money on one v5e pod. In terms of price point money, performance per dollar, you will get more bang for TPU. If I already have a code, because of Google’s help or because of our own work, if I know it already is going to work on a TPU, then at that point it is beneficial for me to just stick with the TPU usage.
In the long run, if I am thinking I need to write a new code base, I need to do a lot more work, then it depends on how long I’m going to train. I would say there is still some, for example, of the workload we have already done on TPUs that in the future because as Google will add newer generation of TPU, they make older ones much cheaper.
For example, when they came out with v4, I remember the price of v2 came down so low that it was practically free to use compared to any NVIDIA GPUs.
Google has got a good promise so they keep supporting older TPUs and they’re making it a lot cheaper. If you don’t really need your model trained right away, if you’re willing to say, “I can wait one week,” even though the training is only three days, then you can reduce your cost 1/5.«
Another valuable interview was with a current AMD employee, acknowledging the benefits of ASICs:
»I would expect that an AI accelerator could do about probably typically what we see in the industry. I’m using my experience at FPGAs. I could see a 30% reduction in size and maybe a 50% reduction in power vs a GPU.«
We also got some numbers from a Former Google employee who worked in the chip segment:
»When I look at the published numbers, they (TPUs) are anywhere from 25%-30% better to close to 2x better, depending on the use cases compared to Nvidia. Essentially, there’s a difference between a very custom design built to do one task perfectly versus a more general purpose design.«
What is also known is that the real edge of TPUs lies not in the hardware but in the software and in the way Google has optimized its ecosystem for the TPU.
A lot of people mention the problem that every Nvidia »competitor« like the TPU faces, which is the fast development of Nvidia and the constant »catching up« to Nvidia problem. This month a former Google Cloud employee addressed that concern head-on as he believes the rate at which TPUs are improving is faster than the rate at Nvidia:
»The amount of performance per dollar that a TPU can generate from a new generation versus the old generation is a much significant jump than Nvidia«
In addition, the recent data from Google’s presentation at the Hot Chips 2025 event backs that up, as Google stated that the TPUv7 is 100% better in performance per watt than their TPUv6e (Trillium).
Even for hard Nvidia advocates, TPUs are not to be shrugged off easily, as even Jensen thinks very highly of Google’s TPUs. In a podcast with Brad Gerstner, he mentioned that when it comes to ASICs, Google with TPUs is a »special case«. A few months ago, we also got an article from the WSJ saying that after the news publication The Information published a report that stated that OpenAI had begun renting Google TPUs for ChatGPT, Jensen called Altman, asking him if it was true, and signaled that he was open to getting the talks back on track (investment talks). Also worth noting was that Nvidia’s official X account posted a screenshot of an article in which OpenAI denied plans to use Google’s in-house chips. To say the least, Nvidia is watching TPUs very closely.
Ok, but after looking at some of these numbers, one might think, why aren’t more clients using TPUs?
Where are the problems for the wider adoption of TPUs
The main problem for TPUs adoption is the ecosystem. Nvidia’s CUDA is engraved in the minds of most AI engineers, as they have been learning CUDA in universities.
Google has developed its ecosystem internally but not externally, as it has used TPUs only for its internal workloads until now. TPUs use a combination of JAX and TensorFlow, while the industry skews to CUDA and PyTorch (although TPUs also support PyTorch now). While Google is working hard to make its ecosystem more supportive and convertible with other stacks, it is also a matter of libraries and ecosystem formation that takes years to develop.
It is also important to note that, until recently, the GenAI industry’s focus has largely been on training workloads. In training workloads, CUDA is very important, but when it comes to inference, even reasoning inference, CUDA is not that important, so the chances of expanding the TPU footprint in inference are much higher than those in training (although TPUs do really well in training as well – Gemini 3 the prime example).
The fact that most clients are multi-cloud also poses a challenge for TPU adoption, as AI workloads are closely tied to data and its location (cloud data transfer is costly). Nvidia is accessible via all three hyperscalers, while TPUs are available only at GCP so far. A client who uses TPUs and Nvidia GPUs explains it well:
»Right now, the one biggest advantage of NVIDIA, and this has been true for past three companies I worked on is because AWS, Google Cloud and Microsoft Azure, these are the three major cloud companies.
Every company, every corporate, every customer we have will have data in one of these three. All these three clouds have NVIDIA GPUs. Sometimes the data is so big and in a different cloud that it is a lot cheaper to run our workload in whatever cloud the customer has data in.
I don’t know if you know about the egress cost that is moving data out of one cloud is one of the bigger cost. In that case, if you have NVIDIA workload, if you have a CUDA workload, we can just go to Microsoft Azure, get a VM that has NVIDIA GPU, same GPU in fact, no code change is required and just run it there.
With TPUs, once you are all relied on TPU and Google says, “You know what? Now you have to pay 10X more,” then we would be screwed, because then we’ll have to go back and rewrite everything. That’s why. That’s the only reason people are afraid of committing too much on TPUs. The same reason is for Amazon’s Trainium and Inferentia.«
These problems are well known at Google, so it is no surprise that internally, the debate over keeping TPUs inside Google or starting to sell them externally is a constant topic. When keeping them internally, it enhances the GCP moat, but at the same time, many former Google employees believe that at some point, Google will start offering TPUs externally as well, maybe through some neoclouds, not necessarily with the biggest two competitors, Microsoft and Amazon. Opening up the ecosystem, providing support, etc., and making it more widely usable are the first steps toward making that possible.
A former Google employee also mentioned that Google last year formed a more sales-oriented team to push and sell TPUs, so it’s not like they have been pushing hard to sell TPUs for years; it is a fairly new dynamic in the organization.
Google’s TPU is the biggest competitive advantage of its cloud business for the next 10 years
The most valuable thing for me about TPUs is their impact on GCP. As we witness the transformation of cloud businesses from the pre-AI era to the AI era, the biggest takeaway is that the industry has gone from an oligopoly of AWS, Azure, and GCP to a more commoditized landscape, with Oracle, Coreweave, and many other neoclouds competing for AI workloads. The problem with AI workloads is the competition and Nvidia’s 75% gross margin, which also results in low margins for AI workloads. The cloud industry is moving from a 50-70% gross margin industry to a 20-35% gross margin industry. For cloud investors, this should be concerning, as the future profile of some of these companies is more like that of a utility than an attractive, high-margin business. But there is a solution to avoiding that future and returning to a normal margin: the ASIC.
The cloud providers who can control the hardware and are not beholden to Nvidia and its 75% gross margin will be able to return to the world of 50% gross margins. And there is no surprise that all three AWS, Azure, and GCP are developing their own ASICs. The most mature by far is Google’s TPU, followed by Amazon’s Trainum, and lastly Microsoft’s MAIA (although Microsoft owns the full IP of OpenAI’s custom ASICs, which could help them in the future).
While even with ASICs you are not 100% independent, as you still have to work with someone like Broadcom or Marvell, whose margins are lower than Nvidia’s but still not negligible, Google is again in a very good position. Over the years of developing TPUs, Google has managed to control much of the chip design process in-house. According to a current AMD employee, Broadcom no longer knows everything about the chip. At this point, Google is the front-end designer (the actual RTL of the design) while Broadcom is only the backend physical design partner. Google, on top of that, also, of course, owns the entire software optimization stack for the chip, which makes it as performant as it is. According to the AMD employee, based on this work split, he thinks Broadcom is lucky if it gets a 50-point gross margin on its part.
Without having to pay Nvidia for the accelerator, a cloud provider can either price its compute similarly to others and maintain a better margin profile or lower costs and gain market share. Of course, all of this depends on having a very capable ASIC that can compete with Nvidia. Unfortunately, it looks like Google is the only one that has achieved that, as the number one-performing model is Gemini 3 trained on TPUs. According to some former Google employees, internally, Google is also using TPUs for inference across its entire AI stack, including Gemini and models like Veo. Google buys Nvidia GPUs for GCP, as clients want them because they are familiar with them and the ecosystem, but internally, Google is full-on with TPUs.
As the complexity of each generation of ASICs increases, similar to the complexity and pace of Nvidia, I predict that not all ASIC programs will make it. I believe outside of TPUs, the only real hyperscaler shot right now is AWS Trainium, but even that faces much bigger uncertainties than the TPU. With that in mind, Google and its cloud business can come out of this AI era as a major beneficiary and market-share gainer.
Recently, we even got comments from the SemiAnalysis team praising the TPU:
»Google’s silicon supremacy among hyperscalers is unmatched, with their TPU 7
th
Gen arguably on par with Nvidia Blackwell. TPU powers the Gemini family of models which are improving in capability and sit close to the pareto frontier of $ per intelligence in some tasks«
How many TPUs does Google produce today, and how big can that get?
Here are the numbers that I researched:
"I'm Not Going to Give Up": Leonard Peltier on Indigenous Rights, His Half-Century in Prison & Coming Home
Democracy Now!
www.democracynow.org
2025-11-27 13:01:50
In September, Democracy Now! host Amy Goodman sat down with longtime political prisoner and Indigenous activist Leonard Peltier for his first extended television and radio broadcast interview since his release to home confinement in February. Before his commutation by former President Joe Biden, the...
This is a rush transcript. Copy may not be in its final form.
AMY
GOODMAN
:
In this special broadcast, we spend the hour with longtime Indigenous activist Leonard Peltier. In February, he was released from a federal prison in Florida after spending nearly half a century behind bars for a crime he says he did not commit. President Biden, on his last day in office, commuted Peltier’s life sentence to home confinement. Biden’s decision followed mounting calls by tribal leaders and supporters around the world in a decadeslong, community-led campaign fighting for his freedom.
In the 1970s, Peltier was involved with the American Indian Movement, known as
AIM
. In 1975, two
FBI
agents and one young
AIM
activist were killed in a shootout on the Pine Ridge Reservation in South Dakota. Two
AIM
members were later arrested for killing the agents. At the trial, the jury acquitted them. Leonard Peltier was arrested later, tried separately and convicted. Peltier has always maintained his innocence.
Notable supporters of Leonard Peltier over the years have included South African President Nelson Mandela, Pope Francis and Amnesty International. Supporters of Peltier say his trial was marked by gross
FBI
and federal prosecutorial misconduct, including the coercion of witnesses, fabricated testimony and suppressed exculpatory evidence.
After being released in February, Leonard Peltier returned home to live on the Turtle Mountain Band of Chippewa Reservation in Belcourt, North Dakota. On September 12th, Leonard Peltier celebrated his 81st birthday. People gathered throughout the day, visiting him and calling from around the world to celebrate. That night and the next day, we spoke to Leonard Peltier in his living room in his first extended TV/radio broadcast interview since his release from prison.
AMY
GOODMAN
:
Hi. I’m Amy Goodman, host of
Democracy Now!
, in the home of Leonard Peltier, just recently freed from prison after 49 years.
LEONARD
PELTIER
:
Plus two months.
AMY
GOODMAN
:
Plus two months.
LEONARD
PELTIER
:
Yeah.
AMY
GOODMAN
:
I’ve spoken to you so many times, Leonard, in prison, in various prisons, several of them supermax prisons. It is quite astonishing to be here with you in person. Tell us where we are. Where are we sitting?
LEONARD
PELTIER
:
We’re sitting in my home, that was given to me by my supporters. This was not given to me by the tribe, or the government had nothing to do with it. I was released by Biden under a commutation of my sentence and home confinement. Actually, what happened was, I was — I was taken out of one prison cell and really put into another type of prison. But this is my home now. This is my home. So it’s a million times better.
AMY
GOODMAN
:
Wait, what do you mean when you say you were taken out of your prison cell after more than 49 years, and you’re saying that you’re not completely free?
LEONARD
PELTIER
:
No, no, I’m on very restrictive restrictions. Even to go to the post office, I got to call my — I call her my handler. I have to call her to go to the post office. Then, when I get back, I have to call her and tell her I’m back. Or if I go anything, if I go shopping or whatever, I have to do that. If I have to go a hundred miles past the nation — I don’t call my place a “reservation,” either; we’re nations of people — I have to get a pass, usually from Washington, D.C., to go to medical, usually medical, or religious ceremonies on different nations, Indian Native nations.
AMY
GOODMAN
:
So, let’s go back to that moment when you were in that prison cell in Coleman, in Florida, and you got word that President Biden had commuted your sentence. It was just hours before he was leaving office? Can you tell us about that process, how it took place?
LEONARD
PELTIER
:
Well, as I went through the years filing for pardons and stuff, Ronald Reagan was the first one to promise to leave me — pardon me. Somebody in Washington stopped it. There’s only one organization that could have stopped it, and didn’t have the power to stop it, but still, somehow, were in power, enough to where they can override a president of the United States, our Congress. It’s the
FBI
. And Reagan promised to let me go. And the
FBI
intervened, and that was stopped. And Bill Clinton and Obama, and, finally, we get to Biden.
And Biden, there was pressure put on him from all over the world. Almost every tribal nation here in the United States filed for my release, demanding my release. The United Nations — the United Nations did a full report on my case, and they demanded that I be released immediately and to be “paid,” quote-unquote. Hundreds of Congress and senators and millions of people —
AMY
GOODMAN
:
And the pope.
LEONARD
PELTIER
:
And the pope, two popes, the last pope and the current pope. And world leaders, many world leaders, demanded my release.
AMY
GOODMAN
:
The Nobel Peace laureate, bishop — Archbishop Desmond Tutu?
LEONARD
PELTIER
:
Yes. I was also nominated for — and nominated four times, because of my work from prisons, for a Nobel Prize. And the board and everything granted it, but somebody intervened again. So, four times, I lost that.
I think somebody was pushing Biden to stop any — any possibility of signing a pardon. So, he didn’t sign it until the last moment. And actually, a day and a half before he actually signed it and he was — his term was completed, I just took the position that, “Nah, he’s not going to do this.” And I just kind of laid back in my cell, and I thought to myself, “Well, I guess I die here, and this is the only ultimate sacrifice I can make, and I have to accept it. I have no other choice.”
And as I laid there and thinking about it, other people came by — even guards would tell me, “Don’t give up, Leonard. Don’t give up.” And other prisoners, and some of them prisoners were telling me that, “Leonard, he’s got to know that if he doesn’t sign this, this is political suicide for the Democratic Party, because there’s millions of people that are going to break away from this if he doesn’t.”
And so, I was laying there, and I was thinking, “Well, let’s try one more thing.” So I called a representative of mine that was working closely with the Biden administration. We got — we have Native people — we had Native people in his administration who were communicating with Biden. And I said, “Tell him to give me a commutation on my sentence and home confinement.” So, she called and did this, and that’s what I ended up with. And that’s what I’m — that’s what I’m living under right now.
AMY
GOODMAN
:
How did you hear that you were going to be free?
LEONARD
PELTIER
:
Well, it was kind of unbelievable in the immediate moment. I thought somebody was just playing games with me. And I thought, “Ah, I’m going to wake up, and this is all a dream, and I’m in the cell, and I’ll be in there.” And I really didn’t believe it until, actually, I walked in the house here.
AMY
GOODMAN
:
What was it like to leave Coleman?
LEONARD
PELTIER
:
Wow! Imagine living in a cubicle larger than some people’s closets for all those years, and then you finally are able to walk out of there. I mean, it was just — for me, it was unbelievable that this was actually happening to me. But, I mean, the feeling of, wow, I can go — I’ll be home. I won’t be able to — I won’t have to go to bed in this cold cell with one blanket, and I won’t have to smell my celly going to the bathroom. I won’t have to eat cold meals. Is this really over for me? Is this really going to be over for me? And it was disbelief. A lot of it was disbelief, really.
AMY
GOODMAN
:
And now we’re sitting here in your living room surrounded by the paintings you did in prison.
LEONARD
PELTIER
:
Yes.
AMY
GOODMAN
:
You are an artist extraordinaire, maybe about to have a gallery showing in New York. Through the years, you sold your paintings. Talk about painting in prison and how you came to be a painter.
LEONARD
PELTIER
:
Well, see, a lot of people think we were allowed to paint in our cells and stuff. We were not. We were not allowed. They had an art room, hobby craft area, and one of the — one of the hobby crafts was painting. So, you have to sign up for that. A lot of people think that all the art supplies was given to you by the prison, the hobby crafter. That’s not true, either. We have to buy our own. And I went and signed up immediately to go into the art hobby craft. And I used to go there every day, and that’s what I did. I painted and painted and painted ’til I was able to create my own style and everything, yeah.
AMY
GOODMAN
:
Can you see your paintings now?
LEONARD
PELTIER
:
No. Two months ago, I think now, I lost 80% of my vision. And I’m in the process of, hopefully, getting my eyesight returned, treated and returned.
AMY
GOODMAN
:
We’re spending the hour with the Indigenous leader, longtime political prisoner, Leonard Peltier. He was released in February from prison after nearly half a century behind bars. Coming up, he talks about being put in an Indian boarding school as a child, his activism and more. We’ll also speak with his daughter, Marquetta Shields-Peltier. She was just a toddler when her father was imprisoned in 1976.
[break]
AMY
GOODMAN
:
This is
Democracy Now!
, democracynow.org,
The War and Peace Report
. I’m Amy Goodman.
We’re continuing with our conversation with longtime Indigenous activist Leonard Peltier in Belcourt, North Dakota. I spoke to him there on his 81st birthday weekend on the Turtle Mountain Band of Chippewa Cree Reservation. He was released in February from the federal prison in Florida after nearly half a century behind bars.
AMY
GOODMAN
:
So, take us back in time. Introduce us to you, starting by saying your name, who your parents were, the nations they were a part of, your family, where you lived growing up.
LEONARD
PELTIER
:
OK. My name is — English name is Leonard Peltier. I’m 81 years old as of yesterday. My father is a Chippewa Cree, from this reservation, this nation right here.
I keep saying “reservations,” because we was trained — we was taught from childhood that we are all reservations, we’re Indians. And we’re not Indians, and this is not a reservation. We made treaties with the United States government, and the Constitution says they shall only make treaties with sovereign nations. So we’re sovereign nations. We’re not — we’re not Indians, as they claim to be — they claim we are.
And my mother is from Fort Totten. But again, that’s not the real name. The real name is Spirit Lake, and that’s of the Lakota/Dakota people.
I was raised, majority of my life, here with my grandparents, which is usually the traditional way of my people. The grandparents will take the children and raise them. But when grandpa died, grandma had no way, no way to support us, so she went to the agency here to ask for help. And in retaliation, they took us and put us in a boarding school.
AMY
GOODMAN
:
What boarding school?
LEONARD
PELTIER
:
Wahpeton, North Dakota, 1953. I was there ’til — for three years, ’56. And it was extremely brutal conditions.
AMY
GOODMAN
:
How old were you?
LEONARD
PELTIER
:
I was 9 then, when I went. And —
AMY
GOODMAN
:
Talk about the point of these boarding schools. Was your hair cut? Did they stop you from speaking your language?
LEONARD
PELTIER
:
Of course. They did all — they did — that was the purpose of the schools, is to take the Indian out of the Indians, is what they literally — was the order. They took us to the boarding schools. The first thing they did was cut all our — buzz cut our hair, took it all off, and then put us and took us into the shower. We showered, and we come out of the shower, and we were poured all over our bodies
DDT
. As you know, that’s poisonous.
AMY
GOODMAN
:
They poured
DDT
over your body?
LEONARD
PELTIER
:
They poured
DDT
, with all the cans, on your head, the whole body. And then they gave us — issued clothes, bedding and assigned us to a bed. And that was the beginning of our treatment. It was an extremely, extremely strict school, and beatings were regular for any little violation of those rules.
I might have been a little hot-headed, I don’t know. But when I first got there, there was a group. They called themselves the Resisters. And I immediately joined them, and I became part of the Resisters. So, we would sneak behind the gymnasium, and we would talk our language. We would sing some song, even do some prayers, yeah. And if we got caught, we got the [bleep] beat out of us.
AMY
GOODMAN
:
You wrote in your book,
Prison Writings: My Life Is My Sun Dance
, that you consider these boarding schools your first imprisonment.
LEONARD
PELTIER
:
Yes, it was. Was. I found the rules more restrictive than when I went — ended up in prison.
AMY
GOODMAN
:
So, you go to this residential school with your cousin and sister for three years. Where do you come back to? And how did you maintain your language and your culture?
LEONARD
PELTIER
:
Well, I came back here to live with my father. And they were still living in log cabins, no electricity, no running water. We had to haul water. We had to haul wood. And we only had $55 to live on, which was my father’s World War II military benefits, and that’s what we had to live on.
And we were facing the — we were facing the time of termination. The United States government wrote a bill, passed by Congress, signed by the president, of termination in 1956. It was supposed to be completed by 1985. And the first one to be terminated was the Menominee Indians of Wisconsin. They had millions of making — millions of prime land, timber and lakes to make hunting lodges and other things out there. It was beautiful, which they do today. They still — they got all those things out there today. But they came and took all that land from them. Then they come here in 1958. Was it ’58? Yeah, ’58. I was 13 years old then. And they came and told us we have been terminated, and we have to accept that. We were supposed to be the second reservation to be terminated.
AMY
GOODMAN
:
The Turtle Mountain Band of Chippewa Cree.
LEONARD
PELTIER
:
Turtle Mountain Band of Chippewa Cree, yes. And my father and all of them and their generation, a lot of people left on relocation. They said, “It’s hopeless. We can’t fight these people. They’re too powerful. They got — they’re just too powerful. You know, maybe life will be better over there,” and stuff like this on this relocation. So they picked the city to go to. A lot of them went to Washington state and Oregon.
And it was a small group of us stayed here and said, “No, we’re not leaving.” So, my dad and his generation said, “Well, what do you mean, 'been terminated'? You can’t come here and tell us that we got to leave and you’re going to just terminate us as a race of people and tell us that we no longer exist. Go [bleep] yourself. Come on. Let’s fight it out.” And literally, they were — I was proud of them. I was 13 years old.
They stopped all provisions. One little girl died over here from malnutrition, and that’s what really got everybody angry.
AMY
GOODMAN
:
So they thought they would starve you out.
LEONARD
PELTIER
:
Yeah, they were making conditions so hard that we were accepting termination. We were leaving. A lot of people took to say, “Well, at least my kids won’t starve to death.”
AMY
GOODMAN
:
And the
BIA
reversed its decision?
LEONARD
PELTIER
:
They reversed their decision and gave us $50 —
AMY
GOODMAN
:
Vouchers?
LEONARD
PELTIER
:
Vouchers to go buy groceries here in Rolla. Everybody got — the whole reservation got $50.
AMY
GOODMAN
:
And they didn’t disband the reservation?
LEONARD
PELTIER
:
No, they got the hell out of here, because we told them we’re going to fight them.
AMY
GOODMAN
:
So, this was really the beginning of the Red Power Movement and the founding of
AIM
, right?
LEONARD
PELTIER
:
Well, I guess so, in so many — yeah, in so many — yeah.
AMY
GOODMAN
:
The American Indian Movement.
LEONARD
PELTIER
:
Yeah.
AMY
GOODMAN
:
Which you want to rename the American —
LEONARD
PELTIER
:
But they were — but they were doing that for — I mean, my people have been fighting back for 500 years, Amy.
AMY
GOODMAN
:
But the modern day.
LEONARD
PELTIER
:
Yeah, the modern-day stuff. But no, we went to war with them. We went to all kinds of different levels of —
AMY
GOODMAN
:
Resistance?
LEONARD
PELTIER
:
Resistance, you know.
AMY
GOODMAN
:
So, talk about the founding of
AIM
, the American Indian Movement, which you today would like to rename the American Indigenous Movement.
LEONARD
PELTIER
:
Well, I was involved in a number of different organizations before I joined
AIM
. And one of the biggest ones that I was — I helped organize the United Tribes of All Indians in Washington state. And we took over Fort Lawton. One of the treaties that we were pushing them days — actually, our people was — older people were pushing this, too, but they just passed. All of our knowledge came from traditionalists. That’s the policies, the American Indian Movement, we followed there. There.
First of all, people got to understand, the American Indian Movement policy is they can’t come onto this reservation, start dictating their policy. They have to be invited by the traditionalists or a tribal government or what else. We can’t just go onto a reservation and say, “You’re going to do this, you’re going to do that.” No, we can’t do that, and we don’t do that. We have to be invited first.
So, anyway, this was before I joined the American Indian Movement. I was involved with the fishing and hunting struggles over there. That was a big area that they really fought hard and got really —
AMY
GOODMAN
:
Fishing and hunting rights.
LEONARD
PELTIER
:
Yes, treaty rights. In fact, Marlon Brando got arrested with us in 1955. He got arrested on one of them lakes. I wasn’t there, but he was — he got arrested fishing and hunting with the Natives out there.
AMY
GOODMAN
:
So, talk about the occupation of the
BIA
offices in Washington, moving on to Wounded Knee and Pine Ridge.
LEONARD
PELTIER
:
Well, it just — our resistance became extremely popular. American Indian Movement was growing, and not just here in America — Canada, Central America. Said, “Wow! There are a lot of full bloods all through Central America,” more than people — more than here in the United States. And we were uniting with all of them, all those Natives across this country — across this whole continent, I mean. And we were uniting. We were pulling them together with the American Indian Movement. That’s why we became a threat to the government so.
And later, later on, when I was arrested — after I got arrested, this one guy was telling me, he said, “You know, I just went down to Mexico someplace, one of them towns.” And he said they were organizing for resistance and stuff like this. He said, “I was down there, down there visiting them.” He said, “I went to this old — this guy told me he was the — he was some kind of medicine man or something. So I went down and visited him, and so I went into his place,” into his — he had kind of a hut-like home, I guess. And he said, “What do I see?” He said, “I see your poster on one of the walls.” That’s so far back. But I wasn’t. We went through all that stuff. And so, anyway —
AMY
GOODMAN
:
But especially for young people to understand, I mean, you’re talking about this critical moment of 1973, ’4 and ’5.
LEONARD
PELTIER
:
’60s, actually.
AMY
GOODMAN
:
What’s that?
LEONARD
PELTIER
:
Started in the ’60s, really.
AMY
GOODMAN
:
And also the height of the antiwar movement. And the role and the effect of the antiwar movement on the Native American movement, and vice versa? If you can talk about those critical moments?
LEONARD
PELTIER
:
We were — I was, and others were, a lot of us Natives were — we were also involved in the peace marches and with the Blacks and the antiwar movements and things like that. We were involved in all that stuff, too. But we were working on trying to get their support, and they were working on trying to get our support. Then the hippies came out, and the hippies really helped us. The hippies did a lot to help us. They started dressing like Natives. They started doing things like Native people. And a lot of them came from very, very wealthy families. A lot of people hated them. That’s one of the reasons the government hated them, is because they were really pushing the Native issues, the culture and stuff like that.
AMY
GOODMAN
:
So, the Trail of Broken Treaties, that was 1972. Explain what it was, which was a takeoff on the Trail of —
LEONARD
PELTIER
:
Well, we knew that we had to get to — get the government to start honoring our treaties, because they never honored our treaties. And the only way we could do this is to go right straight to Washington. And so, we organized a group. We called it the Trail of Broken Treaties. And we all organized from all over the country. They sent representatives in old cars. We had all — nobody had new cars them days. And we all went to Washington.
AMY
GOODMAN
:
You went?
LEONARD
PELTIER
:
Of course, I did. Of course, I was there, too, yeah.
AMY
GOODMAN
:
This is, of course, a takeoff on the Trail of Tears. And most people, in our schools, and maybe less so especially now, will ever even know what the Trail of Tears was.
LEONARD
PELTIER
:
Right, right, right, precisely. That was all past — everything we did, we called it — well, like the Trail of Broken Treaties, that was done out of the Trail of Tears, and the Long Walk, all them other events like that that happened. It wasn’t just the Trail of Tears. That’s what people have to understand. It was — the Trail of Tears was just one of them that became so well known, because I think 10,000 people died on that, and just laying alongside the trails and stuff from dying from sickness, malnutrition, all that stuff, on the Trail of Tears. And that’s why it got so —
AMY
GOODMAN
:
This was President Andrew Jackson?
LEONARD
PELTIER
:
Yes, yeah.
AMY
GOODMAN
:
The president who President Trump reveres.
LEONARD
PELTIER
:
It was [bleep] order, yeah. He was a anti — he was a hater. And so, we were prevented. We organized. We organized under basically the same policies of exposing what was done in the past, what continued to be done.
And we still find it’s still happening today, Amy. Ann Coulter had made a public statement that — about Native people, that we didn’t kill enough of them Indians. That’s a very dangerous thing to say about anybody, because there’s a bunch of nuts out there, like, you know, you take one of them haters and everything, he can end up killing a lot of innocent Natives for — just because those type of words.
You got a president trying to do away with our treaties. If our treaties go, we go. This is the only thing to prove that we are a sovereign nation and a race of people. And if that goes, we go, as a race of people. So, it’s not — I mean, it’s not ending for us. We’re still in danger. Yeah, you see it happening in the streets, you know, I mean, right today. Look at what they’re doing in Palestine, killing women, children, babies, unborn babies. That’s what they did to us, man. And here it is still happening.
AMY
GOODMAN
:
So, 52 years ago, 1973, the start of the American Indian Movement’s 71-day occupation of the village of Wounded Knee on Pine Ridge Reservation, occupation helping draw international attention to the plight of Native Americans, the U.S. government responding to the occupation with a full military siege that included armored personnel carriers, F-4 Phantom jets, U.S. Marshals,
FBI
, state and local enforcement. During the occupation, two Sioux men shot dead by federal agents, and a Black civil rights activist, Ray Robinson, went missing. The
FBI
confirmed in 2014, decades later, that Ray Robinson had been killed during the standoff. Most people don’t know about this history.
LEONARD
PELTIER
:
No.
AMY
GOODMAN
:
Maybe they’ve heard the book
Bury My Heart at Wounded Knee
. So, can you talk about Pine Ridge and Wounded Knee, especially for young people who don’t know the history?
LEONARD
PELTIER
:
Well, I was in jail during the beginning of Wounded Knee, but I got out. It was about halfway through. Then I went, and I went up there, and I helped pack in stuff into Wounded Knee. And I stayed out there on the outside forces.
After we made all those trips to Washington and all that stuff, all those other demonstrations, and all those promises, they were going to do this and do that. They were going to investigate everything, all of our acquisitions and all that stuff. And we soon found out — we knew anyway, but we soon found out that was all a lie. They weren’t going to investigate [bleep]. And they didn’t.
And so, finally, the elders and the chiefs made the decision to go to Wounded Knee.
AIM
had no part in that decision. We cannot go anyplace in Indian Country and make policies. We can’t. That’s not — that is not us. We can’t do that. And we can’t go unless we’re invited by those people. And they just got fed up with so many more false promises and what was happening.
They were being terrorized by a group organized by — a mercenary group, I might add. They were provided with intelligence, armored person ammunition, sophisticated weapons, surveillance and stuff like this, automobiles and stuff. And the leader made that — admitted that on a national interview. So we know that’s all true. They tried to deny it at first, but they called themselves the Guardians of the Oglala Nation. And —
AMY
GOODMAN
:
The GOONs.
LEONARD
PELTIER
:
The GOONs, yeah.
AMY
GOODMAN
:
Dick Wilson’s.
LEONARD
PELTIER
:
Dick Wilson, all them. Nixon ordered the 82nd Airborne to go and investigate what’s going on down there. And if there was — if we were like the government was claiming, that we were communists and Marxists, and we were being financed by the communists, they were to go up there and wipe us out.
When Nixon got the 82nd Airborne involved in it, we filed a lawsuit. And it took us 10 years, but we got all this information out of the files that they had to turn over to us, right? And we found that they had went to the armory and checked out 250,000 rounds of various caliber ammunition, different sophisticated weaponry, armored personnel carriers, and finances and surveillance and stuff like that. See, that was all illegal. And that’s how we found out a lot of stuff about what they were doing there. And it was all illegal. If it would have been left to Nixon, he would have — he was going to wipe us out. But he didn’t, because, we heard, his wife stepped forward and said, “No, don’t you do that.”
AMY
GOODMAN
:
Now, still going back 50 years, what landed you in jail? I want to go to the words in
Democracy Now!
in 2003. The occupation of Wounded Knee is considered the beginning of what Oglala people refer to as the Reign of Terror, from ’73 to ’76, over 60 residents killed in this period. Murders went uninvestigated by the
FBI
, which had jurisdiction. The period culminating in the June 26th shootout for which Leonard Peltier was imprisoned.
LEONARD
PELTIER
:
First of all, I don’t know who the shooter is, or shooters. And I don’t — I wouldn’t tell you if I did know, so I’m not going to tell you anything in that area. But I’ll tell you — I’ll speak on the other issues, because it’s public knowledge, and it’s been — it’s been our attempts to continue to expose that stuff.
But there was a lot of Native people, traditionalists, whose homes were burned, whose houses were — was drive-by shootings. People were pulled over and beaten, and some shot, some killed. And those things are literally recordings on this. We got records of all this stuff now, so people can’t deny this stuff. The only ones that are denying this [bleep] is the United States government and the
FBI
and people like that.
But we faced a time called the Reign of Terror, when they were getting away with all this [bleep]. None of them were getting investigated. A lot of the older people that seen these people identified them, but the
FBI
still wouldn’t investigate. They were able to kill people at random. They were — and getting away with it, because they didn’t have — they had no fear of being prosecuted. The only fear they had was of us, the American Indian Movement, because we wouldn’t take their [bleep]. Every chance we got together, we got a confrontation with them. And that’s the only fear they had of anything, of any retaliations, any arrests or anything else.
AMY
GOODMAN
:
We’re spending the hour with Indigenous elder Leonard Peltier. He was released in February from prison after nearly 50 years behind bars. Stay with us.
[break]
AMY
GOODMAN
:
This is
Democracy Now!
, democracynow.org. I’m Amy Goodman, as we continue our conversation with Indigenous leader Leonard Peltier, released to home confinement in February after nearly half a century behind bars. I asked him about his claims that his extradition from Canada and trial in the United States were marked by prosecutorial misconduct.
AMY
GOODMAN
:
Talk about the coerced testimony of Myrtle Poor Bear, who she is.
LEONARD
PELTIER
:
Who is she?
AMY
GOODMAN
:
I mean, I know you didn’t know her at the time.
LEONARD
PELTIER
:
I never knew her. Her father came to my — was going to come testify at my trial that she had a serious mental problem. And her sister was going to testify that on the day of the shootout, they were sitting there drinking beer. And we got this all on tape. And they were sitting there drinking, drinking beer, and they ran out of beer. And they were watching TV, she said, and they decided to make a run off the reservation, because it was a dry reservation. No alcohol was allowed there. And so, she says, to go buy some more beer, come back and watch some more TV. And they started driving down the road, and all of a sudden a bulletin came over the radio: a big shootout in Oglala between the Marshals,
FBI
,
BIA
cops,
GOON
squads against the American Indian Movement. So they were over 50 miles away. Finally, Myrtle admitted that she didn’t know me.
AMY
GOODMAN
:
But she — her testimony said that —
LEONARD
PELTIER
:
Her testimony in the grand jury is what got us all indicted.
AMY
GOODMAN
:
Said she was your girlfriend and she had seen —
LEONARD
PELTIER
:
She witnessed — oh god.
AMY
GOODMAN
:
Seen all of this.
LEONARD
PELTIER
:
When the lawyers came to me in Canada, they said, “Leonard” — they said, “Leonard, we got bad news for you.” And I said, “Yeah? What kind of bad news?” And they said, “Your girlfriend’s testifying against you.” And I looked at him, and I said, “My girlfriend? What do you mean? My girlfriend?” Said, “Your girlfriend.” And I said, “I don’t have a girlfriend. I got a wife, two kids.”
AMY
GOODMAN
:
So, talk about James Reynolds, the former U.S. attorney in charge of the prosecution, that helped convict you. He later becomes an advocate for your release, stating the prosecution could not prove that you had committed any offense and the conviction was unjust. He wrote to president after president — he himself was appointed by Carter — right through to Biden.
LEONARD
PELTIER
:
Yes. Well, about 10 years ago, James Reynolds started to have a change of heart, I guess. James Reynolds said that there is no evidence Leonard Peltier committed any crimes on that reservation. And that’s pretty — he was in charge of everything.
AMY
GOODMAN
:
What this ultimately leads to is your imprisonment for 49 years.
LEONARD
PELTIER
:
Yeah.
AMY
GOODMAN
:
The majority of your life behind bars, what this institutionalization meant for you, what it meant to be both a prisoner and considered a political prisoner around the world and a symbol of the fight for Native American rights and what happens to you when you engage in it, all of those things?
LEONARD
PELTIER
:
Well, OK, I think — I’ve been asked this question quite a bit, and it’s hard for me to answer. But I think that what really kept me strong was my anger. I was extremely angry about what they did to me and my people. And I’m still — still very, very [bleep] angry. And there was no way in hell I was going to get justice. I could have had — I had at least 14 constitutional issues that I should have been released on, at least that many, and I knew I wasn’t going to get it. I knew what it was — the courts were not going to give it to me. And, I mean, even the Supreme Court would refuse to hear my case and stuff like that. But I knew why. You know, I found evidence of them meeting with Judge Heaney. And Judge Heaney became a strong advocate for my release. But we found evidence he worked with the
FBI
.
And I just — I felt so much hate and anger, and what they — what they did to Native people in this country, this continent, and that kept me strong. It kept me from — oh, kept me from — I’ve been offered numerous times, or a few times anyway, that if I accepted responsibility and made statements, that everything we’ve said negative about the United States government, what their past history was, and their dealings with our — with us, as people in the nation, they would turn me loose. And I refused to do that. I refused to bow down to them. And I still refuse to bow down to them. I’m going to die with my beliefs. And I just refuse to — to me, it’s treason against my nation, my people.
AMY
GOODMAN
:
You’re a major symbol of Indigenous power, not only in the United States, but around the world. What does that mean to you?
LEONARD
PELTIER
:
Well, I hope I can use it to benefit my people. I mean, as I said earlier, we’re still in danger. It’s not over for us. We don’t get to live like the rest of the people in this country, without fear of what would happen to us if we had our treaties taken away from us. We don’t get to live like that. We still have to live under that, that fear of losing our identity, losing our culture, our religion and stuff. Most Americans don’t have to worry about that. We do. And so, the fight for, the struggle still goes on for me. I’m not going to give up. I’m not going to — I have not surrendered. I don’t want to go back to prison, although I heard that Trump was going to try to take away all of Biden’s pardons and everything else like that.
AMY
GOODMAN
:
What would you say to young Indigenous people? I’m looking behind you at a photograph of — is it a picture of your great-granddaughter?
LEONARD
PELTIER
:
Yeah, this one, right here.
AMY
GOODMAN
:
And she’s wearing a T-shirt that says “strong.” How old is she?
LEONARD
PELTIER
:
She’s now 11. Now 11. We adopted her when she was a little baby, been taking care of her ever since. And she loves me and thinks I’m the greatest thing in the world. I love her because she is the greatest thing in the world. And she was — she’s now a champion fly swimmer. She was going to — her plan was, if she wins the Olympics, she was going to take those Olympics and say, “This is for my grandpa, Leonard Peltier, who they illegally put in prison. This is for him.” I said, “Where did you come up with that?” She won’t say, but she just looks at me, yeah.
AMY
GOODMAN
:
You know, we’ve been covering the climate movement for so many years, were here in North Dakota covering the standoff at Standing Rock, the Sioux-led, Indigenous-led global movement to preserve the environment. And this year, the U.N. climate summit is in just the tip of the rainforest, Belém in Brazil. And each of these U.N. climate summits, we see Indigenous people, especially young Indigenous people, there fighting for the planet. Do you see the voice of Indigenous people on the climate movement as hopeful?
LEONARD
PELTIER
:
We’ve been talking about this for 250 years — or, no, since America was first organized. We still — when we pray and when we — whatever we do, we still talk about Mother Earth and other environment stuff. We haven’t stopped. We never will stop. You know, we are still strong on environmentalists.
AMY
GOODMAN
:
Well, Leonard Peltier, I want to say thank you so much for inviting us into your home. I’m so glad we’re not coming to a prison.
LEONARD
PELTIER
:
Well, so am I.
AMY
GOODMAN
:
Indigenous leader Leonard Peltier, speaking in September at his home on the Turtle Mountain Reservation in North Dakota on his 81st birthday weekend.
The original content of this program is licensed under a
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
. Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.
Show HN: MkSlides – Markdown to slides with a similar workflow to MkDocs
Use
mkslides
to easily turn Markdown files into beautiful slides using the power of
Reveal.js
!
MkSlides
is a static site generator that's geared towards building slideshows. Slideshow source files are written in Markdown, and configured with a single YAML configuration file. The workflow and commands are heavily inspired by
MkDocs
and
reveal-md
.
Features
Build static HTML slideshow files from Markdown files.
Turn a single Markdown file into a HTML slideshow.
Turn a folder with Markdown files into a collection of HTML slideshows with an index landing page.
Publish your slideshow(s) anywhere that static files can be served.
Locally on your own device.
On a web server.
Deploy through CI/CD with GitHub/GitLab (like this repo!).
E.g. when your Markdown files are located in the
slides/
folder:
If the
slides
folder doesn't exists, it will fallback to
docs
for backwards compatibility. If
docs
also doesn't exists, it will error.
E.g. when your Markdown files are located in the
somefolder/
folder:
mkslides build somefolder/
E.g. when you have a single Markdown file called
test.md
:
⚠️
When you use a single file as
PATH
, only default static assets will be copied to the output folder. If you want to include images or other files, create a folder instead and pass that as
PATH
. Using a file as
PATH
is more meant for a quick slideshow in a pinch using only text.
Just create a
mkslides.yml
. All options are optional, you only have to add what you want to change to
mkslides.yml
.
Relative file paths are considered relative to the directory containing Markdown files (
PATH
).
Here's an example showcasing all possible options in the config file:
# Configuration for the generated index pageindex:
# Enables or disables the "Documentation built with MkSlides." footer:# booleanenable_footer: true# Favicon of the generated index page: file path or public url to favicon# filefavicon: example-index-favicon.ico# Navigation section describing how to structure the slides on the index# page. This is similar to the `nav` option from MkDocs: list[any]nav:
- Example: example1.md
- "Example 2": somewhere/example1.md
- example3.md
- somewhere/example4.md
- "More examples":
- example5.md
- "Much more examples":
- "Last example": somewhere/much/more/examples/example6.md# Title of the generated index page: stringtitle: example-title# Jinja 2 template to generate index HTML: file path to Jinja2 filetemplate: example.jinja# Theme of the generated index page: file path or public url to CSS filetheme: example-index-theme.css# Configuration for the slidesslides:
# Charset of the slides: string# (see https://revealjs.com/markdown/#external-markdown)charset: utf-8# Favicon of the slides: file path or public url to favicon filefavicon: example-slides-favicon.ico# Theme for syntax highlighting of code fragments on the slides: file path# to CSS file, public url to CSS file, or one of the highlight.js built-in# themes such as `monokai`, `obsidian`, `tokyo-night-dark`, `vs`, ...# (see https://highlightjs.org/examples)highlight_theme: example-slides-highlight-theme.css# Relative path to a python script containing a function# Callable[[str], str] named `preprocess`. Important: a relative file path# here is considered relative to the configuration file, as you probably# don't want to serve the python scripts.# For each Markdown file, the whole file content is given to the function as# a str. The returned string is then further processed as the Markdown to# give to Reveal.jspreprocess_script: tests/test_preprocessors/replace_ats.py# Separator to determine notes of the slide: regexp# (see https://revealjs.com/markdown/#external-markdown)separator_notes: "^Notes?:"# Separator to determine end current/begin new vertical slide: regexp# (see https://revealjs.com/markdown/#external-markdown)separator_vertical: ^\s*-v-\s*$# Separator to determine end current/begin new slide: regexp# (see https://revealjs.com/markdown/#external-markdown)separator: ^\s*---\s*$# Jinja 2 template to generate index HTML: file path to Jinja2 filetemplate: ./example.jinja# Theme of the slides: file path to CSS file, public url to CSS file, or one# of the reveal.js themes such as `black`, `white`, `league`, `solarized`,# `dracula`, ... (see https://revealjs.com/themes/)theme: example-slides-theme.css# Title of the slides. If this is set for a slide, it will be used for the# entry in the generated index HTML: stringtitle: example-title# Options to be passed to reveal.js: options in yaml format, they will be# translated to JSON automatically (see https://revealjs.com/config/)revealjs:
height: 1080width: 1920transition: fadeexample_plugin:
example_plugin_option_A: trueexample_plugin_option_B: qwerty# Plugins or additional CSS/JavaScript files for the slides. These are given as# a list.plugins:
# Name of the plugin (optional, see plugin README): plugin id string# (see https://revealjs.com/creating-plugins/#registering-a-plugin)
- name: RevealExamplePlugin# List of CSS files of the plugin (optional, see plugin README):# public url to CSS file per entryextra_css:
- https://cdn.jsdelivr.net/npm/reveal.js-example-pluging/example.min.css# List of JavaScript files of the plugin (optional, see plugin README):# public url to JavaScript file per entryextra_javascript:
- https://cdn.jsdelivr.net/npm/reveal.js-example-pluging/example.min.js
- name: RevealMermaidextra_javascript:
- https://cdn.jsdelivr.net/npm/reveal.js-mermaid-plugin/plugin/mermaid/mermaid.min.js
- extra_javascript:
- https://cdn.jsdelivr.net/npm/reveal-plantuml/dist/reveal-plantuml.min.js
Default config (also used if no config file is present):
index:
enable_footer: truetemplate: assets/templates/index.html.jinja # Comes with the pip packagetitle: Indexslides:
highlight_theme: monokaitemplate: assets/templates/slideshow.html.jinja # Comes with the pip packagetheme: blackrevealjs:
history: trueslideNumber: c/t
It is also possible to override
slides
,
revealjs
, and
plugins
options on a per Markdown file base using it's frontmatter. Here, relative file paths are considered relative to the Markdown file itself.
---slides:
theme: solarizedhighlight_theme: vsseparator: <!--s-->title: Frontmatter title.revealjs:
height: 1080width: 1920transition: zoom---# Slides with frontmatter<!--s-->## Lorem ipsum
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
<!--s-->
Notes:
title
here is a frontmatter-only available option to set the title of this slideshow in the generated index page. This option is not available in
mkslides.yml
.
The precedence is frontmatter >
mkslides.yml
> defaults.
Full help
Usage: mkslides [OPTIONS] COMMAND [ARGS]...
MkSlides - Slides with Markdown using the power of Reveal.js.
Options:
-V, --version Show the version and exit.
-v, --verbose Enable verbose output
-h, --help Show this message and exit.
Commands:
build Build the MkSlides documentation.
serve Run the builtin development server.
Usage: mkslides build [OPTIONS] [PATH]
Build the MkSlides documentation.
PATH is the path to the directory containing Markdown files. This argument
is optional and will default to 'slides', or 'docs' if the first directory
doesn't exist. If PATH is a single Markdown file or a directory containing a
single Markdown file, it will always be processed into `index.html`
regardless the name of the Markdown file.
Options:
-f, --config-file FILENAME Provide a specific MkSlides-Reveal config file.
-d, --site-dir PATH The directory to output the result of the slides
build. All files are removed from the site dir
before building.
-s, --strict Fail if a relative link cannot be resolved,
otherwise just print a warning.
-h, --help Show this message and exit.
Usage: mkslides serve [OPTIONS] [PATH]
Run the builtin development server.
PATH is the path to the directory containing Markdown files. This argument
is optional and will default to 'slides', or 'docs' if the first directory
doesn't exist. If PATH is a single Markdown file or a directory containing a
single Markdown file, it will always be processed into `index.html`
regardless the name of the Markdown file.
Options:
-f, --config-file FILENAME Provide a specific MkSlides-Reveal config file.
-s, --strict Fail if a relative link cannot be resolved,
otherwise just print a warning.
-a, --dev-addr <IP:PORT> IP address and port to serve slides locally.
-o, --open Open the website in a Web browser after the
initial build finishes.
-h, --help Show this message and exit.
When GitHub Copilot was launched in 2021, the fact that its training data included a vast amount of Open Source code publicly available on GitHub attracted significant attention, sparking lively debates regarding licensing. While there were issues concerning conditions such as attribution required by most licenses, there was a particularly high volume of discourse suggesting that the conditions of copyleft licenses, such as the GNU General Public License (GNU GPL), would propagate to the model itself, necessitating that the entire model be released under the same license. The propagation of the GPL is a concept that many modern software engineers have naturally accepted; thus, for an engineer with a straightforward sensibility, it is a perfectly natural progression to think that if GPL code is included in some form, copyleft applies and the license propagates.
However, as of 2025, the theory that the license of the source code propagates to AI models trained on Open Source code is not seen as frequently as it was back then. Although some ardent believers in software freedom still advocate for such theories, it appears they are being overwhelmed by the benefits of AI coding, which has overwhelmingly permeated the programming field. Amidst this trend, even I sometimes succumb to the illusion that such a theory never existed in the first place.
Has the theory that the license of training code propagates to such AI models been completely refuted?
Actually, it has not. This issue remains an indeterminate problem where lawsuits are still ongoing and the judgments of major national governments have not been made clear. In this article, I will explain the current situation of this license propagation theory, namely “GPL propagates to AI models trained on GPL code,” and connect it to points of discussion such as the legal positioning of models and the nature of the freedom we pursue in the AI domain.
Note:
This article is an English translation of a post
originally written in Japanese.
While it assumes a Japanese reader, I believe it may also be useful for an English-speaking audience.
The Current Standing in Two Lawsuits
First, let us organize what the “GPL propagation theory to AI models” entails. This is the idea that when an AI model ingests GPL code as training data, the model itself constitutes a derivative work (derivative) of the GPL code; therefore, when distributing the model, the copyleft conditions of the GPL, such as the obligation to disclose source code, apply. In other words, it is not a question of whether the output of the model is similar to the GPL code, but a theory that “since the model itself is a derivative containing GPL code, the GPL extends to the model.” While there were many voices supporting this theory around 2021, as mentioned earlier, it is no longer the mainstream of the discussion today. However, two major ongoing lawsuits can be cited as grounds that this theory has not been completely denied. These are
Doe v. GitHub
(the Copilot class action) filed in the United States and
GEMA v. OpenAI
filed in Germany. I will explain the history and current status of each lawsuit below.
Doe v. GitHub (Copilot Class Action): The Persisting Claim of Open Source License Violation
In the Copilot class action filed at the end of 2022 in relation to GitHub Copilot, anonymous developers became plaintiffs and argued that GitHub, Microsoft, and OpenAI trained their models on source code from public repositories without permission, inviting massive license violations through Copilot. Specifically, they viewed it as problematic that when Copilot reproduces part of the code that served as the training source in its output, it does not perform the author attribution or copyright notice required by licenses such as MIT or Apache-2.0 at all, and furthermore, it indiscriminately trains on and outputs code under licenses that impose copyleft conditions like the GPL, thereby trampling on license clauses. The plaintiffs claimed this was a contractual violation of open source licenses and also sought damages and injunctions, asserting that it constituted a violation of the Digital Millennium Copyright Act (DMCA) under copyright law.
In this case, several decisions have already been handed down by the United States District Court for the Northern District of California, and many of the plaintiffs’ claims have been dismissed. What were dismissed were mainly peripheral claims such as DMCA clause violations, privacy policy violations, unjust enrichment, and torts, but some DMCA violations and the claim of “violation of open source licenses” (breach of contract) are still alive. Regarding the latter specifically, the argument is that despite the plaintiffs’ code being published under licenses like GPL or MIT, the defendants failed to comply with the author attribution or the obligation to publish derivatives under the same license, which constitutes a contractual violation. Although the court did not recognize claims for monetary damages because the plaintiffs could not demonstrate a specific amount of damage, it determined that there were sufficient grounds for the claim for injunctive relief against the license violation itself. As a result, the plaintiffs are permitted to continue the lawsuit seeking an order prohibiting the act of Copilot reproducing others’ code without appropriate license indications.
As is clear from the above history, “violation of open source licenses in training data” is still being contested in court in the Copilot litigation, and this is one of the reasons why the theory of license propagation to models has not been completely denied. The plaintiffs’ claim in this lawsuit does not directly demand the release of the model itself under the GPL, but it legally pursues the point that license conditions were ignored in the process of training and output; consequently, it suggests that “if the handling does not follow the license of the training data, the act of providing the model could be illegal.” Furthermore, the court has not clearly rejected this logic at this stage and has indicated a judgment that the use of open source code is accompanied by license obligations, and providing tools that ignore this could constitute a tort subject to injunction.
However, it is necessary to note that the claims in the Copilot litigation are legally framed as breach of contract (license) or DMCA violation, and are not a direct copyright argument that “the model is a derivative work of GPL code.” No judgment has been shown stepping so far as to mandate the disclosure of the entire model under the GPL license. The actual judgment is conservative, stating “monetary damages have not been shown, but there is room for future injunctive relief,” and does not mention the obligation to disclose the model itself. In other words, at present, there is no judicial precedent directly addressing the “GPL propagation theory to models,” and the situation is one where the issue raised regarding license violation of the source code remains alive in the judicial arena.
GEMA v. OpenAI: The Theory Treating “Memory” in Models as Legal Reproduction
Another important lawsuit is the case where the German music copyright collective GEMA sued OpenAI. This is a copyright lawsuit concerning the unauthorized training and output of lyrics by an AI model, not AI code generation, but it carries significant theoretical implications related to “license propagation to models” even if not directly related to GPL.
In November 2025, the Munich I Regional Court handed down a judgment on this lawsuit, indicating regarding the matter where the ChatGPT model had memorized and reproduced the lyrics of 9 famous German songs, that the act of “memory” inside the model itself falls under the act of reproduction under copyright law. According to the judgment, the lyrics under the plaintiff’s management were “fixed” in the models of ChatGPT’s GPT-4 and 4o, and the situation was such that the lyrics were output almost verbatim just by the user giving a simple prompt. Based on this, the court determined that the model contains “parameters that memorized the work” internally, and if it is possible to reproduce an expression substantially identical to the original work for a human by means of an appropriate prompt, that memory itself falls under “reproduction” in Article 16 of the German Copyright Act. Furthermore, it determined that the act of actually outputting lyrics in response to a prompt is also a separate act of reproduction, and providing lyrics to the user falls under the act of making available to the public (public transmission). Also, it ruled that since all of these are done without the permission of the rights holder, they deviate from the scope justified by the TDM (Text and Data Mining) exception in the EU DSM Copyright Directive.
The important point of this judgment is that it clearly acknowledged that “if a work is recorded inside the model in a reproducible form, that state itself can constitute copyright infringement.” The court cited the text of the EU InfoSoc Directive that “reproduction includes copies in any form or manner, and does not need to be directly perceptible to humans,” and stated that in the spirit of this, even if the lyrics are encoded within the model’s parameters, it amounts to the creation of a reproduction. It went as far as to mention that “encoding in the form of probabilistic weights does not prevent it from being considered a copy,” showing a strong recognition that differences in technical formats cannot avoid the nature of reproduction under copyright law. Also, since the fact that the model could output the lyrics was not coincidental but highly consistent, it was factually found that “the direct incorporation of the essential part of the training data” occurred rather than the result of statistical learning. As a result, the Munich District Court recognized OpenAI’s liability for injunction and damages regarding the output act of the lyrics in question, and further ordered the provision of information regarding training data and output content for the future. However, this judgment is the first instance, and since OpenAI has indicated an intention to appeal, it is expected to be a continuing dispute.
The noteworthy theory shown by this GEMA judgment is the extension of the concept of reproduction under copyright law to the interior of the model. That is, if the work used as training data remains within the model and can be reproduced with a simple operation, it means the model already contains a reproduction of that work. This theory is groundbreaking in that it deems “the model contains the source work,” and indeed, in a commentary by Osborne Clarke, it is evaluated that “in contrast to the judgment of the English High Court in the
Getty v. Stability AI
case, the Munich District Court explicitly recognized the possibility that the AI model contains copies of the training material.” Standing on this view, the model is not merely a result of analysis, but depending on the case, can be evaluated as an aggregate of the training data itself.
However, it is necessary to keep in mind that this judgment is based on an extreme case where a complete match output was obtained with short text such as lyrics. The court itself stated, “Normally, temporary reproduction for learning remains within the purpose of analysis and does not infringe on the rights holder’s market, but in this case, the model holds the work in a restorable form and exceeds the scope of analysis,” emphasizing that the judgment is limited to “cases where the model performs complete reproduction.” Also, as the UK case shows, judicial decisions vary by country, and a legal consensus on this issue has not yet been formed.
Nevertheless, the judgment this time, which declared that the recording of a work inside a model is a reproduction, can become a major basis supporting the license propagation theory. This is because, while the premise for discussing GPL propagation is “whether the model can be said to be a reproduction or derivative work of the GPL code,” the logic of the Munich District Court legally certified exactly that “a model can be a reproduction of training data”.
Possibilities Derived from the Current Status of the Two Lawsuits
From the two lawsuits above, we can consider the path through which the theory of license propagation to AI models might be recognized in the future.
Let us assume the worst-case scenario from the perspective of AI operators, where these lawsuits are finalized with the plaintiffs winning. In the Copilot litigation, the judgment that “model providers must comply with the license conditions of the training source code” would be established, and in the GEMA litigation, the legal principle that “the model encompasses reproductions of the work” would be established. When these two intersect, the conclusion that “since an AI model containing GPL code is a reproduction or derivative work of the GPL code, the conditions of the GPL directly apply to its provision” is theoretically derived. That is, the possibility emerges that the theory of GPL propagation to models is effectively ratified by the judiciary.
Specifically, if the model memorizes and contains GPL code fragments internally, the act of distributing or providing that model to a third party may be regarded as the distribution of a reproduction of GPL code; in that case, the act of distribution under conditions other than GPL would be evaluated as a GPL license violation. If a GPL violation is established, there would be room to argue for remedies such as injunctions and claims for damages, as well as forced GPL compliance demanding the disclosure of the entire model under the same license, just as in the case of ordinary software. In fact, the remedies GEMA sought from OpenAI included disclosure regarding training data and output content, and although this is in the context of musical works, this can be said to be a type of disclosure request to make transparent “what the model learned and contains.” In the case of a GPL violation as well, the possibility cannot be denied that demands such as “disclosure of the GPL code parts contained inside the model” or “source disclosure in a form that allows reconstruction of the model” would emerge in seeking license compliance.
Even if not reaching such an extreme conclusion, an intermediate scenario could involve imposing certain restrictions on model providers. For example, the Copilot litigation might be settled or judged by taking measures such as “attaching a license and author attribution at the time of output if existing code of a certain length or more is included in the generated code,” or technically mandating the implementation of filters so that GPL code fragments are not extracted or reproduced from the model. In fact, GitHub, the developer of Copilot, has already introduced an optional feature that “excludes from suggestions if the candidate code matches existing code on large-scale repositories,” attempting to reduce litigation risk. Also regarding OpenAI, there are reports that it strengthened filters so that ChatGPT does not output copyrighted lyrics as they are, in response to the GEMA judgment.
While these are not license propagation itself legally, in practice, they indicate that the industry is steering in the direction of “ensuring the model does not potentially infringe license conditions.” In the future, there is a possibility that guidelines for excluding data with specific license terms like GPL at the model training stage, or mechanisms and systems to guarantee that there is no license-infringing output by conducting output inspections after training, will be established.
In any case, until these two lawsuits are completely settled and the subsequent legislative response is determined, the “theory of GPL propagation to models” has not completely disappeared. It is a scenario that could suddenly become realistic depending on future judgments, and even if the plaintiffs lose in the lawsuits, there is a possibility that support for this theory will reignite within the open source community. It is necessary to note that while it is currently an “undetermined theory not shouted as loudly as before,” that does not mean it has been legally completely denied and resolved. As our community, we need to carefully consider countermeasures while observing these trends and taking into account the legal systems of each country and opposing arguments described in the latter half of this article.
Treatment under Japanese Law
Based on the trends of the overseas lawsuits mentioned above, I will also organize the relationship between AI models, copyrighted works, and licenses under Japanese law. In Japan, Article 30-4 of the Copyright Act, introduced by the 2018 amendment, exists as a provision that comprehensively legalizes reproduction acts associated with machine learning. Furthermore, in March 2024, the Copyright Division of the Council for Cultural Affairs of the Agency for Cultural Affairs published a guideline-like document titled “Thought on AI and Copyright” (hereinafter “the Thought”), presenting a legal organization divided into the development/training stage and the generation/utilization stage of generative AI.
According to “the Thought,” reproduction performed basically for the purpose of AI training is legal as long as it satisfies “information analysis not for the purpose of enjoying the thoughts or sentiments expressed in the work” as defined in Article 30-4. Therefore, acts of collecting and reproducing a wide range of data from the internet to create a training dataset for research and development purposes can be done without the permission of the rights holders in principle. However, what is important is whether an “purpose of enjoyment” is mixed into that training act. “The Thought” states that if training is conducted with the purpose of “intentionally reproducing all or part of the creative expression of a specific work in the training data as the output of generative AI,” it is evaluated as having a concurrent purpose of enjoying the work rather than mere information analysis, and thus lacks the application of Article 30-4. As a typical example of this, “overfitting” is cited, and acts such as making a model memorize specific groups of works through additional training to cause it to output something similar to those works are judged to have a purpose of enjoyment.
Furthermore, “the Thought” also mentions the legal treatment of trained models, stating first that “trained models created by AI training cannot be said to be reproductions of the works used for training in many cases.” This is the view that since the model can generate outputs unrelated to the original in response to various inputs in a general-purpose manner, the model itself is not a copy of any specific work.
However, “the Thought” simultaneously acknowledges the possibility that, exceptionally, in cases where “the trained model is in a state of generating products with similarity to the work that was training data with high frequency,” the creative expression of the original work remains in the model, and it may be evaluated as a reproduction. It also points out that in such cases, the model is positioned as a machine for copyright infringement, and a claim for injunction may be recognized. In short, usually the model is merely statistical data and not the work itself, but if it has turned into a device for spewing out specific works almost as they are, it can be treated as an infringing item; this thinking shares parts with the content of the GEMA judgment.
It is necessary to note that the above organization is strictly a discussion of the scope of application of rights limitation provisions (exception provisions) under the Copyright Act, and does not touch upon the validity of contracts or license clauses. The Agency for Cultural Affairs document discusses from the perspective of “whether it is copyright infringement or not,” and does not deny that even if the training act is legal, contractual liability may arise if it violates terms of service or open source licenses separately. Also, no in-depth view has been shown regarding the propagation of copyleft clauses like the GPL. In Japan’s Copyright Act, there is no override provision where rights limitation provisions like Article 30-4 take precedence over contract conditions, and the “Contract Guidelines on Utilization of AI and Data” by the Ministry of Economy, Trade and Industry suggests the possibility that if there is a contract prohibiting data use between parties, that contract takes precedence.
Therefore, if the license is regarded as a valid contract, even if “training is legal” under Article 30-4 of the Copyright Act, the risk remains that it becomes a “violation of license conditions” under contract law, and it can be said that at least there is no official view organizing the theory of GPL propagation to models. In other words, currently, while the legality of model training acts is recognized quite broadly under the Copyright Act, license violation is left to general civil theory, and there is no clear guideline on, for example, “whether the act of publicly distributing a model trained on GPL code constitutes a GPL license violation.” Overall, the legal organization in Japan is in a situation of “safe in principle at the copyright layer, but blank at the contract layer.” Hence, the discussion in Japan regarding the theory of GPL propagation to models relies on future judicial judgments and legislative trends, and at present, there is no choice but to consider operational guidelines carefully following the organization by the Agency for Cultural Affairs.
Arguments Negating the Theory of License Propagation to Models
As seen in the previous sections, the theory of GPL propagation to models is not legally zero. However, many legal experts and engineers point out that this theory has serious detrimental effects. Here, I present representative arguments negating the theory of license propagation to models from the layers of copyright law, GPL text, technology, and practical policy.
Arguments for Negation at the Copyright Law Layer
First, under copyright law, it is unreasonable to regard an AI model as a “derivative work” or “reproduction” of the training source works. In many cases, the expressions of specific works are not stored inside the model in a form recognizable to humans. The model merely holds statistical abstractions where text and code have been converted into weight parameters, and that itself is not a creative expression to humans at all. A “derivative work” under copyright law refers to a creation that incorporates the essential features of the expression of the original work in a form that can be directly perceived, but one cannot directly perceive the creativity of the original code from the model’s weights. In other words, the model does not show the nature of a work directly enough to be evaluated as encompassing the original code. For example, the High Court of Justice in the UK stated in the judgment of the
Getty v. Stability AI
case that “the Stable Diffusion model itself is not an infringing copy of the training images,” showing a negative view on regarding the model itself as a reproduction of works. Thus, there are many cautious positions internationally regarding regarding the model itself as an accumulation of works or a compilation work.
Also, the output generated by the model involves probabilistic and statistical transformations, and in many cases, things that do not resemble the training source at all are output. Even if a match or similarity occurs by chance, it is difficult to prove whether it is a reproduction relying on the original or an accidental similarity. It is not realistic to conduct the certification of reliance and similarity required to discuss copyright infringement for the entire model. Ultimately, in the framework of copyright law, there is no choice but to judge “whether the model relies on a specific work” on a work-by-work basis, and recognizing uniform copyrightability or infringing nature for the model itself is a large leap. As organized in Japanese law where the model is not considered a reproduction in most cases, the schematic of model equals work is considered unreasonable under copyright law.
Arguments for Negation at the GPL Text Layer
Next, looking at the license text and intent of the GPL itself, doubts are cast on the interpretation that GPL propagates to AI models. For example, in the text of GPLv2, the target of copyleft is limited to “derivative works” of the original code provided under GPL and “works that contain the Program.” Typically, this has been interpreted as software created by modifying or incorporating GPL code, or software combined (linked) with GPL code. In the case of an AI model, it is extremely unclear which part of the original GPL code the model “contains.” Even if the model could memorize fragments of the GPL code used for training, it is a tiny fraction when viewed from the entire model, and most parts are occupied by parameters unrelated to the GPL code. There is no clear assumption shown by the GPL drafters as to whether a statistical model that may partially encapsulate information derived from GPL code can be said to be “a work containing the Program”.
Furthermore, GPLv3 requires the provision of software source code in a “preferred form for modification.” If an AI model is a GPL derivative, the problem arises as to what that preferred form for modification would be. The model weights themselves have low readability and editability for humans, and are hard to call a “preferred form for modification.” If we ask whether the training data is the source code, the original trained GPL code itself cannot be said to be the source of the model, nor is it clear if it refers to the entire vast and heterogeneous training dataset. It is difficult to define what should be disclosed to redistribute the model under GPL compliance, and it could lead to an extreme conclusion that all code and data used for model training must be disclosed. While this is what some freedom believers aim for, it can only be said to be unrealistic in reality, and it deviates from the point of the GPL’s intent to enable users to modify and build from source. Thus, existing GPL provisions are not designed to directly cover products like AI models, and forcing their application causes discrepancies in both text and operation.
In fact, in the “Open Source AI Definition” compiled by the OSI (Open Source Initiative) in 2023, regarding “information necessary for modification” of the model, it stopped at stating that sufficiently detailed information about the training data should be disclosed, and did not require the provision of the training data itself in its entirety. Also, it states that model weights and training code should be published under OSI-approved licenses.
In addition, the FSF (Free Software Foundation) itself does not believe that the current GPL interpretation alone can guarantee freedom in the AI domain, and announced in 2024 that it has started formulating “conditions for machine learning applications to be free.” There, the directionality is shown that “the four freedoms should be guaranteed to users including not only software but also raw training data and model parameters,” but this conversely is a recognition that this is not guaranteed under current licenses. The FSF also points out that “since model parameters cannot be said to be source comprehensible to humans, modification through retraining is more realistic than direct editing,” and can be said to be cautious about treating models on the extension of existing GPL. Overall, claiming GPL propagation univocally to AI models that fall outside the wording and assumptions of GPL provisions is unreasonable from the perspective of interpretation.
Arguments for Negation at the Technical Layer
There are also strong counterarguments from a technical perspective against the theory of GPL propagation to models. AI models, particularly those called large language models, basically hold huge statistical trends internally and do not store the original code or text as they are like a database. Returning a specific output for a specific input is merely generation according to a probability distribution, and it is not guaranteed that the same output as the training data is always obtained. If the model does not perform verbatim reproduction of training data except for a very small number of exceptional cases, evaluating it as “containing GPL code” within the model does not fit the technical reality. In fact, the OpenAI side argued in the GEMA lawsuit that “the model does not memorize individual training data, but merely reflects knowledge learned from the entire dataset in parameters.” This argument was not accepted by the Munich District Court, but that was because there was a clear example of lyric reproduction; conversely, unless there is a clear example of reproduction, the view would be that “the model is a lump of statistical knowledge”.
Furthermore, although it has been confirmed that models can output fragments of training data, that proportion is considered extremely limited when viewed from the whole. Regarding the whole as a reproduction based on the existence of partial memory is like claiming the whole is a reproduction of a photograph just because it contains a tiny mosaic-like fragment in an image, which is an excessive generalization. Technically, it is difficult to quantitatively measure how far specific parameters of the model retain the influence of the original data, and the correspondence between the model and training data remains statistical and difficult to draw a line. Therefore, criteria such as “how similar must it be for GPL to propagate?” cannot be established in the first place. The judgment of infringement or not has to be done on an individual output basis, and this would not be consistent with the idea of applying a single license to the entire model. From the technical aspect, since the model is basically a statistical transformation and the majority is unrelated to GPL code, applying GPL collectively can be said to be irrational.
Practical and Policy Arguments for Negation
Finally, major demerits can be pointed out regarding the theory of license propagation to models from practical and policy perspectives. What would happen if this GPL propagation theory were legally recognized? As an extreme example, if 1 million code repositories were used for training a certain large-scale model, all the various licenses contained in them (GPL, MIT, Apache, proprietary, etc.) would “propagate” to the model, and the model provider would have to distribute the model in a form that complies with all 1 million license clauses. As a practical matter, there would be combinations where conditions contradict, such as GPLv2 and Apache-2.0, and attaching and managing a huge collection of copyright notices for one model is nothing but unrealistic. Applying all licenses to an AI model created from training data with mixed licenses is practically bankrupt, and eventually, the only thing that can be done to avoid it would be to exclude code with copyleft licenses like GPL from the training data from the start.
Is such a situation really desirable for our community? The spirit of the GPL is to promote the free sharing and development of software. However, if asserting excessive propagation to AI models causes companies to avoid using GPL code, and as a result, the value held by GPL software is not utilized in the AI era, it would be putting the cart before the horse. In the field of software development, many companies take a policy of not mixing GPL code into their own products, but similarly, if it becomes “do not include GPL in our AI training data,” GPL projects could lose value as data sources. Furthermore, the current legal battles surrounding AI are leaning more towards monetary compensation and regulatory rule-making, and the reality is that they are proceeding in a different vector from the direction of code sharing idealized by GPL. If only the theory of GPL propagation to models walks alone, in reality, only data exclusion and closing off to avoid litigation risks will progress, and there is a fear that it will not lead to the expansion of free software culture.
Policy-wise as well, governments of each country are carefully considering the use of copyrighted works in AI, but at present, there is no example establishing an explicit rule that “license violation of training data generates legal liability for the model.” Even in the EU AI Act, while there are provisions regarding the quality and transparency of training data, it does not demand compliance with open source licenses. Rather, from the perspective of promoting open science and innovation, the movement to allow text and data mining under rights limitations is strong. In Japan as well, as mentioned earlier, the direction is to broadly recognize information analysis use under Article 30-4, and the policy of forcibly applying licenses to AI models is not mainstream in current international discussions.
Based on the above, the theory of license propagation to models is highly likely to cause disadvantages to open source on both practical and policy fronts, and can be said not to be a realistic solution. What is important is how to realize the “freedom of software,” which is the philosophy of open source, in the AI era; the opinion that this should be attempted through realistic means such as ensuring transparency and promoting open model development rather than extreme legal interpretations is potent, and this is something I have consistently argued as well.
The Stance of OSI and FSF
I will also organize what stance major organizations in the open source (and free software) community are currently taking in relation to the theory of GPL propagation to AI models. Representative organizations are the Open Source Initiative (OSI) and the Free Software Foundation (FSF); while they share the goal of software freedom, they do not necessarily take the same approach regarding AI models and training data.
First, the OSI formulated the “Open Source AI Definition” (OSAID) in 2024, defining the requirements for an AI system to be called open source. This definition states that the four freedoms (use, study, modify, redistribute) similar to software should be guaranteed for AI systems as well, and defines requirements regarding “forms necessary for modification” to realize that, requiring the disclosure of the following three elements.
Data Information: Provide sufficiently detailed information about the data used for training so that a skilled person can reconstruct an equivalent model.
This does not make publishing the training data itself in its entirety mandatory, but requires disclosing the origin, scope, nature, and acquisition method if there is data that cannot be published, listing data that can be published, and providing information on data available from third parties.
Code: Publish the complete set of source code for training and running the model under an OSI-approved license.
Parameters: Publish the model weights (parameters) under OSI-approved conditions.
It should be noted that while OSI states that information regarding the code used for training and training data is indispensable in addition to model weights to realize “Open Source AI,” it does not require the complete disclosure of the training data itself. This is a flexible stance that, for example, if raw data cannot be published due to privacy or confidentiality, explaining the nature of the data by clarifying that fact can substitute. Also, the legal mechanism to ensure free use of model parameters is an issue to be clarified in the future, and at present, no conclusion has been reached on legal rights control (e.g., presence or absence of copyrightability) over parameters either.
As can be read from these, the OSI promotes opening up AI models at the level of the open source definition in principle, but keeps the handling of training data to requirements at the information disclosure level. Thereby, it can be said that the OSI avoids adopting the theory of license propagation to models to demand training data disclosure, and is exploring a realistic solution that first guarantees transparency and reproducibility. In principle, it could be said that the OSI denied the GPL propagation theory at the time of publishing the OSAID definition. Note that I am probably the one who sealed the mandatory argument for training data in the final stage of this definition’s formulation process, and I believe this was the correct judgment.
On the other hand, the FSF and FSF Europe (FSFE) take a stance more faithful to fundamental principles. FSFE declared as of 2021 that “for an AI application to be free, both its training code and training data must be published under a free software license.” That is, to modify or verify the model, one must be able to obtain it including the training data, and therefore both must be free. Also, the FSF itself stated in a 2024 statement, “Under current understanding, for an ML application to be called free, all training data and the scripts processing it must satisfy the four freedoms,” trying to extend the requirements of freedom to data. Thus, FSF/FSFE stands on the position that a model with undisclosed training data is unfree as a whole even if the software part is free.
However, the FSF simultaneously states to the effect that “whether a non-free machine learning application is ethically unjust depends on the case,” mentioning that there can be “legitimate moral reasons” for not being able to publish training data (personal information) of a medical diagnosis AI, for example. In that case, it implies that although that AI is non-free, its use might be ethically permitted due to social utility. One can see an attitude of seeking a compromise between the FSF’s ideal and reality here, but in any case, there is no mistake that the FSF ultimately aims for freedom including training data.
So, does the FSF support the theory of GPL propagation to AI models? Not necessarily. Their claim is closer to an ethical standard or ideal image rather than legal enforceability, and they are not arguing that it applies to models as an interpretation of the current GPL license. Rather, as mentioned before, they are at the stage of trying to create new standards and agreements. Even in the white paper on the Copilot issue funded by the FSF, while legal points such as copyright and license violation are discussed, substantially it has a strong aspect of being told as a GPL compliance problem for users (downstream developers) concerned that they bear the risk of GPL violation if Copilot’s output contains GPL code fragments. This is a caution to developers using AI coding tools rather than GPL application to the model itself, and is different from an approach forcing GPL compliance directly on model providers.
The Software Freedom Conservancy (SFC) naturally has a strong interest in this issue but is also cautious in some respects. The SFC started the protest campaign “Give Up GitHub” against GitHub in 2022, condemning Copilot’s methods as contrary to the philosophy of open source, and is also involved in the Copilot class action. However, in an SFC blog post, regarding this lawsuit, it showed concern about “the risk of interpretations deviating from the principles of the open source community being brought in,” and called on the plaintiffs’ side to comply with community-led GPL enforcement principles as well. The SFC also states that Copilot’s act is an “unprecedented license violation,” and while not fully denying the GPL propagation theory, it can be interpreted as fearing that a judicial precedent undesirable for the community might be created depending on the result of the legal battle. The SFC might be said to be carefully balancing between the aspect of pursuing GPL propagation and the risk of entrusting it to the judiciary.
Finally, what is concerned as the free software camp is that excessive propagation of licenses might conversely invite results that impair freedom. Both OSI and FSF ultimately want to make AI something open that anyone can utilize, but they are carefully assessing whether increasing the purity of legal theory in demands for full data disclosure really leads to achieving the objective. Considering the demerits such as the avoidance of open data due to excessive propagation interpretation or the atrophy effect due to a flurry of lawsuits, I feel that the major organizations share a commonality in that it is essential not to lose sight of the big picture of spreading freedom. Rather than inciting GPL application to models, the pursuit of realistic solutions such as how to make models and data open and which parts should be relaxed in line with reality will likely continue in the future.
Summary
I have looked at the current state of the theory of GPL propagation to AI models above, and as a conclusion, this theory is in a halfway position where “it is not touted as loudly as before, but it has not completely disappeared.” As a result of points such as license violation of training data and reproduction within the model beginning to be scrutinized in lawsuits like the Copilot class action and
GEMA v. OpenAI
, it even appears that the hurdle for infringement certification is lowering. In fact, the Munich District Court’s judgment deemed model memory as reproduction, and the claim of open source license violation survives in the Copilot litigation.
However, on the other hand, the hurdle for the propagation of licenses like GPL remains high. There is a large gap between infringement being recognized and the conclusion that the entire model must be disclosed under GPL etc. immediately. What the current lawsuits are seeking is also injunctions and damages, not the forced GPL-ization of the model. There are zero examples where the judiciary supported the theory of GPL propagation to models itself, and it is a legally uncharted territory. Even if that claim were attempted somewhere in the future, it would face the legal, technical, and practical counterarguments mentioned earlier.
However, the situation has fluid parts, and there is a possibility that the line will shift depending on the policies of each country and the trends of the community. For example, if pressure from rights holder groups strengthens in Europe, there is a possibility that guidelines including license compliance will be formulated. Also, if a consensus is formed within the community regarding the state of copyleft in the AI era, a new license might appear. If such changes occur, a phase where the theory of propagation to models is re-evaluated will also arrive.
To offer my personal opinion, what is important at this moment is the perspective of how to balance software freedom and freedom in the AI domain. Instead of blindly trying to apply the philosophy of copyleft to AI, it is necessary to think about what is best to maximize freedom while considering the technical nature and industrial structure peculiar to AI. Fortunately, solutions to practical problems such as the open publication of large-scale AI models, dataset cleaning methods, and automated attachment of license notices are already being explored by the open source community. Promoting such voluntary efforts and supporting them with legal frameworks as necessary will likely be the key to balancing freedom and development.
The theory of GPL propagation to models is a point where judgment is divided on whether it is an ideal to be pursued or a nightmare to be avoided. However, as stated in this article, seeing the situation in the current year of 2025, it is not a situation where it will become reality immediately, and the majority of the community is likely maintaining a cautious stance. Although it is speculated that trial and error will continue in the judicial, legislative, and technical aspects in the future, as our community, we need to continue exploring the point of compatibility between technological innovation and software freedom without jumping to hasty conclusions. That process itself can be said to be a new challenge in the AI era on the extension of the free software spirit.
In distributed systems, there’s a common understanding that it is not possible to guarantee exactly-once delivery of messages.
What is possible
though is
exactly-once processing
. By adding a unique idempotency key to each message, you can enable consumers to recognize and ignore duplicate messages, i.e. messages which they have received and successfully processed before.
Now, how does this work exactly? When receiving a message, a consumer takes the message’s idempotency key and compares it to the keys of the messages which it already has processed. If it has seen the key before, the incoming message is a duplicate and can be ignored. Otherwise, the consumer goes on to process the message, for instance by storing the message itself, or a view derived from it, in some kind of database.
In addition, it stores the idempotency key of the message. Critically, these two things must happen atomically, typically by wrapping them in a database transaction. Either the message gets processed
and
its idempotency key gets persisted. Or, the transaction gets rolled back and no changes are applied at all. That way, it is ensured that the consumer will process a message again upon redelivery, if it failed to do so before. It also is ensured that duplicates received after successfully processing the message are skipped over.
UUIDs
So let’s discuss what makes for a good idempotency key then. One possible option would be to use a UUIDv4. These random identifiers solve the requirement of uniquely identifying each message. However, they require the consumer to store the UUIDs of all the previous messages it ever has received in order to reliably identify a duplicate. Depending on the message volume, this may not be practical. Pragmatically, you might get away with discarding received UUIDs after some time period, if it is acceptable to occasionally receive and process a duplicate after that period. Unfortunately, neither the producer of the message nor the consumer will have any indication of the duplicated processing in that case.
We can somewhat improve this situation by adding a timestamp to the idempotency key, for instance by using a
UUIDv7
which contains both a timestamp part (first 48 bits) and a random part (remaining bits), or an
ULID
. That way, the consumer can detect when it receives a message with an idempotency key which is "too old". While it can’t decide whether the message is a duplicate or not, it can flag to the producer that it can’t handle that message. It is then upon the producer to decide how to proceed. For instance, if the message is part of a payment flow, the system might suggest to the user to first check in their banking account whether this payment has already been executed or not. Only if that’s not the case, a
new
message with the same payload and a fresh UUID would be sent.
Monotonically Increasing Sequences
All these intricacies can be avoided when it is possible to use a monotonically increasing sequence value as the idempotency key. In that case, the consumer does not need to store all the keys it ever has processed (or a reasonably sized subset thereof). It only needs to store a single value, the one of the latest message which it has processed. If it receives a message with the same or a lower idempotency key, that message must be a duplicate and can be ignored. When receiving messages from a partitioned source, such as a Kafka topic with multiple partitions, or from multiple independent producers (e.g., different clients of a REST API, each using their own separate sequence), then the latest key value per partition must be stored.
Monotonically increasing idempotency keys are a great improvement from the perspective of the message consumer. On the flipside, they may make things more complicated for producers: creating monotonically increasing sequence values isn’t without its own challenges. It is trivial if producers are single-threaded, producing one message at a time. In that case, a database sequence, or even a simple in-memory counter, can be used for creating the idempotency keys. Gaps in the sequence are fine, hence it is possible to increment the persistent state of the sequence or counter in larger steps, and dispense the actual values from an in-memory copy. That way, disk IO can be reduced. From a consumer perspective, Kafka partition offsets fall into that bucket, as they can be considered a monotonically increasing idempotency key for the messages consumed from a given partition.
Things get more complicated when the producer is subject to multiple concurrent requests at once, for instance a REST service with multiple request workers, perhaps even scaled out to multiple compute nodes in a cluster. To ensure monotonicity, retrieval of the idempotency key and emitting a message with that key must happen atomically, uninterrupted by other worker threads. Otherwise, you may end up in a situation where thread A fetches sequence value 100, thread B fetches sequence value 101, B emits a message with idempotency key 101, and then A emits a message with idempotency key 100\. A consumer would then, incorrectly, discard A’s message as a duplicate.
For most cases, ensuring this level of atomicity will impose a severe bottleneck, essentially serializing all requests of the producer system, regardless of how many worker threads or service
instances you deploy. Note that if you really wanted to go down that route, solely using a database sequence for producing the idempotency key will not work. Instead, you’d have to use a mechanism such as
Postgres advisory locks
in order to guarantee monotonicity of idempotency keys in the outgoing messages.
Deriving Idempotency Keys From the Transaction Log
Now, is there a way for us to have this cake and eat it too? Can we get the space efficiency for consumers when using monotonically increasing idempotency keys, without hampering performance of multi-threaded producers? Turns out we can, at least when the emission of messages can be made an asynchronous activity in the producer system, happening independently from processing inbound requests.
This means clients of the producer system receive confirmation that the intent to send a message or request was persisted, but they don’t get the result of the same right away.
If a use case can be modeled with these semantics, the problem can be reduced to the single-threaded situation above: instead of emitting messages directly to the target system, each producer thread inserts them into a queue. This queue is processed by a single-threaded worker process which emits all the messages sequentially. As argued in
The Synchrony Budget
, making activities asynchronous can be generally advantageous, if we don’t require their outcome right away.
One specific way to do so would be a variation of the widely used
outbox pattern
, utilizing the transaction log of the producer service’s database. After all, it’s not necessary to sequence inbound requests ourselves as the database already is doing that for us when serializing the transactions in its log. When producers persist the intent to send a message in the transaction log—for instance by writing a record into a specific table—a process tailing the log can assign idempotency keys to these messages based on their position in the transaction log.
An implementation of this is straight-forward using tools for log-based Change Data Capture (CDC), such as
Debezium
: You retrieve the messages to be sent from the log by capturing the INSERT events from the outbox table, and assign an idempotency key before emitting them, derived from their log offset. The exact details are going to depend on the specific database.
For example, in Postgres
it is ensured
that the log sequence numbers (LSN) of commit events within its write-ahead log (WAL) are monotonically increasing: the commit event of a transaction committing after another transaction will have a higher LSN. Furthermore, it is guaranteed that within a given transaction, the LSNs of the events are also monotonically increasing. This makes the tuple of
{ Commit LSN, Event LSN }
a great fit for an idempotency key. In order to not leak the fact that a producer is using a Postgres database, both values can be encoded into a single 128 bit number value. Note that you don’t need to deploy Kafka or Kafka Connect for this solution. Debezium’s
embedded engine
is a great fit for this use case, allowing you to assign idempotency keys from within a callback method in the producer service itself, not requiring any further infrastructure.
When using Postgres to implement this pattern, you don’t even need a dedicated outbox table, as it lets you write arbitrary contents into the transaction log via
pg_logical_emit_message()
, which is perfect for the use case at hand.
Discussion
So, when to use which kind of idempotency key then?
As always, there are no silver bullets, and the answer depends on your specific use case.
For many scenarios, using UUIDs and dropping them after some time will probably be sufficient,
provided you can tolerate that messages occasionally can be processed a second time when duplicates arrive after the retention period of processed keys.
The more messages you need to process overall,
the more attractive a solution centered around monotonically increasing sequences becomes,
as it allows for space-efficient duplicate detection and exclusion, no matter how many messages you have.
The proposed log-based approach can be an efficient solution for doing so,
but it also adds operational complexity:
your database needs to support logical replication,
you need to run a CDC connector, etc.
However, many organizations already operate CDC pipelines for other purposes
(analytics, search indexing, cache invalidation, etc.). If you’re in that category,
the incremental complexity is minimal. If you’re not, you should weigh the operational
overhead against the benefits (constant-space duplicate detection) for your specific scale.
Don’t buy new tech this Black Friday: expert tips for buying refurbished phones and laptops
Guardian
www.theguardian.com
2025-11-27 12:00:25
Tech is on its last legs? Refurbished can be the cheaper, greener option. Here’s how to choose well and avoid the pitfalls • How to shop smart this Black Friday• How to make your phone last longer Even if you do your best to avoid it, it’s hard to escape the noise of retailers offering implausible-s...
E
ven if you do your best to avoid it, it’s hard to escape the noise of retailers offering implausible-seeming Black Friday discounts on desirable technology. What they won’t be so keen to highlight is the huge environmental cost that comes from feeding the capitalist beast year on year, given what an obvious sales vibe-killer it would be.
While the best approach for the planet is to opt out completely and observe the alternate holiday of
Buy Nothing Day
instead, such an approach can prove self-defeating to your finances in the long term. If you
shop smart on Black Friday
and avoid the lure of impulse buys, it’s a good time to stock up on the things you need, at lower prices than at the rest of the year.
In other words, if your phone, laptop or TV is on its figurative last legs, Black Friday is a sensible time to seek a replacement. But if you’re hoping to limit your ecological impact, it’s definitely worth considering refurbished options.
“As a consumer, you save directly 30-40% versus new, and you also have this feeling of doing the right thing,” says Peter Windischhofer, co-founder and CEO of
Refurbed
. “Because you buy a product that already exists, you don’t have to produce a new one.”
James Rigg, CEO of Trojan Electronics, agrees: “Very often, it’s the better choice: reliable performance, lower prices and a fraction of the environmental cost.
“Buy from someone reputable, look for transparency, and prioritise warranty and repairability over a too-good-to-be-true discount, and you can come out of Black Friday with great devices and a lighter footprint.”
Five tips when buying refurbished
Read the description
Refurbished can mean different things. See what condition is promised, paying special attention to battery health.
Check the warranty and returns policy
You want to know that you’re in good hands should anything go wrong.
Research the seller’s reputation
Look at customer reviews and internet feedback. If on eBay, look for sellers in the company’s
Refurbished
programme.
Research your chosen device
The older the device, the bigger the discount – but this is a false economy if you have to replace it sooner. With phones and laptops, especially, make sure they’re getting updates and will be able to cope with years of use.
Don’t cheap out
A low price is only a bargain if it actually delivers. Prioritise customer service and a transparent refurbishment process over saving a few pounds.
Refurbished vs pre-owned
The process of buying refurbished electronics is often as good as buying new.
Photograph: dikushin/Getty Images
It’s important at this point to define terms. People sometimes use the phrases “preowned”, “secondhand” and “refurbished” interchangeably, which may have given the latter a bad rap.
“They really are quite, quite different,” says Katy Medlock, Back Market’s UK general manager. “Secondhand could be peer to peer: you’re not buying it with a warranty, it hasn’t been through a quality check, you’re not buying it with all the mod cons.”
That separates refurbished marketplaces such as
Back Market
,
MusicMagpie
,
Refurbed
and others from sites where you buy directly from a member of the public, such as Facebook Marketplace or Craigslist. “Our mission is all about trying to make the process of buying refurbished electronics as good as buying something new,” Medlock says, highlighting the ability Back Market offers to split payments, its 30-day money-back promise, 12-month warranty and next-day delivery as ways of seeking parity with retailers shipping new stock.
By contrast, buying preowned or secondhand on the private market is a gamble, even if you sidestep the risk of being scammed.
“I’ve heard so many horror stories from peer-to-peer when it comes to electronics,” says Windischhofer. “I love using those platforms for low-value products; I use a lot of Vinted for clothing, and it’s great. But for electronics, my advice would be to go through a professional where you get a warranty and an invoice, and where, in case this product breaks after five days, they have great customer service to really help you out.”
Items sold privately may also have unseen defects or botched repairs using cheap, third-party parts. “There’s a very different performance from the cheapest phone screen that can be put on a phone and the correct screen,” says Murdock. “Products get traded in at times with a $20 screen on an iPhone or Samsung [device] that really should have a wholesale cost of $150. They’re almost illegible.”
In other words, peer-to-peer buys are a gamble. If you insist on taking that risk, it’s best to buy in person and to take the opportunity to check both the screen and camera alongside signs of cosmetic wear and tear. If buying an iPhone or Android device, check the battery’s health in the phone’s settings and make sure it’s at least 80%.
If buying peer-to-peer on eBay, rather than through a certified refurbished seller, Williams encourages shoppers to look through the feedback and study photos and the product description. “Is this a reliable seller? Have they sold a lot of things in the past? What’s their feedback score? These are all things that are helpful to know that they’ve had good transactions with buyers in the past.”
Crunching the numbers
Buying refurbished tech can shrink your e-waste footprint.
Photograph: Pituk Loonhong/Getty Images
What does that ecological improvement look like in real terms? Let’s take smartphones – the bestselling refurbished category.
According to
research from Back Market
, buying a refurbished model rather than a new one will save 178g of e-waste and 77,000 litres of water, while preventing 77kg of carbon emissions and 244kg of raw materials from being mined.
Trojan Electronics, meanwhile,
estimates
that a refurbished smartphone uses 91.3% fewer raw materials and 86.4% less water while putting 91.6% less carbon emissions into the atmosphere compared with buying a new device.
Plenty of variables make the exact figures impossible to pin down. But while there’s still an impact from refurbished buys – especially if a new battery or screen is installed as part of the refurbishing process – there’s no denying that the harm is lessened.
“Every time you buy a smartphone, it’s about a 58kg CO
2
offset if you don’t buy a new one [and instead] buy one that’s been refurbished,” says James Murdock, co-founder of
Alchemy
, a company that refurbishes old technology for resale. “That includes the operation to actually refurbish it, and the existence of our business. It’s even more if you’re buying a laptop.”
It’s not just phones, either. “Now, people are turning to all different types of electronics, everything from refurbished air fryers through to hair accessories from TVs, coffee machines, ice-cream makers and games consoles,” says Eve Williams, general manager at eBay UK. Unlike its traditional peer-to-peer sales business, eBay has a method by which it
certifies refurbished-reselling businesses
.
“There’s nothing to lose [from buying refurbished],” she says, pointing out that if you get the same 12-month guarantee as you would if buying new, you’re getting the same product at a cheaper price. “You’re getting something that otherwise would have ended up in landfill. We have this amazing seller called
Preloved Tech
. Matt and Andrea, who run it, started their business by working in schools, going in and seeing the times that schools were throwing out laptops and iPads because they didn’t know what to do with them. There are no rules around what some businesses should do to get rid of or recycle technology.
“They’re taking a lot of that tech and are able to give money and funding back to the schools, while also then being able to refurbish these items to sell on, and extend the life of them.”
Avoid smartphones that have stopped receiving security updates.
Photograph: Tim Robberts/Getty Images
“The best advice I can give for buying refurbished is to go via established retailers such as
Back Market
,
Giffgaff
and
Vodafone
, and if you’re buying through eBay then try to get a device that’s listed as ‘certified refurbished’,” says technology journalist Thomas Deehan. “You can also find different perks available depending on where you buy. For instance, Giffgaff provides a two-year warranty on ‘like new’ phones, while Back Market often gives you the option to have an entirely new battery fitted into your device of choice.”
In general, the more information a retailer provides about their refurbishing process, the better. “A proper refurbisher will tell you exactly what’s been checked, what’s been replaced, and how the product is graded,” says Rigg. “If you see vague descriptions like ‘used’ or ‘open box’ with no explanation, that’s your cue to walk away.”
Jon Miller, chief commercial officer at MusicMagpie, agrees. “As a rule of thumb, it’s far wiser to be careful when buying refurbished tech. Try to check the warranty length and what exactly that covers. Similarly, check the fine print and ensure ‘refurbished’ means the tech has been properly restored and tested, not just used.”
Is there any preowned technology you should avoid? “If you had asked me that question 10 years ago, I would have said: ‘Yes, because technology is so fast and, you know, the new iPhone is so much better than the last one,’” says Windischhofer. “But now? I don’t think there’s any product that I would buy new. I would always buy the generation before that.”
Alchemy, similarly, practices what it preaches. “We’ve got hundreds of MacBooks, iPhones and other products in the company. We have about 350 employees, and no one, including me, has ever had a new one,” says Murdock.
There are limits, however, to how old a device you should get, and it varies by category. A
PlayStation 4
will continue to play all disc-based games made for it indefinitely, but something internet-reliant may end up losing functionality with time. Older OLED TVs, too, are considered more prone to burn-in than their modern counterparts.
Equally, buying a five-year-old refurbished phone will be cheaper than buying last year’s model, but if you end up needing to replace it sooner – for example, if it’s an Android that’s stopped receiving security updates – then it’s a false economy.
“It might be tempting to get an old MacBook Air from 2015, but it won’t receive regular updates like M-series MacBooks will,” says Deehan. “I think there’s a good balance to strike where you sometimes buy things new and other times go down the refurbished route. It’s about saving money where it makes sense so that you have the freedom to spend a bit more on a device that you’ll make the most of.”
Finally, don’t forget the age-old adage of “buy cheap, buy twice”. Or, as Rigg puts it: “A gadget that fails quickly and can’t be repaired is only cheap for about five minutes.”
“Don’t be tempted by the cheapest offshore, interesting-looking eBay seller that looks like it’s been there for five minutes,” says Murdock. “Buy from a legitimate-looking local partner that’s offering a warranty, as they would with any other kind of new product, and is willing to stand behind that.”
If you plan to resell your old devices, giving them a good clean will make them look their best.
Photograph: MDV Edwards/Getty Images
So you’ve bought a refurbished phone, laptop or tablet and now have one that’s surplus to requirements. What should you do?
Recycling is an option, but one that should be considered a last resort, according to Medlock, especially as there are limits on capacity. “With the amount of new electronics that are produced every year … there’s no way that … the amount sold in the UK could ever be recycled.”
Many sites will offer a trade-in when you buy the replacement, which will increase the likelihood of a device getting a second lease of life in the refurbished ecosystem.
Alternatively, you could take it to
CeX
or try your luck on the peer-to-peer market through
Gumtree
,
Facebook Marketplace
,
Craigslist
or
eBay
. If you go down this route, do be sure to wipe all your personal data first and give it a good spruce-up.
“This should go without saying, but prior to being listed or sold, your tech should be given a good clean to make it look the best that it can,” says Deehan. “Some retailers will dock the value of your device based on its cosmetic appearance, so make sure to do your part in keeping it all lint- and fingerprint-free.”
Donating is also an option. I recently donated an old laptop to
Screen Share
, which repurposes computers and phones for refugees to help tackle digital exclusion, and a quick search online will uncover many other worthwhile places to donate.
Whatever you choose, Medlock suggests putting some thought into the end of life for today’s Black Friday tech buys. “If people are upgrading, the pathway is obviously to buy refurbished … and encouraging people at that point to buy better.
“There is a cost saving there, so people can use that saving to buy devices that last longer, are built to last and built to be repaired.”
Alan Martin
has more than a decade’s experience of writing about and reviewing consumer technology. He once had to strip off to retrieve a battery-depleted drone from a freezing cold
south London lake, showing his deranged devotion to the career. Nearby swans were left confused
OpenAI discloses API customer data breach via Mixpanel vendor hack
Bleeping Computer
www.bleepingcomputer.com
2025-11-27 11:27:06
OpenAI is notifying some ChatGPT API customers that limited identifying information was exposed following a breach at its third-party analytics provider Mixpanel. [...]...
OpenAI is notifying some ChatGPT API customers that limited identifying information was exposed following a breach at its third-party analytics provider Mixpanel.
Mixpanel offers event analytics that OpenAI uses to track user interactions on the frontend interface for the API product.
According to the AI company, the cyber incident affected “limited analytics data related to some users of the API” and did not impact users of ChatGPT or other products.
“This was not a breach of OpenAI’s systems. No chat, API requests, API usage data, passwords, credentials, API keys, payment details, or government IDs were compromised or exposed,”
OpenAI says
in a press release.
Mixpanel
reported
that the attack “impacted a limited number of our customers” and resulted from a smishing (SMS phishing) campaign that the company detected on November 8.
OpenAI received details of the affected dataset on November 25 after being informed of Mixpanel’s ongoing investigation.
The AI company notes that the exposed information may include:
Name that was provided to us on the API account
Email address associated with the API account
Approximate coarse location based on API user browser (city, state, country)
Operating system and browser used to access the API account
Referring websites
Organization or User IDs associated with the API account
Because no sensitive credentials were exposed, users do not need to reset passwords or regenerate API keys.
Some users are
reporting
that CoinTracker, a cryptocurrency portfolio tracker and tax platform, has also been impacted, with exposed data also including device metadata and limited transaction count.
OpenAI has started an investigation to determine the full scope of the incident. As a precaution, it has removed Mixpanel from its production services and is notifying organizations, administrators, and individual users directly.
While OpenAI underlines that only users of its API are impacted, it notified all its subscribers.
The company warns that the leaked data could be leveraged in phishing or social-engineering attacks and advises users to watch for credible-looking malicious messages related to the incident.
Messages containing links or attachments should be verified to ensure they originate from an official OpenAI domain.
The company also urges users to enable 2FA and never send sensitive information, including passwords, API keys, or verification codes, through email, text, or chat.
Mixpanel’s CEO, Jen Taylor, said that all impacted customers have been contacted directly. “If you have not heard from us, you were not impacted,” she noted.
In response to the attack, Mixpanel secured affected accounts, revoked active sessions and sign-ins, rotated compromised credentials, blocked the threat actor’s IP addresses, and reset passwords for all employees. The company has also implemented new controls to prevent similar incidents in the future.
It's budget season! Over 300 CISOs and security leaders have shared how they're planning, spending, and prioritizing for the year ahead. This report compiles their insights, allowing readers to benchmark strategies, identify emerging trends, and compare their priorities as they head into 2026.
Learn how top leaders are turning investment into measurable impact.
After a long search I've finally figured out how to embed Commodore 64 BASIC listings into blog posts.
This isn't a game, just a visual demo showing off some of the C64's capabilities. It's taken from two one-line programs in the 1985 Special Issue of Run magazine, posted by L.F.S. and Glenn Zuch. I've combined them.
Okay, a few notes here. First off, the Commodore 64 keyboard had special keys for setting and changing color. These are hard to convey in printed listings, so the convention is to use {} brackets and indicate what keys should be pressed.
{yellow}
is a special character on the 8 key accessed by hitting control+8; this turns the following output yellow.
{up}{down}{left}
and
{right}
are the cursor keys. You get them by hitting those keys. They move the on-screen cursor, changing where the next character will print.
{reverse on}
is control+9, and that basically reverses the color of the next printed character; a space will be a full color "block", etc.
Peek and Poke show up as commands in a few different 8-bit BASICs... what they do is let you look at (peek) or change (poke) memory addresses directly. This is different for every computer model, of course, and it isn't obvious what exactly they do without a memory reference chart.
Poke 53280
lets us directly change the color of the "frame" around our graphics window, and
53281
lets us change the color of the background itself.
The
? mid$
line chooses one of the cursor direction codes we have at random, making the asterisk appear to burrow in a random direction each update.
So, a simple program that crams a lot into two lines. The original was line 2; I adapted line 1 (changing the colors) from another one-liner to make the colors pop a bit more.
There’s an old electronics joke that if you want to build an oscillator, you should try building an amplifier. One of the fundamental criteria for oscillation is the presence of signal gain; without it, any oscillation is bound to decay, just like a swing that’s no longer being pushed must eventually come to a stop.
In reality, circuits with gain can occasionally oscillate by accident, but it’s rather difficult to build a good analog oscillator from scratch. The most common category of oscillators you can find on the internet are circuits that don’t work reliably. This is followed by approaches that require exotic components, such as center-tapped inductors or incandescent lightbulbs. The final group are the layouts you can copy, but probably won’t be able to explain to a friend who doesn’t have an EE degree.
In today’s article, I wanted to approach the problem in a different way. I’ll assume that you’re up-to-date on some of the key lessons from earlier articles: that you
can tell the difference between voltage and current
, have a
basic grasp of transistors
, and know what happens when a
capacitor is charged through a resistor
. With this in mind, let’s try to construct an oscillator that’s easy to understand, runs well, and has a predictable operating frequency. Further, let’s do it without peeking at someone else’s homework.
The simplest form of an oscillator is a device that uses negative feedback to cycle back and forth between two unstable states. To illustrate, think of a machine equipped with a light sensor and a robotic arm. In the dark, the machine is compelled to stroll over to the wall switch and flip it on. If it detects light, another part of its programming takes over and toggles the switch off. The machine is doomed to an endless cycle of switch-flipping at a frequency dictated by how quickly it can process information and react.
At first blush, we should be able to replicate this operating principle with a single n-channel MOSFET. After all, a transistor can be used as an electronically-operated switch:
A wannabe oscillator.
The transistor turns on when the voltage between its gate terminal and the source leg (
Vgs
) exceeds a certain threshold, usually around 2 V. When the power supply first ramps up, the transistor is not conducting. With no current flowing through, there’s no voltage drop across the resistor, so
Vgs
is pulled toward the positive supply rail. Once this voltage crosses about 2 V, the transistor begins to admit current. It stands to reason that the process shorts the bottom terminal of the resistor to the ground and causes
Vgs
will plunge to 0 V. If so, that would restart the cycle and produce a square wave on the output leg.
In practice, this is not the behavior you’ll see. For a MOSFET, the relationship between
Vgs
and the admitted current (
Id
) is steep, but the device is not a binary switch:
BS170 Vgs-Id curve for Vds = 1 V. Captured by author.
In particular, there is a certain point on that curve, somewhere in the vicinity of 2 V, that corresponds to the transistor only admitting a current of about 300 µA. From Ohm’s law, this current flowing through a 10 kΩ resistor will produce a voltage drop of 3 V. In a 5 V circuit, this puts
Vgs
at 5 V - 3 V = 2 V. In other words, there exists a stable equilibrium that prevents oscillation. It’s akin to our robot-operated light switch being half-on.
To fix this issue, we need to build an electronic switch that has no stable midpoint. This is known as
Schmitt trigger
and its simple implementation is shown below:
A discrete-transistor Schmitt trigger.
To analyze the design, let’s assume the circuit is running off
Vsupply = 5
V. If the input signal is 0 V, the transistor on the left is not conducting, which pulls
Vgs
for the other MOSFET all the way to 5 V. That input allows nearly arbitrary currents to flow through the right branch of the circuit, making that current path more or less equivalent to a two-resistor a voltage divider. We can calculate the midpoint voltage of the divider:
This voltage is also propagated the source terminal of the input transistor on the left. The actual
Vth
for the
BS170
transistors in my possession is about 2.15 V, so for the input-side transistor to turn on, the supplied signal will need to exceed
Vs + Vth ≈
2.6 V in reference to the ground. When that happens, a large voltage drop appears across R1, reducing the
Vgs
of the output-side transistor below the threshold of conduction, and choking off the current in the right branch.
At this point, there’s still current flowing through the common resistor on the bottom, but it’s now increasingly sourced via the left branch. The left branch forms a new voltage divider; because R1
has a higher resistance than R2,
Vs
is gradually reduced, effectively bumping up
Vgs
for the left transistor and thus knocking it more firmly into conduction even if the input voltage remains constant. This is a positive feedback that gives the circuit no option to linger in a half-on state.
Once the transition is complete, the voltage drop across the bottom resistor is down from 450 mV to about 50 mV. This means that although the left transistor first turned on when the input signal crossed 2.6 V in reference to the ground, it will not turn off until the voltage drops all the way to 2.2 V — a 400 mV gap.
This circuit lets us build what’s known as a
relaxation oscillator
. To do so, we only need to make two small tweaks. First, we need to loop an inverted output signal back onto the input; the most intuitive way of doing this is to add another transistor in a switch-like configuration similar to the failed design of a single-transistor oscillator mentioned earlier on. This building block, marked on the left, outputs
Vsupply
when the signal routed to the gate terminal is 0 V, and produces roughly 0 V when the input is near
Vsupply
:
A Schmitt trigger oscillator.
Next, to set a sensible oscillation speed, we need to add a time delay, which can be accomplished by charging a capacitor through a resistor (middle section). The resistor needs to be large enough not to overload the inverter stage.
For the component values shown in the schematic, the circuit should oscillate at a frequency of almost exactly 3 kHz when supplied with 5 V:
An oscilloscope trace for the circuit, by author.
The frequency is governed by how long it takes for the capacitor to move
Δv =
400 mV between the two Schmitt thresholds voltages:
the “off” point at 2.2 V and the “on” point at 2.6 V.
Because the overall variation in capacitor voltage is small, the we can squint our eyes and say that the voltage across the 100 kΩ resistor is nearly constant in every charge cycle. When the resistor is connected to the positive rail,
V
R
≈ 5 V – 2.4 V ≈ 2.6 V. Conversely, when the resistor is connected to the ground, we get
V
R
≈ 2.4 V. If the voltages across the resistor are nearly constant, so are the resulting capacitor currents:
From the
fundamental capacitor equation
(
Δv = I · t/C
), we can solve for the charging time needed to move the voltage by
Δv
= 400 mV; the result is about 154 µs for the charging period and 167 µs for the discharging period. The sum is 321 µs, corresponding to a frequency of about 3.1 kHz – pretty close to real life.
The circuit can be simplified to two transistors at the expense of readability, but if you need an analog oscillator with a lower component count, an
operational amplifier
is your best bet.
If you’re rusty on op-amps, I suggest pausing to review the article linked in the preceding paragraph. That said, to understand the next circuit, all you need to know is that an op-amp compares two input voltages and that
Vout
swings toward the positive rail if
Vin+
≫
Vin-
or toward the negative rail if
Vin+
≪
Vin-
.
An op-amp relaxation oscillator.
For simplicity, let’s choose R1 = R2 = R3 and then look at the non-inverting (
Vin+
) input of the chip. What we have here is a three-way voltage divider: the signal on the non-inverting input is simple average of three voltages:
Vsupply
(5 V), ground (0 V), and
Vout
. We don’t know the value of
Vout
just yet, but it can only vary from 0 V to
Vsupply
, so the
V
in+
signal will always stay between ⅓ ·
Vsupply
and ⅔ ·
Vsupply.
Next, let’s have a look at the inverting input (
Vin-
). When the circuit is first powered on, the capacitor C isn’t charged, so
Vin-
sits at 0 V. Since the voltage on the non-inverting input can’t be lower than ⅓ ·
Vsupply
, this means that on power-on,
Vin+
≫
Vin-
, sending the output voltage toward the positive rail. When
Vout
shoots up, it also bumps the
Vin+
average to ⅔ ·
Vsupply.
Because
Vout
is now high, this starts the process of charging the capacitor through the bottom resistor (R
cap
). After a while, the capacitor voltage is bound to exceed ⅔ ·
Vsupply
. The capacitor voltage is also hooked up to the amplifier’s inverting input, and at that point,
Vin-
begins to exceeds
Vin+
, nudging the output voltage lower. Stable equilibrium is not possible because this output voltage drop is immediately reflected in the three-way average present on the
Vin+
leg, pulling it down and causing the difference between
Vin-
and
Vin+
to widen. This positive feedback loop puts the amplifier firmly into the
Vin+
≪
Vin-
territory.
At that point,
Vout
must drop to 0 V, thus lowering the voltage on the non-inverting leg to ⅓ ·
Vsupply
. With
Vout
low, the capacitor starts discharging through R
cap
, but it needs to travel from the current charge state of ⅔ ·
Vsupply
all the way to ⅓ ·
Vsupply
before
Vin-
becomes lower than
Vin+
and the cycle is allowed to restart.
The continued charging and discharging of the capacitor between ⅓ ·
Vsupply
and ⅔ ·
Vsupply
results in periodic oscillation. The circuit produces a square wave signal with a period dictated by the value of C and R
cap
. The frequency of these oscillations can be approximated analogously to what we’ve done for the discrete-transistor variant earlier on. In a 5 V circuit with R1 = R2 = R3, the capacitor charges and discharges by
Δv ≈
1.67 V. If R
cap
= 10 kΩ, then the quasi-constant capacitor charging current is
I
≈
2.5 V / 10 kΩ
≈
250 µA.
Knowing
Δv
and
I
, and assuming C = 1 µF, we can tap into the capacitor equation (
Δv = I · t/C
) to solve for
t
. The result is 6.67 ms. This puts the charge-discharge roundtrip at 13.34 ms, suggesting a frequency of 75 Hz. The actual measurement is shown below:
Oscilloscope trace for the relaxation oscillator. By author.
The observed frequency is about 7% lower than predicted: 70 instead of 75 Hz. Although I could pin this on component tolerances, a more honest explanation is that at
Δv ≈
1.67 V, the constant-current approximation of the capacitor charging process is stretched thin; the segments in the bottom oscilloscope trace diverge quite a bit from a straight line.
Short of reducing R3 to bring down
Δv
and thus reduce the variations in current, the way to develop a better formula is to tap into the equation for a capacitor charged by a constant voltage via a resistor, as derived
here
:
\(V_{cap} = V_{in} \cdot (1 - e^{-t \over RC})\)
To make the math simple, we can use ⅓ ·
Vsupply
as the reference point for the calculation. In this view, the “virtual” supply voltage is
Vin =
⅔ ·
Vsupply
(because we took away the unused bottom ⅓) and the capacitor is charging from 0 V to
Vcap
=
50%
·
Vin (
i.e., ⅓
of ⅔
).
To find the charging time, we just need to rearrange the R-C formula for the
Vcap/Vin
ratio, and then solve for
t
at which the value works out to 50% (0.5):
If we plug 1 µF and 10 kΩ into the equation, the value works out to 72 Hz, which is within 3% of the observed behavior, comfortably within the tolerances of standard passive components.
The method outlined earlier on is not the only conceptual approach to build oscillators. Another way is to produce resonance. We can do this by taking a standard op-amp voltage follower which uses negative feedback to control the output — and then mess with the feedback loop in a particular way.
An op-amp voltage follower.
In the basic voltage follower configuration, the op-amp reaches a stable equilibrium when
Vin+
≈
Vin-
≈
Vout
. Again, the circuit works only because of the negative feedback loop; in its absence,
Vin-
would diverge from
Vin+
and the output voltage would swing toward one of the supply rails.
To turn this circuit into an oscillator, we can build a feedback loop that normally provides negative feedback, but that inverts the waveform at a particular sine-wave frequency. This turns negative feedback into positive feedback; instead of stabilizing the output voltage, it produces increasing swings, but only at the frequency at which the inversion takes place.
Such a selective waveform inversion sounds complicated, but we can achieve it a familiar building block: an R-C lowpass filter. The mechanics of these filters are discussed in
this article
; in a nutshell, the arrangement produces a frequency-dependent phase shift of 0° (at DC) to -90° (as the frequency approaches infinity). If we cascade a couple of these R-C stages, we can achieve a -180° phase shift at some chosen frequency, which is the same as flipping the waveform.
A minimalistic but well-behaved op-amp solution is shown below:
A rudimentary phase-shift oscillator.
In this particular circuit, an overall -180° shift happens when each of the R-C stages adds its own -60°. It’s easy to find the frequency at which this occurs. In the aforementioned article on signal filtering, we came up with the following formula describing the shift associated with the filter:
\(\theta = -arctan( 2 \pi f R C )\)
Arctangent is the inverse of the tangent function. In a right triangle, the tangent function describes the ratio of lengths of the opposite to the adjacent for a particular angle; the arctangent goes the other way round, giving us an angle for a particular ratio. In other words, if
x
=
tan(α)
then
α
=
arctan(x).
This allows us to rewrite the equation as:
\(2 \pi f R C = -tan(\theta)\)
We’re trying to solve for
f
at which
θ
= -60°; the value of
-tan(-60°)
is roughly 1.73, so we can plug that into the equation and then move everything except
f
to the right. Throwing in the component values for the first R-C stage in the schematic, we obtain:
You’ll notice that the result is the same for the other two stages: they have higher resistances but proportionally lower capacitances, so the denominator of the fraction doesn’t change.
Oscilloscope traces for the circuit are shown below:
Traces for the three R-C stages.
Because the amplifier’s gain isn’t constrained in any way, the output waveform is a square wave. Nevertheless, in a lowpass circuit with these characteristics, the resulting waveforms are close enough to sinusoids that the sine-wave model approximates the behavior nearly perfectly. We can run a discrete-time simulation to show that the sine-wave behavior of these three R-C stages (gray) aligns pretty well with the square-wave case (blue):
A simulation of a square & sine wave passing through three R-C filters.
To make the output a sine wave, it’s possible to tinker with with the feedback loop to lower the circuit’s gain, but it’s hard to get it right; insufficient gain prevents oscillation while excess gain produces distortion. A simpler trick is to tap into the signal on the non-inverting leg (bottom oscilloscope trace) and use the other part of a dual op-amp IC to amplify this signal to your heart’s desire.
Some readers might be wondering why I designed the stages so that each of them has an impedance ten times larger than the stage before it. This is to prevent the filters from appreciably loading each other. If all the impedances were in the same ballpark, the middle filter could source currents from the left as easily as it could from the right. In that situation, finding the point of -180° phase shift with decent accuracy would require calculating the transfer function for the entire six-component Franken-filter; the task is doable but — to use a mathematical term —
rather unpleasant
.
Footnote: in the literature, the circuit is more often constructed using highpass stages and a discrete transistor as an amplifier. I’d wager that most authors who present the discrete-transistor solution have not actually tried it in practice; otherwise, they would have found it to be quite finicky. The version presented in this article is discussed
here
.
If you enjoyed the content, please subscribe. I’m not selling anything; it’s just a good way to stay in touch with the writers you like.
Discussion about this post
How Arthur Conan Doyle Explored Men's Mental Health Through Sherlock Holmes
Note:
This article is republished from
The Conversation
under a Creative Commons license. It includes links to external sites that may earn a commission for purchases. We did not add these links and have kept the original content intact.
Arthur Conan Doyle was not just one of the world’s best crime fiction writers. He was a progressive wordsmith who brought light to controversial and taboo subjects. One of those taboo subjects was male vulnerability and mental health problems – a topic of personal significance to the author.
Doyle was a
vulnerable child
. His father, Charles,
was an alcoholic
, which led to financial troubles in the family. Charles was admitted to an asylum in 1881 and spent the next 12 years in various
mental care establishments
. So began Doyle’s interest in male vulnerability and mental health.
The character of Sherlock Holmes is a true expression of male vulnerability that does not equate it with weakness. Doyle does not represent Holmes as infallible, but as a man others can relate to – he battles with drug addiction, loneliness and depression. His genius thrives in part because of these vulnerabilities, not despite them.
In The Man with the Twisted Lip, for example, a man named Neville St Clair hides his double life. He tells his family that he is a respectable entrepreneur going to London on business. In reality he is begging on the city streets. He lives this double life due to fear and shame over the inability to pay off his debts. “It was a long fight between my pride and the money,” he explains, “but the dollars won at last.”
“I would have endured imprisonment, ay, even execution, rather than have left my miserable secret as a family blot to my children,” St Clair says. In having his character consider execution to protect his and his family’s reputation, Doyle explored the societal expectations of Victorian masculinity and how men struggled with such pressures.
The Stockbroker’s Clerk also examines male suicide, as well as economic and professional anxieties. When Holmes reveals the crimes of Harry Pinner, the man attempts suicide rather than face prison.
In The Engineer’s Thumb, hydraulic engineer Victor is treated physically by Watson and mentally by Holmes. As Doyle writes: “Round one of his hands he had a handkerchief wrapped, which was mottled all over with bloodstains. He was young, not more than five-and-twenty, I should say, with a strong masculine face; but he was exceedingly pale and gave me the impression of a man who was suffering from some strong agitation, which it took all his strength of mind to control.”
The physical injury marks Victor as a victim of physical violence. Watson suggests that Victor is using all his mental capabilities to keep calm about his severe pain. Holmes treats Victor’s mind as he listens to his story: “Pray lie down there and make yourself absolutely at home. Tell us what you can, but stop when you are tired, and keep up your strength with a little stimulant.”
Holmes is a protector, a confidante and a comforter in this scene. He provides Victor with breakfast, induces him to lie down and offers him a stimulant (more than likely brandy).
The extremity of violence that Victor has endured has escalated to mental trauma. In having Holmes treat Victor’s mental trauma while Watson treats his physical pain, Doyle showed the importance psychological support for men of the age.
Holmes was a highly popular character. To contemporary readers, his drug use and dysfunctional clients were seen as markers of his genius rather than a reflection of the significant social issues that men faced during this period. But today, they offer a window into the mental struggles of Victorian men, and a point of connection between readers of the past and present.
Looking for something good? Cut through the noise with a carefully curated selection of the latest releases, live events and exhibitions, straight to your inbox every fortnight, on Fridays.
Sign up here
.
This article features references to books that have been included for editorial reasons, and may contain links to
bookshop.org
. If you click on one of the links and go on to buy something from
bookshop.org
The Conversation UK may earn a commission.
I’m very proud to announce that “Lazy Linearity for a Core Functional
Language”, a paper by myself and
Bernardo
Toninho
, will be published
at
POPL 26
!
The extended version of the paper, which includes all proofs, is available
here [
arXiv
,
PDF
].
The short-ish story
: In 2023, for my Master’s thesis, I reached out to
Arnaud
Spiwack
to discuss how Linear Types had
been implemented in GHC. I wanted to research compiler optimisations made
possible by linearity. Arnaud was quick to tell me:
“
Well yes, but you can’t!“
“Even though Haskell is linearly typed, Core isn’t!”
1
Linearity is ignored in Core because, as soon as it’s optimised,
previously valid linear programs become invalid.
It turns out that traditional linear type systems are too syntactic, or
strict
, about understanding linearity – but Haskell, regardless of linear
types, is lazily evaluated.
Our paper presents a system which, in contrast, also accepts programs that can
only be understood as linear under non-strict evaluation. Including the vast
majority of optimised linear Core programs (with proofs!).
The key ideas of this paper were developed during my Master’s, but it took a
few more years of on-and-off work (supported by my employer
Well-Typed
) with Bernardo to crystalize the understanding of a
“lazy linearity” and strengthen the theoretical results.
Now, the proof of the pudding is in the eating. Go read it!
Abstract
Traditionally, in linearly typed languages, consuming a linear resource is
synonymous with its syntactic occurrence in the program. However, under the
lens of non-strict evaluation, linearity can be further understood
semantically, where a syntactic occurrence of a resource does not necessarily
entail using that resource when the program is executed. While this distinction
has been largely unexplored, it turns out to be inescapable in Haskell’s
optimising compiler, which heavily rewrites the source program in ways that
break syntactic linearity but preserve the program’s semantics. We introduce
Linear Core, a novel system which accepts the lazy semantics of linearity
statically and is suitable for lazy languages such as the Core intermediate
language of the Glasgow Haskell Compiler. We prove that Linear Core is sound,
guaranteeing linear resource usage, and that multiple optimising
transformations preserve linearity in Linear Core while failing to do so in
Core. We have implemented Linear Core as a compiler plugin to validate the
system against linearity-heavy libraries, including linear-base.
Core is the
intermediate compiler language to which source Haskell is desugared and to
which optimisations are applied
↩︎
Keep Talking About Gaza at Your Thanksgiving Table
Intercept
theintercept.com
2025-11-27 09:00:00
The so-called ceasefire might seem like a good excuse to bury the hatchet and enjoy a quieter family dinner, but it’s not.
The post Keep Talking About Gaza at Your Thanksgiving Table appeared first on The Intercept....
Relatives of Palestinians who lost their lives in Israeli attacks that violated the ceasefire in the Gaza Strip mourn at the Aqsa Martyrs Hospital in Deir al-Balah, Gaza, on Nov. 23, 2025.
Photo: Abdalhkem Abu Riash/Anadolu via Getty Images
If Israel’s genocide
in Gaza has been a site of tension in your family for the last two Thanksgiving holidays, this year should be no different. The so-called ceasefire might seem like a good excuse to bury the hatchet and enjoy a quieter turkey dinner, but when we look at the harrowing status quo for Palestinians in Gaza today, there is no peace to be thankful for — especially not on a day that marks the remembrance of this country’s own
genocide against Indigenous Americans
.
To be clear, if two years of livestreamed annihilation have failed to shift your loved ones’ support away from the Israeli ethnostate, I doubt there is anything a dinner table argument could do to persuade them. There can be no reasoning with a
worldview
that forecloses seeing Palestinians as fully human.
I navigate this with pro-Israel members of my own British Jewish family. It’s painful, and I don’t have any good advice. Whatever your approach with your family, there can be no pretense that the genocide in Gaza is over.
I’ll be thinking of another family this Thanksgiving: that of my student from Gaza.
Families like mine, divided over Israel, are not the important ones here. For my part, I’ll be thinking instead of another family this Thanksgiving: that of my student from Gaza. He escaped in 2024 after Israel bombed his home, killing two of his immediate family members, including his mother. His surviving family are still there, living in tents. He hasn’t heard from them in over two weeks.
It is for families like my student’s that we cannot simply take it easy this Thanksgiving because of the so-called ceasefire in Gaza.
Unending Destruction
While the October 10 agreement has offered some relief for Palestinians, with a significant drop in daily slaughter, displacement, starvation and killings by Israeli forces continue.
Instead
of relentless, Israel’s bombings over the last 45 days have been simply ongoing and regular. Israel has
killed
345 Palestinians in Gaza, including 120 children, while demolishing over 1,500 structures.
At the same time, only a fraction of the aid trucks which were supposed to enter Gaza daily under the ceasefire agreement have been permitted entry by Israeli forces. Mass, enforced hunger continues in the Strip, where 50 million tons of
rubble
sits atop well over 10,000 unrecovered bodies.
In the face of such totalizing and unending destruction, it’s hard to find much solace in the fact that the support for the Palestinian cause has
grown
internationally; that nearly all major international human rights organizations have
recognized
Israel’s actions as
genocidal
; that a major
wave
of nation-states, including France, Canada, and Britain, moved this year to
recognize the state of Palestine
. The dead, displaced, and occupied can do little with declarations that carry no concrete consequences.
“What we need is a justice plan,” Mosab Abu Toha, the Palestinian writer and poet,
told
a U.N. meeting this week. “It is time to stop accepting the illusion of peace processes that only entrench injustices.”
With the state of the world as it stands, it feels unlikely that Israeli leaders will be held
accountable for their war crimes
any time soon. Justice for Palestine is hard to imagine, but we can continue to apply pressure in ways that have already seen paradigms shift. Zohran Mamdani’s victory in the New York City mayoral election was a
genuine victory
against the perverse weaponization of antisemitism against Israel’s critics. Now New Yorkers must push our next mayor to uphold commitments to Palestinian solidarity and international law.
And there is more those of us living in safety can do. We can send funds and share resources, as so many already do. And we can continue heading and supporting Palestinians’ call for boycotts, divestment, and sanctions against Israeli institutions complicit in occupation and apartheid.
Activist sometimes say, “Solidarity begins at home.” Yet not everyone can choose their home. If you have the great fortune of spending the holidays with loved ones who share your commitments to justice and liberation, I hope your time together is full of joy. Most of the time, though, solidarity actually begins anywhere but home. So if you choose to spend time with your family knowing that it will be fraught, I wish you luck. The weekend will pass, and there’s urgent work to be done.
DNS Firewalling with MISP and Technitium DNS Server
Disclaimer: the demos on this page use WebGL features that aren’t available on some mobile devices.
A couple of weeks ago I tweeted a video of a toy graphics project (below). It’s not done, but a lot of people liked it which was surprising and fun! A few people asked how it works, so that’s what this post is about.
Under the hood it uses something called a distance field. A distance field is an image like the one below that tells you how far each pixel is from your shape. Light grey pixels are close to the shape and dark grey pixels are far from it.
When the demo starts up, it draws some text on a 2D canvas and generates a distance field of it. It uses
a library I wrote
that generates distance fields really quickly. If you’re curious how the library works, I wrote about that
here
.
Our lighting scheme works like this: when processing a particular pixel we consider a ray from it to the light, like so…
If the ray intersects a glyph, the pixel we’re shading must be in shadow because there’s something between it and the light.
The simplest way to check this would be to move along the ray in 1px increments, starting from the pixel we’re shading and ending at the light, repeatedly asking the distance field if we’re distance 0 from a shape. This would work, but it’d be really slow.
We could pick some specific length like 30px and move in increments of that size, but then we risk jumping over glyphs that are smaller than 30px. We might think we’re not in shadow when we should be.
Ray marching’s core idea is this: the distance field tells you how far you are from the closest glyph. You can safely advance along your ray by that distance without skipping over any glyphs.
Let’s walk through an example. We start as pictured above and ask the distance field how far we are from any glyph. Turns out in this case that the answer is 95px (pictured left). This means that we can move 95px along our ray without skipping over anything!
Now we’re a little closer to the light. We repeat the process until we hit the ascender of the b! If the b glyph weren’t there, we’d have kept going until we hit the light.
Below is a demo that shows the ray marching steps for a given pixel. The red box is the pixel we’re shading, and each circle along the ray represents a ray marching step and the distance from the scene at that step.
Try dragging the light and the pixel around to build an intuition for it.
Below is GLSL to implement this technique. It assumes you’ve defined a function
getDistance
that samples the distance field.
vec2rayOrigin=...;vec2rayDirection=...;floatrayProgress=0;while(true){if(rayProgress>distance(rayOrigin,lightPosition)){// We hit the light! This pixel is not in shadow.return1.;}floatsceneDist=getDistance(rayOrigin+rayProgress*rayDirection);if(sceneDist<=0.){// We hit a shape! This pixel is in shadow.return0.;}rayProgress+=sceneDist;}
It turns out that some pixels are really expensive to process. So in practice we use a for-loop instead of a while loop – that way we bail out if we’ve done too many steps. A common “slow case” in ray marching is when a ray is parallel to the edge of a shape in the scene…
The approach I’ve described so far will get you a scene that looks like the one below.
It’s cool, but the shadows are sharp which doesn’t look very good. The shadows in the demo look more like this…
One big disclaimer is that they’re not physically realistic! Real shadows look like hard shadows where the edges have been fuzzed. This approach does something slightly different: all pixels that were previously in shadow are still fully in shadow. We’ve just added a penumbra of partially shaded pixels around them.
The upside is that they’re pretty and fast to compute, and that’s what I care about! There are three “rules” involved in computing them.
Rule 1:
The closer a ray gets to intersecting a shape, the more its pixel should be shadowed. In the image below there are two similar rays (their distances to the shape pictured in yellow and green). We want the one that gets closer to touching the corner to be more shadowed.
This is cheap to compute because the variable
sceneDist
tells us how far we are from the closest shape at each ray marching step. So the smallest value of
sceneDist
across all steps is a good approximation for the yellow and green lines in the image above.
Rule 2:
if the pixel we’re shading is far from the point where it almost intersects a shape, we want the shadow to spread out more.
Consider two pixels along the ray above. One is closer to the almost-intersection and is lighter (its distance is the green line). The other is farther and darker (its distance is the yellow line). In general: the further a pixel is from its almost intersection, the more “in shadow” we should make it.
This is cheap to compute because the variable
rayProgress
is the length of the green and yellow lines in the image above.
So: we previously returned
1.0
for pixels that weren’t in shadow. To implement rules 1 and 2, we compute
sceneDist / rayProgress
on each ray marching step, keep track of its minimum value, and return that instead.
vec2rayOrigin=...;vec2rayDirection=...;floatrayProgress=0.;floatstopAt=distance(samplePt,lightPosition);floatlightContribution=1.;for(inti=0;i<64;i++){if(rayProgress>stopAt){returnlightContribution;}// `getDistance` samples our distance field texture.floatsceneDist=getDistance(rayOrigin+rayProgress*rayDirection);if(sceneDist<=0.){// We hit a shape! This pixel is in shadow.return0.;}lightContribution=min(lightContribution,sceneDist/rayProgress);rayProgress+=sceneDist;}// Ray-marching took more than 64 steps!return0.;
This ratio feels kind of magical to me because it doesn’t correspond to any physical value. So let’s build some intuition for it by thinking through why it might take on particular values…
If
sceneDist / rayProgress >= 1
, then either
sceneDist
is big or
rayProgress
is small (relative to each other). In the former case we’re far from any shapes and we shouldn’t be in shadow, so a light value of
1
makes sense. In the latter case, the pixel we’re shadowing is really close to an object casting a shadow and the shadow isn’t fuzzy yet, so a light value of
1
makes sense.
The ratio is
0
only when
sceneDist
is
0
. This corresponds to rays that intersect an object and whose pixels are in shadow.
And here’s a demo of what we have so far…
Rule #3
is the most straightforward one: light gets weaker the further you get from it.
Instead of returning the minimum value of
sceneDist / rayProgress
verbatim, we multiply it by a
distanceFactor
which is
1
right next to the light,
0
far away from it, and gets quadratically smaller as you move away from it.
All together, the code for the approach so far looks like this…
vec2rayOrigin=...;vec2rayDirection=...;floatrayProgress=0.;floatstopAt=distance(samplePt,lightPosition);floatlightContribution=1.;for(inti=0;i<64;i++){if(rayProgress>stopAt){// We hit the light!floatLIGHT_RADIUS_PX=800.;// fadeRatio is 1.0 next to the light and 0. at// LIGHT_RADIUS_PX away.floatfadeRatio=1.0-clamp(stopAt/LIGHT_RADIUS_PX,0.,1.);// We'd like the light to fade off quadratically instead of// linearly.floatdistanceFactor=pow(fadeRatio,2.);returnlightContribution*distanceFactor;}// `getDistance` samples our distance field texture.floatsceneDist=getDistance(rayOrigin+rayProgress*rayDirection);if(sceneDist<=0.){// We hit a shape! This pixel is in shadow.return0.;}lightContribution=min(lightContribution,sceneDist/rayProgress);rayProgress+=sceneDist;}// Ray-marching took more than 64 steps!return0.;
I forget where I found this soft-shadow technique, but I definitely didn’t invent it. Inigo Quilez
has a great post on it
where he talks about using it in 3D.
Inigo’s post also talks about a gotcha with this approach that you might have noticed in the demos above: it causes banding artifacts. This is because Rule 1 assumes that the smallest value of
sceneDist
across all steps is a good approximation for the distance from a ray to the scene. This is not always true because we sometimes take very few ray marching steps.
So in my demo I use an improved approximation that Inigo writes about in his post. I also use another trick that is more effective but less performant: instead of advancing by
sceneDist
on each ray marching step, I advance by something like
sceneDist * randomJitter
where
randomJitter
is between
0
and
1
.
This improves the approximation because we’re adding more steps to our ray march. But we could do that by advancing by
sceneDist * .3
. The random jitter ensures that pixels next to each other don’t end up in the same band. This makes the result a little grainy which isn’t great. But I think looks better than banding… This is an aspect of the demo that I’m still not satisfied with, so if you have ideas for how to improve it please tell me!
Overall my demo has a few extra tweaks that I might write about in future but this is the core of it. Thanks for reading! If you have questions or comments, let me know
on Twitter
.
Thank you to Jessica Liu, Susan Wang, Matt Nichols and Kenrick Rilee for giving feedback on early drafts of this post! Also, if you enjoyed this post you might enjoy working with me at
Figma
!
Teaching first-year university students or high schoolers to use a Unix shell
is not always the easiest or most entertaining of tasks. GameShell was devised
as a tool to help students at the
Université Savoie Mont Blanc
to engage with a
real
shell, in a way that encourages learning while also having fun.
The original idea, due to Rodolphe Lepigre, was to run a standard bash session
with an appropriate configuration file that defined "missions" which would be
"checked" in order to progress through the game.
Here is the result...
GameShell is available in English, French and Italian.
Feel free to send us your remarks, questions or suggestions by opening
issues
or submitting
pull requests
.
We are particularly interested in any new missions you might create!
Getting started
GameShell should work on any standard Linux system, and also on macOS and BSD
(but we have run fewer tests on the latter systems). On Debian or Ubuntu, the
only dependencies (besides
bash
) are the
gettext-base
and
awk
packages
(the latter is generally installed by default). Some missions have additional
dependencies: these missions will be skipped if the dependencies are not met.
On Debian or Ubuntu, run the following command to install all game and mission
dependencies.
The first command will download the latest version of the game in the form of
a self-extracting archive, and the second command will initialise and run the
game from the downloaded archive. Instructions on how to play are provided in
the game directly.
Note that when you quit the game (with
control-d
or the command
gsh exit
)
your progression will be saved in a new archive (called
gameshell-save.sh
).
Run this archive to resume the game where you left it.
If you prefer not running foreign shell scripts on your computer, you can
generate a Docker image with the following:
The game will NOT be saved when you exit, and additional flags are required if
you want to run X programs from inside GameShell. Refer to
this
section
of the user
manual.
Documentation
To find out more about GameShell, refer to the following documents:
The
user manual
provides information on how to run the
game on all supported platforms (Linux, macOS, BSD), explains how to run the
game from the sources, tells you how to generate custom game archives (which
is useful if you want to use GameShell for teaching a class), and more.
The
developer manual
provides information on how to
create new missions, how to translate missions, and how to participate
in the development of the game.
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
What can I do to resolve this?
You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.
Sadly, we have to inform you that this is the last issue of “ECMAScript News”. We have been operating at a loss for too long: The number of advertisers and subscribers has been slowly but steadily decreasing over the last two years (vs. constant growth before that). Therefore, we made the difficult decision to stop publishing this newsletter.
The first issue came out on 2016-09-27. We published a total of 368 issues and are thankful for many loyal readers during many interesting years!
Axel may continue this newsletter in some shape or form next year. If he does, he’ll inform you via one last email in 2026.
A unification of Buddhist phenomenology, active inference, and physical reflexes; a practical theory of suffering, tension, and liberation; the core mechanism for medium-term memory and Bayesian updating; a clinically useful dimension of variation and dysfunction; a description of sensory type safety; a celebration of biological life.
Michael Edward Johnson, Symmetry Institute, July 12, 2023.
I. What is tanha?
By default, the brain tries to grasp and hold onto pleasant sensations and push away unpleasant ones. The Buddha called these ‘micro-motions’ of greed and aversion
taṇhā
, and the Buddhist consensus seems to be that it accounts for an amazingly large proportion (~90%) of suffering.
Romeo Stevens
suggests translating the original Pali term as “fused to,” “grasping,” or “clenching,” and that the mind is trying to make sensations feel
stable, satisfactory, and controllable
. Nick Cammarata suggests “
fast grabby thing
” that happens within ~100ms after a sensation enters awareness; Daniel Ingram suggests this ‘grab’ can occur as quickly as 25-50ms (personal discussion). Uchiyama Roshi describes tanha in terms of its cure, “
opening the hand of thought
”; Shinzen Young suggests “
fixation
”; other common translations of tanha are “
desire
,” “thirst,” “craving.” The vipassana doctrine is that tanha is something the mind instinctively does, and that meditation helps you
see this process
as it happens, which allows you to stop doing it. Shinzen estimates that his conscious experience is
literally 10x better
due to having a satisfying meditation practice.
Tanha is not yet a topic of study in affective neuroscience but I suggest it should be. Neuroscience is generally gated by soluble important mysteries: complex dynamics often arise from complex mechanisms, and complex mechanisms are difficult to untangle. The treasures in neuroscience happen when we find
exceptions
to this rule: complex dynamics that arise from elegantly simple core mechanisms. When we find one it generally leads to breakthroughs in both theory and intervention. Does “tanha” arise from a simple or complex mechanism? I believe
Buddhist phenomenology
is very careful about what it calls
dependent origination
— and this makes items that Buddhist scholarship considers to be ‘basic building-blocks of phenomenology’ particularly likely to have a simple, elegant implementations in the brain — and thus are exceptional mysteries to focus scientific attention on.
I don’t think tanha has 1000 contributing factors; I think it has one crisp, isolatable factor. And I think if we find this factor, it could herald a reorganization of systems neuroscience similar in magnitude to the past shifts of
cybernetics
, predictive coding, and
active inference
.
The first clue is what tanha is trying to do for us. I’ll claim today that tanha is a side-effect of a normal, effective strategy our brains use extensively,
active inference
. Active inference suggests we impel ourselves to action by first creating some predicted sensation (“I have a sweet taste in my mouth” or “I am not standing near that dangerous-looking man”) and then holding it until we act in the world to make this prediction
become true
(at which point we can release the tension). Active inference argues
we store our to-do list as predictions
, which are equivalent to untrue sensory observations that we act to make true.
Formally, the “tanha as unskillful active inference” (TUAI) hypothesis is that this process commonly goes awry (i.e. is applied unskillfully) in three ways:
First, the rate of generating normative predictions can outpace our ability to make them true and overloads a very finite system. Basically we try to control too much, and stress builds up.
Second, we generate normative predictions in domains that we
cannot possibly control
; predicting a taste of cake will linger in our mouth forever, predicting that we did not drop our glass of water on the floor. That good sensations will last forever and the bad did not happen. (This is essentially a “predictive processing” reframe of the story Romeo Stevens has told on his
blog
,
Twitter
, and in person.)[1]
Third, there may be a
context
desynchronization
between the system that represents the world model, and the system that maintains predictions-as-operators on this world model. When
desynchronization
happens and the
basis
of the world model shifts in relation to the basis of the predictions, predictions become nonspecific or
nonsensical
noise and stress.
We may also include a catch-all fourth category for when the prediction machinery becomes altered outside of any semantic context, for example metabolic insufficiency leading to impaired operation.
Core resources:
Safron, A. (2020).
An Integrated World Modeling Theory (IWMT) of Consciousness
: Combining Integrated Information and Global Neuronal Workspace Theories With the Free Energy Principle and Active Inference Framework; Toward Solving the Hard Problem and Characterizing Agentic Causation. Frontiers in Artificial Intelligence, 3. https://doi.org/10.3389/frai.2020.00030
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., Pezzulo, G. (2017).
Active inference: A Process Theory
. Neural Computation, 29(1), 1-49.
At high frequencies these SOHMs will act as feature detectors, at lower frequencies we might think of them as wind chimes: by the presence and absence of particular SOHMs and their interactions we obtain a subconscious feeling about what kind of environment we’re in and where its rewards and dangers are. We can expect SOHMs will be arranged in a way that optimizes differentiability of possible/likely world states,
minimizes crosstalk
, and in aggregate constitutes a
world model
, or in the
Neural Annealing
/
REBUS
/
ALBUS
framework, a belief landscape.
To be in tanha-free “
open awareness
” without greed, aversion, or expectation is to feel the undoctored hum of your SOHMs. However, we doctor our SOHMs *all the time* — when a nice sensation enters our awareness, we reflexively try to ‘grab’ it and stabilize the resonance; when something unpleasant comes in, we try to push away and deaden the resonance. Likewise society puts expectations on us to “
act normal
” and “
be useful
”; we may consider all such SOHM adjustments/predictions as drawing from the same finite resource pool. “Active SOHM management” is effortful (and unpleasant) in rough proportion to how many SOHMs need to be actively managed and how long they need to be managed.
But how can the brain manage SOHMs? And if the Buddhists are right and this creates suffering, why does the brain even try?
Core resources:
Safron, A. (2020).
An Integrated World Modeling Theory (IWMT) of Consciousness
: Combining Integrated Information and Global Neuronal Workspace Theories With the Free Energy Principle and Active Inference Framework; Toward Solving the Hard Problem and Characterizing Agentic Causation. Frontiers in Artificial Intelligence, 3. https://doi.org/10.3389/frai.2020.00030
I propose reframing tanha as an artifact of the brain’s
compression pressure
. I.e. tanha is an artifact of a continual process that subtly but systematically pushes on the complexity of ‘what is’ (the neural patterns represented by undoctored SOHMs) to collapse it into a more simple configuration, and sometimes holds it there until we act to make that simplification true. The result of this compression drive conflates “what is”, “what could be”, “what should be”, and “what will be,” and this conflation is the source of no end of moral and epistemological confusion.
This reframes tanha as both the pressure which collapses complexity into simplicity, and the ongoing stress that comes from maintaining the counterfactual aspects of this collapse (
compression stress
). We can think of this process as balancing two costs: on one hand, applying compression pressure has metabolic and epistemic costs, both immediate and ongoing. On the other hand, the brain is a finite system and if it doesn’t continually “compress away” patterns there will be unmanageable sensory chaos. The right amount of compression pressure is not zero.[2]
Equivalently, we can consider tanha as an excessive forcefulness in the metabolization of uncertainty. Erik P. Hoel has written about energy, information, and uncertainty as equivalent and conserved quantities (
Hoel 2020
):
much like literal digestion
, the imperative of the nervous system is to extract value from sensations then excrete the remaining information, leaving a low-information, low-uncertainty, clean slate ready for the next sensation (thank you Benjamin Anderson for discussion). However, we are often unskillful in the ways we try to extract value from sensations, e.g. improperly assessing context, trying to extract too much or too little certainty, or trying to extract forms of certainty inappropriate for the sensation.
We can define a person’s personality, aesthetic, and a large part of their phenomenology in terms of how they metabolize uncertainty — their library of motifs for (a) initial probing, (b) digestion and integration, and (c) excretion/externalization of any waste products, and the particular reagents for this process they can’t give themselves and
must
seek in the world
.
So far we’ve been discussing brain dynamics on the computational level. But
how
does the brain do all this — what is the
mechanism
by which it attempts to apply compression pressure to SOHMs? This is essentially the question neuroscience has been asking for the last decade. I believe evolution has coupled two very different systems together to selectively apply compression/prediction pressure in a way that preserves the perceptive reliability of the underlying system (undoctored SOHMs as ground-truth perception) but allows near-infinite capacity for adjustment and hypotheticals. One system focused on perception; one on compression, judgment, planning, and action.
The traditional neuroscience approach for locating these executive functions has been to associate them with particular areas of the brain. I suspect the core logic is hiding much closer to the action.
Above: the vertical section of an artery wall (Wikipedia, emphasis added;
video
): the physical mechanism by which we grab sensations and make predictions; the proximate cause of 90% of suffering and 90% of goal-directed behavior.
All blood vessels are wrapped by a thin sheathe of vascular smooth muscle cells (
VSMCs
). The current scientific consensus has the vasculature system as a spiderweb of ever-narrower channels for blood, powered by the heart as a central pump, and supporting systems such as the brain, stomach, limbs, and so on by bringing them nutrients and taking away waste. The sheathe of muscle wrapped around blood vessels undulates in a process called “
vasomotion
” that we think helps blood keep circulating, much like peristalsis in the gut helps keep food moving, and can help adjust blood pressure.
I think all this is true, but is also a product of what’s been easy to measure and misses 90% of what these cells do.
Evolution works in layers, and the most ancient base layers often have rudimentary versions of more specialized capacities (
Levin 2022
) as well as deep control hooks into newer systems that are built around them. The vascular system actually predates neurons and has co-evolved with the nervous system for hundreds of millions of years. It also has mechanical actuators (VSMCs) that have physical access to all parts of the body and can flex in arbitrary patterns and rhythms. It would be extremely surprising if evolution didn’t use this system for something more than plumbing. We can also “follow the money”; the vascular system controls the nutrients and waste disposal for the neural system and will win in any heads-up competition over co-regulation balance.
I expect VSMC contractions to influence nearby neurons through e.g. ephaptic coupling, reducing blood flow, and adjusting local physical resonance, and to be triggered by local dissonance in the electromagnetic field.
I’ll offer three related hypotheses about the computational role of VSMCs[3] today that in aggregate constitute a neural regulatory paradigm I’m calling
vasocomputation
:
Compressive Vasomotion Hypothesis (CVH)
: the vasomotion reflex functions as a compression sweep on nearby neural resonances, collapsing and merging fragile ambivalent patterns (the “Bayesian blur” problem) into a more durable, definite state. Motifs of vasomotion, reflexive reactions to uncertainties, and patterns of tanha are equivalent.
Vascular Clamp Hypothesis (VCH)
: vascular contractions freeze local neural patterns and plasticity for the duration of the contraction, similar to collapsing a superposition or probability distribution, clamping a harmonic system, or pinching a critical network into a definite circuit. Specific vascular constrictions correspond with specific predictions within the Active Inference framework and function as medium-term memory.
Latched Hyperprior Hypothesis (LHH)
: if a vascular contraction is held long enough, it will engage the
latch-bridge mechanism
common to smooth muscle cells. This will durably ‘freeze’ the nearby circuit, isolating it from conscious experience and global updating and leading to a much-reduced dynamical repertoire; essentially creating a durable commitment to a specific hyperprior. The local vasculature will unlatch once the prediction the latch corresponds to is resolved, restoring the ability of the nearby neural networks to support a larger superposition of possibilities.
The initial contractive sweep jostles the neural superposition of interpretations into specificity; the contracted state temporarily freezes the result; if the contraction is sustained, the latch bridge mechanism engages and cements this freeze as a hyperprior. With one motion the door of possibility slams shut. And so we collapse our world into something less magical but more manageable, one clench at a time.
Tanha is cringe.
The claim relevant to the
Free Energy Principle – Active Inference
paradigm is we can productively understand the motifs of smooth muscle cells (particularly in the vascular system) as “where the brain’s top-down predictive models are hiding,” which has been an open mystery in FEP-AI. Specific predictions are held as vascular tension, and vascular tension in turn is released by action, consolidated by
Neural Annealing
, or rendered superfluous by neural remodeling (hold a pattern in place long enough and it becomes the default). Phrased in terms of the
Deep CANALs
framework which imports ideas from machine learning: the neural weights that give rise to SOHMs constitute the learning landscape, and SOHMs+vascular tension constitute the inference landscape.
The claim relevant to Theravada Buddhism is we can productively understand the motifs of the vascular system as the means by which we attempt to manipulate our sensations. Vasomotion corresponds to an attempt to ‘pin down’ a sensation (i.e. tanha); muscle contractions freeze patterns; smooth muscle latches block out feelings of possibility and awareness of that somatic area. Progress on the contemplative path will correspond with both using these forms of tension less, and needing them less. I expect cessations to correspond with a nigh-complete absence of vasomotion (and EEG may measure vasomotion moreso than neural activity).
The claim relevant to practical health is that smooth muscle tension, especially in VSMCs, and especially latched tension, is a system science knows relatively little about but is involved in an incredibly wide range of problems, and understanding this system is hugely helpful for knowing how to take care of yourself and others. The “latch-bridge” mechanism is especially important, where smooth muscle cells have a discrete state where they attach their myosin heads to actin in a way that “locks” or “latches” the tension without requiring ongoing energy. Latches take between seconds to minutes to form & dissolve — a simple way to experience the latch-bridge cycle releasing is to have a hot bath and notice waves of muscle relaxation. Latches can persist for minutes, hours, days, months, or years (depending on what prediction they’re stabilizing), and the sum total of all latches likely accounts for the majority of bodily suffering. If you are “holding tension in your body” you are subject to the mechanics of the latch-bridge mechanism. Migraines and cluster headaches are almost certainly inappropriate VSMC latches; all hollow organs are surrounded by smooth muscle and can latch. A long-term diet of poor food (e.g. seed oils) leads to random latch formation and “lumpy” phenomenology. Sauna + cold plunges are an effective way to force the clench-release cycle and release latches; likewise, simply taking time to feel your body and put your attention into latched tissues can release them. Psychedelics can force open latches. Many issues in neuropathy & psychiatry are likely due to what I call “latch spirals” — a latch forms, which reduces blood flow to that area, which reduces energy available to those tissues, which prevents the latch from releasing (since releasing the latch requires activation energy and returning to a freely cycling state also increases the cell’s rate of energy expenditure).
To summarize the story so far
: tanha is a grabby reflex which is the source of most moment-by-moment suffering. The ‘tanha as unskillful active inference’ (TUAI) hypothesis suggests that we can think of this “grabbing” as part of the brain’s normal predictive and compressive sensemaking, but by default it makes many unskillful predictions that can’t possibly come true and must hold in a costly way. The vascular clamp hypothesis (VCH) is that we store these predictions (both skillful and unskillful) in vascular tension. The VCH can be divided into three distinct hypotheses (CVH, VCH, LHH) that describe the role of this reflex at different computational and temporal scales. An important and non-obvious aspect of smooth muscle (e.g. VSMCs) is they have a discrete “latch” setting wherein energy usage and flexibility drops significantly, and sometimes these latches are overly ‘sticky’; unlatching our sticky latches is a core part of the human condition.
Concluding Part I
: the above work describes a bridge between three distinct levels of abstraction: a central element in Buddhist phenomenology, the core accounting system within active inference, and a specific muscular reflex. I think this may offer a functional route to synthesize the FEP-AI paradigm and Michael Levin’s distributed stress minimization work, and in future posts I plan to explore why this mechanism has been overlooked, and how its dynamics are intimately connected with human problems and capacities.
I view this research program as integral to both human flourishing and AI alignment.
Acknowledgements
: This work owes a great deal to Romeo Stevens’
scholarship on tanha
, pioneering tanha as a ‘clench’ dynamic, intuitions about
muscle tension and prediction
, and notion that we commit to dukkha ourselves until we get what we want; Nick Cammarata’s
fresh perspectives
on Buddhism and his tireless and generative inquiry around the phenomenology & timescale of
tanha
; Justin Mares’ gentle and persistent encouragement; Andrea Bortolameazzi’s many thoughtful comments and observations about the path, critical feedback, and thoughtful support; and Adam Safron’s steadfast belief and support, theorizing on SOHMs, and teachings about predictive coding and active inference. Much of my knowledge of Buddhist psychology comes from the work and teachings of Anthony Markwell; much of my intuition around tantra and interpersonal embodiment dynamics comes from Elena Selezneva. I’m also grateful for conversations with Benjamin Anderson about emergence, to Curran Janssens for supporting my research, and to Ivanna Evtukhova for starting me on the contemplative path. An evergreen thank you to my parents their unconditional support. Finally, a big thank-you to Janine Leger and Vitalik Buterin’s Zuzalu co-living community for creating a space to work on this writeup and make it real.
Footnotes
:
[1] We might attempt to decompose the Active Inference – FEP term of ‘precision weighting’ as (1) the amount of
sensory clarity
(the amount of precision available in stimuli), and (2) the amount of ‘grabbiness’ of the compression system (the amount of precision we empirically try to extract). Perhaps we could begin to put numbers on tanha by calculating the
KL divergence
between these distributions.
[2] We can speculate that the arrow of compression points away from Buddhism’s three attributes: e.g. the brain tries to push and prod its SOHMs toward patterns that are stable (dissonance minimization), satisfactory (harmony maximization), and controllable (compression maximization) — similar yet subtly distinct targets. Thanks to both Romeo and Andrea for discussion about the three attributes and their opposite.
[3] (Added July 19, 2023) Skeletal muscle, smooth muscle, and fascia (which contains myofibroblasts with actin fibers similar to those in muscles) are all found throughout the body and reflexively distribute physical load; it’s likely they do the same for cognitive-emotional load. Why focus on VSMCs in particular? Three reasons: (1) they have the best physical access to neurons, (2) they regulate bloodflow, and (3) they have the latch-bridge mechanism. I.e. skeletal, non-VSMC smooth muscle, and fascia all likely contribute significantly to
distributed stress minimization
, and perhaps do so via similar principles/heuristics, but VSMCs seem to be the only muscle with means, motive, and opportunity to finely puppet the neural system, and I believe are indispensably integrated with its moment-by-moment operation in more ways than are other contractive cells. (Thanks to
@askyatharth
for bringing up fascia.)
Edit, April 6th, 2025:
a friendly Buddhist scholar suggests that common translations of taṇhā conflate two concepts: taṇhā in Pali is most accurately translated as craving or thirst, whereas the act of clinging itself is “upādāna (as in the upādāna-khandhās), and in the links of dependent origination is one step downstream from the thirst (or impulsive craving) of taṇhā.” Under this view we can frame taṇhā as a particular default bias in the
computational-biochemical tuning
of the human nervous system, and upādāna as the impulsive physical (VSMC) clenching this leads to.
Buddhism describes taṇhā as being driven by the three fundamental defilements, greed, fear, & delusion; I expect each defilement maps to a hard truth (aka clearly suboptimal but understandable failure mode) of implementing vasocomputation-based active inference systems.
Sanders, Warren Help Form Senate Democratic ‘Fight Club’ Challenging Schumer’s Leadership
Portside
portside.org
2025-11-27 05:49:27
Sanders, Warren Help Form Senate Democratic ‘Fight Club’ Challenging Schumer’s Leadership
Mark Brody
Thu, 11/27/2025 - 00:49
...
Sanders, Warren Help Form Senate Democratic ‘Fight Club’ Challenging Schumer’s Leadership
Published
Senators Bernie Sanders and Elizabeth Warren | CNN
Angered by the Democratic leadership’s fecklessness and lack of a bold vision for the future, a group of senators including
Bernie Sanders
of
Vermont
and
Elizabeth Warren
of Massachusetts has formed an alliance to push back on Senate Minority Leader
Chuck Schumer
and the party’s campaign arm ahead of next year’s critical
midterm elections
.
The existence of the group, known as the “Fight Club,” was first revealed Monday by the
New York Times
, which
reported
that the senators are pressing the
Democratic Party
to “embrace candidates willing to challenge entrenched corporate interests, fiercely oppose the
Trump administration
, and defy their own party’s orthodoxy.”
Sens. Chris Van Hollen of Maryland, Tina Smith of Minnesota, and
Chris Murphy
of Connecticut are also members of the alliance, and other senators—including
Ed Markey
of Massachusetts and
Jeff Merkley
of Oregon—have taken part in group actions, according to the
Times
.
“The coalition of at least half a dozen senators... is unhappy with how Mr. Schumer and his fellow senator from New York,
Kirsten Gillibrand
, the head of Senate Democrats’ campaign arm, have chosen, recruited and, they argue, favored candidates aligned with the establishment,” the newspaper reported. “The party’s campaign arm, the Democratic Senatorial Campaign Committee, has not made any formal endorsements in contested primaries. However, the senators are convinced that it is quietly signaling support for and pushing donors toward specific Senate candidates: Rep. Angie Craig in Minnesota, Rep. Haley Stevens in Michigan, and Gov. Janet Mills in
Maine
.”
Members of the “Fight Club” have endorsed Minnesota Lt. Gov. Peggy Flanagan’s bid for
US Senate
. In addition to Flanagan, Sanders has backed Abdul El-Sayed’s US Senate run in Michigan and Graham Platner’s campaign to unseat Republican Sen.
Susan Collins
in Maine.
News of the “Fight Club” alliance comes after a small group of centrist Democrats, with Schumer’s tacit blessing,
capitulated
to President
Donald Trump
and
Republicans
earlier this month by agreeing to end the government shutdown without an extension of
Affordable Care Act
subsidies, even as health insurance premiums skyrocket nationwide.
The cave sparked widespread fury, much of it directed at Schumer. Indivisible, a progressive advocacy group that typically aligns with Democrats, has
said
it will not support any Senate Democratic primary candidate who does not call on Schumer to step down as minority leader.
“We must turn the page on this era of cowardice,” Indivisible said following Senate Democrats’ capitulation. “We must nominate and elect Democratic candidates who have an actual backbone. And we must ensure that the kind of failed leadership we see from Sen. Schumer does not doom a future Democratic majority.”
Thus far, no sitting member of the Senate Democratic caucus has demanded Schumer’s resignation. But the emergence of the “Fight Club” is the latest evidence that the Democratic leader’s support is beginning to crumble.
“Absolutely love to see this,” progressive strategist Robert Cruickshank wrote on
social media
in response to the
Times
reporting. “So glad there are some Senate Dems willing to fight back.”
Jake Johnson is a senior editor and staff writer for Common Dreams.
20 States Sue the Trump Administration Over Cuts to Homeless Permanent Housing Funding
Portside
portside.org
2025-11-27 05:34:55
20 States Sue the Trump Administration Over Cuts to Homeless Permanent Housing Funding
Mark Brody
Thu, 11/27/2025 - 00:34
...
The new conditions placed on the program would also give HUD the ability to deny funding for organizations that acknowledge the existence of transgender or nonbinary individuals.
“Communities across the country depend on Continuum of Care funds to provide housing and other resources to our most vulnerable neighbors,” said James in a press release. “These funds help keep tens of thousands of people from sleeping on the streets every night. I will not allow this administration to cut off these funds and put vital housing and support services at risk.”
The coalition of mainly Democratic-led states argues
in the lawsuit
that HUD’s new conditions on the funding are “unlawful and unconstitutional,” alleging that the administration “cannot impose its own conditions on funds that Congress mandated should be distributed based solely on need.”
The lawsuit accuses the Trump administration of violating the Administrative Procedure Act and Congress’ “constitutional power to control spending.”
“HUD is dismayed that the plaintiffs have chosen to misuse the Courts and pursue this delaying tactic to serve their own personal political agenda at the expense of the homeless individuals, youth and families now living on our Nation’s streets. Their use of the courts for political means seeks to prevent nearly $4 billion of aid to flow nationwide to assist those in need. HUD intends to mount a vigorous defense to this meritless legal action,” the spokesperson said in a statement.
The case was filed in the U.S. District Court for the District of Rhode Island and will be decided by Judge Mary S. McElroy, who was appointed by President Donald Trump in 2019 but first nominated by former President Barack Obama.
Earlier this month, HUD imposed a cap on the amount of program funds that can support permanent housing. Previously, there was not a specific limit and around 90 percent of funds supported permanent housing. Under the new cap, no more than 30 percent of these funds can support permanent housing.
HUD Secretary Scott Turner has argued that the policy change is a necessary shift from what the Trump administration considers to be a failed “housing first” model that prioritizes permanent housing without preconditions, such as getting a job or seeking treatment. The agency has said the current policy has fueled a “homeless industrial complex” and does not address the root causes of homelessness.
“What we’ve done is take this Biden-era slush fund, called the Continuum of Care, and turned it into not just housing, but also treatment and transitional housing,” Turner said on Fox Business last week.
The funding cuts could put 170,000 people at risk of experiencing homelessness, according to internal HUD documentation previously obtained by POLITICO. HUD has maintained that the changes will include specific protections for children, veterans and seniors.
Different factions of lawmakers have sent letters to the agency with multiple requests, including extending funding for CoC projects expiring in 2026, reversing the policy changes or answering various questions about implementation.
Additionally, 1,001 national, state and local organizations
sent a letter to Congress
on Monday urging that lawmakers include language directing HUD to renew all existing CoC program grants expiring in 2026 for a full year in the upcoming Transportation-Housing and Urban Development appropriations bill.
A group of 22 House Republicans asked for the same one-year funding extension
in a letter to the agency
earlier this month.
House and Senate Democrats have urged in letters to HUD to rescind the policy change, submit documentation on how the agency will complete the quick application turnaround for housing project funding and extend funding for grants expiring in 2026.
Senate Banking ranking member
Elizabeth Warren
(D-Mass.) said in a statement that Trump’s “draconian changes to the Continuum of Care program could force 170,000 people out of permanent housing and back onto the street. Congress, state leaders, all of us should be pushing back against the Administration’s cruel move that will dramatically exacerbate the homelessness crisis in cities, towns, and suburbs across the country.”
Rep.
Mike Flood
(R-Neb.), chair of the House Financial Services Subcommittee on Housing and Insurance said that while he doesn’t typically discuss pending litigation, he’s “been working with the administration on policy to build more housing, drive housing costs down, and ensure that existing federal funds are spent in a way that rewards success and drives positive results for the American people.”
Other states included as plaintiffs in the lawsuit are Arizona, California, Colorado, Connecticut, Delaware, Illinois, Kentucky, Maine, Maryland, Massachusetts, Michigan, Minnesota, New Jersey, Oregon, Pennsylvania, Rhode Island, Vermont, Washington and Wisconsin, as well as the District of Columbia.
Katherine Hapgood reports on economic and small business policy in Congress at POLITICO.
Russ Allbery: Review: A Matter of Execution
PlanetDebian
www.eyrie.org
2025-11-27 05:34:00
Review: A Matter of Execution, by Nicholas & Olivia Atwater
Series:
Tales of the Iron Rose #0
Publisher:
Starwatch Press
Copyright:
2024
ISBN:
1-998257-08-8
Format:
Kindle
Pages:
131
A Matter of Execution is ...
A Matter of Execution
is the introductory novella that kicked off
the Tales of the Iron Rose series. It is steampunk fantasy with airships.
I previously read and reviewed the subsequent novel,
Echoes of the Imperium
.
As noted in that review, I read the novel first. That was a mistake; this
is a much better place to start.
A Matter of Execution
was clearly
intended as the introduction of all of these characters. More importantly,
I think reading the novella first would have given me enough affinity with
the characters to not mind the worst part of
Echoes of the
Imperium
: the extremely slow first half that seemed filled with the
protagonist's impostor syndrome.
A Matter of Execution
opens, fittingly, with Captain William Blair,
a goblin, former Imperial soldier, Oathbreaker, and series first-person
protagonist being carted to his execution. He is not alone; in the same
prison wagon is an arrogant (and racist) man named Strahl, the killer of
one of the rulers of Lyonesse.
Strahl is rather contemptuous of Blair's claim to be a captain, given that
he's both a goblin and an Oathbreaker. Strahl quickly revises that opinion
when Blair's crew, somewhat predictably given that he is the series
protagonist, creates a daring escape for both of them. The heat of action
gives both a chance to gain some respect for the other, which explains why
Blair is not only willing to invite Strahl to join his crew, but to go
back for Strahl's companion.
Breaking out Strahl's companion will be a more difficult, and surprising,
problem.
Nicholas Atwater is a role-playing game GM, something that you will learn
the "about the author" section at the end of this novella but probably
will have guessed by then. Even more than
Echoes of the Imperium
,
this novella feels like a (good) write-up of an RPG adventure. A wildly
varied cast of characters come together and form a party with a
well-defined objective that has some surrounding mysteries and surprises.
Each of those characters get their individual moments to show off their
specific skills. Readers with a certain gaming background will know
exactly where to insert the
Borderlands
-style title card with a
slightly demented description of each character.
This is not a complaint. You may be able to see the bones of the setup
adventure for a long-running campaign, but I like this style of character
introduction and the story moves right along. There are a ton of varied
characters, some interesting villains and maybe-villains, a rather
satisfying heist setup, and some good chemistry and a bit of banter. This
is not a deep story — it's clearly an introductory episode for both the
characters and the world background — but it's a fun way to spend a few
hours.
I think the best part of this series is the world-building. If you have
read my review of
Echoes of the Imperium
, you have unfortunately
been mildly spoiled for the revelation in this novella. I don't think it
hurt the story that much; you will be able to predict what obvious gaps in
the novel backstory the novella is going to fill in, but it's just as
enjoyable to see how that happens. But the Atwaters aren't going to drop
any of the big world-building bombs in the introductory novella, of
course. Instead, you get a gradual introduction to the nature of magic in
this world, some of the political setup of the recent war, and a quick
introduction to the capabilities of Strahl's mysterious companion.
If you've not yet read this series, I recommend starting here. It's a
quick investment to see if you'll be interested. The novel is heavier and
slower, and the pacing of the first half isn't great, but the
world-building is even better.
If you've already read the novel, this is still worth reading as long as
you enjoyed it. You'll have a few moments of "oh,
that's
how that
happened," and it's a fun and fast-moving way to spend a bit more time
with the characters.
Followed by
Echoes of the Imperium
. The
back matter of the novella says that
The Winds of Fortune
is
supposedly forthcoming.
Rating: 7 out of 10
Reviewed: 2025-11-26
Show HN: Era – Open-source local sandbox for AI agents
Run untrusted or AI-generated code locally inside microVMs that behave like containers for great devX, 200ms launch time, and better security.
There's a fully managed cloud layer, globally deployed Worker/API, jump to
cloudflare/README.md
.
Quick Start
installation options
option 1: homebrew (recommended)
# 1. install the tap
brew tap binsquare/era-agent-cli
# 2. install era agent
brew install binsquare/era-agent-cli/era-agent
# 3. install dependencies
brew install krunvm buildah
# 4. verify the CLI is on PATH
agent vm exec --help
# 4. follow platform-specific setup (see below)
option 2: from source
# 1. install dependencies
brew install krunvm buildah # on macos# 2. clone the repository
git clone https://github.com/binsquare/era
cd era-agent
# 3. build the agent
make
# 4. follow platform-specific setup (see below)
Installation (macOS)
brew tap binsquare/era-agent-cli
brew install era-agent-cli
brew install krunvm buildah
Run the post-install helper to prepare the case-sensitive volume/state dir on macOS:
$(brew --prefix era-agent)/libexec/setup/setup.sh
platform setup details
homebrew installation setup
if you installed era agent via homebrew, use the setup script from the installed location:
# for macos users with homebrew installation$(brew --prefix era-agent)/libexec/setup/setup.sh
# or run the setup script directly after installation$(brew --prefix)/bin/era-agent-setup # if setup script is linked separately
macos setup
Run
scripts/macos/setup.sh
to bootstrap dependencies, validate (or create) a case-sensitive volume, and prepare an agent state directory (the script may prompt for your password to run
diskutil
). The script will also detect your Homebrew installation and recommend the correct value for the
DYLD_LIBRARY_PATH
environment variable, which may be required for
krunvm
to find its dynamic libraries.
If you prefer to create the dedicated volume manually, open a separate terminal and run (with
sudo
as required):
(replace
disk3
with the identifier reported by
diskutil list
). The operation is non-destructive, does not require
sudo
, and shares space with the source container volume.
When prompted by the setup script, accept the default mount point (
/Volumes/krunvm
) or provide your own. Afterwards, export the environment variables printed by the script (at minimum
AGENT_STATE_DIR
,
KRUNVM_DATA_DIR
, and
CONTAINERS_STORAGE_CONF
) before invoking
agent
or running
krunvm
/
buildah
directly. The helper now prepares a matching container-storage configuration under the case-sensitive volume so the CLI can run without extra manual steps.
The script also writes
policy.json
/
registries.conf
under the same directory so Buildah doesn't look for root-owned files in
/etc/containers
. Export the variables it prints (
CONTAINERS_POLICY
,
CONTAINERS_REGISTRIES_CONF
) if you invoke Buildah manually.
Linux Setup
Install
krunvm
and
buildah
using your package manager (the specific installation method may vary)
Ensure the system is properly configured to run microVMs (may require kernel modules or specific privileges)
Consider setting
AGENT_STATE_DIR
to a writable location if running as non-root
Runtime Requirements
krunvm
must be installed and available on
$PATH
(Homebrew:
brew install krunvm
; see upstream docs for other platforms).
buildah
must also be present because
krunvm
shells out to it for OCI image handling.
On macOS,
krunvm
requires a case-sensitive APFS volume; see the macOS setup notes above.
Build
make # builds the agent CLI
make clean # removes build artifacts (Go cache)
Full platform-specific steps (macOS volume setup, Linux env vars, troubleshooting) live in
era-agent/README.md
.
🎥 Demo Video
A demo video showing how to install and use the CLI tool is available in the
era-agent directory
. This video covers:
Installing dependencies and compiling the CLI tool
Creating and accessing local VMs
Running code and agents through commands or scripts
Uploading and downloading files to/from a VM
Core Commands
# create a long-running VM
agent vm create --language python --cpu 1 --mem 256 --network allow_all
# run something inside it
agent vm exec --vm <id> --cmd "python -c 'print(\"hi\")'"# ephemeral one-off execution
agent vm temp --language javascript --cmd "node -e 'console.log(42)'"# inspect / cleanup
agent vm list
agent vm stop --all
agent vm clean --all
Supported
--language
values:
python
,
javascript
/
node
/
typescript
,
go
,
ruby
. Override the base image with
--image
if you need a custom runtime.
⚙ Configuration Highlights
AGENT_STATE_DIR
: writable directory for VM metadata, krunvm state, and Buildah storage. The macOS setup script prints the correct exports.
AGENT_LOG_LEVEL
(
debug|info|warn|error
) and
AGENT_LOG_FILE
: control logging.
In New York, one of the toughest challenges that Mayor-elect Zohran Mamdani faces will be preserving and increasing the supply of affordable housing. Same story in Boston, where I live and where our progressive mayor, Michelle Wu, is constrained by similar forces.
The immediate obstacles are a scarcity of buildable land and subsidy dollars. In both cities, higher taxes to support more housing requires the approval of state government.
New York has a form of rent control, known as rent stabilization, but most New Yorkers do not live in rent-stabilized apartments. Boston once had rent control, but the state legislature took it away in 1994. Local option rent control will be back before voters next year via a ballot initiative.
But behind all of these challenges is the sheer political power of developers. Let me give a couple of emblematic examples
Thirty years ago, Boston had massive tracts of vacant developable land in a part of the waterfront that was a jumble of parking lots, warehouses, and piers. It had not been developed partly because its ownership was patchwork, and partly because Boston was still emerging from a prolonged recession.
The land area totaled about 1,000 acres, only slightly less than Boston’s entire historic downtown. It represented the city’s last large-scale building opportunity.
The Boston Redevelopment Authority (BRA) gradually got control of the land, rebranded it as the Seaport District, then as the Innovation District, and in 1999 began working with private developers to create a whole new section of the city with hotels, office buildings, restaurants, and luxury housing. Number of affordable housing units: fewer than 500.
Why? Because the BRA and the two mayors of that era, Tom Menino (1993–2014) and Marty Walsh (2014–2021), were close allies of developers, and luxury pays. The total public subsidy for the Seaport/Innovation District is hard to calculate, because it is a mix of land assembly, roads, infrastructure, and tax breaks, but it easily runs into the billions. Think of the affordable housing that might have been built.
In addition to being a case study of how not to develop affordable housing, the Innovation District is a case study of how not to do transportation and climate remediation. It is exactly at sea level, and the city imposed hardly any building standards to protect against sea level rise. Thus its nickname: The Inundation District. And no subway line was extended to the new district, creating parking problems.
This all occurred not because planners are stupid. It occurred because of the political power of developers.
Now, Boston finally has a mayor who is not in the pocket of developers, Michelle Wu. But that one last giant tract is pretty well filled up.
Developers were so anxious about not having an ally in City Hall that they poured money into the campaign of billionaire Josh Kraft, a carpetbagger from the suburbs whom Wu so thoroughly trounced in the September preliminary election that he dropped out before the November final.
But winning an election overwhelmingly is not the same as having adequate resources. And even if developers no longer control City Hall, they pretty well control the legislature. So Boston is unlikely to get the taxing resources that it needs to build more affordable housing.
IN NEW YORK, THERE IS NOTHING QUITE COMPARABLE
to the Seaport District, but a wasted opportunity on a smaller scale is the development called Hudson Yards on the far West Side of Manhattan. Built on giant platforms over rail lines, the heart of Hudson Yards is a giant indoor mall plus luxury housing.
Think about New York City for a minute. One of its many great qualities is the street-level retail of all kinds. New York needs a suburban-style indoor shopping mall like the proverbial bull needs proverbial teats. Plus, the region already has them: It’s called New Jersey.
But there was money to be made, so in 2005 the city cut a deal (finalized by the city council in 2013) with billionaire Steve Ross and his Related Companies to develop Hudson Yards with a total of 13,500 housing units, of which some 4,000 were to be affordable. In the end, only about 600 affordable units were produced. The average Hudson Yards condo in 2025 has sold for $7.4 million.
At the time, the mayor was (of course) Michael Bloomberg, a civic liberal in some respects but the ultimate ally of real estate developers.
Here is another telling irony. One of the nearby public amenities that makes Hudson Yards and the surrounding areas so commercially valuable is a wonderful quirky walkway called the High Line. It began life as an abandoned elevated railroad track. When I lived in West Greenwich Village as a young writer, it went right through my neighborhood.
In the 1990s, a local group, Friends of the High Line, came up with the improbable idea of developing it into a greened pathway. They persuaded very skeptical city officials to let them try, and the idea has succeeded spectacularly. The High Line is now a charming elevated park. It is so attractive that luxury housing has been built all along it, and it is one of the attractions of the nearby Hudson Yards.
So something that began as a loving, volunteer civic endeavor has become one more subsidy to billionaires. The ghost of the economist Henry George would understand. He proposed a tax on the unearned increment in land values.
There is a second phase of Hudson Yards still in the planning stage. The original developer bailed out, and various city agencies are still in final negotiations with the latest developer. The city has spent billions in various subsidies for Hudson Yards. Here is where Mayor-elect Mamdani comes in. He could demand a lot more affordable housing.
THERE IS ONE MORE WAY THAT DEVELOPERS
have choked off the supply of affordable housing in places like Boston and New York. That is by converting subsidized apartments intended for low- or middle-income people into luxury housing. Many federal programs allow this to be done as soon as the original mortgage is paid off.
In New York, many moderate-income complexes built with tax subsidies, such as Stuyvesant Town and Peter Cooper Village, have been converted to luxury housing. Likewise for many New York apartments built as middle-class housing under a city-state program that used tax-exempt bonds and loans with low interest rates, called the Mitchell-Lama program.
One of the prime offenders, who got very rich converting Mitchell-Lama apartments, was a developer named … wait for it … Steve Witkoff. Yes, the same Trump crony who sold out middle-income New York renters is now reborn as Trump’s foreign-policy guy in charge of selling out Ukraine.
These conversions could not have been done without the approval of city officials. This is another reflection of the same political power of developers. The big developers were huge campaign contributors to the opponents of Mamdani because they appreciated that he could put a stop to this thievery. Let’s hope that he does.
Robert Kuttner is co-founder and co-editor of The American Prospect, and professor at Brandeis University’s Heller School. His latest book is Going Big: FDR’s Legacy, Biden’s New Deal, and the Struggle.
Evaluating Uniform Memory Access Mode on AMD's Turin
NUMA, or Non-Uniform Memory Access, lets hardware expose affinity between cores and memory controllers to software. NUMA nodes traditionally aligned with socket boundaries, but modern server chips can subdivide a socket into multiple NUMA nodes. It’s a reflection of how non-uniform interconnects get as core and memory controller counts keep going up. AMD designates their NUMA modes with the NPS (Nodes Per Socket) prefix.
NPS0 is a special NUMA mode that goes in the other direction. Rather than subdivide the system, NPS0 exposes a dual socket system as a single monolithic entity. It evenly distributes memory accesses across all memory controller channels, providing uniform memory access like in a desktop system. NPS0 and similar modes exist because optimizing for NUMA can be complicated and time intensive. Programmers have to specify a NUMA node for each memory allocation, and take are to minimize cross-node memory accesses. Each NUMA node only represents a fraction of system resources, so code pinned to a NUMA node will be constrained by that node’s CPU core count, memory bandwidth, and memory capacity. Effort spent getting an application to scale across NUMA nodes might be effort not spent on a software project’s other goals.
From AMD’s EPYC 9005 Series Architecture Overview, showing a dual socket Zen 5 (Turin) setup in NPS1 mode
A massive thank you goes to
Verda (formerly DataCrunch)
for proving an instance with 2 AMD EPYC 9575Fs and 8 Nvidia B200 GPUs.
Verda
gave us about 3 weeks with the instance to do with as we wished. While this article looks at the AMD EPYC 9575Fs, there will be upcoming coverage of the B200s found in the VM.
This system appears to be running in NPS0 mode, giving an opportunity to see how a modern server acts with 24 memory controllers providing uniform memory access.
A simple latency test immediately shows the cost of providing uniform memory access. DRAM latency rises to over 220 ns, giving a nearly 90 ns penalty over the EPYC 9355P running in NPS1 mode. It’s a high penalty compared to using the equivalent of NPS0 on older systems. For example, a dual socket Broadwell system has 75.8 ns of DRAM latency when each socket is treated as a NUMA node, and 104.6 ns with uniform memory access[1].
NPS0 mode does have a bandwidth advantage from bringing twice as many memory controllers into play. But the extra bandwidth doesn’t translate to a latency advantage until bandwidth demands reach nearly 400 GB/s. The EPYC 9355P seems to suffer when a latency test thread is mixed with bandwidth heavy ones. A bandwidth test thread with just linear read patterns
can achieve 479 GB/s in NPS1 mode
. However, my bandwidth test produces low values on the EPYC 9575F because not all test threads finish at the same time. I avoid this problem in the loaded memory latency test, because I have bandwidth load threads check a flag. That lets me stop all threads at approximately the same time.
Per-CCD bandwidth is barely affected by the different NPS modes. Both the EPYC 9355P and 9575F use “GMI-Wide” links for their Core Complex Dies, or CCDs. GMI-Wide provides 64B/cycle of read and write bandwidth at the Infinity Fabric clock. On both chips, each CCD enjoys more bandwidth to the system compared to standard “GMI-Narrow” configurations. For reference, a GMI-Narrow setup running at a typical desktop 2 GHz FCLK would be limited to 64 GB/s of read and 32 GB/s of write bandwidth.
Higher memory latency could lead to lower performance, especially in single threaded workloads. But the EPYC 9575F does surprisingly well in SPEC CPU2017. The EPYC 9575F runs at a higher 5 GHz clock speed, and DRAM latency is only one of many factors that affect CPU performance.
Individual workloads show a more complex picture. The EPYC 9575F does best when workloads don’t miss cache. Then, its high 5 GHz clock speed can shine. 548.exchange2 is an example. On the other hand, workloads that hit DRAM a lot suffer in NPS0 mode. 502.gcc, 505.mcf, and 520.omnetpp see the EPYC 9575F’s higher clock speed count for nothing, and the higher clocked chip underperforms compared to 4.4 GHz setups with lower DRAM latency.
SPEC CPU2017’s floating point suite also shows diverse behavior. 549.fotonik3d and 554.roms suffer in NPS0 mode as the EPYC 9575F struggles to keep itself fed. 538.imagick plays nicely to the EPYC 9575F’s advantages. In that test, high cache hitrates let the 9575F’s higher core throughput shine through.
NPS0 mode performs surprisingly well in a single threaded SPEC CPU2017 run. Some sub-tests suffer from higher memory latency, but enough other tests benefit from the higher 5 GHz clock speed to make up the difference. It’s a lesson about the importance of clock speeds and good caching in a modern server CPU. Those two factors go together, because faster cores only provide a performance advantage if the memory subsystem can feed them. The EPYC 9575F’s good overall performance despite having over 220 ns of memory latency shows how good its caching setup is.
As for running in NPS0 mode, I don’t think it’s worthwhile in a modern system. The latency penalty is very high, and bandwidth gains are minor for NUMA-unaware code. I expect those latency penalties to get worse as server core and memory controller counts continue to increase. For workloads that need to scale across socket boundaries, optimizing for NUMA looks to be an unfortunate necessity.
Again, a massive thank you goes to
Verda (formerly DataCrunch)
without which this article, and the upcoming B200 article, would not be possible!
Seems weird to say, but I've been posting here for seventeen years now. And in that time, can I say that the quality of the discourse has slipped some? Well... if I'm being honest, probably yeah. A little. But at the same time, I can still honestly say that HN is still easily the best community of this sort on the 'net, at least that I'm aware of. OK, Lobste.rs has some merit, but the problem there is that the community there is arguably still a little too small, and you just don't get the variety and volume of interesting discussion you get here. But the level of discourse is high there as well.
Anyway, I find HN to be a wonderful refuge from a lot of the absurdity that's "out there" and I will happily throw in my own "Thanks, guys!" to dang and tomhow. And to pg for starting this whole thing back in the day.
Happy Thanksgiving, everyone, and here's to more years to come!
$96M AUD revamp of Bom website bombs out on launch
Australia's beloved weather website got a makeover - and infuriated users
Getty Images
Farmers are angry - they argue the information they need is now hard to find
It was an unseasonably warm spring day in Sydney on 22 October, with a forecast of 39C (99F) - a real scorcher.
The day before, the state of New South Wales had reported its hottest day in over a century, a high of 44.8C in the outback town of Bourke.
But little did the team at the national Bureau of Meteorology foresee that they, in particular, would soon be feeling the heat.
Affectionately known by Australians as the Bom, the agency's long-awaited website redesign went live that morning, more than a decade after the last update.
Within hours, the Bom was flooded with a deluge of complaints. The hashtag #changeitback went viral.
Gripes ranged from the new colour scheme for the rain radar, to furious farmers and fishermen who could no longer put in GPS coordinates to find forecasts for a specific location.
And then, this week it was revealed that the site's redesign had cost about A$96.5m ($62.3m; £48m), 20 times more than the previously stated A$4.1m.
"First you violate expectations by making something worse, then you compound the injury by revealing the violation was both expensive and avoidable," psychologist and neuroscientist Joel Pearson told the BBC, explaining the public outrage.
"It's the government IT project equivalent of ordering a renovation, discovering the contractor has made your house less functional, and then learning they charged you for a mansion."
'Game of hide and seek'
A consensus was quickly clear: "Please bring back the previous format," one person surmised on social media.
"It's awful, the most useful features are gone and it's not user-friendly. A waste of taxpayer money," another added.
Others said the timing was poor: "Why change it on a day of severe weather?"
There were some fans, including one who posted: "I like the new site. The front page is much cleaner". But they were few and far between.
Less than 48 hours after the launch, the Bom released a
list of tips
on how to use the new site, but this was further mocked by disgruntled users.
"Terrible! You shouldn't need step-by-step instructions to navigate the site," one post read.
Social media has been flooded with complaints about the new site
With more than 2.6 billion views a year, Bom tried to explain that the site's refresh - prompted by a major cybersecurity breach in 2015 - was aimed at improving stability, security and accessibility. It did little to satisfy the public.
Some frustrated users turned to humour: "As much as I love a good game of hide and seek, can you tell us where you're hiding synoptic charts or drop some clues?"
Malcolm Taylor, an agronomist in Victoria, told the Australian Broadcasting Corporation (ABC) that the redesign was a complete disaster.
"I'm the person who needs it and it's not giving me the information I need," the plant and soil scientist said.
Others appeared to accept their fate: "I am sure we will get used to it but it is not intuitive at all."
Bureau of Meteorology
Many users say they found the old version easier to navigate
Exactly a week after the debacle, the acting head of the agency was forced to apologise. There were concerns that people had been underprepared for storms in Queensland because of the site's poor usability.
The outpouring prompted the federal government to issue a scathing rebuke of the Bom and order immediate changes to the site.
"The bureau clearly has work to do, in that it has lost community confidence in the new website," Energy Minister Chris Bowen said at the time.
In a bid to calm the storm, parts of the previous site were brought back to life, giving people the option to use the old features.
A month after the relaunch, the new head of the Bom - who started his role during the saga - admitted the changes had been "challenging for some" and again apologised for the confusion.
"Inherently, we don't, and won't, always get it perfectly right. But, we are constantly striving to get better," Dr Stuart Minchin said.
But he kicked off another round of criticism by revealing the revamp actually cost $96m, a figure which covered a full website rebuild and testing of the "systems and technology that underpin" it.
Immediately, the government demanded Bom explain how taxpayers' money had been spent "efficiently and appropriately," according to the Sydney Morning Herald.
Barnaby Joyce, a member of the Nationals, which mainly represents regional communities, said: "We spent $96m to put a B at the end of the Bom site. It's now bomb, it's hopeless."
New site 'scrambling' people's brains
On the day of the launch, the Bom assured Australians that the community had been consulted on the changes. A test site in the months leading up to the relaunch found customer satisfaction rates were consistently above 70%, they told the BBC.
"The tsunami of complaints suggests that consultation was either perfunctory or they listened to the wrong people," Mr Pearson said.
For years, farmers and emergency workers had developed what neuroscientists call "procedural memory" for reading weather patterns using the site, he explained. It's muscle memory like touch-typing or driving a familiar route home.
"Your fingers know where the keys are, your hands know when to turn."
But when the new site changed the radar's colour scale, long-time users were left scratching their heads as their "hard-won intuition for reading storm intensity became unreliable overnight".
Steve Turton/Bom
The old colour scheme included black which users said was a useful indicator
The new site, Mr Pearson said, "was scrambling the neurological shortcuts that people had spent a decade building".
"It's like rearranging all the furniture in your house and then expecting you to navigate it in the dark without stubbing your toe. Except the 'furniture' in this case determines whether you move your livestock before the flood arrives."
For sociologist Ash Watson, the collective reaction to the site reflected its special status in Australia.
"Australia has always been a large country of weather extremes, and Bom's cultural importance has really been cemented in recent years as we've experienced more severe weather and the rising impacts of climate change."
As a regular user of Bom's site, Ms Watson acknowledged the good intentions behind the changes, but said her research - on the social impact of tech - showed that people are getting fatigued by change.
"It can be hard for people to get excited by new updates and see their immediate benefits when they don't want to have to learn how to use yet another new platform, app or website."
AFP via Getty Images
The Bom website performs an essential role in times of disaster
This is not the first time the Bom has weathered a publicity storm.
In 2022, it spent hundreds of thousands of dollars on a rebrand, asking to be called either its full name or "the bureau", not the "weather bureau" or "the Bom", given the negative connotations.
But the campaign was short-lived. They eventually released a statement saying the public was welcome to use whatever name they wished.
The incident reflected a fundamental misunderstanding of how the culture of naming works, Mr Pearson said.
Australians had organically adopted "Bom" as a term of affection, like a nickname for a friend, he said.
"When the institution tried to correct this, it felt like being told you're pronouncing your mate's name wrong."
He said the site's redesign revealed a similar "cultural blindness but with higher stakes".
In a statement, Bom's spokesperson told the BBC it had received about 400,000 items of feedback on the new site, which accounted for less than 1% of the 55 million visits in the past month.
The responses were "both positive and negative", they said, with fans saying they liked the new design and presentation, the accuracy and reliability of the forecasts, and greater ease in using the site on different types of mobile devices.
But it was clear that people had "formed strong habits", the spokesperson said, and further changes may be made based on the feedback.
The Goal of Socialism Is Everything
Portside
portside.org
2025-11-27 05:01:47
The Goal of Socialism Is Everything
Mark Brody
Thu, 11/27/2025 - 00:01
...
On Saturday, November 22,
Jacobin
founding editor Bhaskar Sunkara delivered the keynote at New York City Democratic Socialists of America’s (DSA) biannual organizing conference at the First Unitarian Congregational Society in Brooklyn. Below is a transcript of his remarks on why the Left must win real gains today — but also keep fighting for a socialist society beyond them.
I'm so excited to be here with you all. It feels to me that this is the political moment so many of us have been waiting for and working to build for years.
We’re a month away from one of our comrades becoming mayor. We’ve built a network of socialist elected officials, we have a real organization to call home, and there’s a growing base of support in this city for our immediate demand of taxing the rich to expand public goods.
This moment extends beyond New York — we have a huge political opening in the United States as whole. But we know that we have that opportunity because millions of people are living through hard times. We have an erratic and authoritarian president, we have an affordability crisis, with millions struggling to pay their bills and to live lives where they’re treated with dignity and respect. We’ve seen the return of forms of nativism and racism that should have been long vanquished by now.
And at a social and economic level, things may get worse very soon.
The country — not just this city — is crying out for principled political leadership. Not just a kind of populist leadership through great figures, though I’m grateful we have one of the greatest figures on our side. I mean class leadership through organization.
The leadership that says that the disparities that we see in our country and the world are not the natural laws of God but the result of a world that human beings have created. The leadership that says that the interests of the working-class majority are distinct from the interest of capitalist elites, and that we need to organize around those interests to win not only a better distribution of wealth
within capitalism
but a different type of society all together.
God’s Children Can Govern
Ijoined the Democratic Socialists of America when I was seventeen years old. I don’t need to tell you what DSA was in New York back in 2007. Some of you here remember it. I made so many good friends, but we were lucky if a dozen people showed up to a meeting.
We made progress through the patient, steady work and commitment of those people and the many more who joined later. We were marathon runners for socialism.
This, though, is a moment for sprinting. This is the biggest opening our movement has had in decades. The time we devote to political work in the next few months and years will have an outsize impact in our city and country — for now and for posterity.
But what exactly should we be doing, and how should we relate to both the new mayor’s administration and our other comrades in elected office? In my mind, our tasks as organized socialists outside of government are both different and largely compatible with theirs.
The key demands of our moment are around the affordability agenda. Our mayor-elect will be leading an effort to raise revenue to fund social programs and empower the city’s working class. If Zohran , our other electeds, and the grassroots movement around them deliver positive change in people’s lives, we’ll build a deeper social base for the Left.
Right now, our electoral strength has far outpaced our base. But people are ready for our message and ready for results.
But fundamentally, there are constraints to any sort of social democratic governance. Just as under capitalism, workers are dependent on having profitable firms for jobs. Cities are dependent on big corporations and wealthy people for tax revenue.
Zohran needs to navigate these constraints. He can’t undermine the old regime of accumulation and redistribution without having a replacement for it, and certainly there can’t be a total replacement in one city.
These concerns aren’t new. This is the dilemma of social democracy. This is the tension between our near-term and long-term goals that has existed in the socialist movement for 150 years.
Our elected officials in the near term need to manage capitalism in the interest of workers, while our movement also has a long-term goal of constructing a new system through the self-emancipation of those workers.
We need to see the constraints that Zohran will be under in these structural terms, rather than moral ones. But having patience and being supportive of him doesn’t answer how we reconcile the
near
and the
long
— social democracy and socialism.
At the very least, it’s important that we remember the end goal. The great theorist of reformism, Eduard Bernstein, once said that “the goal is nothing, the movement everything.” I think that’s not quite right. If we don’t talk about socialism after capitalism, no one else will. The historic dream of our movement, a world without exploitation or oppression, will be lost.
But we shouldn’t just avoid reformism because we want to feel pure as “true socialists” or as an intellectual pursuit. We should avoid reformism and remember the goal of rupture with capitalism because it can offer a compelling vision of the world to those we’re trying to reach.
Socialism isn’t “Sweden” like Bernie sometimes says. Socialism isn’t even
just
, as Martin Luther King Jr said and Zohran has beautifully invoked, “a better distribution of wealth for all of God’s children.”
Socialism means a better distribution but also democratic control over the things we all depend on — workers holding the levers of production and investment, and the state guaranteeing the basics of life as social rights.
Socialism means no longer begging corporations to invest in our communities or the rich to stay and pay their taxes.
Socialism means overcoming the labor-capital dialectic through the triumph of labor itself, not a more favorable class compromise.
Socialism means that the people who’ve kept this world alive — the caregivers, the drivers, the machinists, the farmworkers, the cleaners — stop being an invisible backdrop and become the authors of their futures.
Socialism means a society where those who have always given without having any say finally show their true capabilities. Where, as C. L. R. James
said
, every cook can govern.
Socialism means replacing an economy built on hierarchy and exclusion with one built on the intelligence and creativity of working people themselves.
That is the goal we keep alive. Not because it’s utopian, but because it is the only horizon equal to the dignity and potential of ordinary people.
And because, it’s
compelling
. This isn’t just offering workers some of their surplus value back in exchange for their votes. It’s offering them
the future
, a society that they can own, a chance to assume their rightful place as agents of history.
Something like this is real socialism. It isn’t an interest group or a label to distinguish ourselves from other progressives. It’s a fundamentally more radical goal than those of our allies. It’s premised on a different analysis of the world around us and the world that can be built.
Perhaps we can think of ways to bridge some of the gap between near and long through a set of demands that at least raise the concept of socialization immediately. Ideas that offer not just more badly needed social welfare but a taste of ownership and control. A hint at a different political economy.
Just one example: when a business closes down or its owners are retiring, workers supported by a public fund could get the first crack at saving it by converting it into a labor-managed firm. At the city level, we can have a municipal office to help workers turn shuttered shops into cooperatives by providing the backbone of legal and accounting support and fast-tracking permits.
We’ve already been talking about city-owned grocery stores and the need for public housing. We need more ideas like these. Reforms that fit within social democracy but gesture beyond it.
Socialism in Our Time
It’s been thrilling to meet people who’ve just joined DSA. It’s been nice to see old friends too. I’ve been complaining about missing the first half of the Knicks game, but even Jalen Brunson can’t keep me away from here.
I’m really enthusiastic about what we can do in the next couple of years. We
will
improve the lives of millions of people. And we will grow our movement.
But in addition to enthusiasm, we need
honesty about how far we still have to go to root ourselves in working-class communities. We need more power not just at the ballot box but at the points of production and exchange. And we need to be honest about the battles and constraints that Zohran will face, and be ready to support him when times get tough.
Zohran’s mayoralty will be a fight for what’s winnable right now. Our job is to let that fight expand, not narrow, our horizon — and to keep alive the goal of socialism in our time.
Bhaskar Sunkara is the founding editor of
Jacobin
, the president of the
Nation
magazine, and the author of
The Socialist Manifesto: The Case for Radical Politics in an Era of Extreme Inequality
.
Music eases surgery and speeds recovery, study finds
Music eases surgery and speeds recovery, Indian study finds
Soutik Biswas
India correspondent
BBC
A patient with headphones playing music during surgery in a hospital in Delhi
Under the harsh lights of an operating theatre in the Indian capital, Delhi, a woman lies motionless as surgeons prepare to remove her gallbladder.
She is under general anaesthesia: unconscious, insensate and rendered completely still by a blend of drugs that induce deep sleep, block memory, blunt pain and temporarily paralyse her muscles.
Yet, amid the hum of monitors and the steady rhythm of the surgical team, a gentle stream of flute music plays through the headphones placed over her ears.
Even as the drugs silence much of her brain, its auditory pathway remains partly active. When she wakes up, she will regain consciousness more quickly and clearly because she required lower doses of anaesthetic drugs such as propofol and opioid painkillers than patients who heard no music.
That, at least, is what a
new peer-reviewed study
from Delhi's Maulana Azad Medical College and Lok Nayak Hospital suggests. The research, published in the journal Music and Medicine, offers some of the strongest evidence yet that music played during general anaesthesia can modestly but meaningfully reduce drug requirements and improve recovery.
The study focuses on patients undergoing laparoscopic cholecystectomy, the standard keyhole operation to remove the gallbladder. The procedure is short - usually under an hour - and demands a particularly swift, "clear-headed" recovery.
To understand why the researchers turned to music, it helps to decode the modern practice of anaesthesia.
"Our aim is early discharge after surgery," says Dr Farah Husain, senior specialist in anaesthesia and certified music therapist for the study. "Patients need to wake up clear-headed, alert and oriented, and ideally pain-free. With better pain management, the stress response is curtailed."
Achieving that requires a carefully balanced mix of five or six drugs that together keep the patient asleep, block pain, prevent memory of the surgery and relax the muscles.
Getty Images
Patients need to wake up clear-headed and ideally pain-free after surgery
In procedures like laparoscopic gallbladder removal, anaesthesiologists now often supplement this drug regimen with regional "blocks" - ultrasound-guided injections that numb nerves in the abdominal wall.
"General anaesthesia plus blocks is the norm," says Dr Tanvi Goel, primary investigator and a former senior resident of Maulana Azad Medical College. "We've been doing this for decades."
But the body does not take to surgery easily. Even under anaesthesia, it reacts: heart rate rises, hormones surge, blood pressure spikes. Reducing and managing this cascade is one of the central goals of modern surgical care. Dr Husain explains that the stress response can slow recovery and worsen inflammation, highlighting why careful management is so important.
The stress starts even before the first cut, with intubation - the insertion of a breathing tube into the windpipe.
To do this, the anaesthesiologist uses a laryngoscope to lift the tongue and soft tissues at the base of the throat, obtain a clear view of the vocal cords, and guide the tube into the trachea. It's a routine step in general anaesthesia that keeps the airway open and allows precise control of the patient's breathing while they are unconscious.
"The laryngoscopy and intubation are considered the most stressful response during general anaesthesia," says Dr Sonia Wadhawan, director-professor of anaesthesia and intensive care at Maulana Azad Medical College and supervisor of the study.
"Although the patient is unconscious and will remember nothing, their body still reacts to the stress with changes in heart rate, blood pressure, and stress hormones."
To be sure, the drugs have evolved. The old ether masks have vanished. In their place are intravenous agents - most notably propofol, the hypnotic made infamous by
Michael Jackson's death
but prized in operating theatres for its rapid onset and clean recovery. "Propofol acts within about 12 seconds," notes Dr Goel. "We prefer it for short surgeries like laparoscopic cholecystectomy because it avoids the 'hangover' caused by inhalational gases."
The team of researchers wanted to know whether music could reduce how much propofol and fentanyl (an opioid painkiller) patients required. Less drugs means faster awakening, steadier vital signs and reduced side effects.
So they designed a study. A pilot involving eight patients led to a full 11-month trial of 56 adults, aged roughly 20 to 45, randomly assigned to two groups. All received the same five-drug regimen: a drug that prevents nausea and vomiting, a sedative, fentanyl, propofol and a muscle relaxant. Both groups wore noise-cancelling headphones - but only one heard music.
"We asked patients to select from two calming instrumental pieces - soft flute or piano," says Dr Husain. "The unconscious mind still has areas that remain active. Even if the music isn't explicitly recalled, implicit awareness can lead to beneficial effects."
A pilot involving eight patients led to a full trial of 56 adults randomly assigned to two groups
The results were striking.
Patients exposed to music required lower doses of propofol and fentanyl. They experienced smoother recoveries, lower cortisol or stress-hormone levels and a much better control of blood pressure during the surgery. "Since the ability to hear remains intact under anaesthesia," the researchers write, "music can still shape the brain's internal state."
Clearly, music seemed to quieten the internal storm. "The auditory pathway remains active even when you're unconscious," says Dr Wadhawan. "You may not remember the music, but the brain registers it."
The idea that the mind behind the anaesthetic veil is not entirely silent has long intrigued scientists. Rare cases of "intraoperative awareness" show patients recalling fragments of operating-room conversation.
If the brain is capable of picking up and remembering stressful experiences during surgery - even when a patient is unconscious - then it might also be able to register positive or comforting experiences, like music, even without conscious memory.
"We're only beginning to explore how the unconscious mind responds to non-pharmacological interventions like music," says Dr Husain. "It's a way of humanising the operating room."
Music therapy is not new to medicine; it has long been used in psychiatry, stroke rehabilitation and palliative care. But its entry into the intensely technical, machine-governed world of anaesthesia marks a quiet shift.
If such a simple intervention can reduce drug use and speed recovery - even modestly - it could reshape how hospitals think about surgical wellbeing.
As the research team prepares its next study exploring music-aided sedation, building on earlier findings, one truth is already humming through the data: even when the body is still and the mind asleep, it appears a few gentle notes can help the healing begin.
Worley noise
is a type of noise used for procedural texturing in computer graphics. In its most basic form, it looks like this:
(color r2 (worley q 50 :oversample true))
That’s ugly and boring, but it’s a quick way to see what the effect looks like. If we use Worley noise to distort a 3D shape, we can get something like a hammered or cratered texture:
(def s (osc t 5 | ss 0.2 0.8 * 30 + 10))
(ball 100
| expound (worley p s) (sqrt s)
| slow 0.8
| shade sky
| rotate y (t / 10))
Like many procedural textures, it looks a lot better if you repeat the effect a few times with different frequencies:
(def s (osc t 5 | ss 0.2 0.8 * 30 + 10))
(ball 100
| expound (fbm 3 worley p s) (sqrt s)
| slow 0.8
| shade sky
| rotate y (t / 10))
There are some visual artifacts in these renderings, because they’re using a fast approximation of Worley noise that gives the wrong answer for some values.
To explain these artifacts in more detail, we have to understand a little bit about how Worley noise works.
It’s pretty simple: you start with a grid of points.
When you’re writing a shader, you don’t actually have the ability to generate random numbers, so we’re using a hash function to produce random-
looking
offsets based on the logical position of each point (that is,
$i = [0 0]
for the center point,
$i = [1 0]
for the point to the right of that, etc).
Finally, once you have the points at random-looking positions, you compute the distance to the nearest point for every simple pixel that you sample – and that’s Worley noise.
How do you compute the distance to the nearest point for any pixel you ask about? It’s actually pretty simple: you know that you started with a perfectly even square grid. For any pixel, you can compute the “grid cell” that that pixel falls into (
[0 0]
,
[0 1]
, etc). It’s just the pixel divided by the grid size, rounded to the nearest integer.
And you know that the nearest point is either in this grid, or it’s in one of the immediately adjacent grids, because we only offset our points by
at most
half the grid size, so each randomly distributed point is still inside its original grid cell. Which means there’s no point inside any other cell that could be nearer than any point in one of the adjacent cells.
So that leaves you nine points to check, for every single pixel in your shader. Here’s the optimization that’s causing visual artifacts: instead of checking all nine adjacent cells, only check the current cell and the three cells closest to the point in question. The nearest point to your sample position is
probably
in one of those cells, but it doesn’t
have
to be. So you might get some visual artifacts occasionally.
Nowhere does that code compute cell coordinates or check for the nearest point. I just constructed this
thing
, said
shape/distance
, and somehow that just… gave me the distance to the nearest point.
I was able to do that because
Bauble
is a playground for making 3D graphics with
signed distance functions
. Bauble’s
whole deal
is computing distances to things! And Worley noise is just the signed distance function of a bunch of randomly distributed points. I’m used to thinking of signed distance functions as defining implicit surfaces of 3D shapes, but Worley noise uses the distance as a scalar in its own right.
So.
This is interesting.
What if… we took
other
signed distance functions, and used them as procedural noise distortions?
We’ll start simple. Instead of points, what if we randomly distribute a bunch of squares?
It’s a lot harder to visualize the distance field in 3D. What you’re seeing there is the distance field at the plane that passes through the origin and faces towards the camera. I know it’s not a great visualization, but the point is that this technique generalizes to 3D (even if it’s hard to imagine the distance field at every point in 3D space).
Let’s see how this looks when we use it to distort a 3D shape:
I think it’s kind of more interesting without the randomization.
We’ve constructed an interesting 3D noise function, and we’re using it to distort 3D space. But of course, we can go back to considering this a “noise texture” in the original sense of the word:
The point of all of this is: Worley noise invites us to reconsider signed distance functions as more than implicit surfaces. And since Bauble makes it easy to construct signed distance functions, it’s a good playground for experimenting with textures like this.
Even if we never found anything particularly attractive, it’s fun to play around with space.
If this is your first time seeing
Bauble
, hey welcome! This post is an expansion of something
I briefly talked about in a YouTube video once
. The video has many more examples of the sorts of things that Bauble can do. Check it out if this piqued your interest!
White Poverty
How Exposing Myths About Race and Class Can Reconstruct American Democracy
Liveright
William J. Barber II with Jonathan Wilson-Hartgrove
ISBN: 978-1-324-09675-7
For
progressives
to win, we need a powerful multiracial coalition. That includes the people of color who disproportionately suffer
poverty
and structural violence, but it also includes the white people who make up the largest share of poor people in this country.
As the Reverend Dr.
William J. Barber
II points out in his new book,
White Poverty
, there are more poor white people than any other racial group, and more effort should be put into pulling them into this coalition.
I’m a white man from a wealthy family—and a lawyer who took on tough
civil rights
cases and fought them as if my life depended on it. My goal from the beginning was to join those who are trying to make America a better place—a country where
racism
and sexism would slowly fade away and where the possibility of equal opportunity would shine through.
I see that road forward in Rev. Barber’s new book, co-written with Jonathan Wilson-Hartgrove.
White Poverty
‘s great value is to teach and motivate both Black and white leaders to create a multiracial movement which demands legislation that benefits
all
poor people.
Talking to white people in all walks of life—from taxi drivers to restaurant workers as well as bankers and stockbrokers—has been very revealing. When I say I’m a civil rights lawyer, their voices often take on a certain unsympathetic tone—and many times they inject the “Black crime rate” into the conversation. Sometimes the person will shift the conversation to discuss Black children being raised by single women who use food stamps to put food on the table or who benefit from other welfare programs.
As Barber points out, there are “more than twice as many poor white people as there are poor Black people in this nation.” But if I mention that, the person sometimes appears not to hear me, or lets me know in no uncertain terms that it’s Black people themselves who are at fault for their poverty—and they should look to their own lives rather than blame whites. The government taxes “us,” I’m often told, to give “them” a free ride.
When I hear this, I know there’s something major missing.
De-racializing Poverty
I’ve been encouraged by the many articles, books, and memoirs that have been written about
racial justice
since the protests over George Floyd’s murder, but few suggest an effective way forward.
For example, a new book by Kellie Carter Jackson,
We Refuse: A Forceful History of Black Resistance
(Seal Press, 2024), highlights how Black women fought back against racism, some with weapons, some without, but none took the path that Reverend Barber takes in
White Poverty
. Reverend Barber, by contrast, argues that Blacks and whites must join together to address their common needs.
Another prominent civil rights advocate, Heather McGhee, traveled across America to write
The Sum of Us: What Racism Costs Everyone and How We Can Prosper Together
(One World, 2021), which documents how some progressives were beginning to engage in cross-racial solidarity through collective action to achieve higher wages and benefits for working people.
As Barber points out, the political establishment invariably markets itself to the needs of “the middle class” and ignores the poor, and whites especially look the other way.
In effect, Barber’s
White Poverty
builds upon McGhee’s book. It’s the work of a man of action to not only test cross-racial solidarity, but to put that theory into action. Barber lays it on the line in his very first sentence: “This is a book by a Black man about white poverty in America.” That initial signal points to where he is headed.
As a lifelong civil rights lawyer, I find that his signal resonates. As Barber persuasively argues, the public and the country’s legislatures—federal, state, and local—accept the myth that poverty is only a Black issue, as do the people I talk to daily. They view poverty through this lens to the detriment of Black and white people alike, as well as people of all other colors and races.
As Barber points out, the political establishment invariably markets itself to the needs of “the middle class” and ignores the poor, and whites especially look the other way. The same is true even in our country’s religious establishments. Barber notes that “a Pew Research Center study of nearly 50,000 sermons found that neither the words ‘poverty’ nor ‘poor’ register as commonly used in American pulpits.”
A Multiracial Fusion Movement
Much of
White Poverty
concerns the history of how American racism came into being and how the myths evolved around it. Barber explains how the manipulation of these myths has preserved the power of white elites, who use their political and economic power to downgrade the needs of poor white people as well as Black people, while benefiting the wealthy.
To this reader then,
White Poverty
‘s great value is to teach and motivate both Black and white leaders to create a multiracial movement which demands legislation that benefits
all
poor people. As an additional benefit,
White Poverty
gives examples of Black and white movements fusing themselves together.
Not least, Barber has spent a huge amount of energy over the past seven years in building a multiracial
Poor People’s Campaign
. Co-chaired by Rev. Barber along with Rev. Liz Theoharis of the Kairos Center, the Poor People’s Campaign has thousands in the field to help poor white and poor Black communities understand each others’ community needs and the advantages of working together to fight against “policy violence” and to turn out the vote.
This beautifully written book offers a road map to the powerful multiracial organizing that can turn this country around, lift up poor people, and deepen our democracy.
In the last election for governor in
Kentucky
, the campaign and its allies worked with both white and Black rural communities to get out the vote. The result was an upset in electing the state’s present governor, Democrat Andy Beshear. In rural counties, an enlarged electorate turned out to vote and that tipped the election.
The Poor People’s Campaign has built durable alliances with other organizations to advance its multiracial vision. It’s currently collaborating with the AFL-CIO on voter engagement. It pursues legal challenges with Forward Justice. It coordinates actions with national Christian and Jewish organizations. With the Institute for Policy Studies, on whose board I serve, it has produced the data and the analysis to back up its bold agenda.
Barber is a man of the cloth who takes his
religion
seriously. As a result, the book is sprinkled with words from other religious figures who offer moral reasons for organizing poor people to struggle for their needs nonviolently but willing to cross police lines and stand up to authority.
In short, this beautifully written book offers a road map to the powerful multiracial organizing that can turn this country around, lift up poor people, and deepen our democracy.
Lewis M. Steel is a former senior counsel at Outten & Golden LLP and an Institute for Policy Studies board member. He's the author of The Butler's Child: White Privilege, Race, and a Lawyer's Life in Civil Rights
Fourteen years ago, my storage needs outpaced my capacity and I began to look into building a network attached storage server. I had a few criteria in mind and was curious to see if anyone had _ recently_ shared something similar, but I couldn’t find anything that was relevant.
In fact, I found that the communities I was looking for answers in were actively hostile towards what I wanted to do. This resulted in my decision
to build my own DIY NAS
and share that as one of my very first blogs.
Much to my surprise, people were very interested in that blog! Ever since, I’ve been building a similar DIY NAS machine almost every year trying to satisfy the curiosity of other prospective DIY NAS builders.
Here are those criteria:
Small form factor
: It’s not the case for me any more, but at the time the space was limited in my office. I always assume that space in everybody’s office is limited. As a result, I want my DIY NAS builds to occupy as little of that office space as I can.
At least six drive bays
: Back when I built my NAS, it took about four drives’ worth of storage to meet my storage needs. Plus I desired two empty drive bays for future use. However, in the years since hard drive capacities have increased dramatically. At some point in the future, I may reduce this to four drive bays.
An integrated, low power CPU
: I intend my DIY NAS to run 24 hours a day, 7 days a week, and 52 weeks a year. When it comes to power consumption, that can do some damage on your electric bill! Thankfully our electricity here isn’t as expensive as others’ in the United States, or even further outside its borders, but I try and keep power consumption in mind when picking components for a DIY NAS build.
Homelab potential
: It does not take up a lot of CPU horsepower for a NAS to serve up files, which means that on modern hardware there’s a lot of untapped potential in a DIY NAS for virtual machines or containers to self-host services.
It’s important to remember that
these are my criteria
, and not necessarily yours. Every DIY NAS builder should be making their own list of criteria and reconcile all of their component purchases against the criteria that’s important to them.
Is it even a good time to build a NAS?
As I prepared to build this NAS, component prices disappointed me. Hard drives, SSDs, and RAM prices were all rising. Based on what I’ve been told, I expect Intel CPU prices to increase as well. My contact at Topton has been encouraging me to stock up on motherboards while they still have some in inventory. Based on what’s been explained to me, I expect the motherboard’s prices to rise and for their availability to potentially dwindle.
In short, the economy sucks and the price of DIY NAS components is a pretty good reflection of just how sucky things are becoming. I briefly considered not publishing a DIY NAS build this year hoping that things would improve a few months down the road. But then I asked myself, “What if it’s even worse in a few months?”
I sure hope things get better, but I fear and expect that they’ll get worse.
Motherboard and CPU
I built
my first DIY NAS with a Topton motherboard in 2023
. Each DIY NAS since then has also featured a Topton motherboard. My only complaint about the motherboards has been that buying them from one of the Chinese e-tail sites like AliExpress is considered problematic by some. With every DIY NAS build, I try and go through all the motherboards that I can find while searching for something with a better value proposition, but for each of the past three years I’ve landed on the latest offering from Topton.
For the
DIY NAS: 2026 Edition
, I chose the
Topton N22 motherboard
with the
Intel Core 3 N355
CPU. The motherboard is similar to last year’s
Topton N18
but has incrementally more compelling features, particularly the extra 2 SATA ports, the PCI-e x1 slot, and the N355 CPU!
I opted for the motherboard with the
Intel Core 3 N355
CPU. This makes the server a more capable homelab machine than prior years’ DIY NAS builds. The extra cores and threads come in handy for streaming media, replacing your cloud storage, facilitating home automation, hosting game servers, etc.
Case
Just like Topton has been making great motherboards for DIY NAS machines, JONSBO has been steadily releasing great cases for DIY NAS machines. This year SilverStone Technology released a new case, the
CS383
(
specs
) which I was
very interested
in buying one for the
DIY NAS: 2026 Edition
. Unfortunately it carries a pretty hefty price tag to go along with all of its incredible features!
The
JONSBO N4
(
specs
) is a third the price, adheres to my “smaller footprint” criteria, and it is rather impressive on its own. It’s a
tiny
bit larger case than last year’s DIY NAS, but I really like that it has drive bays for six 3.5” drives and two 2.5” drives.
Although, it’s peculiar in that two of the 3.5” drive bays (and the two 2.5” drive bays) aren’t attached to a SATA backplane and can’t be swapped anywhere as easily as the other four 3.5” bays. However, this peculiar decision seems to have caused the JONSBO N4 to sell for a bit less ($20-$40) than similar offerings from JONSBO. At its price, it’s a compelling value proposition!
Case Fan
In the past, I’ve found that the fans which come with JONSBO cases are too noisy. They’ve been noisy for two reasons; the design quality of the fans make them loud. And the fans are constantly running at their top speed because of the fan header they’re plugged into on the cases’ SATA backplanes.
I anticipated that fan efficiency and noise would be a problem, so I picked out the
Noctua NF-A12x25 PWM
to solve it. Firstly, swapping in a high-quality fan that pushes more air
and
generates less noise–especially at its top speed–is a good first step. Secondly, I’d address the problem by plugging the fan into the motherboard’s
SYS_FAN
header instead of on the SATA backplane. This provides the opportunity to tune the fan’s RPMs directly in the BIOS and generate far less noise.
RAM
The first time I first asked myself, “Should I even build the
DIY NAS: 2026 Edition
?” came as I was checking prices on DDR5 memory. Thankfully for me I had leftover RAM after purchasing DDR5 4800MHz SODIMMs for the
DIY NAS: 2025 Edition
,
the Pocket Mini NAS
, and then again for the
DIY NAS that I built and gave away at 2025’s Texas Linux Fest
. I was personally thankful that I had one brand new 32GB DDR5 4800MHz SODIMM laying around, but I was wildly disappointed for everybody who will try and follow this build when I saw the price of those same SODIMMs.
Regardless, I felt a
Crucial 32GB DDR5 4800MHz SODIMM
(
specs
) was the right amount of RAM to get started with for a DIY NAS build in 2025. Whether you just need storage or you wish to also host virtual machines, you will benefit from having more than the bare minimum recommendation of RAM. I really wanted to buy a
48GB DDR5 4800MHZ SODIMM
for this DIY NAS build, but I couldn’t talk myself into spending the $250-$300 that it would’ve wound up costing.
Storage
A quick disclaimer about all the drives that I purchased for the
DIY NAS: 2026 Edition
, I already had all of them! I tend to buy things when I see them on sale and as a result, I have a collection of brand new parts for machines in my homelab or for upcoming projects. I raided that collection of spare parts for the
DIY NAS: 2026 Edition
.
Boot Drive
If you ranked the drives in your DIY NAS in order of importance, the boot drive should be the least-important drive. That is
not
saying that boot drive isn’t performing an important function, but I am suggesting that you shouldn’t invest a bunch of energy and money into picking the optimal boot drive.
Because the
JONSBO N4
has a pair of 2.5” drive bays, I decided that a 2.5” SATA SSD would be ideal for the boot drives. As a rule of thumb, I try and spend less than $30 per boot drive in my DIY NAS builds.
Ultimately I selected a pair of
128GB Silicon Power A55 SSDs
(
specs
). I’ve used these before, I’d use them again in the future, and I even have four of their higher capacity (1TB) SSDs in a pool in my own NAS.
App and Virtual Machine NVMe SSDs
Self-hosting apps and virtual machines on your DIY NAS has really exploded in the past few years. The developers of NAS appliance packages have made it much easier and the self-hosted products themselves have become as good–or often better–than things you’re probably subscribing to today. Because of that, I saved the highest-performing storage options on the
Topton N22 motherboard
for apps and VMs.
However, it’s important to point out that these M.2 slots are PCI-e version 3 and capped at a single PCI-e lane. This is a consequence of the limited number of PCI-e lanes available for each of the CPU options available for the
Topton N22 motherboard
(N100, N150, N305, and N355).
Thanks to rising prices, I opted to do like I’ve done with past DIY NAS builds and skip buying hard drives for the
DIY NAS: 2026 Edition
.
When planning your DIY NAS, it is good to always remember that
storage will ultimately be your costliest and most important expense
.
Here’s a few things to consider when buying hard drives:
Determine your hardware redundancy preferences. I recommend having two hard disk drives’ worth of redundancy (RAIDZ2, RAID6, etc.)
Focus on price-per-terabyte when comparing prices of drives.
Do some
burn in testing
of your hard drives before putting them to use.
When buying new drives of the same model, try and buy them from multiple vendors to increase the chances of buying drives manufactured in separate batches.
Plan Ahead! Understand the rate that your storage grows so that you can craft a strategy to grow your storage down the road.
Being cheap today can and will paint you into a corner that’s quite expensive to get out of.
Understand that RAID is not a backup!
Thankfully, I’ve collected a bunch of my own decomissioned hard drives which I used to thoroughly test this DIY NAS build.
SATA Cables
One of the under-the-radar features of the
Topton N22 motherboard
might be one of my favorite features! The motherboard’s Asmedia ASM1164 SATA controllers sit behind two SFF-8643 connectors. These connectors provide two advantages for these motherboards:
The one thing that I have routinely disliked about building small form factor DIY NAS machines is the price tag that accompanies a small form factor power supply (SFX) like is required with the
JONSBO N4
.
Regardless of whether it was called FreeNAS, TrueNAS, TrueNAS CORE, TrueNAS SCALE, or now
TrueNAS Community Edition
, the storage appliance product(s) from iXSystems have always been my go-to choice. For each yearly DIY NAS build, I wander over to the
TrueNAS Software Status page
and look at the state of the current builds.
I’m conservative with my personal NAS setup. However, for these blog builds, I typically choose Early Adopter releases. This year that’s
TrueNAS 25.10.0.1 (aka Goldeye)
. I enjoy being able to use these DIY NAS builds as a preview to the latest and greatest that TrueNAS has to offer.
I repeatedly choose TrueNAS because it’s what I’ve become accustomed to; it’s legitimately an enterprise-grade storage product, which is exactly the quality of solution that I want my data to depend on. At the same time it does not feel like you need a specialized certification and a truckload of enterprise storage experience to meet set up a NAS that exceeds your needs at home.
Many times I have been asked, “Why not
<insert NAS appliance or OS here>
?” My answer to that question is, TrueNAS has always done everything that I need it to and they haven’t given me any reason to consider anything else. As a result, there’s never been a need for me to evaluate something else.
Hardware Assembly, BIOS Configuration, and Burn-In
Hardware Assembly
I wanted the smallest possible DIY NAS. The
JONSBO N4
case initially felt too large since it accommodates Micro ATX motherboards. However, I grew to accept its slightly larger footprint. However, putting the
Topton N22 motherboard
into the case felt roomy and luxurious. Building the
DIY NAS: 2026 Edition
compared to prior years’ felt a lot like coming home to put on sweatpants and a t-shirt after wearing a suit and tie all day long.
I wasn’t too fond of the cable-management of the power supply’s cables. The layout of the case pretty much makes the front of the power supply inaccessible once it is installed. One consequence of this is that the power cable which powered the SATA backplane initially prevented the 120mm case fan from spinning up. That issue was relatively minor and was resolved with zip ties.
Overall, I felt pretty good about the assembly of the
DIY NAS: 2026 Edition
, but things would take a turn for the worse when I decided to fill all the 3.5-inch drive bays up with some of my decommissioned 8TB HDDs. Now this is probably my fault, I wouldn’t be surprised at all that the manual of the
JONSBO N4
warned me against this, but putting the drives in last turned out to be a major pain in the neck for each of the four drive bays
without
a SATA backplane.
I had wrongly guessed that you accessed those drives’ power and data ports from the front of the case. I worked really hard to route the cables and even managed to install all of the drives before realizing my error and learning my lesson. I’m understanding now why the
JONSBO N4
is cheaper than all of its siblings. Partly because there’s a missing SATA backplane, but also because those other 4 drive bays’ layout is frustrating.
Don’t let my last couple paragraphs sour you on the
JONSBO N4
, though. I still really like its size, it feels big when you’re working in it with a Mini ITX motherboard. If you wind up deciding to use the JONSBO N4, then I suggest that you put those four drives and their cables in first before you do anything else. That would’ve made a world of difference for me. Actually looking at the documentation before getting started might have saved me quite a bit of aggravation, too!
If I have ruined the JONSBO N4 for you, then check out the
JONSBO N3
. It’s
eight
3.5-inch drive bays pair up really nicely with the
Topton N22 motherboard
. You can see what I thought of the JONSBO N3 by reading the
DIY NAS: 2024 Edition
blog.
BIOS Configuration
Generally speaking, I do as little as I possibly can in the BIOS. Normally I strive to only set the time and change the boot order. However, I did a bit more for the
DIY NAS: 2026 Edition
since I’m using the
SYS_FAN
header for the fan which is responsible for cooling the hard drives. Here are the changes that I made in the BIOS:
Set the
System Date
and
System Time
to Greenwich Mean Time
Advanced
Hardware Monitor ( Advanced)
Set
SYS SmartFan Mode
to
Disabled
.
Set the
Manual PWM Setting
(for
SYS_FAN
) to 180.
Set
PWRON After Power Loss
to
Always On
Boot
Set
Boot Option #1
to the TrueNAS boot device.
I’m not at all interested in venturing into the rabbit’s hole of trying to completely minimize how much power the NAS uses. However, I imagine there’s some opportunities for power savings lurking in the BIOS. I didn’t go looking for them myself, but if you’re intrepid enough to do so here’s a few suggestions that I have to save some additional power:
Disable the onboard audio.
Disable any network interfaces that you don’t wind up using.
Tinker with the CPU settings.
Got other suggestions?
Share them in the comments!
Burn-In
Because all of the hardware is brand-new to me brand-new components are not guaranteed to be free of defects, I always do a little bit of burn-in testing to establish some trust in the hardware that I’ve picked out for each DIY NAS build. While I think doing
some
burn-in testing critically important, I also think the value of subsequent burn-in testing drops the more that you do. Don’t get too carried away and do your own burn-in testing in moderation!
I
always
use Memtest86+ to burn-in the RAM. I always run at least 3+ passes of Memtest86+. Typically, I run many more passes because I tend to let the system keep running additional passes overnight. Secondarily, running these many passes give the CPU a little bit of work to do and there’s enough information displayed by Memtest86+ to give me confidence in the CPU and its settings.
Hard Drives
The failure rate of hard drives is highest when the drives are new and then again when they’re old. Regardless of type of hard drives that I buy or when I buy them, I always do some disk burn in. I tend to run
Spearfoot’s Disk Burn-in and Testing script
on all of my new drives. However executing this script against all of the drives can take quite a long time, even if you use something like
tmux
to run the tests in parallel.
Initial TrueNAS CE Setup
There’s always a little bit of setup that I do for a new TrueNAS machine. This isn’t intended to be an all inclusive step-by-step guide for all the things you should do with your DIY NAS. Instead, it’s more of a list of things I kept track of while I made sure that the
DIY NAS: 2026 Edition
was functional enough for me to finish writing this blog. That being said, I do think your NAS would be rather functional if you decided to do the same configuration.
Updated the hostname to
diynas2026
Note: This is only to avoid issues with another NAS on my network.
Updated the timezone.
Enabled the following services and set them to start automatically.
SMB
SSH
NFS
Enabled password login for the
truenas_admin
user.
Note: If I were planning to use this DIY NAS long-term, I wouldn’t have done this. Using SSH keys for authentication is a better idea
.
Edited the TrueNAS Dashboard widgets to reflect the 10Gb interface (
enp1s0
).
Created a pool named
rust
which consisted of a single RAID-Z2 vdev using eight hard drives that I had sitting on my shelf after they were decomissioned.
Configured the Apps to use the
flash
pool for the apps’ dataset.
Made sure that the System Dataset Pool was set to
flash
.
Confirmed that there were Scrub Tasks set up for the
flash
and
rust
pools.
Created a dataset on each pool for testing;
flash-test
and
rust-test
Installed the
Scrutiny
app found in the App Catalog.
If I were planning to keep this NAS and use it for my own purposes, I would also:
As the NAS is seeded with data, create and maintain a suite of
snapshot tasks
tailored to the importance of the different data being stored on the NAS.
Set up S.M.A.R.T. tests for all of the drives:
Weekly Short Test
Monthly Long Test
Benchmarks
Just about every year, I benchmark each DIY NAS build and almost always come to the same conclusion; the NAS will outperform your network at home. Your first bottleneck is almost always going to be the network and the overlwhelming majority of us have gigabit networks at home–but that’s slowly changing since 2.5Gbps and 10Gbps network hardware has started to get reasonable lately.
Even though I always come to the same conclusion, I still like to do the benchmarks for two reasons:
It helps me build confidence that the
DIY NAS: 2026 Edition
works well.
People tend to enjoy consuming benchmarks
and
it’s fun for me to see the DIY NAS’ network card get saturated during the testing.
Throughput
I like to do three categories of tests to measure the throughput of the NAS:
Use
iperf3
to benchmark throughput between my NAS and another machine on my network.
Benchmark the throughput of the pool(s) locally on the NAS using
fio
.
Set up SMB shares on each of the pools and then benchmark the throughput when using those shares.
What do I think these benchmarks and my use of the
DIY NAS: 2026 Edition
tell me? In the grand scheme of things, not a whole lot.
However, these benchmarks do back up what I expected, the
DIY NAS: 2026 Edition
is quite capable and more than ready to meet my storage needs. I especially like that the CrystalDiskMark benchmark of the SMB shares were
both faster than a SATA SSD
, and the throughput to the share on the
flash
pool practically saturated the NAS’ 10GbE network connection.
FIO Tests
Every time I benchmark a NAS, I seem to either be refining what I tried in prior years or completely reinventing the wheel. As a result, I wouldn’t recommend comparing these results with results that I shared in prior years’ DIY NAS build blogs. I haven’t really put a ton of effort into developing a standard suite of benchmarks. Things in my homelab change enough between DIY NAS blogs that trying to create and maintain an environment for a standard suite of benchmarks is beyond what my budget, spare time, and attention span will allow.
I’m going to paste these
fio
commands here in the blog for my own use in future DIY NAS build blogs. If you wind up building something similar, these
might
be helpful to measure your new NAS’ filesystem’s performance and compare it to mine!
One not-so-obvious cost of running a DIY NAS is how much power it consumes. While I specifically tried to pick items that were efficient in terms of power consumption, it’s also important to realize that all the other bells and whistles on the awesome
Topton N18 NAS motherboard
consume power, too. And that the biggest consumer of power in a NAS is almost always the hard disk drives.
Thanks to my tinkering with
home automation
, I have a plethora of
smart outlets
which are capable of power monitoring. I used those smart outlets for most of my power monitoring. But I also have a
Kill a Watt P400
that I also use for some of the shorter tests:
Power consumed during a handful of specific tasks:
Idle while running TrueNAS
RAM Burn-in (~14 passes of Memtest86+)
An 8-hour throughput benchmark copying randomly-sized files to the NAS using SMB.
Total consumed during the build, burn-in, and use of the
DIY NAS: 2026 Edition
.
Task
Duration
Max Wattage
Avg. Wattage
Total Consumption
Boot
10 min.
200.00 W
120.00 W
0.02 kWh
Idle
3 hr.
90.00 W
66.67 W
0.20 kWh
RAM Burn-in
18 hr.
104.00 W
91.67 W
1.65 kWh
SMB Benchmark of HDDs
8 hr.
107.00 W
85.00 W
0.68 kWh
Total
108 hr.
237.80 W
66.49 W
7.17 kWh
What about an EconoNAS?
Shortly before prices skyrocketed, I decided I wasn’t very interested in doing a separate EconoNAS builds. Several months ago, I realized that there were several off-the-shelf NAS machines that were more-than-capable of running TrueNAS and they were selling at economical prices that couldn’t be topped by a DIY approach. I will dive deeper into this in a future blog, eventually …
maybe
?
All that being said–it’d be incredibly easy to make some compromises which result in the
DIY NAS: 2026 Edition
becoming quite a bit more economical. Here’s a list of changes that I would consider to be more budget-friendly:
Altogether, these savings could add up to more than $400, which is pretty considerable! If you made all of these changes, you’d have something that’s going to be nearly equivalent to the
DIY NAS: 2026 Edition
but at a fraction of the price.
What am I going to do with the DIY NAS: 2026 Edition?!
My DIY NAS is aging quite gracefully, but I’ve recently been wondering about replacing it. Shortly before ordering all the parts for the
DIY NAS: 2026 Edition
, I briefly considered using this year’s DIY NAS build to replace my personal NAS. However, I decided not to do that. Then prices skyrocketed and I shelved the idea of building a replacement for my own NAS and I nearly shelved the idea of a DIY NAS in 2026!
So that begs the question, “What is Brian going to do with the
DIY NAS: 2026 Edition
?”
I’m going to auction it off on the
briancmosesdotcom store on eBay
! Shortly after publishing this blog, I’ll list it on eBay. In response to skyrocketing prices for PC components, I’m going to do a no-reserve auction. At the end of the auction, the highest bidder wins and hopefully they’ll get a pretty good deal!
Final Thoughts
Overall, I’m pleased with the
DIY NAS: 2026 Edition
. The
Topton N22 motherboard
is a significant improvement over last year’s
Topton N18 motherboard
, primarily due to its extra two SATA ports. This provides 33.3% more gross storage capacity.
While testing, I found the
Intel Core 3 N355 CPU
somewhat excessive for basic NAS functions. However, the substantial untapped CPU horsepower offers luxurious performance potential. This makes the build compelling for anyone planning extensive self-hosting projects.
I have mixed feelings about the
JONSBO N4 case
. The four right-side drive bays lack SATA backplane connectivity. Without creative cabling solutions, individual drive replacement becomes challenging. However, the case’s ~$125 price point compensates for this inconvenience. I anticipate that those the cost savings will justify the compromise for most builders. If I were to build the
DIY NAS: 2026 Edition
all over again, I’d be tempted to use the
JONSBO N3 case
or even the
JONSBO N6
which isn’t quite obtainable, yet.
The DIY NAS: 2026 Edition delivers excellent performance and superior specifications. In my opinion, it represents better value than off-the-shelf alternatives:
Building your own NAS provides significant advantages. Years later, you can upgrade RAM, motherboard, case, or add PCI-e (x1) expansion cards. These off-the-shelf alternatives offer severely limited upgrade paths.
Penpot is the first
open-source
design tool for design and code collaboration. Designers can create stunning designs, interactive prototypes, design systems at scale, while developers enjoy ready-to-use code and make their workflow easy and fast. And all of this with no handoff drama.
Available on browser or self-hosted, Penpot works with open standards like SVG, CSS, HTML and JSON, and it’s free!
The latest updates take Penpot even further. It’s the first design tool to integrate native
design tokens
—a single source of truth to improve efficiency and collaboration between product design and development.
With the
huge 2.0 release
, Penpot took the platform to a whole new level. This update introduces the ground-breaking
CSS Grid Layout feature
, a complete UI redesign, a new Components system, and much more.
For organizations that need extra service for its teams,
get in touch
🎇 Design, code, and Open Source meet at
Penpot Fest
! Be part of the 2025 edition in Madrid, Spain, on October 9-10.
Penpot expresses designs as code. Designers can do their best work and see it will be beautifully implemented by developers in a two-way collaboration.
Plugin system
Penpot plugins
let you expand the platform's capabilities, give you the flexibility to integrate it with other apps, and design custom solutions.
Designed for developers
Penpot was built to serve both designers and developers and create a fluid design-code process. You have the choice to enjoy real-time collaboration or play "solo".
Inspect mode
Work with ready-to-use code and make your workflow easy and fast. The inspect tab gives instant access to SVG, CSS and HTML code.
Self host your own instance
Provide your team or organization with a completely owned collaborative design tool. Use Penpot's cloud service or deploy your own Penpot server.
Integrations
Penpot offers integration into the development toolchain, thanks to its support for webhooks and an API accessible through access tokens.
Building Design Systems: design tokens, components and variants
Penpot brings design systems to code-minded teams: a single source of truth with native Design Tokens, Components, and Variants for scalable, reusable, and consistent UI across projects and platforms.
Getting started
Penpot is the only design & prototype platform that is deployment agnostic. You can use it in our
SAAS
or deploy it anywhere.
Learn how to install it with Docker, Kubernetes, Elestio or other options on
our website
.
Community
We love the Open Source software community. Contributing is our passion and if it’s yours too, participate and
improve
Penpot. All your designs, code and ideas are welcome!
If you need help or have any questions; if you’d like to share your experience using Penpot or get inspired; if you’d rather meet our community of developers and designers,
join our Community
!
Anyone who contributes to Penpot, whether through code, in the community, or at an event, must adhere to the
code of conduct
and foster a positive and safe environment.
Contributing
Any contribution will make a difference to improve Penpot. How can you get involved?
Participate in the
Community
space by asking and answering questions; reacting to others’ articles; opening your own conversations and following along on decisions affecting the project.
Contribute to Penpot's code:
Watch this video
by Alejandro Alonso, CIO and developer at Penpot, where he gives us a hands-on demo of how to use Penpot’s repository and make changes in both front and back end
To find (almost) everything you need to know on how to contribute to Penpot, refer to the
contributing guide
.
Resources
You can ask and answer questions, have open-ended conversations, and follow along on decisions affecting the project.
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
Copyright (c) KALEIDOS INC
This book is an introduction to data structures and algorithms for
functional languages, with a focus on proofs. It covers both
functional correctness and running time analysis. It does so in a
unified manner with inductive proofs about functional programs and
their running time functions. All proofs have been
machine-checked by the proof assistant
Isabelle
. The pdf contains
links to the corresponding Isabelle theories.
Click on an image to download the pdf of the whole book:
This book is meant to evolve over time. If you would like to contribute, get in touch!
Migrating the Main Zig Repository from GitHub to Codeberg
Putting aside GitHub’s
relationship with ICE
, it’s abundantly clear that the talented folks who used to work on the product have moved on to bigger and better things, with the remaining losers eager to inflict some kind of bloated, buggy JavaScript framework on us in the name of progress. Stuff that used to be snappy is now sluggish and often entirely broken.
More importantly, Actions is
created by monkeys
and
completely neglected
. After the
CEO of GitHub said to “embrace AI or get out”
, it seems the lackeys at Microsoft took the hint, because GitHub Actions started “vibe-scheduling”; choosing jobs to run seemingly at random. Combined with other bugs and inability to manually intervene, this causes our CI system to get so backed up that not even master branch commits get checked.
Rather than wasting donation money on more CI hardware to work around this crumbling infrastructure, we’ve opted to switch Git hosting providers instead.
As a bonus, we look forward to fewer violations (exhibit
A
,
B
,
C
) of our
strict no LLM / no AI policy
, which I believe are at least in part due to GitHub aggressively pushing the “file an issue with Copilot” feature in everyone’s face.
GitHub Sponsors
The only concern we have in leaving GitHub behind has to do with GitHub Sponsors. This product was key to Zig’s early fundraising success, and it
remains a large portion of our revenue today
. I can’t thank
Devon Zuegel
enough. She appeared like an angel from heaven and single-handedly made GitHub into a viable source of income for thousands of developers. Under her leadership, the future of GitHub Sponsors looked bright, but sadly for us, she, too, moved on to bigger and better things. Since she left, that product as well has been neglected and is already starting to decline.
Although GitHub Sponsors is a large fraction of Zig Software Foundation’s donation income,
we consider it a liability
. We humbly ask if you, reader, are currently donating through GitHub Sponsors, that you consider
moving your recurring donation to Every.org
, which is itself a non-profit organization.
As part of this, we are sunsetting the GitHub Sponsors perks. These perks are things like getting your name onto the home page, and getting your name into the release notes, based on how much you donate monthly. We are working with the folks at Every.org so that we can offer the equivalent perks through that platform.
Migration Plan
Effective immediately, I have made
ziglang/zig on GitHub
read-only, and the canonical origin/master branch of the main Zig project repository is
https://codeberg.org/ziglang/zig.git
.
Thank you to the Forgejo contributors who helped us with our issues switching to the platform, as well as the Codeberg folks who worked with us on the migration - in particular
Earl Warren
,
Otto
,
Gusted
, and
Mathieu Fenniak
.
In the end, we opted for a simple strategy, sidestepping GitHub’s aggressive vendor lock-in: leave the existing issues open and unmigrated, but start counting issues at 30000 on Codeberg so that all issue numbers remain unambiguous. Let us please consider the GitHub issues that remain open as metaphorically “copy-on-write”.
Please leave all your existing GitHub issues and pull requests alone
. No need to move your stuff over to Codeberg unless you need to make edits, additional comments, or rebase.
We’re still going to look at the already open pull requests and issues
; don’t worry.
In this modern era of acquisitions, weak antitrust regulations, and platform capitalism leading to extreme concentrations of wealth, non-profits remain a bastion defending what remains of the commons.
The
Wild linker
makes very extensive use of
rayon
for parallelism. Much of this parallelism is in the
form of
par_iter
and friends. However, some parts of the linker don’t fit neatly because the amount of work isn’t
known in advance. For example, the linker has two places where it explores a graph. When we start,
we know some roots of that graph, but we don’t know all the nodes that we’ll need to visit. We’ve
gone through a few different approaches for how we implement such algorithms. This post covers those
approaches and what we’ve learned along the way.
Spawn broadcast
Our first approach was to spawn a task for each thread (rayon’s
spawn_broadcast
)
then do our own work sharing and job control between those threads. By “our own job control” I mean
that each thread would pull work from a channel and if it found no work, it’d
park the
thread
. If new work came up, the thread that
produced the work would wake a parked thread.
This was complex. Worse, it didn’t allow us to use other rayon features while it was running. For
example, if we tried to do a par_iter from one of the threads, it’d only have the current thread to
work with because all the others were doing their own thing, possibly parked, but in any case, not
available to rayon.
The idea here is that we create a scope and spawn some initial tasks into that scope. Those tasks
then spawn additional tasks and so on until eventually there are no more tasks.
The rayon documentation warns that this is more expensive than other approaches, so should be
avoided if possible. The reason it’s more expensive is that it heap-allocates the task. Indeed, when
using this approach, we do see increased heap allocations.
Channel + par_bridge
Another approach that I’ve tried recently and which arose out of the desire to reduce heap
allocations is to put work into a
crossbeam
channel
. The work
items can be an enum if there are different kinds. Our work scope is then just something like the
following:
let(work_send,work_recv)=crossbeam_channel::unbounded();// Add some initial work items. fornodeinroots{work_send.send(WorkItem::ProcessNode(node,work_send.clone()));}// Drop sender to ensure we can terminate. Each work item has a copy of the sender. drop(work_send);work_recv.into_iter().par_bridge().for_each(|work_item|{matchwork_item{WorkItem::ProcessNode(node,work_send)=>{explore_graph(node,work_send);}}});
The trick with this approach is that each work item needs to hold a copy of the send-end of the
channel. That means that when processing work items, we can add more work to the queue. Once the
last work item completes, the last copy of the sender is dropped and the channel closes.
This approach works OK. It does avoid the heap allocations associated with scoped spawning. It is a
little bit complex, although not as complex as doing all the job control ourselves. One downside is
that like doing job control ourselves, it doesn’t play nicely with using
par_iter
inside of worker
tasks. The reason why is kind of subtle and is due to the way rayon is implemented. What can happen
is that the
par_iter
doesn’t just process its own tasks. It can also steal work from other
threads. When it does this, it can end up blocking trying to pull another work item from the
channel. The trouble is that because the
par_iter
was called from a work item that holds a copy of
the send-end of the channel, we can end up deadlocked. The channel doesn’t close because we hold a
sender and we don’t drop the sender because we’re trying to read from the read-end of the channel.
Another problem with this approach that I’ve just come to realise is that it doesn’t compose well. I
had kind of imagined just getting more and more options in my
WorkItem
enum as the scope of the
work increased. The trouble is that working with this kind of work queue doesn’t play nicely with
the borrow checker. An example might help. Suppose we have some code written with rayon’s
par_chunks_mut
and we want to flatten that work into some other code that uses a channel with work items. First we
need to convert the
par_chunks_mut
code into a channel of work items.
letfoo=create_foo();foo.par_chunks_mut(chunk_size).for_each(|chunk|{// Do work with mutable slice `chunk` });
If we want the creation of
foo
to be a work item and each bit of processing to also be work items,
there’s no way to do that and have the borrow checker be happy.
matchwork_item{WorkItem::CreateAndProcessFoo=>{letfoo=create_foo();// Split `foo` into chunks and queue several `WorkItem::ProcessChunk`s….? }WorkItem::ProcessChunk(chunk)=>{// Do work with mutable slice `chunk`. }}
So that clearly doesn’t work. There’s no way for us to take our owned
foo
and split it into chunks
that can be processed as separate
WorkItem
s. The borrow checker won’t allow it.
Another problem arises if we’ve got two work-queue-based jobs and we’d like to combine them, but the
second job needs borrows that were taken by the first job to be released before it can run. This
runs into similar problems.
The kinds of code structures we end up with here feel a bit like we’re trying to write async code
without async/await. This makes me wonder if async/await could help here.
Async/await
I don’t know exactly what this would look like because I haven’t yet tried implementing it. But I
imagine it might look a lot like how the code is written with rayon’s scopes and spawning. Instead
of using rayon’s scopes, it’d use something like
async_scoped
.
One problem that I have with rayon currently is, I think, solved by using async/await. That problem,
which I briefly touched on above is described in more detail here. Suppose we have a
par_iter
inside some other parallel work:
outer_work.par_iter().for_each(|foo|{letfoo=inputs.par_iter().map(|i|...).collect();// < Some other work with `foo` here, hence why we cannot merge the two par_iters >foo.par_iter().map(|i|...).for_each(|i|...);});
If the thread that we’re running this code on becomes idle during the first inner
par_iter
, that
thread will try to steal work from other threads. If it succeeds, then even though all the work of
the
par_iter
is complete, we can’t continue to the second inner
par_iter
until the stolen work
also completes. However, with async/await, tasks are not tied to a specific thread once started.
Threads steal work, but tasks don’t, so the task that’s running the above code would become runnable
as soon as the
par_iter
completed even if the thread that had originally been running that task
had stolen work - the task could just be run on another thread.
It’d be very interesting to see what async/await could contribute to the parallel computation space.
I don’t have any plans to actually try this at this stage, but maybe in future.
Return to scoped spawning and future work
In the meantime, I’m thinking I’ll return to scoped spawning. Using a channel works fine for simple
tasks and it avoids the heap allocations, but it really doesn’t compose at all well.
I am interested in other options for avoiding the heap allocations. Perhaps there’s options for
making small changes to rayon that might achieve this. e.g. adding support for spawning tasks
without boxing, provided the closure is less than or equal to say 32 bytes. I’ve yet to explore such
options though.
Thanks
Thanks to everyone who has been
sponsoring
my work on
Wild, in particular the following, who have sponsored at least $15 in the last two months:
CodeursenLiberte
repi
rrbutani
Rafferty97
wasmerio
mati865
Urgau
mstange
flba-eb
bes
Tudyx
twilco
sourcefrog
simonlindholm
petersimonsson
marxin
joshtriplett
coreyja
binarybana
bcmyers
Kobzol
HadrienG2
+3 anonymous
The Tesla Model Y Just Scored the Worst Reliability Rating in a Decade
Bonsai_term is a library that lets you write Terminal UIs (TUIs) using
OCaml. It uses the same programming model as the
bonsai_web
library.
Getting started
If you are new to OCaml - or if you haven't already -
install
opam
. It is OCaml's package manager
and we'll be using it to install
bonsai_term
and its dependencies.
The specific installation instructions depend on your platform. You
can find platform-specific instructions
here
.
bonsai_term
uses
OxCaml
so the next thing
you'll want to do is install
oxcaml
by following the instructions
here
.
Run
opam install bonsai_term
. (This will install
bonsai_term
and its dependencies).
At this point you should now have
bonsai_term
"installed".
To learn how to use
bonsai_term
you can read its MLI
src/bonsai_term.mli
and / or look
at some examples in the
bonsai_term_examples
repo.
To learn how to use
bonsai
, you can read the docs in
bonsai_web
.
(most of those docs are aimed at the "web" version of bonsai, so the "vdom" bits may not
apply, but the "effect" / "state-fulness" and ways of doing "incrementality" all should
transfer from
bonsai_web
into
bonsai_term
).
To learn how to use
ocaml
here are some good resources:
Timed out getting readerview for https://www.analog.com/en/resources/analog-dialogue/articles/dsp-101-part-1.html
Russell Coker: PineTime Band
PlanetDebian
etbe.coker.com.au
2025-11-27 00:37:27
I’ve had a Pine Time for just over 2 years [1]. About a year ago I had a band break and replaced it from a spare PineTime and now I just had another break. Having the band only last one year isn’t that great, but it’s fortunate that the break only affects the inner layer of plastic so there is no ri...
I started writing this post while using the band from a
Colmi P80 [3]
. I bought one for a relative who wanted the metal band and the way the Aliexpress seller does it is to sell the package with the plastic band and include the metal band in the package so I had a spare band. It fits quite well and none of the reported problems of the PineTime having insufficient space between the spring bar and the watch. The Colmi band in question is described as “rose gold” but is more like “pinkish beige” and doesn’t match the style of the black PineTime.
I ordered a couple of cheap bands from AliExpress which cost $9.77 and $13.55 including postage while the ones that Pine64 recommend have over $15 postage from Amazon!
There are claims that getting a replacement band for a PineTime is difficult. My experience is that every band with a 20mm attachment works as long as it’s designed for a square watch, some of the bands are designed to partly go around a round face and wouldn’t fit. I expect that some bands won’t fit, but I don’t think that it’s enough of a problem to be worried about buying a random band from AliExpress. The incidence of bands not fitting will probably be lower than the incidence of other AliExpress products not doing quite what you want (while meeting the legal criteria of doing what they are claimed to do) and not being used.
I’m now wearing the PineTime with the “Magnetic Buckle Watch Strap Band” and plan to wear it for the next year or so.
Sam Roberts, reporting for The New York Times:
David Lerner, a high school dropout and self-taught computer geek
whose funky foothold in New York’s Flatiron district, Tekserve,
was for decades a beloved discount mecca for Apple customers
desperate to retrieve lost data and repair frozen hard dri...
“It was inevitable,” said Jake Ostrovskis, head of OTC trading at Wintermute, referring to the sell-off in digital asset treasury stocks. “It got to the point where there’s too many of them.”
Several companies have begun selling their crypto stockpiles in an effort to fund share buybacks and shore up their stock prices, in effect putting the crypto treasury model into reverse.
North Carolina-based ether holder FG Nexus sold about $41.5 million of its tokens recently to fund its share buyback program. Its market cap is $104 million, while the crypto it holds is worth $116 million. Florida-based life sciences company turned ether buyer ETHZilla recently sold about $40 million worth of its tokens, also to fund its share buyback program.
Sequans Communications, a French semiconductor company, sold about $100 million of its bitcoin this month in order to service its debt, in a sign of how some companies that borrowed to fund crypto purchases are now struggling. Sequans’ market capitalization is $87 million, while the bitcoin it holds is worth $198 million.
Georges Karam, chief executive of Sequans, said the sale was a “tactical decision aimed at unlocking shareholder value given current market conditions.”
While bitcoin and ether sellers can find buyers, companies with more niche tokens will find it more difficult to raise money from their holdings, according to Morgan McCarthy. “When you’ve got a medical device company buying some long-tail asset in crypto, a niche in a niche market, it is not going to end well,” he said, adding that 95 percent of digital asset treasuries “will go to zero.”
Strategy, meanwhile, has doubled down and bought even more bitcoin as the price of the token has fallen to $87,000, from $115,000 a month ago. The firm also faces the looming possibility of being cut from some major equity indices, which could heap even more selling pressure on the stock.
But Saylor has brushed off any concerns. “Volatility is Satoshi’s gift to the faithful,” he said this week, referring to the pseudonymous creator of bitcoin.
Foreign interference or opportunistic grifting: why are so many pro-Trump X accounts based in Asia?
Guardian
www.theguardian.com
2025-11-27 00:01:12
A new feature on the social media platform formerly known as Twitter allows users to see the location of other accounts. It has resulted in a firestorm of recriminations When X rolled out a new feature revealing the locations of popular accounts, the company was acting to boost transparency and clam...
W
hen
X
rolled out a new feature revealing the locations of popular accounts, the company was acting to boost transparency and clamp down on disinformation. The result, however, has been a circular firing squad of recriminations, as users turn on each other enraged by the revelation that dozens of popular “America first” and pro-Trump accounts originated overseas.
The new feature
was enabled over the weekend
by X’s head of product, Nikita Bier, who called it the first step in “securing the integrity of the global town square.” Since then many high-engagement accounts that post incessantly about US politics have been “unmasked” by fellow users.
An Ivanka Trump fan account that posts about illegal immigration to the US was shown to be based in Nigeria. MAGAStorm, spreading conspiracy theories about the assassination attempt on Trump, was found to be in eastern Europe. AmericanVoice which posts anti-Islam content, is based in India.
Users have noted that a high proportion of these potentially misleading accounts – many of which claim to be in America – are operating from Asia, but experts are in disagreement over whether they may be state-backed influence campaigns or even opportunists trying to make a quick buck.
Monetising ‘rage bait’
In 2024 the
Centre for Information Resilience
(CIR) revealed that a network of accounts on X were posing as young American women, stealing images from European influencers to burnish their credibility. Often these images were manipulated to include pro-Trump hats and clothing.
The new location feature on X has allowed Benjamin Strick, who ran the original investigation, to confirm that almost all of these accounts purporting to be “independent Trump supporting” women are located in Thailand.
Strick noted that while promising to “follow patriots” and “stand with Trump”, these accounts often also posted anti-Islamic content too.
In their 2024 report, the CIR found that these accounts exploited “pre-existing societal tensions” in their efforts to spread disinformation.
“Accounts seized upon news stories relating to gender and LGBTQ+ rights, in some cases allowing them to undermine Democratic policies and promote Republican views.”
Fears that foreign actors are using social media to influence US voters reached their zenith in the months after Trump’s 2016 election win over Hillary Clinton. An intelligence assessment the following year detailed the steps that the Russian state took to bolster Trump using bot farms.
In the years since, experts have warned that foreign influence campaigns are becoming more sophisticated, but as America’s politics has become more partisan and voters more siloed, those warnings appear to have been forgotten.
However it’s possible though that the sheer number of pro-Trump accounts around the world might have as much to do with turning a profit as political influence, says Simon Copland, a researcher at the Australian National University.
“Social media is really based on attention … [and] on places like X or Twitter you can get money from that,” he says, adding that at the moment, the best way to get attention “is to be posting about
Donald Trump
.”
Changes to the way X monetises its content could be a factor as well. In 2024, the platform announced that creators would now be paid based on the levels of engagement with their content. At the time, some expressed concern that this would incentivise users to create more and more controversial content.
“When platforms begin to reward engagement, creators will begin posting anything that drives a discussion of some sort, including posts that are designed to enrage users, forcing them to reply or comment,” TechCrunch wrote at the time.
“That’s where things like rage bait come about,” says Copland. “People deliberately induce rage to try to encourage people to go on to the platforms” and engage in the content.
The calculations used to determine a user’s payments remain opaque and it’s not clear how much money overseas users posing as Maga-faithful could be making. A BBC investigation from 2024 suggested that for some, it could be thousands of dollars. Experts in southeast Asia’s disinformation space say such figures could be highly motivating for people in the region.
A 2021 report into southeast Asia’s “disinformation crisis” found that many accounts pushing xenophobic and misogynistic messages to appeal to the US right were not particularly invested ideologically, but “driven by almost purely entrepreneurial motivations.”
The ‘dark corners’ of the internet
While the perpetually online cadre of Trump’s followers erupt in anger over the origins of some accounts – many of which have now been suspended – others have been left questioning why the issue matters at all.
Copland points to the flow of rightwing ideas, and how policies dreamed up in dank corners of the internet can make their way to the heights of US and European politics.
On the night that X began to reveal the location of accounts, Donald Trump shared a post from an account called Trump_Army_. With nearly 600,000 followers, the account regularly amplifies conspiracy theories; in a recent post it asked its followers if “JFK was killed for trying to expose the same crooks Trump is now exposing”. Soon after, another user pointed out that Trump_Army_ was based in India.
It’s among the more innocuous examples, but illustrative of the way the wider ecosystem of right-wing politics operates online.
“Extreme ideas start in these dark corners of the internet. They spread, they become memes, they go on to more mainstream platforms and then you see politicians pick them up,” says Copland. ‘
In May, Trump
ambushed South African president Cyril Ramaphosa
in the Oval Office, accusing him of turning a blind eye to a “white genocide” against South African farmers. These widely discredited claims are thought to have in-part originated in far-right chatrooms.
“We have to be taking this stuff seriously,” he warns, because these ideas “are suddenly becoming mainstream.”
X was approached for comment.
Valhalla's Things: PDF Planners 2026
PlanetDebian
blog.trueelena.org
2025-11-27 00:00:00
Posted on November 27, 2025
Tags: madeof:atoms, madeof:bits, craft:bookbinding
A few years ago I wrote some planner generating code to make myself a
custom planner; in November 2023 I generated a few, and posted them
here on the blog...
A few years ago I wrote some
planner generating code
to make myself a
custom planner; in
November 2023
I generated a few, and posted them
here on the blog, in case somebody was interested in using them.
In 2024 I tried to do the same, and ended up being even more late, to
the point where I didn’t generate any (uooops).
I did, however, start to write a Makefile to automate the generation
(and got stuck on the fact that there wasn’t an easy way to deduce the
correct options needed from just the template name); this year, with the
same promptness as in 2023 I got back to the Makefile and finished it, so
maybe next year I will be able to post them early enough for people to
print and bind them? maybe :)
Anyway, these are all of the variants I currently generate, for 2026.
The files with
-book
in the name have been imposed on A4 paper for a
16 pages signature. All of the fonts have been converted to paths, for
ease of printing (yes, this means that customizing the font requires
running the script, but the alternative also had its drawbacks).
Some of the planners include ephemerids and moon phase data: these have
been calculated for the town of Como, and specifically for
geo:45.81478,9.07522?z=17
, because that’s what
everybody
needs, right?
If you need the ephemerids for a different location and can’t run the
script yourself (it depends on pdfjam, i.e. various GB of LaTeX, and a
few python modules such as dateutil, pypdf and jinja2), feel free to
ask: unless I receive too many requests to make this sustainable I’ll
generate them and add them to this post.
I hereby release all the PDFs linked in this blog post under the
CC0
license
.
You may notice that I haven’t decided on a license for the code dump
repository; again if you need it for something (that is compatible with
its unsupported status) other than running it for personal use (for
which afaik there is an implicit license) let me know and I’ll push
“decide on a license” higher on the stack of things to do :D
Finishing the Makefile meant that I had to add a tiny feature to one of
the scripts involved, which required me to add a dependency to
pypdf
:
up to now I have been doing the page manipulations with
pdfjam
, which
is pretty convenient to use, but also uses LaTeX, and apparently not
every computer comes with texlive installed (shocking, I know).
If I’m not mistaken, pypdf can do all of the things I’m doing with
pdfjam, so maybe for the next year I could convert my script to use that
one instead.
But then the planners 2027 will be quick and easy, and I will be able to
publish them
promptly
, right?
Running to the Press
Daring Fireball
daringfireball.net
2025-11-26 23:55:20
Regarding my earlier post on similarities between the 2010 App Store Guidelines and today’s: Notably absent from the current guidelines (I think for a very long time) is the specious but very Jobsian claim that “If you run to the press and trash us, it never helps.” Getting the press on your side is...
(a) Come up with your own ideas. We know you have them, so make
yours come to life. Don’t simply copy the latest popular app
on the App Store, or make some minor changes to another app’s
name or UI and pass it off as your own. In addition to risking
an intellectual property infringement claim, it makes the App
Store harder to navigate and just isn’t fair to your fellow
developers.
(b) Submitting apps which impersonate other apps or services is
considered a violation of the Developer Code of Conduct and
may result in removal from the Apple Developer Program.
(c) You cannot use another developer’s icon, brand, or product
name in your app’s icon or name, without approval from the
developer.
It’s guideline (c) that’s new, but I like guideline (a) here. Not just the intent of it, but the language. It’s clear, direct, and human. It reminds me of the tone of the very early guidelines, when it seemed like Steve Jobs’s voice was detectable in some of them.
In a post back in 2010, I wrote
:
This new document is written in remarkably casual language. For
example, a few bullet items from the beginning:
We have over 250,000 apps in the App Store. We don’t need any
more Fart apps.
If your app doesn’t do something useful or provide some form of
lasting entertainment, it may not be accepted.
If your App looks like it was cobbled together in a few days, or
you’re trying to get your first practice App into the store to
impress your friends, please brace yourself for rejection. We
have lots of serious developers who don’t want their quality
Apps to be surrounded by amateur hour.
We will reject Apps for any content or behavior that we believe
is over the line. What line, you ask? Well, as a Supreme Court
Justice once said, “I’ll know it when I see it”. And we think
that you will also know it when you cross it.
If your app is rejected, we have a Review Board that you can
appeal to. If you run to the press and trash us, it never helps.
Some of that language remains today. Here’s the current guideline for section 4.3:
4.3 Spam [...]
(b) Also avoid piling on to a category that is already saturated;
the App Store has enough fart, burp, flashlight, fortune
telling, dating, drinking games, and Kama Sutra apps, etc.
already. We will reject these apps unless they provide a
unique, high-quality experience. Spamming the store may lead
to your removal from the Apple Developer Program.
I could be wrong, but my sense is that Apple has, without much fanfare, cracked down on scams and rip-offs in the App Store. That doesn’t mean there’s none. But it’s like crime in a city: a low amount of crime is the practical ideal, not zero crime. Maybe Apple has empowered something like the “
bunco squad
” I’ve wanted for years? If I’m just unaware of blatant rip-offs running wild in the App Store, send examples my way.
★
Wednesday, 26 November 2025
Tesla's European sales tumble nearly 50% in October
Tesla's (
TSLA
) Europe woes are only getting worse.
According to the
European Automobile Manufacturers' Association
(ACEA), Tesla electric vehicle registrations (a proxy for sales) in Europe fell to just 6,964 units in October, a 48.5% drop compared to a year ago. Meanwhile, total EV registrations in the region, which includes the UK and the European Free Trade Association, rose 32.9% in October, with overall registrations regardless of powertrain up 4.9%.
October's total marks the 10th straight month of declining Tesla sales in Europe. Meanwhile, the overall market share of EVs in the broader European region grew to 16.4%.
Tesla's sales hangover rolled on in certain key European territories, with the introduction of the revamped Model Y not enough to blunt the effect of rising competition and CEO Elon Musk's deep unpopularity.
October's sales slide follows a rough 2025 for Tesla year to date in broader Europe.
In the first 10 months of the year, Tesla sales dropped 29.6% to 180,688 units, per the ACEA. Conversely, Tesla's overall market share in Europe dropped to 1.6% from 2.4% a year ago.
Meanwhile, Tesla's Chinese competitor BYD (
BYDDY
), which sells a mix of pure EVs and hybrids, reported sales jumping 207% to 17,470 units sold in Europe. Another major China rival, SAIC, saw sales climb 46% to just under 24,000 vehicles sold.
While weakening sales in a key, EV-centric region should be a concern, it hasn't been a significant issue for Tesla stock.
The interior of the new Tesla Model 3 with Full Self-Driving activated, highlighting the advanced autonomous driving system and design of Tesla's electric vehicles, in Bari, Italy, on Sept. 6, 2025. (Matteo Della Torre/NurPhoto via Getty Images)
·
NurPhoto via Getty Images
"One of the reasons we called Tesla a 'must own' in our recent launch — despite all the obvious risks — is that the world is about to change, dramatically," analyst Rob Wertheimer wrote. "Autonomy is coming very soon, and it will change everything about the driving ecosystem.”
The main spark appears to be the latest version of Tesla's full self-driving (FSD) software, which is available in the US and select territories.
While investors own Tesla stock mostly for the AI and autonomous potential, there could be good news from the self-driving front for European buyers.
The Netherlands RDW automotive governing body said it has set up a schedule allowing Tesla to demonstrate in February
whether FSD meets requirements
but has not approved it yet.
Getting at least one automotive regulator in Europe to approve FSD would be a huge step in the right direction for Tesla and may help staunch the sales slide in the region.
The Babushka Lady was seen to be holding a camera by eyewitnesses and was also seen in film accounts of the assassination.
[
1
]
[
2
]
She was observed standing on the grass between Elm and Main streets, standing amongst onlookers in front of the Dallas County Building, and is visible in the
Zapruder film
, as well as in the films of
Orville Nix
,
[
3
]
Marie Muchmore
, and Mark Bell,
[
4
]
44 minutes and 47 seconds into the Bell film; even though the shooting had already taken place and most of her surrounding witnesses took cover, she can be seen still standing with the camera at her face. After the shooting, she crossed Elm Street and joined the crowd that went up the
grassy knoll
.
The Babushka Lady is last seen in photographs walking east on Elm Street. Neither she, nor the film she may have taken, have ever been positively identified. Her first appearance on film chronologically is on the sidewalk in front of the Dallas County Building, which is visible in an image as being on Kennedy's right. She would have crossed Houston Street and onto Dealey Plaza in order to be visible in the Dealey Plaza images. This may imply that the images show two different women of similar appearance. It is plausible that once the motorcade passed by she was able to cross the street to catch a second motorcade drive past on Dealey Plaza where she would be on Kennedy's left.
In 1970, a woman named Beverly Oliver told conspiracy researcher Gary Shaw at a church
revival meeting
in
Joshua, Texas
, that she was the Babushka Lady.
[
5
]
Oliver stated that she filmed the assassination with a Super 8 film
Yashica
and that she turned the undeveloped film over to two men who identified themselves to her as FBI agents.
[
5
]
According to Oliver, she obtained no receipt from the men, who told her that they would return the film to her within ten days. She did not follow up with an inquiry.
[
5
]
Oliver reiterated her claims in the 1988 documentary
The Men Who Killed Kennedy
.
[
5
]
According to
Vincent Bugliosi
, Oliver has "never proved to most people's satisfaction that she was in Dealey Plaza that day".
[
5
]
Confronted with the fact that the Yashica Super-8 camera was not made until 1969, she stated that she received the "experimental" camera from a friend and was not sure the manufacturer's name was on it.
[
5
]
Oliver's claims were the basis for a scene in
Oliver Stone
's 1991 film
JFK
, in which a character named "Beverly" meets
Jim Garrison
in a Dallas nightclub.
[
6
]
Played by
Lolita Davidovich
, she is depicted in the
director's cut
as wearing a headscarf at Dealey Plaza and speaking of having given the film she shot to two men claiming to be FBI agents.
In March 1979, the Photographic Evidence Panel of the
United States House Select Committee on Assassinations
indicated that they were unable to locate any film attributed to the Babushka Lady.
[
7
]
According to their report: "Initially,
Robert Groden
, a photographic consultant to the committee advised the panel as to pertinent photographic issues and related materials. Committee investigators located many of the suggested films and photographs, however, some items were never located, i.e. the Babushka Lady film, a color photograph by Norman Similas, and the original negative of the Betzner photograph."
[
7
]
Public hearings of the Assassination Records Review Board
On November 18, 1994, assassination researcher Gary Mack testified before the
Assassination Records Review Board
that he had recently been told by an executive in
Kodak
's Dallas office that a woman in her early 30s with brunette hair brought in film purported to be of the assassination scene while they were processing the
Zapruder film
.
[
8
]
According to Mack, the executive said the woman explained to federal investigators already at the film processing office that she ran from Main Street across the grass to Elm Street where she stopped and snapped a photo with some people in the foreground of the
presidential limousine
and the
Texas School Book Depository
.
[
8
]
Mack said that he was told by the Kodak executive that the photo was extremely blurry and "virtually useless" and indicated that the woman likely went home without anyone recording her identity.
[
8
]
After suggesting that the woman in the story may have been the Babushka Lady, Mack then told the Board: "I do not believe that Beverly Oliver is the Babushka Lady, or, let me rephrase that, she certainly could be but the rest of the story is a fabrication."
[
8
]
Also appearing that same day before the ARRB as "Beverly Oliver Massegee", Oliver stated that she was 17 years old at the time of the assassination.
[
8
]
She told the Board that she was filming with an "experimental"
8 mm movie camera
approximately 20 to 30 feet (6 to 9 m) from Kennedy when he was shot and that the film was confiscated by a man who identified himself as an FBI agent.
[
8
]
According to Oliver, she handed over the camera because the man was an authority figure and because she feared being caught in possession of
marijuana
.
[
8
]
Oliver's claims were addressed point by point and debunked by conspiracy theory researcher John McAdams.
[
9
]
When two of the most influential people in AI both say that
today’s large language models are hitting their limits
, it’s worth paying attention.
In a recent long-form interview,
Ilya Sutskever
– co-founder of OpenAI and now head of Safe Superintelligence Inc. – argued that the industry is moving from an
“age of scaling”
to an
“age of research”
. At the same time,
Yann LeCun
, VP & Chief AI Scientist at Meta, has been loudly insisting that
LLMs are not the future of AI at all
and that we need a completely different path based on “world models” and architectures like
JEPA
.
As developers and founders, we’re building products right in the middle of that shift.
This article breaks down Sutskever’s and LeCun’s viewpoints and what they mean for people actually shipping software.
1. Sutskever’s Timeline: From Research → Scaling → Research Again
Sutskever divides the last decade of AI into three phases:
1.1. 2012–2020: The first age of research
This is the era of “try everything”:
convolutional nets for vision
sequence models and attention
early reinforcement learning breakthroughs
lots of small experiments, new architectures, and weird ideas
There
were
big models, but compute and data were still limited. The progress came from
new concepts
, not massive clusters.
1.2. 2020–2025: The age of scaling
Then scaling laws changed everything.
The recipe became:
More data + more compute + bigger models = better results.
You didn’t have to be extremely creative to justify a multi-billion-dollar GPU bill. You could point to a curve: as you scale up parameters and tokens, performance climbs smoothly.
This gave us:
GPT-3/4 class models
state-of-the-art multimodal systems
the current wave of AI products everyone is building on
1.3. 2025 onward: Back to an age of research (but with huge computers)
Now Sutskever is saying that
scaling alone is no longer enough
:
The industry is already operating at
insane scale
.
The internet is finite, so you can’t just keep scraping higher-quality, diverse text forever.
The returns from “just make it 10× bigger” are getting smaller and more unpredictable.
We’re moving into a phase where:
The clusters stay huge, but
progress depends on new ideas
, not only new GPUs.
2. Why the Current LLM Recipe Is Hitting Limits
Sutskever keeps circling three core issues.
2.1. Benchmarks vs. real-world usefulness
Models look god-tier on paper:
they pass exams
solve benchmark coding tasks
reach crazy scores on reasoning evals
But everyday users still run into:
hallucinations
brittle behavior on messy input
surprisingly dumb mistakes in practical workflows
So there’s a gap between
benchmark performance
and
actual reliability
when someone uses the model as a teammate or co-pilot.
2.2. Pre-training is powerful, but opaque
The big idea of this era was: pre-train on enormous text + images and you’ll learn “everything”.
It worked incredibly well… but it has downsides:
you don’t fully control
what
the model learns
when it fails, it’s hard to tell if the issue is data, architecture, or something deeper
pushing performance often means
more of the same
, not better understanding
That’s why there’s so much focus now on
post-training
tricks: RLHF, reward models, system prompts, fine-tuning, tool usage, etc. We’re papering over the limits of the pre-training recipe.
2.3. The real bottleneck: generalization
For Sutskever, the biggest unsolved problem is
generalization
.
Humans can:
learn a new concept from a handful of examples
transfer knowledge between domains
keep learning continuously without forgetting everything
Models, by comparison, still need:
huge amounts of data
careful evals to avoid weird corner-case failures
extensive guardrails and fine-tuning
Even the best systems today
generalize much worse than people
. Fixing that is not a matter of another 10,000 GPUs; it needs new theory and new training methods.
3. Safe Superintelligence Inc.: Betting on New Recipes
Sutskever’s new company,
Safe Superintelligence Inc. (SSI)
, is built around a simple thesis:
scaling was the driver of the last wave
research will drive the next one
SSI is not rushing out consumer products. Instead, it positions itself as:
focused on
long-term research into superintelligence
trying to invent
new training methods and architectures
putting
safety and controllability
at the core from day one
Instead of betting that “GPT-7 but bigger” will magically become AGI, SSI is betting that
a different kind of model
, trained with different objectives, will be needed.
4. Have Tech Companies Overspent on GPUs?
Listening to Sutskever, it’s hard not to read between the lines:
Huge amounts of money have gone into GPU clusters on the assumption that scale alone would keep delivering step-function gains.
We’re discovering that the
marginal gains from scaling
are getting smaller, and progress is less predictable.
That doesn’t mean the GPU arms race was pointless. Without it, we wouldn’t have today’s LLMs at all.
But it does mean:
The next major improvements will likely come from
smarter algorithms
, not merely
more expensive hardware
.
Access to H100s is slowly becoming a
commodity
, while genuine innovation moves back to
ideas and data
.
For founders planning multi-year product strategies, that’s a big shift.
5. Yann LeCun’s Counterpoint: LLMs Aren’t the Future at All
If Sutskever is saying “scaling is necessary but insufficient,”
Yann LeCun
goes further:
LLMs, as we know them, are not the path to real intelligence.
He’s been very explicit about this in talks, interviews and posts.
5.1. What LeCun doesn’t like about LLMs
LeCun’s core criticisms can be summarized in three points:
Limited understanding
LLMs are great at manipulating text but have a
shallow grasp of the physical world
.
They don’t truly “understand” objects, physics or causality – all the things you need for real-world reasoning and planning.
A product-driven dead-end
He sees LLMs as an amazing product technology (chatbots, assistants, coding helpers) but believes they are
approaching their natural limits
.
Each new model is larger and more expensive, yet delivers smaller improvements.
Simplicity of token prediction
Under the hood, an LLM is just predicting the next token. LeCun argues this is a
very narrow, simplistic proxy for intelligence
.
For him, real reasoning can’t emerge from next-word prediction alone.
5.2. World models and JEPA
Instead of LLMs, LeCun pushes the idea of
world models
– systems that:
learn by watching the world (especially video)
build an internal representation of objects, space and time
can
predict what will happen next
in that world, not just what word comes next
One of the architectures he’s working on is
JEPA – Joint Embedding Predictive Architecture
:
it learns representations by predicting future embeddings rather than raw pixels or text
it’s designed to scale to complex, high-dimensional input like video
the goal is a model that can support
persistent memory, reasoning and planning
5.3. Four pillars of future AI
LeCun often describes four pillars any truly intelligent system needs:
Understanding of the physical world
Persistent memory
Reasoning
Planning
His argument is that today’s LLM-centric systems mostly
hack around
these requirements instead of solving them directly. That’s why he’s increasingly focused on world-model architectures instead of bigger text models.
6. Sutskever vs. LeCun: Same Diagnosis, Different Cure
What’s fascinating is that Sutskever and LeCun
agree on the problem
:
current LLMs and scaling strategies are
hitting limits
simply adding more parameters and data is delivering
diminishing returns
new ideas are required
Where they differ is
how radical the change needs to be
:
Sutskever
seems to believe that the next breakthroughs will still come from the same general family of models – big neural nets trained on massive datasets – but with better objectives, better generalization, and much stronger safety work.
LeCun
believes we need a
new paradigm
: world models that learn from interaction with the environment, closer to how animals and humans learn.
For people building on today’s models, that tension is actually good news: it means there is still a lot of frontier left.
7. What All This Means for Developers and Founders
So what should you do if you’re not running an AI lab, but you
are
building products on top of OpenAI, Anthropic, Google, Meta, etc.?
7.1. Hardware is becoming less of a moat
If the next big gains won’t come from simply scaling, then:
the advantage of “we have more GPUs than you” decreases over time
your real edge comes from
use cases, data, UX and integration
, not raw model size
This is good for startups and agencies: you can piggyback on the big models and still differentiate.
7.2. Benchmarks are not your product
Both Sutskever’s and LeCun’s critiques are a warning against obsessing over leaderboards.
Ask yourself:
Does this improvement meaningfully change what my users can do?
Does it reduce hallucinations in
their
workflows?
Does it make the system more reliable, debuggable and explainable?
User-centric metrics matter more than another +2% on some synthetic reasoning benchmark.
7.3. Expect more diversity in model types
If LeCun’s world models, JEPA-style architectures, or other alternatives start to work, we’ll likely see:
specialized models for
physical reasoning and robotics
LLMs acting as a
language interface
over deeper systems that actually handle planning and environment modeling
more hybrid stacks, where multiple models collaborate
For developers, that means learning to
orchestrate multiple systems
instead of just calling one chat completion endpoint.
7.4. Data, workflows and feedback loops are where you win
No matter who is right about the far future, one thing is clear for product builders:
Owning
high-quality domain data
Designing
tight feedback loops
between users and models
Building
evaluations that match your use case
…will matter more than anything else.
You don’t need to solve world modeling or superintelligence yourself. You need to:
pick the right model(s) for the job
wrap them in workflows that make sense for your users
keep improving based on real-world behavior
8. A Quiet Turning Point
In 2019–2021, the story of AI was simple:
“scale is all you need.”
Bigger models, more data, more GPUs.
Now, two of the field’s most influential figures are effectively saying:
scaling is
not enough
(Sutskever)
LLMs themselves may be a
dead end for real intelligence
(LeCun)
We’re entering a new phase where research, theory and new architectures matter again as much as infrastructure.
For builders, that doesn’t mean you should stop using LLMs or pause your AI roadmap. It means:
focus less on chasing the next parameter count
focus more on
how
intelligence shows up inside your product: reliability, reasoning, planning, and how it fits into real human workflows
The GPU race gave us today’s tools. The next decade will be defined by what we
do
with them – and by the new ideas that finally move us beyond “predict the next token.”
Last week Italy’s metal workers secured a major victory as the unions Fim, Fiom and Uilm, all affiliated to IndustriALL Global Union, signed the renewed National Collective Labour Agreement (NCLA) with Federmeccanica and Assistal after four days of continuous and intense negotiations. The agreement covers more than 1.5 million workers across the country and guarantees a €205(US$ 237.17) increase on minimum contractual salaries over four years, which the unions say is essential to protecting wages amid rising living costs and economic uncertainty.
In June 2025, FIOM, FIM and UILM
staged an eight-hour strike accompanied
by regional demonstrations across Italy, calling out what they described as the employers’ irresponsible refusal to negotiate. Workers across the sector, including those in small and medium-sized enterprises, joined the strike action and additional measures, such as overtime and flexibility blockades, were enforced. Demonstrations sent a clear and unified message: workers would not accept stagnation, wage erosion or further delays. The strike movement strengthened union resolve and demonstrated to employers that metal workers were mobilized, united and prepared to continue the fight to defend purchasing power and secure fair working conditions.
Union negotiators have described this as a crucial victory that ensures long-term wage defence at a moment when many families are facing mounting financial strain. The revised wage structure maintains a system designed to safeguard purchasing power through inflation. The agreement also includes an additional salary quota and a safeguard clause should inflation surge beyond forecasts during the contract period.
General secretaries Ferdinando Uliano, Michele De Palma and Rocco Palombella, from Fim, Fiom and Uilm, said the contract represents not only a negotiation victory, but also the defence of Italy’s national collective bargaining system itself. They emphasized the unity and resolve of the unions throughout the process:
“It was a very hard negotiation, but we closed the gap and signed a strong contract. We protected the purchasing power of metal workers and strengthened rights and protections. The wage increase, the start of a working-time reduction trial and the stabilization of precarious work were our pillars and we achieved them. Today, we can say we saved the national contract, which has never stopped being challenged. This agreement ensures dignity for those who build the industrial heart of Italy. Metal workers are once again writing the history of this country at a time when it needs stability, courage and lasting solutions.”
The contract delivers significant gains in the fight against job insecurity and precarious work, issues that have been central to the unions’ platform. Employers will now be required to stabilize a share of fixed-term workers after 12 months if they wish to extend temporary contracts under specific conditions. Workers employed through staffing agencies will gain the right to permanent employment at the host company after 48 months, an important shift toward fairer and more secure employment for thousands of metal workers.
The agreement also introduces forward-looking changes, including a structured trial to reduce working hours under the guidance of a dedicated commission. Additional improvements include stronger health and safety protections, expanded rights to workplace training, enhanced safeguards for seriously ill and disabled workers and new provisions specifically aimed at preventing violence against women.
IndustriALL general secretary, Atle Høie, praised the agreement and the determination of the Italian unions:
“This is an important victory not only for metal workers in Italy, but for workers everywhere who are fighting against insecurity, declining wages and the erosion of fundamental rights. By securing real wage protection, pathways to stable employment and groundbreaking progress on working-time reduction, Fim, Fiom and Uilm have shown what strong, united unions can achieve. This agreement sends a clear message: collective bargaining remains one of the most powerful tools workers have to build fairer, safer and more dignified workplaces.”
Earlier this year I
demoed
iOS 6 running on an iPod touch 3 - a device that Apple never gave iOS 6 to, making iOS 5.1.1 the latest build it can run
A few months later I also released a
script
that generates an iOS 6 restore image installable on that iPod touch model
This article describes technical details behind this work. Certain proficiency in iOS internals is assumed
I'll show you what iOS is made of
First of all, let's recap what software components iOS consists of:
iBoot
- the bootloader. Has 4 different types for different scenarios - iBSS, iBEC, LLB and iBoot
Kernelcache
- the OS kernel + kernel extensions (drivers) built into a single binary blob
DeviceTree
- structured list of hardware used by specific device model + some parameters that specify software behavior. The copy included in an IPSW is more of a template that is heavily modified by iBoot before jumping into kernel
Userspace filesystem - tiny
restore ramdisk
used purely for OS installation or the actual
root filesystem
of iOS installed persistently
Various firmwares for coprocessors, be they internal or external to the main SoC - like, baseband, Wi-Fi, Bluetooth, multitouch and etc.
iPhone 3GS tests
iPhone 3GS was released the same year as iPod touch 3 (2009), and has a very similar hardware (
S5L8920X
SoC vs.
S5L8922X
). But the most important part is that it actually got iOS 6 officially
Before doing anything on the iPod I decided to try to boot iOS 6.0 with iOS 5.1.1 iBoot & DeviceTree on the iPhone and see what's gonna break and how
DeviceTree
The most broken thing was DeviceTree - iOS 6 added a lot of new nodes and properties. To fix it in automated manner I wrote a stupid Python script that decodes and computes a diff between 2 DeviceTrees. Such diff can also be applied to another DeviceTree
As I mentioned above a lot of things in a DeviceTree is filled by iBoot at runtime. One of such new properties is
nvram-proxy-data
in
chosen
node
The property must contain a raw NVRAM dump - leaving it empty will make kernel get stuck somewhere very early
For iPod touch 3 I also had to clean-up the diff out of iPhone-specific things before applying it to iPod's 5.1.1 DeviceTree
iBoot
iBoot didn't require any major changes in this case. Just typical Image3 signature check patch, boot-args injection and
debug-enabled
patch so kernel is going to actually respect AMFI boot-args
One important thing is to actually populate
nvram-proxy-data
dynamically, at least for normal boots (aka non-restore). Restore boot will be fine with some random NVRAM hardcoded into DeviceTree, but normal one will overwrite your actual NVRAM with the random one if it decides to sync it at some point
I do it by replacing a call to
UpdateDeviceTree()
with my own little function that calls the real
UpdateDeviceTree()
, but also populates actual
nvram-proxy-data
and
random-seed
(this one shouldn't be of any importance)
For boot-args I always add
amfi=0xff
to disable code-signing, but that's pretty cannonical as well
Please note that other iBoot+kernel combos might require more changes - if you ever try something and it doesn't work, I recommend looking into DeviceTree differences (both the initial template and how iBoot fills it) and also
boot_args
structure iBoot passes to kernel (not to be confused with boot-args
string
, the
boot_args
structure
is a different thing)
Kernelcache
The most complex part. iPod touch 3 never got iOS 6 officialy, yes, but it was rumored that initially it was meant to have it, but Apple's marketing team said no. Either way, almost every internal iOS 6 build got both standalone S5L8922X kernel and even standalone kexts (including ones specific to iPod touch 3)
The question is how to load them all simultaneously. My initial idea was to do it just as older Mac OS X could do - load all kexts dynamically on bootloader level. Long story short, my strategy was the following:
In iBoot context, load all kexts from filesystem - binary itself + Info.plist
Lay them out in memory and add corresponding entries to
chosen/memory-map
node of DeviceTree
Boot standalone kernel which will then pick them up and load
The sad outcome:
panic(cpu 0 caller 0x802e5223): "kern_return_t kxld_link_file(KXLDContext *, u_char *, u_long, const char *, void *, KXLDDependency *, u_int, u_char **, kxld_addr_t *) (com.apple.kec.corecrypto) called in kernel without kxld support"
The kernel has all the code to pick them up, but not to actually link...
Glueing a prelinked kernelcache
So creating a legit kernelcache is the only way after all. I was already imagining all the horrors of writing software to parse and apply
LINKEDIT
and etc., but then it occured to me! Mac OS X (before Apple Silicon) was generating such kernelcaches somehow! What if we use that logic to build our iOS kernelcache?
I used
/usr/local/bin/kcgen
from internal Sierra build (can be found online as "Phoenix A1708.dmg"), but it seems that even latest macOS
kextcache
can do it (included by default)
Here is a breakdown of the options:
-c output.bin
- output file to write resulting kernelcache to
$(cat n18.10A403.kextlist | sed 's/^/--bundle-id /')
- this weird expression appends
--bundle-id
to every line from the file at
n18.10A403.kextlist
. This is to specify which kexts we'd like to include. How I created such list is described below
-arch armv7
- obviously only build armv7 slice
-all-personalities
- very important flag that prevents
irrelevant
IOKit personalities to be stripped. "Irrelevant" as in "irrelevant to current machine", meaning everything
relevant
to iPod touch 3 is going to be stripped
-strip-symbols
- strips unnecessary symbols. This flag can be omitted theoretically, but I recommend keeping it to make resulting kernelcache smaller
-uncompressed
- do not apply compression. Since we'll have to change one little thing later, compression would have to be reapplied anyway
--
means the rest of the args will point to directories to grab kexts from
kernels_kexts_10A63970m/Extensions
is a path to a folder containing kexts
The little thing to do is to remove fat header. For some reason, it creates a fat Mach-O with a single slice. iBoot doesn't like it, so let's strip it:
lipo -thin armv7 output.bin -o output.thin.bin
The kernel cache is ready now! Just needs to be compressed and packaged into Image3 container
About kext lists
Once again I compared iPhone 3GS' iOS 5.1.1 vs. 6.0 - some kexts were added, some removed, some changed their bundle IDs, some were irrelevant for iPod touch 3
Do not forget to include the pseudo-extensions as well!
In this specific case I had to patch up Info.plist of the Wi-Fi kext. As always there is a sample in the
repo
Restore ramdisk filesystem
Pretty cannonical here. I patched
asr
as usual and also had to move
options.n88.plist
to
options.n18.plist
so it can lay out partitions properly
However, I also have to install the iBoot exploit. To do that I reimplement
rc.boot
binary:
Remount ramdisk and set
umask
just like the original one does
Call
restored_external
, but with
-server
argument, so it doesn't reboot after finishing restore
If restore was completed properly, I add a third partition, write the exploit there and set
boot-partition
to
2
Reboot the device
My implementation is available guess where? Yes, in the
repository
Root filesystem
This needed a lot of changes:
Add matching SpringBoard's hardware feature plist (
/System/Library/CoreServices/SpringBoard.app/N18AP.plist
in this case)
I took the iOS 5.1.1 variant as a base and added iOS 6 specific capabilities
I tried to keep
original
enough Home screen icon order by
merging
iPod touch 3 iOS 5.1.1 and iPod touch 4 6.x layouts
Add multitouch & Wi-Fi firmwares
I use versions from 5.1.1
Add Bluetooth firmware and scripts
This is more complicated, as those are all hardcoded into
/usr/sbin/BlueTool
Luckily, they can also be overriden by files in
/etc/bluetool
- as always check my code for reference
I extracted both firmware and scripts from 5.1.1
BlueTool
FairPlay
daemon is limited to
N88AP
(iPhone 3GS)
It has
LimitLoadToHardware
key in its' LaunchDaemon plist
But if we simply remove the key, it works on iPod touch 3 as well
This is important, because otherwise we cannot activate device through Apple's servers
This trick will be harder to pull off on iOS 6.1+ because they load LaunchDaemons from a signed cache. Still can be bypassed in many ways - for instance, patching
launchd
or forcefully loading another plist via
launchctl
DYLD shared cache patches
Product ID map patch
iOS 6 brings a concept of "product ID" in the form of a long byte sequence
It is filled by iBoot into
product
node of DeviceTree (which didn't even exist before)
I hardcode the value of iPhone 3GS straight into DeviceTree (
8784AE8D7066B0F0136BE91DCFE632A436FFD6FB
)
There is also a short form of this identifier - 16-bit integer - which existed before iOS 6
iPhone 3GS is
0x2714
and the iPod is
0x2715
MobileGestalt
framework has a table that matches the short form by the long one - I swap
0x2714
with
0x2715
there
I believe it's better for iTunes and etc.
getDeviceVariant()
patch
MobileGestalt
once again messes us up our business
Device variant
is a letter - usually "A" or "B"
It seems to depend on Wi-Fi transciever vendor used in exact device (?)
iOS 6 fails miserably to determine this value for iPod touch 3
This crashes activation process, for example
To fix it, I patch the function to always return "A" (in form of
CFString
)
Fixing code signature
This is much easier than most people think
Shared cache files have the same format of signature as normal Mach-Os
And since it's just ad-hoc, all you need to do is to recalculate SHA-1 hash for pages you modified and update the signature
So easy, it can be done with just a hex-editor
The iBoot exploit
iOS 5 iBoot had a bug in
HFS+
filesystem driver. I did make an exploit many years ago but it was
bad
. Like, truly
bad
. I reimplemented it from scratch for this project making it deterministic (hopefully...)
This subject probably deserves a separate article
Conclusion & future plans
This was not easy to do, and yet easier than I expected initially
After releasing the tool many people asked me about jailbreaking. The old tools are not going to work, but it should be easy to just patch the kernel and drop Cydia tarball onto the filesystem. I guess I will give it a try later
There was another device that Apple dropped support for in that year - iPad 1. I will try that soon enough as well
I hope that the information from this write-up will help you making other crazy combinations, like iOS 4 on iPhone 4S or iOS 5 on iPad mini 1
PyPI and Shai-Hulud: Staying Secure Amid Emerging Threats
An attack on the npm ecosystem continues to evolve, exploiting compromised accounts to publish malicious packages.
This campaign, dubbed
Shai-Hulud
, has targeted large volumes of packages in the JavaScript ecosystem,
exfiltrating credentials to further propagate itself.
PyPI has not been exploited
, however some PyPI credentials were found exposed in compromised repositories.
We've revoked these tokens as a precaution, there's no evidence they have been used maliciously.
This post raises awareness about the attack and encourages proactive steps to secure your accounts,
especially if you're using build platforms to publish packages to PyPI.
How does this relate to PyPI?
This week, a security researcher disclosed long-lived PyPI credentials exposed as part of the Shai-Hulud campaign.
The credentials were found in GitHub repositories (stored as repository secrets), and were still valid.
We saw an attack with insecure workflow settings for
Ultralytics in 2024
.
While the campaign primarily targets npm, some projects use
monorepo
setups,
publishing both JavaScript packages to npmjs.com and Python packages to PyPI from the same repository.
When attackers compromise these repositories, they can extract credentials for multiple platforms.
We investigated the reported credentials and found they were associated with accounts that hadn't published recently.
We've revoked these credentials and reached out to affected users to advise them to rotate any remaining tokens.
What can I do to protect my PyPI account?
Here are security practices to protect your PyPI account:
Use Trusted Publishing:
If you are using a build platform to publish packages to PyPI,
consider using a
Trusted Publisher
.
This eliminates the need to manage long-lived authentication tokens, reducing the risk of credential exposure.
Trusted Publishing uses short-lived, scoped tokens for each build, minimizing the impact of any potential compromise.
This approach has
risen in popularity
,
with other registries like
Crates.io
,
RubyGems
,
and
npmjs.com
adopting similar models.
When using GitHub Actions, consider layering in additional security measures,
like requiring human approval via
GitHub Environments
before publishing.
This blog post from pyOpenSci
has detailed guidance on adding manual review steps to GitHub Actions workflows.
Audit your workflows for misconfiguration:
Review your GitHub Actions workflows for any potential security issues.
Tools like
zizmor
and
CodeQL
can help identify vulnerabilities in your CI/CD pipelines.
Adopt scanning as automated actions for the repository to catch future issues.
Review your account activity:
Regularly check your PyPI account activity for any unauthorized actions.
If you notice any suspicious activity,
report it to the PyPI security team
immediately.
Taking any of these steps helps mitigate the risk of compromise and keeps packages secure.
References
Some blog posts covering the attack behaviors and mitigation steps:
EFF to Arizona Federal Court: Protect Public School Students from Surveillance and Punishment for Off-Campus Speech
Electronic Frontier Foundation
www.eff.org
2025-11-26 22:33:54
Legal Intern Alexandra Rhodes contributed to this blog post.
EFF filed an amicus brief urging the Arizona District Court to protect public school students’ freedom of speech and privacy by holding that the use of a school-issued laptop or email account does not categorically mean a student is “on c...
Legal Intern Alexandra Rhodes contributed to this blog post.
EFF filed an
amicus brief
urging the Arizona District Court to protect public school students’ freedom of speech and privacy by holding that the use of a school-issued laptop or email account does not categorically mean a student is “on campus.” We argued that students need private digital spaces beyond their school’s reach to speak freely, without the specter of constant school surveillance and punishment.
Surveillance Software Exposed a Bad Joke Made in the Privacy of a Student’s Home
The case,
Merrill v. Marana Unified School District
, involves a Marana High School student who, while at home one morning before school started, asked his mother for advice about a bad grade he received on an English assignment. His mother said he should talk to his English teacher, so he opened his school-issued Google Chromebook and started drafting an email. The student then wrote a series of jokes in the draft email that he deleted each time. The last joke stated: “GANG GANG GIMME A BETTER GRADE OR I SHOOT UP DA SKOOL HOMIE,” which he narrated out loud to his mother in a silly voice before deleting the draft and closing his computer.
Within the hour, the student’s mother received a phone call from the school principal, who said that Gaggle surveillance software had flagged a threat from her son and had sent along the screenshot of the draft email. The student’s mother attempted to explain the situation and reassure the principal that there was no threat. Nevertheless, despite her reassurances and the student’s lack of disciplinary record or history of violence, the student was ultimately suspended over the draft email—even though he was physically off campus at the time, before school hours, and had never sent the email.
After the student’s suspension was unsuccessfully challenged, the family
sued the school district
alleging infringement of the student’s right to free speech under the First Amendment and violation of the student’s right to due process under the Fourteenth Amendment.
Public School Students Have Greater First Amendment Protection for Off-Campus Speech
The U.S. Supreme Court has addressed the First Amendment rights of public school students in a
handful of cases
.
Most notably, in
Tinker v. Des Moines Independent Community School District
(1969), the Court held
that students may not be punished for their
on-campus
speech unless the speech “materially and substantially” disrupted the school day or invaded the rights of others.
Decades later, in
Mahanoy Area School District v. B.L. by and through Levy
(2021)
,
in which
EFF filed a brief
,
the Court further held that schools have less leeway to regulate student speech when that speech occurs off campus. Importantly, the Court stated that schools should have a limited ability to punish off-campus speech because “from the student speaker’s perspective, regulations of off-campus speech, when coupled with regulations of on-campus speech, include all the speech a student utters during the full 24-hour day.”
The Ninth Circuit has further held that off-campus speech is only punishable if it bears a “
sufficient nexus
” to the school and poses a credible threat of violence.
In this case, therefore, the extent of the school district’s authority to regulate student speech is tied to whether the high schooler was
on or off campus
at the time of the speech. The student here was at home and thus physically off campus when he wrote the joke in question; he wrote the draft before school hours; and the joke was not emailed to anyone on campus or anyone associated with the campus.
Yet the school district is arguing that his use of a school-issued Google Chromebook and Google Workspace for Education account (including the email account) made his speech—and makes all student speech—automatically “on campus” for purposes of justifying punishment under the First Amendment.
Schools Provide Students with Valuable Digital Tools—But Also Subject Them to Surveillance
EFF supports the plaintiffs’ argument that the student’s speech was “off campus,” did not bear a sufficient nexus to the school, and was not a credible threat. In our amicus brief, we urged the trial court at minimum to
reject
a rule that the use of a school-issued device or cloud account always makes a student’s speech “on campus.”
Our amicus brief supports the plaintiffs’ First Amendment arguments through the lens of surveillance, emphasizing that digital speech and digital privacy are inextricably linked.
As we explained, Marana Unified School District, like many schools and districts across the country, offers students free Google Chromebooks and requires them to have an online Google Account to access the various cloud apps in Google Workspace for Education, including the Gmail app.
Marana Unified School District also uses three surveillance technologies that are integrated into Chromebooks and Google Workspace for Education: Gaggle, GoGuardian, and Securly. These surveillance technologies collectively can monitor virtually everything students do on their laptops and online, from the emails and documents they write (or even just
draft
) to the websites they visit.
School Digital Surveillance Chills Student Speech and Further Harms Students
In our amicus brief, we made four main arguments against a blanket rule that categorizes any use of a school-issued device or cloud account as “on campus,” even if the student is geographically off campus or outside of school hours.
First, we pointed out that such a rule will result in students having no reprieve from school authority, which runs counter to the Supreme Court’s admonition in
Mahanoy
not to regulate “all the speech a student utters during the full 24-hour day.” There must be some place that is “off campus” for public school students even when using digital tools provided by schools, otherwise schools will reach too far into students’ lives.
Second, we urged the court to reject such an “on campus” rule
to mitigate the
chilling effect of digital surveillance
on students’ freedom of speech—that is, the risk that students will self-censor and choose not to express themselves in certain ways or access certain information that may be disfavored by school officials. If students know that no matter where they are or what they are doing with their Chromebooks and Google Accounts, the school is watching
and
the school has greater legal authority to punish them because they are always “on campus,” students will undoubtedly curb their speech.
Third, we argued that such an
“on campus” rule
will
exacerbate existing inequities in public schools
among students of different socio-economic backgrounds. It would
distinctly disadvantage lower-income students who are
more likely to rely on school-issued devices
because their families cannot afford a personal laptop or tablet. This creates a
“pay for privacy” scheme
: lower-income students are subject to greater school-directed surveillance and related discipline for digital speech, while wealthier students can limit surveillance by using personal laptops and email accounts, enabling them to have more robust free speech protections.
Fourth,
such an “on campus” rule will incentivize public schools to continue eroding student privacy by subjecting them to near constant digital surveillance. The student surveillance technologies schools use are notoriously
privacy invasive
and
inaccurate
, causing various harms to students—including unnecessary investigations and discipline, disclosure of sensitive
information, and frustrated learning.
We urge the Arizona District Court to protect public school students’ freedom of speech and privacy by rejecting this approach to school-managed technology
. As we said in our brief, students, especially high schoolers,
need some sphere of digital autonomy, free of surveillance, judgment, and punishment,
as much as anyone else—to express themselves, to develop their identities, to learn and explore, to be silly or crude, and even to make mistakes
.
Bring Back Doors – Bring Bathroom Doors Back to Hotels
I’m done. I’m done arriving at hotels and discovering that they have removed the bathroom door. Something that should be as standard as having a bed, has been sacrificed in the name of “aesthetic”.
I get it, you can save on material costs and make the room feel bigger, but what about my dignity??? I can’t save that when you don’t include a bathroom door.
It’s why I’ve built this website, where I compiled hotels that are guaranteed to have bathroom doors, and hotels that need to work on privacy.
I’ve emailed hundreds of hotels and I asked them two things: do your doors close all the way, and are they made of glass? Everyone that says yes to their doors closing, and no to being made of glass has been sorted by price range and city for you to easily find places to stay that are
guaranteed
to have a bathroom door.
Quickly check to see if the hotel you’re thinking of booking has been reported as lacking in doors by a previous guest.
Finally, this passion project could not exist without people submitting hotels without bathroom doors for public shaming. If you’ve stayed at a doorless hotel send me an email with the hotel name to bringbackdoors@gmail.com, or send me a
DM on Instagram
with the hotel name and a photo of the doorless setup to be publicly posted.
Let’s name and shame these hotels to protect the dignity of future travelers.
New ShadowV2 botnet malware used AWS outage as a test opportunity
Bleeping Computer
www.bleepingcomputer.com
2025-11-26 22:24:14
A new Mirai-based botnet malware named 'ShadowV2' has been observed targeting IoT devices from D-Link, TP-Link, and other vendors with exploits for known vulnerabilities. [...]...
A new Mirai-based botnet malware named ‘ShadowV2’ has been observed targeting IoT devices from D-Link, TP-Link, and other vendors with exploits for known vulnerabilities.
Fortinet’s FortiGuard Labs researchers spotted the activity during the major
AWS outage in October
. Although the two incidents are not connected, the botnet was active only for the duration of the outage, which may indicate that it was a test run.
ShadowV2 spread by leveraging at least eight vulnerabilities in multiple IoT products:
Among these flaws, CVE-2024-10914 is a
known-to-be-exploited
command injection flaw impacting EoL D-Link devices, which the vendor announced that it
would not fix
.
Regarding CVE-2024-10915, for which there’s a
NetSecFish report
from November 2024, BleepingComputer initially did not find the vendor's advisory for the flaw. After reaching out to the company, we received confirmation that the issue would not be fixed for the impacted models.
D-Link
updated an older bulletin
to add the particular CVE-ID and
published a new one
referring to the ShadowV2 campaign, to warn users that end-of-life or end-of-support devices are no longer under development and will not receive firmware updates.
CVE-2024-53375, which was also
presented in detail
in November 2024, was reportedly fixed via a beta firmware update.
Various exploits used by ShadowV2
Source: Fortinet
According to FortiGuard Labs researchers, the ShadowV2 attacks originated from 198[.]199[.]72[.]27, and targeted routers, NAS devices, and DVRs across seven sectors, including government, technology, manufacturing, managed security service providers (MSSPs), telecommunications, and education.
The impact was global, with attacks observed in North and South America, Europe, Africa, Asia, and Australia.
The botnet's global impact
Source: Fortinet
The malware identifies itself as "ShadowV2 Build v1.0.0 IoT version," and is similar to the Mirai LZRD variant, the
researchers say
in a report that provides technical details on how ShadowV2 functions.
It is delivered to vulnerable devices through an initial access stage using a downloader script (binary.sh) that fetches it from a server at 81[.]88[.]18[.]108.
Downloader script
Source: Fortinet
It uses XOR-encoded configuration for filesystem paths, User-Agent strings, HTTP headers, and Mirai-style strings.
In terms of functional capabilities, it supports distributed denial-of-service (DDoS) attacks on UDP, TCP, and HTTP protocols, with various flood types for each. The command-and-control (C2) infrastructure triggers these attacks via commands sent to the bots.
DDoS attack trigger
Source: Fortinet
Typically, DDoS botnets make money by renting their firepower to cybercriminals or by directly extorting targets, demanding payments for stopping the attacks. However, it is not yet known who is behind Shadow V2 and what their monetization strategy is.
Fortinet shared indicators of compromise (IoCs) to help identify this emerging threat at the bottom of the report, while warning about the importance of keeping firmware updated on IoT devices.
As MCP (Model Context Protocol) becomes the standard for connecting LLMs to tools and data, security teams are moving fast to keep these new services safe.
This free cheat sheet outlines 7 best practices you can start using today.
Image credit: Pixabay
(Image credit: Photo by Thomas Traasdahl / Ritzau Scanpix / AFP) / Denmark OUT (Photo by THOMAS TRAASDAHL/Ritzau Scanpix/AFP via Getty Images)
The EU Council reached an agreement on the Child Sexual Abuse Regulation
Voluntary chat scanning remains in the bill despite privacy backlash
The Council now prepares to start negotiations with the Parliament
The EU Council has finally reached an agreement on the controversial Child Sexual Abuse Regulation (CSAR) after more than three years of failed attempts.
Nicknamed
Chat Control
by its critics, the agreement has kept cryptographers, technologists, encrypted service providers, and privacy experts alike in turmoil since its inception.
Presidency after presidency, the bill has taken many shapes. But its most controversial feature is an obligation for all messaging service providers operating in the EU – including those using end-to-end-encryption – to scan their users' private chats on the lookout for child sexual abuse material (CSAM).
At the beginning of the month, the Danish Presidency decided to change its approach with a
new compromise text
that makes the chat scanning voluntary, instead. That turned to be a winning move, with the proposal managing to reach an agreement in the Council on Wednesday, November 26, 2025.
Privacy experts are unlikely to celebrate, though. The decision came a few days after a group of scientists wrote yet another open letter warning that the latest text still "
brings high risks to society
." That's after other privacy experts deemed the new proposal a "
political deception
" rather than an actual fix.
The EU Council is now preparing to start negotiations with the European Parliament, hoping to agree on the final terms of the regulation.
What we know about the Council agreement
(Image credit: Pixabay)
As per the
EU Council announcement
, the new law imposes a series of obligations on digital companies. Under the new rules, online service providers will be required to assess how their platforms could be misused and, based on the results, may need to "implement mitigating measures to counter that risk," the Council notes.
The Council also introduces three risk categories of online services. Those deemed to be a high-risk can be forced "to contribute to the development of technologies to mitigate the risks relating to their services." Voluntary scanning also remains in the bill.
A new EU agency is then tasked to oversee the implementation of the new rules.
"I'm glad that the member states have finally agreed on a way forward that includes a number of obligations for providers of communication services to combat the spread of child sexual abuse material," said Danish Minister for Justice, Peter Hummelgaard.
But concerns about how the agreement threatens our digital rights persist, with one person on the forum,
Hacker News
, saying the Danish "government has today turned the EU into a tool for total surveillance, I don't know if there can be any return from."
As trilogue negotiations approach, the ongoing challenge for legislators remains striking the right balance between halting abuse online, without compromising on fundamental rights and strong
encryption
.
Chiara is a multimedia journalist committed to covering stories to help promote the rights and denounce the abuses of the digital side of life – wherever cybersecurity, markets, and politics tangle up. She believes an open, uncensored, and private internet is a basic human need and wants to use her knowledge of VPNs to help readers take back control. She writes news, interviews, and analysis on data privacy, online censorship, digital rights, tech policies, and security software, with a special focus on VPNs, for TechRadar and TechRadar Pro. Got a story, tip-off, or something tech-interesting to say? Reach out to chiara.castro@futurenet.com
November Update to the App Store Review Guidelines
Daring Fireball
developer.apple.com
2025-11-26 21:46:25
Here’s the updated full guideline for section 4.1:
4.1 Copycats
(a) Come up with your own ideas. We know you have them, so make
yours come to life. Don’t simply copy the latest popular app
on the App Store, or make some minor changes to another app’s
name or UI and pass it off as yo...
The
App Review Guidelines
have been revised to support updated policies and to provide clarification. Please review the changes below:
1.2.1(a): This new guideline specifies that creator apps must provide a way for users to identify content that exceeds the app’s age rating, and use an age restriction mechanism based on verified or declared age to limit access by underage users.
2.5.10: This language has been deleted (“Apps should not be submitted with empty ad banners or test advertisements.”).
3.2.2(ix): Clarified that loan apps may not charge a maximum APR higher than 36%, including costs and fees, and may not require repayment in full in 60 days or less.
4.1(c): This new guideline specifies that you cannot use another developer’s icon, brand, or product name in your app’s icon or name, without approval from the developer.
4.7: Clarifies that HTML5 and JavaScript mini apps and mini games are in scope of the guideline.
4.7.2: Clarifies that apps offering software not embedded in the binary may not extend or expose native platform APIs or technologies to the software without prior permission from Apple.
4.7.5: Clarifies that apps offering software not embedded in the binary must provide a way for users to identify content that exceeds the app’s age rating, and use an age restriction mechanism based on verified or declared age to limit access by underage users.
5.1.1(ix): Adds crypto exchanges to the list of apps that provide services in highly regulated fields.
5.1.2(i): Clarifies that you must clearly disclose where personal data will be shared with third parties, including with third-party AI, and obtain explicit permission before doing so.
Google’s Pixel 10 works with AirDrop, and other phones should follow later.
Google's Pixel 10 series now features compatibility with Apple's AirDrop.
Credit:
Ryan Whitwam
Google's Pixel 10 series now features compatibility with Apple's AirDrop.
Credit:
Ryan Whitwam
Last year, Apple
finally added support
for Rich Communications Services (RCS) texting to its platforms, improving consistency, reliability, and
security
when exchanging green-bubble texts between the competing iPhone and Android ecosystems. Today, Google is announcing another small step forward in interoperability, pointing to a slightly less annoying future for friend groups or households where not everyone owns an iPhone.
Google
has updated
Android’s Quick Share feature to support Apple’s AirDrop, which allows users of Apple devices to share files directly using a local peer-to-peer Wi-Fi connection. Apple devices with AirDrop enabled and set to “everyone for 10 minutes” mode will show up in the Quick Share device list just like another Android phone would, and Android devices that support this new Quick Share version will also show up in the AirDrop menu.
Google will only support this feature on the Pixel 10 series, at least to start. The company is “looking forward to improving the experience and expanding it to more Android devices,” but it didn’t announce anything about a timeline or any hardware or software requirements. Quick Share also won’t work with AirDrop devices working in the default “contacts only” mode, though Google “[welcomes] the opportunity to work with Apple to enable ‘Contacts Only’ mode in the future.” (Reading between the lines: Google and Apple are not currently working together to enable this, and
Google confirmed to The Verge
that Apple hadn’t been involved in this at all.)
Like AirDrop, Google notes that files shared via Quick Share are transferred directly between devices, without being sent to either company’s servers first.
Google shared a little more information in
a separate post about Quick Share’s security
, crediting Android’s use of the memory-safe Rust programming language with making secure file sharing between platforms possible.
“Its compiler enforces strict ownership and borrowing rules at compile time, which guarantees memory safety,” writes Google VP of Platforms Security and Privacy Dave Kleidermacher. “Rust removes entire classes of memory-related bugs. This means our implementation is inherently resilient against attackers attempting to use maliciously crafted data packets to exploit memory errors.”
Why is this happening now?
Google doesn’t mention it in either Quick Share post, but if you’re wondering why it’s suddenly possible for Quick Share to work with AirDrop, it can almost certainly be credited to European Union regulations imposed under the Digital Markets Act (DMA).
Let’s start with how AirDrop works. Like many of Apple’s “
Continuity
” features that rely on wireless communication between devices, AirDrop uses Bluetooth to allow devices to find each other, and a fast peer-to-peer Wi-Fi connection to actually transfer files and other data. This isn’t exotic hardware; all smartphones, tablets, and computers sold today include some flavor of Bluetooth and Wi-Fi.
But to make those Continuity features work, Apple also developed a proprietary protocol called Apple Wireless Direct Link (AWDL) to facilitate the actual connection between devices and the data transfer. Because this wasn’t a standard anyone could use, other companies couldn’t try to make their own wireless sharing features compatible with AirDrop.
But
earlier this year
, the EU
adopted new specification decisions
that required Apple to adopt new interoperable wireless standards, starting in this year’s iOS 26 release. If you don’t want to wade through the regulatory documents,
this post
from cloud services company Ditto is a useful timeline of events written in plainer language.
Setting AirDrop to “everyone for 10 minutes” mode on an iPhone.
Credit:
Andrew Cunningham
The rulings required Apple to add support for the Wi-Fi Alliance’s
Wi-Fi Aware standard
instead of AWDL—and in fact required Apple to deprecate AWDL and to help add its features to Wi-Fi Aware so that any device could benefit from them. This wasn’t quite the imposition it sounded like; Wi-Fi Aware
was developed with Apple’s help
, based on the work Apple had already done on AWDL. But it meant that Apple could no longer keep other companies out of AirDrop by using a functionally similar but private communication protocol instead of the standardized version.
In some ways, Apple’s journey to Wi-Fi Aware recalls the iPhone’s journey to USB-C: first, Apple developed a proprietary port that achieved some of the same goals as USB-C; Apple then contributed work to what would become the standardized USB-C connector; but then the company hesitated to actually adopt the standardized port in its phones until its hand was
forced by regulators
.
In any case, Wi-Fi Aware was added to iOS 26 and iPadOS 26, and
Apple’s developer documentation
lists the specific hardware that supports it (the iPhone 12 and later, and most iPads released within the last three or four years). For Android users, that likely means that Quick Share will only work with AirDrop on those devices, if they’ve been updated to iOS/iPadOS 26 or later. Google
has supported Wi-Fi Aware
in Android since version 8.0, so it should at least theoretically be possible for most modern Android phones to add support for the feature in software updates somewhere down the line.
Apple’s hardware support list also suggests that Android phones
won’t
work with AirDrop on the Mac, since macOS 26 isn’t listed as a supported operating system on Apple’s Wi-Fi Aware (it’s likely not a coincidence that macOS is not considered to be a “gatekeeper” operating system under the DMA, as both iOS and iPadOS are).
If I had to guess why neither of Google’s Quick Share posts mentions Wi-Fi interoperability standards or the DMA, it may be because Google has been
complaining
about
various aspects of the law
and its enforcement since
before it was even passed
(as have many US tech companies designated as gatekeepers by the law). Google has occasionally tried to take advantage of the DMA, as it did
when it argued
that Apple’s iMessage service should be opened up. But it may be that Google doesn’t want to explicitly credit or praise the DMA in its press releases when the company is
facing the possibility of huge fines
under the same law.
The New York Times
reported earlier this week
that EU regulators are considering changes to some of its tech regulations, citing concerns about “overregulation” and “competitiveness,” but that the EU was not currently considering changes to the DMA. For its part, Apple recently
called for the DMA to be repealed entirely
.
Andrew is a Senior Technology Reporter at Ars Technica, with a focus on consumer tech including computer hardware and in-depth reviews of operating systems like Windows and macOS. Andrew lives in Philadelphia and co-hosts a weekly book podcast called
Overdue
.
Out of nowhere,
Google brought cross-platform AirDrop support
to the Pixel 10 this week, allowing the company’s latest lineup of flagships to safely and securely send photos, files, and more to the iPhone. While it initially seemed like this was a rogue move made by Google to coerce Apple into another boundary-breaking decision, it might actually be part of the repercussions that also led to USB-C on iPhone and the adoption of RCS.
If you’ve been scratching your head trying to figure out just
how
— not to mention
why
— Google was able to get this up and running, the answer might be a little more simple than you could think. While this certainly brought back memories of, say, Beeper’s attempt at getting iMessage up and running on Android two years ago, as well as Palm’s war of attrition on iTunes support in the earliest days of the Pre, it sounds like this particular example was far less hostile towards Apple than any of its predecessors, all thanks to some of the changes made by the EU.
As reported by
Ars Technica
, the answer to this week’s mysterious Quick Share upgrade lies in the EU’s interoperability requirements designed for the DMA. The ruling out of the European Commission pushed Apple to begin supporting interoperable wireless standards beginning with this year’s set of OS upgrades, replacing the previous proprietary standard the company used to power its various Continuity features. That forced Apple to add support for the Wi-Fi Alliance’s Wi-Fi Aware standard of multi-directional file sharing, at the cost of completely phasing out its previous walled-in protocol.
So yes, while Apple wasn’t officially involved with opening up AirDrop clients to Android, it’s a little unfair to paint this company as having no involvement at all. Thanks to actions Apple was required to make under the DMA in Europe, Pixel 10 users — and soon, Android users at large — now have effectively native AirPlay support through Quick Share without any sacrifice to security, so long as the hardware has proper support for Wi-Fi Aware.
Still, just because this isn’t the quiet workaround some of us might’ve assumed Google was relying on doesn’t mean you should expect Apple to join in on the fun any time soon. As
Ars Technica
points out in its report,
Europe has been rethinking
its heavy-handed approach to tech firms, specifically in reaction to the absence of AI-centric firms in the region — and Apple, for its part, still wants the DMA revoked.
Try out AirDrop while your phone still supports it
, Pixel 10 owners. While it seems unlikely, you never know if this could disappear overnight.
FTC: We use income earning auto affiliate links.
More.
Why 90s Movies Feel More Alive Than Anything on Netflix
I was rewatching The Silence of the Lambs the other night, and something hit me hard. This movie, made in 1991, feels more alive, more gripping, more
real
than most things coming out today. And it got me thinking: why do 80s and 90s movies seem so much better than what we're getting now?
There's something about the way older films were crafted that modern cinema seems to have lost. Take Goodfellas from 1990. Scorsese doesn't just tell you a story about mobsters, he pulls you into their world. The tracking shot through the Copacabana, the narration that feels like a conversation, the way violence erupts suddenly and brutally. You feel the seduction of that lifestyle and the paranoia that comes with it. Every frame has purpose. Every scene builds character. Compare that to The Irishman from 2019, which is actually good but feels bloated, overly long, relying too heavily on “de-aging” technology that never quite convinces you.
Or think about Pulp Fiction from 1994. Tarantino took narrative structure and shattered it into pieces, then reassembled it into something that shouldn't work but does, brilliantly. The dialogue crackles. The characters feel lived-in. Vincent and Jules aren't just hitmen, they're more like philosophers debating foot massages and divine intervention between murders. Now look at something like Bullet Train from 2022. It's stylish, sure, but it feels like it's trying too hard to be quirky. The characters are archetypes. The dialogue is clever for cleverness' sake. It's entertaining in the moment but fades away from your memory almost immediately.
Even The Silence of the Lambs itself proves the point. Every interaction between Clarice and Hannibal is a chess match. You feel her vulnerability, his intelligence, the way he gets under her skin. The horror isn't in jump scares, it's in the psychological warfare. Modern thrillers like The Woman in the Window from 2021 have twists and atmosphere, but they lack that deep character work that makes you actually care what happens.
I think the difference comes down to this: older movies took risks. They trusted audiences to pay attention, to feel something, to think. Scorsese and Tarantino had visions and the freedom to execute them without endless studio interference. They weren't chasing demographics or worrying about franchise potential. They were making
films
, not products.
Today's cinema often feels designed by committee, optimized for streaming algorithms and opening weekend numbers rather than lasting impact. We have better technology, way bigger budgets, more sophisticated effects, but somewhere along the way, we forgot that movies are supposed to move us, not just occupy our time between scrolling sessions.
Maybe I'm just nostalgic. Maybe I'm romanticizing the past. But when I finish a good movie, I can sit there thinking about them for hours, even days depending on the movie. When I finish most modern blockbusters, I'm already thinking about dinner. And that difference, I think, says everything.
Crews Claim Boring Company Failed to Pay Workers and Snubbed OSHA Concerns
Willie Shane broke the asphalt on Elon Musk’s Music City Loop project this summer. Seven of his crew had been the sole excavators, fabricators and dump trucking company on The Boring Company’s proposed tunnel through Nashville for months.
Then came Monday night, when they walked off the site.
“I moved the equipment myself,” Shane said in an interview with the
Banner
on Tuesday.
“We were really skeptical from the beginning, and then since then, things pretty much just went downhill,” he added.
Musk’s company has
a spotty record of completing similar tunnels in other cities
, often snagging on government regulations and contractual issues. When Shane’s company, Shane Trucking and Excavating, which works with major local clients like the Grand Ole Opry and the Nashville International Airport, was approached by The Boring Company, he said he had some reservations.
“I told them very bluntly — and I don’t want this to come across like egotistical — but I told them, ‘Hey, my dad worked really hard to build a reputation in Nashville, and my brother and I work very hard to keep that reputation,’” Shane said. “If you guys are actually serious about doing this, you need to be 100 percent serious, because this is going to be our reputation as part of this too.”
After being reassured, Shane’s team took the job in July.
He and his crew left the state-owned property on Rosa L Parks Boulevard, where they had been working on the proposed 9-mile tunnel from the state capitol to the airport after months of safety and financial issues with Musk’s company.
It started about a month in with a change in pay.
“We were supposed to be paid every 15 days. And then they switched accounting firms, and then it went from 15 days to 60,” Shane said. Now it’s been 123 days since they started digging, and Shane says The Boring Company has only paid out about five percent of what he’s owed.
According to Shane, he has still been able to pay his employees on time, but the local trucking company is left holding the bag for money unpaid by The Boring Company. Other subcontractors, he says, have also severed ties due to nonpayment on the project.
The final straw that caused Shane to pull his crew from the site was when multiple employees reported that a representative of The Boring Company was soliciting them to bail on Shane and work directly for TBC on Monday.
“One of their head guys texts two of my welders, offering them a job for $45 an hour from his work phone,” Shane described, noting that the same TBC employee denied sending the texts when confronted with screenshots. “That’s actually a breach of contract.”
Shane also says he and other vendors have filed multiple OSHA safety complaints since working on the site but have gotten no response. His biggest concerns have been Boring employees on the jobsite not wearing proper personal protective equipment, such as hard hats, and unsafe shoring, which he says he’s repeatedly complained about to the Boring Company.
“Where we’re digging, we’re so far down, there should be concrete and different structures like that to hold the slope back from falling on you while you’re working,” Shane explained. “Where most people use concrete, they currently have — I’m not even kidding — they currently have wood. They had us install wood 2x12s.”
The safety concerns are why Shane says he decided to make the issue public.
“We’re not coming forward in like a vindictive way,” Shane said. “I just don’t want someone to get hurt, sure, and then, in the future, I have to be like, ‘Dang, I worked on there, and I turned a blind eye to it.’”
In the meantime, Shane said that the amount of backpay owed to his company is in the six figures and that he has retained a lawyer.
Boring Company response
After the
Banner
contacted The Boring Company about Shane’s claims, Vice President David Buss said he connected with Shane and would make good on the outstanding invoices by the end of the day Wednesday and would do a “full audit” on the error.
“It does look like we had some invoicing errors on that,” Buss told the
Banner
. “It was, you know, unfortunately, too common of a thing, but I assured them that we are going to make sure that invoices are wired tomorrow.”
Buss later clarified that he does not believe The Boring Company has a “common” practice of missing payments to vendors, but rather missed payments happen sometimes during “the normal course of business.”
“You hate to have an unhappy vendor. We certainly aim to have great relationships,” Buss said. “And so my goal will be to figure out what happened in this incident and then make sure that that’s not extrapolated to any other incidents.”
Buss also said he was looking into Shane’s claims about The Boring Company trying to hire contractors.
“It is definitely not our practice to try to poach anybody, so I understand the frustrations on their side,” Buss said. “Hopefully it’s something where we’re able to smooth that over and correct some of the things that happened on site and that led to this.”
Asked about the safety complaints, Buss said Shane did not raise any concerns on their call Tuesday and said he was unaware of any OSHA complaints, but would look into it.
“Safety is existential to our company,” Buss said. “We thankfully have a long history of seven years of tunneling in Las Vegas, and we’ve had one construction-related injury that was not the company’s fault in a violation.”
Hiring headaches
According to Buss, the projected timeline had not changed, and work had not been slowed by the crews’ departure from the site. Shane, however, painted a different picture.
“Actually, we were the crew that was building the tunnel boring machine. So there’s nobody building the tunnel boring machine right now, and the Boring Company has been trying to hire welders, but they haven’t been able to secure any help,” Shane said Tuesday, noting that many prospective employees won’t work on the project because of Musk’s reputation.
“A lot of people don’t like Elon and their payment terms; the way that they pay their employees, is not traditional,” Shane said.
Buss denied any hiring trouble.
“We’ve had zero issues finding great talent thus far in Nashville,” Buss said. “I think we’ve hired about 14 people now, and we’re going to start to grow the team as we begin mining operations.”
As reports of a second Boring tunnel under Broadway and West End surfaced, Boring Company CEO Steve Davis hosted a two-hour live update session on X, the social media website also owned by Musk Monday evening, in which he touted progress on the Music City Loop and described the project as smoothly underway, with boring set to begin around January after the proper permits are secured.
An hour later, Shane’s team left the site.
During Davis’ virtual meeting, members of the public could submit questions, some of which were answered by Boring Company leadership. Many of those questions came from State Sen. Heidi Campbell (D-Nashville), who represents the area and has been a vocal critic of the project since it was announced.
“I would say the promotional session that they had last night on on Twitter was disingenuous at best, if not dishonest, because it was, it sounded like a utopian project and then, lo and behold, the very next day, we find out that there are people leaving the site because they’re not getting paid and they’re not being treated well,” Campbell told the
Banner
.
In addition to her concerns about irreparable damage to the site and whether the project would even be completed, Campbell said she was concerned about the state’s liability if there were unsafe working conditions on the leased property and whether there was any way for lawmakers to stop the process.
“There is nothing to hold The Boring Company accountable for any of these things,” Campbell said of the lease. “They’ve already dug a big hole. But then on top of it, if they move forward, forward in any capacity, they have not proven that they are reliable to take care of the damage that they cause.”
When Shane first spoke to the
Banner
, he said he did not intend to return to the job even if they received payment, noting that his employees had expressed discomfort “because they didn’t feel the management there was very good.”
Hours later, after hearing from Buss, Shane said he would consider returning “if they correct the situation on their end.”
Demetria Kalodimos contributed to this report
.
The most male and female reasons to end up hospital
The first post I wrote for this blog was about
people being injured by dogs
. Specifically, how much of this goes on, and what counts as a lot.
We can measure this reasonably well in England, because the health service publishes annual data for
hospital admissions
showing what people were admitted
for
.
This often includes not just the physical condition that needed treatment, but the event that led to that condition in the first place. So not just the tissue damage on someone’s hand, in other words, but the story of a dog bite behind it.
These second-order reasons for admission—known as “external causes”—cover a whole world of horrible mishaps beyond the ones that I looked at last time. The data also records whether the patient was male or female, so I wondered what the
most male
and
most female
external causes might be.
To cut to the chase, here they are.
When I began the crunching that produced these numbers, I’d given no thought at all to what I would find. If I had, it would have been obvious that pregnancy would top the charts on the female side.
But I don’t think I could have imagined what a stark dossier of male and female stereotypes I was compiling. Because to me, the chart above basically says that violence, physical labour, sport and machines are the most typically male ways to end up in hospital, while pregnancy, beauty and animals and mental health are the most typically female.
I’m having to choose my words carefully, because I need to stress one thing:
these are not the most common reasons for men and women to be admitted to hospital
. They are the most
typically male
and
typically female
.
So only about 400 men in the whole of England go to hospital after falls from scaffolding each year. But that cause is at the top of the chart because it is the reason for admission that’s most male-dominated—just as the various pregnancy-related reasons are the most female. (I’ve put the total number of admissions in the column on the right, to give an actual sense of scale.)
In practice, I’d guess that these causes are the things that men or women do more often, or more dangerously.
Some minor points: I excluded all the external causes with less than 1,000 admissions in the last three years, so everything you see here happens at least fairly frequently, and amounts to a reasonable sample. I also excluded a small number of admissions (less than half a percent) that are classified “Gender Unknown”.
Some of the external causes have
very longwinded names
, so I’ve made them as simple as possible. “Agents primarily acting on smooth and skeletal muscles and the respiratory system” is especially unenlightening, although I suspect it might have something to do with Botox.
In the next few days I plan to upload all the data in a searchable table (if I can make that work) so you can explore it in other ways too.
NordVPN Black Friday Deal: Unlock 77% off VPN plans in 2025
Bleeping Computer
www.bleepingcomputer.com
2025-11-26 20:00:37
The NordVPN Black Friday Deal is now live, and you can get the best discount available: 77% off that applies automatically when you follow our link. If you've been waiting for the right moment to upgrade your online security, privacy, and streaming freedom, this is the one VPN deals this Black Frida...
Want one of the best VPN discounts of 2025? This NordVPN Black Friday deal gives you the fastest VPN with strong digital security and US Netflix access – all at an unbeatable price.
NordVPN Black Friday Deal: Unlock up to 77% off VPN plans in 2025
The
NordVPN Black Friday Deal
is now live, and you can get the best discount available: 77% off that applies automatically when you follow our link. If you’ve been waiting for the right moment to upgrade your online security, privacy, and streaming freedom, this is the one VPN deal we can guarantee will have you smiling all year round.
There’s no better time to buy a VPN than Black Friday or Cyber Monday. You get the same premium VPN that costs more at any other time of year, but at a fraction of the price. What’s more, if you grab a 1-year, 2-year plan, or even a 3-year plan right now, your renewal will fall during Black Friday. That means you’ll be able to hunt for another discount each time you need a VPN subscription.
So, why NordVPN? Besides having one of the best discounts, NordVPN ranks as the fastest VPN thanks to its NordLynx protocol (WireGuard fork). Fast VPN speeds make Nord perfect for Netflix access, HD streaming, gaming, and torrenting.
It enforces a strict no-logs policy, offers powerful Threat Protection Pro, and bundles valuable extras like NordPass, NordLocker, and NordProtect (Identity Theft Protection) for better everyday protection online.
NordVPN offers a more comprehensive privacy suite. Plus, with a 30-day money-back guarantee, you can try it risk-free while the discount lasts. If you want the biggest NordVPN savings of 2025, Black Friday is the perfect time to act.
NordVPN: The best Black Friday deal of 2025
The top promo this year is
NordVPN’s 2-year plan
. It is available with a massive 77% discount plus three extra months free. Best of all?
NordVPN’s Black Friday pricing
immediately surpasses VPN promotions advertised by competing providers.
In 2025, NordVPN confirmed that its Black Friday and Cyber Monday promotion runs from October 16 through December 10. That gives you nearly two months to grab the most impressive VPN deals of 2025.
Here’s what NordVPN had to say:
"Black Friday is a busy time — and not just for shoppers. Cybercriminals are also highly active during this period, so remember to take the necessary steps to protect yourself online. NordVPN protects your online traffic with encryption, making you safer on every network and device."
Get the discount – with no strings attached
When you follow the link in this article, the Black Friday deal will activate automatically – no codes or hoops to jump through.
The deal brings the total down to just $80.73 for 27 months of NordVPN Basic.
To put that into perspective, the regular subscription costs $12.99 per month, which means you’d normally pay $77.94 for just six months of VPN protection.
With this Black Friday deal, you’re getting well over two years of protection, unbeatable streaming access on vacation, and some of the best online security tools we have ever tested – for a fraction of the usual cost.
This is exactly why NordVPN’s Black Friday bundle is turning heads across the VPN industry. And why it’s easily the most competitive VPN offer we’ve managed to land on this season.
NordVPN’s Black Friday deals means you’ll get security, privacy, Netflix access, and WiFi privacy at the lowest cost.
NordVPN bundle deals
NordVPN
didn't stop at its Basic plan this Black Friday. The leading privacy provider has also slashed prices across its premium bundles. This gives you access to the Ultimate Nord Security ecosystem at prices we’ve never seen outside the Black Friday window.
NordVPN Plus
The first standout option for bargain hunters seeking better all-around protection is the NordVPN Plus subscription.
This plan includes full VPN access, Threat Protection Pro (an always‑on security layer that blocks malware, phishing websites, intrusive ads, and trackers in real time, even when the VPN is disconnected), and Nord’s secure password manager.
This Black Friday, you can get this all bundled for just $3.89 per month: turning a standard VPN subscription into a full-blown online security suite, at a price point that beats most competitors' basic plans.
If you’re looking for an online protection suite with reliable filtering against trackers, ads, and malware, NordVPN delivers exactly that. It also includes a top-tier password manager that helps secure your accounts against hackers and phishing.
What’s more, NordVPN’s pricing is unusually generous for the amount of protection you get. It’s genuinely rare to find such a comprehensive security bundle at a cost that beats what most providers charge for the VPN alone.
NordVPN Ultimate
Hunting for the ultimate VPN deal of the year? NordVPN’s “Ultimate” plan is the centerpiece of its 2025 Black Friday event.
Normally valued at $626.13, the 27-month Ultimate plan is currently discounted to just $159. That works out to $5.89 per month, which is a massive 77% price cut.
Ultimate includes every service and feature that Nord Security offers. You get unlimited VPN use, the password manager, upgraded anti-malware and anti-tracking tools, 1TB of encrypted cloud storage, and even $5,000 in scam loss insurance through NordProtect. Just bear in mind that insurance is only available to US residents.
When you consider that Google charges $5 per month for just 1TB of cloud storage, Nord’s Black Friday pricing really comes out swinging! For only 89 cents more, you’ll get cloud storage
plus
a VPN, password manager, advanced threat filtering, and identity theft protection.
For anyone looking to build a full security stack at the lowest possible cost, these Black Friday bundles are among the strongest tech deals of the year.
Which VPN features does NordVPN come with?
No matter whether you choose NordVPN Basic, Plus, or Ultimate, you'll get full access to NordVPN’s complete VPN feature set. All core tools, including global server options, VPN protocol options, privacy settings, and security features, remain identical across all plans.
The higher-tier bundles simply add extra services such as password management, advanced threat filtering, encrypted cloud storage, and identity protection.
That means you can stick with NordVPN Basic if all you want is a powerful, fast, and fully featured VPN. The upgrades are optional add-ons and will not change how the VPN itself performs.
Full NordVPN feature list:
Strong encryption of all traffic
(AES‑256 with modern VPN protocols like NordLynx/
WireGuard
, OpenVPN, and IKEv2 for both security and speed).
Protection against ISP or network surveillance
by hiding all browsing activity inside an encrypted tunnel.
IP address masking
so websites and services see the VPN server’s IP instead of your real one, improving privacy and helping avoid IP‑based tracking.
Location spoofing
lets you choose from thousands of servers in 127+ countries, useful for
bypassing geo‑restrictions
and regional blackouts.
Ad blocking
at the server level to strip many ads before they reach your device (via Threat Protection/Threat Protection Pro).
Tracking prevention
by blocking common tracking domains and cookies so advertisers and analytics tools collect less data on you.
Malicious site blocking
that stops connections to known phishing, malware, and scam domains before they load.
Malware download scanning
(on supported desktop apps) that checks downloaded files.
MultiHop VPN routing (Double VPN)
, sending your traffic through two VPN servers with two layers of encryption for extra anonymity in high‑risk situations.
Tor over VPN
sends your traffic first through the VPN and then into the
Tor network
for stronger identity protection on .onion sites.
Automatic
kill switch
that cuts your internet connection if the VPN drops, preventing any data from leaking outside the encrypted tunnel.
DNS leak protection
by forcing all DNS lookups through NordVPN’s own DNS resolvers, so your ISP cannot see what domains you visit.
Obfuscated servers
(NordWhisper / obfuscation) to hide the fact that you are using a VPN. Useful to connect on restrictive networks and to use the VPN in high-censorship countries.
P2P‑optimized servers
for safer torrenting and other peer‑to‑peer traffic without sacrificing too much speed.
Streaming‑optimized servers
(SmartPlay) that automatically use working DNS/routes to access major streaming platforms when they try to block VPN IPs.
Split tunneling
(on supported apps) so you can choose which apps use the VPN and which go directly to the internet—for example, routing only your browser through the VPN while games or banking apps use a normal connection.
Private DNS servers
operated by NordVPN instead of your ISP’s DNS, reducing data exposure and some forms of DNS‑based censorship.
High‑speed connections
(including 10 Gbps locations and NordLynx) to minimize the performance hit usually associated with VPNs.
Support for up to 10 simultaneous devices
under one subscription, so you can cover multiple personal devices or family members at once.
Optional dedicated IP addresses
so you can get a consistent IP (useful for hosting, remote access, avoiding CAPTCHA, and accessing strict streaming accounts).
Native apps for Windows,
macOS
, Linux, Android, iOS/iPadOS, Android TV
, and many
smart TVs
, Amazon Fire TV/Firestick, Apple TV, and Apple Vision (via native/tvOS/visionOS support).
Browser extensions
(proxy-based protection) for Chrome, Firefox, and Microsoft Edge.
Why NordVPN is the standout Black Friday VPN deal of 2025
NordVPN
is one of the most trusted VPN brands on the market, and its Black Friday and Cyber Monday deals make 2025 the perfect time to lock in long-term savings.
The service is headquartered in privacy-friendly Panama, a location that puts it well out of reach of data-hungry jurisdictions like the US, the UK, and the EU. Thanks to Panama's lack of mandatory data retention laws, NordVPN can maintain a
strict no-logging policy
. That means Nord has no records of your activities, even if the government comes knocking with a warrant.
Add to this its wide feature set and excellent third-party audit results, and you can see why NordVPN continues to stand out as one of the best value VPN options for netizens who care about strong privacy and watertight online security.
With the NordVPN Black Friday Deal, you will get access to all the premium features that helped NordVPN earn its reputation. This includes its NordLynx protocol (built on WireGuard to make NordVPN the
fastest VPN
), advanced encryption, and reliable privacy settings for users in countries where surveillance and censorship are a part of daily life.
Fully optimized for streaming
When it comes to streaming, NordVPN is exceptional. During our tests, its international network successfully accessed
multiple Netflix regions
, Hulu, HBO Max, Disney+,
Prime Video
, YouTube TV, DirecTV, SlingTV, BBC iPlayer, Joyn, Canal+,
Crave
, ESPN+, FOX, ABC, NBC, and Peacock.
And its fast connection speeds make it perfect for HD streaming without buffering, as well as for
gaming
, torrenting, and making video calls.
Does NordVPN have apps for all platforms?
Yes, NordVPN gives you comprehensive coverage for every gadget you own.
NordVPN provides custom apps for all major platforms (including Windows, macOS, iOS, Android,
Linux
, and Amazon Firestick), making it a practical, versatile option for households with mixed devices.
Each subscription supports up to 10
simultaneous connections
, allowing you to protect phones, tablets, laptops, smart TVs, and even school or work devices under one account.
With this year’s Black Friday pricing, NordVPN has turned one of the most polished premium VPNs on the market into a cheap VPN we can confidently recommend.
It's budget season! Over 300 CISOs and security leaders have shared how they're planning, spending, and prioritizing for the year ahead. This report compiles their insights, allowing readers to benchmark strategies, identify emerging trends, and compare their priorities as they head into 2026.
Learn how top leaders are turning investment into measurable impact.
Popular Forge library gets fix for signature verification bypass flaw
Bleeping Computer
www.bleepingcomputer.com
2025-11-26 19:32:42
A vulnerability in the 'node-forge' package, a popular JavaScript cryptography library, could be exploited to bypass signature verifications by crafting data that appears valid. [...]...
A vulnerability in the ‘node-forge’ package, a popular JavaScript cryptography library, could be exploited to bypass signature verifications by crafting data that appears valid.
The flaw is tracked as CVE-2025-12816 and received a high severity rating. It arises from the library’s ASN.1 validation mechanism, which allows malformed data to pass checks even when it is cryptographically invalid.
“An interpretation-conflict vulnerability in node-forge versions 1.3.1 and earlier enables unauthenticated attackers to craft ASN.1 structures to desynchronize schema validations, yielding a semantic divergence that may bypass downstream cryptographic verifications and security decisions,” reads the flaw's
description
in the National Vulnerabilities Database (NVD).
Hunter Wodzenski of Palo Alto Networks discovered the flaw and reported it responsibly to the node-forge developers.
The researcher warned that applications that rely on node-forge to enforce the structure and integrity of ASN.1-derived cryptographic protocols can be tricked into validating malformed data, and provided a proof-of-concept demonstrating how a forged payload could trick the verification mechanism.
A security advisory from the Carnegie Mellon CERT-CC explains that the impact varies per application, and may include authentication bypass, signed data tampering, and misuse of certificate-related functions.
“In environments where cryptographic verification plays a central role in trust decisions, the potential impact can be significant,”
CERT-CC warns
.
The impact may be significant considering that node-forge is massively popular with close to
26 million weekly downloads
on the Node Package Manager (NPM) registry.
The library is used by projects that need cryptographic and public-key infrastructure (PKI) functionality in JavaScript environments.
A fix was released earlier today in version 1.3.2. Developers using node-forge are advised to switch to the latest variant as soon as possible.
Flaws in widely used open-source projects
can persist for a long time
after their public disclosure and the availability of a patch. This may happen due to various reasons, the complexity of the environment and the need to test the new code being some of them.
Maybe it’s because my eyes are getting old or maybe it’s because
the contrast between windows
on macOS keeps getting worse. Either way, I built a tiny Mac app last night that draws a border around the active window. I named it “Alan”.
In Alan’s preferences, you can choose a preferred border width and colors for both light and dark mode.
Fara-7B: An Efficient Agentic Model for Computer Use
Overview
Fara-7B
is Microsoft's first
agentic small language model (SLM)
designed specifically for computer use. With only 7 billion parameters, Fara-7B is an ultra-compact Computer Use Agent (CUA) that achieves state-of-the-art performance within its size class and is competitive with larger, more resource-intensive agentic systems.
Try Fara-7B locally as follows (see
Installation
for detailed instructions):
vllm serve "microsoft/Fara-7B" --port 5000 --dtype auto
Then you can iterative query it with:
fara-cli --task "whats the weather in new york now"
Hint: might need to do
--tensor-parallel-size 2
with vllm command if you run out of memory
What Makes Fara-7B Unique
Unlike traditional chat models that generate text-based responses, Fara-7B leverages computer interfaces—mouse and keyboard—to perform multi-step tasks on behalf of users. The model:
Operates visually
by perceiving webpages and taking actions like scrolling, typing, and clicking on directly predicted coordinates
Uses the same modalities as humans
to interact with computers—no accessibility trees or separate parsing models required
Enables on-device deployment
due to its compact 7B parameter size, resulting in reduced latency and improved privacy as user data remains local
Completes tasks efficiently
, averaging only ~16 steps per task compared to ~41 for comparable models
Fara-7B is trained using a novel synthetic data generation pipeline built on the
Magentic-One
multi-agent framework, with 145K trajectories covering diverse websites, task types, and difficulty levels. The model is based on
Qwen2.5-VL-7B
and trained with supervised fine-tuning.
Key Capabilities
Fara-7B can automate everyday web tasks including:
Searching for information and summarizing results
Filling out forms and managing accounts
Booking travel, movie tickets, and restaurant reservations
Shopping and comparing prices across retailers
Finding job postings and real estate listings
Performance Highlights
Fara-7B achieves state-of-the-art results across multiple web agent benchmarks, outperforming both comparable-sized models and larger systems:
Model
Params
WebVoyager
Online-M2W
DeepShop
WebTailBench
SoM Agents
SoM Agent (GPT-4o-0513)
-
90.6
57.7
49.1
60.4
SoM Agent (o3-mini)
-
79.3
55.4
49.7
52.7
SoM Agent (GPT-4o)
-
65.1
34.6
16.0
30.8
GLM-4.1V-9B-Thinking
9B
66.8
33.9
32.0
22.4
Computer Use Models
OpenAI computer-use-preview
-
70.9
42.9
24.7
25.7
UI-TARS-1.5-7B
7B
66.4
31.3
11.6
19.5
Fara-7B
7B
73.5
34.1
26.2
38.4
Table: Online agent evaluation results showing success rates (%) across four web benchmarks. Results are averaged over 3 runs.
WebTailBench: A New Benchmark for Real-World Web Tasks
We are releasing
WebTailBench
, a new evaluation benchmark focusing on 11 real-world task types that are underrepresented or missing in existing benchmarks. The benchmark includes 609 tasks across diverse categories, with the first 8 segments testing single skills or objectives (usually on a single website), and the remaining 3 evaluating more difficult multi-step or cross-site tasks.
WebTailBench Detailed Results
Task Segment
Tasks
SoM GPT-4o-0513
SoM o3-mini
SoM GPT-4o
GLM-4.1V-9B
OAI Comp-Use
UI-TARS-1.5
Fara-7B
Single-Site Tasks
Shopping
56
62.5
71.4
38.1
31.0
42.3
41.1
52.4
Flights
51
60.1
39.2
11.1
10.5
17.6
10.5
37.9
Hotels
52
68.6
56.4
31.4
19.9
26.9
35.3
53.8
Restaurants
52
67.9
59.6
47.4
32.1
35.9
22.4
47.4
Activities
80
70.4
62.9
41.7
26.3
30.4
9.6
36.3
Ticketing
57
58.5
56.7
37.4
35.7
49.7
30.4
38.6
Real Estate
48
34.0
17.4
20.1
16.0
9.0
9.7
23.6
Jobs/Careers
50
49.3
44.0
32.7
22.7
20.7
20.7
28.0
Multi-Step Tasks
Shopping List (2 items)
51
66.0
62.7
17.0
7.8
34.0
20.9
49.0
Comparison Shopping
57
67.3
59.1
27.5
22.8
1.2
8.8
32.7
Compositional Tasks
55
51.5
39.4
26.7
17.0
10.3
9.1
23.0
Overall
Macro Average
609
59.7
51.7
30.1
22.0
25.3
19.9
38.4
Micro Average
609
60.4
52.7
30.8
22.4
25.7
19.5
38.4
Table: Breakdown of WebTailBench results across all 11 segments. Success rates (%) are averaged over 3 independent runs. Fara-7B achieves the highest performance among computer-use models across all task categories.
Coming Soon:
Task Verification pipeline for LLM-as-a-judge evaluation
Official human annotations of WebTailBench (in partnership with BrowserBase)
Evaluation Infrastructure
Our evaluation setup leverages:
Playwright
- A cross-browser automation framework that replicates browser environments
Abstract Web Agent Interface
- Allows integration of any model from any source into the evaluation environment
Fara-Agent Class
- Reference implementation for running the Fara model
Note:
Fara-7B is an experimental release designed to invite hands-on exploration and feedback from the community. We recommend running it in a sandboxed environment, monitoring its execution, and avoiding sensitive data or high-risk domains.
Installation
Install the package using either UV or pip:
or
Then install Playwright browsers:
Hosting the Model
Recommended:
The easiest way to get started is using Azure Foundry hosting, which requires no GPU hardware or model downloads. Alternatively, you can self-host with VLLM if you have GPU resources available.
Azure Foundry Hosting (Recommended)
Deploy Fara-7B on
Azure Foundry
without needing to download weights or manage GPU infrastructure.
Setup:
Deploy the Fara-7B model on Azure Foundry and obtain your endpoint URL and API key
Add your endpoint details to the existing
endpoint_configs/
directory (example configs are already provided):
# Edit one of the existing config files or create a new one# endpoint_configs/fara-7b-hosting-ansrz.json (example format):
{
"model": "Fara-7B",
"base_url": "https://your-endpoint.inference.ml.azure.com/",
"api_key": "YOUR_API_KEY_HERE"
}
Run the Fara agent:
fara-cli --task "how many pages does wikipedia have" --start_page "https://www.bing.com"
That's it! No GPU or model downloads required.
Self-hosting with VLLM
If you have access to GPU resources, you can self-host Fara-7B using VLLM. This requires a GPU machine with sufficient VRAM.
All that is required is to run the following command to start the VLLM server:
vllm serve "microsoft/Fara-7B" --port 5000 --dtype auto
Testing the Fara Agent
Run the test script to see Fara in action:
fara-cli --task "how many pages does wikipedia have" --start_page "https://www.bing.com" --endpoint_config endpoint_configs/azure_foundry_config.json [--headful] [--downloads_folder "/path/to/downloads"] [--save_screenshots] [--max_rounds 100] [--browserbase]
In self-hosting scenario the
endpoint_config
points to
endpoint_configs/vllm_config.json
from the VLLM server above.
If you set
--browserbase
, export environment variables for the API key and project ID.
Expected Output
Initializing Browser...
Browser Running... Starting Fara Agent...
##########################################
Task: how many pages does wikipedia have
##########################################
Running Fara...
Thought #1: To find the current number of Wikipedia pages, I'll search for the latest Wikipedia page count statistics.
Action #1: executing tool 'web_search' with arguments {"action": "web_search", "query": "Wikipedia total number of articles"}
Observation#1: I typed 'Wikipedia total number of articles' into the browser search bar.
Thought #2: Wikipedia currently has 7,095,446 articles.
Action #2: executing tool 'terminate' with arguments {"action": "terminate", "status": "success"}
Observation#2: Wikipedia currently has 7,095,446 articles.
Final Answer: Wikipedia currently has 7,095,446 articles.
Enter another task (or press Enter to exit):
Reproducibility
We provide a framework in
webeval/
to reproduce our results on WebVoyager and OnlineMind2Web.
Agentic evaluations on live websites present unique challenges due to day-to-day changes. We implement several measures to ensure reliable and comparable evaluations:
BrowserBase Integration
We employ BrowserBase to manage browser session hosting, enabling reliable browser instance management.
Time-sensitive Task Updates
Tasks in benchmarks like WebVoyager can become stale or impossible. We:
Removed ~48 impossible tasks from the original WebVoyager benchmark
Updated ~50 tasks with future dates to keep them achievable
Example:
"Search for a hotel in Bali from Jan 1 to Jan 4, 2024"
→
"Search for a hotel in Bali from Jan 1 to Jan 4, 2026"
Our updated WebVoyager benchmark is available at
webeval/data/webvoyager/WebVoyager_data_08312025.jsonl
Trajectories are retried up to 5 times when environment errors occur
Complete yet incorrect trajectories are never retried
Each retry starts with a fresh browser session, with no retained state
Step Budget
Each trajectory is capped at a maximum of 100 actions across all online benchmarks. Trajectories exceeding this budget without choosing to stop are considered incorrect.
screenshot_X.png
- Screenshots captured before each action X
Running Analysis
Use the analysis notebook to compute metrics:
cd webeval/scripts/analyze_eval_results/
jupyter notebook analyze.ipynb
The script:
Identifies trajectories aborted mid-execution and diagnostic reasons
Computes average scores across non-aborted trajectories
Distinguishes between aborted trajectories (errors during sampling) and completed trajectories (with terminate() call or step budget exceeded)
To re-run failed tasks, execute the evaluation script again with the same
run_id
and
username
- it will skip non-aborted tasks.
Example WebVoyager GPT Eval Result
{
"score": 1.0,
"gpt_response_text": "To evaluate the task, we need to verify if the criteria have been met:\n\n1. **Recipe Requirement**: A vegetarian lasagna recipe with zucchini and at least a four-star rating.\n\n2. **Search and Results**:\n - The screenshots show that the search term used was \"vegetarian lasagna zucchini.\"\n - Among the search results, \"Debbie's Vegetable Lasagna\" is prominently featured.\n\n3. **Evaluation of the Recipe**:\n - Rating: \"Debbie's Vegetable Lasagna\" has a rating of 4.7, which satisfies the requirement of being at least four stars.\n - The presence of zucchini in the recipe is implied through the search conducted, though the screenshots do not explicitly show the ingredients list. However, the result response confirms the match to the criteria.\n\nGiven the information provided, the task seems to have fulfilled the requirement of finding a vegetarian lasagna recipe with zucchini and a four-star rating or higher. \n\n**Verdict: SUCCESS**"
}
Citation
If you use Fara in your research, please cite our work:
Releasing Packages with a Valet Key: npm, PyPI, and beyond
Disclaimer: This post should have been written about 5 years ago but I never got around to it; with the most recent
Shai-Hulud attack
, I thought it would be a good time to finally check this off the list and hopefully help others avoid supply-chain attacks.
About 5 years ago, I sat in on a meeting at Sentry in the midst of their SOC 2 compliance efforts. There was
Armin
, telling us that we needed a secret storage service for our package repository tokens. The tokens we used to deploy Sentry SDKs to package repositories such as npm, PyPI etc. This was to ensure there were no unauthorized releases of our SDKs which were embedded into all Sentry customers’ products. There were a limited set of people who had access to these tokens back in the time. Now, they became the bottleneck for more and more frequent releases. There was also the auditability issue at hand: releases were performed from individuals’ workstations and there was no easy way to trace a release back to where it originated from or whether it was authorized or not.
For some reason I intuitively was against such a secret storage service and felt like the answer was somewhere in GitHub, GitHub Actions, and their secret storage service we already used. We already had the repo permissions, personnel structure, and all the visibility for auditing there. Heck, even the approval mechanics were there with pull requests. So I said “give me a week and I’ll get you a proof of concept” which Armin did and I delivered - though I think it took a bit more than a week 😅
Secrets in Plain Sight
Before we dive into the solution, let me paint a picture of the problem. Publishing packages to registries like npm, PyPI, or crates.io requires access tokens. These tokens are essentially the keys to the kingdom - whoever has them can publish anything under your organization’s name. At the time, these tokens were either distributed to select individuals, or lived in GitHub repository secrets, accessible to anyone with write access to the repository.
1
Now, here’s the scary part: at Sentry, we had 90-100+ engineers with commit rights to our SDK repositories. Any one of them could:
Create a new workflow or modify an existing one
Access these secrets within that workflow
Exfiltrate them to any web service they controlled
Do all of the above without triggering any alarms
And the truly terrifying bit? Even if someone
did
steal these tokens, there would be no indication whatsoever. No alerts, no logs, nothing. They could sit on these credentials and use them months later, long after they’ve left the company. We’ve seen this exact scenario play out recently with supply-chain attacks like the
Shai-Hulud npm takeover
where attackers compromised maintainer accounts to publish malicious versions of popular packages.
The Valet Key
Some fancy cars come with a “valet key” - a special key you give to parking attendants or car wash folks. Unlike your regular key, this one has limited capabilities: maybe it can only go up to 20mph, can’t open the trunk, or won’t let you disable the alarm. It’s the same car, but with reduced privileges for reduced risk of theft.
This concept maps beautifully to our problem. Instead of giving everyone the full keys (the publishing tokens), why not give them a way to
request
the car to be moved (a release be made)? The actual keys stay with a very small, trusted (and monitored) group who are the builders and maintainers of the infrastructure. Even the approvers don’t actually have access to the keys!
Here’s what we wanted:
Secrets in a secure, limited-access location
- only 3-4 release engineers should have access
Clear approval process
- every release needs explicit sign-off from authorized personnel
Low friction for developers
- anyone should be able to
request
a release easily
Full audit trail
- everything logged being compliance-friendly
No new infrastructure
- we didn’t want to build or maintain a separate secrets service
As a side note, trusted publishing through OIDC and OAuth with limited and very short-lived tokens is the
actual
digital equivalent of valet keys. npm is slowly rolling this out
2
, but at the time we built this system, it wasn’t an option. And even today, it’s not available at the organization/scope level which is what we’d need. Also, we publish way more places than npm so we need a more generic solution.
Another approach worth mentioning is Google’s
Wombat Dressing Room
- an npm registry proxy that funnels all publishes through a single bot account with 2FA enabled. It’s a clever solution if you’re npm-only and want something off-the-shelf. That said it still requires running a separate service.
3
Enter getsentry/publish
The solution we landed on is beautifully simple in hindsight: a
separate repository
dedicated entirely to publishing. Here’s the trick:
Write access is extremely limited
- only 3-4 release engineers can actually modify the repo
Release managers get “triage” access
- GitHub’s triage role lets you manage issues and labels, but not code - perfect for approving releases
Everyone else can create issues
- that’s all you need to request a release
Approval happens via labels
- a release manager adds the “accepted” label to trigger the actual publish
The beauty of this setup is that the publishing tokens live
only
in this repo’s secrets. The repo itself is mostly static - we rarely need to modify the actual code - so the attack surface is minimal.
The Implementation (with Craft)
Under the hood, we use
Craft
, our CLI tool for managing releases. Craft was designed with a crucial architectural decision that predates the publish repo: it separates releases into two distinct phases -
prepare
and
publish
.
The
prepare
phase is where all the “dangerous” work happens:
npm install
, build scripts, test runs, changelog generation. This phase runs in the SDK repository
without
any access to publishing tokens. The resulting artifacts are uploaded to GitHub as,
well
, build artifacts.
The
publish
phase simply downloads these pre-built artifacts and pushes them to the registries. No
npm install
, no build scripts, no arbitrary code execution - just download and upload. This dramatically reduces the attack surface during the privileged publishing step. Even if an attacker managed to inject malicious code into a dependency, it would only execute during the prepare phase which has no access to publishing credentials.
This two-phase architecture is what makes supply-chain attacks like
Shai-Hulud
much harder to pull off against Sentry’s SDKs. The malicious code would need to somehow persist through the artifact upload/download cycle and execute during a phase that deliberately runs no code.
The magic happens with our GitHub Actions setup:
Developer triggers release workflow
in their SDK repo (e.g.,
sentry-javascript
)
action-prepare-release
runs
craft prepare
: creates the release branch, updates changelogs, builds artifacts, uploads them to GitHub
An issue is automatically created
in
getsentry/publish
with all the details: what changed, what’s being released, which targets
Release manager reviews and approves
by adding the “accepted” label
Publishing workflow triggers
craft publish
: downloads artifacts from GitHub and pushes to npm, PyPI, crates.io, etc. - no build step, just upload
Fighting Overprotective Parents
GitHub, bless their security-conscious hearts, put up quite a few guardrails that we had to work around. Here’s where things got… creative:
The Token Trigger Problem
: For the automation, we had to use the
Sentry Release Bot
, a GitHub App that generates short-lived tokens. This is crucial because
GITHUB_TOKEN
(default token GitHub Actions creates) has a security restriction: actions triggered by it don’t trigger other actions
4
. We needed workflows in
getsentry/publish
to trigger based on issues created from SDK repos, so we had to work around this.
The Admin Bot Account
: We needed a bot that could commit directly to protected branches. GitHub’s branch protection rules
are
were all-or-nothing - you can’t say “this bot can commit, but only to update
CHANGELOG.md
”. So our bot ended up with admin access on all repos. Not ideal, but necessary
5
.
Composite Actions and Working Directories
: If you’ve ever tried to use GitHub’s composite actions with custom working directories, you know the pain. There’s no clean way to say “run this composite action from this subdirectory”. We ended up with various hacks involving explicit
cd
commands and careful path management.
Some More
Creative
Workarounds
: We maintain a small collection of ugly-but-necessary workarounds in our action definitions. They’re not pretty, but they work. Sometimes pragmatism beats elegance
6
.
Happily Ever After
After all this work, what did we actually achieve?
Compliance-friendly
✓ - every release is logged, approved, and traceable
Centralized secrets
- tokens live in one place, accessible to very few
Developer convenience
- anyone can request a release with a few clicks
Enterprise security
- no individual has publishing credentials on their machine
Full transparency
- the entire
publish repo
is open, notifications enabled for stakeholders
We’ve made more than
6,000 releases
through this system and happily counting upwards. Every single one is traceable: who requested it, who approved it, what changed, when it shipped.
Why This Matters Today
Recent supply-chain attacks like
Shai-Hulud
show exactly why this architecture matters. When attackers compromise a maintainer’s npm account, they can publish malicious versions of packages that millions of developers will automatically install. With our system:
No individual at Sentry has npm/PyPI/crates.io credentials on their machine
Every release requires explicit approval from a release manager
The approval happens in a public repo with full audit trail
Any suspicious activity would be immediately visible
Is it perfect? No. Could a determined attacker with inside access still cause damage? Probably. But we’ve dramatically reduced the attack surface and made any compromise immediately visible and auditable.
Closing Thoughts
Looking back, this is one of my proudest achievements at Sentry. It’s not flashy - no one’s going to write a blog post titled “Revolutionary New Way to Click a Label” - but it’s the kind of infrastructure that quietly makes everything more secure and more convenient at the same time.
7
If you’re dealing with similar challenges, I encourage you to check out
getsentry/publish
and the
Craft
. The concepts are transferable even if you don’t use our exact implementation.
And hey, it only took me 5 years to write about it. Better late than never, right? 😅
Thanks
I’d like to thank the following people:
Armin
and
Daniel
for their trust and support in building this system.
Jeffery
for reviewing this post thoroughly and being my partner in crime for many things security at Sentry.
Michael
for giving me the push I needed to write this post, coming up with the awesome post image idea, and for his support and guidance on the post itself.
This was before GitHub introduced “environment secrets” which allow more granular access control. Even with those, the problem isn’t fully solved for our use case.
↩
If only someone could make this run directly in GitHub Actions…
↩
This is actually a smart security feature - imagine a workflow that creates a commit that triggers itself. Infinite loop, infinite bills, infinite sadness.
↩
This is now fixed with special
by-pass rules via rule sets
recently and we also no longer have admin access for the bots, phew.
↩
If you peek at the repo, you’ll see what I mean. I’m not proud of all of it, but I’m proud it works.
↩
Especially while “security means more friction” is still a thing.
↩
China Has Three Reusable Rockets Ready for Their Debut Flights
Three of China’s space enterprises are near the debut flights of their partially reusable rockets, expected to liftoff before the end of the year.
Around
November
25th
, the Shanghai Academy of Spaceflight Technology’s Long March 12A partially reusable launch vehicle
1
was spotted heading for its launch pad the the Jiuquan Satellite Launch Center, for its first public appearance of a full vehicle. The liquid methane and liquid oxygen burning rocket has two 3.8-meter wide stages, with the first equipped with seven
Longyun engines
from Jiuzhou Yunjian (九州云箭) and the second with a single vacuum optimized
YF-209
, to carry up to 12,000 kilograms. First-stage reuse will be achieved by an engine performing a landing burn to touchdown on four legs, with grid fins guiding it before that.
Due to the opaque nature of the Long March 12A’s development, it is unknown if the launch vehicle at Jiuquan will wrap up the overall development campaign, possibly with a static fire, before a debut flight later in December.
The Shanghai Academy of Spaceflight Technology’s Long March 12A launch vehicle atop of its transporter-erector at the Jiuquan Satellite Launch Center in November 2025.
Meanwhile, LandSpace’s 66-meter-tall, 4.5-meter-wide Zhuque-3
is on its Jiuquan launch pad
too, following delivery
in October
. Like the Long March 12A, the rocket burns liquid methane and liquid oxygen, but has two more engines, LandSpace’s TQ-12A, on its first-stage and one vacuum-optimized TQ-15A engine on the second-stage, to deliver up to 11,800 kilograms in its ‘
block one
’ configuration. Similar to the Shanghai Academy’s rocket, Zhuque-3’s first-stage will touchdown on four landing legs following an engine burn, with four grid fins guiding it through the atmosphere.
Zhuque-3 has had a highly successful test campaign during its just
over two-year-long development
process. In September 2024, the launch vehicle’s in-atmosphere hop-testing campaign was completed with
a 10-kilometer flight
that saw an engine relight for touchdown. That was followed by a
45-second static fire in June
, later matched by flight hardware performing a similar static fire with a second-stage on top. Hardware has also been flown with the company’s Zhuque-2 and
Zhuque-2E
launch vehicles as well.
LandSpace’s Zhuque-3 Y1 vehicle at Launch Complex 96B at the Jiuquan Satellite Launch Center in October 2025.
Along with the two methane-fueled rockets, Space Pioneer’s Tianlong-3 is also at Jiuquan, having
arrived sometime in November
. The two-stage 72-meter-tall, 3.8-meter-wide launch burns rocket-grade kerosene and liquid oxygen to carry up to 17,000 kilograms to low Earth orbit, with nine TH-12 engines on the first-stage and a single vacuum-optimized one on the second-stage. Tianlong-3's first-stage is planned to land on four landing legs, guided by four grid fins, with an engine burn providing the soft touchdown needed.
In the lead-up to launch, Tianlong-3 conducted its first
wholly successful static fire in September
and skipped a second-stage firing, having confidence in the singular engine powering it following its development campaign. At the moment, the launch vehicle is on its dedicated launchpad at the launch site for integrated testing with ground systems. Notably, no reuse hardware has been installed yet, and mounting points appear to be missing.
Space Pioneer’s Tianlong-3 Y1 vehicle on its launch pad at the Jiuquan Satellite Launch Center in November 2025.
Out of the Long March 12A, Zhuque-3, and Tianlong-3, LandSpace may fly China’s first reusable rocket. Despite a current lack of hazard notices,
news outlets
are saying November 29th is the first targeted date.
LandSpace has vaguely denied that date
, asking enthusiasts to do diligent research. As for the other two rockets, Space Pioneer and the Shanghai Academy of Spaceflight Technology are yet to share relevant information
2
.
First-stage booster landing sites have been completed for both
Zhuque-3
and the
Long March 12A
in previous months. Those sites are expected to have systems for safing the boosters following touchdown as well as fire suppression systems in the event of an anomaly. LandSpace and the
Shanghai Academy
are eyeing first-stage landings during the debut flights. Whichever lands first will be the third globally and the first outside of the United States, following SpaceX’s
Falcon 9 in 2015
and Blue Origin’s New Glenn
on November 13th 2025
.
When the three rockets do debut, they will be a boon to the deployment efforts of
China’s various mega-constellations
, as reuse will allow for cheaper and more frequent launch missions. Back in August, Shanghai Spacesail Technologies, the operator of the Qianfan (千帆) constellation,
awarded contracts to
LandSpace and Space Pioneer to prove they can launch satellite batches with their partially reusable rockets, with Tianlong-3
looking to deliver
larger satellite groups.
Thanks for reading China in Space! This post is public so feel free to share it.
Effortless to create. Enjoyable to answer. Designed for real insights.
Build Smarter Forms. Capture Honest Answers.
Fabform makes it easy to create flexible, powerful forms that invite real responses — so you can connect, understand, and grow with confidence.
Conversational Forms
Guide your respondents through one question at a time, creating a natural, friendly flow that feels like a conversation. This approach increases completion rates and captures thoughtful, honest answers.
No-Code Logic
Build complex branching logic and customized form paths easily — no coding required. Set up conditional questions and personalized flows that make your forms smarter, saving you time and ensuring your data is relevant.
Everything your form should be — smart, conversational, and effortless
Build interactive forms that guide people one step at a time — no code, no stress.
Smart Branching
Show or hide questions based on what users say. Branching logic makes your forms feel natural, not robotic.
Conversational UI
Ask one question at a time, just like a real conversation. Your form becomes a friendly guide.
Design Without Code
Drop in questions, style layouts, and brand everything — all with a visual editor.
Seamless Integrations
Connect your forms effortlessly with apps and tools you already use — automations start the moment a form is submitted.
Build Smarter Forms — Without Limits
Drag, drop, and deploy forms in minutes. No code, no caps, no nonsense. Perfect for teams, startups, and creators who just want it to work.
Collect unlimited responses for free. Customize every pixel. Integrate with over 6,000 tools like Google Sheets, Slack, and Zapier.
From lead gen to surveys, our builder adapts to whatever you're building. It's everything you need — nothing you don’t.
Build Smarter Forms Faster
Fabform makes building smart, responsive, and beautiful forms easier than ever. Whether you’re collecting data, payments, or signatures, Fabform’s rich feature set helps you get the job done — quickly and effortlessly.
Unlimited Forms & Responses
Build and manage unlimited forms without worrying about restrictions. Collect as many responses as you need to fuel your business insights and operations.
Intuitive Drag-and-Drop Builder
Design beautiful, customized forms effortlessly using our visual drag-and-drop interface. No coding skills required—just drag, drop, and create.
Fully Responsive Design
Fabform’s forms automatically adjust to look perfect on any device, from smartphones and tablets to desktops, ensuring an excellent user experience everywhere.
Advanced Conditional Logic
Create dynamic forms that adapt in real time. Show or hide questions, sections, or entire pages based on previous answers for a personalized experience.
500+ Professionally Crafted Templates
Get a head start with a huge library of ready-made templates designed for surveys, quizzes, registrations, feedback forms, and more.
File Upload Support
Easily collect documents, images, or any other files directly through your forms. Users can upload files up to 10MB, with options for higher limits on premium plans.
Easy Embedding & Sharing
Embed Fabform forms seamlessly on your website or share them via direct links on social media, emails, or messaging platforms with zero hassle.
Real-Time Email Notifications
Stay instantly informed about new submissions with customizable email alerts, so you never miss important data or leads.
Powerful Integrations
Connect Fabform effortlessly with popular tools like Google Sheets, Slack, Zapier, and Calendly to automate tasks and streamline your workflow.
Webhooks for Instant Data Delivery
Push form submissions in real time directly to your own servers or apps with secure webhooks, enabling seamless integrations and automation.
Digital Signature Collection
Collect legally binding e-signatures right inside your forms — perfect for contracts, consent forms, and agreements.
Integrated Payment Processing
Accept payments securely and effortlessly via Stripe without forcing users to leave the form, simplifying order and donation workflows.
Custom Branding & Domains
Make Fabform yours by adding logos, customizing fonts and colors, and hosting forms on your own domain for a fully branded experience.
Save & Resume Partial Submissions
Allow users to save their progress and return later to complete forms — improving completion rates for longer surveys or applications.
Collaborative Team Workspaces
Work together with your team in shared spaces to build, manage, and analyze forms efficiently.
Multilingual Forms
Reach and engage a global audience by creating forms in multiple languages with ease.
Custom Redirects & Thank You Pages
Personalize the post-submission experience by redirecting users to custom pages or displaying tailored thank-you messages.
Google Tag Manager Integration
Track form performance, conversions, and user behavior easily with full Google Tag Manager support.
Powerful dashboard to manage your forms
and get the insight you need.
Easily navigate all of FabForm's features through its powerful yet easy to use dashboard.
Rocking reviews
from our customers
Here is what our loyal customers have to say.
Roberta Johnson
I.T. Recruiter
"I'm amazed by the quality of the features offered on the FabForm platform. I evaluated other Form Builders on the market and FabForm comes out on top. The UI design feels better. It's feature rich, terrific pricing and it works a charm. "
Emilio Pirelli
Digital Marketing
"As a full-time Digital Marketing professional, I needed a flexible and easy way to create and monitor various marketing forms for some picky clients. FabForm has exceeded my expectations. It is -- in my humble opinion -- the best Form Builder out there barnone."
"FabForm is my absolute favorite form builder. I can throw together a beautiful form in minutes. It's reliable and has all the features and ease of use that one needs -- and then some."
Counting years backwards unlocks faster speed.
The first date algorithm to use only 4 multiplications, instead of 7+.
23 November 2025
In this article I present my final
very
fast date conversion algorithm. It represents a significant speed gain — being similar in magnitude to the speed gains achieved by the previous fastest algorithm (Neri-Schneider 2021) over its predecessor (C++ Boost). The full algorithm
implementation in C++
is released as free open source software (BSL-1.0 License).
The algorithm provides accurate results over a period of ±1.89 Trillion years, making it suitable to process the full UNIX 64–bit time (in seconds).
The entire algorithm has been re-written top-to-bottom, with various micro-optimisations, but three main new ideas:
Years are calculated
backwards
, which removes various intermediate steps.
The step to calculate day-of-year is
skipped
, instead using a year-modulus-bitshift technique which removes a division.
The
"Julian Map"
technique is utilised
from my previous article, which speeds up the 100/400 year calculation, removing two more hardware multiplications.
While fast date algorithms have always used 7 or more expensive computations (multiplication, division, or modulus by non-power-of-2 numbers),
this algorithm uses only 4 multiplications
. The speed-gain can be seen at a glance.
Relative Speeds of Fastest Algorithms
As tested on Intel x64 and Apple M4 Pro processors
(smaller numbers = faster)
See
benchmark section
for further information.
C++ Boost
(2012)
2.4×
Neri-Schneider
(2021)
1.6×
This Algorithm
(2025)
1×
The benchmark results match what is expected by hand-counting operations:
It is no surprise that most (if not all) fast date conversion algorithms have counted years in the forward direction, it is the default intuitive choice.
When doing so, one ends up with terms of the format
(foo * 4 + 3) / N
. This is one of the most common patterns you'll see across fast date algorithms. An example from C++ Boost is shown below:
The multiplication by
4
and division by
146097
and
1461
is not too hard to understand, the larger constants being the number of days in 400-years and 4-years respectively — but what about those “
+ 3
” terms?
The reason for those is due to the longer years and longer centuries being offset slightly from the epoch. This is best understood by viewing the tables below. The first table shows the distribution of leap years when counting from the year zero:
Traditional forward counting
years
— epoch: 0000-03-01
Year
0
1
2
3
Start
End
0000-03-01
0001-02-28
0001-03-01
0002-02-28
0002-03-01
0003-02-28
0003-03-01
0004-02-29
Length
+ Type
365
Normal
365
Normal
365
Normal
366
Leap
Since the Gregorian leap year rule omits a usual leap year every 100 years, except years divisible by 400, we get the same pattern for centuries:
Traditional forward counting
centuries
— epoch: 0000-03-01
Century
0
1
2
3
Start
End
0000-03-01
0100-02-28
0100-03-01
0200-02-28
0200-03-01
0300-02-28
0300-03-01
0400-02-29
Length
+ Type
36,524
Short
36,524
Short
36,524
Short
36,525
Long
If we could find a way to count the dates from an epoch where these both start with a
Leap/Long
immediately (instead of offset-by-3) we could delete the “
+ 3
” terms. Not only that, we will have a way of merging the terms:
foo * 4 / N
into a single multiplication and bit-shift. This has the potential to remove up to four CPU-cycles from the algorithm
(although the exact speed gain will be platform dependent)
.
I first tried by setting the epoch at the year
-100
(101 BC), but that only solves half the problem: the century slicing
must
end on a date
XX00-02-2Y
, which means it must start immediately
after
a February in a year divisible by
4
.
As you know, it turns out counting
backwards
is the way to solve this problem fully:
Backwards
counting
years
— epoch: 2400-02-29
Year
0
1
2
3
Start
End
2400-02-29
2399-03-01
2399-02-28
2398-03-01
2398-02-28
2397-03-01
2397-02-28
2396-03-01
Length
+ Type
366
Leap
365
Normal
365
Normal
365
Normal
This allows the epoch to begin in a leap-year as shown above, as well as to start in a long century as shown below:
Backwards
counting
centuries
— epoch: 2400-02-29
Century
0
1
2
3
Start
End
2400-02-29
2300-03-01
2300-02-28
2200-03-01
2200-02-28
2100-03-01
2100-02-28
2000-03-01
Length
+ Type
36,525
Long
36,524
Short
36,524
Short
36,524
Short
Fortunately, reversing the timeline comes at zero speed cost, as all date algorithms tend to have
epoch
adjustments. It will just be a matter of using a minus sign where an addition would normally be.
Now that this general concept is out of the way, we can proceed with the nitty-gritty of the individual steps of the algorithm.
The Algorithm - Line by Line
With the timeline reversed, we can now begin explaining the algorithm from the top:
Given:
days
=
Days since epoch, where "1970-01-01" is zero
— Then:
The Algorithm — Step 1
constERAS = 4726498270
constD_SHIFT = 146097 * ERAS - 719469
rev = D_SHIFT - days
The constants
D_SHIFT
is calculated at compile-time:
719469
is the number of days to count back from the UNIX epoch of
1970-01-01
to
0000-02-29
— our natural alignment point.
Note that this is one day earlier than many other algorithms which typically count from the date
0000-03-01
.
146097
is the number of days per 400 year
era
, and we will add a large multiple
ERAS
of these to set our far future count-back point.
In the example tables, we counted back from
2400
, which would mean in that case
ERAS = 6
.
The specific value for
ERAS = 4726498270
was selected to provide the maximum possible range centred around the UNIX epoch in 64–bit.
For 32–bit applications, the value of
14704
is appropriate.
Finally, the calculation of
rev
involves that minus sign to reverse the timeline.
The Algorithm — Step 2
constC1 = 505054698555331
cen = (rev*C1) >> 64
In this step, we are performing what was previously
cen = (days * 4 + 3) / 146097
, but with just a single multiplication.
Performing a division by doing multiplication followed by bit-shift ("mul-shift") is a very common technique in low-level algorithm design. If one uses actual division, the compiler will typically output something similar, but with a larger bit-shift.
There are two reasons that we are hand-writing our own mul-shift in this case
We want to integer-divide by
36524.25
— something that cannot be done with a standard integer division.
We can specify a bit-shift of
precisely
64
, which is “free” in 64–bit computers.
The reason a bit-shift of 64 in this case is “free” is due to the way 64–bit computers work. Since 64–bit computers cannot hold a number larger than 64–bits in a single register, a multiplication will place the full 128–bit result from the multiplication into two adjacent registers. The bit-shift of 64 is therefore just the compiled machine-code making use of the first of these two registers, and hence, does not actually involve any work.
The reason a compiler would normally use a larger bit-shift when using a normal division, is that eventually, with astronomically large values, this calculation will no longer be correct. We don't need this to be correct forever, just for a very large range to make the algorithm useful. A compiler does not know such requirements, and will usually err on the safe side and go with the more accurate, larger, and costlier, bit-shift.
The Algorithm — Step 3
jul = rev + cen - cen / 4
This is also a “new” step not found in
recent
fast date algorithms.
What we are doing here is quickly, and efficiently, accounting for the 100-year and 400-year leap year rules. Boost and Neri-Schneider both use alternative steps to calculate the specific day within the associated 400-year block (costly modulus or subtraction and multiplication), and later construct the year with another costly addition and multiplication.
I wrote about this technique in a
previous article
earlier this month. It is a technique used by some former date-algorithm designers, but the speed gains were either not well understood, or not communicated clearly to later designers. I recommend reading the previous article for more details.
The Algorithm — Step 4
constY_SHIFT = 400 * ERAS - 1
constC2 = 50504432782230121
num = jul*C2
yrs = Y_SHIFT - (num>> 64)
low = num % (1 << 64)
Once again, we are doing a fast division with a single multiplication, where previous algorithms would have to
first
calculate
4 * jul + 3
.
This time we are not immediately performing the bit-shift by
64
, as we will utilise both the high and low parts of the result. This technique was pioneered by
Neri‑Schneider
.
The high 64–bit part of
num
, represents the number of years that have elapsed backwards since our forward-epoch, so subtracting this by a carefully selected constant will yield the correct year. As with most other fast date algorithms, this year will later need to be incremented if it is determined that the month is either January or February, since the internal logic of the algorithm is using treating the start of the year as 1 March.
While the upper-64 bits are easy to explain, the lower 64 bits are going to be treated different to usual.
The Algorithm — Step 5
ypt = (782432 *low) >> 64
This is where things start to get very different. In the Neri-Schneider algorithm, the lower bits are
divided
by a constant in order to calculate the exact
day_of_year
(0-365). That
day_of_year
is then later
multiplied
again to recover the month and day.
At first glance, a division followed by a multiplication sounds like it could be merged into a single combined operation. However, the division step introduces an important integer rounding, and that rounding cannot be removed without changing the result.
In this algorithm, we deliberately skip the explicit
day_of_year
step and merge the multiplication and division together to get just a
year-part
. This temporarily loses that rounding effect, causing a drift in the value of
year-part
equal to 1/4 of a day per year — resetting every 4 years. We will allow this “error” to occur and correct it later using a year-modulus-bitshift technique. Hold that thought for now — we will return to it when we construct
N
on line 27 (Step 7).
The Algorithm — Step 6
bump = ypt < 126464
shift = bump ? 191360 : 977792
We will adopt the term
bump
used by Neri-Schneider to indicate that the year beginning
1 March
has overflowed to the next calendar year — i.e. the month is January or February. Other algorithms often use
day_of_year
or
month
to calculate this, however we are not computing
day_of_year
, and we'll require
bump
in order to calculate
month
. As such, we'll determine this value early via this more cryptic looking method.
Note that
ypt
(year part) is still in the reverse direction, so the “final” months of January and February
(final for a computational year beginning 1 March)
are actually low numbers instead of the more natural high numbers one usually expects.
Next,
shift
is a value that will be used in the next line to shift a linear equation by 12-months to achieve a similar “bump” overflow to our year.
Note that the difference between the two “shift” options is 786,432, which is equal to 12 × 2
16
. Since the next linear equation uses 2
16
as the denominator, this achieves the shift by 12 months where required, in only 1 CPU-cycle.
Other fast date algorithms often use a step near the end such as
month = bump ? month - 12 : month
, however on x64 processors that takes 2 CPU-cycles to compute, as
M - 12
must always be calculated before the ternary check is performed.
The Algorithm — Step 7
N = (yrs % 4) * 512 + shift - ypt
This is the
year-modulus-bitshift
magic hinted at earlier. The value of
N
will be split into two parts, where the high 16 bits will become the
month
and the low 16 bits will map to the
day
.
We have already explained
shift
, but
(
yrs
% 4) * 512
is certainly an unusual looking term.
In simple terms, if we pretend that months are 32-days long, this represents a yearly shift of exactly 1/4 of a day (resetting every four years). This cancels the error discussed in Step 5.
How is it 1/4 of a day?
In our fake 32-day-month model, 512 units correspond to exactly one quarter of a day within the 16–bit space. In binary terms,
512
is (2
16
/ 32) / 4 — that is, one quarter of one fake day. The value 512 is also a power of two, so it compiles to a simple left-shift by
9
.
Of course, actual Gregorian months are not exactly 32 days long, but they are
just close enough
to 32 days long to make this work. There is just enough wiggle room in the valid range of
shift
for the rounding to land correctly across all months.
Note:
at first this trickery looks like it might only save minimal cycles, but it is actually a necessary step to avoid those pesky
+3
terms that we worked so hard to remove earlier. If we used the Neri–Schneider technique from line 17 onwards, that offset would have reappeared.
It is very fortunate that the Julian and Gregorian calendars have month-lengths close to a power of two to make this trick possible. Other calendars that have month-lengths closer to the actual synodic lunar month might not be able to adopt this technique.
The Algorithm — Step 8
constC3 = 8619973866219416
D = ((N% 65536) * C3) >> 64
This is the last non-trivial step, where we take the lower 16–bits and map them to the day-number.
This day number will be a value in the range of 0-30, and will need to be incremented at the end.
Neri-Schneider use division by
2141
here, however I found the value of
2140
was seemingly required in order to achieve the
year-modulus-bitshift
technique.
We also use a hand-crafted mul-shift in order to ensure that the bit-shift is the “free” 64–bit variation.
The Algorithm — Step 9
day = D + 1
month = N / 65536
year = yrs + bump
Finally, we clean up the values as previously described:
Increment day so it is 1-indexed
Take the upper 16 bits of
N
for
month
Overflow the
year
when the month is January or February.
That's it.
The date has now been calculated faster than ever before.
Optimisations for ARM64 and Apple Silicon
Many ARM chips take longer to load integers that are larger than 16 bits (max: 65,535).
There are a handful of integers between lines 17-25 that are larger than this threshold, namely:
782432
,
126464
,
191360
,
977792
,
65536
.
The reason for their size is that they allow calculation of
D
to involve
N % 65536
, which compiles to taking the lowest 16–bits of
N
, which is a
free
operation on x64.
These constants have been carefully selected to each be divisible by 32. One can use a compile-time check to divide these constants by 32 if the target is ARM (or use the smaller constants and scale by 32 for x64). When doing so, they all fit under 16–bit in size, leading to a speed bump for many ARM chips. Other constants also need adjusting in this case: divide
512
by 32 (for calculating
N
), and the constant of
C3
should be
increased
by a factor
32
.
Additionally, in Step-6 of the explanation, I noted that the new approach using
shift
specifically improves the performance on x64 based computers, and
should
have no effect on ARM devices. From my testing, I have found that this particular optimisation negatively impacts the performance on Apple M4 Pro, presumably due to some parallelisation that is lost. As such, my recommendation is to use a compile-time check to limit this improvement specifically to x64 devices.
Adopting the above changes results in the following changes to the algorithm:
Note: terms highlighted in Green are simplified at compile-time.
ARM64 / Apple Silicon Improvement
#if IS_ARM
constSCALE = 1
#else
constSCALE = 32
#endif
constC3 = 8619973866219416 * 32 / SCALE
ypt = ((24451 * SCALE) * low) >> 64
#if IS_ARM
shift = (30556 * SCALE)
#else
bump = ypt < 126464
shift = bump ? (24412 * SCALE) : (30556 * SCALE)
#endif
N = (yrs % 4) * (16 * SCALE) + shift - ypt
D = ((N % (2048 * SCALE)) * C3) >> 64
M = N / (2048 * SCALE)
#if IS_ARM
bump = M > 12
month = bump ? M - 12 : M
#else
month = M
#endif
Accuracy and Range
Fortunately, the range of accurate values of the algorithm is very wide:
Total Days
1,381,054,434,006,886
~1.381 × 10
15
~1.381 Quadrillion
~2
50.29
Total Years
3,781,198,611,900
~3.781 × 10
12
~3.781 Trillion
~2
41.78
Max Date
+1,890,599,308,000-02-29 — Rata Die: +690,527,217,032,721
Min Date
−1,890,599,303,900-03-01 — Rata Die: −690,527,216,974,164
The first date above this range will overflow.
The first date below this range will incorrectly return February 29 in a non-leap year.
64–bit UNIX time (measured in seconds) covers a range of around 585 Billion years in total, while this algorithm is accurate in the range of Trillions of years. This makes it sufficient to handle the full 64–bit UNIX time range.
While I have not developed a formal proof of this range, there is a
testcase
in the benchmark codebase which verifies the above range. It is very slow to check the entire range (around a month on Apple M4 Pro), so it first quickly checks:
[-2
32
... 2
32
]
[MAX_DATE − 2
32
... MAX_DATE]
[MIN_DATE ... MIN_DATE + 2
32
]
A random sample of
2
32
dates
Portability
Since this algorithm is 64–bit only, some library authors may be interested in techniques to safely deploy this in a portable manner that can support 32–bit computers.
The fast 32–bit algorithms from
Article 1
(fastest) and
Article 2
(full 32–bit input range safe), have been updated with techniques from this algorithm. They are each now notably faster than as stated in their respective articles.
If your API only needs to support a restricted day range of around 2
30
days (~1 Billion days / ~2.9 Million years), then the following combination is the fastest known for each platform type:
My algorithm from Article 2 is the only fast 32–bit date algorithm that I am aware of that is 100% overflow safe within the 32–bit input range. It is both overflow-safe, and very fast. It is faster than Boost but either slower or faster than Neri-Schneider depending on the device. Note that Boost and Neri-Schneider only support around 25% of the 32–bit input range.
This overflow-safe algorithm was created with the sole purpose of supporting the scenario noted above.
Benchmark Results
The benchmark code is a direct fork of the benchmarks provided by Neri‑Schneider
(GitHub link
)
. The relative speed ratios are calculated by first subtracting the "scan" performance, which removes the overhead of calling the function from the benchmarks, as is also used by Neri‑Schneider.
MacBook Pro 2024 (MacOS 15.6.1)
Apple M4 Pro
Compiler: Apple clang 17.0.0
-
(3720)
2.45x
(32147)
1.62x
(22520)
1.38x
(19751)
1.92x
(25957)
1.00x
(15304)
38.4%
The above platforms were chosen because they produce stable, consistent results that match expectations. I also tested the algorithm on a Lenovo IdeaPad Slim 5 (Snapdragon ARM64), but it reported nearly a 60% speed gain. This strongly suggests a thermal or power-management behaviour that distorts short benchmark loops. Similarly, x86-based MacBook Pros from 2016 and 2020 reported speed improvements of only 2–10%, while other algorithms appeared twice as slow as expected. This again indicates that the CPU is applying aggressive battery or thermal optimisation, making those measurements unreliable.
If you found this interesting, you should
follow me on X
to get notified when I publish more date and algorithm related articles.
Hell Gate’s 2025 Guide to Navigating Difficult Thanksgiving Conversations
Thanksgiving is a time to contemplate what we are grateful for in our lives, to prepare an elaborate dinner, and, for many of us, to enjoy the company of family. But as we all know, family members don't always see things the same way, and sometimes conversation around the Thanksgiving dinner table can get a little rocky.
Navigating these difficult family encounters can be stressful, frustrating, or downright traumatic. It's important to remember that you can't control how your family behaves, but you can control how you respond. With that in mind, Hell Gate would like to offer some examples of how to productively engage with loved ones over the Thanksgiving holiday.
Scenario 1
Your uncle Danny, who owns a car dealership in Long Island, interrupts a story about your co-op shift to say that he'd never live in New York City, because the crime is out of control, the subways are murder traps, and "illegals" are taking over.
Solution:
You could blow your top, call him a racist MAGA blowhard who doesn't know what he's talking about, storm away from the table, and poison the whole family gathering. Or, you could try saying something like this:
"Well Danny, when I talk to everyday New Yorkers—no matter where they're from originally—they all share many of the same concerns. They all want safety, and they all want justice. But what they want most of all is a city that they can afford to live in. I bet affordability is even an issue for you guys up in Syosset, right?"
Boom! You can bet Danny has some gripes about the cost of living he's ready to share—now you're back on the same page and the conversation is back on track.
Scenario 2
Every Thanksgiving, you make cheddar mashed potatoes as a side dish. And every year, they're a hit. But this year, your father-in-law is insisting that you make the gross sweet potato pecan casserole instead—
the one with marshmallows in it
. No one actually wants to eat this, especially because the turkey he prepares is dry and flavorless, but Frank is adamant: "My house, my side dishes."
You could stand on your principles and stick with the cheddar potatoes, but that might involve an uncomfortable scene. Everyone is looking to you to save a Thanksgiving tradition.
Solution:
Make the disgusting marshmallow abomination. After all, your father-in-law has done a decent job raising such a wonderful family, and this is arguably a small price to pay to stay in his good graces.
To smooth it over with the rest of the family, tell them, "We need not choose between justice and safety, and with this decision, we are affirming our commitment to both. And while Frank and I don't agree on everything, we both share the same passion for furthering an agenda of a successful Thanksgiving, one that delivers on a promise of fellowship, football, and a pumpkin pie at the end."
If one of your cousins points out that this sweet potato side dish is basically a dessert, and one that is far too similar to pumpkin pie, sidestep his question with something like, "No, what I believe, is that both cheddar potatoes and sweet potato casseroles should be taken seriously, but that at this juncture, the imperative to execute on the casserole should take precedence so that our larger goals, those that involve a restful and meaningful holiday, ultimately prevail."
Scenario 3
Your aunt Glenda starts to say the blessing, but is rudely cut off by your cousin Sarah, who promised one of her kids, Dylan, who is 3, that
he
could say the blessing this year. Awkward silence at the table ensues, as the pro-Glenda and pro-Dylan factions glower at each other. How do you stop this from going off the rails?
Solution:
Remind your family about why they are here: to appreciate the fact that everyone at this table agrees that the cost of living is simply too high, but that if we all work together, we can take actions to reverse this trend.
Say something like, "I'm talking about groceries for Glenda, and I'm talking about day care for Dylan. I'm talking about utility bills too—Frank, you were saying earlier that your PSEG bill was insane last month?" (at this point nod at your father-in-law, who will appreciate this, even if he didn't specifically talk to you about his utility bill, there is a good chance he complained about it to someone else).
Definitely add something like, "Let us all appreciate this time we have together. As Eugene Debs
once said
, 'Upon every hand we see the signs of preparation,' and I definitely see the hands here at this table ready to dig into this delicious turkey and stuffing and sweet potato casserole! Right, Frank? Now let's raise our glasses to a new dawn, one where we usher in a new era of leadership, one that speaks for all of us—young and old—without condescension, and without compromising our basic commitment to ensure a decent standard of living. Years from now, may our only regret be that this day took so long to come. Cheers!"
Scenario 4
After the big meal, uncle Phil wants to hide the wishbone for the little kids, and whoever finds it, gets to break it. Everyone else is tired, or engaged in cleaning, no one is jumping at the chance to play a game with a group of hyperactive children. Many of the adults are rolling their eyes, but Uncle Phil, who is kindhearted and genuine, approaches you for backup. "Will you help me play this game? Maybe start a
new
tradition?" he asks, poking you in the ribs and handing you another way-too-strong Nitro Double IPA that he brought, and has been sitting in the trunk of his Impala since last Thanksgiving.
Solution:
Put your hand on Phil's shoulder and say that you will help him. Look around for something to stand on—maybe an ottoman, or a step stool—and climb up it before announcing, "Far too often, the traditions of the past have been forgotten by the politics of the present. Tonight let us speak in a clear voice: Hope is alive. Hope for a time when we can play some low-effort games to entertain the kids after a satisfying meal. Hope that those same kids can afford to grow up and raise their own families in the city that they call home. And we will build a Thanksgiving tradition defined by competence and a compassion that have for too long been placed at odds with one another."
At this point, everyone in the room should be applauding, so add this: "Together, we will usher in a generation of change. And if we embrace this brave new course, rather than fleeing from it, we can respond to cynicism and solipsism with the strength it fears, not the appeasement it craves."
Scenario 5
Your partner corners you outside of the bathroom and whispers, "Why are you talking in platitudes and acting like a fucking stooge? Can't you be normal for two fucking hours? Jesus fucking Christ."
Solution:
Walk up to the crowd of uncles gathered around the TV watching American football and say, "Hey did any of you catch that
Arsenal game yesterday
? It was amazing! I think we have
Paramount+ on this puppy
, let's watch some highlights at halftime!"
Comcast to pay $1.5M fine for vendor breach affecting 270K customers
Bleeping Computer
www.bleepingcomputer.com
2025-11-26 18:30:10
Comcast will pay a $1.5 million fine to settle a Federal Communications Commission investigation into a February 2024 vendor data breach that exposed the personal information of nearly 275,000 customers. [...]...
Comcast will pay a $1.5 million fine to settle a Federal Communications Commission investigation into a February 2024 vendor data breach that exposed the personal information of nearly 275,000 customers.
The breach
occurred in February 2024
, when attackers hacked into the systems of Financial Business and Consumer Solutions (FBCS), a debt collector Comcast had stopped using two years earlier.
The FCBS data breach was initially believed to have affected 1.9 million people in total, but the tally was
raised to 3.2 million
in June and, finally,
to 4.2 million
in July.
FBCS, which filed for bankruptcy before revealing a data breach in August 2024, notified Comcast on July 15 (five months after the attack) that customer data had been compromised,
affecting 273,703 Comcast customers
. Previously, it had assured Comcast in March that the breach did not affect any of its customers.
The threat actors stole personal and financial information between February 14 and February 26, including the names, addresses, Social Security numbers, dates of birth, and Comcast account numbers of affected current and former customers. Affected customers had used Comcast's Xfinity-branded internet, television, streaming, VoIP, and home security services.
Under the
consent decree
announced by the FCC
on Monday
, Comcast has also agreed to implement a compliance plan that includes enhanced vendor oversight to protect data and ensure customer privacy, ensuring its vendors properly dispose of customer information they no longer need for business purposes, as required by the Cable Communications Policy Act of 1984.
The telecommunications giant must also appoint a compliance officer, conduct risk assessments of vendors handling customer data every two years, file compliance reports with the FCC every six months over the next three years, and report any material violations within 30 days of discovery.
However, Comcast said in a statement to
Reuters
that it "was not responsible for and has not conceded any wrongdoing in connection with this incident," noting that its network wasn't breached and that FBCS was contractually required to comply with security requirements.
A Comcast spokesperson was not immediately available for comment when contacted by BleepingComputer.
Comcast is an American mass media, telecommunications, and entertainment multinational company, and the fourth-largest telecom firm in the world by revenue, after AT&T, Verizon, and China Mobile.
It also has over 182,000 employees, hundreds of millions of customers worldwide, and reported revenues of $123.7 billion in 2024.
Whether you're cleaning up old keys or setting guardrails for AI-generated code, this guide helps your team build securely from the start.
Get the cheat sheet and take the guesswork out of secrets management.
✋ Get A Warrant | EFFector 37.17
Electronic Frontier Foundation
www.eff.org
2025-11-26 18:16:27
Even with the holidays coming up, the digital rights news doesn't stop. Thankfully, EFF is here to keep you up-to-date with our EFFector newsletter!
In our latest issue, we’re explaining why politicians latest attempts to ban VPNs is a terrible idea; asking supporters to file public comments opposin...
Since 1990 EFF has
published EFFector
to help keep readers on the bleeding edge of their digital rights. We know that the intersection of technology, civil liberties, human rights, and the law can be complicated, so EFFector is a great way to stay on top of things. The newsletter is chock full of links to updates, announcements, blog posts, and other stories to help keep readers—and listeners—up to date on the movement to protect online privacy and free expression.
Thank you to the supporters around the world who make our work possible! If you're not a member yet,
join EFF today
to help us fight for a brighter digital future.
This guide covers ~30 pro-tips for effectively using Gemini CLI for agentic coding
Gemini CLI
is an open-source AI assistant that brings the power of Google's Gemini model directly into your
terminal
. It functions as a conversational, "agentic" command-line tool - meaning it can reason about your requests, choose tools (like running shell commands or editing files), and execute multi-step plans to help with your development
workflow
.
In practical terms, Gemini CLI acts like a supercharged pair programmer and command-line assistant. It excels at coding tasks, debugging, content generation, and even system automation, all through natural language prompts. Before diving into pro tips, let's quickly recap how to set up Gemini CLI and get it running.
Installation:
You can install Gemini CLI via npm. For a global install, use:
npm install -g @google/gemini-cli
Or run it without installing using
npx
:
Gemini CLI is available on all major platforms (it's built with Node.js/TypeScript). Once installed, simply run the
gemini
command in your terminal to launch the interactive
CLI
.
Authentication:
On first use, you'll need to authenticate with the Gemini service. You have two options: (1)
Google Account Login (free tier)
- this lets you use Gemini 2.5 Pro for free with generous usage limits (about 60 requests/minute and 1,000 requests per
day
. On launch, Gemini CLI will prompt you to sign in with a Google account (no billing
required
. (2)
API Key (paid or higher-tier access)
- you can get an API key from Google AI
Studio
and set the environment variable
GEMINI_API_KEY
to use
it
.
API key usage can offer higher quotas and enterprise data‑use protections; prompts aren't used for training on paid/billed usage, though logs may be retained for
safety
.
For example, add to your shell profile:
export GEMINI_API_KEY="YOUR_KEY_HERE"
Basic Usage:
To start an interactive session, just run
gemini
with no arguments. You'll get a
gemini>
prompt where you can type requests or commands. For instance:
$ gemini
gemini> Create a React recipe management app using SQLite
You can then watch as Gemini CLI creates files, installs dependencies, runs tests, etc., to fulfill your request. If you prefer a one-shot invocation (non-interactive), use the
-p
flag with a prompt, for example:
gemini -p "Summarize the main points of the attached file. @./report.txt"
This will output a single response and
exit
. You can also pipe input into Gemini CLI: for example,
echo "Count to 10" | gemini
will feed the prompt via
stdin
.
CLI Interface:
Gemini CLI provides a rich REPL-like interface. It supports
slash commands
(special commands prefixed with
/
for controlling the session, tools, and settings) and
bang commands
(prefixed with
!
to execute shell commands directly). We'll cover many of these in the pro tips below. By default, Gemini CLI operates in a safe mode where any action that modifies your system (writing files, running shell commands, etc.) will ask for confirmation. When a tool action is proposed, you'll see a diff or command and be prompted (
Y/n
) to approve or reject it. This ensures the AI doesn't make unwanted changes without your consent.
With the basics out of the way, let's explore a series of pro tips and hidden features to help you get the most out of Gemini CLI. Each tip is presented with a simple example first, followed by deeper details and nuances. These tips incorporate advice and insights from the tool's creators (e.g. Taylor Mullen) and the Google Developer Relations team, as well as the broader community, to serve as a
canonical guide for power users
of Gemini CLI.
Tip 1: Use
GEMINI.md
for Persistent Context
Quick use-case:
Stop repeating yourself in prompts. Provide project-specific context or instructions by creating a
GEMINI.md
file, so the AI always has important background knowledge without being told every
time
.
When working on a project, you often have certain overarching details - e.g. coding style guidelines, project architecture, or important facts - that you want the AI to keep in mind. Gemini CLI allows you to encode these in one or more
GEMINI.md
files. Simply create a
.gemini
folder (if not already present) in your project, and add a Markdown file named
GEMINI.md
with whatever notes or instructions you want the AI to persist. For example:
# Project Phoenix - AI Assistant- All Python code must follow PEP 8 style.
- Use 4 spaces for indentation.
- The user is building a data pipeline; prefer functional programming paradigms.
Place this file in your project root (or in subdirectories for more granular context). Now, whenever you run
gemini
in that project, it will automatically load these instructions into
context
. This means the model will
always
be primed with them, avoiding the need to prepend the same guidance to every prompt.
How it works:
Gemini CLI uses a hierarchical context loading
system
. It will combine
global context
(from
~/.gemini/GEMINI.md
, which you can use for cross-project defaults) with your
project-specific
GEMINI.md
, and even context files in subfolders. More specific files override more general ones. You can inspect what context was loaded at any time by using the command:
This will display the full combined context the AI
sees
. If you make changes to your
GEMINI.md
, use
/memory refresh
to reload the context without restarting the
session
.
Pro Tip:
Use the
/init
slash command to quickly generate a starter
GEMINI.md
. Running
/init
in a new project creates a template context file with information like the tech stack detected, a summary of the project,
etc
.. You can then edit and expand that file. For large projects, consider breaking the context into multiple files and
importing
them into
GEMINI.md
with
@include
syntax. For example, your main
GEMINI.md
could have lines like
@./docs/prompt-guidelines.md
to pull in additional context
files
. This keeps your instructions organized.
With a well-crafted
GEMINI.md
, you essentially give Gemini CLI a "memory" of the project's requirements and conventions. This
persistent context
leads to more relevant responses and less back-and-forth prompt engineering.
Tip 2: Create Custom Slash Commands
Quick use-case:
Speed up repetitive tasks by defining your own slash commands. For example, you could make a command
/test:gen
that generates unit tests from a description, or
/db:reset
that drops and recreates a test database. This extends Gemini CLI's functionality with one-liners tailored to your workflow.
Gemini CLI supports
custom slash commands
that you can define in simple configuration files. Under the hood, these are essentially pre-defined prompt templates. To create one, make a directory
commands/
under either
~/.gemini/
for global commands or in your project's
.gemini/
folder for project-specific
commands
. Inside
commands/
, create a TOML file for each new command. The file name format determines the command name: e.g. a file
test/gen.toml
defines a command
/test:gen
.
Let's walk through an example. Say you want a command to generate a unit test from a requirement description. You could create
~/.gemini/commands/test/gen.toml
with the following content:
# Invoked as: /test:gen "Description of the test"
description \= "Generates a unit test based on a requirement."
prompt \= """
You are an expert test engineer. Based on the following requirement, please write a comprehensive unit test using the Jest framework.
Requirement: {{args}}
"""
Now, after reloading or restarting Gemini CLI, you can simply type:
/test:gen "Ensure the login button redirects to the dashboard upon success"
Gemini CLI will recognize
/test:gen
and substitute the
{{args}}
in your prompt template with the provided argument (in this case, the requirement). The AI will then proceed to generate a Jest unit test
accordingly
. The
description
field is optional but is used when you run
/help
or
/tools
to list available commands.
This mechanism is extremely powerful - effectively, you can script the AI with natural language. The community has created numerous useful custom commands. For instance, Google's DevRel team shared a set of
10 practical workflow commands
(via an open-source repo) demonstrating how you can script common flows like creating API docs, cleaning data, or setting up boilerplate
code
. By defining a custom command, you package a complex prompt (or series of prompts) into a reusable shortcut.
Pro Tip:
Custom commands can also be used to enforce formatting or apply a "persona" to the AI for certain tasks. For example, you might have a
/review:security
command that always prefaces the prompt with "You are a security auditor..." to review code for vulnerabilities. This approach ensures consistency in how the AI responds to specific categories of tasks.
To share commands with your team, you can commit the TOML files in your project's repo (under
.gemini/commands
directory). Team members who have Gemini CLI will automatically pick up those commands when working in the project. This is a great way to
standardize AI-assisted workflows
across a team.
Tip 3: Extend Gemini with Your Own
MCP
Servers
Quick use-case:
Suppose you want Gemini to interface with an external system or a custom tool that isn't built-in - for example, query a proprietary database, or integrate with Figma designs. You can do this by running a custom
Model Context Protocol (MCP) server
and plugging it into Gemini
CLI
. MCP servers let you add new tools and abilities to Gemini, effectively
extending the agent
.
Gemini CLI comes with several MCP servers out-of-the-box (for instance, ones enabling Google Search, code execution sandboxes, etc.), and you can add your own. An MCP server is essentially an external process (it could be a local script, a microservice, or even a cloud endpoint) that speaks a simple protocol to handle tasks for Gemini. This architecture is what makes Gemini CLI so
extensible
.
Examples of MCP servers:
Some community and Google-provided MCP integrations include a
Figma MCP
(to fetch design details from Figma), a
Clipboard MCP
(to read/write from your system clipboard), and others. In fact, in an internal demo, the Gemini CLI team showcased a "Google Docs MCP" server that allowed saving content directly to Google
Docs
. The idea is that whenever Gemini needs to perform an action that the built-in tools can't handle, it can delegate to your MCP server.
How to add one:
You can configure MCP servers via your
settings.json
or using the CLI. For a quick setup, try the CLI command:
This would register a server named "myserver" that Gemini CLI will launch by running the given command (here a Python module) on port 8080. In
~/.gemini/settings.json
, it would add an entry under
mcpServers
. For example:
This configuration (based on the official docs) tells Gemini how to start the MCP server and
where
. Once running, the tools provided by that server become available to Gemini CLI. You can list all MCP servers and their tools with the slash command:
This will show any registered servers and what tool names they
expose
.
Power of MCP:
MCP servers can provide
rich, multi-modal results
. For instance, a tool served via MCP could return an image or a formatted table as part of the response to Gemini
CLI
. They also support OAuth 2.0, so you can securely connect to APIs (like Google's APIs, GitHub, etc.) via an MCP tool without exposing
credentials
. Essentially, if you can code it, you can wrap it as an MCP tool - turning Gemini CLI into a hub that orchestrates many services.
Default vs. custom:
By default, Gemini CLI's built-in tools cover a lot (reading files, web search, executing shell commands, etc.), but MCP lets you go beyond. Some advanced users have created MCP servers to interface with internal systems or to perform specialized data processing. For example, you could have a
database-mcp
that provides a
/query_db
tool for running SQL queries on a company database, or a
jira-mcp
to create tickets via natural language.
When creating your own, be mindful of security: by default, custom MCP tools require confirmation unless you mark them as trusted. You can control safety with settings like
trust: true
for a server (which auto-approves its tool actions) or by whitelisting specific safe tools and blacklisting dangerous
ones
.
In short,
MCP servers unlock limitless integration
. They're a pro feature that lets Gemini CLI become a glue between your AI assistant and whatever system you need it to work with. If you're interested in building one, check out the official
MCP guide
and community examples.
Tip 4: Leverage Memory Addition & Recall
Quick use-case:
Keep important facts at your AI's fingertips by adding them to its long-term memory. For example, after figuring out a database port or an API token, you can do:
/memory add "Our staging RabbitMQ is on port 5673"
This will store that fact so you (or the AI) don't forget it
later
. You can then recall everything in memory with
/memory show
at any time.
The
/memory
commands provide a simple but powerful mechanism for
persistent memory
. When you use
/memory add <text>
, the given text is appended to your project's global context (technically, it's saved into the global
~/.gemini/GEMINI.md
file or the project's
GEMINI.md
. It's a bit like taking a note and pinning it to the AI's virtual bulletin board. Once added, the AI will always see that note in the prompt context for future interactions, across sessions.
Consider an example: you're debugging an issue and discover a non-obvious insight ("The config flag
X_ENABLE
must be set to
true
or the service fails to start"). If you add this to memory, later on if you or the AI are discussing a related problem, it won't overlook this critical detail - it's in the context.
Using
/memory
:
/memory add "<text>"
- Add a fact or note to memory (persistent context). This updates the
GEMINI.md
immediately with the new entry.
/memory show
- Display the full content of the memory (i.e. the combined context file that's currently loaded).
/memory refresh
- Reload the context from disk (useful if you manually edited the
GEMINI.md
file outside of Gemini CLI, or if multiple people are collaborating on it).
Because the memory is stored in Markdown, you can also manually edit the
GEMINI.md
file to curate or organize the info. The
/memory
commands are there for convenience during conversation, so you don't have to open an editor.
Pro Tip:
This feature is great for "decision logs." If you decide on an approach or rule during a chat (e.g., a certain library to use, or an agreed code style), add it to memory. The AI will then recall that decision and avoid contradicting it later. It's especially useful in long sessions that might span hours or days - by saving key points, you mitigate the model's tendency to forget earlier context when the conversation gets long.
Another use is personal notes. Because
~/.gemini/GEMINI.md
(global memory) is loaded for all sessions, you could put general preferences or information there. For example, "The user's name is Alice. Speak politely and avoid slang." It's like configuring the AI's persona or global knowledge. Just be aware that global memory applies to
all
projects, so don't clutter it with project-specific info.
In summary,
Memory Addition & Recall
helps Gemini CLI maintain state. Think of it as a knowledge base that grows with your project. Use it to avoid repeating yourself or to remind the AI of facts it would otherwise have to rediscover from scratch.
Tip 5: Use Checkpointing and
/restore
as an Undo Button
Quick use-case:
If Gemini CLI makes a series of changes to your files that you're not happy with, you can
instantly roll back
to a prior state. Enable checkpointing when you start Gemini (or in settings), and use the
/restore
command to undo changes like a lightweight Git
revert
.
/restore
rolls back your workspace to the saved checkpoint; conversation state may be affected depending on how the checkpoint was captured.
Gemini CLI's
checkpointing
feature acts as a safety net. When enabled, the CLI takes a snapshot of your project's files
before
each tool execution that modifies
files
. If something goes wrong, you can revert to the last known good state. It's essentially version control for the AI's actions, without you needing to manually commit to Git each time.
How to use it:
You can turn on checkpointing by launching the CLI with the
--checkpointing
flag:
Alternatively, you can make it the default by adding to your config (
"checkpointing": { "enabled": true }
in
settings.json
). Once active, you'll notice that each time Gemini is about to write to a file, it says something like "Checkpoint saved."
If you then realize an AI-made edit is problematic, you have two options:
Run
/restore list
(or just
/restore
with no arguments) to see a list of recent checkpoints with timestamps and descriptions.
Run
/restore <id>
to rollback to a specific checkpoint. If you omit the id and there's only one pending checkpoint, it will restore that by
default
.
For example:
Gemini CLI might output:
0: [2025-09-22 10:30:15] Before running 'apply_patch'
1: [2025-09-22 10:45:02] Before running 'write_file'
You can then do
/restore 0
to revert all file changes (and even the conversation context) back to how it was at that checkpoint. In this way, you can "undo" a mistaken code refactor or any other changes Gemini
made
.
What gets restored:
The checkpoint captures the state of your working directory (all files that Gemini CLI is allowed to modify) and the workspace files (conversation state may also be rolled back depending on how the checkpoint was captured). When you restore, it overwrites files to the old version and resets the conversation memory to that snapshot. It's like time-traveling the AI agent back to before it made the wrong turn. Note that it won't undo external side effects (for example, if the AI ran a database migration, it can't undo that), but anything in the file system and chat context is fair game.
Best practices:
It's a good idea to keep checkpointing on for non-trivial tasks. The overhead is small, and it provides peace of mind. If you find you don't need a checkpoint (everything went well), you can always clear it or just let the next one overwrite it. The development team recommends using checkpointing especially before multi-step code
edits
. For mission-critical projects, though, you should still use a proper version control (
git
) as your primary safety
net
- consider checkpoints as a convenience for quick undo rather than a full VCS.
In essence,
/restore
lets you use Gemini CLI with confidence. You can let the AI attempt bold changes, knowing you have an
"OH NO" button
to rewind if needed.
Tip 6: Read Google Docs, Sheets, and More. With a Workspace MCP server configured, you can paste a Docs/Sheets link and have the MCP fetch it, subject to permissions
Quick use-case:
Imagine you have a Google Doc or Sheet with some specs or data that you want the AI to use. Instead of copy-pasting the content, you can provide the link, and with a configured Workspace MCP server Gemini CLI can fetch and read it.
For example:
Summarize the requirements from this design doc: https://docs.google.com/document/d/<id>
Gemini can pull in the content of that Doc and incorporate it into its response. Similarly, it can read Google Sheets or Drive files by link.
How this works:
These capabilities are typically enabled via
MCP integrations
. Google's Gemini CLI team has built (or is working on) connectors for Google Workspace. One approach is running a small MCP server that uses Google's APIs (Docs API, Sheets API, etc.) to retrieve document content when given a URL or
ID
. When configured, you might have slash commands or tools like
/read_google_doc
or simply an auto-detection that sees a Google Docs link and invokes the appropriate tool to fetch it.
For example, in an Agent Factory podcast demo, the team used a
Google Docs MCP
to save a summary directly to a
doc
- which implies they could also read the doc's content in the first place. In practice, you might do something like:
@https://docs.google.com/document/d/XYZ12345
Including a URL with
@
(the context reference syntax) signals Gemini CLI to fetch that resource. With a Google Doc integration in place, the content of that document would be pulled in as if it were a local file. From there, the AI can summarize it, answer questions about it, or otherwise use it in the conversation.
Similarly, if you paste a Google Drive
file link
, a properly configured Drive tool could download or open that file (assuming permissions and API access are set up).
Google Sheets
could be made available via an MCP that runs queries or reads cell ranges, enabling you to ask things like "What's the sum of the budget column in this Sheet [link]?" and have the AI calculate it.
Setting it up:
As of this writing, the Google Workspace integrations may require some tinkering (obtaining API credentials, running an MCP server such as the one described by
Kanshi Tanaike
, etc.). Keep an eye on the official Gemini CLI repository and community forums for ready-to-use extensions - for example, an official Google Docs MCP might become available as a plugin/extension. If you're eager, you can write one following guides on how to use Google APIs within an MCP
server
. It typically involves handling OAuth (which Gemini CLI supports for MCP servers) and then exposing tools like
read_google_doc
.
Usage tip:
When you have these tools, using them can be as simple as providing the link in your prompt (the AI might automatically invoke the tool to fetch it) or using a slash command like
/doc open <URL>
. Check
/tools
to see what commands are available - Gemini CLI lists all tools and custom commands
there
.
In summary,
Gemini CLI can reach out beyond your local filesystem
. Whether it's Google Docs, Sheets, Drive, or other external content, you can pull data in by reference. This pro tip saves you from manual copy-paste and keeps the context flow natural - just refer to the document or dataset you need, and let the AI grab what's needed. It makes Gemini CLI a true
knowledge assistant
for all the information you have access to, not just the files on your disk.
(Note: Accessing private documents of course requires the CLI to have the appropriate permissions. Always ensure any integration respects security and privacy. In corporate settings, setting up such integrations might involve additional auth steps.)
Tip 7: Reference Files and Images with
@
for Explicit Context
Quick use-case:
Instead of describing a file's content or an image verbally, just point Gemini CLI directly to it. Using the
@
syntax, you can attach files, directories, or images into your prompt. This guarantees the AI sees exactly what's in those files as
context
. For example:
Explain this code to me: @./src/main.js
This will include the contents of
src/main.js
in the prompt (up to Gemini's context size limits), so the AI can read it and explain
it
.
This
@
file reference
is one of Gemini CLI's most powerful features for developers. It eliminates ambiguity - you're not asking the model to rely on memory or guesswork about the file, you're literally handing it the file to read. You can use this for source code, text documents, logs, etc. Similarly, you can reference
entire directories
:
Refactor the code in @./utils/ to use async/await.
By appending a path that ends in a slash, Gemini CLI will recursively include files from that
directory
(within reason, respecting ignore files and size limits). This is great for multi-file refactors or analyses, as the AI can consider all relevant modules together.
Even more impressively, you can reference
binary files like images
in prompts. Gemini CLI (using the Gemini model's multimodal capabilities) can understand images. For example:
Describe what you see in this screenshot: @./design/mockup.png
The image will be fed into the model, and the AI might respond with something like "This is a login page with a blue sign-in button and a header image,"
etc
.. You can imagine the uses: reviewing UI mockups, organizing photos (as we'll see in a later tip), or extracting text from images (Gemini can do OCR as well).
A few notes on using
@
references effectively:
File limits:
Gemini 2.5 Pro has a huge context window (up to 1 million
tokens
), so you can include quite large files or many files. However, extremely large files might be truncated. If a file is enormous (say, hundreds of thousands of lines), consider summarizing it or breaking it into parts. Gemini CLI will warn you if a reference is too large or if it skipped something due to size.
Automatic ignoring:
By default, Gemini CLI respects your
.gitignore
and
.geminiignore
files when pulling in directory
context
. So if you
@./
a project root, it will not dump huge ignored folders (like
node_modules
) into the prompt. You can customize ignore patterns with
.geminiignore
similarly to how
.gitignore
works.
Explicit vs implicit context:
Taylor Mullen (the creator of Gemini CLI) emphasizes using
@
for
explicit context injection
rather than relying on the model's memory or summarizing things yourself. It's more precise and ensures the AI isn't hallucinating content. Whenever possible, point the AI to the source of truth (code, config files, documentation) with
@
references. This practice can significantly improve accuracy.
Chaining references:
You can include multiple files in one prompt, like:
Compare @./foo.py and @./bar.py and tell me differences.
The CLI will include both files. Just be mindful of token limits; multiple large files might consume a lot of the context window.
Using
@
is essentially how you
feed knowledge into Gemini CLI on the fly
. It turns the CLI into a multi-modal reader that can handle text and images. As a pro user, get into the habit of leveraging this - it's often faster and more reliable than asking the AI something like "Open the file X and do Y" (which it may or may not do on its own). Instead, you explicitly give it X to work with.
Tip 8: On-the-Fly Tool Creation (Have Gemini Build Helpers)
Quick use-case:
If a task at hand would benefit from a small script or utility, you can ask Gemini CLI to create that tool for you - right within your session. For example, you might say, "Write a Python script to parse all JSON files in this folder and extract the error fields." Gemini can generate the script, which you can then execute via the CLI. In essence, you can
dynamically extend the toolset
as you go.
Gemini CLI is not limited to its pre-existing tools; it can use its coding abilities to fabricate new ones when needed. This often happens implicitly: if you ask for something complex, the AI might propose writing a temporary file (with code) and then running it. As a user, you can also guide this process explicitly:
Creating scripts:
You can prompt Gemini to create a script or program in the language of your choice. It will likely use the
write_file
tool to create the file. For instance:
Generate a Node.js script that reads all '.log' files in the current directory and reports the number of lines in each.
Gemini CLI will draft the code, and with your approval, write it to a file (e.g.
script.js
). You can then run it by either using the
!
shell command (e.g.
!node script.js
) or by asking Gemini CLI to execute it (the AI might automatically use
run_shell_command
to execute the script it just wrote, if it deems it part of the plan).
Temporary tools via MCP:
In advanced scenarios, the AI might even suggest launching an MCP server for some specialized tasks. For example, if your prompt involves some heavy text processing that might be better done in Python, Gemini could generate a simple MCP server in Python and run it. While this is more rare, it demonstrates that the AI can set up a new "agent" on the fly. (One of the slides from the Gemini CLI team humorously referred to "MCP servers for everything, even one called LROwn" - suggesting you can have Gemini run an instance of itself or another model, though that's more of a trick than a practical use!).
The key benefit here is
automation
. Instead of you manually stopping to write a helper script, you can let the AI do it as part of the flow. It's like having an assistant who can create tools on-demand. This is especially useful for data transformation tasks, batch operations, or one-off computations that the built-in tools don't directly provide.
Nuances and safety:
When Gemini CLI writes code for a new tool, you should still review it before running. The
/diff
view (Gemini will show you the file diff before you approve writing it) is your chance to inspect the
code
. Ensure it does what you expect and nothing malicious or destructive (the AI shouldn't produce something harmful unless your prompt explicitly asks, but just like any code from an AI, double-check logic, especially for scripts that delete or modify lots of data).
Example scenario:
Let's say you have a CSV file and you want to filter it in a complex way. You ask Gemini CLI to do it, and it might say: "I will write a Python script to parse the CSV and apply the filter." It then creates
filter_data.py
. After you approve and it runs, you get your result, and you might never need that script again. This ephemeral creation of tools is a pro move - it shows the AI effectively extending its capabilities autonomously.
Pro Tip:
If you find the script useful beyond the immediate context, you can promote it into a permanent tool or command. For instance, if the AI generated a great log-processing script, you might later turn it into a custom slash command (Tip #2) for easy reuse. The combination of Gemini's generative power and the extension hooks means your toolkit can continuously evolve as you use the CLI.
In summary,
don't restrict Gemini to what it comes with
. Treat it as a junior developer who can whip up new programs or even mini-servers to help solve the problem. This approach embodies the agentic philosophy of Gemini CLI - it will figure out what tools it needs, even if it has to code them on the spot.
Tip 9: Use Gemini CLI for System Troubleshooting & Configuration
Quick use-case:
You can run Gemini CLI outside of a code project to help with general system tasks - think of it as an intelligent assistant for your OS. For example, if your shell is misbehaving, you could open Gemini in your home directory and ask: "Fix my
.bashrc
file, it has an error." Gemini can then open and edit your config file for you.
This tip highlights that
Gemini CLI isn't just for coding projects - it's your AI helper for your whole development environment
. Many users have used Gemini to customize their dev setup or fix issues on their machine:
Editing dotfiles:
You can load your shell configuration (
.bashrc
or
.zshrc
) by referencing it (
@~/.bashrc
) and then ask Gemini CLI to optimize or troubleshoot it. For instance, "My
PATH
isn't picking up Go binaries, can you edit my
.bashrc
to fix that?" The AI can insert the correct
export
line. It will show you the diff for confirmation before saving changes.
Diagnosing errors:
If you encounter a cryptic error in your terminal or an application log, you can copy it and feed it to Gemini CLI. It will analyze the error message and often suggest steps to resolve it. This is similar to how one might use StackOverflow or Google, but with the AI directly examining your scenario. For example: "When I run
npm install
, I get an
EACCES
permission error - how do I fix this?" Gemini might detect it's a permissions issue in
node_modules
and guide you to change directory ownership or use a proper node version manager.
Running outside a project:
By default, if you run
gemini
in a directory without a
.gemini
context, it just means no project-specific context is loaded - but you can still use the CLI fully. This is great for ad-hoc tasks like system troubleshooting. You might not have any code files for it to consider, but you can still run shell commands through it or let it fetch web info. Essentially, you're treating Gemini CLI as an AI-powered terminal that can
do
things for you, not just chat.
Workstation customization:
Want to change a setting or install a new tool? You can ask Gemini CLI, "Install Docker on my system" or "Configure my Git to sign commits with GPG." The CLI will attempt to execute the steps. It might fetch instructions from the web (using the search tool) and then run the appropriate shell commands. Of course, always watch what it's doing and approve the commands - but it can save time by automating multi-step setup processes. One real example: a user asked Gemini CLI to "set my macOS Dock preferences to auto-hide and remove the delay," and the AI was able to execute the necessary
defaults write
commands.
Think of this mode as using Gemini CLI as a
smart shell
. In fact, you can combine this with Tip 16 (shell passthrough mode) - sometimes you might drop into
!
shell mode to verify something, then go back to AI mode to have it analyze output.
Caveat:
When doing system-level tasks, be cautious with commands that have widespread impact (like
rm -rf
or system config changes). Gemini CLI will usually ask for confirmation, and it doesn't run anything without you seeing it. But as a power user, you should have a sense of what changes are being made. If unsure, ask Gemini to explain a command before running (e.g., "Explain what
defaults write com.apple.dock autohide-delay -float 0
does" - it will gladly explain rather than just execute if you prompt it in that way).
Troubleshooting bonus:
Another neat use is using Gemini CLI to parse logs or config files looking for issues. For instance, "Scan this Apache config for mistakes" (with
@httpd.conf
), or "Look through syslog for errors around 2 PM yesterday" (with an
@/var/log/syslog
if accessible). It's like having a co-administrator. It can even suggest likely causes for crashes or propose fixes for common error patterns.
In summary,
don't hesitate to fire up Gemini CLI as your assistant for environment issues
. It's there to accelerate all your workflows - not just writing code, but maintaining the system that you write code on. Many users report that customizing their dev environment with Gemini's help feels like having a tech buddy always on call to handle the tedious or complex setup steps.
Tip 10: YOLO Mode - Auto-Approve Tool Actions (Use with Caution)
Quick use-case:
If you're feeling confident (or adventurous), you can let Gemini CLI run tool actions without asking for your confirmation each time. This is
YOLO mode
(You Only Live Once). It's enabled by the
--yolo
flag or by pressing
Ctrl+Y
during a
session
. In YOLO mode, as soon as the AI decides on a tool (like running a shell command or writing to a file), it executes it immediately, without that "Approve? (y/n)" prompt.
Why use YOLO mode?
Primarily for speed and convenience
when you trust the AI's actions
. Experienced users might toggle YOLO on if they're doing a lot of repetitive safe operations. For example, if you ask Gemini to generate 10 different files one after another, approving each can slow down the flow; YOLO mode would just let them all be written automatically. Another scenario is using Gemini CLI in a completely automated script or CI pipeline - you might run it headless with
--yolo
so it doesn't pause for confirmation.
To start in YOLO mode from the get-go, launch the CLI with:
Or the short form
gemini -y
. You'll see some indication in the CLI (like a different prompt or a notice) that auto-approve is
on
. During an interactive session, you can toggle it by pressing
Ctrl+Y
at any
time
- the CLI will usually display a message like "YOLO mode enabled (all actions auto-approved)" in the footer.
Big warning:
YOLO mode is powerful but
risky
. The Gemini team themselves labels it for "daring users" - meaning you should be aware that the AI could potentially execute a dangerous command without asking. In normal mode, if the AI decided to run
rm -rf /
(worst-case scenario), you'd obviously decline. In YOLO mode, that command would run immediately (and likely ruin your day). While such extreme mistakes are unlikely (the AI's system prompt includes safety guidelines), the whole point of confirmations is to catch any unwanted action. YOLO removes that safety net.
Best practices for YOLO:
If you want some of the convenience without full risk, consider
allow-listing
specific commands. For example, you can configure in settings that certain tools or command patterns don't require confirmation (like allowing all
git
commands, or read-only actions). In fact, Gemini CLI supports a config for skipping confirmation on specific commands: e.g., you can set something like
"tools.shell.autoApprove": ["git ", "npm test"]
to always run
those
. This way, you might not need YOLO mode globally - you selectively YOLO only safe commands. Another approach: run Gemini in a sandbox or container when using YOLO, so even if it does something wild, your system is insulated (Gemini has a
--sandbox
flag to run tools in a Docker
container
).
Many advanced users toggle YOLO on and off frequently - turning it on when doing a string of minor file edits or queries, and off when about to do something critical. You can do the same, using the keyboard shortcut as a quick toggle.
In summary,
YOLO mode eliminates friction at the cost of oversight
. It's a pro feature to use sparingly and wisely. It truly demonstrates trust in the AI (or recklessness!). If you're new to Gemini CLI, you should probably avoid YOLO until you clearly understand the patterns of what it tends to do. If you do use it, double down on having version control or backups - just in case.
(If it's any consolation, you're not alone - many in the community joke about "I YOLO'ed and Gemini did something crazy." So use it, but... well, you only live once.)
Tip 11: Headless & Scripting Mode (Run Gemini CLI in the Background)
Quick use-case:
You can use Gemini CLI in scripts or automation by running it in
headless mode
. This means you provide a prompt (or even a full conversation) via command-line arguments or environment variables, and Gemini CLI produces an output and exits. It's great for integrating with other tools or triggering AI tasks on a schedule.
For instance, to get a one-off answer without opening the REPL, you've seen you can use
gemini -p "...prompt..."
. This is already headless usage: it prints the model's response and returns to the
shell
. But there's more you can do:
System prompt override:
If you want to run Gemini CLI with a custom system persona or instruction set (different from the default), you can use the environment variable
GEMINI_SYSTEM_MD
. By setting this, you tell Gemini CLI to ignore its built-in system prompt and use your provided file
instead
. For example:
export GEMINI_SYSTEM_MD="/path/to/custom_system.md"
gemini -p "Perform task X with high caution"
This would load your
custom_system.md
as the system prompt (the "role" and rules the AI follows) before executing the
prompt
. Alternatively, if you set
GEMINI_SYSTEM_MD=true
, the CLI will look for a file named
system.md
in the current project's
.gemini
directory
. This feature is very advanced - it essentially allows you to
replace the built-in brain
of the CLI with your own instructions, which some users do for specialized workflows (like simulating a specific persona or enforcing ultra-strict policies). Use it carefully, as replacing the core prompt can affect tool usage (the core prompt contains important directions for how the AI selects and uses
tools
).
Direct prompt via CLI:
Aside from
-p
, there's also
-i
(interactive prompt) which starts a session with an initial prompt, and then keeps it open. For example:
gemini -i "Hello, let's debug something"
will open the REPL and already have said hello to the model. This is useful if you want the first question to be asked immediately when starting.
Scripting with shell pipes:
You can pipe not just text but also files or command outputs into Gemini. For example:
gemini -p "Summarize this log:" < big_log.txt
will feed the content of
big_log.txt
into the prompt (after the phrase "Summarize this log:"). Or you might do
some_command | gemini -p "Given the above output, what went wrong?"
. This technique allows you to compose Unix tools with AI analysis. It's headless in the sense that it's a single-pass operation.
Running in CI/CD:
You could incorporate Gemini CLI into build processes. For instance, a CI pipeline might run a test and then use Gemini CLI to automatically analyze failing test output and post a comment. Using the
-p
flag and environment auth, this can be scripted. (Of course, ensure the environment has the API key or auth needed.)
One more headless trick:
the
--format=json
flag
(or config setting). Gemini CLI can output responses in JSON format instead of the human-readable text if you configure
it
. This is useful for programmatic consumption - your script can parse the JSON to get the answer or any tool actions details.
Why headless mode matters:
It transforms Gemini CLI from an interactive assistant into a
backend service
or utility that other programs can call. You could schedule a cronjob that runs a Gemini CLI prompt nightly (imagine generating a report or cleaning up something with AI logic). You could wire up a button in an IDE that triggers a headless Gemini run for a specific task.
Example:
Let's say you want a daily summary of a news website. You could have a script:
gemini -p "Web-fetch \"https://news.site/top-stories\" and extract the headlines, then write them to headlines.txt"
With
--yolo
perhaps, so it won't ask confirmation to write the file. This would use the web fetch tool to get the page and the file write tool to save the headlines. All automatically, no human in the loop. The possibilities are endless once you treat Gemini CLI as a scriptable component.
In summary,
Headless Mode
enables automation. It's the bridge between Gemini CLI and other systems. Mastering it means you can scale up your AI usage - not just when you're typing in the terminal, but even when you aren't around, your AI agent can do work for you.
(Tip: For truly long-running non-interactive tasks, you might also look into Gemini CLI's "Plan" mode or how it can generate multi-step plans without intervention. However, those are advanced topics beyond this scope. In most cases, a well-crafted single prompt via headless mode can achieve a lot.)
Tip 12: Save and Resume Chat Sessions
Quick use-case:
If you've been debugging an issue with Gemini CLI for an hour and need to stop, you don't have to lose the conversation context. Use
/chat save <name>
to save the session. Later (even after restarting the CLI), you can use
/chat resume <name>
to pick up where you left
off
. This way, long-running conversations can be paused and continued seamlessly.
Gemini CLI essentially has a built-in chat session manager. The commands to know are:
/chat save <tag>
- Saves the current conversation state under a tag/name you
provide
. The tag is like a filename or key for that session. Save often if you want, it will overwrite the tag if it exists. (Using a descriptive name is helpful - e.g.,
chat save fix-docker-issue
.)
/chat list
- Lists all your saved sessions (the tags you've
used
. This helps you remember what you named previous saves.
/chat resume <tag>
- Resumes the session with that tag, restoring the entire conversation context and history to how it was when
saved
. It's like you never left. You can then continue chatting from that point.
/chat share
- (saves to file) This is useful as you can share the entire chat with someone else who can continue the session. Almost collaboration-like.
Under the hood, these sessions are stored likely in
~/.gemini/chats/
or a similar location. They include the conversation messages and any relevant state. This feature is super useful for cases such as:
Long debugging sessions:
Sometimes debugging with an AI can be a long back-and-forth. If you can't solve it in one go, save it and come back later (maybe with a fresh mind). The AI will still "remember" everything from before, because the whole context is reloaded.
Multi-day tasks:
If you're using Gemini CLI as an assistant for a project, you might have one chat session for "Refactor module X" that spans multiple days. You can resume that specific chat each day so the context doesn't reset daily. Meanwhile, you might have another session for "Write documentation" saved separately. Switching contexts is just a matter of saving one and resuming the other.
Team hand-off:
This is more experimental, but in theory, you could share the content of a saved chat with a colleague (the saved files are likely portable). If they put it in their
.gemini
directory and resume, they could see the same context. The
practical simpler approach
for collaboration is just copying the relevant Q&A from the log and using a shared
GEMINI.md
or prompt, but it's interesting to note that the session data is yours to keep.
Usage example:
(Session saved as "api-upgrade")
(Later, reopen CLI)
$ gemini
gemini> /chat list
(Shows: api-upgrade)
gemini> /chat resume api-upgrade
Now the model greets you with the last exchange's state ready. You can confirm by scrolling up that all your previous messages are present.
Pro Tip:
Use meaningful tags when saving
chats
. Instead of
/chat save session1
, give it a name related to the topic (e.g.
/chat save memory-leak-bug
). This will help you find the right one later via
/chat list
. There is no strict limit announced on how many sessions you can save, but cleaning up old ones occasionally might be wise just for organization.
This feature turns Gemini CLI into a persistent advisor. You don't lose knowledge gained in a conversation; you can always pause and resume. It's a differentiator compared to some other AI interfaces that forget context when closed. For power users, it means
you can maintain parallel threads of work
with the AI. Just like you'd have multiple terminal tabs for different tasks, you can have multiple chat sessions saved and resume the one you need at any given time.
Tip 13: Multi-Directory Workspace - One Gemini, Many Folders
Quick use-case:
Do you have a project split across multiple repositories or directories? You can launch Gemini CLI with access to
all of them
at once, so it sees a unified workspace. For example, if your frontend and backend are separate folders, you can include both so that Gemini can edit or reference files in both.
There are two ways to use
multi-directory mode
:
Launch flag:
Use the
--include-directories
(or
-I
) flag when starting Gemini CLI. For example:
This assumes you run the command from, say, a
scripts
directory and want to include two sibling folders. You provide a colon-separated list of paths. Gemini CLI will then treat all those directories as part of one big workspace.
Persistent setting:
In your
settings.json
, you can define
"includeDirectories": ["path1", "path2", [...]](https://www.philschmid.de/gemini-cli-cheatsheet#:~:text=,61AFEF%22%2C%20%22AccentPurple)
. This is useful if you always want certain common directories loaded (e.g., a shared library folder that multiple projects use). The paths can be relative or absolute. Environment variables in the paths (like
~/common-utils
) are
allowed
.
When multi-dir mode is active, the CLI's context and tools consider files across all included locations. The
> /directory show
command will list which directories are in the current
workspace
. You can also dynamically add directories during a session with
/directory add [<path>](https://medium.com/@ferreradaniel/gemini-cli-free-ai-tool-upgrade-5-new-features-you-need-right-now-04cfefac5e93#:~:text=How%20to%20add%20multiple%20directories,step)
- it will then load that on the fly (potentially scanning it for context like it does on startup).
Why use multi-directory mode?
In microservice architectures or modular codebases, it's common that one piece of code lives in one repo and another piece in a different repo. If you only ran Gemini in one, it wouldn't "see" the others. By combining them, you enable cross-project reasoning. For example, you could ask, "Update the API client in the frontend to match the backend's new API endpoints" - Gemini can open the backend folder to see the API definitions and simultaneously open the frontend code to modify it accordingly. Without multi-dir, you'd have to do one side at a time and manually carry info over.
Example:
Let's say you have
client/
and
server/
. You start:
cd client
gemini --include-directories "../server"
Now at the
gemini>
prompt, if you do
> !ls
, you'll see it can list files in both
client
and
server
(it might show them as separate paths). You could do:
Open server/routes/api.py and client/src/api.js side by side to compare functionnames.
The AI will have access to both files. Or you might say:
The API changed: the endpoint "/users/create" is now "/users/register". Update both backend and frontend accordingly.
It can simultaneously create a patch in the backend route and adjust the frontend fetch call.
Under the hood, Gemini merges the file index of those directories. There might be some performance considerations if each directory is huge, but generally it handles multiple small-medium projects fine. The cheat sheet notes that this effectively creates one workspace with multiple
roots
.
Tip within a tip:
Even if you don't use multi-dir all the time, know that you can still reference files across the filesystem by absolute path in prompts (
@/path/to/file
). However, without multi-dir, Gemini might not have permission to edit those or know to load context from them proactively. Multi-dir formally includes them in scope so it's aware of all files for tasks like search or code generation across the whole set.
Remove directories:
If needed,
/directory remove <path>
(or a similar command) can drop a directory from the workspace. This is less common, but maybe if you included something accidentally, you can remove it.
In summary,
multi-directory mode unifies your context
. It's a must-have for polyrepo projects or any situation where code is split up. It makes Gemini CLI act more like an IDE that has your entire solution open. As a pro user, this means no part of your project is out of the AI's reach.
Tip 14: Organize and Clean Up Your Files with AI Assistance
Quick use-case:
Tired of a messy
Downloads
folder or disorganized project assets? You can enlist Gemini CLI to act as a smart organizer. By providing it an overview of a directory, it can classify files and even move them into subfolders (with your approval). For instance, "Clean up my
Downloads
: move images to an
Images
folder, PDFs to
Documents
, and delete temporary files."
Because Gemini CLI can read file names, sizes, and even peek into file contents, it can make informed decisions about file
organization
. One community-created tool dubbed
"Janitor AI"
showcases this: it runs via Gemini CLI to categorize files as important vs junk, and groups them
accordingly
. The process involved scanning the directory, using Gemini's reasoning on filenames and metadata (and content if needed), then moving files into categories. Notably, it didn't automatically delete junk - rather, it moved them to a
Trash
folder for
review
.
Here's how you might replicate such a workflow with Gemini CLI manually:
Survey the directory:
Use a prompt to have Gemini list and categorize. For example:
List all files in the current directory and categorize them as "images", "videos", "documents", "archives", or "others".
Gemini might use
!ls
or similar to get the file list, then analyze the names/extensions to produce categories.
Plan the organization:
Ask Gemini how it would like to reorganize. For example:
Propose a new folder structure for these files. I want to separate by type (Images, Videos, Documents, etc.). Also identify any files that seem like duplicates or unnecessary.
The AI might respond with a plan: e.g.,
"Create folders:
Images/
,
Videos/
,
Documents/
,
Archives/
. Move
X.png
,
Y.jpg
to
Images/
; move
A.mp4
to
Videos/
; etc. The file
temp.txt
looks unnecessary (maybe a temp file)."
Execute moves with confirmation:
You can then instruct it to carry out the plan. It may use shell commands like
mv
for each file. Since this modifies your filesystem, you'll get confirmation prompts for each (unless you YOLO it). Carefully approve the moves. After completion, your directory will be neatly organized as suggested.
Throughout, Gemini's natural language understanding is key. It can reason, for instance, that
IMG_001.png
is an image or that
presentation.pdf
is a document, even if not explicitly stated. It can even open an image (using its vision capability) to see what's in it - e.g., differentiating between a screenshot vs a photo vs an icon - and name or sort it
accordingly
.
Renaming files by content:
A particularly magical use is having Gemini rename files to be more descriptive. The Dev Community article "7 Insane Gemini CLI Tips" describes how Gemini can
scan images and automatically rename them
based on their
content
. For example, a file named
IMG_1234.jpg
might be renamed to
login_screen.jpg
if the AI sees it's a screenshot of a login
screen
. To do this, you could prompt:
For each .png image here, look at its content and rename it to something descriptive.
Gemini will open each image (via vision tool), get a description, then propose a
mv IMG_1234.png login_screen.png
action
. This can dramatically improve the organization of assets, especially in design or photo folders.
Two-pass approach:
The Janitor AI discussion noted a two-step process: first broad categorization (important vs junk vs other), then refining
groups
. You can emulate this: first separate files that likely can be deleted (maybe large installer
.dmg
files or duplicates) from those to keep. Then focus on organizing the keepers. Always double-check what the AI flags as junk; its guess might not always be right, so manual oversight is needed.
Safety tip:
When letting the AI loose on file moves or deletions, have backups or at least be ready to undo (with
/restore
or your own backup). It's wise to do a dry-run: ask Gemini to print the commands it
would
run to organize, without executing them, so you can review. For instance: "List the
mv
and
mkdir
commands needed for this plan, but don't execute them yet." Once you review the list, you can either copy-paste execute them, or instruct Gemini to proceed.
This is a prime example of using Gemini CLI for "non-obvious" tasks - it's not just writing code, it's doing
system housekeeping with AI smarts
. It can save time and bring a bit of order to chaos. After all, as developers we accumulate clutter (logs, old scripts, downloads), and an AI janitor can be quite handy.
Tip 15: Compress Long Conversations to Stay Within Context
Quick use-case:
If you've been chatting with Gemini CLI for a long time, you might hit the model's context length limit or just find the session getting unwieldy. Use the
/compress
command to summarize the conversation so far, replacing the full history with a concise
summary
. This frees up space for more discussion without starting from scratch.
Large language models have a fixed context window (Gemini 2.5 Pro's is very large, but not infinite). If you exceed it, the model may start forgetting earlier messages or lose coherence. The
/compress
feature is essentially an
AI-generated tl;dr
of your session that keeps important points.
How it works:
When you type
/compress
, Gemini CLI will take the entire conversation (except system context) and produce a summary. It then replaces the chat history with that summary as a single system or assistant message, preserving essential details but dropping minute-by-minute dialogue. It will indicate that compression happened. For example, after
/compress
, you might see something like:
--- Conversation compressed ---
Summary of discussion: The user and assistant have been debugging a memory leak in an application. Key points: The issue is likely in
DataProcessor.js
, where objects aren't being freed. The assistant suggested adding logging and identified a possible infinite loop. The user is about to test a fix.
--- End of summary ---
From that point on, the model only has that summary (plus new messages) as context for what happened before. This usually is enough if the summary captured the salient info.
When to compress:
Ideally before you
hit
the limit. If you notice the session is getting lengthy (several hundred turns or a lot of code in context), compress proactively. The cheat sheet mentions an automatic compression setting (e.g., compress when context exceeds 60% of
max
). If you enable that, Gemini might auto-compress and let you know. Otherwise, manual
/compress
is in your toolkit.
After compressing:
You can continue the conversation normally. If needed, you can compress multiple times in a very long session. Each time, you lose some granularity, so don't compress too frequently for no reason - you might end up with an overly brief remembrance of a complex discussion. But generally the model's own summarization is pretty good at keeping the key facts (and you can always restate anything critical yourself).
Context window example:
Let's illustrate. Suppose you fed in a large codebase by referencing many files and had a 1M token context (the max). If you then want to shift to a different part of the project, rather than starting a new session (losing all that understanding), you could compress. The summary will condense the knowledge gleaned from the code (like "We loaded modules A, B, C. A has these functions... B interacts with C in these ways..."). Now you can proceed to ask about new things with that knowledge retained abstractly.
Memory vs Compression:
Note that compression doesn't save to long-term memory, it's local to the conversation. If you have facts you
never
want lost, consider Tip 4 (adding to
/memory
) - because memory entries will survive compression (they'll just be reinserted anyway since they are in
GEMINI.md
context). Compression is more about ephemeral chat content.
A minor caution:
after compression, the AI's style might slightly change because it's effectively seeing a "fresh" conversation with a summary. It might reintroduce itself or change tone. You can instruct it like "Continue from here... (we compressed)" to smooth it out. In practice, it often continues fine.
To summarize (pun intended),
use
/compress
as your session grows long
to maintain performance and relevance. It helps Gemini CLI focus on the bigger picture instead of every detail of the conversation's history. This way, you can have marathon debugging sessions or extensive design discussions without running out of the "mental paper" the AI is writing on.
Tip 16: Passthrough Shell Commands with
!
(Talk to Your Terminal)
Quick use-case:
At any point in a Gemini CLI session, you can run actual shell commands by prefixing them with
!
. For example, if you want to check the git status, just type
!git status
and it will execute in your
terminal
. This saves you from switching windows or context - you're still in the Gemini CLI, but you're essentially telling it "let me run this command real quick."
This tip is about
Shell Mode
in Gemini CLI. There are two ways to use it:
Single command:
Just put
!
at the start of your prompt, followed by any command and arguments. This will execute that command in the current working directory and display the output
in-line
. For example:
will list the files in the
src
directory, outputting something like you'd see in a normal terminal. After the output, the Gemini prompt returns so you can continue chatting or issue more commands.
Persistent shell mode:
If you enter
!
alone and hit Enter, Gemini CLI switches into a sub-mode where you get a shell prompt (often it looks like
shell>
or
similar
. Now you can type multiple shell commands interactively. It's basically a mini-shell within the CLI. You exit this mode by typing
!
on an empty line again (or
exit
). For instance:
After the final
!
, you're back to the normal Gemini prompt.
Why is this useful?
Because development is a mix of actions and inquiries. You might be discussing something with the AI and realize you need to compile the code or run tests to see something. Instead of leaving the conversation, you can quickly do it and feed the result back into the chat. In fact, Gemini CLI often does this for you as part of its tool usage (it might automatically run
!pytest
when you ask to fix tests, for
example
). But as the user, you have full control to do it manually too.
Examples:
After Gemini suggests a fix in code, you can do
!npm run build
to see if it compiles, then copy any errors and ask Gemini to help with those.
If you want to open a file in
vim
or
nano
, you could even launch it via
!nano filename
(though note that since Gemini CLI has its own interface, using an interactive editor inside it might be a bit awkward - better to use the built-in editor integration or copy to your editor).
You can use shell commands to gather info for the AI: e.g.,
!grep TODO -R .
to find all TODOs in the project, then you might ask Gemini to help address those TODOs.
Or simply use it for environment tasks:
!pip install some-package
if needed, etc., without leaving the CLI.
Seamless interplay:
One cool aspect is how the conversation can refer to outputs. For example, you could do
!curl http://example.com
to fetch some data, see the output, then immediately say to Gemini, "Format the above output as JSON" - since the output was printed in the chat, the AI has it in context to work with (provided it's not too large).
Terminal as a default shell:
If you find yourself always prefacing commands with
!
, you can actually make the shell mode persistent by default. One way is launching Gemini CLI with a specific tool mode (there's a concept of default tool). But easier: just drop into shell mode (
!
with nothing) at session start if you plan to run a lot of manual commands and only occasionally talk to AI. Then you can exit shell mode whenever you want to ask a question. It's almost like turning Gemini CLI into your normal terminal that happens to have an AI readily available.
Integration with AI planning:
Sometimes Gemini CLI itself will propose to run a shell command. If you approve, it effectively does the same as
!command
. Understanding that, you know you can always intervene. If Gemini is stuck or you want to try something, you don't have to wait for it to suggest - you can just do it and then continue.
In summary, the
!
passthrough
means
you don't have to leave Gemini CLI for shell tasks
. It collapses the boundary between chatting with the AI and executing commands on your system. As a pro user, this is fantastic for efficiency - your AI and your terminal become one continuous environment.
Tip 17: Treat Every CLI Tool as a Potential Gemini Tool
Quick use-case:
Realize that Gemini CLI can leverage
any
command-line tool installed on your system as part of its problem-solving. The AI has access to the shell, so if you have
cURL
,
ImageMagick
,
git
,
Docker
, or any other tool, Gemini can invoke it when appropriate. In other words,
your entire
$PATH
is the AI's toolkit
. This greatly expands what it can do - far beyond its built-in tools.
For example, say you ask: "Convert all PNG images in this folder to WebP format." If you have ImageMagick's
convert
utility installed, Gemini CLI might plan something like: use a shell loop with
convert
command for each
file
. Indeed, one of the earlier examples from a blog showed exactly this, where the user prompted to batch-convert images, and Gemini executed a shell one-liner with the
convert
tool
.
Another scenario: "Deploy my app to Docker." If
Docker CLI
is present, the AI could call
docker build
and
docker run
steps as needed. Or "Use FFmpeg to extract audio from
video.mp4
" - it can construct the
ffmpeg
command.
This tip is about mindset:
Gemini isn't limited to what's coded into it
(which is already extensive). It can figure out how to use other programs available to achieve a
goal
. It knows common syntax and can read help texts if needed (it could call
--help
on a tool). The only limitation is safety: by default, it will ask confirmation for any
run_shell_command
it comes up with. But as you become comfortable, you might allow certain benign commands automatically (see YOLO or allowed-tools config).
Be mindful of the environment:
"With great power comes great responsibility." Since every shell tool is fair game, you should ensure that your
$PATH
doesn't include anything you wouldn't want the AI to run inadvertently. This is where Tip 19 (custom PATH) comes in - some users create a restricted
$PATH
for Gemini, so it can't, say, directly call system destructive commands or maybe not call
gemini
recursively (to avoid loops). The point is, by default if
gcc
or
terraform
or anything is in
$PATH
, Gemini could invoke it. It doesn't mean it will randomly do so - only if the task calls for it - but it's possible.
Train of thought example:
Imagine you ask Gemini CLI: "Set up a basic HTTP server that serves the current directory." The AI might think: "I can use Python's built-in server for this." It then issues
!python3 -m http.server 8000
. Now it just used a system tool (Python) to launch a server. That's an innocuous example. Another: "Check the memory usage on this Linux system." The AI might use the
free -h
command or read from
/proc/meminfo
. It's effectively doing what a sysadmin would do, by using available commands.
All tools are extensions of the AI:
This is somewhat futuristic, but consider that any command-line program can be seen as a "function" the AI can call to extend its capability. Need to solve a math problem? It could call
bc
(calculator). Need to manipulate an image? It could call an image processing tool. Need to query a database? If the CLI client is installed and credentials are there, it can use it. The possibilities are expansive. In other AI agent frameworks, this is known as tool use, and Gemini CLI is designed with a lot of trust in its agent to decide the right
tool
.
When it goes wrong:
The flip side is if the AI misunderstands a tool or has a hallucination about one. It might try to call a command that doesn't exist, or use wrong flags, resulting in errors. This isn't a big deal - you'll see the error and can correct or clarify. In fact, the system prompt of Gemini CLI likely guides it to first do a dry-run (just propose the command) rather than executing blindly. So you often get a chance to catch these. Over time, the developers are improving the tool selection logic to reduce these missteps.
The main takeaway is to
think of Gemini CLI as having a very large Swiss Army knife
- not just the built-in blades, but every tool in your OS. You don't have to instruct it on how to use them if it's something standard; usually it knows or can find out. This significantly amplifies what you can accomplish. It's like having a junior dev or devops engineer who knows how to run pretty much any program you have installed.
As a pro user, you can even install additional CLI tools specifically to give Gemini more powers. For example, if you install a CLI for a cloud service (AWS CLI, GCloud CLI, etc.), in theory Gemini can utilize it to manage cloud resources if prompted to. Always ensure you understand and trust the commands run, especially with powerful tools (you wouldn't want it spinning up huge cloud instances accidentally). But used wisely, this concept -
everything is a Gemini tool
- is what makes it
exponentially
more capable as you integrate it into your environment.
Tip 18: Utilize Multimodal AI - Let Gemini See Images and More
Quick use-case:
Gemini CLI isn't limited to text - it's multimodal. This means it can analyze images, diagrams, or even PDFs if given. Use this to your advantage. For instance, you could say "Here's a screenshot of an error dialog,
@./error.png
- help me troubleshoot this." The AI will "see" the image and respond accordingly.
One of the standout features of Google's Gemini model (and its precursor PaLM2 in Codey form) is image understanding. In Gemini CLI, if you reference an image with
@
, the model receives the image data. It can output descriptions, classifications, or reason about the image's content. We already discussed renaming images by content (Tip 14) and describing screenshots (Tip 7). But let's consider other creative uses:
UI/UX feedback:
If you're a developer working with designers, you can drop a UI image and ask Gemini for feedback or to generate code. "Look at this UI mockup
@mockup.png
and produce a React component structure for it." It could identify elements in the image (header, buttons, etc.) and outline code.
Organizing images:
Beyond renaming, you might have a folder of mixed images and want to sort by content. "Sort the images in
./photos/
into subfolders by theme (e.g., sunsets, mountains, people)." The AI can look at each photo and categorize it (this is similar to what some photo apps do with AI - now you can do it with your own script via Gemini).
OCR and data extraction:
If you have a screenshot of error text or a photo of a document, Gemini can often read the text from it. For example, "Extract the text from
invoice.png
and put it into a structured format." As shown in a Google Cloud blog example, Gemini CLI can process a set of invoice images and output a table of their
info
. It basically did OCR + understanding to get invoice numbers, dates, amounts from pictures of invoices. That's an advanced use-case but entirely possible with the multimodal model under the hood.
Understanding graphs or charts:
If you have a graph screenshot, you could ask "Explain this chart's key insights
@chart.png
." It might interpret the axes and trends. Accuracy can vary, but it's a nifty try.
To make this practical: when you
@image.png
, ensure the image isn't too huge (though the model can handle reasonably large images). The CLI will likely encode it and send it to the model. The response might include descriptions or further actions. You can mix text and image references in one prompt too.
Non-image modalities:
The CLI and model potentially can handle PDFs and audio too, by converting them via tools. For example, if you
@report.pdf
, Gemini CLI might use a PDF-to-text tool under the hood to extract text and then summarize. If you
@audio.mp3
and ask for a transcript, it might use an audio-to-text tool (like a speech recognition function). The cheat sheet suggests referencing PDFs, audio, video files is
supported
, presumably by invoking appropriate internal tools or APIs. So, "transcribe this interview audio:
@interview.wav
" could actually work (if not now, likely soon, since underlying Google APIs for speech-to-text could be plugged in).
Rich outputs:
Multimodal also means the AI can return images in responses if integrated (though in CLI it usually won't
display
them directly, but it could save an image file or output ASCII art, etc.). The MCP capability mentioned that tools can return
images
. For instance, an AI drawing tool could generate an image and Gemini CLI could present it (maybe by opening it or giving a link).
Important:
The CLI itself is text-based, so you won't
see
the image in the terminal (unless it's capable of ASCII previews). You'll just get the analysis. So this is mostly about reading images, not displaying them. If you're in VS Code integration, it might show images in the chat view.
In summary,
don't forget the "I" in GUI when using Gemini CLI
- it can handle the visual just as well as the textual in many cases. This opens up workflows like visual debugging, design help, data extraction from screenshots, etc., all under the same tool. It's a differentiator that some other CLI tools may not have yet. And as models improve, this multimodal support will only get more powerful, so it's a future-proof skill to exploit.
Tip 19: Customize the
$PATH
(and Tool Availability) for Stability
Quick use-case:
If you ever find Gemini CLI getting confused or invoking the wrong programs, consider running it with a tailored
$PATH
. By limiting or ordering the available executables, you can prevent the AI from, say, calling a similarly named script that you didn't intend. Essentially, you sandbox its tool access to known-good tools.
For most users, this isn't an issue, but for pro users with lots of custom scripts or multiple versions of tools, it can be helpful. One reason mentioned by the developers is avoiding infinite loops or weird
behavior
. For example, if
gemini
itself is in
$PATH
, an AI gone awry might recursively call
gemini
from within Gemini (a strange scenario, but theoretically possible). Or perhaps you have a command named
test
that conflicts with something - the AI might call the wrong one.
How to set PATH for Gemini:
Easiest is inline on launch:
PATH=/usr/bin:/usr/local/bin gemini
This runs Gemini CLI with a restricted
$PATH
of just those directories. You might exclude directories where experimental or dangerous scripts lie. Alternatively, create a small shell script wrapper that purges or adjusts
$PATH
then exec's
gemini
.
Another approach is using environment or config to explicitly disable certain tools. For instance, if you absolutely never want the AI to use
rm
or some destructive tool, you could technically create an alias or dummy
rm
in a safe
$PATH
that does nothing (though this could interfere with normal operations, so maybe not that one). A better method is the
exclude list
in settings. In an extension or
settings.json
, you can exclude tool
names
. E.g.,
"excludeTools": ["run_shell_command"]
This extreme example would stop
all
shell commands from running (making Gemini effectively read-only). More granular, there was mention of skipping confirmation for some; similarly you might configure something like:
"tools": {
"exclude": ["apt-get", "shutdown"]
}
(This syntax is illustrative; consult docs for exact usage.)
The principle is, by controlling the environment, you reduce risk of the AI doing something dumb with a tool it shouldn't. It's akin to child-proofing the house.
Prevent infinite loops:
One user scenario was a loop where Gemini kept reading its own output or re-reading files
repeatedly
. Custom
$PATH
can't directly fix logic loops, but one cause could be if the AI calls a command that triggers itself. Ensuring it can't accidentally spawn another AI instance (like calling
bard
or
gemini
command, if it thought to do so) is good. Removing those from
$PATH
(or renaming them for that session) helps.
Isolation via sandbox:
Another alternative to messing with
$PATH
is using
--sandbox
mode (which uses Docker or Podman to run tools in an isolated
environment
). In that case, the AI's actions are contained and have only the tools that sandbox image provides. You could supply a Docker image with a curated set of tools. This is heavy-handed but very safe.
Custom PATH for specific tasks:
You might have different
$PATH
setups for different projects. For example, in one project you want it to use a specific version of Node or a local toolchain. Launching
gemini
with the
$PATH
that points to those versions will ensure the AI uses the right one. Essentially, treat Gemini CLI like any user - it uses whatever environment you give it. So if you need it to pick
gcc-10
vs
gcc-12
, adjust
$PATH
or
CC
env var accordingly.
In summary:
Guard rails.
As a power user, you have the ability to fine-tune the operating conditions of the AI. If you ever find a pattern of undesirable behavior tied to tool usage, tweaking
$PATH
is a quick remedy. For everyday use, you likely won't need this, but it's a pro tip to keep in mind if you integrate Gemini CLI into automation or CI: give it a controlled environment. That way, you know exactly what it can and cannot do, which increases reliability.
Tip 20: Track and reduce token spend with token caching and stats
If you run long chats or repeatedly attach the same big files, you can cut cost and latency by turning on token caching and monitoring usage. With an API key or Vertex AI auth, Gemini CLI automatically reuses previously sent system instructions and context, so follow‑up requests are cheaper. You can see the savings live in the CLI.
How to use it
Use an auth mode that enables caching. Token caching is available when you authenticate with a Gemini API key or Vertex AI. It is not available with OAuth login today.
Google Gemini
Inspect your usage and cache hits. Run the
stats
command during a session. It shows total tokens and a
cached
field when caching is active.
The command's description and cached reporting behavior are documented in the commands reference and FAQ.
Google Gemini+1
Capture metrics in scripts. When running headless, output JSON and parse the
stats
block, which includes
tokens.cached
for each model:
gemini -p "Summarize README" --output-format json
The headless guide documents the JSON schema with cached token counts.
Google Gemini
Save a session summary to file: For CI or budget tracking, write a JSON session summary to disk.
With API key or Vertex auth, the CLI automatically reuses previously sent context so later turns send fewer tokens. Keeping
GEMINI.md
and large file references stable across turns increases cache hits; you'll see that reflected in stats as cached tokens.
Tip 21: Use
/copy
for Quick Clipboard Copy
Quick use-case:
Instantly copy the latest answer or code snippet from Gemini CLI to your system clipboard, without any extraneous formatting or line
numbers
. This is perfect for quickly pasting AI-generated code into your editor or sharing a result with a teammate.
When Gemini CLI provides an answer (especially a multi-line code block), you often want to reuse it elsewhere. The
/copy
slash command makes this effortless by copying
the last output produced by the CLI
directly to your
clipboard
. Unlike manual selection (which can grab line numbers or prompt text),
/copy
grabs only the raw response content. For example, if Gemini just generated a 50-line Python script, simply typing
/copy
will put that entire script into your clipboard, ready to paste - no need to scroll and select text. Under the hood, Gemini CLI uses the appropriate clipboard utility for your platform (e.g.
pbcopy
on macOS,
clip
on
Windows
. Once you run the command, you'll typically see a confirmation message, and then you can paste the copied text wherever you need it.
How it works:
The
/copy
command requires that your system has a clipboard tool
available
. On macOS and Windows, the required tools (
pbcopy
and
clip
respectively) are usually pre-installed. On Linux, you may need to install
xclip
or
xsel
for
/copy
to
function
. After ensuring that, you can use
/copy
anytime after Gemini CLI prints an answer. It will capture the
entire
last response (even if it's long) and omit any internal numbering or formatting the CLI may show on-screen. This saves you from dealing with unwanted artifacts when transferring the content. It's a small feature, but a huge time-saver when you're iterating on code or compiling a report generated by the AI.
Pro Tip:
If you find the
/copy
command isn't working, double-check that your clipboard utilities are installed and accessible. For instance, Ubuntu users should run
sudo apt install xclip
to enable clipboard
copying
. Once set up,
/copy
lets you share Gemini's outputs with zero friction - copy, paste, and you're done.
Tip 22: Master
Ctrl+C
for Shell Mode and Exiting
Quick use-case:
Cleanly interrupt Gemini CLI or exit shell mode with a single keypress - and quit the CLI entirely with a quick double-tap - thanks to the versatile
Ctrl+C
shortcut
. This gives you immediate control when you need to stop or exit.
Gemini CLI operates like a REPL, and knowing how to break out of operations is essential. Pressing
Ctrl+C
once will cancel the current action or clear any input you've started typing, essentially acting as an "abort"
command
. For example, if the AI is generating a lengthy answer and you've seen enough, hit
Ctrl+C
- the generation stops immediately. If you had started typing a prompt but want to discard it,
Ctrl+C
will wipe the input line so you can start
fresh
. Additionally, if you are in
shell mode
(activated by typing
!
to run shell commands), a single
Ctrl+C
will exit shell mode and return you to the normal Gemini prompt (it sends an interrupt to the shell process
running
. This is extremely handy if a shell command is hanging or you simply want to get back to AI mode.
Pressing
Ctrl+C twice
in a row is the shortcut to exit Gemini CLI
entirely
. Think of it as "
Ctrl+C
to cancel, and
Ctrl+C
again to quit." This double-tap signals the CLI to terminate the session (you'll see a goodbye message or the program will close). It's a faster alternative to typing
/quit
or closing the terminal window, allowing you to gracefully shut down the CLI from the keyboard. Do note that a single
Ctrl+C
will not quit if there's input to clear or an operation to interrupt - it requires that second press (when the prompt is idle) to fully
exit
. This design prevents accidentally closing the session when you only meant to stop the current output.
Pro Tip:
In shell mode, you can also press the
Esc
key to leave shell mode and return to Gemini's chat mode without terminating the
CLI
. And if you prefer a more formal exit, the
/quit
command is always available to cleanly end the session. Lastly, Unix users can use
Ctrl+D
(EOF) at an empty prompt to exit as well - Gemini CLI will prompt for confirmation if
needed
. But for most cases, mastering the single- and double-tap of
Ctrl+C
is the quickest way to stay in control.
Tip 23: Customize Gemini CLI with
settings.json
Quick use-case:
Adapt the CLI's behavior and appearance to your preferences or project conventions by editing the
settings.json
config file, instead of sticking with one-size-fits-all
defaults
. This lets you enforce things like theme, tool usage rules, or editor mode across all your sessions.
Gemini CLI is highly configurable. In your home directory (
~/.gemini/
) or project folder (
.gemini/
within your repo), you can create a
settings.json
file to override default
settings
. Nearly every aspect of the CLI can be tuned here - from visual theme to tool permissions. The CLI merges settings from multiple levels: system-wide defaults, your user settings, and project-specific settings (project settings override user
settings
. For example, you might have a global preference for a dark theme, but a particular project might require stricter tool sandboxing; you can handle this via different
settings.json
files at each level.
Inside
settings.json
, options are specified as JSON key-value pairs. Here's a snippet illustrating some useful customizations:
In this example, we set the theme to "GitHub" (a popular color scheme), disable
autoAccept
(so the CLI will always ask before running potentially altering tools), enable Vim keybindings for the input editor, and enforce using Docker for tool sandboxing. We also added some directories to the workspace context (
includeDirectories
) so Gemini can see code in shared paths by
default
. Finally, we kept
usageStatisticsEnabled
true to collect basic usage stats (which feeds into telemetry, if
enabled
. There are many more settings available - like defining custom color themes, adjusting token limits, or whitelisting/blacklisting specific tools - all documented in the configuration
guide
. By tailoring these, you ensure Gemini CLI behaves optimally for
your
workflow (for instance, some developers always want
vimMode
on for efficiency, while others might prefer the default editor).
One convenient way to edit settings is via the built-in settings UI. Run the command
/settings
in Gemini CLI, and it will open an interactive editor for your
configuration
. This interface lets you browse and search settings with descriptions, and prevents JSON syntax errors by validating inputs. You can tweak colors, toggle features like
yolo
(auto-approval), adjust checkpointing (file save/restore behavior), and more through a friendly
menu
. Changes are saved to your
settings.json
, and some take effect immediately (others might require restarting the CLI).
Pro Tip:
Maintain separate project-specific
settings.json
files for different needs. For example, on a team project you might set
"sandbox": "docker"
and
"excludeTools": ["run_shell_command"]
to lock down dangerous operations, while your personal projects might allow direct shell commands. Gemini CLI will automatically pick up the nearest
.gemini/settings.json
in your project directory tree and merge it with your global
~/.gemini/settings.json
. Also, don't forget you can quickly adjust visual preferences: try
/theme
to interactively switch themes without editing the file, which is great for finding a comfortable
look
. Once you find one, put it in
settings.json
to make it permanent.
Tip 24: Leverage IDE Integration (VS Code) for Context & Diffs
Quick use-case:
Supercharge Gemini CLI by hooking it into VS Code - the CLI will automatically know which files you're working on and even open AI-proposed code changes in VS Code's diff editor for
you
. This creates a seamless loop between AI assistant and your coding workspace.
One of Gemini CLI's powerful features is its
IDE integration
with Visual Studio Code. By installing the official
Gemini CLI Companion
extension in VS Code and connecting it, you allow Gemini CLI to become "context-aware" of your
editor
. What does this mean in practice? When connected, Gemini knows about the files you have open, your current cursor location, and any text you've selected in VS
Code
. All that information is fed into the AI's context. So if you ask, "Explain this function," Gemini CLI can see the exact function you've highlighted and give a relevant answer, without you needing to copy-paste code into the prompt. The integration shares up to your 10 most recently opened files, plus selection and cursor info, giving the model a rich understanding of your
workspace
.
Another huge benefit is
native diffing
of code changes. When Gemini CLI suggests modifications to your code (for example, "refactor this function" and it produces a patch), it can open those changes in VS Code's diff viewer
automatically
. You'll see a side-by-side diff in VS Code showing the proposed edits. You can then use VS Code's familiar interface to review the changes, make any manual tweaks, and even accept the patch with a click. The CLI and editor stay in sync - if you accept the diff in VS Code, Gemini CLI knows and continues the session with those changes applied. This tight loop means you no longer have to copy code from the terminal to your editor; the AI's suggestions flow straight into your development environment.
How to set it up:
If you start Gemini CLI inside VS Code's integrated terminal, it will detect VS Code and usually prompt you to install/connect the extension
automatically
. You can agree and it will run the necessary
/ide install
step. If you don't see a prompt (or you're enabling it later), simply open Gemini CLI and run the command:
/ide install
. This will fetch and install the "Gemini CLI Companion" extension into VS Code for
you
. Next, run
/ide enable
to establish the
connection
- the CLI will then indicate it's linked to VS Code. You can verify at any time with
/ide status
, which will show if it's connected and list which editor and files are being
tracked
. From then on, Gemini CLI will automatically receive context from VS Code (open files, selections) and will open diffs in VS Code when needed. It essentially turns Gemini CLI into an AI pair programmer that lives in your terminal but operates with full awareness of your IDE.
Currently, VS Code is the primary supported editor for this
integration
. (Other editors that support VS Code extensions, like VSCodium or some JetBrains via a plugin, may work via the same extension, but officially it's VS Code for now.) The design is open though - there's an IDE Companion Spec for developing similar integrations with other
editors
. So down the road we might see first-class support for IDEs like IntelliJ or Vim via community extensions.
Pro Tip:
Once connected, you can use VS Code's Command Palette to control Gemini CLI without leaving the
editor
. For example, press
Ctrl+Shift+P
(Cmd+Shift+P on Mac) and try commands like
"Gemini CLI: Run"
(to launch a new CLI session in the terminal),
"Gemini CLI: Accept Diff"
(to approve and apply an open diff), or
"Gemini CLI: Close Diff Editor"
(to reject
changes
. These shortcuts can streamline your workflow even further. And remember, you don't always have to start the CLI manually - if you enable the integration, Gemini CLI essentially becomes an AI co-developer inside VS Code, watching context and ready to help as you work on code.
Tip 25: Automate Repo Tasks with
Gemini CLI GitHub Action
Quick use-case:
Put Gemini to work on GitHub - use the
Gemini CLI GitHub Action
to autonomously triage new issues and review pull requests in your repository, acting as an AI teammate that handles routine dev
tasks
.
Gemini CLI isn't just for interactive terminal sessions; it can also run in CI/CD pipelines via GitHub Actions. Google has provided a ready-made
Gemini CLI GitHub Action
(currently in beta) that integrates into your repo's
workflows
. This effectively deploys an AI agent into your project on GitHub. It runs in the background, triggered by repository
events
. For example, when someone opens a
new issue
, the Gemini Action can automatically analyze the issue description, apply relevant labels, and even prioritize it or suggest duplicates (this is the "intelligent issue triage"
workflow
. When a
pull request
is opened, the Action kicks in to provide an
AI code review
- it will comment on the PR with insights about code quality, potential bugs, or stylistic
improvements
. This gives maintainers immediate feedback on the PR before any human even looks at it. Perhaps the coolest feature is
on-demand collaboration
: team members can mention
@gemini-cli
in an issue or PR comment and give it an instruction, like "
@gemini-cli
please write unit tests for this". The Action will pick that up and Gemini CLI will attempt to fulfill the request (adding a commit with new tests, for
instance
. It's like having an AI assistant living in your repo, ready to do chores when asked.
Setting up the Gemini CLI GitHub Action is straightforward. First, ensure you have Gemini CLI version
0.1.18 or later
installed locally (this ensures compatibility with the
Action
. Then, in Gemini CLI run the special command:
/setup-github
. This command generates the necessary workflow files in your repository (it will guide you through authentication if needed). Specifically, it adds YAML workflow files (for issue triage, PR review, etc.) under
.github/workflows/
. You will need to add your Gemini API key to the repo's secrets (as
GEMINI_API_KEY
) so the Action can use the Gemini
API
. Once that's done and the workflows are committed, the GitHub Action springs to life - from that point on, Gemini CLI will autonomously respond to new issues and PRs according to those workflows.
Because this Action is essentially running Gemini CLI in an automated way, you can customize it just like you would your CLI. The default setup comes with three workflows (issue triage, PR review, and a general mention-triggered assistant) which are **fully open-source and
editable**
. You can tweak the YAML to adjust what the AI does, or even add new workflows. For instance, you might create a nightly workflow that uses Gemini CLI to scan your repository for outdated dependencies or to update a README based on recent code changes - the possibilities are endless. The key benefit here is offloading mundane or time-consuming tasks to an AI agent so that human developers can focus on harder problems. And since it runs on GitHub's infrastructure, it doesn't require your intervention - it's truly a "set and forget" AI helper.
Pro Tip:
Keep an eye on the Action's output in the GitHub Actions logs for transparency. The Gemini CLI Action logs will show what prompts it ran and what changes it made or suggested. This can both build trust and help you refine its behavior. Also, the team has built enterprise-grade safeguards into the Action - e.g., you can require that all shell commands the AI tries to run in a workflow are allow-listed by
you
. So don't hesitate to use it even on serious projects. And if you come up with a cool custom workflow using Gemini CLI, consider contributing it back to the community - the project welcomes new ideas in their repo!
Tip 26: Enable Telemetry for Insights and Observability
Quick use-case:
Gain deeper insight into how Gemini CLI is being used and performing by turning on its built-in
OpenTelemetry
instrumentation - monitor metrics, logs, and traces of your AI sessions to analyze usage patterns or troubleshoot
issues
.
For developers who like to measure and optimize, Gemini CLI offers an observability feature that exposes what's happening under the hood. By leveraging
OpenTelemetry (OTEL)
, Gemini CLI can emit structured telemetry data about your
sessions
. This includes things like metrics (e.g. how many tokens used, response latency), logs of actions taken, and even traces of tool calls. With telemetry enabled, you can answer questions like:
Which custom command do I use most often? How many times did the AI edit files in this project this week? What's the average response time when I ask the CLI to run tests?
Such data is invaluable for understanding usage patterns and
performance
. Teams can use it to see how developers are interacting with the AI assistant and where bottlenecks might be.
By default, telemetry is
off
(Gemini respects privacy and performance). You can opt-in by setting
"telemetry.enabled": true
in your
settings.json
or by starting Gemini CLI with the flag
--telemetry
. Additionally, you choose the
target
for the telemetry data: it can be logged
locally
or sent to a backend like Google Cloud. For a quick start, you might set
"telemetry.target": "local"
- with this, Gemini will simply write telemetry data to a local file (by default) or to a custom path you specify via
["outfile"](https://google-gemini.github.io/gemini-cli/docs/cli/telemetry.html#:~:text=disable%20telemetry%20,file%20path)
. The local telemetry includes JSON logs you can parse or feed into tools. For more robust monitoring, set
"target": "gcp"
(Google Cloud) or even integrate with other OpenTelemetry-compatible systems like Jaeger or
Datadog
. In fact, Gemini CLI's OTEL support is vendor-neutral - you can export data to just about any observability stack you prefer (Google Cloud Operations, Prometheus,
etc.
. Google provides a streamlined path for Cloud: if you point to GCP, the CLI can send data directly to Cloud Logging and Cloud Monitoring in your project, where you can use the usual dashboards and alerting
tools
.
What kind of insights can you get? The telemetry captures events like tool executions, errors, and important milestones. It also records metrics such as prompt processing time and token counts per
prompt
. For usage analytics, you might aggregate how many times each slash command is used across your team, or how often code generation is invoked. For performance monitoring, you could track if responses have gotten slower, which might indicate hitting API rate limits or model changes. And for debugging, you can see errors or exceptions thrown by tools (e.g., a
run_shell_command
failure) logged with context. All this data can be visualized if you send it to a platform like Google Cloud's Monitoring - for example, you can create a dashboard of "tokens used per day" or "error rate of tool X". It essentially gives you a window into the AI's "brain" and your usage, which is especially helpful in enterprise settings to ensure everything runs
smoothly
.
Enabling telemetry does introduce some overhead (extra data processing), so you might not keep it on 100% of the time for personal use. However, it's fantastic for debugging sessions or for intermittent health checks. One approach is to enable it on a CI server or in your team's shared environment to collect stats, while leaving it off locally unless needed. Remember, you can always toggle it on the fly: update settings and use
/memory refresh
if needed to reload, or restart Gemini CLI with
--telemetry
flag. Also, all telemetry is under your control - it respects your environment variables for endpoint and credentials, so data goes only where you intend it to. This feature turns Gemini CLI from a black box into an observatory, shining light on how the AI agent interacts with your world, so you can continuously improve that interaction.
Pro Tip:
If you just want a quick view of your current session's stats (without full telemetry), use the
/stats
command. It will output metrics like token usage and session length right in the
CLI
. This is a lightweight way to see immediate numbers. But for long-term or multi-session analysis, telemetry is the way to go. And if you're sending telemetry to a cloud project, consider setting up dashboards or alerts (e.g., alert if error rate spikes or token usage hits a threshold) - this can proactively catch issues in how Gemini CLI is being used in your team.
Tip 27: Keep an Eye on the Roadmap (Background Agents & More)
Quick use-case:
Stay informed about upcoming Gemini CLI features - by following the public
Gemini CLI roadmap
, you'll know about major planned enhancements (like
background agents for long-running tasks
) before they
arrive
, allowing you to plan and give feedback.
Gemini CLI is evolving rapidly, with new releases coming out frequently, so it's wise to track what's on the horizon. Google maintains a
public roadmap
for Gemini CLI on GitHub, detailing the key focus areas and features targeted for the near
future
. This is essentially a living document (and set of issues) where you can see what the developers are working on and what's in the pipeline. For instance, one exciting item on the roadmap is support for
background agents
- the ability to spawn autonomous agents that run in the background to handle tasks continuously or
asynchronously
. According to the roadmap discussion, these background agents would let you delegate long-running processes to Gemini CLI without tying up your interactive session. You could, say, start a background agent that monitors your project for certain events or periodically executes tasks, either on your local machine or even by deploying to a service like Cloud
Run
. This feature aims to "enable long-running, autonomous tasks and proactive assistance" right from the
CLI
, essentially extending Gemini CLI's usefulness beyond just on-demand queries.
By keeping tabs on the roadmap, you'll also learn about other planned features. These could include new tool integrations, support for additional Gemini model versions, UI/UX improvements, and more. The roadmap is usually organized by "areas" (for example,
Extensibility
,
Model
,
Background
, etc.) and often tagged with milestones (like a target quarter for
delivery
]. It's not a guarantee of when something will land, but it gives a good idea of the team's priorities. Since the project is open-source, you can even dive into the linked GitHub issues for each roadmap item to see design proposals and progress. For developers who rely on Gemini CLI, this transparency means you can anticipate changes - maybe an API is adding a feature you need, or a breaking change might be coming that you want to prepare for.
Following the roadmap can be as simple as bookmarking the GitHub project board or issue labeled "Roadmap" and checking periodically. Some major updates (like the introduction of Extensions or the IDE integration) were hinted at in the roadmap before they were officially announced, so you get a sneak peek. Additionally, the Gemini CLI team often encourages community feedback on those future features. If you have ideas or use cases for something like background agents, you can usually comment on the issue or discussion thread to influence its development.
Pro Tip:
Since Gemini CLI is open source (Apache 2.0 licensed), you can do more than just watch the roadmap - you can participate! The maintainers welcome contributions, especially for items aligned with the
roadmap
. If there's a feature you really care about, consider contributing code or testing once it's in preview. At the very least, you can open a feature request if something you need isn't on the roadmap
yet
. The roadmap page itself provides guidance on how to propose changes. Engaging with the project not only keeps you in the loop but also lets you shape the tool that you use. After all, Gemini CLI is built with community involvement in mind, and many recent features (like certain extensions and tools) started as community suggestions.
Tip 28: Extend Gemini CLI with
Extensions
Quick use-case:
Add new capabilities to Gemini CLI by installing plug-and-play
extensions
- for example, integrate with your favorite database or cloud service - expanding the AI's toolset without any heavy lifting on your
part
. It's like installing apps for your CLI to teach it new tricks.
Extensions are a game-changer introduced in late 2025: they allow you to
customize and expand
Gemini CLI's functionality in a modular
way
. An extension is essentially a bundle of configurations (and optionally code) that connects Gemini CLI to an external tool or service. For instance, Google released a suite of extensions for Google Cloud - there's one that helps deploy apps to Cloud Run, one for managing BigQuery, one for analyzing application security, and
more
. Partners and community developers have built extensions for all sorts of things: Dynatrace (monitoring), Elastic (search analytics), Figma (design assets), Shopify, Snyk (security scans), Stripe (payments), and the list is
growing
. By installing an appropriate extension, you instantly grant Gemini CLI the ability to use new domain-specific tools. The beauty is that these extensions come with a pre-defined
"playbook"
that teaches the AI how to use the new tools
effectively
. That means once installed, you can ask Gemini CLI to perform tasks with those services and it will know the proper APIs or commands to invoke, as if it had that knowledge built-in.
Using extensions is very straightforward. The CLI has a command to manage them:
gemini extensions install <URL>
. Typically, you provide the URL of the extension's GitHub repo or a local path, and the CLI will fetch and install
it
. For example, to install an official extension, you might run:
gemini extensions install https://github.com/google-gemini/gemini-cli-extension-cloud-run
. Within seconds, the extension is added to your environment (stored under
~/.gemini/extensions/
or your project's
.gemini/extensions/
folder). You can then see it by running
/extensions
in the CLI, which lists active
extensions
. From that point on, the AI has new tools at its disposal. If it's a Cloud Run extension, you could say "Deploy my app to Cloud Run," and Gemini CLI will actually be able to execute that (by calling the underlying
gcloud
commands through the extension's tools). Essentially, extensions function as first-class expansions of Gemini CLI's capabilities, but you opt-in to the ones you need.
There's an
open ecosystem
around extensions. Google has an official Extensions page listing available
extensions
, and because the framework is open, anyone can create and share their own. If you have a particular internal API or workflow, you can build an extension for it so that Gemini CLI can assist with it. Writing an extension is easier than it sounds: you typically create a directory (say,
my-extension/
) with a file
gemini-extension.json
describing what tools or context to
add
. You might define new slash commands or specify remote APIs the AI can call. No need to modify Gemini CLI's core - just drop in your extension. The CLI is designed to load these at runtime. Many extensions consist of adding custom
MCP tools
(Model Context Protocol servers or functions) that the AI can use. For example, an extension could add a
/translate
command by hooking into an external translation API; once installed, the AI knows how to use
/translate
. The key benefit is
modularity
: you install only the extensions you want, keeping the CLI lightweight, but you have the option to integrate virtually anything.
To manage extensions, besides the
install
command, you can update or remove them via similar CLI commands (
gemini extensions update
or just by removing the folder). It's wise to occasionally check for updates on extensions you use, as they may receive improvements. The CLI might introduce an "extensions marketplace" style interface in the future, but for now, exploring the GitHub repositories and official catalog is the way to discover new ones. Some popular ones at launch include the GenAI
Genkit
extension (for building generative AI apps), and a variety of Google Cloud extensions that cover CI/CD, database admin, and more.
Pro Tip:
If you're building your own extension, start by looking at existing ones for examples. The official documentation provides an
Extensions Guide
with the schema and
capabilities
. A simple way to create a private extension is to use the
@include
functionality in
GEMINI.md
to inject scripts or context, but a full extension gives you more power (like packaging tools). Also, since extensions can include context files, you can use them to preload domain knowledge. Imagine an extension for your company's internal API that includes a summary of the API and a tool to call it - the AI would then know how to handle requests related to that API. In short, extensions open up a new world where Gemini CLI can interface with anything. Keep an eye on the extensions marketplace for new additions, and don't hesitate to share any useful extension you create with the community - you might just help thousands of other
developers
.
Additional Fun: Corgi Mode Easter Egg 🐕
Lastly, not a productivity tip but a delightful easter egg - try the command
*/corgi*
in Gemini CLI. This toggles
"corgi mode"
, which makes a cute corgi animation run across your
terminal
! It doesn't help you code any better, but it can certainly lighten the mood during a long coding session. You'll see an ASCII art corgi dashing in the CLI interface. To turn it off, just run
/corgi
again.
This is a purely for-fun feature the team added (and yes, there's even a tongue-in-cheek
debate
about spending dev time on corgi mode). It shows that the creators hide some whimsy in the tool. So when you need a quick break or a smile, give
/corgi
a try. 🐕🎉
(Rumor has it there might be other easter eggs or modes - who knows? Perhaps a "/partyparrot" or similar. The cheat sheet or help command lists
/corgi
, so it's not a secret, just underused. Now you're in on the joke!)
Conclusion:
We've covered a comprehensive list of pro tips and features for Gemini CLI. From setting up persistent context with
GEMINI.md
, to writing custom commands and using advanced tools like MCP servers, to leveraging multi-modal inputs and automating workflows, there's a lot this AI command-line assistant can do. As an external developer, you can integrate Gemini CLI into your daily routine - it's like a powerful ally in your terminal that can handle tedious tasks, provide insights, and even troubleshoot your environment.
Gemini CLI is evolving rapidly (being open-source with community contributions), so new features and improvements are constantly on the horizon. By mastering the pro tips in this guide, you'll be well-positioned to harness the full potential of this tool. It's not just about using an AI model - it's about integrating AI deeply into how you develop and manage software.
Happy coding with Gemini CLI, and have fun exploring just how far your "AI agent in the terminal" can take you.
You now have a Swiss-army knife of AI at your fingertips - use it wisely, and it will make you a more productive (and perhaps happier) developer
!
A Lone Astronomer Has Reported a Dark Matter ‘Annihilation’ Breakthrough
403 Media
www.404media.co
2025-11-26 17:50:20
“It was like playing the lottery,” said astronomer Tomonori Totani, adding that he hopes other scientists will verify the possible detection of a new dark matter signature....
Subscribe
to 404 Media to get
The Abstract
, our newsletter about the most exciting and mind-boggling science news and studies of the week.
An astronomer has reported a possible new signature of dark matter, a mysterious substance that makes up most of the universe, according to
a study published on Tuesday
in the
Journal of Cosmology and Astroparticle Physics
.
Dark matter accounts for 85 percent of all matter in the universe, but its existence has so far been inferred only from its indirect effects on the familiar “baryonic” matter that makes up stars, planets, and life.
Tomonori Totani, a professor of astronomy at the University of Tokyo and the author of the study, believes he has spotted novel indirect traces of dark matter particles in the “halo” surrounding the center of our galaxy using new observations from NASA’s Fermi Gamma-ray Space Telescope. When these speculative particles collide—a process called dark matter annihilation—the crash is predicted to emit bright gamma rays, which is the light that Totani thinks he has identified.
“The discovery was made possible by focusing on the halo region (excluding the galactic center), which had received little attention, and by utilizing data accumulated over 15 years from the Fermi satellite,” Totani told 404 Media in an email. “After carefully removing all components other than dark matter, a signal resembling dark matter appeared.”
“It was like playing the lottery, and at first I was skeptical,” he added. “But after checking meticulously and thinking it seemed correct, I got goosebumps!”
If the detection is corroborated by follow-up studies, it could confirm a leading hypothesis that dark matter is made of a hypothetical class of weakly interacting massive particles, or “WIMPs”—potentially exposing the identity of this mysterious substance for the first time. But that potential breakthrough is still a ways off, according to other researchers in the field.
“Any new structure in the gamma-ray sky is interesting, but the dark matter interpretation here strikes me as quite preliminary,” said Danielle Norcini, an experimental particle physicist and
assistant professor at Johns Hopkins University, in an email to 404 Media.
Gamma-ray intensity map excluding components other than the halo, spanning approximately 100 degrees in the direction of the Galactic center. The horizontal gray bar in the central region corresponds to the Galactic plane area, which was excluded from the analysis to avoid strong astrophysical radiation. Image: Tomonori Totani, The University of Tokyo
Dark matter has flummoxed scientists for almost a century. In the 1930s, astronomer Fritz Zwicky observed that the motions of galaxies hinted that they are much more massive than expected based solely on visible baryonic matter. Since then, astronomers have confirmed that dark matter, which accumulates into dense halos at the centers of galaxies, acts like a gravitational glue that holds structures together. Dark matter is also the basis of a vast cosmic web of gaseous threads that links galaxy clusters across billions of light years.
But while dark matter is ubiquitous, it does not interact with the electromagnetic force, which means it does not absorb, reflect, or emit light. This property makes it difficult to spot with traditional astronomy, a challenge that has inspired the development of novel instruments designed to directly detect dark matter such as the subterranean LUX-ZEPLIN in South Dakota and the forthcoming DAMIC-M in France.
For years, scientists have been
probing possible emission from dark matter annihilation
at the center of the Milky Way, which is surrounded by a halo of densely-clustered dark matter. Those previous studies focus on an excess emission pattern of about 2 gigaelectronvolts (GeV). Tontani’s study spotlights a new and different pattern with extremely energetic gamma rays at 20 GeV.
“A part of the Fermi data showed a peculiar excess that our model couldn't explain, leading me to suspect it might be due to radiation originating from dark matter,” he said. “The most difficult part is removing gamma-ray emissions of origins other than dark matter, such as those from cosmic rays and celestial objects.”
This tentative report may finally fill in a major missing piece of our understanding of the universe by exposing the true nature of dark matter and confirming the existence of WIMPs. But given that similar claims have been made in the past, more research is needed to assess the significance of the results.
“For any potential indirect signal, the key next steps are independent checks: analyses using different background models, different assumptions about the Milky Way halo, and ideally complementary data sets,” Norcini said.
“Gamma-ray structures in the halo can have many astrophysical origins, so ruling those out requires careful modeling and cross-comparison,” she continued. “At this point the result seems too new for that scrutiny to have played out, and it will take multiple groups looking at the same data before a dark matter interpretation could be considered robust.”
Though Totani is confident in his interpretation of his discovery, he also looks forward to the input of other dark matter researchers around the world.
“First, I would like other researchers to independently verify my analysis,” he said. “Next, for everyone to be convinced that this is truly dark matter, the decisive factor will be the detection of gamma rays with the same spectrum from other regions, such as dwarf galaxies. The accumulation of further data from the Fermi satellite and large ground-based gamma-ray telescopes, such as the Cherenkov Telescope Array Observatory (CTAO) will be crucial.”
🌘
Subscribe
to 404 Media to get
The Abstract
, our newsletter about the most exciting and mind-boggling science news and studies of the week.
Scaleway turns Mac minis into high‑density, Raspberry Pi–managed servers
Take a behind-the-scenes look at how Scaleway brought the Mac mini as-a-Service to life — transforming Apple’s compact desktop into a highly available cloud server hosted in state-of-the-art datacenters.
From Consumer Machine to Cloud Server: A Fully Controlled Pipeline
Apple designs the Mac mini. inmac wstore supplies it. Scaleway transforms it into a
ready-to-use dedicated server
, accessible remotely from anywhere in the world.
Scaleway’s mission is clear: to provide iOS and macOS developers, macOS software users, and businesses of all sizes with remote access to the power of Apple silicon (M-series) chips — all within a controlled, secure, and high-performance environment.
Each Mac mini is managed automatically. Once installed in the racks, Scaleway’s teams add a custom Mobile Device Management (MDM) profile to deploy system settings remotely, along with a set of server-specific tools that compensate for the lack of a Baseboard Management Controller (BMC). This enables
granular management
of each machine.
Thanks to this process, we at Scaleway can deliver a consumer-grade Mac mini as a fully reliable dedicated server, seamlessly integrated into our cloud ecosystem — ready to meet even the most demanding production needs.
A Datacenter Designed for Efficiency and Resilience
All Scaleway Mac minis are hosted exclusively in French datacenters, ensuring sovereign hosting that meets the highest standards for security, privacy, and data locality in Europe.
At the heart of this infrastructure lies Opcore DC2, Scaleway’s strategic datacenter located in Vitry-sur-Seine, where hundreds of Mac minis run side by side with traditional bare-metal servers — all within a
resilient, high-performance network architecture
monitored in real time.
Scaleway’s datacenter design reflects its commitment to performance and reliability:
Power & Redundancy
: 3N electrical system with automatic failover, three backup generators, and a total power capacity of up to 8,000 kW.
Precision Cooling
: Cold corridors with underfloor air distribution optimize temperature and prevent hot spots — minimizing energy use.
Advanced Security
: 24/7 monitoring, biometric access controls, and a water-vapor fire suppression system that protects equipment without damage.
A Custom-Built Rack for Mac minis
The Mac mini wasn’t originally designed for datacenter environments: there’s no BMC (Baseboard Management Controller), no native remote firmware access, and no standard rackmount format.
To overcome this, Scaleway engineered a
custom chassis
where each Mac mini is placed in an individual sliding tray. This allows any unit to be removed for maintenance without disrupting the others — ensuring maximum density and ease of access. Ethernet cabling is carefully organized to guarantee fast, stable network connections.
Each rack can hold up to
96 Mac minis
, an impressive density compared to traditional servers. This is made possible by two key factors:
The
compact size
of the Mac mini, which packs a powerful System on a Chip (SoC) into a tiny footprint.
The
energy efficiency
of Apple silicon (M-series) chips, which allows high density without overheating or excessive power draw.
As a result,
Scaleway’s Mac mini racks are among the most energy-efficient server setups in the cloud industry.
However, the absence of a BMC posed a major challenge: how to perform critical remote operations without physical access?
Scaleway’s solution to that problem was ingenious:
embedding a Raspberry Pi module
with each Mac mini.
Each Raspberry Pi acts as a control layer, sending commands such as reboot or remote reinstall to the Mac mini. This makes the machines virtually autonomous throughout their cloud lifecycle, while remaining fully compliant with Apple’s hardware requirements.
The Future of Mac Minis in the Scaleway Cloud
Scaleway plans to keep expanding its Mac mini fleet as cloud-native development evolves
. Future versions of macOS, the rise of AI workloads, and the growing need for macOS environments in cross-platform development are all driving demand.
With Mac mini as-a-Service, Scaleway delivers a powerful, flexible solution designed for developers, tech companies, and demanding freelancers alike.
ai-PULSE
, Europe’s premier Artificial Intelligence conference powered by Scaleway, is returning!
Gathering key players from across Europe, the event will be back once again at
STATION F on December 4
for a unique blend of deep technical expertise and crucial business insights.
You’ll hear from:
Micah Hill-Smith, Co-Founder & CEO of Artificial Analysis, on which metrics truly matter in the new AI stack
Boris Gamazaychikov, Head of AI Sustainability at Salesforce, on how we can make “energy-efficient” AI measurable
Pauline Pham, Strategy & Operations at Dust, on building and orchestrating agentic fleets
... and dozens more leaders and engineers shaping the technology’s future.
s&box is a modern game engine, built on Valve's Source 2 and the latest .NET technology, it provides a modern intuitive editor for creating games.
If your goal is to create games using s&box, please start with the
getting started guide
.
This repository is for building the engine from source for those who want to contribute to the development of the engine.
Getting the Engine
Steam
You can download and install the s&box editor directly from
Steam
.
Compiling from Source
If you want to build from source, this repository includes all the necessary files to compile the engine yourself.
# Clone the repo
git clone https://github.com/Facepunch/sbox-public.git
Once you've cloned the repo simply run
Bootstrap.bat
which will download dependencies and build the engine.
The game and editor can be run from the binaries in the game folder.
Contributing
If you would like to contribute to the engine, please see the
contributing guide
.
If you want to report bugs or request new features, see
sbox-issues
.
Documentation
Full documentation, tutorials, and API references are available at
sbox.game/dev/
.
License
The s&box engine source code is licensed under the
MIT License
.
Certain native binaries in
game/bin
are not covered by the MIT license. These binaries are distributed under the s&box EULA. You must agree to the terms of the EULA to use them.
This project includes third-party components that are separately licensed.
Those components are not covered by the MIT license above and remain subject
to their original licenses as indicated in
game/thirdpartylegalnotices
.
Suppose Alice runs a confidential restaurant. Alice doesn’t want there to be any record of who visited her restaurant but does want to get paid for her food. She accepts Monero, and instead of a cash register there are two QR codes on display, one corresponding to her
public view key
A
and the other corresponding to her
public spend key
S
.
How Bob buys his burger
A customer Bob walks into the restaurant and orders a burger and fries. When Bob pays Alice, here’s what’s going on under the hood.
Bob is using software that generates a random integer
r
and multiplies it by a point
G
on an elliptic curve, specifically ed25519, obtaining the point
R
=
rG
on the curve. The software also multiplies Alice’s view key
A
, a point on the same elliptic curve, by
r
, then runs a hash function
H
on the produce
rA
that returns an integer
k
.
k
=
H
(
rA
).
Finally, Bob’s software computes the point
P
=
k
G
+
S
and sends Alice’s cash register, i.e. her crypto wallet, the pair of points (
P
,
R
). The point
P
is a
stealth address
, an address that will only be used this one time and cannot be linked to Alice or Bob [1]. The point
R
is additional information that helps Alice receive her money.
How Alice gets paid
Alice and Bob share a secret: both know
k
. How’s that?
Alice’s public view key
A
is the product of her private view key
a
and the group generator
G
[2]. So when Bob computes
rA
, he’s computing
r
(
aG
). Alice’s software can multiply the point
R
by
a
to obtain
a
(
rG
).
rA
=
r
(
aG
) =
a
(
rG
) =
aR.
Both Alice and Bob can hash this point—which Alice thinks of as
aR
and Bob thinks of as
rA
—to obtain
k
. This is
ECDH
: elliptic curve Diffie-Hellman key exchange.
Next, Alice’s software scans the blockchain for payments to
P
=
k
G
+
S.
Note that
P
is on the blockchain, but only Alice and Bob know how to factor
P
into
kG
+
S
because only Alice and Bob know
k
. And only Alice can spend the money because only she knows the private key
s
corresponding to the public spend key
S
where
S
=
sG.
She knows
P
=
kG
+
sG
= (
k
+
s
)
G
and so she has the private key (
k
+
s
) corresponding to
P
.
[1] Bob sends money to the address
P
, so there could be some connection between Bob and
P
on the Monero blockchain. However, due to another feature of Monero, namely ring signatures, someone analyzing the blockchain could only determine that Bob is one of 16 people who may have sent money to the address
P
, and there’s no way to know who received the money. That is, there is no way, using only information on the blockchain, who received the money. A private investigator who saw Bob walk into Alice’s restaurant would have additional information outside the blockchain.
[2] The key assumption of elliptic curve cryptography is that it’s computationally infeasible to “divide” on an elliptic curve, i.e. to recover
a
from knowledge of
G
and
aG
. You could recover
a
by brute force if the group were small, but the elliptic curve ed25519 has on the order of 2
255
points, and
a
is some integer chosen randomly between 1 and the size of the curve.
Multiple London councils' IT systems disrupted by cyberattack
Bleeping Computer
www.bleepingcomputer.com
2025-11-26 17:26:11
The Royal Borough of Kensington and Chelsea (RBKC) and the Westminster City Council (WCC) announced that they are experiencing service disruptions following a cybersecurity issue. [...]...
The Royal Borough of Kensington and Chelsea (RBKC) and the Westminster City Council (WCC) announced that they are experiencing service disruptions following a cybersecurity issue.
Multiple systems have been impacted by the attack, including phone lines, which prompted the two councils to activate emergency plans to make sure that residents still receive critical services.
The two authorities have been impacted at the same time because they share some IT infrastructure as part of joint arrangements.
A third council, the London Borough of Hammersmith and Fulham (LBHF), also shares some services with RBKC and WCC and decided to take "enhanced measures to isolate and safeguard our networks," which led to business disruptions.
Westminster City Council is a major local authority in the U.K., with important landmarks in the area, like the Palace of Westminster (Houses of Parliament), the Buckingham Palace, 10 Downing Street, national institutions, important shopping streets, and significant tourist hotspots.
The councils, which provide services for 360,000 residents, shut down several computerised systems as a precaution to limit further possible damage.
RBKC is one of the smallest boroughs in London (in terms of size and population) but also the wealthiest (in terms of GDP per capita) in the UK, while LBHF is a mid-sized but still significant council serving 180,000 residents.
In an announcement yesterday, the RBKC said that it had an issue that prevented residents from contacting the council through online services or the contact center.
The council later published a statement saying that it was "responding to a cyber security issue" that occurred on Monday and also affected Westminster City Council.
The local authority stated that investigations into the perpetrators and their motives are ongoing and that it will publish updates as soon as more information becomes available.
"[...] the two authorities have been working closely together and with the help of specialist cyber incident experts and the National Cyber Security Centre, with the focus on protecting systems and data, restoring systems, and maintaining critical services to the public."
"We don’t have all the answers yet, as the management of this incident is still ongoing,"
RBKC says
, adding that “we know people will have concerns, so we will be updating residents and partners further over the coming days.”
“At this stage, it is too early to say who did this and why, but we are investigating to see if any data has been compromised.”
The council states that it has already informed the UK Information Commissioner’s Office (ICO), in accordance to established protocols.
The other two councils,
WCC
and
LBHF
, have published short statements about the disruption via banners on their websites, listing alternative phone numbers people can use right now to contact them.
BleepingComputer has contacted RBKC to ask more details about the shared IT system, but a spokesperson declined to disclose any additional information at this time.
Security expert
Kevin Beaumont said
that the incident is a ransomware attack at a services provider used by the three councils.
At the time of writing, no ransomware groups publicly claimed the attack.
As MCP (Model Context Protocol) becomes the standard for connecting LLMs to tools and data, security teams are moving fast to keep these new services safe.
This free cheat sheet outlines 7 best practices you can start using today.
Meet Rey, the Admin of ‘Scattered Lapsus$ Hunters’
Krebs
krebsonsecurity.com
2025-11-26 17:22:36
A prolific cybercriminal group that calls itself "Scattered LAPSUS$ Hunters" made headlines regularly this year by stealing data from and publicly mass extorting dozens of major corporations. But the tables seem to have turned somewhat for "Rey," the moniker chosen by the technical operator and publ...
A prolific cybercriminal group that calls itself “
Scattered LAPSUS$ Hunters
” has dominated headlines this year by regularly stealing data from and publicly mass extorting dozens of major corporations. But the tables seem to have turned somewhat for “Rey,” the moniker chosen by the technical operator and public face of the hacker group: Earlier this week, Rey confirmed his real life identity and agreed to an interview after KrebsOnSecurity tracked him down and contacted his father.
Scattered LAPSUS$ Hunters (SLSH) is thought to be an amalgamation of three hacking groups —
Scattered Spider
,
LAPSUS$
and
ShinyHunters
. Members of these gangs hail from many of the same chat channels on the
Com
, a mostly English-language cybercriminal community that operates across an ocean of Telegram and Discord servers.
In May 2025, SLSH members launched
a social engineering campaign
that used voice phishing to trick targets into connecting a malicious app to their organization’s Salesforce portal. The group later launched a data leak portal that threatened to publish the internal data of three dozen companies that allegedly had Salesforce data stolen, including
Toyota
,
FedEx
,
Disney/Hulu
, and
UPS
.
The new extortion website tied to ShinyHunters, which threatens to publish stolen data unless Salesforce or individual victim companies agree to pay a ransom.
Last week, the SLSH Telegram channel featured an offer to recruit and reward “insiders,” employees at large companies who agree to share internal access to their employer’s network for a share of whatever ransom payment is ultimately paid by the victim company.
SLSH has solicited insider access previously, but their latest call for disgruntled employees started making the rounds on social media at the same time news broke that the cybersecurity firm
Crowdstrike
had fired an employee for
allegedly sharing screenshots of internal systems
with the hacker group (Crowdstrike said their systems were never compromised and that it has turned the matter over to law enforcement agencies).
The Telegram server for the Scattered LAPSUS$ Hunters has been attempting to recruit insiders at large companies.
Members of SLSH have traditionally used other ransomware gangs’ encryptors in attacks, including malware from ransomware affiliate programs like ALPHV/BlackCat, Qilin, RansomHub, and DragonForce. But last week, SLSH announced on its Telegram channel
the release of their own ransomware-as-a-service operation
called
ShinySp1d3r
.
The individual responsible for releasing the ShinySp1d3r ransomware offering is a core SLSH member who goes by the handle “Rey” and who is currently one of just three administrators of the SLSH Telegram channel. Previously, Rey was an
administrator
of the data leak website for
Hellcat
, a ransomware group that surfaced in late 2024 and was involved in attacks on companies including
Schneider Electric
,
Telefonica
, and
Orange Romania
.
A recent, slightly redacted screenshot of the Scattered LAPSUS$ Hunters Telegram channel description, showing Rey as one of three administrators.
Also in 2024, Rey would take over as administrator of the
most recent incarnation of BreachForums
, an English-language cybercrime forum whose domain names have been seized on multiple occasions by the FBI and/or by international authorities. In April 2025, Rey
posted on Twitter/X
about another FBI seizure of BreachForums.
On October 5, 2025, the FBI
announced
it had once again seized the domains associated with BreachForums, which it described as a major criminal marketplace used by ShinyHunters and others to traffic in stolen data and facilitate extortion.
“This takedown removes access to a key hub used by these actors to monetize intrusions, recruit collaborators, and target victims across multiple sectors,” the FBI said.
Incredibly, Rey would make a series of critical operational security mistakes last year that provided multiple avenues to ascertain and confirm his real-life identity and location. Read on to learn how it all unraveled for Rey.
WHO IS REY?
According to the cyber intelligence firm
Intel 471
, Rey was an active user on various
BreachForums
reincarnations over the past two years, authoring more than 200 posts between February 2024 and July 2025. Intel 471 says Rey previously used the handle “
Hikki-Chan
” on BreachForums, where their first post shared data allegedly stolen from the
U.S. Centers for Disease Control and Prevention
(CDC).
In that February 2024 post about the CDC, Hikki-Chan says they could be reached at the Telegram username
@wristmug
. In May 2024, @wristmug posted in a Telegram group chat called “Pantifan” a copy of an extortion email they said they received that included their email address and password.
The message that @wristmug cut and pasted appears to have been part of an
automated email scam
that claims it was sent by a hacker who has compromised your computer and used your webcam to record a video of you while you were watching porn. These missives threaten to release the video to all your contacts unless you pay a Bitcoin ransom, and they typically reference a real password the recipient has used previously.
“Noooooo,” the @wristmug account wrote in mock horror after posting a screenshot of the scam message. “I must be done guys.”
A message posted to Telegram by Rey/@wristmug.
In posting their screenshot, @wristmug redacted the username portion of the email address referenced in the body of the scam message. However, they did not redact their previously-used password, and they left the domain portion of their email address (@proton.me) visible in the screenshot.
O5TDEV
Searching on @wristmug’s rather unique 15-character password in the breach tracking service
Spycloud
finds it is known to have been used by just one email address:
cybero5tdev@proton.me
. According to Spycloud, those credentials were exposed at least twice in early 2024 when this user’s device
was infected with an infostealer trojan
that siphoned all of its stored usernames, passwords and authentication cookies.
Intel 471 shows the email address cybero5tdev@proton.me belonged to a BreachForums member who went by the username
o5tdev
. Searching on this nickname in Google brings up at least two website defacement archives showing that a user named o5tdev was previously involved in
defacing sites with pro-Palestinian messages
. The screenshot below, for example, shows that 05tdev was part of a group called
Cyb3r Drag0nz Team
.
A 2023 report from
SentinelOne
described Cyb3r Drag0nz Team as a hacktivist group with a history of launching DDoS attacks and cyber defacements as well as engaging in data leak activity.
“Cyb3r Drag0nz Team claims to have leaked data on over a million of Israeli citizens spread across multiple leaks,” SentinelOne
reported
. “To date, the group has released multiple .RAR archives of purported personal information on citizens across Israel.”
The cyber intelligence firm
Flashpoint
finds the Telegram user @05tdev was active in 2023 and early 2024, posting in Arabic on anti-Israel channels like “Ghost of Palestine” [full disclosure: Flashpoint is currently an advertiser on this blog].
‘I’M A GINTY’
Flashpoint shows that Rey’s Telegram account (ID7047194296) was particularly active in a cybercrime-focused channel called
Jacuzzi
, where this user shared several personal details, including that their father was an airline pilot. Rey claimed in 2024 to be 15 years old, and to have family connections to Ireland.
Specifically, Rey mentioned in several Telegram chats that he had Irish heritage, even posting a graphic that shows the prevalence of the surname “
Ginty
.”
Rey, on Telegram claiming to have association to the surname “Ginty.” Image: Flashpoint.
Spycloud indexed hundreds of credentials stolen from cybero5dev@proton.me, and those details indicate that Rey’s computer is a shared Microsoft Windows device located in Amman, Jordan. The credential data stolen from Rey in early 2024 show there are multiple users of the infected PC, but that all shared the same last name of Khader and the address Hamad Al-Qanawi Street, Building 11, in Amman, Jordan.
The “autofill” data lifted from Rey’s family PC contains an entry for a 46-year-old
Zaid Khader
that says his mother’s maiden name was Ginty. The infostealer data also shows Zaid Khader frequently accessed internal websites for employees of
Royal Jordanian Airlines
.
MEET SAIF
The infostealer data makes clear that Rey’s full name is
Saif Al-Din Khader
. Having no luck contacting Saif directly, KrebsOnSecurity sent an email to his father Zaid. The message invited the father to respond via email, phone or Signal, explaining that his son appeared to be deeply enmeshed in a serious cybercrime conspiracy.
Less than two hours later, I received a Signal message from Saif, who said his dad suspected the email was a scam and had forwarded it to him.
“I saw your email, unfortunately I don’t think my dad would respond to this because they think its some ‘scam email,'” said Saif, who told me he turns 16 years old next month. “So I decided to talk to you directly.”
Saif explained that he’d already heard from European law enforcement officials, and had been trying to extricate himself from SLSH. When asked why then he was involved in releasing SLSH’s new ShinySp1d3r ransomware-as-a-service offering, Saif said he couldn’t just suddenly quit the group.
“Well I cant just dip like that, I’m trying to clean up everything I’m associated with and move on,” he said.
The former Hellcat ransomware site. Image: Kelacyber.com
He also shared that ShinySp1d3r is just a rehash of Hellcat ransomware, except modified with AI tools. “I gave the source code of Hellcat ransomware out basically.”
“I’m already cooperating with law enforcement,” Saif said. “In fact, I have been talking to them since at least June. I have told them nearly everything. I haven’t really done anything like breaching into a corp or extortion related since September.”
Saif suggested that a story about him right now could endanger any further cooperation he may be able to provide. He also said he wasn’t sure if the U.S. or European authorities had been in contact with the Jordanian government about his involvement with the hacking group.
“A story would bring so much unwanted heat and would make things very difficult if I’m going to cooperate,” Saif Khader said. “I’m unsure whats going to happen they said they’re in contact with multiple countries regarding my request but its been like an entire week and I got no updates from them.”
Saif shared a screenshot that indicated he’d contacted Europol authorities late last month. But he couldn’t name any law enforcement officials he said were responding to his inquiries, and KrebsOnSecurity was unable to verify his claims.
“I don’t really care I just want to move on from all this stuff even if its going to be prison time or whatever they gonna say,” Saif said.
This Commission That Regulates Crypto Could Be Just One Guy: An Industry Lawyer
Intercept
theintercept.com
2025-11-26 17:14:55
Mike Selig had dozens of crypto clients. Now he will be a key industry regulator.
The post This Commission That Regulates Crypto Could Be Just One Guy: An Industry Lawyer appeared first on The Intercept....
Republicans in the Senate
are racing to confirm a lawyer with a long list of crypto industry clients as the next Commodity Futures Trading Commission chair, a position that will hold wide sway over the industry.
CFTC nominee Mike Selig has served dozens of crypto clients ranging from venture capital firms to a bear-themed blockchain company based in the Cayman Islands, according to ethics records obtained by The Intercept.
Those records show the breadth of potential conflicts of interest for Selig, who, if confirmed, will serve on the CFTC alone due to an exodus of other commissioners.
With a Bitcoin crash
wiping out a trillion dollars of value
in the past few weeks, the industry is counting on friendly regulators in Washington to give it a boost.
Senate Agriculture Committee members
voted 12-11 on party lines
in favor of Selig on November 20, setting up a vote in the full Senate. The committee vote came a day after a hearing in which Selig dodged straightforward questions about whether CFTC staffing should be expanded as it takes on a role overseeing digital assets, and whether Donald Trump was right to
pardon Binance founder Changpeng Zhao.
One thing Selig was committal on, however, was the danger of over-enforcement — leading the consumer group Better Markets to criticize him as the “wrong choice” to lead the CFTC.
“The CFTC is facing unprecedented strain as crypto and prediction market oversight has been layered into its traditional derivatives market oversight responsibilities,”
said
Benjamin Schiffrin, the nonprofit group’s director of securities policy. “During his hearing, Mr. Selig showed little interest in regulation on either count and was unable to answer the simplest of questions.”
Selig’s resume shows why the industry is so comfortable with him. Early in his career he was a law clerk for J. Christopher Giancarlo, the CFTC chair during Trump’s first term who calls himself CryptoDad.
After the CFTC, Selig joined Giancarlo at the white-shoe law firm Willkie Farr & Gallagher. His client list there extended from major crypto investors to smaller startups, many of them with some presence in the derivatives or commodities worlds, according to a form he filed with the Office of Government Ethics after his nomination.
Selig’s clients included Amir Haleem, the CEO of a crypto company that was the target of a
yearslong Securities and Exchange Commission probe
; Architect Financial Technologies, which last year
announced
a CFTC-regulated digital derivatives brokerage; Berachain, the Caymans-based blockchain company whose pseudonymous co-founders include
“Smokey the Bera” and “Papa Bear
”; CoinList, a crypto exchange that allows traders to access newly listed digital tokens; Deribit, a crypto options exchange; Diamond Standard, which offers commodities products that combine diamonds and the blockchain; Input Output Global, one of the developers of the decentralized blockchain Cardano; and the U.S. branch of eToro, an Israeli crypto trading platform.
“Yes, I think the crypto community is excited about Mike.”
At least one of Selig’s former clients, Alluvial Finance, met with staffers of the crypto task force where Selig has served as chief counsel since the start of the second Trump administration,
according to SEC records
.
Selig’s clients have also included trade groups including the Proof of Stake Alliance, which advocates for friendly tax policies for a type of blockchain, and the Blockchain Association, which represents dozens of investment firms and large crypto companies in Washington.
Pushing back against the idea that Selig was a one-trick pony in a recent podcast interview, Giancarlo said that Selig’s interests extended to other industries overseen by the CFTC such as agriculture.
“Yes, I think the crypto community is excited about Mike. But so is the whole CFTC community,” Giancarlo said. “It’s not, ‘Crypto bro goes to CFTC.’ This is somebody who has had a decadelong practice in all aspects of CFTC law and jurisdiction, and is accomplished in all those areas.”
Revolving Door
It is far from unusual for Republican presidents to tap industry-friendly lawyers to serve as financial regulators. Selig, though, is poised to assume a uniquely powerful position thanks to a more unusual circumstance: an exodus of CFTC commissioners this year.
The commission’s other members fled for the doors since Trump’s second term began, with only a single, crypto-friendly Republican left to serve as acting chair. She has said that she will step down once her replacement is confirmed.
Trump so far has yet to nominate any Democratic commissioners on the body that is typically split 3-2 along party lines, with the majority going to the party that controls the White House.
That appears to have been the sticking point for the Democratic senators who unanimously voted against Selig at the committee vote.
Selig may not have to recuse himself from matters involving his former clients as CFTC chair, it appears. In his government ethics filing, Selig pledged not to involve himself in matters involving his former clients for the standard period of a year after he represented them. However, Selig has been in government service for most of 2025, meaning that there are only a few weeks remaining of that blackout period.
A White House spokesperson did not answer questions about potential conflicts of interest if Selig is confirmed.
“Mike Selig is a highly qualified crypto and industry leader, who will do an excellent job in leading the Commodity Futures Trading Commission under President Trump,” White House spokesperson Davis Ingle said in a statement. “We look forward to his swift confirmation.”
Backwater to Bleeding Edge
If confirmed, Selig will lead an agency that was once considered a relative backwater until it was put in charge of
regulating derivates after the 2008 financial crash
. More recently, Congress advanced legislation that would put the CFTC on the bleeding edge of overseeing digital assets.
Nonetheless, even relatively crypto-friendly Democrats, such as Sen. Cory Booker of New Jersey, noted at the hearing last week that the agency has nowhere near the staff needed to take on a major new role in the financial markets. The CFTC has only 161 employees dedicated to enforcement actions compared to about 1,500 at the SEC, Booker said.
“There is a real problem right now with capacity in the agency that you are up to lead,” Booker told Selig.
Despite the dearth of both commissioners and staff, Selig was unwilling to commit to growing the agency if he is confirmed. Pressed by Democrats whether he would ask Trump for a bigger staff, Selig repeatedly said that he needed to study the issue.
Selig also avoided giving direct answer to questions from Democrats as to whether the CFTC should crack down on the emerging world of “prediction markets” offering sports gambling
outside the auspices of state regulation
, and whether crypto exchanges should be allowed to “vertically integrate” by investing in the same tokens they allow customers to trade.
Selig did signal a general openness toward cryptocurrencies — and skepticism of regulation — in his statement to the committee.
“I have seen firsthand how regulators, unaware of the real-world impact of their efforts, and zeal for regulation-by-enforcement, can drive businesses offshore and smother entrepreneurs with red tape,”
Selig said
. “Everyday Americans pay the price for these regulatory failures. If confirmed, I am committed to instituting common sense, principles-based regulations that facilitate well-functioning markets and keep pace with the rapid speed of innovation.”
DRAM prices are spiking, but I don't trust the industry's why
Use promo code:
n8RjrbAt
at checkout for 20% Off 🎉 with Optery’s Thanksgiving Sale!
🍁 🦃
💡Page not loading? Optery’s Career page uses Cookies to display the full page content. If you’re not seeing anything, try opening the cookie banner (cookie icon in the bottom left corner) and Accept Personalization cookies.
Ready to
safeguard
your personal data?
Join the movement of people strengthening their privacy
I considered it a possibility. Now it's set in stone. Instead of fully shutting down in the coming year due to tumbling revenue, leadership decided "What if we use someone else's platform?" It just so happens,
the platform they chose is vibe coded
.
Like many tech companies during the pandemic, we over-hired and had to contract over and over again. Without the VC-funded war chest that our competitors had, we couldn't compete in marketing and sales. Our brand-awareness shrunk into obscurity.
So, in all fairness, we lost the capitalism game. And, I'm fine with that.
If you're curious, I'm sorry to disappoint. I haven't name-dropped, nor will I now or in the future.
We had a plan to gracefully wind down, unlike
Redbox
(
archived
). Once the balance hit a certain threshold, a plan (prepared a year in advance) would have made everyone whole and return the remaining funds to the investors.
Except, the investors changed their mind and would rather take a chance on a future sale than admit defeat.
What's changed their mind?
The allure and promise of AI workforce reduction.
The technology costs are but a single digit percentage of the monthly spend – the majority is tied to headcount and benefits. When I saw the numbers going towards headcount costs, I fully understood the situation we were in.
The previous reduction truly cut headcount to the bare minimum that can still keep the technology we have operating. Any fewer, and there's a high risk of business interruption within a few months.
At the same time, the current revenue projection calls for the end of the business within a few more months.
We used to have a thousand people. Today, I can count everyone on my hands. A cut beyond this will fundamentally need a different operating model.
Given that our revenue can no longer support the staff needed to run our own technology, how do the finances work on someone else's platform?
Assuming that this Software as a Service (SaaS) can deliver what leadership believes, the napkin math suggests it'll work out.
With this SaaS, they expect...
No engineering headcount
No implementation headcount
No support headcount
Contracted sales teams to pick up the rest
So if they're going to lay everyone off and migrate to a SaaS, who's going to do the migration?
I'll be on my own for an extra month or two to migrate it all over.
Somehow, I need to keep the tech coasting in its last days while migrating all the data that I can.
Thankfully, AWS is
not
a source of stress for me. Stuff still works, even if it complains years later.
I've expected either a winding down or a transition for over a year now. I've come to terms with an ending
like
this already.
While my peers are bitter about having a closer end date than me, I'm not as emotionally invested into
when
or
how
it ends.
What I didn't expect is how
a vibe coded app
passed as legitimate to the board of directors. We don't even have a contract with this platform yet and people are told they're being laid off.
How could a platform be that bad? This SaaS has no customers in the United States. Their team is based in another country without similar laws or regulations.
Even so, I'm confident that vibe coded platforms made by people in the United States also unknowingly violate state and federal laws around privacy, communications, and accessibility.
One of our tech acquisitions was through a bankruptcy fire sale after the original company could not make penalty payments for violating the Telephone Consumer Protection Act. These issues cannot be ignored to do business in the United States.
Things don't work
I've used LLM assisted auto complete. I've generated inline functionality. I've generated classes and modules. And I've generated entire web apps. I've seen what GPT, Claude, Z.ai GLM, Grok Code, and Gemeni do across the entire spectrum of software development.
Everyone has a different definition of "vibe coding", and as
Theo described the spectrum of its definitions
(at 4:30), I'll be using the slice of the spectrum "Ignoring the code entirely and only prompting" as my definition of vibe coding.
Within a minute, I could tell it was made with Claude or GLM. Every picture zooms in on hover for no reason. There are cards everywhere. Links go to
#
in the footer. Modals have an closing
X
button that doesn't work. The search button up top doesn't do anything...
It's like someone took some screenshots of a competitor, asked an LLM agent to create design documents around all of them, and then implement those design documents without any human review.
At the shallowest depth, I can see how a CEO got bamboozled. The happiest path is implemented. The second happiest path is rough. The third happiest path is unhinged.
No hacks. No reading the source code. Just innocent clicking around allowed me to break a critical invariant to running a business: I could place orders without giving my contact details or payment.
Besides displacing jobs
, issues like this concern me deeply.
LLM-generated code
can
enable a business process quicker and cheaper than hiring a full team with benefits. With the experts that still value their craft steering the development, software can be produced just as well as without these tools. Business processes meaningfully affect people's lives, whether staff, customer, vendor, or share-holder.
At its extreme with vibe coding
, LLM-generated code will have such poor quality that
it is
negligent
to use LLM-generated code
without
expert oversight and verification
. More lives are going to be affected by negligent software than ever before.
It is so much easier to accept that my life is changing because my employer couldn't stay fit in the economy than to accept it being displaced because of broken software made by a machine. The fiscal performance of my employer in this economy is the root cause, of course. And I accept that. Having to pivot everything to some broken SaaS that breaks the law? That's harder to accept.
While it is hard to accept, I'll still do my part and will move on after a job well done. How well the new platform operates after the domain swap is not my problem.
KDE Plasma 6.8 will be Wayland-only
Linux Weekly News
lwn.net
2025-11-26 16:49:45
KDE's Plasma team has announced
that KDE Plasma will drop X11 session support with Plasma 6.8:
The Plasma X11 session will be supported by KDE into early
2027.
We cannot provide a specific date, as we're exploring the
possibility of shipping some extra bug-fix releases for Plasma
6.7. The ex...
KDE's Plasma team has
announced
that KDE Plasma will drop X11 session support with Plasma 6.8:
The Plasma X11 session will be supported by KDE into early
2027.
We cannot provide a specific date, as we're exploring the
possibility of shipping some extra bug-fix releases for Plasma
6.7. The exact timing of the last one will only be known when we get
closer to its actual release, which we expect will be sometime in
early 2027.
What if I still really need X11?
This is a perfect use case for long term support (LTS)
distributions shipping older versions of Plasma. For example,
AlmaLinux 9 includes the Plasma X11 session and will be supported
until sometime in 2032.
See the blog post for information on running X11 applications
(still supported), accessibility, gaming, and more.
Yet again
, another global IT outage happen (deja vu strikes again in our
industry). This time at
cloudflare(
Prince 2025
). Again, taking down
large swats of the internet with
it(
Booth 2025
).
And yes, like my previous analysis of the GCP and CrowdStrike’s outages,
this post critiques Cloudflare’s root cause analysis (RCA), which —
despite providing a great overview of what happened — misses the real
lesson.
Here’s the key section of their RCA:
Unfortunately, there were assumptions made in the past, that the list of
columns returned by a query like this would only include the “default”
database:
SELECT
name,
type
FROM system.columns
WHERE
table = ‘http_requests_features’
order by name;
Note how the query does not filter for the database name. With us
gradually rolling out the explicit grants to users of a given ClickHouse
cluster, after the change at 11:05 the query above started returning
“duplicates” of columns because those were for underlying tables stored
in the r0 database.
This, unfortunately, was the type of query that was performed by the Bot
Management feature file generation logic to construct each input
“feature” for the file mentioned at the beginning of this section.
The query above would return a table of columns like the one displayed
(simplified example):
However, as part of the additional permissions that were granted to the
user, the response now contained all the metadata of the r0 schema
effectively more than doubling the rows in the response ultimately
affecting the number of rows (i.e. features) in the final file output.
A central database query didn’t have the right constraints to express
business rules. Not only it missed the database name, but it clearly
needs a distinct and a limit, since these seem to be crucial business
rules.
So, a new underlying security work manifested the (unintended) potential
already there in the query. Since this was by definition unintended, the
application code didn’t expect that value to be what it was, and reacted
poorly. This caused a crash loop across seemingly all of cloudflare’s
core systems. This bug wasn’t caught during rollout because the faulty
code path required data that was assumed to be impossible to be
generated.
Sounds familiar? It should. Any senior engineer has seen this pattern
before. This is classic database/application mismatch. With this in
mind, let’s review how Cloudflare is planning to prevent this from happening
again:
Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input
Enabling more global kill switches for features
Eliminating the ability for core dumps or other error reports to overwhelm system resources
Reviewing failure modes for error conditions across all core proxy modules
These are all solid, reasonable steps. But here’s the problem: they
already do most of this—and the outage happened anyway.
Why? Because of they seem to mistake physical replication with not
having a single point of failure. This mistakes the physical layer with
the logical layer. One can have a logical single point of failure
without having any physical one, which was the case in this
situation.
I base my paragraph on their choice of abandoning PostgreSQL and
adopting
ClickHouse(
Bocharov 2018
). The
whole post is a great overview on trying to process data fast, without a
single line on how to garantee its logical correctness/consistency in
the face of changes.
They are treating a logical problem as if it was a physical problem
I’ll repeat the
same advice
I offered in my previous article on GCP’s outage:
The real cause
These kinds of outages stem from the uncontrolled interaction between
application logic and database schema. You can’t reliably catch that
with more tests or rollouts or flags. You prevent it by
construction—through analytical design.
FAANG-style companies are unlikely to adopt formal methods or relational
rigor wholesale. But for their most critical systems, they should. It’s
the only way to make failures like this impossible by design, rather
than just less likely.
The internet would thank them. (Cloud users too—caveat emptor.)
References
Chapman, Roderick, Claire Dross, Stuart Matthews, and Yannick Moy. 2024. “Co-Developing Programs and Their Proof of Correctness.”
Commun. Acm
67 (3): 84–94.
https://doi.org/10.1145/3624728
.
Figure 1:
The Cluny library was one of the richest and most important in France and Europe. In 1790 during the French Revolution, the abbey was sacked and mostly destroyed, with only a small part surviving
London councils enact emergency plans after three hit by cyber-attack
Guardian
www.theguardian.com
2025-11-26 16:34:05
Kensington and Westminster councils investigating whether data has been compromised as Hammersmith and Fulham also reports hack Three London councils have reported a cyber-attack, prompting the rollout of emergency plans and the involvement of the National Crime Agency (NCA) as they investigate whet...
Three
London
councils have reported a cyber-attack, prompting the rollout of emergency plans and the involvement of the National Crime Agency (NCA) as they investigate whether any data has been compromised.
The Royal Borough of Kensington and Chelsea (RBKC), and Westminster city council, which share some IT infrastructure, said a number of systems had been affected across both authorities, including phone lines. The councils shut down several computerised systems as a precaution to limit further possible damage.
The Information Commissioner’s Office (ICO) said the London Borough of Hammersmith and Fulham had also reported an attack. Together the three authorities provide services for more than half a million Londoners.
RBKC said it had established the cause of the cyber incident on Wednesday but declined to give further details, citing the involvement of the NCA and National Cyber Security Centre (NCSC), part of GCHQ.
In 2020 a ransomware attack on Hackney council accessed and encrypted 440,000 files, resulting in a reprimand from the ICO.
Engineers at RBKC worked through the night on Monday, when the incident occurred, and Tuesday. Services including checking council tax bills and paying parking fines are likely to be limited at RBKC, which said its website would probably go up and down during Wednesday as security fixes progressed.
In a
statement, the council said
: “We don’t have all the answers yet, as the management of this incident is still ongoing. But we know people will have concerns, so we will be updating residents and partners further over the coming days. At this stage it is too early to say who did this, and why, but we are investigating to see if any data has been compromised – which is standard practice.”
It said it and Westminster had been working with specialist cyber-incident experts and the NCSC, “with the focus on protecting systems and data, restoring systems, and maintaining critical services to the public”.
The boroughs also share some IT systems with the London borough of Hammersmith and Fulham. It was not immediately clear to what extent that borough had been affected.
RBKC said it had “invoked business continuity and emergency plans to ensure we are still delivering critical services to residents, focusing on supporting the most vulnerable”.
Westminster city council
said in a statement
: “We apologise to residents for any inconvenience and thank them for being flexible and understanding. People may see some delays in responses and the services we provide over the coming days. We will continue working with our cyber specialists and the NCSC to restore all systems as quickly as possible, and we will be in touch with more information as it becomes available. If there are any further changes to services, we endeavour to keep everyone updated.”
The incident, which was spotted on Monday morning, led to concern at other councils. Hackney in east London, which was not affected but saw land searches, housing and planning services affected in 2020, told staff it had “received intelligence that multiple London councils have been targeted by cyber-attacks within the last 24-48 hours, with potential disruption to systems and services”.
Rob Miller, a former IT director at Hackney council and now a senior director at the consultancy Public Digital, said: “If you want to impact on people’s lives, councils are a good target. [When it happens] it feels like your stomach has dropped out and quickly it becomes apparent how hard it is to get things back. It’s a genuinely traumatic experience.”
European parliament calls for social media ban on under-16s
Guardian
www.theguardian.com
2025-11-26 16:28:31
MEPs pass resolution to help parents tackle growing dangers of addictive internet platforms Children under 16 should be banned from using social media unless their parents decide otherwise, the European parliament says. MEPs passed a resolution on age restrictions on Wednesday by a large majority. ...
Children under 16 should be banned from using social media unless their parents decide otherwise, the European parliament says.
MEPs passed a resolution on age restrictions on Wednesday by a large majority. Although not legally binding, it raises pressure for European legislation amid growing alarm about the
mental health
risks to children of unfettered internet access.
The European Commission, which is responsible for initiating EU law, is already studying Australia’s
world-first social-media ban
for under-16s, which is due to take effect next month.
In a speech in September, the commission’s president,
Ursula von der Leyen
, said she would watch the implementation of Australia’s policy. She spoke out against “algorithms that prey on children’s vulnerabilities with the explicit purpose of creating addictions” and said parents felt powerless against “the tsunami of big tech flooding their homes”.
Von der Leyen promised a panel of experts would be set up by the end of the year to advise on the best approach to protecting children.
Interest is growing in restricting children’s social media and smartphone access. An expert report commissioned last year by France’s president, Emmanuel Macron, said children should not be allowed to use smartphones
until the age of 13
and social media, such as TikTok, Instagram and Snapchat, until they were 18.
Christel Schaldemose, the Danish Social Democrat MEP who drafted the resolution, told reporters that politicians needed to act to protect children: “It is not just parents. Society also needs to step up and make sure that platforms are a safe place for minors to be, but only if they are above a certain age.”
Her report called for the default disabling of addictive features on internet platforms when used by minors, such as infinite scrolling (endless content as the user scrolls down), videos that automatically play, excessive push notifications and rewards for repeated use of a site.
The resolution noted that “addictive design features are often inherent to the business model of platforms, notably social media”. An earlier draft of the Schaldemose report cited
a study
stating that one in four children and young people displayed “problematic” or “dysfunctional” smartphone use – behavioural patterns mirroring addiction. The resolution said children should be 16 before they could access social media, although parents could give consent from the age of 13.
The White House is urging the EU to roll back its digital laws and some supporters of a social media ban explicitly framed the vote in this context. At a meeting in Brussels on Monday, Howard Lutnick, the US commerce secretary, said EU rules on tech companies needed to be more “balanced” in exchange for lower US steel and aluminium tariffs.
Referring to Lutnick’s visit, Stéphanie Yon-Courtin, a French MEP from Macron’s party, said Europe was not “a regulatory colony”. In a statement after the vote, she added: “Our digital laws are not for sale. We will not back down on children’s protections because a foreign billionaire or big tech tells us to.”
The EU already seeks to protect internet users from online harms, such as disinformation, cyberbullying and illegal content, via its Digital Services Act. But the resolution said this law had gaps and could do more to protect children from addictive design features and online exploitation, such as financial incentives to become influencers.
Schaldemose said the act, which she co-authored, was strong “but we could go further, especially in areas of addictive design features and harmful dark pattern practices where we are not so specific, not so precise”.
Dark patterns refer to app or website design features to influence decision-making, such as countdown timers to encourage users to make purchases, or nagging requests to turn on location trackers and notifications.
Schaldemose’s resolution was adopted by 483 MEPs and opposed by 92, with 86 abstentions.
Eurosceptic MEPs criticised the plan, saying the EU would be overreaching if it banned social media access for children. “Decisions about children’s access must be taken as close to families as possible – in the member states, not in Brussels,” said Kosma Złotowski, a Polish member of the European Conservatives and Reformists group.
The resolution was passed only one week after the
commission announced delays
to changes to its Artificial Intelligence Act and other digital laws in a push to lighten regulation on companies in the name of “simplification”.
Schaldemose said she appreciated the need to avoid creating too many laws but added “there is a willingness to do more when it comes to kids and protection of our children in the EU”.
"Flash crowd" redirects here. For the short story by Larry Niven, see
Flash Crowd
. For the social gathering in the real world, see
Flash mob
.
The
Slashdot effect
, also known as
slashdotting
or the
hug of death
occurs when a popular
website
links to a smaller website, causing a massive increase in traffic. This
overloads
the smaller site, causing it to slow down or even temporarily become unavailable. Typically, less robust sites are unable to cope with the huge increase in traffic and become unavailable – common causes are lack of sufficient
data bandwidth
,
servers
that fail to cope with the high number of requests, and traffic
quotas
. Sites that are maintained on
shared hosting
services often fail when confronted with the Slashdot effect. This has the same effect as a
denial-of-service attack
, albeit accidentally. The name stems from the huge influx of
web traffic
which would result from the technology news site
Slashdot
linking to websites. The term
flash crowd
is a more generic term.
[
1
]
The original circumstances have changed, as flash crowds from
Slashdot
were reported in 2005 to be diminishing due to competition from
similar sites
,
[
2
]
and the general adoption of elastically scalable cloud hosting platforms.
The term "Slashdot effect" refers to the phenomenon of a website becoming virtually unreachable because too many people are hitting it after the site was mentioned in an interesting article on the popular Slashdot news service. It was later extended to describe any similar effect from being listed on a popular site.
[
3
]
Sites such as
Slashdot
, Digg, Reddit, StumbleUpon, and Fark consist of brief submitted stories and a self-moderated discussion on each story. The typical submission introduces a news item or website of interest by
linking
to it. In response, large masses of readers tend to simultaneously rush to view the referenced sites. The ensuing flood of page requests from readers can exceed the site's available bandwidth or the ability of its servers to respond, and render the site temporarily unreachable.
Google Doodles
, which link to search results on the doodle topic, also result in high increases of traffic from the search results page.
[
7
]
MRTG
graph from a web server statistics generator showing a moderate
Slashdot
effect in action in 2005
Major news sites or corporate websites are typically engineered to serve large numbers of requests and therefore do not normally exhibit this effect. Websites that fall victim may be hosted on home servers, offer large images or movie files or have inefficiently generated dynamic content (e.g. many database hits for every web hit even if all web hits are requesting the same page). These websites often became unavailable within a few minutes of a story's appearance, even before any comments had been posted. Occasionally, paying
Slashdot
subscribers (who have access to stories before non-paying users) rendered a site unavailable even before the story was posted for the general readership.
Few definitive numbers exist regarding the precise magnitude of the
Slashdot
effect, but estimates put the peak of the mass influx of page requests at anywhere from several hundred to several thousand hits per minute.
[
8
]
[
9
]
[
10
]
The flood usually peaked when the article was at the top of the site's front page and gradually subsided as the story was superseded by newer items. Traffic usually remained at elevated levels until the article was pushed off the front page, which could take from 12 to 18 hours after its initial posting. However, some articles had significantly longer lifetimes due to the popularity, newsworthiness, or interest in the linked article.
By 2005, reporters were commenting that the
Slashdot
effect had been diminishing.
[
2
]
However, the effect has been seen involving Twitter when some popular users mention a website.
[
11
]
When the targeted website has a
community
-based structure, the term can also refer to the secondary effect of having a large group of new users suddenly set up accounts and start to participate in the community. While in some cases this has been considered a good thing, in others it is viewed with disdain by the prior members, as quite often the sheer number of new people brings many of the unwanted aspects of
Slashdot
along with it, such as
trolling
,
vandalism
, and
newbie
-like behavior. This bears some similarity to the 1990s Usenet concept of
Eternal September
.
Many solutions have been proposed for sites to deal with the Slashdot effect.
[
12
]
There are several systems that automatically mirror any Slashdot-linked pages to ensure that the content remains available even if the original site becomes unresponsive.
[
13
]
Sites in the process of being Slashdotted may be able to mitigate the effect by temporarily redirecting requests for the targeted pages to one of these mirrors. Slashdot does not
mirror
the sites it links to on its own servers, nor does it endorse a third party solution. Mirroring of content may constitute a breach of
copyright
and, in many cases, cause ad revenue to be lost for the targeted site.
^
Ari, Ismail; Hong, Bo; Miller, Ethan L.; Brandt, Scott A.; Long, Darrell D. E. (October 2003).
"Managing Flash Crowds on the Internet"
(PDF)
. University of California Santa Cruz Storage Systems Research Center. Archived from
the original
(PDF)
on 9 May 2013
. Retrieved
15 March
2010
.
^
"Slashdotting graphs"
. Princeton University Department of Astrophysical Sciences. Archived from
the original
on 27 February 2009
. Retrieved
13 January
2004
.
Bits from Debian: New Debian Developers and Maintainers (September and October 2025)
PlanetDebian
bits.debian.org
2025-11-26 16:00:00
The following contributors got their Debian Developer accounts in the last two
months:
Evangelos Ribeiro Tzaras (devrts)
Andrea Bolognani (abologna)
The following contributors were added as Debian Maintainers in the last two
months:
Rylie Pavlik
Yuchin Tsai
Daniel Markstedt
Guido Berhörster
Renzo...
Every systems engineer at some point in their journey yearns to write
a filesystem. This sounds daunting at first - and writing a battle-tested filesystem
is
hard - but the minimal surface area for a “working” FS is surprisingly small, simple, and in-distribution for coding agents.
In fact, one of my smoke tests for new coding models is seeing how good of
a filesystem they can one-shot! At some point, I had quite a few filesystems lying around - and coding models were getting pretty good - which made me wonder if the models were intelligent enough to actually model the filesystem engine itself?
A filesystem is the perfect black-box API to model with wacky backends (see
“Harder drives”
), and besides the joy of training an LLM for fun - there were a few deeper truths about language models that I wanted to explore.
So I set upon training a filesystem. Building on top of one of my throwaway
FUSEs, a few rounds with Claude repurposed it to loopback against the host
with added logging, two things I needed to generate reference fine-tuning data:
classLoggingLoopbackFS(LoggingMixIn, Operations):
"""
A loopback FUSE filesystem that logs all operations for training data.
This implementation delegates all filesystem operations to a real directory
on the host filesystem, ensuring perfect semantic correctness while logging
every operation for LLM training data.
"""
I then wrote a filesystem interaction simulator, which sampled various
operations against a sandboxed
LoggingLoopbackFS
to generate diverse FUSE
prompt/completion pairs. Concretely, I captured only the minimal set of
operations needed
for
R/W-ish capability (no open, xattrs, fsync etc).
Alongside the FUSE operation, I captured the full filesystem state at every
turn. I experimented with various formats, including an ASCII-art
representation, but ultimately settled on XML since it enforces prompt
boundaries clearly and had canonical parsers available.
With prompts including the FUSE operation + XML filesystem tree, the model learned two forms of completions:
Reads (<R>) requested the content / metadata as per the operation
(
getattr
/
readdir
/
read
)
Writes (<W>) requested the model to output the full filesystem tree state,
after modification (
unlink
/
chmod
/
truncate
/
write
)
Once I had clean, representative, and diverse filesystem simulation data, actually running SFT was pretty straightforward on Modal. Over a few iteration cycles spread across nibbles of spare time, I ended up with ~98% accuracy on a hold-out eval after 8 epochs of SFT on a N=15000 dataset with Qwen3-4b.
Most of my time here was spent cleaning generated data and ensuring we represented every FUSE operation sufficiently + generated enough “complex” trees to learn on.
At this point, I wrote … possibly the smallest filesystem I’ve seen… to give my model a spin in
the real world. Every FUSE operation was a passthrough to the LLM, for example:
Nice! I now had a mountable FUSE that was entirely “implemented” by a language
model. As you can see below, I was able to
ls
around it,
echo
into files, and
cat
them back out.
Poking around a Docker container with a mounted LLMFuse.
Perhaps the largest glaring inefficiency in this set up is the sheer verbosity
of the XML-based representation. I was using many bytes to represent attributes
and tree structure that could be encoded far more efficiently (~O(bits)) in a standard
C struct.
However, as I was fine-tuning on the XML filesystem tree representation, I was
baking in this very structure into the weights and probability distributions of my Qwen fork! If only there was a way to leverage this to compress state…
As it turns out, compression and AI are intimately related. Using LLMs to lossily
compress text is one of the most common applications, so it’s not entirely
unintuitive. However, one researcher (Marcus Hutter) claimed back in 2006 that they are
equivalent
(and in fact
bet $500K on this claim!
).
Presciently, Hutter appears to be absolutely right. His
enwik8
and
enwik9
’s benchmark datasets are, today, best compressed by a
169M parameter LLM
(trained by none other than Fabrice Bellard in 2023).
That’s a bit perplexing on the first glance. Surely LLM compression isn’t reversible? What kind of voodoo magic was going on here?
The algorithm that enables reversible compression using LLMs is called “arithmetic coding” and it builds upon a
1948 result by Claude Shannon
.
Researchers at DeepMind (including Hutter himself) have
explained the math in
detail
, so I’ll direct the most inquisitive of you readers there, but for a simplified understanding of what’s going on, forget everything you might know about working with LLMs today. There’s no prompting involved!
Let’s assume the following is true for some predictive model \(M\)
Lorem has first-word probability = 0.57.
Ipsum has second-word conditional probability = 0.67 (joint 0.38).
Dolor has a third word conditional probability = 0.5 (joint 0.19).
…
so on and so forth until you reach the end of the string you want to compress and you end up with some “final interval width” \(P(m)\) on the real interval \([0,1]\) which represents your string.
Let’s suppose in our example this turns out to be 0.012. We can represent this decimal in roughly \(- \log_{2}{P(m)} = 6.4\) bits, which is our final compression size.
There’s a few elegant things about this algorithm:
Any
number within this interval is uniquely determined by tracing the arithmetic coding algorithm through the specific probabilistic model’s weights. “Decoding” is simply a retracing operation (see the line through the probability distributions above)
The inverse log relationship between predictive power \(P(m)\) and compression
pushes the burden of the “hard compression problem” to deep learning machinery which can encode high-dimensional text patterns within model weights, yielding far better compression ratios than deterministic algorithms.
Sounds cool! But
how good really
is this compression? On comparing
arithmetic coding backed by
Qwen3-4B
against
gzip
for
lipsum.txt
,
we already see pretty dramatic results:
Method
Size (bytes)
Compression Impact
Original (plain)
446
—
gzip
298
~33% smaller
llmencode
13
~97% smaller
(note:
llmencode
is my implementation of arithmetic coding)
22x better compression than
gzip
is pretty ridiculous! A caveat here is that
lipsum.txt
is heavily represented in training data, but 5-20x efficiency gains broadly hold for all text data that (looks like) it’s been on the internet.
Now, back to our filesystem. The XML overhead we were worried about now can be
“compressed away” by the fine-tuned model. Using the same toy filesystem from
the Docker container demo above:
The fine-tuned model achieves
44.7% better compression
on XML filesystem
trees - the very format it was trained to predict. This is the “self-compression”
effect: by baking the XML structure into the model weights during fine-tuning,
the arithmetic coder can represent that structure in fewer bits.
Self-compression in filesystems isn’t a novel concept. For example, there exists the
squashfs
tool (created in 2002) to create R/O compressed filesystems. Squashfs compresses
files, inodes, and directories together, not unlike what we’re doing here!
Under the hood,
squashfs
just wraps
gzip
/
zstd
/your favourite compression
algorithm. So for plain-text data,
squashfs
compression stats pale in the face of
llmfuse
:
For the same filesystem tree (one directory, one 14-byte text file), llmfuse
achieves
~8x better compression
than squashfs (see methodology in appendix).
The difference comes down to
llmencode
being far better than
gzip
on
text data + XML structure - especially when the model has been fine-tuned on exactly
that structure.
What started off as a little experiment mostly to get my hands dirty with
training and inference evolved into a full blown
nerd
snipe
and intellectual adventure. Thanks for making it
this far!
I entirely recognize that this is a “toy”
experiment under a very specific setup; with that said, the numbers above are pretty eye-popping, and the question I’ve been trying to answer as I write this up is: does this have any real-world potential?
Of course, in the short term, there’s a whole host of caveats: you need an
LLM, likely a GPU, all your data is in the context window (which we know scales
poorly), and this only works on text data.
Still, it’s intriguing to wonder whether the very engines that will likely
dominate all “text generation” going forward can be used to compress their own
data? Perhaps in a distant future, where running LLMs at the edge makes sense, or for specific kinds of workflows where data is read very infrequently.
Overall, I’m grateful to Peyton at
Modal
for the compute credits. Running
a somewhat unconventional experiment like this wouldn’t have been possible
without full control over the training and inference code, and extremely
tedious without the simplicity of running ML infra on Modal! It’s truly awesome
to be able to just
modal deploy
and get my own private inference endpoints,
or just
modal run
to prototype some code on the cloud.
All of the source code for this experiment, particularly
llmfuse
and
llmencode
are
open-sourced
under MIT.
llmencode
is abstracted into a CLI utility that you can run locally.
Inference on 4B models is slow, but entirely possible on consumer hardware.
I prototyped most of this code by running on a 2021 MacBook Pro, before
productionizing on Modal.
A fun experiment / party trick to identify how “common” a certain
string is in training data is to look at its
llmencode
compression ratio!
The raw
.sqsh
file is 4096 bytes due to block alignment padding. To find the
actual compressed size, I used
xxd
to inspect the binary and found the last
non-zero byte at offset 266 (267 bytes total). Subtracting the fixed 96-byte
superblock header gives us 171 bytes of actual gzip-compressed content -
everything needed to reconstruct the filesystem.
It’s equally interesting to think about compression as a metric. An angle I’d
considered is doing some kind of RL on the arithmetic coded compression number itself.
Is that simply equivalent to the pre-training objective (due to the prediction-compression duality)? Or does the “sequence-level” objective add something more… interesting to the mix. Please reach out if you have thoughts!
From blood sugar to brain relief: GLP-1 therapy slashes migraine frequency
Notice: News releases are not subject to review by
MedLink Neurology
’s Editorial Board.
Researchers at the Headache Centre of the
University of Naples “Federico II” gave the glucagon-like peptide-1
(GLP-1) receptor agonist liraglutide to 26 adults with obesity and
chronic migraine (defined as 15 or more headache days per month). Patients
reported an average of 11 fewer headache days per month, while
disability scores on the Migraine Disability Assessment Test dropped by
35 points, indicating a clinically meaningful improvement in work,
study, and social functioning.
GLP-1 agonists have gained recent
widespread attention, reshaping treatment approaches for several
diseases, including diabetes and cardiovascular disease.
2
In
the treatment of type 2 diabetes, liraglutide helps lower blood sugar
levels and reduce body weight by suppressing appetite and reducing
energy intake.
3,4,5
Importantly, while participants’ body mass
index declined slightly (from 34.01 to 33.65), this change was not
statistically significant. An analysis of covariance confirmed that BMI
reduction had no effect on headache frequency, strengthening the
hypothesis that pressure modulation, not weight loss, drives the
benefit.
“Most patients felt better within the
first two weeks and reported quality of life improved significantly”,
said lead researcher Dr Simone Braca. “The benefit lasted for the full
three-month observation period, even though weight loss was modest and
statistically non-significant.”
Patients were screened to exclude
papilledema (optic disc swelling resulting from increased intracranial
pressure) and sixth nerve palsy, ruling out idiopathic intracranial
hypertension (IIH) as a confounding factor. Growing evidence closely
links subtle increases in intracranial pressure to migraine attacks.
6
GLP-1-receptor agonists, such as liraglutide, reduce cerebrospinal fluid
secretion and have already proved effective in treating IIH.
7
Therefore,
building on these observations, Dr Braca and colleagues hypothesised
that exploiting the same mechanism of action might ultimately dampen
cortical and trigeminal sensitisation that underlie migraine.
“We think that, by modulating
cerebrospinal fluid pressure and reducing intracranial venous sinuses
compression, these drugs produce a decrease in the release of calcitonin
gene-related peptide (CGRP), a key migraine-promoting peptide”, Dr
Braca explained. “That would pose intracranial pressure control as a
brand-new, pharmacologically targetable pathway.”
Mild gastrointestinal side effects (mainly
nausea and constipation) occurred in 38% of participants but did not
lead to treatment discontinuation.
Following this exploratory 12-week pilot
study, a randomised, double-blind trial with direct or indirect
intracranial pressure measurement is now being planned by the same
research team in Naples, led by Professor Roberto De Simone. “We also
want to determine whether other GLP-1 drugs can deliver the same relief,
possibly with even fewer gastrointestinal side effects”, Dr Braca
noted.
If confirmed, GLP-1-receptor agonists
could offer a new treatment option for the estimated one in seven people
worldwide who live with migraine,
8
particularly those who do
not respond to current preventives. Given liraglutide’s established use
in type 2 diabetes and obesity, it may represent a promising case of
drug repurposing in neurology.
References:
Braca S, Russo C, et al.
GLP-1R Agonists for the Treatment of Migraine: A Pilot Prospective Observational Study.
Abstract A-25-13975. Presented at the 11
th
EAN Congress (Helsinki, Finland).
Zheng Z, Zong Y, Ma Y, et al. Glucagon-like peptide-1 receptor: mechanisms and advances in therapy. Signal Transduct Target Ther 2024;9(1):234.
Lin CH, Shao L, Zhang YM, et al. An evaluation of liraglutide including its efficacy and safety for the treatment of obesity. Expert Opin Pharmacother 2020;21(3):275-85.
Moon S, Lee J, Chung HS, et al. Efficacy and safety of the new appetite suppressant, liraglutide: a meta-analysis of randomized controlled trials. Endocrinol Metab (Seoul) 2021;36(3):647-60.
Jacobsen LV, Flint A, Olsen AK, Ingwersen SH. Liraglutide in type 2 diabetes mellitus: clinical pharmacokinetics and pharmacodynamics. Clin Pharmacokinet 2016;55(6):657-72.
De Simone R, Sansone M, Russo C, Miele A, Stornaiuolo A, Braca S.
The putative role of trigemino-vascular system in brain perfusion
homeostasis and the significance of the migraine attack. Neurol Sci 2022;43(9):5665-72.
Mitchell JL, Lyons HS, Walker JK, et al. The effect of GLP-1RA exenatide on idiopathic intracranial hypertension: a randomized clinical trial. Brain 2023;146(5):1821-30.
Steiner TJ, Stovner LJ, Jensen R, Uluduz D, Katsarava Z. Lifting The Burden: the Global Campaign against Headache. Migraine remains second among the world's causes of disability, and
first among young women: findings from GBD2019. J Headache Pain 2020;21(1):137.
It’s hard to believe it’s only been a few years since generative AI tools started flooding the internet with low quality content-slop. Just over a year ago, you’d have to peruse certain corners of Facebook or spend time wading through the cultural cesspool of Elon Musk’s X to find people posting bizarre and repulsive synthetic media. Now, AI slop feels inescapable — whether you’re
watching TV
, reading
the news
, or
trying to find a new apartment
.
That is, unless you’re using
Slop Evader
, a new browser tool that filters your web searches to only include results from before November 30, 2022 — the day that ChatGPT was released to the public.
The tool is available for Firefox and Chrome, and has one simple function: Showing you the web as it was before the deluge of AI-generated garbage. It uses Google search functions to index popular websites and filter results based on publication date, a scorched earth approach that virtually guarantees your searches will be slop-free.
Slop Evader was created by artist and researcher
Tega Brain
, who says she was motivated by the growing dismay over the tech industry’s unrelenting, aggressive rollout of so-called “generative AI”—despite widespread criticism and the wider public’s distaste for it.
Slop Evader in action. Via Tega Brain
“This sowing of mistrust in our relationship with media is a huge thing, a huge effect of this synthetic media moment we’re in,” Brain told 404 Media, describing how tools like Sora 2 have short-circuited our ability to determine reality within a sea of artificial online junk. “I’ve been thinking about ways to refuse it, and the simplest, dumbest way to do that is to only search before 2022.”
One under-discussed impact of AI slop and synthetic media, says Brain, is how it increases our “cognitive load” when viewing anything online. When we can no longer immediately assume any of the media we encounter was made by a human, the act of using social media or browsing the web is transformed into a never-ending procession of
existential double-takes
.
This cognitive dissonance extends to everyday tasks that require us to use the internet—which is practically everything nowadays. Looking for a house or apartment? Companies are using genAI tools to
generate pictures of houses and rental properties
, as well as the ads themselves. Trying to sell your old junk on Facebook Marketplace? Meta’s embrace of generative AI means you may have to compete with bots, fake photos, and
AI-generated listings
. And when we shop for beauty products or view ads, synthetic media tools are taking our filtered and impossibly-idealized beauty standards to
absurd and disturbing new places
.
In all of these cases, generative AI tools further thumb the scales of power—saving companies money while placing a higher cognitive burden on regular people to determine what’s real and what’s not.
“I open up Pinterest and suddenly notice that half of my feed are these incredibly idealized faces of women that are clearly not real people,” said Brain. “It’s shoved into your face and into your feed, whether you searched for it or not.”
Currently, Slop Evader can be used to search pre-GPT archives of seven different sites where slop has become commonplace, including YouTube, Reddit, Stack Exchange, and the parenting site MumsNet. The obvious downside to this, from a user perspective, is that you won’t be able to find anything time-sensitive or current—including this very website, which did not exist in 2022. The experience is simultaneously refreshing and harrowing, allowing you to browse freely without having to constantly question reality, but always knowing that this freedom will be forever locked in time—nostalgia for a human-centric world wide web that
no longer exists
.
Of course, the tool’s limitations are part of its provocation. Brain says she has plans to add support for more sites, and release a new version that uses DuckDuckGo’s search indexing instead of Google’s. But the real goal, she says, is prompting people to question how they can collectively refuse the dystopian, inhuman version of the internet that Silicon Valley’s AI-pushers have forced on us.
“I don’t think browser add-ons are gonna save us,” said Brain. “For me, the purpose of doing this work is mostly to act as a provocation and give people examples of how you can refuse this stuff, to furnish one’s imaginary for what a politics of refusal could look like.”
With enough cultural pushback, Brain suggests, we could start to see alternative search engines like DuckDuckGo adding options to filter out search results suspected of having synthetic content (DuckDuckGo added the
ability to filter out
AI images in search earlier this year). There’s also been a growing movement
pushing back against the new AI data centers
threatening to pollute communities and
raise residents’ electricity bills
. But no matter what form AI slop-refusal takes, it will need to be a group effort.
“It’s like with the climate debate, we’re not going to get out of this shitshow with individual actions alone,” she added. “I think that’s the million dollar question, is what is the relationship between this kind of individual empowerment work and collective pushback.”
KDE Plasma 6.8 Will Go Wayland-Exclusive in Dropping X11 Session Support
KDE developers announced they are going "all-in on a Wayland future" and with the Plasma 6.8 desktop it will become Wayland-exclusive. The Plasma X11 session is going away.
KDE developers announced with Plasma 6.8 it will be Wayland-exclusive in removing Plasma X11 session support although continuing to support X11 apps/games via XWayland.
KDE developers report that "the vast majority of our users are already using the Wayland session" and longer-term this change will allow for new features, optimizations, and more development speed with foregoing X11 session support.
With the Plasma release timing, this means Plasma X11 session support will remain supported into early 2027 with the Plasma 6.7 series. The Plasma 6.7 release may end up seeing some extra bug-fix releases for X11 holdouts.
More details on Plasma 6.8 going Wayland-exclusive and other details via the
KDE.org blog
.
Chinatown's 'Secret' Sandwich Window Gets a Nifty New Dining Room
hellgate
hellgatenyc.com
2025-11-26 15:37:47
The Sandwich Board has a muffaletta, as well as chicken, duck, and breakfast sandwiches, and now you can even sit inside while you eat them....
Michael Brafman was born and raised in Brooklyn, currently lives in Peter Cooper Village, and has been a professional chef in NYC for more than 30 years. He clocked his first kitchen job when he was 17, and has bounced between fancy places (like Jean-Georges and Gramercy Tavern) and corporate dining gigs (where the hours are
so
much more conducive to raising a family) ever since.
During an unemployment stint a couple of years ago though, Brafman was helping a buddy devise a sandwich menu for Solely Tea on Eldridge Street, when something clicked. "I'm just going to make all the stuff that inspires me," he remembers thinking. "There's no boundaries! To me, the most important thing is, I don't want to limit my inspiration to just making versions of other, existing sandwiches. It's more like, I look at plated food that I like, and try to translate those not-sandwich dishes into sandwiches."
Brafman's vision proved to be too mighty for the tea shop, so instead he opened his own place in September of 2024, a simple, semi-discreet ordering window just a few doors down from Solely called
the Sandwich Board
. "Our whole goal was to become a local, neighborhood staple," he said and, if you spend even a few minutes with Brafman on Eldridge, it's clear that he's succeeded. During our brief chat on the sidewalk outside the shop at least a half dozen people walking by gave Brafman a wave, or a fist-bump, or a "say hi to the family." He has strong "mayor-of-the-block" vibes, for sure.
Thing is though, for most of the past year the Sandwich Board didn't provide us non-locals with anywhere to eat. Yes, there were four chairs set up guerilla-style on the sidewalk, which was great when the weather was pleasant, but much less appealing in February and March. So when the folks running the Forever Mart snacks-and-curios shop in the adjacent space called it quits, Brafman knew the time had come to expand. A few weeks ago he unveiled his new dining room, complete with stools, high tops and counters, and a second, indoor ordering window.
Massachusetts Institute of Technology on Wednesday released a study that found that
artificial intelligence
can already replace 11.7% of the U.S. labor market, or as much as $1.2 trillion in wages across finance, health care and professional services.
The study was conducted using a labor simulation tool called the
Iceberg Index
, which was created by MIT and Oak Ridge National Laboratory. The index simulates how 151 million U.S. workers interact across the country and how they are affected by AI and corresponding policy.
The Iceberg Index, which was
announced
earlier this year, offers a forward-looking view of how AI may reshape the
labor market
, not just in coastal tech hubs but across every state in the country. For lawmakers preparing billion-dollar reskilling and training investments, the index offers a detailed map of where disruption is forming down to the zip code.
"Basically, we are creating a digital twin for the U.S. labor market," said Prasanna Balaprakash, ORNL director and co-leader of the research. ORNL is a Department of Energy research center in eastern Tennessee, home to the Frontier supercomputer, which powers many large-scale modeling efforts.
The index runs population-level experiments, revealing how AI reshapes tasks, skills and labor flows long before those changes show up in the real economy, Balaprakash said.
The index treats the 151 million workers as individual agents, each tagged with skills, tasks, occupation and location. It maps more than 32,000 skills across 923 occupations in 3,000 counties, then measures where current AI systems can already perform those skills.
What the researchers found is that the visible tip of the iceberg — the
layoffs
and role shifts in tech, computing and information technology — represents just 2.2% of total wage exposure, or about $211 billion. Beneath the surface lies the total exposure, the $1.2 trillion in wages, and that includes routine functions in human resources, logistics, finance, and office administration. Those are areas sometimes overlooked in automation forecasts.
The index is not a prediction engine about exactly when or where jobs will be lost, the researchers said. Instead, it's meant to give a skills-centered snapshot of what today's AI systems can already do, and give policymakers a structured way to explore what-if scenarios before they commit real money and legislation.
The researchers partnered with state governments to run proactive simulations. Tennessee, North Carolina and Utah helped validate the model using their own labor data and have begun building policy scenarios using the platform.
Tennessee moved first, citing the Iceberg Index in its official
AI Workforce Action Plan
released this month. Utah state leaders are preparing to release a similar report based on Iceberg's modeling.
North Carolina state Sen. DeAndrea Salvador, who has worked closely with MIT on the project, said what drew her to the research is how it surfaces effects that traditional tools miss. She added that one of the most useful features is the ability to drill down to local detail.
"One of the things that you can go down to is county-specific data to essentially say, within a certain census block, here are the skills that is currently happening now and then matching those skills with what are the likelihood of them being automated or augmented, and what could that mean in terms of the shifts in the state's GDP in that area, but also in employment," she said.
Salvador said that kind of simulation work is especially valuable as states stand up overlapping AI task forces and working groups.
The Iceberg Index also challenges a common assumption about AI risk — that it will stay confined to tech roles in coastal hubs. The index's simulations show exposed occupations spread across all 50 states, including inland and rural regions that are often left out of the AI conversation.
To address that gap, the Iceberg team has built an interactive simulation environment that allows states to experiment with different policy levers — from shifting workforce dollars and tweaking training programs to exploring how changes in technology adoption might affect local employment and gross domestic product.
"Project Iceberg enables policymakers and business leaders to identify exposure hotspots, prioritize training and infrastructure investments, and test interventions before committing billions to implementation," the report says.
Balaprakash, who also serves on the Tennessee Artificial Intelligence Advisory Council, shared state-specific findings with the governor's team and the state's AI director. He said many of Tennessee's core sectors — health care, nuclear energy, manufacturing and transportation — still depend heavily on physical work, which offers some insulation from purely digital automation. The question, he said, is how to use new technologies such as robotics and AI assistants to strengthen those industries rather than hollow them out.
For now, the team is positioning Iceberg not as a finished product but as a sandbox that states can use to prepare for AI's impact on their workforces.
"It is really aimed towards getting in and starting to try out different scenarios," Salvador said.
ChatGPT firm blames boy’s suicide on ‘misuse’ of its technology
Guardian
www.theguardian.com
2025-11-26 15:31:58
OpenAI responds to lawsuit claiming its chatbot encouraged California teenager to kill himself The maker of ChatGPT has said the suicide of a 16-year-old was down to his “misuse” of its system and was “not caused” by the chatbot. The comments came in OpenAI’s response to a lawsuit filed against the ...
The maker of
ChatGPT
has said the suicide of a 16-year-old was down to his “misuse” of its system and was “not caused” by the chatbot.
The comments came in OpenAI’s response to a lawsuit filed against the San Francisco company and its chief executive, Sam Altman, by the family of California teenager Adam Raine.
Raine
killed himself in April
after extensive conversations and “months of encouragement from ChatGPT”, the family’s lawyer has said.
The lawsuit alleges the teenager discussed a method of suicide with ChatGPT on several occasions, that it guided him on whether a suggested method would work, offered to help him write a suicide note to his parents and that the version of the technology he used was “rushed to market … despite clear safety issues”.
According to filings at the superior court of the state of California on Tuesday, OpenAI said that “to the extent that any ‘cause’ can be attributed to this tragic event” Raine’s “injuries and harm were caused or contributed to, directly and proximately, in whole or in part, by [his] misuse, unauthorised use, unintended use, unforeseeable use, and/or improper use of ChatGPT”.
It said that its terms of use prohibited asking ChatGPT for advice about self-harm and highlighted a limitation of liability provision that states “you will not rely on output as a sole source of truth or factual information”.
OpenAI, which is valued at $500bn (£380bn), said its goal was to “handle mental health-related court cases with care, transparency, and respect” and that “independent of any litigation, we’ll remain focused on improving our technology in line with our mission”.
The blogpost added: “Our deepest sympathies are with the Raine family for their unimaginable loss. Our response to these allegations includes difficult facts about Adam’s mental health and life circumstances.
“The original complaint included selective portions of his chats that require more context, which we have provided in our response. We have limited the amount of sensitive evidence that we’ve publicly cited in this filing, and submitted the chat transcripts themselves to the court under seal.”
The family’s lawyer, Jay Edelson, called OpenAI’s response “disturbing” and said the company “tries to find fault in everyone else, including, amazingly, by arguing that Adam himself violated its terms and conditions by engaging with ChatGPT in the very way it was programmed to act”.
Earlier this month, OpenAI was
hit by seven further lawsuits
in California courts relating to ChatGPT, including an allegation it acted as a “suicide coach”.
A spokesperson for the company said at the time: “This is an incredibly heartbreaking situation, and we’re reviewing the filings to understand the details. We train ChatGPT to recognise and respond to signs of mental or emotional distress, de-escalate conversations, and guide people toward real-world support.”
In August, Open AI said it was
strengthening the safeguards
in ChatGPT when people engage in long conversations because experience had shown that parts of the model’s safety training might degrade in these situations.
“For example, ChatGPT may correctly point to a suicide hotline when someone first mentions intent, but after many messages over a long period of time, it might eventually offer an answer that goes against our safeguards,” it said. “This is exactly the kind of breakdown we are working to prevent.”
OpenAI needs to raise at least $207B by 2030 so it can continue to lose money
The era-defining Xbox 360 reimagined gaming and Microsoft never matched it
Guardian
www.theguardian.com
2025-11-26 15:00:28
Two decades on, its influence still lingers, marking a moment when gaming felt thrillingly new again • Don’t get Pushing Buttons delivered to your inbox? Sign up here Almost 20 years ago (on 1 December 2005, to be precise), I was at my very first video game console launch party somewhere around Lond...
A
lmost 20 years ago (on 1 December 2005, to be precise), I was at my very first video game console launch party somewhere around London’s Leicester Square. The Xbox 360 arrived on 22 November 2005 in the US and 2 December in the UK, about three months after I got my first job as a junior staff writer on GamesTM magazine. My memories of the night are hazy because a) it was a worryingly long time ago and b) there was a free bar, but I do remember that DJ Yoda played to a tragically deserted dancefloor, and everything was very green. My memories of the console itself, however, and the games I played on it, are still as clear as an
Xbox Crystal
. It is up there with
the greatest consoles
ever.
In 2001, the first Xbox had muscled in on a scene dominated by
Japanese consoles
, upsetting the established order (it outsold Nintendo’s GameCube by a couple of million) and dragging console gaming into the online era with Xbox Live, an online multiplayer service that was leagues ahead of what the PlayStation 2 was doing. Nonetheless, the PS2 ended up selling over 150m to the original Xbox’s 25m. The Xbox 360, on the other hand, would sell over 80m, neck and neck with the PlayStation 3 for most of its eight-year life cycle (and well ahead in the US). It turned Xbox from an upstart into a market leader.
In a very un-Microsoft way, the Xbox 360 was cool. Its design was interesting, an inwards double curve described by its designers as an “inhale”, with a swappable front faceplate. It had a memorably Y2K startup animation and clean, futuristic menus that brought messaging, friends lists and music. I remember finding Microsoft’s marketing powerfully cringe at the time – witness this
developer video
, featuring former Microsoft entertainment boss J Allard and his infamous earring, in which a guy juggles while saying the words “Three symmetric cores”. But, despite that, the machine they built felt modern and exciting. The controller, too, white with its pops of colour, was such a tremendous improvement on the uncomfortably gigantic original Xbox controller that it’s become a design standard. I know people who will still only use wired Xbox 360 pads to play PC games.
Powerfully cringe … Microsoft’s Xbox 360 promo video.
As the first properly, seamlessly connected console, it brought a lot of things together to form a sense of gamer identity: playing different games online under one unified gamertag; messages and social features, as well as the inspired idea of achievements, which created a personal gaming history via the little challenges you completed in everything you played. (Sony would soon copy this with trophies.) Attaching a number to this, the gamerscore, was devilish genius, encouraging players to compete for ultimately meaningless clout, and creating a powerful incentive for people to stick with the console rather than buying games elsewhere. The
Xbox
360 was the first console to understand that people stay where their friends are. If you had the choice between buying a game on PS3 or 360, you’d choose 360 because that’s where everyone else was playing.
By late 2006, when a complacent Sony released an overpriced and awkward-looking follow-up to the PlayStation 2, the Xbox 360 had already had a year to convert people to its vision for high-definition gaming. People had already built up a collection of games and an online identity that was tied to Xbox. The big third-party game publishers, who found the PS3’s proprietary technology awkward to develop for, had started to prioritise Xbox for multi-platform games. The 360 never cracked Japan, but in the rest of the world it became the default console, an extraordinary thing for Microsoft to achieve considering how comprehensively Sony had dominated the previous two generations with the PlayStation.
The weird, monochrome realm of Limbo.
Photograph: TriplePoint
Xbox Live Arcade also helped to usher in the modern era of indie games. Between the 90s and the late 00s, publishers and bricks-and-mortar retailers largely controlled which games did and didn’t make it into players’ hands, especially on consoles. In 2008, Xbox Live Arcade started letting people download smaller, cheaper games direct to their consoles – no shop or publisher required. It did for console gaming what Steam would later do on PC, getting players comfortable with the idea of digital distribution. Games released via the arcade included
Geometry Wars
, Braid, Limbo, Bastion and, just as importantly, the best-ever digital version of Uno. I remember sinking many, many hours into Oblivion, Mass Effect and BioShock in my late teens, but I also eagerly awaited each new batch of Xbox Live Arcade games.
Looking back, the architects of the Xbox 360 really understood how and why people played games, and what they wanted from a next-generation console at the time. They understood how the internet could transform not just multiplayer gaming, but the social experience around games, and the way people found and bought them. This knowledge was apparently lost in a few short years, because when Microsoft announced the Xbox One in 2013, it was an absolute shitshow. By then, Microsoft apparently thought that people wanted to play games while watching sports picture-in-picture, as a mandatory connected camera watched your every move.
Microsoft has never again come close to market leadership in video games. A resurgent Sony took all the best lessons from the Xbox 360 and packaged them into the PlayStation 4, and then the Nintendo Switch arrived in 2018 and blew everything else out of the water. With Xbox now in distant third place in the waning console wars, it seems to see its future as a
quasi-monopolistic video game subscription service
, rather than a hardware maker. Series that defined the 360 era, such as Halo and Gears of War, are now playable on PC and PlayStation. Others, such as Fable, have been languishing for over a decade.
The 360 era was an exciting time in games, a period of great change and competition brought about by online gaming. The console market was a lot smaller back then, but also less predictable. There was still room for those “interesting, 7/10” B-games that sometimes proved even more memorable than the blockbusters when free-to-play games were not yet a thing – games were yet to consolidate into the five established mega-franchises that now dominate everything. And, in bringing indie games to console players, it genuinely changed the trajectory of my gaming taste.
Writing about Xbox Live Arcade had me hankering for
Geometry Wars: Retro Evolved
, the spectacularly compulsive Xbox Live Arcade
top-down shooter
that looks like fireworks and feels like a sensory bath for your brain. So I downloaded it on Steam and was instantly hooked once again. Made by Bizarre Creations, of Project Gotham Racing game, this game was constantly trading places with Uno as the 360’s most downloaded digital game, and it still holds up beautifully. I’d forgotten how the grid background ripples beautifully when things explode, a little high-definition-era flair for a very arcade-era game.
Available on:
Steam, Xbox (if you’re happy to play the sequel instead)
Estimated playtime:
10 minutes to, well, 20 years
What to read
Obstinately difficult and painfully funny … Baby Steps.
Photograph: Devolver Digital
I’ve been thinking a lot lately about
difficult games
, and what it is that keeps me coming back to them, which has led to reading quite a bit about challenge from a game designer’s perspective. And then this
exceptionally succinct article
by Raph Koster, veteran designer of Ultima Online and much else, dropped into my feed. It’s called
Game Design is Simple, Actually
, and it’s a must-read.
If you are more of an OG Xbox fan, you’ll be delighted to learn that
Crocs
have just launched an
Xbox clog
, inspired by the original Xbox’s black and green beast of a controller. It is
fantastically
ugly.
Poncle, makers of Bafta game of the year winning Vampire Survivors have announced a new game,
Vampire Crawlers
, with a
tongue-in-cheek trailer
. This one’s a blend of card-game and old school first-person dungeon crawler.
Last week, reader Jude asked me which video game world I would most want to live in (Cyrodiil from Elder Scrolls, obviously), and we threw the question back to you. We had so many delightful and/or deranged responses – here’s what you had to say.
“If you want somewhere to go get a beer, the world of
Cyberpunk 2077
looks amazingly hard to top.” –
Spence Bromage
“I know it’s silly but I was so enthralled with the ship in
System Shock 2
, I wanted to live there!”
– Charles Rouleau
“The
Dragon Age
universe in a heartbeat. Give me Fereldan and Denerim and yes, even Orlais. Give me a Skyhold to live in and a warble to manage, and I may never leave.” –
Kateland Vernon
“Call me weird, but I’ll take
Fallout 3
to live in. It had a massive impact on me, seeing pockets of humanity enduring the wasteland, with an overarching battle between good and evil.” –
Toby Durnall
“I have strange one:
Animal Well
. The freedom to explore this self-contained little map full of hidden corners has meant that I have a really good sense of where I am on the map. Even though I’ve ‘done’ the game’s activities, I have had some strange comfort in the last two weeks after finishing the game, just in wandering the space for the sheer joy of it.” –
Ben Gibb-Reid
If you’ve got a question for Question Block – or anything else to say about the newsletter –
email us on
pushingbuttons@theguardian.com
.
Dirk Eddelbuettel: tidyCpp 0.0.8 on CRAN: Maintenance
PlanetDebian
dirk.eddelbuettel.com
2025-11-26 14:57:00
Another maintenance release of the tidyCpp
package arrived on CRAN this morning, the first in about two years. The
packages offers a clean C++ layer (as well as one small C++ helper
class) on top of the C API for R which aims to make use of this robust
(if awkward) C API a little easier and more con...
Another maintenance release of the
tidyCpp
package arrived on CRAN this morning, the first in about two years. The
packages offers a clean C++ layer (as well as one small C++ helper
class) on top of the C API for R which aims to make use of this robust
(if awkward) C API a little easier and more consistent. See the (now
updated, see below) vignette for
motivating
examples
.
This release contains mostly internal upkeep of the usual type:
refreshing continuous integration, updating links, switching to
Authors@R. But as we wrap the C API of R here too, changes made in
R-devel this week affected the two reverse-dependencies
(i.e. “downstream”) packages (of mine) using this. So we commented-out
the definitions for the five now-hidden accessors so that these
downstream packages can build again under R-devel.
The NEWS entry follows.
Changes in tidyCpp
version 0.0.8 (2025-11-25)
Updated continuous integration setup several times
Updated README.md documentation with link to R API site
Updated example snippets to use of
Protect
Updated documentation in
defines.h
header
Updated
internals.h
header reflecting in R API
changes
As it happens, hours after the release at
CRAN
a helpful issue ticket was
opened detailing more than a handful of typos in the vignette. This has
been corrected, and I am not exporting the vignette via GitHub Pages so
the
motivating
examples vignette
contains the corrections.
Computer maker HP to cut up to 6,000 jobs by 2028 as it turns to AI
Guardian
www.theguardian.com
2025-11-26 14:53:31
US firm says plan to speed up product development and improve customer satisfaction would save $1bn a yearBusiness live – latest updatesUp to 6,000 jobs are to go at HP worldwide in the next three years as the US computer and printer maker increasingly adopts AI to speed up product development. Anno...
Up to 6,000 jobs are to go at HP worldwide in the next three years as the US computer and printer maker increasingly adopts AI to speed up product development.
Announcing a lower-than-expected profit outlook for the coming year,
HP said it would cut
between 4,000 and 6,000 jobs by the end of October 2028. It has about 56,000 employees.
“As we look ahead, we see a significant opportunity to embed AI into HP to accelerate product innovation, improve customer satisfaction and boost productivity,” said the California company’s chief executive, Enrique Lores.
He said teams working on product development, internal operations and customer support would be affected by the job cuts. He added that this would lead to $1bn (£749m) savings a year by 2028, although the cuts would cost an estimated $650m.
News of the job cuts came as a leading educational research charity warned that
up to 3m low-skilled jobs could disappear in the UK
by 2035 because of automation and AI. The jobs most at risk are those in occupations such as trades, machine operations and administrative roles, the National Foundation for Educational Research said.
In the US, about 40% of jobs could be replaced by AI, in sectors ranging from education and healthcare to business and legal, according to a report by the McKinsey Global Institute released this week.
AI agents and robots could automate more than half of US work hours using technology that is available today, the US consultancy’s analysis found. It estimated that $2.9tn of economic value could be unlocked in the US by 2030.
Most roles at risk are in legal and administrative services, with tasks such as data entry, financial processing and drafting documents likely to be handled by AI systems, although people will still be needed to design, supervise and verify. Dangerous, physical jobs such as machine operation could be replaced by robots.
HP had already cut between 1,000 and 2,000 staff in February as part of a restructuring plan.
It is the latest in a run of companies to cite AI when announcing cuts to workforce numbers. Last week the law firm Clifford Chance revealed it was
reducing
business services staff at its London base by 10% – about 50 roles – attributing the change partly to the adoption of the new technology.
The head of PwC also publicly cut back on plans to hire 100,000 people between 2021 and 2026, saying “the world is different” and AI had changed its hiring needs.
Klarna said last week that AI-related savings had helped the buy now, pay later company
almost halve its workforce over the past three years
through natural attrition, with departing staff replaced by technology rather than by new staff members, hinting at further role reductions to come.
Several US technology companies have announced job reductions in recent months as consumer spending cooled amid higher prices and a government shutdown.
Executives across industries are hoping to use AI to speed up software development and automate customer service. Cloud providers are buying large supplies of memory to meet computing demand from companies that build advanced AI models, such as Anthropic and OpenAI, leading to an increase in memory costs.
The push by big tech companies to build out AI infrastructure has triggered price increases for dynamic random access memory and NAND semiconductors – two commonly used types of memory chips – amid high competition in the server market.
Analysts at Morgan Stanley have said that soaring prices for memory chips, driven by rising demand from datacentres, could push up costs and dent profits at HP and rivals such as Dell and Acer.
“Memory costs are currently 15% to 18% of the cost of a typical PC, and while an increase was expected, its rate has accelerated in the last few weeks,” Lores said.
HP posted better-than-expected revenues of $14.6bn for its fourth quarter. Demand for AI-enabled PCs continues to increase, and they made up more than 30% of HP’s shipments in the fourth quarter to 31 October.
However, it forecast between $2.90 and $3.20 in adjusted net earnings per share for the coming year, below analysts’ forecasts of $3.33. It said the outlook reflected added costs from US trade tariffs.
HP shares fell as much as 6% after announcing the job cuts.
Solving the Partridge Packing Problem using MiniZinc
The
Partridge Packing Problem
is a packing puzzle that was originally proposed by Robert T. Wainwright at G4G2 (the Second Gathering for Gardner conference) in 1996.
In this post we will model and solve the Partridge Packing Problem using
MiniZinc
.
The inspiration was Matt Parker’s
fun video on the problem
.
Packing problems are a classic use-case for combinatorial solvers.
In fact, the original paper that introduced the idea of global constraints for constraint programming,
“Introducing global constraints in CHIP”
by Beldiceanu and Contejean 1994
included the so-called
diffn
constraint
for packing problems.
The constraint ensures that a set of (n-dimensional) boxes are not overlapping.
1
This post assumes some familiarity with MiniZinc.
For some background on MiniZinc, see the
previous posts
in the collection.
The puzzle will be explained fully, and no specific knowledge of packing problems is assumed.
The Partridge Packing Problem is a packing problem for squares in a larger square.
For size
, the goal is to pack:
into a square of size
.
2
The name comes from the song
“The Twelve Days of Christmas,”
where the first gift is a partridge in a pear tree, then two turtle doves, and so on going up to twelve drummers drumming.
The sum of the areas of all the smaller squares equals the area of the larger square
But just because the area matches does not mean that it is possible.
It is known that sizes 2 to 7 have no solution, and sizes from 8 to 33 have at least one solution.
The problem becomes increasingly difficult as
grows larger, as the number of parts grows quadratically.
Let’s look at the first interesting size with a solution, size 8.
Here are all the parts to pack.
3
These parts can be packed in a square of size
, where
comes from
, and here is one such solution.
This visualization shows how all the squares pack together perfectly to fill the 36×36 grid.
As mentioned, for sizes below 8 the problem is infeasible (except 1, which is the trivial case). Consider size 2,
which includes 1 part of size
and 2 parts of size
that should be packed in a
square.
As can be seen below, while the sum of the areas of the parts equals the area to pack in, there is no way to put the two
larger squares on the area without them overlapping.
Following previous parts in this
collection
,
we will split up the model in parts.
In this section the first basic model will be presented, including the data, the viewpoint,
the basic constraints, and the search and output.
In the next section, improvements to the model will be discussed.
Several of the improvements were suggested by
Mats Carlsson
, and made the model a lot better and faster.
The problem is parameterized by a single value
,
which determines both the number of different square sizes
and the size of the target square.
int: n;
setofint: N =1..n;
% Triangular number of n is both the total number of parts and
% the board size length
int: triangular_n = (n * (n+1)) div 2;
enum Parts =P(1..triangular_n);
setofint: Pos =1..triangular_n;
array[Parts] of N: sizes =array1d(Parts, reverse([
size
| size in N, copy in1..size
]));
constraintassert(sum(s in sizes) (s * s) == triangular_n * triangular_n,
"The squares fill the board completely");
The computed value
triangular_n
is the
triangular number
of the size parameter
n
.
This is both the total number of parts to pack as well as the side length of the board where the parts are to be placed.
The enum
Parts
is used to separate the set of parts from the
Pos
positions to place them at.
4
The sizes are generated in increasing order but are reversed,
resulting in the larger boxes being first in the array.
This is useful since many solvers will use the input-order as a tie-breaker for heuristics,
promoting packing hard-to-pack boxes (i.e., the larger ones) first.
Similar to the
LinkedIn Queens
post, we can use instance files to set the parameter
n
.
However, running the model from the MiniZinc IDE the user is prompted for all unknown values,
and for a single integer this is very easy to supply.
There are many ways that one can model a packing problem.
The most common way for box packings is to set one corner as the reference point,
and to use the position of that reference point as the position for the box.
The most natural expression for this is to use two arrays representing the x and y
coordinates of the bottom-left corner of each square.
% Main variables for placement of parts, the x and y coordinate
array[Parts] ofvar Pos: x;
array[Parts] ofvar Pos: y;
MiniZinc has a feature where records can be used to structure data,
and using that, we could declare the variables like this instead.
% Main variables for placement of squares, the x and y coordinate
array[Parts] ofrecord(var Pos: x, var Pos: y): box;
However, there are several places in the model where a constraint is formulated
over the
x
variables only, and then over the
y
variables.
Therefore, it is easier to use two arrays instead of a single one.
5
The base variables allow placement of the reference point anywhere inside the packing area.
However, the allowed positions need to be adjusted based on the size of a part.
This is done by adjusting the upper bounds of the x and y value based on the size,
ensuring that the point is also in the
Pos
set.
constraint :: "Parts fit in x direction"
forall(p in Parts) (
x[p] + sizes[p] -1in Pos
);
constraint :: "Parts fit in y direction"
forall(p in Parts) (
y[p] + sizes[p] -1in Pos
);
In the above (and the rest of the constraint here), constraints are named using the
:: string
annotation.
These names, such as
"Parts fit in x direction"
, are translated into the FlatZinc format and are useful for debugging
and for tools such as
findMUS
.
The main constraint for a packing problem is that no parts should overlap.
The classic way to ensure this is to use the no-overlap constraint,
which for historic reasons is named the
diffn
constraint
in MiniZinc.
constraint :: "No-overlap packing constraint"
diffn(x, y, sizes, sizes);
The arguments to
diffn
are the x and y positions of the rectangles, and their extent in the x and y direction
(that is, the width and the height).
Since the parts are squares, their extents are the same in both directions.
This is a satisfaction problem and we will leave the search strategy to the solver.
There are two output blocks for this model. The first block will print an ASCII-art representation of the packing to the standard output.
/**
* Get the unique singleton value in the supplied set, assert if it is not a singleton.
*/
function $$T: singleton_value(setof $$T: values) =
assert(card(values) ==1, "Values must have exactly one element, was \(values)",
min(values)
);
/**
* Return a character representation of the value v.
*
* Support values v in 1..36.
*/
functionstring: to_char(int: v) =
if v in0..9then
"\(v)"
else
["a", "b", "c", "d", "e", "f", "g", "h",
"i", "j", "k", "l", "m", "n", "o", "p",
"q", "r", "s", "t", "u", "v", "w",
"x", "y", "z"][v-9]
endif;
% Base command-line output mapping the placed parts to their sizes.
%
output [
let {
any: fx =fix(x),
any: fy =fix(y),
any: board =array2d(Pos, Pos, [
let {
Parts: part_id =singleton_value({p | p in Parts where
tx in fx[p]..(fx[p] + sizes[p]-1) /\
ty in fy[p]..(fy[p] + sizes[p]-1)
})
} in
to_char(sizes[part_id])
| tx in Pos, ty in Pos
])
} in
concat(tx in Pos) (
concat(board[tx, ..]) ++"\n"
)
];
While long, this code is reasonably straightforward.
First, there are two helper functions:
singleton_value
, which transforms a set that is known to be just one element to the element,
and
to_char
, which transforms a size to a character that represents it in base 36 (0-9 and a-z).
Next, a matrix is constructed where for each position, the part that is covering that position is found, and the size of that part is used to get the character.
Finally, this matrix is concatenated into a set of strings.
The second output-block uses a feature of the MiniZinc IDE where
custom visualizations can be used
.
These work by starting a webserver serving a webpage that receives the solutions as they are produced.
For this problem, the existing
vis_geost_2d
visualization
is used.
outputvis_geost_2d(
% Internal x and y offset of each part, 0 since each part is its own shape
[p:0 | p in Parts], [p:0 | p in Parts],
% Size of each part in x and y direction
sizes, sizes,
% Map each shape to the corresponding single part
[p:{p} | p in Parts],
% Reference points for each shape
x, y,
% The kind of each part
array1d(Parts, Parts)
);
The
vis_geost_2d
family of visualizations can show packing problems with
shapes made out of rectangles using internal offsets to a common shape reference point,
matching the input for the
geost
constraint
.
As each part is just a square, each kind of shape will be a single part, and the internal offsets are just 0.
Note that the construction
[p:0 | p in Parts]
will create an array with
Parts
as the index set,
skipping the
p:
part would create an array with
1..card(Parts)
as the index set.
An alternative way to write this is to coerce the base array to the right index set:
array1d(Parts, [0 | p in Parts])
.
In all the tests here, we will use OR-Tools CP-SAT 9.14 bundled with MiniZinc IDE 2.9.4 on a MacBook Pro M1 Max with 64 GiB of memory.
The configuration is set to use 10 threads (same as the number of cores in the CPU), and use free search.
As mentioned, sizes 2 to 7 are unsatisfiable, so the smallest interesting problem with a solution is size 8.
However, this base model is not efficient at all.
Finding a solution took about 3 and a half hours in one run, which makes it not very practical.
777777744448888888888888888333666666
777777744448888888888888888333666666
777777744448888888888888888333666666
777777744448888888888888888333666666
777777744448888888888888888333666666
777777744448888888888888888333666666
777777744448888888888888888227777777
444433344448888888888888888227777777
444433322777777777777776666667777777
444433322777777777777776666667777777
444455555777777777777776666667777777
444455555777777777777776666667777777
444455555777777777777776666667777777
444455555777777777777776666667777777
444455555777777777777776666667777777
777777788888888888888886666667777777
777777788888888888888886666667777777
777777788888888888888886666667777777
777777788888888888888886666667777777
777777788888888888888886666667777777
777777788888888888888885555588888888
777777788888888888888885555588888888
666666188888888888888885555588888888
666666555555555577777775555588888888
666666555555555577777775555588888888
666666555555555577777775555588888888
666666555555555577777775555588888888
666666555555555577777775555588888888
888888888888888877777775555588888888
888888888888888877777775555588888888
888888888888888866666666666688888888
888888888888888866666666666688888888
888888888888888866666666666688888888
888888888888888866666666666688888888
888888888888888866666666666688888888
888888888888888866666666666688888888
----------
==========
%%%mzn-stat:nSolutions=1
%%%mzn-stat-end
%%%mzn-stat:boolVariables=1023
%%%mzn-stat:failures=88389736
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:propagations=3870695549
%%%mzn-stat:solveTime=12697.9
%%%mzn-stat-end
Finishedin3h31m38s.
While the ASCII art is nice, the visualization is much easier to understand.
Below you can see first the visualization from MiniZinc, and then the visualization for this post
where squares of equal size get the same color and all squares are marked with their size.
The above model is the base, with just the constraints that are needed for a correct solution.
In this part, we will add additional constraints that improve the model significantly.
These constraints are of two types,
implied constraints
and
symmetry breaking constraints
.
An implied constraint is a constraint that strengthens the model by adding additional constraints that are true in every solution.
The goal is to add additional propagation that makes more deductions.
A symmetry breaking constraint is used to reduce the number of solutions, by limiting the symmetries of solutions.
Symmetries often arise from modeling decisions, but sometimes also from the problem itself.
For example, in the classic 8-queens problem there is a symmetry from the problem definition:
the chessboard for a single solution can be rotated and mirrored diagonally to create 8 different solutions.
If the model were to name the queens, then that would introduce a symmetry for which queen is placed where.
This symmetry would occur because of modeling decisions, not from the problem itself where queens are indistinguishable.
6
We will use a feature of MiniZinc to mark constraints with their type by enclosing the constraint in calls to
implied_constraint
and
symmetry_breaking_constraint
.
While not useful for many solvers, some (such as
Constraint-Based Local Search solvers
)
can use this information to decide what constraints to soften and what constraints to use for moves.
For each improvement, we will test it to see the effects.
Note that the configuration that is used, OR-Tools CP-SAT with 10 threads, is not a deterministic system.
One single run might not be indicative for all runs, but in most cases it will be a good indication.
A classic implied constraint for packing problems is to add a
cumulative
profile constraint
for the x and y direction.
Cumulative is a classic scheduling constraint, and is typically used for tasks that use some set of resources while they are active.
Below is an example of 8 tasks that are scheduled, with a capacity limit of 8 and varying amounts of usage at different points.
Note that the tasks do not have a fixed y-position; they only have a start, an end, and a resource usage (height).
This means that tasks like the green task 4 and the purple task 6 are not shown as a rectangle but staggered based on the amount of other tasks.
For the packing case, looking along one dimension, the orthogonal dimension can be seen as a resource, and the squares as tasks to be scheduled.
This is a classic implied constraint that can strengthen the propagation, and OR-Tools CP-SAT even has several
parameters
that can be set to include cumulative-style reasoning.
Here, the
cumulative
constraint is instead added as a MiniZinc constraint so that it can be used with all different solvers.
constraint :: "Cumulative profile of parts along the x axis."implied_constraint(
cumulative(x, sizes, sizes, card(Pos))
);
constraint :: "Cumulative profile of parts along the y axis."implied_constraint(
cumulative(y, sizes, sizes, card(Pos))
);
Running this, however, does not give better results at all.
The simple model took three and a half hours, but this model takes more than an hour more!
%%%mzn-stat:boolVariables=2184
%%%mzn-stat:failures=99470613
5 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:propagations=7359764734
%%%mzn-stat:solveTime=17031.9
%%%mzn-stat-end
Finishedin4h43m52s.
Unfortunately, this type of behavior is not uncommon when a learning system with automatic heuristics and randomization is combined with changes to a model.
This shows the importance of benchmarking and testing all changes to see how the model behaves.
Even well-known improvements might make it worse.
The
cumulative
constraint above adds to the reasoning, but it is also a lot weaker than it could have been.
The Partridge Packing Problem is a tight packing, where the board is fully covered.
The cumulative constraint “just” says that too much area can’t be used.
Consider instead a constraint that, for each row and column, checks which parts overlap it and requires that the sum of the sizes of overlapping parts equals the board size exactly.
% The sizes of the parts that overlap rc in the xy direction
% on_rc[p] is true iff the part overlaps the row/column rc
array[Parts] ofvarbool: on_rc = [
rc-sizes[p] < xy[p] /\ xy[p] <= rc
| p in Parts
]
} in
sum(p in Parts) (
sizes[p] * on_rc[p]
) =card(Pos);
constraint :: "Exact profile of parts along the x axis."implied_constraint(
forall(rc in Pos) (
exact_fill(x, rc)
)
);
constraint :: "Exact profile of parts along the y axis."implied_constraint(
forall(rc in Pos) (
exact_fill(y, rc)
)
);
Here, a utility function is added so that the right sum can be constructed for each column and for each row.
The
exact_fill
function takes the positions of all the parts along either the x or y axis, and a specified row or column.
Inside, a local array
on_rc
indexed by
Parts
of Boolean variables is constructed that indicates whether each part overlaps that row or column.
Multiplying by the size of each part gives how much of the dimension is used, and that is required to be equal to the cardinality of the
Pos
set.
This addition is a huge improvement over the base model!
A solution is found in less than 4 minutes instead of 3 and a half hours.
%%%mzn-stat:boolVariables=3960
%%%mzn-stat:failures=6155170
5 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:propagations=1762225146
%%%mzn-stat:solveTime=225.119
%%%mzn-stat-end
Finishedin3m45s.
This is starting to look like a viable model to use.
Checking if the cumulative constraint might help now shows that it is still not a good addition,
and it increased the search time to 4 minutes 33 seconds.
%%%mzn-stat:boolVariables=3960
%%%mzn-stat:failures=7544566
5 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:propagations=2046103308
%%%mzn-stat:solveTime=272.594
%%%mzn-stat-end
Finishedin4m33s.
From the work that
Mats Carlsson
and
Nicolas Beldiceanu
did creating the
geost
constraint
,
there are several additional deductions that can be made based on placements of boxes.
The core insight in this case is that since the board should be filled completely,
then for every area created there must be parts that can fill it.
Consider the below packing where a part has been placed on the board close to the edge.
The red area next to the border has a width of 2 and a height of 6.
It can only be packed with parts that are at most size 2, and a total area of
needs to be available.
However, for parts up to size 2, this is not possible since there is one
square and two
squares, for a total area of 9.
Trying to fill up the area between the size 6 part and the border would look like this.
Given the above reasoning, it is clear that any part of size 6 must either be placed next to a border,
or at a distance of more than 2 from a border.
In general, for a given size
, the sum of the areas of the smaller parts (up to size
) is the square of the triangular number for
.
This reasoning can be generalized and implemented with the following MiniZinc code.
% The amount of available area from parts up to given size
For each size of part, there is a custom calculation of the
allowed_placements
for that part.
Since the parts and the board are squares, the same set can be used for both x and y placements.
The calculation of
min_distance_from_edge
uses the idea that if the part is
d
steps away from the edge,
then the available area of parts up to size
d
must be greater than the
size
length times the value
d
for it to be valid.
Using this, the set of
forbidden_placements
is computed close to the edges,
and the
allowed_placements
are the complement of that with respect to
Pos
.
This is a conservative approximation of the packability:
if this requirement is not satisfied, then there is no packing that would work.
Adding this constraint reduces the time significantly again.
Running three times the time varies between 45 and 105 seconds due to the stochastic nature of the solving process.
The median has the following statistics.
2 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:boolVariables=3812
%%%mzn-stat:failures=2251053
%%%mzn-stat:propagations=558765030
3 collapsed lines
%%%mzn-stat:solveTime=73.3695
%%%mzn-stat:nSolutions=1
%%%mzn-stat-end
Finishedin1m13s.
In the original
geost
work, this type of reasoning is not just limited to placements close to an edge,
but for all different types of induced areas during the search.
This is much stronger reasoning, but would not be readily expressible as fixed constraints.
It requires careful implementation as a propagator in a system.
SICStus Prolog has the original and probably most advanced implementation of
geost
with a large domain-specific language to express placements.
Symmetry breaking is often crucial in many problems.
Here, the focus is a symmetry that is introduced by the modeling: parts of the same size should be indistinguishable.
The three parts of size
are the same, but since they have different identifiers they are different to the solvers.
A common way to break symmetries is to introduce an ordering among the alternatives.
For each size, the set of parts with that size are collected.
Then, a matrix of
placements
is constructed where each column represents the
x
and
y
coordinates of a part for that size.
7
The
lex_chain_less
constraint
is used to order these tuples using lexicographic ordering.
Adding the symmetry reduces the solving time significantly again.
In 10 runs, it is between 0.8 and 3.6 seconds, with an average of 1.9 seconds.
The median has the following statistics.
2 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:boolVariables=3700
%%%mzn-stat:failures=15018
%%%mzn-stat:propagations=8614720
3 collapsed lines
%%%mzn-stat:solveTime=1.4264
%%%mzn-stat:nSolutions=1
%%%mzn-stat-end
Finishedin1s830msec.
As mentioned above, the board has 8 symmetries (four rotations times flipping), and it is common to break them in many puzzle cases.
Matt Parker argues in the video that for the purposes of this puzzle, they should be kept in.
Also, it can be quite tricky to combine symmetry breaking techniques.
For any way to order the symmetries of the board, that ordering would have to work jointly with the ordering of the parts.
For testing, you can download the
full MiniZinc model
.
Remember to set OR-Tools CP-SAT to use at least as many threads as you have cores, and to also check the free search box.
In all the above examples, size 8 has been the instance solved.
Using the model developed, let’s try larger sizes and see the performance for that.
In
Matt Parker’s video
that inspired this post, size 9 was the instance that was discussed.
This is because size 9 has a side-length of 45, and thus the area of the board is
, which is the year the video was published.
Remember, even though the step from 8 to 9 sounds small, the number of parts grows from 36 to 45.
In a couple of tests, it took between 61 and 86 seconds to solve size 9.
2 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:boolVariables=6060
%%%mzn-stat:failures=651221
%%%mzn-stat:propagations=323892330
3 collapsed lines
%%%mzn-stat:solveTime=61.1534
%%%mzn-stat:nSolutions=1
%%%mzn-stat-end
Finishedin1m1s.
At size 10, there are 55 parts to pack on a board of 3025 squares, increasing the difficulty even more.
Here OR-Tools CP-SAT is starting to struggle a bit more, and in two runs took about 13 and a half minutes.
Here are the statistics for one of them.
2 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:boolVariables=9319
%%%mzn-stat:failures=6516108
%%%mzn-stat:propagations=3208777732
3 collapsed lines
%%%mzn-stat:solveTime=804.504
%%%mzn-stat:nSolutions=1
%%%mzn-stat-end
Finishedin13m25s.
As can be seen below, the two solutions found are quite different from each other.
Turning it up to eleven, it took OR-Tools CP-SAT a bit more than 51 minutes to solve the problem.
With 66 parts and an area of 4356, it is significantly larger than size 10.
2 collapsed lines
%%%mzn-stat:objective=0
%%%mzn-stat:objectiveBound=0
%%%mzn-stat:boolVariables=13850
%%%mzn-stat:failures=15611863
%%%mzn-stat:propagations=10240280820
3 collapsed lines
%%%mzn-stat:solveTime=3078.61
%%%mzn-stat:nSolutions=1
%%%mzn-stat-end
Finishedin51m19s.
Finding a solution of size 12 turned out to be too hard for the model.
Running OR-Tools CP-SAT for 12 hours gave no result.
In the above tests, only the OR-Tools CP-SAT solver has been used.
This is both because initial experiments showed it was probably the best solver for this
and because it has been dominant in the
MiniZinc Challenge
for more than a decade.
A benefit of MiniZinc is that many different solvers can be tested, so let’s look at some alternatives.
The new
Huub solver
was quite impressive in this year’s
MiniZinc Challenge
coming in third after OR-Tools CP-SAT and Chuffed in the Open category.
Huub uses an external SAT solver, and runs single threaded.
Running the model for size 8 with free search for ten rounds solves it in between 7.8 and 7.9 seconds,
which is remarkably stable.
%%%mzn-stat:solveTime=7.474344917
%%%mzn-stat:failures=103390
%%%mzn-stat:peakDepth=4796
%%%mzn-stat:propagations=20878861
%%%mzn-stat:restarts=145
3 collapsed lines
%%%mzn-stat:oracleDecisions=123909
%%%mzn-stat:userDecisions=0
%%%mzn-stat-end
Finishedin7s839msec.
This looked very promising, but increasing to size 9 Huub timed out after 12 hours.
Pumpkin
is also an LCG solver like Huub, but it is more focused on proof logging.
It is single-threaded like Huub, and uses a custom internal SAT solver.
Here, solving size 8 took around 2 minutes (2 test runs).
%%%mzn-stat:nodes=838498
%%%mzn-stat:failures=427421
%%%mzn-stat:restarts=1706
%%%mzn-stat:variables=12219
%%%mzn-stat:propagators=14931
%%%mzn-stat:propagations=422962879
%%%mzn-stat:peakDepth=570
4 collapsed lines
%%%mzn-stat:nogoods=427421
%%%mzn-stat:backjumps=307815
%%%mzn-stat:solveTime=147.569079042
%%%mzn-stat-end
Finishedin2m28s.
While size 8 was significantly slower for Pumpkin than for Huub, Pumpkin could actually solve size 9 in around 10 minutes.
%%%mzn-stat:nodes=3080585
%%%mzn-stat:failures=1547051
%%%mzn-stat:restarts=5243
%%%mzn-stat:variables=19208
%%%mzn-stat:propagators=23411
%%%mzn-stat:propagations=1673243823
%%%mzn-stat:peakDepth=925
4 collapsed lines
%%%mzn-stat:nogoods=1547051
%%%mzn-stat:backjumps=1108974
%%%mzn-stat:solveTime=642.870503792
%%%mzn-stat-end
Finishedin10m43s.
Running size 10 with Pumpkin failed with an unspecified error after around 5 hours.
None of these solvers were really useful for this problem.
Chuffed is often a very good solver with really great automatic search heuristics, but sometimes it doesn’t work that well.
Here, it took just over two hours to find a solution to the base size 8 packing.
Chuffed is single-threaded, same as Huub and Pumpkin.
%%%mzn-stat:nodes=123635519
%%%mzn-stat:failures=60596049
%%%mzn-stat:restarts=73648
%%%mzn-stat:variables=34838
%%%mzn-stat:intVars=2734
%%%mzn-stat:boolVariables=32102
%%%mzn-stat:propagators=5521
%%%mzn-stat:propagations=96201866795
%%%mzn-stat:peakDepth=272
10 collapsed lines
%%%mzn-stat:nogoods=60596049
%%%mzn-stat:backjumps=59399516
%%%mzn-stat:peakMem=0.00
%%%mzn-stat:time=7432.788
%%%mzn-stat:initTime=0.078
%%%mzn-stat:solveTime=7432.710
%%%mzn-stat:baseMem=0.00
%%%mzn-stat:trailMem=0.12
%%%mzn-stat:randomSeed=-499155368
%%%mzn-stat-end
Finishedin2h3m53s.
Gecode
is a competent classical constraint programming solver, and as such it doesn’t really have any effective automatic search heuristics.
This is clearly visible for this problem, where it fails to solve the problem in 12 hours.
3 collapsed lines
%%%mzn-stat:initTime=0.0371863
%%%mzn-stat:solveTime=43199.8
%%%mzn-stat:solutions=0
%%%mzn-stat:variables=9550
%%%mzn-stat:propagators=9245
%%%mzn-stat:propagations=2213651468397
%%%mzn-stat:nodes=5060871988
%%%mzn-stat:failures=2530435922
%%%mzn-stat:restarts=0
%%%mzn-stat:peakDepth=108
%%%mzn-stat-end
Finishedin12h.
Since Gecode can really benefit from a search heuristic, I tried adding one.
This heuristic uses the well-known
left-bottom placement
strategy,
prioritizing placement of larger parts before placing smaller parts.
This did not help.
% The position of a Part is essentially the index of the square.
array[Parts] ofvarint: position = [
x[p] *card(Pos) + y[p]
| p in Parts
];
% Search by placing the part with the smallest position/index at that position,
% breaking ties by input order (where larger parts are earlier).
Finally,
HiGHS
is a modern open source MIP solver.
Unfortunately, it also fails to solve this problem in 12 hours.
As mentioned above, the original development of the
geost
constraint was done in the SICStus Prolog solver.
However, the MiniZinc model here does not translate to the geost constraint,
nor is there support for using the specialized settings for the geost constraint.
Running the base MiniZinc model takes more than 4 hours to solve size 8.
2 collapsed lines
%%%mzn-stat:initTime=0.075
%%%mzn-stat:solveTime=15488.9
%%%mzn-stat:propagations=32188610389
3 collapsed lines
%%%mzn-stat:entailments=17947678933
%%%mzn-stat:prunings=58276738702
%%%mzn-stat:backtracks=381834851
%%%mzn-stat:restarts=0
%%%mzn-stat:solutions=1
%%%mzn-stat:optimalities=0
%%%mzn-stat:propagators=6651
%%%mzn-stat:variables=16508
%%%mzn-stat-end
Finishedin4h24m11s.
However, in the SICStus distribution, there is a partridge packing example with suitable geost arguments and a custom search predicate.
Here we get the chance to compare a generic model with one that is customized for a solver,
using that particular solver’s special features.
Solving size 8 is really quick at around half a second (the timing is reported in milliseconds).
Note also that SICStus is a single-threaded system.
Size 9 took about a minute, size 10 took around 23 minutes, size 11 took 4 and a half minutes, and size 12 took 1 hour 11 minutes.
It is expected that a larger instance can sometimes be faster (11 vs 10) when searching for a satisfying solution.
Another things that is also worth noting that SICStus uses less than 20 MiB of memory when searching for a solution for size 12,
while OR-Tools CP-SAT uses over 3 GiB of memory.
Here is the size 12 partridge packing that SICStus found.
Since size 12 is the reason the Partridge Packing Problem got its name,
it feels good to find a solution for this size as well.
At size 13, SICStus also starts to struggle with the search, with no solution produced in 12 hours.
Solving the Partridge Packing Problem using MiniZinc is an interesting challenge.
The base model performs poorly, and the usual trick (adding cumulative constraints) for improving a packing problem was not that useful.
However, with some custom implied constraints and symmetry breaking, it was possible to get solutions for size 8 and 9 quite quickly.
As is common for CP problems modeled in MiniZinc, OR-Tools CP-SAT dominates in performance.
However, it was interesting to see that the relatively new solvers Huub and Pumpkin are both promising.
8
Moving from MiniZinc to the custom SICStus Partridge program showed the benefits of using a system with smart propagators and a custom search strategy.
There are better ways to solve this packing problem, giving faster solutions in a more scalable way.
Still, it is a good example of how to incrementally develop a MiniZinc model and how to add strengthening constraints.
A benefit of using a high-level modeling language for this type of problem is that it can be adapted to new constraints
and changes in requirements.
In many industrial problems, it is quite common for requirements to change frequently.
In the end though, the most important part was that it was fun to experiment with.
A National Mission to Accelerate Science Through Artificial Intelligence
A National Mission to Accelerate Science Through Artificial Intelligence
Video Url
Genesis Mission video
US Dept of Energy
Genesis Mission is a national initiative to build the world's most powerful scientific platform to accelerate discovery science, strengthen national security, and drive energy innovation.
Goal
Genesis Mission will develop an integrated platform that connects the world's best supercomputers, experimental facilities, AI systems, and unique datasets across every major scientific domain to double the productivity and impact of American research and innovation within a decade.
Collaborators
Energy
Fusion you can plug into
Harnessing the power of the stars to deliver abundant, affordable energy. Through real-time collaboration between scientists, supercomputers, and AI systems, researchers can design, test, and stabilize fusion reactors far faster than before, accelerating the realization of sustainable fusion power.
Advanced nuclear, faster and safer
Creating a new generation of more efficient reactor designs, including new modular reactors, that provide reliable, around-the-clock energy. Engineers and AI tools work together to optimize reactor design, materials, licensing, and operations, shortening development timelines while strengthening safety and performance.
An intelligent, resilient grid
Building a power network that grows as fast as the technologies it fuels. By combining human expertise in energy planning with AI-enabled forecasting and simulation, teams can modernize the nation's grid, improving reliability and accelerating deployment of new infrastructure.
Discovery Science
Seeing molecules in action
Revealing chemical and biological processes as they unfold in real time. AI will work with ultrafast experiments to observe molecular dynamics and uncover insights that accelerate breakthroughs in materials and medicine.
Understanding the universe, from quarks to cosmos
Connecting the smallest particles to the largest structures. Physicists, guided by AI tools that reason across astronomical and particle-physics data, work together to test new theories about dark matter, dark energy, and the laws of nature.
Discovering new quantum algorithms
Unlocking the next frontier of computation. AI serves as a reasoning partner for researchers, generating and verifying new classes of quantum algorithms while scientists interpret and validate the results, bringing practical quantum computing closer to reality.
National Security
Securing critical materials
Reducing dependence on foreign supply chains. Materials scientists and AI systems co-design substitutes, responsibly utilize Earth's resources, and recover rare elements from waste, building a stable, self-reliant foundation for the nation's future industries.
Accelerating advanced manufacturing
Turning design into production at the speed of need. Engineers and AI-driven digital twins share a continuous feedback loop between design, sensors, and fabrication, cutting qualification time and boosting efficiencies.
Discovering mission-ready materials
Delivering alloys, polymers, and composites vital to defense and industry. Human insight and AI-guided discovery converge to fuse simulation, literature mining, and autonomous labs, pointing toward a future where years of materials research could unfold in a fraction of the time.
Essential Information and Guidance
A national initiative led by the Department of Energy and its 17 National Laboratories to build the world’s most powerful scientific platform to accelerate discovery, strengthen national security, and drive energy innovation.
We are amid a revolution in computing, driven by artificial intelligence and quantum information technologies, that will transform how science is done. Genesis Mission has the goal of doubling the productivity and impact of U.S. research and development by pairing scientists with intelligent systems that reason, simulate, and experiment at extraordinary speed.
Genesis Mission will create a national discovery platform that unites the world’s most powerful supercomputers, AI systems, and emerging quantum technologies with the nation’s most advanced scientific instruments. Together, they form an integrated infrastructure for scientific exploration—an intelligent network capable of sensing, simulating, and understanding nature at every scale.
By connecting these systems, Genesis Mission will transform how science is done. It will generate a new class of high-fidelity data to train advanced AI models, empower researchers to solve the hardest scientific challenges, and accelerate discovery from years to months. In doing so, it will serve as both a national accelerator for innovation and a proving ground for the next generation of AI and quantum and robotics technologies.
From fusion energy and new materials to quantum computing and life-saving medicines, Genesis Mission expands what’s possible in energy, discovery science, and national security.
Unlike commercial models trained on the open internet, Genesis Mission draws from the government’s secure, multi-domain scientific data, decades of experiments unavailable anywhere else.
No. Genesis Mission enables them. It’s AI for discovery, not automation, helping researchers explore and understand the universe faster.
The Department of Energy, in partnership with the White House Office of Science and Technology Policy.
Genesis Mission brings together the Department of Energy’s 17 National Laboratories with America’s leading universities and industry, including pioneers in artificial intelligence, computing, materials, and energy, to build the most powerful scientific platform ever to solve national challenges.
The initial collaborators listed below. Together, they represent the strength of the U.S. innovation ecosystem, uniting public and private sectors to accelerate discovery and maintain America’s scientific and technological leadership.
Genesis Mission is a movement to transform how science is done. DOE will open parts of Genesis Mission platform to qualified researchers, innovators, and companies, ensuring the benefits of this national effort are shared across the American scientific ecosystem.
Learn more
.
Follow The Mission
The Next Era Begins Now. Subscribe for more information.
Rights Organizations Demand Halt to Mobile Fortify, ICE's Handheld Face Recognition Program
Electronic Frontier Foundation
www.eff.org
2025-11-26 14:46:12
Mobile Fortify, the new app used by Immigration and Customs Enforcement (ICE) to use face recognition technology (FRT) to identify people during street encounters, is an affront to the rights and dignity of migrants and U.S. citizens alike. That's why a coalition of privacy, civil liberties and civi...
Mobile Fortify
, the new app used by Immigration and Customs Enforcement (ICE) to use face recognition technology (FRT) to identify people during street encounters, is an affront to the rights and dignity of migrants and U.S. citizens alike. That's why a coalition of privacy, civil liberties and civil rights organizations are demanding the Department of Homeland Security (DHS) shut down the use of Mobile Fortify, release the agency's privacy analyses of the app, and clarify the agency's policy on face recognition.
As the organizations, including EFF, Asian Americans Advancing Justice and the Project on Government Oversight, write in
a letter
sent by
EPIC
:
ICE’s reckless field practices compound the harm done by its use of facial recognition. ICE does not allow people to opt-out of being scanned, and ICE agents apparently have the discretion to use a facial recognition match as a definitive determination of a person’s immigration status even in the face of contrary evidence. Using face identification as a definitive determination of immigration status is immensely disturbing, and ICE’s cavalier use of facial recognition will undoubtedly lead to wrongful detentions, deportations, or worse. Indeed, there is already at least one reported incident of ICE mistakenly determining a U.S. citizen “could be deported based on biometric confirmation of his identity.”
As if this dangerous use of
nonconsensual
face recognition isn't bad enough, Mobile Fortify also queries a wide variety of government databases. Already there have been reports that federal officers may be using this FRT to target protesters engaging in First Amendment-protected activities. Yet ICE concluded it did not need to conduct a new Privacy Impact Assessment, which is standard practice for proposed government technologies that collect people's data.
While Mobile Fortify is the latest iteration of ICE’s mobile FRT, EFF has been tracking this type of technology for more than a decade. In 2013, we identified how a San Diego agency had distributed
face recognition-equipped phones
to law enforcement agencies across the region, including federal immigration officers. In 2019, EFF helped pass a law temporarily banning collecting biometric data with mobile devices, resulting in the
program's cessation
.
We fought against handheld FRT then, and we will fight it again today.
Courtesy from
Courtesy from
Courtesy from
Courtesy from
Shortly before prices skyrocketed, I decided I wasn’t very interested in doing a separate EconoNAS builds. Several months ago, I realized that there were several off-the-shelf NAS machines that were more-than-capable of running TrueNAS and they were selling at economical prices that couldn’t be topped by a DIY approach. I will dive deeper into this in a future blog, eventually …
maybe
?
.png)
Poking around a Docker container with a mounted LLMFuse.
