2026: A Year of Reckoning

Portside
portside.org
2025-12-08 01:00:27
2026: A Year of Reckoning jay Sun, 12/07/2025 - 20:00 ...
Original Article

Solidarity Forever--a Depression era poster by Anita Wilcox // Heretic, Rebel, a Thing to Flout

Millions demonstrated. Cities mobilized in defense of their people. Judges and juries upheld the law. Voters laid the basis for revoking the 2024 balance of power and issuing a new mandate for progressive change.

We have the power to make 2026 a year of reckoning, of decisive defeats for the MAGA movement. We believe that a revitalized Left, with its vision of a multiracial democratic and working-class movement, is key to ousting the MAGA crowd at every level of government in every region of the country.

This is a time for incisive analysis and bold initiatives, for strategizing and organizing for real change. For devising new tactics and thinking big about what can be achieved. We at Portside will be working to provide you and other readers the best strategic thinking and analysis we can find from a multitude of sources. We will continue to reflect the struggles, in our country and globally, for peace, security and justice. Once a year we ask you to help us do that.

Support This Vision

This year showed what it looks like for people to make their own history.

New York voters generated a political thunderclap by electing a democratic socialist mayor. California answered Trump’s gerrymander. Chicago gave new meaning to whistleblowing and Portland launched the Frog Brigade. Each such creative act inspires new actions.

By these actions and many more, people punctured the facade of racist and reactionary omnipotence and created a new political reality. We believe that is a signal of what is to come. We look forward to many more reckonings in 2026.

Every day we search the Internet for examples of people making history, including frontline reporting, cogent argument, culture and humor. We look for and share insights from science. Every day, we share the best that we find with you.

To receive a short daily update of these materials, subscribe to Portside Snapshot .

As you probably know, we moderators of Portside work on an entirely volunteer basis. We’re rewarded by the readers who put the information we provide to use to secure a better future, to advance toward a qualitatively more just society.

We pledge to keep doing what we've been doing. We ask you to help us by donating to keep our servers running and our website current.

Support This Vision

We are delighted that in the last year visits to the Portside website tripled. More people are recommending material and more authors are submitting their writings for consideration. We are dedicated to serving as your eyes and ears in the digital universe. Keep sending your input to either portside@portside.org or reader comments .

Please contribute to keep this project going. We promise to make every donation go a long way toward the future we seek together. We don’t ask our readers for financial support often. If you want to be a part of this project and to keep it going strong, this is the time to support Portside.

Yours in struggle,

The entire Portside crew

Judy Atkins, Jonathan Bennett, Mark Brody, Barry Cohen, David Cohen, Ira Cohen, Jeannette Ferrary, Marti Garza, Greg Heires, Geoffrey Jacques, Will Jones, Maureen LaMar, Stephanie Luce, Ray Markey, John P. Pittman, Natalie Reuss, Lee Rossi, Nan Rubin, Meredith Schafer, Jay Schaffner, Kurt Stand, Ethan Young

Checks should be made payable to PORTSIDE and sent to:

Portside
355 Eighth Avenue #1J
New York, NY 10001-4839

Spinlocks vs. Mutexes: When to Spin and When to Sleep

Hacker News
howtech.substack.com
2025-12-08 00:38:44
Comments...
Original Article

You’re staring at perf top showing 60% CPU time in pthread_mutex_lock . Your latency is in the toilet. Someone suggests “just use a spinlock” and suddenly your 16-core server is pegged at 100% doing nothing useful. This is the synchronization primitive trap, and most engineers step right into it because nobody explains when each primitive actually makes sense.

Mutexes sleep. Spinlocks burn CPU. Both protect your critical section, but they fail in opposite ways. A mutex that sleeps for 3 microseconds is a disaster when your critical section is 50 nanoseconds. A spinlock that burns CPU for 10 milliseconds is a waste when you could’ve let the thread do other work.

Here’s what actually happens. A spinlock does a LOCK CMPXCHG in userspace—an atomic compare-and-swap that keeps looping until it wins. Zero syscalls, but 100% CPU usage while waiting. Every failed attempt bounces the cache line between CPU cores at ~40-80ns per bounce. With four threads fighting over one lock, you’re just burning electricity.

A mutex tries userspace first with a futex fast path, but when contention hits, it calls futex(FUTEX_WAIT) . That’s a syscall (~500ns), a context switch (~3-5μs), and your thread goes to sleep. When the lock releases, another syscall wakes you up. The thread scheduler gets involved. You pay the full cost of sleeping and waking.

Spinlocks are dangerous in preemptible contexts. Your thread holds the spinlock, gets preempted by the scheduler, and now three other threads are spinning for the full timeslice (100ms on Linux). They’re burning CPU waiting for a thread that isn’t even running. This is why the Linux kernel disables preemption around spinlocks—but you can’t do that in userspace.

Mutex fast paths are actually pretty fast. Glibc’s pthread mutex does an atomic operation first, just like a spinlock. Only when that fails does it call futex(). An uncontended mutex costs 25-50ns, not the microseconds you’d expect from syscall overhead. The syscall only happens under contention.

Priority inversion will bite you. Low-priority thread holds a spinlock. High-priority thread starts spinning. Low-priority thread never gets CPU time because the high-priority thread is hogging the CPU spinning. Deadlock. Priority Inheritance (PI) mutexes solve this by temporarily boosting the lock holder’s priority. Spinlocks can’t do that.

Cache line bouncing is your enemy. Every atomic operation invalidates the cache line on all other CPUs. Put two different spinlocks on the same 64-byte cache line (false sharing) and they contend even though they’re protecting different data. Modern allocators pad locks to cache line boundaries— alignas(64) in C++ or __attribute__((aligned(64))) in C.

Critical section under 100ns, low contention (2-4 threads): Spinlock. You’ll waste less time spinning than you would on a context switch.

Critical section 100ns-10μs, moderate contention: Hybrid mutex (glibc adaptive mutex spins briefly then sleeps). PostgreSQL’s LWLock does exactly this.

Critical section over 10μs or high contention: Regular mutex. Let the scheduler do its job. Spinning wastes CPU that could run other threads.

Real-time requirements: Priority Inheritance mutex on a PREEMPT_RT kernel. Spinlocks cause priority inversion. Bounded latency matters more than average-case performance.

Run perf stat -e context-switches,cache-misses on your process. High context-switches with low CPU usage means mutex overhead might be killing you—consider adaptive mutexes. High cache-misses with 100% CPU usage means cache line bouncing—your locks are too contended or you have false sharing.

Use strace -c to count syscalls. Every futex() call is a contended mutex. If you’re seeing millions per second, you have a hot lock that might benefit from sharding or lock-free techniques.

Check /proc/PID/status for voluntary vs involuntary context switches. Voluntary switches are threads yielding or blocking—normal. Involuntary switches mean the scheduler is preempting threads, possibly while they hold spinlocks.

Redis uses spinlocks for its tiny job queue because critical sections are under 50ns. PostgreSQL’s buffer pool uses spinlocks for lookup operations (nanoseconds) but mutexes for I/O operations (milliseconds). Nginx avoids the problem entirely with a multi-process architecture—no shared memory means no locks.

The Linux kernel learned this the hard way. Early 2.6 kernels used spinlocks everywhere, wasting 10-20% CPU on contended locks because preemption would stretch what should’ve been 100ns holds into milliseconds. Modern kernels use mutexes for most subsystems.

Your job is knowing your hold time and contention level. Profile with perf, measure with rdtsc, and choose the primitive that wastes the least time. A spinlock that spins for 200ns is fine. A spinlock that spins for 10ms is catastrophic. A mutex that sleeps for 5μs is fine. A mutex that sleeps for 20ns worth of work is wasteful.

The right answer isn’t “always use X.” It’s “measure your critical section, count your threads, and pick the tool that matches your problem.”

Let’s prove everything we just discussed with working code. You’ll build two programs—one using a spinlock, one using a mutex—and watch them behave exactly as described above. 

https://github.com/sysdr/howtech/tree/main/spinlock_vs_mutexes/generated

Three programs that work together:

A spinlock test that keeps your CPU at 100% while threads fight for the lock. A mutex test where threads politely sleep when blocked. A monitor that watches both programs live and shows you the CPU usage and context switch differences in real time.

First, we build a custom spinlock using C11 atomic operations. The key is atomic_compare_exchange_weak —it reads the lock, checks if it’s 0 (unlocked), and if so, sets it to 1 (locked) all in one atomic instruction. If someone else holds the lock, it keeps trying in a loop.

Save this as spinlock_test.c :

c 

#define _GNU_SOURCE
#include <stdio.h>
#include <pthread.h>
#include <stdatomic.h>
#include <time.h>
#include <sched.h>

#define NUM_THREADS 4
#define ITERATIONS 1000000
#define HOLD_TIME_NS 100

typedef struct {
    atomic_int lock;
    long counter;
} spinlock_t;

void spinlock_acquire(spinlock_t *s) {
    int expected;
    do {
        expected = 0;
    } while (!atomic_compare_exchange_weak(&s->lock, &expected, 1));
}

void spinlock_release(spinlock_t *s) {
    atomic_store(&s->lock, 0);
}

void* worker_thread(void *arg) {
    spinlock_t *lock = (spinlock_t *)arg;
    
    for (long i = 0; i < ITERATIONS; i++) {
        spinlock_acquire(lock);
        lock->counter++;
        // Simulate 100ns of work
        for (volatile int j = 0; j < 10; j++);
        spinlock_release(lock);
    }
    return NULL;
}

int main() {
    pthread_t threads[NUM_THREADS];
    spinlock_t lock = { .lock = 0, .counter = 0 };
    struct timespec start, end;
    
    clock_gettime(CLOCK_MONOTONIC, &start);
    
    for (int i = 0; i < NUM_THREADS; i++) {
        pthread_create(&threads[i], NULL, worker_thread, &lock);
    }
    
    for (int i = 0; i < NUM_THREADS; i++) {
        pthread_join(threads[i], NULL);
    }
    
    clock_gettime(CLOCK_MONOTONIC, &end);
    
    double elapsed = (end.tv_sec - start.tv_sec) + 
                     (end.tv_nsec - start.tv_nsec) / 1e9;
    
    printf(”SPINLOCK Results:\n”);
    printf(”  Final counter: %ld\n”, lock.counter);
    printf(”  Time: %.3f seconds\n”, elapsed);
    printf(”  Operations/sec: %.0f\n”, (NUM_THREADS * ITERATIONS) / elapsed);
    printf(”  CPU usage: 100%% (busy-waiting)\n”);
    
    return 0;
}

The atomic_compare_exchange_weak does the magic. It’s a single CPU instruction ( LOCK CMPXCHG on x86) that atomically checks and sets the lock. The “weak” version might fail spuriously on some architectures, but that’s fine—we’re looping anyway.

Now the mutex version. This uses pthread_mutex_t which calls futex() when contention happens. Threads sleep instead of spinning.

Save this as mutex_test.c :

c 

#define _GNU_SOURCE
#include <stdio.h>
#include <pthread.h>
#include <time.h>

#define NUM_THREADS 4
#define ITERATIONS 1000000

typedef struct {
    pthread_mutex_t mutex;
    long counter;
} mutex_lock_t;

void* worker_thread(void *arg) {
    mutex_lock_t *lock = (mutex_lock_t *)arg;
    
    for (long i = 0; i < ITERATIONS; i++) {
        pthread_mutex_lock(&lock->mutex);
        lock->counter++;
        for (volatile int j = 0; j < 10; j++);
        pthread_mutex_unlock(&lock->mutex);
    }
    return NULL;
}

int main() {
    pthread_t threads[NUM_THREADS];
    mutex_lock_t lock = { .mutex = PTHREAD_MUTEX_INITIALIZER, .counter = 0 };
    struct timespec start, end;
    
    clock_gettime(CLOCK_MONOTONIC, &start);
    
    for (int i = 0; i < NUM_THREADS; i++) {
        pthread_create(&threads[i], NULL, worker_thread, &lock);
    }
    
    for (int i = 0; i < NUM_THREADS; i++) {
        pthread_join(threads[i], NULL);
    }
    
    clock_gettime(CLOCK_MONOTONIC, &end);
    
    double elapsed = (end.tv_sec - start.tv_sec) + 
                     (end.tv_nsec - start.tv_nsec) / 1e9;
    
    printf(”MUTEX Results:\n”);
    printf(”  Final counter: %ld\n”, lock.counter);
    printf(”  Time: %.3f seconds\n”, elapsed);
    printf(”  Operations/sec: %.0f\n”, (NUM_THREADS * ITERATIONS) / elapsed);
    printf(”  CPU efficient (threads sleep when blocked)\n”);
    
    pthread_mutex_destroy(&lock.mutex);
    return 0;
}

Compile both programs with these flags:

bash 

gcc -Wall -Wextra -O2 -pthread spinlock_test.c -o spinlock_test
gcc -Wall -Wextra -O2 -pthread mutex_test.c -o mutex_test

The -pthread flag links the pthread library. -O2 enables optimizations without messing up our timing measurements.

Run the spinlock test and watch your CPU usage spike to 100%:

bash 

./spinlock_test

While it’s running, open another terminal and check CPU usage:

bash 

top -p $(pgrep spinlock_test)

You’ll see all four threads at 100% CPU. They’re spinning, waiting for the lock.

Now run the mutex test:

bash 

./mutex_test

Check its CPU usage the same way. You’ll see much lower CPU usage because threads sleep when blocked instead of spinning.

This is where it gets interesting. Use strace to see what syscalls each program makes:

bash 

strace -c ./spinlock_test

Look at the syscall summary. You’ll see almost no futex calls. The spinlock never talks to the kernel—it’s all userspace atomic operations.

Now trace the mutex version:

bash 

strace -c ./mutex_test

Count those futex calls. Thousands of them. Every time a thread can’t acquire the lock, it calls futex(FUTEX_WAIT) to sleep. When the lock is released, another futex(FUTEX_WAKE) wakes it up. That’s the syscall overhead we talked about.

Look at the /proc filesystem to see context switches:

bash 

# While spinlock_test is running in another terminal:
cat /proc/$(pgrep spinlock_test)/status | grep ctxt

# Then for mutex_test:
cat /proc/$(pgrep mutex_test)/status | grep ctxt

Spinlock will show very few voluntary context switches—threads never voluntarily give up the CPU. Mutex will show thousands of voluntary switches—threads sleeping and waking.

If you have perf installed, this shows cache behavior:

bash 

perf stat -e cache-misses,cache-references ./spinlock_test
perf stat -e cache-misses,cache-references ./mutex_test

Spinlock will show a higher cache miss rate. That’s the cache line bouncing between CPUs as threads fight for the lock.

Try changing NUM_THREADS to 2, 8, or 16. Watch how spinlock performance degrades with more threads—more CPUs spinning means more wasted cycles. Mutex handles it better because only one thread runs at a time while others sleep.

Change HOLD_TIME_NS by adjusting the busy-wait loop. Make it longer (more iterations) and watch the spinlock waste even more CPU. This proves the point: spinlocks are only good for very short critical sections.

You built working implementations of both primitives and saw the exact behavior we described. Spinlocks burn CPU but have low latency for short holds. Mutexes use syscalls and context switches but keep your CPU available for real work. The choice depends on your critical section duration and contention level.

Now you understand why Redis uses spinlocks for nanosecond operations but PostgreSQL uses mutexes for longer database operations. You’ve seen the futex syscalls, measured the context switches, and watched the CPU usage yourself. This isn’t theory—it’s how production systems actually work.

Show HN: Cdecl-dump - represent C declarations visually

Hacker News
github.com
2025-12-08 00:26:04
Comments...
Original Article

Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign up

Appearance settings

Repository files navigation

cdecl-dump

Dump C declarations visually on the command line.

How to use 

./cdecl-dump "int a"
./cdecl-dump "void f(int a)"
./cdecl-dump "unsigned char *const *arr[20][30]"
./cdecl-dump "int (*const fp[20])(void)"

Building

  • ./build.sh produces a debug build with additional tracing of the parsing stages
  • DEBUG=0 ./build.sh produces an optimised executable

Bugs

  • The program doesn't do strict validation of the declarator in certain cases. Unlike the rules of C, it allows functions to return arrays and arrays to have functions as their elements.
  • Only built-in types are supported. For example, size_t or uint64_t will not be accepted.

Screenshot

alt tag

How it works

The program uses a hand-written, table-driven lexer and parser.

An Attempt at a Compelling Articulation of Forth's Practical Strengths and Eternal Usefulness

Lobsters
im-just-lee.ing
2025-12-08 00:22:52
Comments...
Original Article
                  An Attempt at a Compelling Articulation
                      of Forth's Practical Strengths
                          and Eternal Usefulness


                                   PRELUDE

                              Problem Statement

    It's done! We have that crucial ingredient giving life to that little
    Forth, allowing it to walk by itself! Unfortunately, you'll soon see
    that after a few steps, this little creature stumbles and falls.
    What is it lacking, a little balance maybe?

      -- Virgil Dupras

Explaining why Forth is still relevant has become a bittersweet, ironic comedy 
routine enthusiasts and operators are finding themselves in more frequently.
We usually begin with lavish statements of simplicity, and ease the
reader into a fog of what Forth is. Finally we slam back to Earth with examples.
Examples so contrived that they have the complete opposite effect, a tint and a
dulling of any allure that may have existed as a spark in the prospective
Forther's mind. Each time a speech of grandeur falling flat. 

Virgil's quote is a perfect specimen of this phenomenon. The miracle of popping
into existence - walking nonetheless - is a direct testament to Forth's
evolutionary nature. It's such a strong positive to begin with, yet immediately
follows up with "it's useless".

Virgil needs no defense: he's right. It is useless. The point is, that statement
is a negative wrinkle for what is a positive feature: a foundational, minimal
core to build on. It's these instances of faulter that contribute to Forth's
continual uncertainty in today's context.

On Saturday I saw the final act causing me to want to write this piece.


                  Here Use This Honda Civic Made of Lambos

A long-running project called eforth was linked on lobste.rs, a link aggregation
site, attempting to sell Forth on still having relevancy:
https://github.com/chochain/eforth.

The author details their history, how they worked with a prolific Forth user and
contributor, Dr. Chen Hanson Ting (the creator of  eforth, passed away 2022),
and why their particular Forth project is a successor to his work.

But it's the first paragraph that takes the cake.

The first paragraph unironically reads:

    With all the advantages, it is unfortunate that Forth lost out to C
    language over the years and have been reduced to a niche. Per ChatGPT:
    due to C's broader appeal, standardization, and support ecosystem
    likely contributed to its greater adoption and use in mainstream
    computing.

The project's README then proceeds into lengthy detail about how to implement 
Forth in C.

Yep.

It is the fastest open-closed case I've seen to date of someone explaining why
you'd use Forth and then miraclously shoot both their feet at once.

It seriously left me empty-minded after reading.

The only hint of relevancy revolves around compilation and being able to build
the system for many architectures, thanks to C compilers and ease of
implementation.


                Why Do My Words Have Weight And Why They Don't

I'm not a Forth guru; a great sage like those of LISP, C, or Haskell. No
universal promise can be made to you that what I say is unyielding truth. You
will have to verify that for yourself. What I can promise is the perspective
from a simple, passionate programmer of a couple decades, and 3 years of those
belong to Forth. Never has my heart been completely enveloped by an idea. The
allure and compulsion has practically  caused me to fall in love with a
programming language. Maybe this is how Pythagoreans felt about their triangles.
With this great fire, I will tell you why Forth is forever.


                              WHAT IS FORTH

It's very possible many readers will be hearing about Forth for the first time
when they encounter this essay. There are two recommended readings for all
Forth first timers by Leo Brodie: Starting Forth, and Thinking Forth. The former
focuses more on concrete code, and the latter is a philosophical read. They go
hand-in-hand and I can't really recommend one over the other: it really depends
on the kind of person you are.

I will do my best to give an extremely short description of what Forth is in
this essay to remain self-contained.


                      Explain Like I'm 5: Forth Edition

Forth is a pancake stacking machine with superpowers.

  top      pancake5
           pancake4
           pancake3
           pancake2
  bottom   pancake1

You eat these pancakes like you normally would: from top to bottom.

What do you normally eat pancakes with? A fork and a knife.

In Forth you can make your own super special fork and knife, maybe they shoot
lasers, but you can also make robots that help you.

For example, the Gigafork-o-tron 9000 lets you eat 3 pancakes at once, by
smushing them into 1 big pancake for you to eat:

           smushed_pancake
           pancake2
           pancake1

With 100s of robots you can do some really cool stuff when you start feeding
them things other than pancakes, like build video games.

Why use Forth and not something else? Because all you need are some elastic
bands, pencils and paper to build this stacking machine with superpowers! These
are all items you've got lying around the house! Get going!


                  Explain Like I'm Familiar With Programming

Ok let's try again: so Forth is a stack-based programming language.

On the surface that statement means nothing except somehow the stack data
structure is somehow involved.

In Forth it's core to data passing (arguments) and your memory pool (allocated
memory).

To program arguments onto the stack, list them out before functions (words):

   1 2 3 4 + + +
-> 1 2 7 + +
-> 1 9 +
-> 10

We're left with 10 by the end of the reduction.

You're intended to create 100s of words (functions), which are inserted into
the dictionary (a global stack), which make up a vocabulary. Yes, each Forth
program is a domain specific language (DSL).

  \ A comment describing what's going on.
  \ ( stuff ) is an inline comment but it is used commonly to denote expected
  \ function arguments coming in on the stack. They do nothing.
  : my-func  ( int int -- int ) * 10 + ;

Forth can manipulate its own interpreter at "compile time". You can leverage
this when needing to parse formats, or create new syntax.

  s" something-to-parse-until" parse

Doing common work like splitting strings or concatenating lists is all provided
as words too.

That's Forth. It's so simple the idea is you can bootstrap it from assembly
relatively quickly.


                          SUPERNATURAL BOOTSTRAPPING

                     Programming Language By Synthesis

Programmers will often use other programming languages to implement their new
programming language. The choice of parent language is usually determined by
what problems the child language is intended to solve. Choosing a suitable
parent reduces time to implementation, keeping the programmer motivated and
actually achieving a finished program.

A side effect of using a parent language is the child will inherit its traits.
The performance ceiling of the parent becomes the ceiling of the child, unless
it's able to evolve a code generator. All available parent language
packages and libraries are available to the child for use toward its own
development and extensions. Any memory management and access faults are
transfered.

It's understandable why using an existing language to bootstrap your own is very
reasonable. 

A familiar case of the preceeding is Python. Python has and continues to be
written in the C programming language. Its C core hardly minimal, clocking in
at 35% of the whole codebase as of writing.

The Glasgow Haskell Compiler, the most prolific functional programming language
compiler, does the same, but with an exceptionally smaller, desireable core
size: 8% is C. Unfortunately it uses Python as well, pulling in 952 C source
files if we are to require it to build. So is it really small then, or simply
abstracted away into other software packages?

Clojure is a case of being a far descendant of C. Bootstrapped from Java,
which itself written in C++, further evolved from C. Clojure benefits from
Java's garbage collection and being able to interoperate with other Java source
files, while offering unparalleled expressiveness. CPUs that implement JVM
bytecode compatible hardware (such as the "Jazelle ARM extension") are an
interesting case of removing ancestrial ties, simplifying lineage. Similar
can be said with CPU architectures that favor C-like code.


                              C is Not Free

What's the problem then? C is the lingua-franca of the programming world. It
exists for nearly every practical architecture imaginable. It's free!

At the risk of getting absolutely roasted by two (or more) very strongly
opinionated communities, I will say pretty much (not every!) every mainstream
development in programming languages, Rust and Zig included, are guilty of this
fallacy. They are built around the illusion of their parent language having low
to zero cost.

LLVM is the typical basis of these languages: a heavy technology largely funded
by mega corporations. A Zig and Rust enthusiast argument about their language
being less complex than the other is off-putting. The reality is the complexity
comparison is surface level. The shape of two fruit are compared but the genetic
structure of the fruit is 90% the same. Theirs arguments revolve around
particular expressions of logic in various situations but ignores everything
else.

Zig and Rust experts will try to reason about this cost, that "they only make
up 1% of the code base". Uh, no, without that 1% there is nothing; the code will
not compile, and cannot be invoked. That 1% is made up of millions of dollars
and lines of code. It is not free, far from it for these LLVM-based languages!

Let me make an evidentary statement:

    A C compiler takes a non-trivial amount of energy to create.

Following are projects that I believe cover a range of full-featured to smallest
C compilers, and ran `sloccount` on them. The evidence leans toward a good
estimate, as the timelines match up with the commit histories and the amount of
contributors.* The results are organized from most to least expensive.


    clang**

    Total Physical Source Lines of Code (SLOC)                = 11,200,038
    Development Effort Estimate, Person-Years                 = 3,570.35
    Total Estimated Cost to Develop                           = $ 482,305,367

    gcc**

    Total Physical Source Lines of Code (SLOC)                = 8,208,908
    Development Effort Estimate, Person-Years                 = 2,576.50
    Total Estimated Cost to Develop                           = $ 348,049,714

    tcc

    Total Physical Source Lines of Code (SLOC)                = 108,038
    Development Effort Estimate, Person-Years                 = 27.31
    Total Estimated Cost to Develop                           = $ 3,688,901

    chibicc

    Total Physical Source Lines of Code (SLOC)                = 9,487
    Development Effort Estimate, Person-Years                 = 2.12
    Total Estimated Cost to Develop                           = $ 286,832

    *1 Of course you can call bullshit and verify yourself. The `sloccount`
       website has further testimonials to its usefulness in estimates. Even
       if the margin of error is a whole magnitude, the cost remains high.
    *2 Fun fact these took like 8+ minutes to crunch, the others were instant.
    *3 It must be made clear for transparency that clang and gcc SLOC counts
       include GIMPLE, LLVM, and other tooling. Generally speaking both
       compilers require them to work, but they include additional code for
       non-C compilers, such as ALGOL, Fortran, and GO, as well.

Now add on top the person-hours required to build the child language. Bonkers.


                      Programming Language By Evolution

New programming languages are also created through more natural, evolutionary
means. Common examples are C++ and TypeScript. C++ came to be from C programmers
required to handle ever-growning requirements and complexity of systems
software. Web developers turned to type-systems to deal with their own
ecosystem and social complexity. structuring JavaScript code written by people
(and now robots) from all walks of life. What could such an evolution look like
when we go back to the roots of programming: assembly?

Assembly is the lowest level of abstraction we can get from the computer. The
IBM 1401 from the 1960s forced humans to write code in literal 1s and 0s.
Quickly we learned mnemonics are easier on the brain than monotonous numerals.
Textual code is a one-to-one abstraction that arose from naturally using the
technology of the time. As programs grew in complexity, and humans being
incredibly good pattern matchers, they realized a lot of code could be
deduplicated into reuseable chunks. At first this took the form of subroutines,
but it was not enough. The macro was created to resolve this complexity.
Once again a natural solution to a natural problem. Now assembly programmers
could create systems beyond their normal comprehension - literally!

This is where Charles Moore (aka Chuck) enters the scene. The idea of Forth is
not particularly genius or crazy, but his resolve toward simplicity is something
to really celebrate. The story goes Chuck worked at various astronomical
observatories throughout the years, and needed to program the telescopes. He
found himself needing to interact with all the different hardware frequently.
Bringing a set of macros to each new telescope, this is the system that became
Forth. An evolution of assembly macros; Forth is a true example of holistic
design.

While it took years to refine the idea of what Forth was, the implementation of
a Forth-like takes place all the time, to this day. The first thing language
implementation learners do: implement a stack machine. When a technology
accidentally exists time-again the cost is even better than free. It just is.
For me this equivalent to discovering how to create your own molecules. A true
lingua-franca of the Turing machine.


                   Masterclass Example of a Forth Bootstrap

SmithForth is a golden textbook example of bootstrapping a Forth from nothing.
SmithForth literally constructs executable headers using binary, and each line
of binary maps to an instruction, with each commented in great detail.

Here is an excerpt:


################ Interpreter subroutines #######################################

99 05 43 4F 4D 50 4C #### COMPL Forth's COMPILE, B9 ( ebx=xt -- )
B0 FF AA                # compile >>>>>>>>>>>>>>>>> call r/m64          FF /2
B0 14 AA                #     al = _                mov r8, imm8        B0+rb ib
B0 25 AA                #         [rdi++] = al      stos m8             AA
93                      # eax = ebx                 xchg eax, r32       90+rd
AB                      # [rdi(++4)] = eax          stos m32            AB
C3                      # return                    ret                 C3


Absolutely psycho. And it goes on for ~500 lines. I am envious of the mind that
yields the mental model to parse this fine.

A C programmer in comparison would have to write a minimal C compiler this way,
to then write some C to bootstrap other C features. All while on that same
computer. Sounds painful. The person who wrote the smallest useful C compiler,
chibicc, shown in the previous section, also wrote 8cc. Both took over a year
to write.

Meanwhile the Forth programmer is off to solving their original problem in a
week. A full working environment. Their power continues to build as the
dictionary fills out with useful words. Maybe they go a little overboard and
develop a full operating system. That's the principle behind Virgil's incredible
Dusk OS. It's a testment to the practicality of doing that in Forth.


                     EXTENSION THROUGH SELF-MODIFICATION

Programming languages rarely expose methods to extend the language syntax
itself, or modify existing syntax. The latter is more common in interpreted
languages like Python or JavaScript, where an object property can be overridden
with a new one. Forget about this in systems languages. C++ has method override
but it's not the same: you cannot change the behavior of how addition works
on two 64-bit integers (such as treating them both as fixed-point numbers).
At most these languages expose complex macro or template systems, and it's a
discouraged practice depending on where, or how, you work. Adding more syntax to
already complex languages literally creates more complexity.

The opposite approach is taken in Forth: syntax is non-existent, so adding it
to simplify reasoning about the solution to a problem is encouraged. Each
program then is a narrow, well-focused expression of a solution.

For example, there's a complex tax situation, and you want to record and
calculate everything with some sort of special text record file. The playbook
for this problem is to write a parser. Design decisions such as number precision
and operations are made. Finally the the actual computations take place. Then
there's the output format.

In Forth, we leverage the language itself. Your program code doubles as the
record file. There is no separation because the Forth programmer can leverage
the Forth compiler, morphing it to their needs at run-time. Changing addition
to work on "bignum"s (numbers that have arbitrarily large precision) is a good
usecase of overriding the addition operation.

As you may have guessed, this is a real situation I found myself in. Encoding
tax brackets and having a system that calculated tax became trivial once I 
defined the "tax bracket" word. Cute how it's a literal bracket:

    ...

    \ Calculate a bracket.
    \ ac:accumulated capital, rb:remaining balance
    \ r:tax rate, l: upper limit
    : ]> ( ac rb r l -- ac rb )
      ( ac rb r l ) frot
      ( ac r l rb ) f2dup f< if
        fremaining frot frot f*
        ( ac rb tax ) frot
        ( rb tax ac ) f+ fswap
        ( ac rb ) else
        fswap fdrop f* f+ 0e \ rb is zero
        then
    ;

    : tax-cra-income ( income-to-tax -- collected )
      0e fswap
      ( collected remaining ) 0.150e  55867e ]>
                              0.205e 111733e ]>
                              0.260e 173205e ]>
                              0.290e 246752e ]>
                              0.330e fover   ]>
      fdrop
    ;

    ...

    accounts LeeBusiness
    accounts ACompany BCompany

    -5674.84e tax-sales   ACompany
    remaining  tax-income LeeBusiness

    -2145.66e            BCompany
    remaining tax-income LeeBusiness

    ...


Then we arrive at the point of divergence: most languages stop at a parser for a
new file format. Little Forth goes beyond: you can also modify and extend the
"outer interpreter".

This is the REPL most Forths offer to developers so they can poke and prod at
hardware. In this case the hardware is literally the computer you're working at.
It is common for Forth developers to create their own  rudimentary text editors
as an exercise. Quickly it becomes obvious how such a holistic system gives a
natural extension power very few other languages have.

Some people have taken this strength to the extreme: Virgil Dupras's DuskOS and
mschwartz's mykesforth both build entire operating systems off these principles.

DuskOS is intended to solve the goal of bootstrapping other C-based operating
systems with a Forth core that implements a C compiler.

mykesforth is a complete pure Forth OS.


                              EXECUTABLE COMPRESSION

Texas Instruments' introduction of the MSPM0C1104 this year (2025) has ushered
us into the world of nano-controllers. A computer that can sit on a pin-head.
With only 16 KiB of storage and 1KiB of memory. The 1980s were more generous:
most home computers had 64 KiB of memory and much more storage. It's no surprise
Forth found itself popular in that era. The ability to create essentially
compressed executable code was and still is unmatched and greatly desired. This
is achieved through a particular implementation called Indirect Threading Code
(ITC). Combined with heavy code refactoring into small functions, programs
become executable Lempel–Ziv–Welch compressions, which astonishingly also have a
concept of a "dictionary" and reusable parts. The parallels are remarkable.
There is literally no other language I'm aware of that does this kind of
optimization to this degree.

Briefly I will explain what ITC is.

Here is a pseudo example of some program code (& means address, : is a label):

  my_func:
   &other_func1
   &other_func2
   &other_func3

  other_func1:
    &deeper1
  other_func2:
    &deeper2
  other_func3:
    &deeper3

  deeper1:
    push a
    ld a, 7
    ...

  deeper2:
    ...

Each function, which is extremely small because we've done the correct practice
of refactoring them, reside somewhere in memory like usual. Compilers typically
will then use a CPU's `call` instruction to jump to the address and set the
return register. Forth instead opts to place these addresses to functions one
after the other. A function call may do tens to hundreds of lookups - so yes,
the trade-off is slower code... except you can decide to switch out that model
of function calling! It's a win-win. There is an even more compact variant
called Huffman Threading and it's exactly what it describes: using Huffman codes
to create indexes to functions. On the opposite end a developer can invoke
particular Forth compilers, such as gForth, to use familiar code optimizations
found in LLVM and other code generators, to generate fast code.

The best part of the above is you get the advantages by doing good practice.
Refactoring code into small composable functions is what everyone strives for in
the first place!


               ENLIGHTENMENT GAINED FROM EXPLICIT DATA-FLOW

Time for a little detour into the higher level topic of data-flow.

Being a stack-based language, Forth passes arguments to functions using the
stack. Yes, pretty much every other language in existence also does this, but
usually only after registers are exhausted, and it's implicit. Never does a
programmer explicitly need to say "this goes on the stack", it just sort of
happens, and is placed in whatever position is deemed most optimal by the
compiler.

In Forth once again the opposite is true: argument passing on the stack is
explicit! The most compelling reason for this is how natural it is to pass an
arbitrary amount of arguments to a function this way, and how portable that is.

As long as the target machine has memory that can be linearly accessed, it can
express a stack machine. The original Turing machine is essentially a stack
machine. There is something inherently natural about these machines.

After you use Forth for a little you begin to realize, there is no logical sense
to have it reversed: a computer cannot be told to do something, expecting input
to act on, without the input!

It makes way more sense to have that data preceed the operation: 4 7 +.
Personally seeing and taking this fact in for the first time kind of unlocked
something in my head. I'm led to try algebra this way. I'm led to doing every
day quick math this way. No more does order of operations or parenthesis matter,
the order is baked into the expression. And there's a trick to reading
expressions written this way: you can read them like operations consuming their
operators, sweeping your eyes from right to left.

With explict data-flow, value copies and reuses become extremely obvious. They
become painful to see in code, similar to when encountering `.unwrap`s in Rust.
There is a new drive to remove any copies, because it causes the stack to thrash
more. You  literally feel it in your brain when trying to reason about the stack
manipulations. These new experiences teach that we've had the blinders on this
whole time on data-flow.


                             AN ALTERNATIVE TO

                            An alternative to C

Python, JavaScript, Ruby being interpreted languages are incapable of writing
low level code to poke at memory registers, use pointers or inline assembly.

For years these needs have been fulfilled by C and C++, followed by some niche
languages like D. Within the last 10 years there has been an explosion of
contenders for a better systems / low-level programming language, with Rust and
Zig being the most well-known.

Meanwhile Forth has been busy benchwarming. Never do I see it recommended
as an alternative to C in programming discussions. It's surprising, because 
Forth  has the same low-level mechanisms as C and can express the same high
level constructs. It has pointers. It can poke arbitrary memory. The default
memory allocation model, the dictionary, can grow arbitrarily big, unlike C's
`malloc`. Forth has `alloc` too, but it's heavily discouraged. Sticking to
growing the dictionary forces the programmer to construct their program in such
a way that it only uses the memory it absolutely needs at any point in
execution.

All these C alternatives are also missing another Forth superpower: the REPL.
(Just like LISP!)

Forth is designed to play with hardware at run-time, being able to poke memory
with the keyboard as the program runs. The closest thing we've ever had to that
I think is Terry Davis's HolyC, that provides a C shell (lol), and could allow
the same technique.

And there's all the benefits that have been mentioned in previous sections:
cheap, fast, compact, simple, portable.


                           An alternative to LISP

If you've gotten this far, and love LISP, you have probably seen many parallels
to LISP throughout the essay. especially after the mention of REPL in the
previous section. The evidence is strong enough for me to believe that Forth
is some sort of LISP sibling.

LISP's birth took place in the sterile environment of academia at MIT, as a
language to further develop artificial intelligence. Forth was born in the
messy work world, with ever changing requirements and environments. I tell
people that Forth is a "working man's LISP", because of how closely it looks
like a LISP but born out of completely different circumstances.

Here are some direct code comparisons for a quick visual:

                   (+ 1 2 3 4 5) vs 5 4 3 2 1 + + + +

       (defun my_func (a b) ...) vs : my_func ( a b -- ) ... ;

                        'my_func vs 'add

    (LOOP (PRINT (EVAL (READ)))) vs BEGIN ACCEPT EVALUATE TYPE AGAIN


Is that not the just the coolest thing to see in awhile? A simple, fast,
low-level LISP is possible! It's called Forth! With a skip and a hop you can get
LISP-like list operations very easily:

  : E+ 1 - 0 do + loop ;
  1 2 3 4 5 5 E+


Now take into account how there is no garbage collection. Inline assembly is
possible. It is even more syntax-less than LISP. These are aspects I find myself
desiring more than hygenic macros, or data structures (lists) that also define
the language. Don't interpret that as me saying LISP sucks. It's me saying
I'm so incredibly glad there is a LISP-like with trade-offs I'm happy with. 
Forth is a chimera of the Turing Machine and the Lambda Calculus.


                         UNSUITABLE CLASSES OF USE

The purpose of this section is to trick the reader into thinking of their own
reasons why they would use Forth.

Imagine a venn diagram with these aspects:

                        Against       Both       For
                                    |        |
                      security      |        |
                      memory safety | cost   |   ???
                      logical rigor | no-ast |
                      teamwork      |        |

Since file formats parsed by Forth are Forth programs themselves, we can't
really say Forth is suitable for security, since someone could simply craft a
file that is also some malicious Forth code. On the opposite end, if you work
in a secure environment, this is no longer a problem. So Forth is suitable to
use in all secure environments. 

Working in a large team can be difficult with a growing list of vocabulary in
a custom Forth DSL. In contrast, Forth is an excellent language to work as a 
solo developer, allowing you to work in your own comfy mental models.

Type systems and logical constructions are not concepts available to a Forth
programmer - which allows them to achieve some of the most unholy logic ever,
and get away with it! Of course, they can always develop their own type system
to work within.

While the grand total cost of Forth is significantly less than pretty much any
other mainstream language, the human costs can be much higher. A C or JavaScript
programmer cost significantly less because they can use abstractions that are
consistent across thousands of codebases. We can ignore the underlying
technology cost "just because". We have agreed as a society, or at least with
some sort of inherent bias, to using free compilers and interpreters for the
most part. This is why Forth is generally rarely, if ever, suitable for a 
business, but thrives in individualism.

Those examples hopefully convey the duality of Forth's weaknesses.


                            CLOSING COMMENTARY

If there's one thing I'd like to drive home, it's the natural evolution of
Forth from the primoridial assembly ooze, and all the power for the low cost
it brings along.

I look forward to what challenges the next generation of Forth programmers will
face, and the beautiful spawn that will rise from them.


                  An Attempt at a Compelling Articulation
                      of Forth's Practical Strengths
                          and Eternal Usefulness

                          Thank you for reading.

                            - Lee

    Special thanks to paultag, austin, appledash, and jon, for peer reviewing!

                          2025-11-09T21:25-04:00

François Marier: Learning a new programming language with an LLM

PlanetDebian
feeding.cloud.geek.nz
2025-12-08 00:15:00
I started learning Go this year. First, I picked a Perl project I wanted to rewrite, got a good book and ignored AI tools since I thought they would do nothing but interfere with learning. Eventually though, I decided to experiment a bit and ended up finding a few ways to use AI assistants effective...
Original Article

I started learning Go this year. First, I picked a Perl project I wanted to rewrite, got a good book and ignored AI tools since I thought they would do nothing but interfere with learning . Eventually though, I decided to experiment a bit and ended up finding a few ways to use AI assistants effectively even when learning something new.

Searching more efficiently

The first use case that worked for me was search. Instead of searching on a traditional search engine and then ending up on Stack Overflow , I could get the answer I was looking for directly in an AI side-window in my editor. Of course, that's bad news for Stack Overflow.

I was however skeptical from the beginning since LLMs make mistakes, sometimes they making up function signatures or APIs that don't exist. Therefore I got into the habit of going to the official standard library documentation to double-check suggestions. For example, if the LLM suggests using strings.SplitN , I verify the function signature and behaviour carefully before using it. Basically, " don't trust and do verify."

I stuck to the standard library in my project , but if an LLM recommends third-party dependencies for you, make sure they exist and that Socket doesn't flag them as malicious. Research has found that 5-20% of packages suggested by LLMs don't actually exist , making this a real attack vector (dubbed "slopsquatting").

Autocomplete is too distracting

A step I took early on was to disable AI autocomplete in my editor. When learning a new language, you need to develop muscle memory for the syntax. Also, Go is no Java. There's not that much boilerplate to write in general.

I found it quite distracting to see some almost correct code replace my thinking about the next step. I can see how one could go faster with these suggestions, but being a developer is not just about cranking out lines of code as fast as possible, it's also about constantly learning new things (and retaining them).

Asking about idiomatic code

One of the most useful prompts when learning a new language is "Is this the most idiomatic way to do this in Go?". Large language models are good at recognizing patterns and can point out when you're writing code that works but doesn't follow the conventions of the language. This is especially valuable early on when you don't yet have a feel for what "good" code looks like in that language.

It's usually pretty easy (at least for an experience developer) to tell when the LLM suggestion is actually counter productive or wrong. If it increases complexity or is harder to read/decode, it's probably not a good idea to do it.

Reviews

One way a new dev gets better is through code review. If you have access to a friend who's an expert in the language you're learning, then you can definitely gain a lot by asking for feedback on your code.

If you don't have access to such a valuable resource, or as a first step before you consult your friend, I found that AI-assisted code reviews can be useful:

  1. Get the model to write the review prompt for you. Describe what you want reviewed and let it generate a detailed prompt.
  2. Feed that prompt to multiple models. They each have different answers and will detect different problems.
  3. Be prepared to ignore 50% of what they recommend. Some suggestions will be stylistic preferences, others will be wrong, or irrelevant.

The value is in the other 50%: the suggestions that make you think about your code differently or catch genuine problems.

Similarly for security reviews:

  • A lot of what they flag will need to be ignored (false positives, or things that don't apply to your threat model).
  • Some of it may highlight areas for improvement that you hadn't considered.
  • Occasionally, they will point out real vulnerabilities.

But always keep in mind that AI chatbots are trained to be people-pleasers and often feel the need to suggest something when nothing was needed

An unexpected benefit

One side effect of using AI assistants was that having them write the scaffolding for unit tests motivated me to increase my code coverage. Trimming unnecessary test cases and adding missing ones is pretty quick when the grunt work is already done, and I ended up testing more of my code (being a personal project written in my own time) than I might have otherwise.

In the end, I continue to believe in the value of learning from quality books (I find reading paper-based most effective). In addition, I like to create Anki questions for common mistakes or things I find I have to look up often. Remembering something will always be faster than asking an AI tool.

So my experience this year tells me that LLMs can supplement traditional time-tested learning techniques, but I don't believe it obsoletes them.

P.S. I experimented with getting an LLM to ghost-write this post for me from an outline (+ a detailed style guide ) and I ended up having to rewrite at least 75% of it. It was largely a waste of time.

Chris Hani’s Murder Robbed South Africa of a Great Leader

Portside
portside.org
2025-12-07 23:55:12
Chris Hani’s Murder Robbed South Africa of a Great Leader Ira Sun, 12/07/2025 - 18:55 ...
Original Article

The assassination of Chris Hani in April 1993 was a decisive moment in South Africa’s transition to democracy. Nelson Mandela used this tragic event to pressure President F. W. De Klerk to conclude negotiations and announce a date for the elections that would bring the African National Congress (ANC) to power the following year, marking the formal end of apartheid.

Hani was born in 1942, the same year as two of Mandela’s successors as president, Thabo Mbeki and Jacob Zuma. While Mbeki and Zuma both remain alive and politically active today, more than thirty years later, Hani did not live long enough to hold office in the new South Africa. He is nevertheless remembered today as one of the greatest of South Africans.

He played a leading role in three anti-apartheid organizations: the ANC, the South African Communist Party (SACP), and the armed wing that the ANC and SACP formed together in the 1960s, uMkhonto weSizwe (usually known as MK). By discussing in more detail his role in these organizations, we can show why his death was such a great loss to the new South Africa.

Background and Education

Hani was born in the Transkei, now the Eastern Cape. He attended Catholic primary schools in the Transkei and did his matriculation exams at Lovedale at the early age of sixteen before moving on to the University College of Fort Hare. He graduated at nineteen, having taken courses in Latin and English literature, as well as the Greek classics in English. His parents discouraged him from seeking ordination as a Catholic priest, but he valued his classical education.

He had an inherited interest in the ANC as his father, Gilbert, a labor migrant who became a small trader in Cape Town, was an active member. His uncle, Milton Hani, was also an active member of the Communist Party of South Africa, which was the SACP’s legal predecessor before the authorities banned it in 1950. His own political education began at school at Lovedale where he was drawn firstly toward the Sons of Young Africa, the youth wing of the Trotskyist Non-European Unity Movement, and then to the ANC Youth League.

It was at Fort Hare that Hani joined a Marxist study group under the influence of Govan Mbeki, who was a leader of both the ANC and the SACP. South Africa’s Communists had reconstituted their party in 1953 as a clandestine organization after the government ban; the ANC was also outlawed in 1960. Hani’s study group read the Communist Manifesto and Emile Burns’s What is Marxism? , and he enlisted in an underground cell of the SACP during his time in Fort Hare.

After graduation in 1961, Hani joined his father in Cape Town and became an articled clerk in a firm of attorneys. He soon became a member of MK and did elementary military training. In 1963, he moved north to Johannesburg and then through Bechuanaland (now Botswana) and Northern Rhodesia (now Zambia) to Tanganyika (now Tanzania).

From there, he traveled with other activists to the Soviet Union, where he spent a year, undergoing military training and political education in Moscow. He traveled widely in the USSR and especially valued his exposure to ballet, opera, and Russian literature.

The Wankie Campaign

In 1965, Hani returned to newly independent Zambia, where he became one of a group of ANC and MK leaders who were planning a return to South Africa. When it became clear in 1966 that newly independent Botswana would not provide a transit route for freedom fighters to South Africa, the ANC leadership resolved to form an alliance with the Zimbabwean African Political Union, seeking to open a route through Rhodesia, which was ruled by the white-settler dictatorship of Ian Smith.

A group of seventy-nine men, mainly MK members, crossed the Zambezi river near Livingstone on July 31, 1967, in what came to be known as the Wankie Campaign. Hani was one of the leaders of a group who first clashed with Rhodesian forces three weeks after crossing the Zambezi.

In the wake of another clash, he led a group of about twenty men who took refuge in Botswana where they surrendered to the local paramilitary police. The authorities charged them with having entered Botswana carrying weapons of war and they received two-year prison sentences, although they were released after one year.

Hani returned to Zambia with others in September 1968. Although the members of what was called the Luthuli Detachment had failed in their military objectives, they had shown great bravery and sustained heavy losses. Hani was later certain that the campaign was a good example of armed propaganda and helped to inspire resistance within South Africa, including the Black Consciousness Movement, which had its origins in the late 1960s.

The Hani Memorandum

Hani’s role in the Wankie Campaign earned him a reputation for physical courage. It was his role in the campaign’s aftermath that also gave him a reputation for moral courage. He became the lead signatory, one of seven, of a three-thousand-word document that was written in January 1969 and became known as the “Hani Memorandum.” The opening sentence stated: “The ANC in Exile is in deep crisis as a result of which a rot has set in.”

Although the document did not attack the ANC president Oliver Tambo personally, this was a devastating critique of the Congress leadership as a whole and of its failure to recognize the “heroes and martyrs” of the Wankie Campaign and the subsequent, equally unsuccessful Sipolilo Campaign. The main targets of the memorandum were Joe Modise, MK’s commander-in-chief, and Duma Nokwe, ANC secretary-general and head of the security department. The authors saw the security department as being more closely focused on detecting subversion among the membership than on defending the organization against external attack.

The memorandum concluded with a demand for a conference to discuss the issues it raised. Modise and Nokwe responded with fury. When the signatories refused to attend a tribunal that they considered to be stacked against them, they were first suspended and then expelled from the ANC. They were even threatened with execution, and Hani believed that his life was in danger. He withdrew for a while to the Zambian Copperbelt where he stayed with expatriate friends.

However, Tambo agreed to hold the conference that the memorandum demanded, which took place in the Tanzanian city of Morogoro in April 1969. While Hani and the other signatories were unable to attend as they were no longer ANC members, the conference did recommend their reinstatement, which took effect in June. Modise was demoted, though he remained commander of MK, and Nokwe was replaced as secretary-general and head of security.

Among the important policy decisions of the conference were the formal opening of membership of the ANC and its national executive to white, colored, and Indian members. This brought the Congress in line with the practice of MK and signified the end of the multiracial Congress Alliance and a step towards non-racialism. The conference also adopted a new “Strategy and Tactics” document, which, in response to the Memorandum, specifically rejected “militarism” and emphasized the primacy of political struggle.

Political Work From Lesotho

While Hani’s reinstatement did not occur without controversy, he was rapidly promoted to positions of leadership. He became the SACP’s deputy-general-secretary in 1972, and was elected to the ANC’s National Executive Committee (NEC) in 1974, along with Thabo Mbeki. He began to play a diplomatic role and traveled to Scandinavia in 1972–73, establishing close links with Sweden.

In September 1974, he became the first serving member of the NEC to enter South Africa for underground work, although he was unable to stay long. He moved on to Lesotho, a country where he had connections through Limpho Sekamane, who he had recently married, and through his father, Gilbert. Hani Sr had been running a café in Mafeteng with his partner, Elizabeth Mafikeng, a trade unionist, since 1963.

Hani was to remain based in Lesotho until 1982. Although he carried out some recruiting for MK, he did very little military work during this time. Most of his activity involved political liaisons with established and newly emerging trade unions; with Black Consciousness organizations, such as the Port Elizabeth Black Civic Organization; and with the opposition to the Bantustan government in the Transkei. Hani and the Lesotho branch of the ANC were putting into practice the Morogoro Conference’s resolution about the primacy of political work.

His stay in Lesotho became increasingly dangerous, and there were attempts on his life in 1981 and 1982. The ANC eventually withdrew Hani from the country at the request of its government in 1982. His wife and family were lucky to escape death in December of that year when there was a massive South African raid on Lesotho’s capital, Maseru, which killed many ANC members and local citizens.

Military Work and Negotiations

In 1983, Hani was appointed deputy commander and political commissar of MK, and he became the organization’s chief of staff in 1987. In these capacities, he traveled frequently between Angola, Zambia, and Mozambique. However, the Nkomati Accord of March 1984 between the Mozambican government and the apartheid regime meant that he was henceforth excluded from the country, and MK later had to withdraw from its Angolan camps to Uganda in 1988.

In the early months of 1984, Hani had to contend with two major mutinies in the Angolan camps, which were prompted by poor living conditions; MK casualties in fighting with UNITA, which was engaged in a civil war with the Angolan government; and the frustration of cadres who saw no prospect of getting into action in South Africa. While Hani could not escape some responsibility for the crises in Angola, he and Joe Slovo had an image of being on the side of the troops, in contrast with Joe Modise, the MK commander, and the hated security apparatus.

His continuing popularity was demonstrated when he came top of the poll in the NEC elections at the Kabwe Consultative Conference in 1985. MK activity in South Africa reached a peak in 1988 and then declined as pressure on its bases in the frontline states increased.

In 1985, the movement toward a negotiated settlement began after the meeting of ANC leaders with the Anglo-American delegation of South African businessmen and journalists, as guests of Zambian President Kenneth Kaunda, in the Luangwa National Park. Hani represented MK and made only one intervention in the discussions, saying that while the ANC was accused of violence, it was the government of President P. W. Botha that was the truly violent actor.

In the years leading up to the ANC’s unbanning and the release of Mandela in February 1990, and during the subsequent period of negotiations, observers generally portrayed Hani as a hard-line figure in contrast with Thabo Mbeki, the conciliator. Although he was sometimes involved in the unfolding process — for example, through participating in the CODESA talks in December 1991 — he did not play a major role overall. He and Slovo acted with Cyril Ramaphosa, the ANC secretary-general, to remove Mbeki and Jacob Zuma from leadership of the talks in August 1991 on the grounds that they were moving too slowly.

In December 1991, Hani stepped down as MK chief of staff and as a member of the ANC’s National Working Committee, taking over from Slovo as general secretary — in other words, leader — of the SACP. This came at a moment when the Soviet Union was disintegrating, and half the members of the SACP’s central committee (including Mbeki and Zuma) had resigned from the party, not wishing to be identified as communists. Yet Hani, with characteristic courage, took on the leadership role.

Crisis of Socialism

He had always been politically close to Slovo, and the two men had in general welcomed the ascension of Mikhail Gorbachev and his reform agenda of glasnost and perestroika . Hani agreed with the thrust of Slovo’s pamphlet , Has Socialism Failed?, published in January 1990, in which he argued that Soviet communism was a distortion of socialist ideals, but that its failure did not discredit those ideals as such.

In an interview conducted shortly before his death, Hani said that while he and his comrades might have been blind not to see the lack of democracy in the Soviet Union, socialism was no more invalidated by the bad things done in its name than Christianity had been. He did not believe that the crisis of international socialism represented the “end of history.”

Hani was a progressive on many issues, including his support for feminism, his response to the HIV/AIDS pandemic, and his views on rural feudalism and the role of chiefs. He saw the function of the SACP as the promotion of “democratic socialism” within the ANC and of the “national democratic revolution” in South Africa as a whole.

There was room in this framework for “born-again socialists and born-again communists,” committed to pluralism and a multiparty system. Hani and his co-thinkers did not wish to impose socialism, and they rejected concepts such as that of the “dictatorship of the proletariat.” He had personally come to accept that there was a role for the market in economic life, but he argued that “the great majority of South Africa’s people are not even in the market.”

As Hani acknowledged, he was in the unusual position of heading a communist party that was growing rapidly at a time when most communist parties in the rest of the world were in decline. However, he rejected the suggestion of his friend Wolfie Kodesh that he might want to lead the SACP into an oppositional stance toward the ANC.

It is not entirely clear why he decided, not long before his death, that he would not participate in the projected government of national unity, which would bring together the ANC and the outgoing nationalist party for a period of up to five years. It is possible that he had doubts about the viability of the government of national unity, which ultimately lasted less than two years after being established a year after his assassination.

A Lost Leader

Hani had always been more skeptical about the negotiating process than his rival Thabo Mbeki, and he took a harder line as it unfolded. He appears to have believed that it was important for him, as the SACP’s leader, to assume an independent and critical role in the transitional phase: “The perks of a new government are not really appealing to me . . . the real problems of the country are not whether one is in cabinet, or a key minister, but what we do for social upliftment of the working masses of our people.”

There are still unresolved questions about Hani’s murder in 1993. Were the two men found guilty of his assassination, Janusz Waluś and Clive Derby-Lewis, acting independently, or were they acting, as seems likely, as agents of the apartheid state? We do not know for sure.

It is also impossible to say what the wider political consequences of his death proved to be. Hani was the ANC’s most popular and highly respected leader, receiving 95 percent of the vote in the NEC elections at the first ANC conference held inside the country in July 1991.

Would he, and not Mbeki, have succeeded Nelson Mandela as president of South Africa in 1999, and would that have affected the course of South African history? All that we can say for sure is that his death represented an incalculable loss to South Africa.

Hugh Macmillan is a prominent South African historian, researcher, and author known for his extensive work on Southern African history, particularly the African National Congress (ANC) in exile, business history (like Susman Brothers & Wulfsohn), and biographies of key figures (Chris Hani, Oliver Tambo) . He taught at universities in Zambia, Swaziland, and South Africa before becoming a Research Associate at Oxford University's African Studies Centre and Senior Research Associate at the University of Johannesburg.

Jacobin is a leading voice of the American left, offering socialist perspectives on politics, economics, and culture. The print magazine is released quarterly and reaches 75,000 subscribers, in addition to a web audience of over 3,000,000 a month.

Subscribe to Jacobin today , get four beautiful editions a year, and help us build a real, socialist alternative to billionaire media.

Since 2010, Jacobin has emerged as a leading voice of the American left. Every month we reach millions of people, most of whom browse online for free.

We are committed to ensuring that socialist thought is both as rigorously produced as possible and available for public use. Over the past year, our website registered 20 million visitors, offering them perspectives to not just understand, but change the world.

Over the next year, we’ll produce 2,500 original essays, release paperback books, start reading groups in new locations, and host an international event series.

But even as our audience grows, with rising postal costs and all the other expenses involved with producing a print magazine, we have to count on you, our most committed readers, to thrive.

As a 501(c)(3) nonprofit organization, donations to Jacobin Foundation are tax-deductible. If you are a grant maker or would like to learn about other ways to give, including bequests or stocks and bonds, please contact us at development jacobin.com.

Donate to Jacobin.

Pulldash: Fast, filterable GitHub PR review. Entirely client-side

Lobsters
github.com
2025-12-07 23:18:13
Comments...
Original Article

pulldash logo Pulldash

GitHub Release GitHub License GitHub Actions Workflow Status

Fast, filterable PR review. Entirely client-side.

Warning

Pulldash is WIP. Expect bugs.

Try It

Browser : pulldash.com . Replace github.com with pulldash.com in any PR URL.

Desktop : Latest release for Linux, macOS, Windows.

Example

Features

  • Custom filters : Add repos and filter by review requests, authored PRs, or all activity.

    Filtering PRs

  • Keyboard-driven : j / k to navigate files, arrows for lines, c to comment, s to submit.

    Keybinds

  • Fast file search : Ctrl+K to fuzzy-find across hundreds of changed files.

    Search

Why

  • GitHub's review UI is slow (especially for large diffs)
  • No central view to filter PRs you care about
  • AI tooling has produced more PRs than ever before—making a snappy review UI essential

How It Works

GitHub's API supports CORS , so Pulldash runs entirely client-side. No backend proxying your requests.

  • Web Worker pool : Diff parsing and syntax highlighting run in workers sized to navigator.hardwareConcurrency . The main thread stays free for scrolling.

  • Pre-computed navigation : When a diff loads, we index all navigable lines. Arrow keys are O(1)—no DOM queries.

  • External store : State lives outside React ( useSyncExternalStore ). Focusing line 5000 doesn't re-render the file tree.

  • Virtualized rendering : Diffs, file lists, and the command palette only render visible rows.

Development

License

AGPL

Bag of words, have mercy on us

Hacker News
www.experimental-history.com
2025-12-07 22:31:22
Comments...
Original Article
photo cred: my dad

Look, I don’t know if AI is gonna kill us or make us all rich or whatever, but I do know we’ve got the wrong metaphor.

We want to understand these things as people . When you type a question to ChatGPT and it types back the answer in complete sentences, it feels like there must be a little guy in there doing the typing. We get this vivid sense of “ it’s alive!! ”, and we activate all of the mental faculties we evolved to deal with fellow humans: theory of mind , attribution , impression management , stereotyping , cheater detection , etc.

We can’t help it; humans are hopeless anthropomorphizers. When it comes to perceiving personhood, we’re so trigger-happy that we can see the Virgin Mary in a grilled cheese sandwich :

A human face in a slice of nematode:

And an old man in a bunch of poultry and fish atop a pile of books:

Apparently, this served us well in our evolutionary history—maybe it’s so important not to mistake people for things that we err on the side of mistaking things for people . 1 This is probably why we’re so willing to explain strange occurrences by appealing to fantastical creatures with minds and intentions: everybody in town is getting sick because of WITCHES, you can’t see the sun right now because A WOLF ATE IT, the volcano erupted because GOD IS MAD. People who experience sleep paralysis sometimes hallucinate a demon-like creature sitting on their chest, and one explanation is that the subconscious mind is trying to understand why the body can’t move, and instead of coming up with “I’m still in REM sleep so there’s not enough acetylcholine in my brain to activate my primary motor cortex”, it comes up with “BIG DEMON ON TOP OF ME”.

This is why the past three years have been so confusing—the little guy inside the AI keeps dumbfounding us by doing things that a human wouldn’t do. Why does he make up citations when he does my social studies homework? How come he can beat me at Go but he can’t tell me how many “r”s are in the word “strawberry” ? Why is he telling me to put glue on my pizza ? 2

Trying to understand LLMs by using the rules of human psychology is like trying to understand a game of Scrabble by using the rules of Pictionary. These things don’t act like people because they aren’t people. I don’t mean that in the deflationary way that the AI naysayers mean it. They think denying humanity to the machines is a well-deserved insult; I think it’s just an accurate description. 3 As long we try to apply our person perception to artificial intelligence, we’ll keep being surprised and befuddled.

We are in dire need of a better metaphor. Here’s my suggestion: instead of seeing AI as a sort of silicon homunculus, we should see it as a bag of words.

An AI is a bag that contains basically all words ever written, at least the ones that could be scraped off the internet or scanned out of a book. When users send words into the bag, it sends back the most relevant words it has. There are so many words in the bag that the most relevant ones are often correct and helpful, and AI companies secretly add invisible words to your queries to make this even more likely.

This is an oversimplification, of course. But it’s also surprisingly handy. For example, AIs will routinely give you outright lies or hallucinations, and when you’re like “Uhh hey that was a lie”, they will immediately respond “Oh my god I’m SO SORRY!! I promise I’ll never ever do that again!! I’m turning over a new leaf right now, nothing but true statements from here on” and then they will literally lie to you in the next sentence. This would be baffling and exasperating behavior coming from a human, but it’s very normal behavior coming from a bag of words. If you toss a question into the bag and the right answer happens to be in there, that’s probably what you’ll get. If it’s not in there, you’ll get some related-but-inaccurate bolus of sentences. When you accuse it of lying, it’s going to produce lots of words from the “I’ve been accused of lying” part of the bag. Calling this behavior “malicious” or “erratic” is misleading because it’s not behavior at all, just like it’s not “behavior” when a calculator multiplies numbers for you.

“Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. “Give me a list of the ten worst transportation disasters in North America” is an easy task for a bag of words, because disasters are well-documented. On the other hand, “Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is a hard task for a bag of words, because the bag just doesn’t contain that many words on the topic. 4 And a question like “What are the most important lessons for life?” won’t give you anything outright false, but it will give you a bunch of fake-deep pablum, because most of the text humans have produced on that topic is, no offense, fake-deep pablum.

When you forget that an AI is just a big bag of words, you can easily slip into acting like it’s an all-seeing glob of pure intelligence. For example, I was hanging with a group recently where one guy made everybody watch a video of some close-up magic, and after the magician made some coins disappear, he exclaimed, “I asked ChatGPT how this trick works, and even it didn’t know!” as if this somehow made the magic extra magical. In this person’s model of the world, we are all like shtetl-dwelling peasants and AI is like our Rabbi Hillel, the only learned man for 100 miles. If Hillel can’t understand it, then it must be truly profound!

If that guy had instead seen ChatGPT as a bag of words, he would have realized that the bag probably doesn’t contain lots of detailed descriptions of contemporary coin tricks. After all, magicians make money from performing and selling their tricks, not writing about them at length on the internet. Plus, magic tricks are hard to describe—“He had three quarters in his hand and then it was two pennies!”—so you’re going to have a hard time prompting the right words out of the bag. The coin trick is not literally magic, and neither is the bag of words.

The “bag of words” metaphor can also help us guess what these things are gonna do next. If you want to know whether AI will get better at something in the future, just ask: “can you fill the bag with it?” For instance, people are kicking around the idea that AI will replace human scientists . Well, if you want your bag of words to do science for you, you need to stuff it with lots of science. Can we do that?

When it comes to specific scientific tasks, yes, we already can. If you fill the bag with data from 170,000 proteins, for example, it’ll do a pretty good job predicting how proteins will fold. Fill the bag with chemical reactions and it can tell you how to synthesize new molecules . Fill the bag with journal articles and then describe an experiment and it can tell you whether anyone has already scooped you .

All of that is cool, and I expect more of it in the future. I don’t think we’re far from a bag of words being able to do an entire low-quality research project from beginning to end—coming up with a hypothesis, designing the study, running it, analyzing the results, writing them up, making the graphs, arranging it all on a poster, all at the click of a button—because we’ve got loads of low-quality science to put in the bag. If you walk up and down the poster sessions at a psychology conference, you can see lots of first-year PhD students presenting studies where they seemingly pick some semi-related constructs at random, correlate them, and print out a p-value (“Does self-efficacy moderate the relationship between social dominance orientation and system-justifying beliefs?”). A bag of words can basically do this already ; you just need to give it access to an online participant pool and a big printer. 5

But science is a strong-link problem ; if we produced a million times more crappy science, we’d be right where we are now. If we want more of the good stuff, what should we put in the bag? You could stuff the bag with papers, but some of them are fraudulent, some are merely mistaken, and all of them contain unstated assumptions that could turn out to be false. And they’re usually missing key information—they don’t share the data, or they don’t describe their methods in adequate detail. Markus Strasser, an entrepreneur who tried to start one of those companies that’s like “we’ll put every scientific paper in the bag and then ??? and then profit”, eventually abandoned the effort , saying that “close to nothing of what makes science actually work is published as text on the web.” 6

Here’s one way to think about it: if there had been enough text to train an LLM in 1600, would it have scooped Galileo? My guess is no. Ask that early modern ChatGPT whether the Earth moves and it will helpfully tell you that experts have considered the possibility and ruled it out . And that’s by design. If it had started claiming that our planet is zooming through space at 67,000mph, its dutiful human trainers would have punished it: “Bad computer!! Stop hallucinating!!”

In fact, an early 1600s bag of words wouldn’t just have the right words in the wrong order. At the time, the right words didn’t exist . As the historian of science David Wootton points out 7 , when Galileo was trying to describe his discovery of the moons of Jupiter, none of the languages he knew had a good word for “discover”. He had to use awkward circumlocutions like “I saw something unknown to all previous astronomers before me”. The concept of learning new truths by looking through a glass tube would have been totally foreign to an LLM of the early 1600s, as it was to most of the people of the early 1600s, with a few notable exceptions.

You would get better scientific descriptions from a 2025 bag of words than you would from a 1600 bag of words. But both bags might be equally bad at producing the scientific ideas of their respective futures. Scientific breakthroughs often require doing things that are irrational and unreasonable for the standards of the time and good ideas usually look stupid when they first arrive, so they are often—with good reason!—rejected, dismissed, and ignored. This is a big problem for a bag of words that contains all of yesterday’s good ideas. Putting new ideas in the bag will often make the bag worse, on average, because most of those new ideas will be wrong. That’s why revolutionary research requires not only intelligence, but also stupidity . I expect humans to remain usefully stupider than bags of words for the foreseeable future.

The most important part of the “bag of words” metaphor is that it prevents us from thinking about AI in terms of social status . Our ancestors had to play status games well enough to survive and reproduce—losers, by and large, don’t get to pass on their genes. This has left our species exquisitely attuned to who’s up and who’s down. Accordingly, we can turn anything into a competition: cheese rolling , nettle eating , phone throwing , toe wrestling , and ferret legging , where male contestants, sans underwear, put live ferrets in their pants for as long as they can. (The world record is five hours and thirty minutes .)

When we personify AI, we mistakenly make it a competitor in our status games. That’s why we’ve been arguing about artificial intelligence like it’s a new kid in school: is she cool? Is she smart? Does she have a crush on me? The better AIs have gotten, the more status-anxious we’ve become. If these things are like people, then we gotta know: are we better or worse than them? Will they be our masters, our rivals, or our slaves? Is their art finer, their short stories tighter, their insights sharper than ours? If so, there’s only one logical end: ultimately, we must either kill them or worship them.

But a bag of words is not a spouse, a sage, a sovereign, or a serf. It’s a tool. Its purpose is to automate our drudgeries and amplify our abilities. Its social status is NA; it makes no sense to ask whether it’s “better” than us. The real question is: does using it make us better?

That’s why I’m not afraid of being rendered obsolete by a bag of words. Machines have already matched or surpassed humans on all sorts of tasks. A pitching machine can throw a ball faster than a human can, spellcheck gets the letters right every time, and autotune never sings off key. But we don’t go to baseball games, spelling bees, and Taylor Swift concerts for the speed of the balls, the accuracy of the spelling, or the pureness of the pitch. We go because we care about humans doing those things. It wouldn’t be interesting to watch a bag of words do them—unless we mistakenly start treating that bag like it’s a person.

(That’s also why I see no point in using AI to, say, write an essay, just like I see no point in bringing a forklift to the gym. Sure, it can lift the weights, but I’m not trying to suspend a barbell above the floor for the hell of it. I lift it because I want to become the kind of person who can lift it. Similarly, I write because I want to become the kind of person who can think.)

But that doesn’t mean I’m unafraid of AI entirely. I’m plenty afraid! Any tool can be dangerous when used the wrong way—nail guns and nuclear reactors can kill people just fine without having a mind inside them. In fact, the “bag of words” metaphor makes it clear that AI can be dangerous precisely because it doesn’t operate like humans do. The dangers we face from humans are scary but familiar: hotheaded humans might kick you in the head, reckless humans might drink and drive, duplicitous humans might pretend to be your friend so they can steal your identity. We can guard against these humans because we know how they operate. But we don’t know what’s gonna come out of the bag of words. For instance, if you show humans computer code that has security vulnerabilities, they do not suddenly start praising Hitler. But LLMs do . 8 So yes, I would worry about putting the nuclear codes in the bag. 9

Anyone who has owned an old car has been tempted to interpret its various malfunctions as part of its temperament . When it won’t start on a cold day, it feels like the appropriate response is to plead, the same way you would with a sleepy toddler or a tardy partner: “C’mon Bertie, we gotta get to the dentist!” But ultimately, person perception is a poor guide to vehicle maintenance. Cars are made out of metal and plastic that turn gasoline into forward motion; they are not made out of bones and meat that turn Twinkies into thinking. If you want to fix a broken car, you need a wrench, a screwdriver, and a blueprint, not a cognitive-behavioral therapy manual.

Similarly, anyone who sees a mind inside the bag of words has fallen for a trick. They’ve had their evolution exploited. Their social faculties are firing not because there’s a human in front of them, but because natural selection gave those faculties a hair trigger. For all of human history, something that talked like a human and walked like a human was, in fact, a human. Soon enough, something that talks and walks like a human may, in fact, be a very sophisticated logistic regression. If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper.

Unlike moths, however, we aren’t stuck using the instincts that natural selection gave us. We can choose the schemas we use to think about technology. We’ve done it before: we don’t refer to a backhoe as an “artificial digging guy” or a crane as an “artificial tall guy”. We don’t think of books as an “artificial version of someone talking to you”, photographs as “artificial visual memories”, or listening to recorded sound as “attending an artificial recital”. When pocket calculators debuted, they were already smarter than every human on Earth, at least when it comes to calculation—a job that itself used to be done by humans . Folks wondered whether this new technology was “a tool or a toy”, but nobody seems to have wondered whether it was a person .

(If you covered a backhoe with skin, made its bucket look like a hand, painted eyes on its chassis, and made it play a sound like “hnngghhh!” whenever it lifted something heavy, then we’d start wondering whether there’s a ghost inside the machine. That wouldn’t tell us anything about backhoes, but it would tell us a lot about our own psychology.)

The original sin of artificial intelligence was, of course, calling it artificial intelligence. Those two words have lured us into making man the measure of machine: “Now it’s as smart as an undergraduate...now it’s as smart as a PhD!” These comparisons only give us the illusion of understanding AI’s capabilities and limitations, as well as our own, because we don’t actually know what it means to be smart in the first place. Our definitions of intelligence are either wrong (“Intelligence is the ability to solve problems”) or tautological (“Intelligence is the ability to do things that require intelligence”). 10

It’s unfortunate that the computer scientists figured out how to make something that kinda looks like intelligence before the psychologists could actually figure out what intelligence is, but here we are. There’s no putting the cat back in the bag now. It won’t fit—there’s too many words in there.

PS it’s been a busy week on Substack—

and I discussed why people get so anxious about conversations, and how to have better ones:

And

at Can't Get Much Higher answered all of my questions about music. He uncovered some surprising stuff, including an issue that caused a civil war on a Beatles message board, and whether they really sang naughty words on the radio in the 1970s:

Derek and Chris both run terrific Substacks, check ‘em out!

How I block all online ads

Hacker News
troubled.engineer
2025-12-07 22:18:59
Comments...
Original Article

Ads support content creators and free services. If you value specific creators or platforms, consider supporting them directly through memberships or donations rather than relying solely on ad blocking.

Intro

A couple of years ago, I decided I'd had enough of ads. Not just the occasional banner or a quick pre-roll video — I mean all of them. They have to go.

So I embarked on a holy crusade to get rid of them as much as possible. I tried the obvious solutions first, then dug deeper into less conventional approaches. It took a long time, tons of experiments, and many observations, but today I am finally happy where I stand.

There are many techniques out there, some well-known and others surprisingly obscure. Here's what I learned over the years and what actually worked for me.

Let's start with the basics and work our way up to the more unconventional methods. The first few are straightforward and widely used. The later ones require more setup and maintenance but can block ads in places where traditional methods fail.

Browser extensions

Browser ad blockers are the biggest boycott in history . You're probably using one already!

I use Firefox with uBlock Origin — it's the best ad blocking combo out there. It's harder if you're on Chromium-based browser, since Google transitioned to Manifest V3 which conveniently limits ad blockers.

I keep my filter lists minimal — they cover almost everything I need:

  • Built-in uBlock filters
  • EasyList
  • AdGuard - Ads

I also maintain my own filters . They don't focus on ads, but rather on other annoyances.

DNS filtering

DNS filtering complements browser extensions by catching ads that slip through — particularly in mobile apps. Mobile apps typically load ads from dedicated ad-serving domains, making them straightforward to block at the DNS level.

Pi-hole and AdGuard Home are the most popular self-hosted options for this. If you're looking for a cloud-based solution, I don't use them myself, but I've heard good things about NextDNS .

I use Pi-hole, and it's been smooth so far. I don't expose it publicly — instead, I connect via WireGuard and set Pi-hole as the DNS server in my WireGuard config. If you're looking for blocklists, The Firebog is a great starting point. You'll also want to maintain an allowlist — blocklists occasionally include legitimate domains that break functionality on websites or in apps.

There are multiple ways to install Pi-hole , I keep it in Docker and suggest you do the same.

VPN via cloud

Now here comes a secret ingredient. If you route all your traffic through a popular cloud provider (via VPN or proxy), then many online platforms are less likely to show you ads.

That happens because to these platforms you look like a fraudster doing something sketchy with their ads. Imagine this scenario: a small business spends $1000 on ads. Their competitors figure out the targeting, mimic that behavior, spin up 10 VMs, and waste the entire advertising budget on fake interactions. The small business isn't coming back to spend more money on ads after that experience.

Online platforms are well aware of this, so they fight fraud. Not serving ads to traffic from public cloud providers is one of the first steps they take.

However, this will negatively affect your experience on some sites — you'll hit Cloudflare captchas and HTTP errors due to sites blocking cloud provider IPs. I'm fine with it and just turn the VPN off occasionally when something breaks. Just keep in mind that even a few requests with your real IP might be enough for an online platform to start showing you ads again.

I host WireGuard on a $5 DigitalOcean droplet, but Hetzner, Azure, Google Cloud, AWS, and others work just as well. DigitalOcean also provides a detailed guide on how to set it up.

Other useful stuff

Below you'll find some other useful things, although they aren't exactly related to ad-blocking:

  • Browser extensions against annoyances:
  • I'd also suggest SponsorBlock — it has saved me so much time. There's also an option for TVs and streaming devices .
  • If you're on iOS, consider turning off Background App Refresh . Only a few apps use Background App Refresh as Apple designed it, the majority are simply abusing it to get more data about you. If you don't have always-on VPN, you risk exposing your real IP.
  • Patched apps are also a thing, and it's also possible to patch mobile apps yourself via ReVanced . While it's a decent option, it's also a security risk — I'm careful with it and don't use it with sensitive accounts.

Personal experience

I've been using all these things mentioned above for over 3 years now. I barely see any ads nowadays. If you're curious about specifics, I keep track of what works where:

Platform Web iOS / Android
YouTube uBlock Origin NewPipe or Invidious
Instagram uBlock Origin VPN via cloud (takes a week to a month)
Twitch VPN via cloud (takes a few days) -
TikTok uBlock Origin VPN via cloud (takes a few hours)
Apps with AdMob - DNS blocking

These are the tricky outliers. For most sites and apps, DNS filtering and a browser ad blocker catch 99% of ads without any extra effort. The VPN approach helps with that remaining 1%, though it usually takes time to kick in — these platforms don't make decisions based on seeing your IP once, they need to observe patterns over days or weeks.

Measuring Agents In Production (survey paper)

Lobsters
arxiv.org
2025-12-07 21:31:40
Comments...
Original Article

View PDF HTML (experimental)

Abstract: AI agents are actively running in production across diverse industries, yet little is publicly known about which technical approaches enable successful real-world deployments. We present the first large-scale systematic study of AI agents in production, surveying 306 practitioners and conducting 20 in-depth case studies via interviews across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and what the top development challenges are. We find that production agents are typically built using simple, controllable approaches: 68% execute at most 10 steps before requiring human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability remains the top development challenge, driven by difficulties in ensuring and evaluating agent correctness. Despite these challenges, simple yet effective methods already enable agents to deliver impact across diverse industries. Our study documents the current state of practice and bridges the gap between research and deployment by providing researchers visibility into production challenges while offering practitioners proven patterns from successful deployments.

Submission history

From: Melissa Pan [ view email ]
[v1] Tue, 2 Dec 2025 16:45:10 UTC (337 KB)

Quoting Cory Doctorow

Simon Willison
simonwillison.net
2025-12-07 21:28:28
Now I want to talk about how they're selling AI. The growth narrative of AI is that AI will disrupt labor markets. I use "disrupt" here in its most disreputable, tech bro sense. The promise of AI – the promise AI companies make to investors – is that there will be AIs that can do your job, and when ...
Original Article

Now I want to talk about how they're selling AI. The growth narrative of AI is that AI will disrupt labor markets. I use "disrupt" here in its most disreputable, tech bro sense.

The promise of AI – the promise AI companies make to investors – is that there will be AIs that can do your job, and when your boss fires you and replaces you with AI, he will keep half of your salary for himself, and give the other half to the AI company.

That's it.

That's the $13T growth story that MorganStanley is telling. It's why big investors and institutionals are giving AI companies hundreds of billions of dollars. And because they are piling in, normies are also getting sucked in, risking their retirement savings and their family's financial security.

— Cory Doctorow , The Reverse Centaur’s Guide to Criticizing AI

Using LLMs at Oxide

Simon Willison
simonwillison.net
2025-12-07 21:28:17
Using LLMs at Oxide Thoughtful guidance from Bryan Cantrill, who evaluates applications of LLMs against Oxide's core values of responsibility, rigor, empathy, teamwork, and urgency. Via Lobste.rs Tags: ai, generative-ai, llms, oxide, bryan-cantrill...
Original Article

Using LLMs at Oxide ( via ) Thoughtful guidance from Bryan Cantrill, who evaluates applications of LLMs against Oxide's core values of responsibility, rigor, empathy, teamwork, and urgency.

Posted 7th December 2025 at 9:28 pm

Russia Blocks FaceTime and Snapchat

Daring Fireball
apnews.com
2025-12-07 21:19:00
Dasha Litvinova, reporting for the AP: Russian authorities said Thursday they have imposed restrictions on Apple’s video calling service FaceTime, the latest step in an effort to tighten control over the internet and communications online. State internet regulator Roskomnadzor alleged in a state...
Original Article

Russian authorities said Thursday they have imposed restrictions on Apple’s video calling service FaceTime, the latest step in an effort to tighten control over the internet and communications online.

State internet regulator Roskomnadzor alleged in a statement that the service is being “used to organize and conduct terrorist activities on the territory of the country, to recruit perpetrators (and) commit fraud and other crimes against our citizens.” Apple did not respond to an emailed request for comment.

The Russian regulator also announced that it has blocked Snapchat, a messaging app for sharing photos, videos and text messages, citing the same grounds it gave for restricting FaceTime. It said that it took the action Oct. 10 even though it only reported the move on Thursday.

Under President Vladimir Putin, authorities have engaged in deliberate and multipronged efforts to rein in the internet. They have adopted restrictive laws and banned websites and platforms that don’t comply. Technology also has been perfected to monitor and manipulate online traffic.

After Russia’s full-scale invasion of Ukraine in 2022, the government blocked major social media like Twitter, Facebook and Instagram.

Access to YouTube was disrupted last year in what experts called deliberate throttling of the widely popular site by the authorities. The Kremlin blamed YouTube owner Google for not properly maintaining its hardware in Russia.

While it’s still possible to circumvent some of the restrictions by using virtual private network services, those are routinely blocked, too.

Authorities further restricted internet access this summer with widespread shutdowns of cellphone internet connections . Officials have insisted the measure was needed to thwart Ukrainian drone attacks, but experts argued it was another step to tighten internet control. In dozens of regions, “white lists” of government-approved sites and services that are supposed to function despite a shutdown have been introduced.

The government also has acted against popular messaging platforms. Encrypted messenger Signal and another popular app, Viber, were blocked in 2024. This year the authorities banned calls via WhatsApp, the most popular messaging app in Russia, and Telegram, a close second. Roskomnadzor justified the measure by saying the two apps were being used for criminal activities.

At the same time, authorities actively promoted a “national” messenger app called MAX, which critics see as a surveillance tool. The platform, touted by developers and officials as a one-stop shop for messaging, online government services, making payments and more, openly declares it will share user data with authorities upon request. Experts also say it doesn’t use end-to-end encryption.

Earlier this week, the government also said it was blocking Roblox, a popular online game platform, saying the step aimed at protecting children from illicit content and “pedophiles who meet minors directly in the game’s chats and then move on to real life.”

Stanislav Seleznev, cyber security expert and lawyer with the Net Freedom rights group, told The Associated Press that Russian law views any platform where users can message each other as “organizers of dissemination of information.”

This label mandates that platforms have an account with Roskomnadzor so that it could communicate its demands, and give Russia’s security service, the FSB, access to accounts of their users for monitoring; those failing to comply are in violation and can get blocked, Seleznev said.

He suggested that these regulations could have been applied to both Roblox and FaceTime.

Roblox in October was the second most popular game platform in Russia, with nearly 8 million monthly users, according to media monitoring group Mediascope.

Seleznev estimated that possibly tens of millions of Russians have been using FaceTime, especially after calls were banned on WhatsApp and Telegram. He called the restrictions against the service “predictable” and warned that other sites failing to cooperate with Roskomnadzor “will be blocked, that’s obvious.”

★ Meta Says Fuck That Metaverse Shit

Daring Fireball
daringfireball.net
2025-12-07 21:18:13
Joz might want to use the word now, just to make jokes....
Original Article

Mike Isaac, reporting for The New York Times, “ Meta Weighs Cuts to Its Metaverse Unit ” (gift link):

Meta is considering making cuts to a division in its Reality Labs unit that works on the so-called metaverse, said three employees with knowledge of the matter.

The cuts could come as soon as next month and amount to 10 to 30 percent of employees in the Metaverse unit, which works on virtual reality headsets and a V.R.-based social network, the people said. The numbers of potential layoffs are still in flux, they said. Other parts of the Reality Labs division develop smart glasses, wristbands and other wearable devices. The total number of employees in Reality Labs could not be learned.

Meta does not plan to abandon building the metaverse, the people said. Instead, executives expect to shift the savings from the cuts into investments in its augmented reality glasses, the people said.

Meta confirmed the cuts to the Wall Street Journal , and Blooomberg’s Kurt Wagner broke the news Thursday .

I’m so old that I remember ... checks notes ... four years ago, when Facebook renamed itself Meta in late 2021 with this statement : “Meta’s focus will be to bring the metaverse to life and help people connect, find communities and grow businesses.” And Mark Zuckerberg, announcing the change, wrote :

But all of our products, including our apps, now share a new vision: to help bring the metaverse to life. And now we have a name that reflects the breadth of what we do.

From now on, we will be metaverse-first, not Facebook-first. That means that over time you won’t need a Facebook account to use our other services. As our new brand starts showing up in our products, I hope people around the world come to know the Meta brand and the future we stand for.

Many of us never fell for this metaverse nonsense. For example, I’m also old enough to remember just one year later, near the end of Joanna Stern’s on-stage interview with Craig Federighi and Greg Joswiak at a 2022 WSJ event, seven months before Vision Pro was announced ( at the 29:30 mark ):

Stern: You have to finish this sentence, both of you. The metaverse is...

Joz: A word I’ll never use.

He might want to use the word now, just to make jokes.

Om Malik, writing in April this year :

Some of us are old enough to remember that the reason Mark renamed the company is because the Facebook brand was becoming toxic, and associated with misinformation and global-scale crap. It was viewed as a tired, last-generation company. Meta allowed the company to rebrand itself as something amazing and fresh .

Lastly, yours truly, linking to Malik’s post :

And so while “Meta” will never be remembered as the company that spearheaded the metaverse — because the metaverse never was or will be an actual thing — it’s in truth the perfect name for a company that believes in nothing other than its own success.

Why the Sanitizer API is just `setHTML()`

Lobsters
frederikbraun.de
2025-12-07 21:16:39
Comments...
Original Article

Sanitizing HTML is the practice of taking a piece of HTML and removing some unwanted elements and attributes. Most often this is done to allow user-generated content with HTML but without causing XSS bugs. When imported from a library, a sanitizer typically looks like this: 

const clean = DOMPurify.sanitize(input);
context.innerHTML = clean;

However, the API that we are building doesn't look like this at all. The core feature of the Sanitizer API is actually just Element.setHTML(input) .

This blog post will explain why.

To do so, we have to study the two lines of code from the DOMPurity example above. They result in the following steps:

  1. Take an input string (and optionally a list of allowed elements as parameter).
  2. Parse the input into an HTML fragment (no context element given).
  3. Traverse the HTML fragment and remove elements as configured.
  4. Serialize the remaining fragment into a string.
  5. Parse the sanitized string (again), this time with context as context node into a fragment.
  6. Insert the new fragment below context in the DOM tree.

Quick exercise for the reader: Can you spot where line 1 ( DOMPurify.sanitize() ) stops and line 2 (the innerHTML assignment) starts?

Solution DOMPurify.sanitize() includes steps 1 through 4. The innerHTML assignment. is steps 5-6.

This is pretty similar to the Sanitizer that I wanted to build into the browser: 

const mySanitizer = new Sanitizer(/* config */);
//XXX This never shipped.
context.innerHTML = Sanitizer.sanitize(input);

But that is NOT the Sanitizer we ended up with.

And the reason is essentially Mutated XSS (mXSS). To quickly recap, the idea behind mXSS is that HTML parsing is not stable and a line of HTML being parsed and serialized and parsed again may turn into something rather different. (See this description of mXSS bugs collected by SonarSource if you need a refresher.)

Another key point with mXSS is that HTML parsing can be quite context-sensitive : How an input string will be interpreted depends on the current node it is being inserted into.

Now let's go back to the algorithm steps 1-6. Did you notice that step 2 and 5 both perform HTML parsing? DOMPurify and most other sanitizers do this without any supplied context element. Typically, they parse into a new document and only return the content of the resulting <body> . The second parse step (step 5), however, does include a context element.

This means that we are parsing the input subtly different each time. We accidentally built a weird machine that will turn HTML into mXSS.

A better HTML sanitizer therefore needs to do away with all of that. How about the following:

  • Use the right context when parsing HTML input.
  • Remove the need for parsing twice.

Starting from an API design with a constructor like new Sanitizer() , it felt pretty hard to think of a context-sensitive method. I wanted something like Sanitizer.sanitize(input, context) . But how would we actually ensure that the return value can not be used another, potentially wrong context ?

What we settled on was an API that has no return value: 

context.setHTML(input, {sanitizer: ... } );

The internal algorithm is now the following:

  1. Parse the input (with the right context element) into a document fragment
  2. Traverse the resulting fragment and sanitize. (Using safe defaults or a user-specified configuration).
  3. Replace the child nodes below context with the sanitized up fragment.

No superfluous parsing. No ambiguous contexts. Just setting HTML.

As a nice side-effect, you can replace existing code in the style of ctx.innerHTML = input with context.setHTML(input) and it should just work the same.

Except that there's no XSS.

To learn more about the Sanitizer API, please continue on MDN , in the Sanitizer Playground , or the Specification ).

OpenAI denies rolling out ads on ChatGPT paid plans

Bleeping Computer
www.bleepingcomputer.com
2025-12-07 20:51:08
ChatGPT is allegedly showing ads to those who pay $20 for the Plus subscription, but OpenAI says this is an app recommendation feature, not an ad. [...]...
Original Article

OpenAI

OpenAI has denied the reports that it has rolled out ads on ChatGPT Plus after users spotted recommendations for shopping apps.

As potted on X , a ChatGPT Plus user casually asked a normal question about Windows BitLocker. While the AI answered the question, it also recommended shopping at Target for groceries.

Now, groceries or home food are clearly not related to BitLocker, but the "Shop for home and groceries" bubble still appears, and it's quite fair to assume that it's an ad.

GPT-ad
ChatGPT showing recommendation (or ad?) for Target

However, an OpenAI executive argues that this is not “not an ad” but an app recommendation from a pilot partner, and that the company wants app suggestions to appear more “organic” inside ChatGPT.

"We've launched apps from some of our pilot partners since DevDay, including Target, and have been working to make the discovery mechanism for apps more organic inside ChatGPT," Daniel McAuley wrote in a post on X.

"Our goal is that apps augment the ux when relevant to a conversation, and we're still working on it. Anyone can build apps using the apps SDK, and we plan to open submissions and the app directory soon," he explained.

For most people, it still looks and feels like an ad. You see a brand logo, a short shopping message, and a call-to-action, inside a paid product, even though you never asked about shopping or Target.

ChatGPT is automatically pushing a commercial suggestion into an answer, just like how recommendation appear in the Windows 11 Start menu, and still defending it.

tines

Break down IAM silos like Bitpanda, KnowBe4, and PathAI

Broken IAM isn't just an IT problem - the impact ripples across your whole business.

This practical guide covers why traditional IAM practices fail to keep up with modern demands, examples of what "good" IAM looks like, and a simple checklist for building a scalable strategy.

Iustin Pop: Yes, still alive!

PlanetDebian
k1024.org
2025-12-07 20:37:00
Yeah, again three months have passed since my last (trivial) post, and I really don’t know where the time has flown. I suppose the biggest problem was the long summer vacation, which threw me off-track, and then craziness started. Work work work, no time for anything, which kept me fully busy in Aug...
Original Article

Yeah, again three months have passed since my last (trivial) post, and I really don’t know where the time has flown.

I suppose the biggest problem was the long summer vacation, which threw me off-track, and then craziness started. Work work work, no time for anything, which kept me fully busy in August, and then “you should travel”.

So mid-September I went on my first business trip since Covid, again to Kirkland, which in itself was awesome. Flew out Sunday, and as I was concerned I was going to lose too much fitness—had a half-marathon planned on the weekend after the return—I ran every morning of the four days I was there. And of course, on the last day, I woke up even earlier (05:30 AM), went out to run before sunrise, intending to do a very simple “run along the road that borders the lake for 2.5K, then back”. And right at the farthest point, a hundred metres before my goal of turning around, I tripped, started falling, and as I was falling, I hit—sideways—a metal pole. I was in a bus station, it was the pole that has the schedule at the top, and I hit it at relatively full speed, right across my left-side ribs. The crash took the entire air out of my lungs, and I don’t remember if I ever felt pain/sensation like that—I was seriously not able to breathe for 20 seconds or so, and I was wondering if I’m going to pass out at this rate.

Only 20 seconds, because my Garmin started howling like a police siren, and the screen was saying something along the lines of: “Incident detected; contacting emergency services in 40…35…” and I was fumbling to cancel that, since a) I wasn’t that bad, b) notifying my wife that I had a crash would have not been a smart idea.

My left leg was scraped in a few places, my left hand pretty badly, or more than just scraped, so my focus was on limping back, and finding a fountain to wash my injuries, which I did, so I kept running with blood dripping down my hand. Fun fun, everything was hurting, I took an Uber for the ~1Km to the office, had many meetings, took another Uber and flew back to Zurich. Seattle → San Francisco → Zürich, I think 14 hours, with my ribs hurting pretty badly. But I got home (Friday afternoon), and was wondering if I can run or not on Saturday.

Saturday comes, I feel pretty OK, so I said let’s try, will stop if the pain is too great. I pick up my number, I go to the start, of course in the last block and not my normal block, and I start running. After 50 metres, I knew this won’t be good enough, but I said, let’s make it to the first kilometre. Then to the first fuelling point, then to the first aid point, at which moment I felt good enough to go to the second one.

Long story short, I ran the whole half marathon, with pain. Every stop for fuelling was mentally hard, as the pain stopped, and I knew I had to start running again, and the pain would resume. In the end, managed to finish: two and a half hours, instead of just two hours, but alive and very happy. Of course, I didn’t know what was waiting for me… Sunday I wake up in heavy pain, and despite painkillers, I was not feeling much better. The following night was terrible, Monday morning I went to the doctor, had X-rays, discussion with a radiologist. “Not really broken, but more than just bruised. See this angle here? Bones don’t have angles normally”. Painkillers, chest/abdomen wrapping, no running! So my attempts to “not lose fitness” put me off running for a couple of weeks.

Then October came, and I was getting better, but work was getting even more crazy. I don’t know where November passed, honestly, and now we’re already in December. I did manage to run, quite well, managed to bike a tiny bit and swim a little, but I’m not in a place where I can keep a regular and consistent schedule.

On the good side, I managed this year, for the first time since Covid, to not get sick. Hey, a sport injury is 100× better than a sickness, like I had in previous years, taking me out for two weeks. But life was crazy enough that I didn’t read some of my email accounts for months, and I’m just now starting to catch up to, well, baseline.

Of course, “the” rib—the lowest one on the left side—is long-healed, or so I thought. After some strength training early this week, I was very sore the next day, and I wanted to test whether my rib is still sore. I touched it at “the point”, and it hurt so badly I couldn’t believe. Two and a half months, and it’s not done-done.

And now it’s just two weeks before Christmas and New Year’s, and that time off will ruin my rhythm again. At least ski vacation is booked, ski service is done, and slowly, work is getting in good enough shape to actually enjoy thinking about vacation.

So, in the end, a very adventurous last third of the year, and that wasn’t even all. As I’m writing this, my right wrist is bandaged and for the past 24 hours it hasn’t hurt too much, but that’s another, and not so interesting, story.

I’ll close with a yay for always being behind/backlogged, but alive and relatively well. My sport injuries are “elective injuries” so to speak, and I’m very thankful for that. See you in the next post!

Quoting David Crespo

Simon Willison
simonwillison.net
2025-12-07 20:33:54
What to try first? Run Claude Code in a repo (whether you know it well or not) and ask a question about how something works. You'll see how it looks through the files to find the answer. The next thing to try is a code change where you know exactly what you want but it's tedious to type. Describe it...
Original Article

What to try first?

Run Claude Code in a repo (whether you know it well or not) and ask a question about how something works. You'll see how it looks through the files to find the answer.

The next thing to try is a code change where you know exactly what you want but it's tedious to type. Describe it in detail and let Claude figure it out. If there is similar code that it should follow, tell it so. From there, you can build intuition about more complex changes that it might be good at. [...]

As conversation length grows, each message gets more expensive while Claude gets dumber. That's a bad trade! [...] Run /reset (or just quit and restart) to start over from scratch. Tell Claude to summarize the conversation so far to give you something to paste into the next chat if you want to save some of the context.

— David Crespo , Oxide's internal tips on LLM use

Mechanical power generation using Earth's ambient radiation

Hacker News
www.science.org
2025-12-07 21:55:01
Comments...
Original Article

iced 0.14 has been released (Rust GUI library)

Hacker News
github.com
2025-12-07 21:27:28
Comments...
Original Article

@hecrj hecrj released this

07 Dec 20:53

Added

  • Reactive rendering. #2662
  • Time travel debugging. #2910
  • Animation API for application code. #2757
  • Headless mode testing. #2698
  • First-class end-to-end testing. #3059
  • Input method support. #2777
  • Hot reloading. #3000
  • Concurrent image decoding and uploading (and more cool stuff). #3092
  • comet debugger and devtools foundations. #2879
  • Presentation metrics for comet . #2881
  • Custom performance metrics for comet . #2891
  • Smart scrollbars. #2922
  • System theme reactions. #3051
  • table widget. #3018
  • grid widget. #2885
  • sensor widget. #2751
  • float widget and other cool stuff. #2916
  • pin widget. #2673
  • wrap method for column widget. #2884
  • auto_scroll support for scrollable widget. #2973
  • delay support for tooltip widget. #2960
  • Auto strategy to text::Shaping . #3048
  • Incremental markdown parsing. #2776
  • Customizable markdown rendering and image support. #2786
  • Quote support for markdown widget. #3005
  • Tasklist support for markdown widget. #3022
  • crisp feature for default quad snapping. #2969
  • Basic layer merging for graphics::layer::Stack . #3033
  • Headless mode for iced_wgpu and concurrency foundations. #2857
  • Primitive culling in column and row widgets. #2611
  • Lazy Compositor initialization in winit shell. #2722
  • Support for Justified text alignment. #2836
  • Support for double click event to mouse_area . #2602
  • Default implementation for iced_wgpu::geometry::Cache . #2619
  • physical_key field to KeyReleased event. #2608
  • total_size method for qr_code widget. #2606
  • PartialEq implementations for widget styles. #2637
  • Send marker to iced_wgpu::Renderer by using Arc in caches. #2692
  • Disabled Status for scrollbar widget. #2585
  • warning color to theme::Palette . #2607
  • maximized and fullscreen fields to window::Settings . #2627
  • window tasks for controlling sizes and resize increments. #2633
  • window task for drag resizing. #2642
  • Helper functions for alignment to widget module. #2746
  • time::repeat subscription. #2747
  • Vertical support for progress_bar . #2748
  • scale support for image widget. #2755
  • LineEnding support for text_editor . #2759
  • Mul<Transformation> implementation for mouse::Cursor and mouse::Click . #2758
  • animation module support for Wasm target. #2764
  • Flake for a dev shell in DEPENDENCIES . #2769
  • unfocus widget operation. #2804
  • sipper support and some QoL. #2805
  • Variable text size for preedit IME window. #2790
  • is_focused widget operation. #2812
  • Notification of window pre-presentation to windowing system. #2849
  • Customizable vertical spacing for wrapped rows. #2852
  • Indent and unindent actions for text_editor . #2901
  • Floating Images. #2903
  • min_size method to PaneGrid . #2911
  • Generic key for sensor widget. #2944
  • Debug implementation for Task . #2955
  • draw_with_bounds method to canvas::Cache . #3035
  • Synchronous Task Execution and RedrawRequested Consistency. #3084
  • id method to text_editor . #2653
  • horizontal and vertical methods to Padding . #2655
  • is_focused selector and find / find_all operations. #2664
  • push and into_options methods to combo_box::State . #2684
  • Hidden variant to mouse::Interaction . #2685
  • menu_height method to pick_list and combo_box widgets. #2699
  • text_color to toggler::Style . #2707
  • text_shaping method to combo_box widget. #2714
  • transparent field for window::Settings . #2728
  • closeable and minimizable fields to window::Settings . #2735
  • window::monitor_size task. #2754
  • Division operation for Size and Vector . #2767
  • hidden method to scrollable widget. #2775
  • Support for macOS-specific key shortcuts with Control modifier. #2801
  • Additional variants to mouse::Interaction . #2815
  • vsync field to window::Settings . #2837
  • wgpu-bare feature flag to disable default wgpu features. #2828
  • ratio method for Size . #2861
  • Support for ⌘ + Backspace and ⌘ + Delete macOS shortcuts. #2862
  • Expandable selection-by-word after double click in text editors. #2865
  • x11 and wayland feature flags. #2869
  • label method for checkbox widget. #2873
  • shader::Pipeline trait for easier wgpu resource management. #2876
  • select_range widget operation. #2890
  • grid! macro helper. #2904
  • warning style for container widget. #2912
  • Current toggle state to toggler::Status::Disabled . #2908
  • Cursor size awareness for input methods. #2918
  • allow_automatic_tabbing task to runtime::window . #2933
  • FromStr and Display implementations for Color . #2937
  • text::Renderer trait in iced_graphics with fill_raw method. #2958
  • font_maybe helper for text widget. #2988
  • filter_map method to Subscription . #2981
  • repeat field to keyboard::Event::KeyPressed . #2991
  • Additional settings to control the fonts used for markdown rendering. #2999
  • Rescaled variant to window::Event . #3001
  • Environment variable to define beacon server listen address. #3003
  • push_under method to stack widget. #3010
  • NONE constant to keyboard::Modifiers . #3037
  • shadow field to overlay::menu::Style . #3049
  • draw_mesh_cache method in mesh::Renderer trait. #3070
  • Efficient is_empty method for text_editor::Content . #3117
  • *Assign implementations for Point and Vector . #3131
  • Support Background instead of Color styling for scrollable . #3127
  • CornerPreference window setting for Windows. #3128
  • move_to method for Editor API. #3125
  • Background and padding_ratio support for toggler styling. #3129
  • More syntaxes for iced_highlighter . #2822
  • Implement Sub<Vector> for Cursor . #3137

Changed

  • Replace Rc with Arc for markdown caching. #2599
  • Improved button::Catalog and Style documentation. #2590
  • Improved clock example to display ticks and numbers. #2644
  • Derived PartialEq and Eq for mouse::click::Kind . #2741
  • Marked Color::from_rgb8 and Color::from_rgba8 as const. #2749
  • Replaced unmaintained directories-next crate with directories . #2761
  • Changed Widget::update to take Event by reference. #2781
  • Improved gallery example with blurhash previews. #2796
  • Replaced wasm-timer with wasmtimer . #2780
  • Tweaked Palette Generation. #2811
  • Relaxed Task::perform bound from Fn to FnOnce . #2827
  • Improved quad shader to use a single SDF in iced_wgpu . #2967
  • Leveraged Limits::min directly in scrollable::layout . #3004
  • Overhauled theme::Palette generation by leveraging Oklch . #3028
  • Mutable Widget Methods. #3038
  • Prioritized Shrink over Fill in layout logic. #3045
  • Replaced format! with concat! for string literals. #2695
  • Replaced window::run_with_handle with a more powerful window::run . #2718
  • Made color helpers in palette module public. #2771
  • Changed default PowerPreference to HighPerformance in iced_wgpu . #2813
  • Made button::DEFAULT_PADDING public. #2858
  • Replaced Url parsing in markdown widget with String URIs. #2992
  • Improved alignment docs of container . #2871
  • Made input_method module public. #2897
  • iced logo to built-in icons font. #2902
  • Made Layout::children return an ExactSizeIterator . #2915
  • Enabled fancy-regex instead of onig for syntect . #2932
  • Added warning status to toast example. #2936
  • Improved scroll_to and snap_to to allow operating on a single axis. #2994
  • Disabled png-format feature from iced_tiny_skia . #3043
  • Unified keyboard subscriptions into a single listen subscription. #3135
  • Updated to Rust 2024. #2809
  • Updated wgpu to 22.0 . #2510
  • Updated wgpu to 23.0 . #2663
  • Updated wgpu to 24.0 . #2832
  • Updated wgpu to 26.0 . #3019
  • Updated wgpu to 27.0 . #3097
  • Updated image to 0.25 . #2716
  • Updated cosmic-text to 0.13 . #2834
  • Updated cosmic-text to 0.14 . #2880
  • Updated cosmic-text to 0.15 . #3098
  • Updated resvg to 0.45 . #2846
  • Updated wasmtimer to 0.4.2 . #3012
  • Updated dark-light to 2.0 . #2724
  • Updated openssl to 0.10.70 . #2783
  • Updated our winit fork with 0.30.8 fixes. #2737

Fixed

  • Slow wgpu documentation. #2593
  • Documentation for open_events . #2594
  • Layout for wrapped row with spacing . #2596
  • Flex layout of Fill elements in a Shrink cross axis. #2598
  • Incorrect triangle mesh counting in wgpu . #2601
  • Dropped images and meshes when pasting Frame . #2605
  • loading_spinners example skipping part of the animation cycle. #2617
  • Window File* events not marked as unsupported for Wayland. #2615
  • Coupling of markdown::view iterator lifetime with resulting Element . #2623
  • Delete key not working in text_editor widget. #2632
  • Consecutive clicks triggering independently of distance. #2639
  • pane_grid losing continuity when adding or removing panes. #2628
  • Synthetic keyboard events not being discarded. #2649
  • sort_by without total ordering in tiny-skia damage tracking. #2651
  • Outdated docs of Scrollable::with_direction and direction . #2668
  • button calling its on_press handler unnecessarily. #2683
  • system_information example getting stuck at boot. #2681
  • tooltip widget not redrawing when hovered. #2675
  • pane_grid::DragEvent::Canceled not emitted within deadband. #2691
  • Inconsistent positions in window-related operations. #2688
  • text::Wrapping not being applied to Paragraph . #2723
  • Broken nested markdown lists without empty line. #2641
  • Unnecessary cast in the_matrix example. #2731
  • Incorrect layer counting in iced_wgpu . #2701
  • Image not respecting viewport bounds. #2752
  • Attempting to draw empty meshes in iced_wgpu . #2782
  • Input placeholder text not clearing when IME is activated. #2785
  • Missing redraw request in image::Viewer . #2795
  • Wrong position of preedit text on scrolled content. #2798
  • Wrong initial candidate position for IME. #2793
  • Text spans in IME preedit not being properly cached. #2806
  • cpu_brand in system_information always being empty. #2797
  • Horizontal text alignment being ignored on multi-line text. #2835
  • Missing redraw request in mouse_area when hovered. #2845
  • futures-executor being pulled even when it's not the default executor. #2841
  • WebGPU failing to boot in Chromium. #2686
  • Crash when using WebGL due to wrong binding alignment. #2883
  • Wrong calculation of rows in grid widget when evenly distributed. #2896
  • Panic in combo_box due to cleared children during diff . #2905
  • OpenGL backend in wgpu interpreting atlas texture as cube map instead of texture array. #2919
  • quad shader blending without pre-multiplication. #2925
  • Inconsistent primitive pixel snapping in iced_wgpu . #2962
  • Inconsistent Rectangle::is_within implementation. #2966
  • Text damage calculation in iced_tiny_skia . #2964
  • Leftover title mention in documentation. #2972
  • Text bounds cutoff in iced_wgpu . #2975
  • Rectangle vertices not being snapped to the pixel grid independently. #2768
  • Lints for Rust 1.89. #3030
  • debug builds on macOS Tahoe. #3056
  • Typo in documentation comment for filter_map . #3052
  • container::Style not respecting crisp feature. #3112
  • Incorrect padding in text_editor . #3115
  • Outdated documentation of Widget::mouse_interaction . #2696
  • Incorrect render pass viewport in custom_shader example. #2738
  • Capturing ButtonReleased event inside image::Viewer . #2744
  • Incomplete docs for on_link_click in rich_text . #2803
  • Stale syntax highlighting on text_editor after theme changes. #2818
  • Wrong background color for window::Preedit on translucent themes. #2819
  • Panic on Chromium-like browsers when canvas initial size is (0, 0) . #2829
  • Outdated dev shell templates. #2840
  • Missing derive feature for serde dependency. #2854
  • bezier_tool listed as an example in the Widget trait docs. #2867
  • Incomplete doc comment of Length::is_fill . #2892
  • scrollable touch scrolling when out of bounds. #2906
  • Element::explain being hidden by multi-layer widgets. #2913
  • Missing Shell::request_redraw on component . #2930
  • Text clipping in iced_tiny_skia . #2929
  • Inconsistent naming of tree parameter in Widget trait. #2950
  • text_editor syntax highlighting not updating on paste. #2947
  • svg scaling in iced_tiny_skia . #2954
  • Stroke bounds calculation and clip transformations in iced_tiny_skia . #2882
  • Artifacts when drawing small arcs in canvas widget. #2959
  • Path not being closed in Path::circle . #2979
  • Incorrect transformation of cached primitives in iced_tiny_skia . #2977
  • Panic when drawing empty image in iced_tiny_skia . #2986
  • Incorrect mapping of navigation keys on higher keyboard layers. #3007
  • Status of svg widget not being updated on cursor movement. #3009
  • hover widget ignoring events in certain conditions. #3015
  • OpenGL backend in iced_wgpu choosing wrong texture format in wgpu::image::atlas . #3016
  • Missing redraw request in geometry example. #3020
  • Buffer presentation logic in iced_tiny_skia . #3032
  • combo_box text not getting cleared on selection. #3063
  • wgpu surface not being reconfigured on SurfaceError::Lost or Outdated . #3067
  • Incorrect cursor for slider widget on Windows . #3068
  • Paragraph::hit_span returning false positives at end of content. #3072
  • Incorrect Limits::loose documentation. #3116
  • Missing semicolon triggering a clippy lint. #3118
  • iced_tiny_skia using a Window instead of a Display handle for softbuffer::Context creation. #3090
  • Missing fn operate in tooltip widget. #3132
  • Panic when rendering problematic svg . #3123
  • Hotkey combinations not working on non-latin keyboard layouts. #3134
  • keyboard::listen reporting captured key events. #3136

Removed

  • is_over method in Overlay trait. #2921
  • Short-hand notation support for color! macro. #2592
  • surface argument of Compositor::screenshot . #2672
  • once_cell dependency. #2626
  • winapi dependency. #2760
  • palette dependency. #2839

Many thanks to...

Proxmox delivers its software-defined datacenter contender and VMware escape

Hacker News
www.theregister.com
2025-12-07 21:26:35
Comments...
Original Article

Open source virtualization project Proxmox has delivered the first full and stable release of its Datacenter Manager product, making it a more viable alternative as a private cloud platform.

Proxmox’s Virtual Environment, a platform that hosts virtual machines and containers, and includes software-defined storage and networking, has become increasingly prominent in recent years as Broadcom’s VMware business unit focused on large enterprise customers. Proxmox has become a popular alternative to VMware for organizations whose needs don’t go far beyond basic server virtualization. Even one VMware partner The Register recently spoke to decided Proxmox was sufficient for some internal workloads it felt did not need all the features of VMware’s Cloud Foundation platform.

Proxmox, however, has bigger ambitions and on Thursday started chasing them by releasing a new product called Datacenter Manager that offers centralized management for multiple, independent Proxmox-based environments.

As explained in Proxmox’s launch announcement, the product “… provides an aggregated view of all your connected nodes and clusters and is designed to manage complex and distributed infrastructures, from local installations to globally scaled data centers.”

Datacenter Manager also enables migration of VMs across clusters without having to manually reconfigure networks. That’s a trick VMware invented decades ago and has since become table stakes for serious private cloud players.

Other features also help to make Proxmox a contender, such as VM fleet management tools that allow admins to identify VMs that need patches and arrange installation, lifecycle management for VMs, and a dashboard that allows a view of all hosts and the workloads they host – and their status.

Bardia Khalilifar, CEO of Australian Proxmox service provider Multiportal.io, welcomed the debut of Datacenter Manager.

"I think it is fantastic that Proxmox has released this," he said, as it will help enable service providers to manage multiple Proxmox rigs on behalf of their clients. He said the product "opens the floodgates" for wider Proxmox adoption, especially across multiple datacenters and for use in private clouds.

Proxmox Server Solutions GmbH, the entity that develops Proxmox products and makes them available under the GNU AGPLv3 license, wrote Datacenter Manager in Rust. That’s no guarantee of strong security, but it’s a decent start. Proxmox’s developers based the platform on Debian Trixie 13.2, using version 6.17 of the Linux kernel, and with ZFS 2.3.4 included.

Datacenter Manager downloads are available here . If you take it for a spin, let us know how it goes! ®

The Ways the AI Bubble Might Burst

Hacker News
www.wheresyoured.at
2025-12-07 21:13:28
Comments...
Original Article

[Editor's Note: this piece previously said "Blackstone" instead of "Blackrock," which has now been fixed.]

I've been struggling to think about what to write this week, if only because I've written so much recently and because, if I'm honest, things aren't really making a lot of sense.

NVIDIA claims to have shipped six million Blackwell GPUs in the last four quarters — as I went into in my last premium piece — working out to somewhere between 10GW and 12GW of power (based on the power draw of B100 and B200 GPUs and GB200 and GB300 racks), which...does not make sense based on the amount of actual data center capacity brought online.

Similarly, Anthropic claims to be approaching $10 billion in annualized revenue — so around $833 million in a month — which would make it competitive with OpenAI's projected $13 billion in revenue, though I should add that based on my reporting extrapolating OpenAI's revenues from Microsoft's revenue share , I estimate the company will miss that projection by several billion dollars, especially now that Google's Gemini 3 launch has put OpenAI on a " Code Red, " shortly after an internal memo revealed that Gemini 3 could “create some temporary economic headwinds for [OpenAI]."

Which leads me to another question: why?

Gemini 3 is "better," in the same way that every single new AI model is some indeterminate level of "better." Nano Banana Pro is, to Simon Willison, " the best available image generation model. " But I can't find a clear, definitive answer as to why A) this is "so much better," B) why everybody is freaking out about Gemini 3, and C) why this would have created "headwinds" for OpenAI, headwinds so severe that it has had to rush out a model called Garlic "as soon as possible" according to The Information :

Last week, OpenAI’s chief research officer Mark Chen told some colleagues about the new model, which was performing well on the company’s evaluations, at least when compared to Gemini 3 and Anthropic’s Opus 4.5 in tasks involving coding and reasoning, according to a person with knowledge of the remarks.

But Garlic may be a bigger deal. Chen said OpenAI is looking to release a version of Garlic as soon as possible, which we think means people shouldn’t be surprised to see GPT-5.2 or GPT-5.5 release by early next year.

Garlic is a different model from Shallotpeat, a new large language model under development which Altman told staff in October would help OpenAI challenge Gemini 3. Garlic incorporates bug fixes that the company used in developing Shallotpeat during the pretraining process, the first stage of model training in which an LLM is shown data from the web and other sources so it can learn connections between them.

Right, sure, cool, another model. Again, why is Gemini 3 so much better and making OpenAI worried about "economic headwinds"? Could this simply be a convenient excuse to cover over, as Alex Heath reported a few weeks ago , ChatGPT's slowing download and usage growth ?

Experts I've talked to arrived at two conclusions:

  • Gemini 3 is good/better at the stuff tested on benchmarks compared to what OpenAI has.
  • OpenAI's growth and usage was decelerating before this happened, and this just allows OpenAI to point to something.

I don't know about garlic or shallotpeat or whatever , but one has to wonder at some point what it is that OpenAI is doing all day :

Altman said Monday in an internal Slack memo that he was directing more employees to focus on improving features of ChatGPT, such as personalizing the chatbot for the more than 800 million people who use it weekly, including letting each of those people customize the way it interacts with them.

Altman also said other key priorities covered by the code red included Imagegen, the image-generating AI that allows ChatGPT users to create anything from interior-design mockups to turning real-life photos into animated ones. Last month, Google released its own image generation model, Nano Banana Pro, to strong reviews.

Altman said other priorities consisted of improving “model behavior” so that people prefer the AI models that powers ChatGPT more than models from competitors, including in public rankings such as LMArena; boosting ChatGPT’s speed and reliability; and minimizing overrefusals, a term that refers to when the chatbot refuses to answer a benign question.

So, OpenAI's big plan is to improve ChatGPT , make the image generation better , make people like the models better , improve rankings , make it faster, and make it answer more stuff.

I think it's fair to ask: what the fuck has OpenAI been doing this whole time if it isn't "make the model better" and "make people like ChatGPT more"? I guess the company shoved Sora 2 out the door — which is already off the top 30 free Android apps in the US and at 17 on the US free iPhone apps rankings as of writing this sentence after everybody freaked out about it hitting number one . All that attention, and for what?

Indeed, signs seem to be pointing towards reduced demand for these services. As The Information reported a few days ago ...

Multiple Microsoft divisions, for instance, have lowered how much salespeople are supposed to grow their sales of certain AI products after many of them missed sales-growth goals in the fiscal year that ended in June, according to two salespeople in Microsoft’s Azure cloud unit.

Microsoft, of course, disputed this, and said...

A Microsoft spokesperson said “aggregate sales quotas for AI products have not been lowered” but declined to comment specifically on the lowered growth targets. The spokesperson pointed to growth in the company’s overall cloud business, which has been lifted by rentals of AI servers by OpenAI and other AI developers.

Well, I don't think Microsoft has any problems selling compute to OpenAI — which paid it $8.67 billion just for inference between January and September — as I doubt there is any "sales team" having to sell compute to OpenAI.

But I also want to be clear that Microsoft added a word: "aggregate." The Information never used that word, and indeed nobody seems to have bothered to ask what "aggregate" means. I do, however, know that Microsoft has had trouble selling stuff. As I reported a few months ago, in August 2025 Redmond only had 8 million active paying licenses for Microsoft 365 Copilot out of the more-than-440 million people paying for Microsoft 365 .

In fact, here's a rundown of how well AI is going for Microsoft:

  • Its chips effort is falling behind , with its "Maya" AI chip delayed to 2026, and according to The Information, "when it finally goes into mass production next year, it’s expected to fall well short of the performance of Nvidia’s flagship Blackwell chip."
  • According to The Information in late October 2025 , "more customers have been using Microsoft’s suite of AI copilots, but many of them aren’t paying for it."
  • In October , Australian's Competition and Consumer Commission sued Microsoft for "allegedly misleading 2.7 million Australians over Microsoft 365 subscriptions," by making it seem like they had to pay extra and integrate Copilot into their subscription rather than buy the, and I quote, "undisclosed third option, the Microsoft 365 Personal or Family Classic plans, which allowed subscribers to retain the features of their existing plan, without Copilot, at the previous lower price."
  • According to The Information in September 2025 , Microsoft had to "partly" replace OpenAI's models with Anthropic's for some of its Copilot software. Microsoft has, at this point, sunk over ten billion dollars into OpenAI, and part of its return for doing so was exclusively being able to use its models. Cool!
  • According to The Information in September 2025 , Microsoft has had to push discounts for Office 365 Copilot as customers had "found Copilot adoption slow due to high cost and unproven ROI."
  • In late 2024 , customers had paused purchasing further Copilot assistants due to performance and cost issues.

Yet things are getting weird. Remember that OpenAI-NVIDIA deal? The supposedly "sealed" one where NVIDIA would invest $100 billion in OpenAI , with each tranche of $10 billion gated behind a gigawatt of compute? The one that never really seemed to have any fundament to it, but people reported as closed anyway? Well, per NVIDIA's most-recent 10-Q (emphasis mine):

Investment commitments are $6.5 billion as of October 26, 2025, including $5 billion in Intel Corporation which is subject to regulatory approval. In the third quarter of fiscal year 2026, we entered into a letter of intent with an opportunity to invest in OpenAI .

A letter of intent "with an opportunity" means jack diddly squat. My evidence? NVIDIA's follow-up mention of its investment in Anthropic:

In November 2025, we entered into an agreement, subject to certain closing conditions, to invest up to $10 billion in Anthropic.

This deal, as ever, was reported as effectively done , with NVIDIA investing $10 billion and Microsoft $5 billion, saying the word "will" as if the money had been wired, despite the "closing conditions" and the words "up to" suggesting NVIDIA hasn't really agreed how much it will really invest. A few weeks later, the Financial Times would report that Anthropic is trying to go public as early as 2026 and that Microsoft and NVIDIA's money would "form part of a funding round expected to value the group between $300bn and $350bn."

For some reason, Anthropic is hailed as some sort of "efficient" competitor to OpenAI, at least based on what both The Information and Wall Street Journal have said, yet it appears to be raising and burning just as much as OpenAI . Why did a company that's allegedly “reducing costs” have to raise $13 billion in September 2025 after raising $3.5 billion in March 2025 , and after raising $4 billion in November 2024 ? Am I really meant to read stories about Anthropic hitting break even in 2028 with a straight face? Especially as other stories say Anthropic will be cash flow positive “ as soon as 2027 .”

And if this company is so efficient and so good with money , why does it need another $15 billion, likely only a few months after it raised $13 billion? Though I doubt the $15 billion round closes this year, if it does, it would mean that Anthropic would have raised $31.5 billion in 2025 — which is, assuming the remaining $22.5 billion comes from SoftBank, not far from the $40.8 billion OpenAI would have raised this year.

In the event that SoftBank doesn't fund that money in 2025, Anthropic will have raised a little under $2 billion less ($16.5 billion) than OpenAI ($18.3 billion, consisting of $10 billion in June split between $7.5 billion from SoftBank and $2.5 billion from other investors, and an $8.3 billion round in August ) this year.

I think it's likely that Anthropic is just as disastrous a business as OpenAI, and I'm genuinely surprised that nobody has done the simple maths here, though at this point I think we're in the era of "not thinking too hard because when you do so everything feels crazy.”

Which is why I'm about to think harder than ever!

I feel like I'm asked multiple times a day both how and when the bubble will burst, and the truth is that it could be weeks or months or another year , because so little of this is based on actual, real stuff. While our markets are supported by NVIDIA's eternal growth engine, said growth engine isn't supported by revenues or real growth or really much of anything beyond vibes. As a result, it's hard to say exactly what the catalyst might be, or indeed what the bubble bursting might look like.

Today, I'm going to sit down and give you the scenarios — the systemic shocks — that would potentially start the unravelling of this era, as well as explain what a bubble bursting might actually look like, both for private and public companies.

This is the spiritual successor to August's AI Bubble 2027 , except I'm going to have a little more fun and write out a few scenarios that range from likely to possible , and try and give you an enjoyable romp through the potential apocalypses waiting for us in 2026.

XKeyscore

Hacker News
en.wikipedia.org
2025-12-07 20:54:16
Comments...
Original Article

From Wikipedia, the free encyclopedia

XKeyscore ( XKEYSCORE or XKS ) is a secret computer system used by the United States National Security Agency (NSA) for searching and analyzing global Internet data, which it collects in real time. The NSA has shared XKeyscore with other intelligence agencies, including the Australian Signals Directorate , Canada's Communications Security Establishment , New Zealand's Government Communications Security Bureau , Britain's Government Communications Headquarters , Japan's Defense Intelligence Headquarters , Germany's Bundesnachrichtendienst , and the Danish Defense Intelligence Service , the latter of which proceeded to use it to spy on the UK, Germany, and other key allies for the US. [ 1 ] [ 2 ] [ 3 ] [ 4 ]

In July 2013, Edward Snowden publicly revealed the program's purpose and use by the NSA in The Sydney Morning Herald and O Globo newspapers. The code name was already public knowledge because it was mentioned in earlier articles, and, like many other code names, it appears in job postings and online résumés of employees. [ 5 ] [ 6 ]

On July 3, 2014, German public broadcaster Norddeutscher Rundfunk , a member of ARD , published excerpts of XKeyscore's source code. [ 7 ] [ 8 ]

Scope and functioning

[ edit ]

XKeyscore is a complicated system, and various authors have different interpretations of its actual capabilities. Edward Snowden and Glenn Greenwald have said that XKeyscore is a system that enables almost unlimited surveillance of anyone anywhere in the world, while the NSA has claimed that usage of the system is limited and restricted. [ citation needed ]

According to The Washington Post and national security reporter Marc Ambinder , XKeyscore is an NSA data-retrieval system which consists of a series of user interfaces, backend databases , servers and software that selects certain types of data and metadata that the NSA has already collected using other methods. [ 9 ] [ 10 ]

According to Snowden and Greenwald

[ edit ]

On January 26, 2014, the German broadcaster Norddeutscher Rundfunk asked Edward Snowden in its TV interview: "What could you do if you would use XKeyscore?" and he answered: [ 4 ]

You could read anyone's email in the world, anybody you've got an email address for. Any website: You can watch traffic to and from it. Any computer that an individual sits at: You can watch it. Any laptop that you're tracking: you can follow it as it moves from place to place throughout the world. It's a one-stop-shop for access to the NSA's information. ... You can tag individuals ... Let's say you work at a major German corporation and I want access to that network, I can track your username on a website on a forum somewhere, I can track your real name, I can track associations with your friends and I can build what's called a fingerprint, which is network activity unique to you, which means anywhere you go in the world, anywhere you try to sort of hide your online presence, your identity.

According to The Guardian ' s Glenn Greenwald , low-level NSA analysts can, via systems like XKeyscore, "listen to whatever emails they want, whatever telephone calls, browsing histories, Microsoft Word documents. And it's all done with no need to go to a court, with no need to even get supervisor approval on the part of the analyst." [ 11 ]

He added that the NSA's database of collected communications allows its analysts to listen "to the calls or read the emails of everything that the NSA has stored, or look at the browsing histories or Google search terms that you've entered, and it also alerts them to any further activity that people connected to that email address or that IP address do in the future". [ 11 ]

According to the NSA

[ edit ]

Further information: SIGINT

In an official statement from July 30, 2013, the NSA said "XKeyscore is used as a part of NSA's lawful foreign signals intelligence collection system" to legally obtain information about "legitimate foreign intelligence targets in response to requirements that our leaders need for information necessary to protect our nation and its interests. ... to collect the information, that enables us to perform our missions successfully – to defend the nation and to protect U.S. and allied troops abroad." [ 12 ] In terms of access, an NSA press statement reads that there is no "unchecked analyst access to NSA collection data. Access to XKeyscore, as well as all of NSA's analytic tools, is limited to only those personnel who require access for their assigned tasks." and that there are "stringent oversight and compliance mechanisms built in at several levels. One feature is the system's ability to limit what an analyst can do with a tool, based on the source of the collection and each analyst's defined responsibilities." [ 13 ]

XKeyscore logo
XKeyscore logo
Slide from a 2008 NSA presentation about XKeyscore, showing a world map with the locations of XKeyscore servers
Slide from a 2008 NSA presentation about XKeyscore, showing the query hierarchy

According to an NSA slide presentation about XKeyscore from 2013, it is a " DNI Exploitation System/Analytic Framework". DNI stands for Digital Network Intelligence, which means intelligence derived from internet traffic. [ 14 ]

Edward Snowden said about XKeyscore: "It's a front end search engine" in an interview with the German Norddeutscher Rundfunk . [ 4 ]

XKeyscore is a "piece of Linux software that is typically deployed on Red Hat servers. It uses the Apache web server and stores collected data in MySQL databases". [ 15 ]

XKeyscore is considered a "passive" program, in that it listens, but does not transmit anything on the networks that it targets. [ 8 ] But it can trigger other systems, which perform "active" attacks through Tailored Access Operations which are "tipping", for example, the QUANTUM family of programs, including QUANTUMINSERT, QUANTUMHAND, QUANTUMTHEORY, QUANTUMBOT and QUANTUMCOPPER and Turbulence . These run at so-called "defensive sites" including the Ramstein Air Force base in Germany, Yokota Air Base in Japan, and numerous military and non-military locations within the US. Trafficthief, a core program of Turbulence, can alert NSA analysts when their targets communicate, and trigger other software programs, so select data is "promoted" from the local XKeyscore data store to the NSA's "corporate repositories" for long-term storage. [ 8 ]

XKeyscore consists of over 700 servers at approximately 150 sites where the NSA collects data, like "US and allied military and other facilities as well as US embassies and consulates" in many countries around the world. [ 16 ] [ 17 ] [ 18 ] Among the facilities involved in the program are four bases in Australia and one in New Zealand . [ 17 ]

According to an NSA presentation from 2008, these XKeyscore servers are fed with data from the following collection systems: [ 19 ]

  1. F6 (Special Collection Service) – joint operation of the CIA and NSA that carries out clandestine operations including espionage on foreign diplomats and leaders
  2. FORNSAT – which stands for "foreign satellite collection", and refers to intercepts from satellites
  3. SSO (Special Source Operations) – a division of the NSA that cooperates with telecommunication providers

In a single, undated slide published by Swedish media in December 2013, the following additional data sources for XKeyscore are mentioned: [ 20 ]

  1. Overhead – intelligence derived from American spy planes, drones and satellites
  2. Tailored Access Operations – a division of the NSA that deals with hacking and cyberwarfare
  3. FISA – all types of surveillance approved by the Foreign Intelligence Surveillance Court
  4. Third party – foreign partners of the NSA such as the (signals) intelligence agencies of Belgium, Denmark, France, Germany, Italy, Japan, the Netherlands, Norway, Sweden, etc. However the Netherlands is out of any cooperation concerning intelligence gathering and sharing for illegal spying.

From these sources, XKeyscore stores "full-take data", which is scanned by plug-ins that extract certain types of metadata (like phone numbers, e-mail addresses, log-ins, and user activity) and indexs them in metadata tables, which can be queried by analysts. XKeyscore has been integrated with MARINA , which is NSA's database for internet metadata. [ 14 ]

However, the system continuously gets so much Internet data that it can be stored only for short periods of time. Content data remains on the system for only three to five days, while metadata is stored for up to thirty days. [ 21 ] A detailed commentary on an NSA presentation published in The Guardian in July 2013 cites a document published in 2008 declaring that "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours." [ 22 ]

According to a document from an internal GCHQ website which was disclosed by the German magazine Der Spiegel in June 2014, there are three different types of the XKeyscore system: [ 23 ]

  • Traditional : The initial version of XKeyscore is fed with data from low-rate data signals, after being processed by the WEALTHYCLUSTER system. This traditional version is not only used by NSA but also at many intercept sites of GCHQ.
  • Stage 2 : This version of XKeyscore is used for higher data rates. The data is first processed by the TURMOIL system, which sends 5% of the internet data packets to XKeyscore. GCHQ only uses this version for collection under the MUSCULAR program.
  • Deep Dive : This latest version can process internet traffic at data rates of 10 gigabits per second. Data that could be useful for intelligence purposes is then selected and forwarded by using the "GENESIS selection language". GCHQ also operates a number of Deep Dive versions of XKeyscore at three locations under the codename TEMPORA . [ 24 ]
Slide from a 2008 NSA presentation about XKeyscore, showing the differences between the various NSA database systems

For analysts, XKeyscore provides a "series of viewers for common data types", which allows them to query terabytes of raw data gathered at the aforementioned collection sites. This enables them to find targets that cannot be found by searching only the metadata, and also to do this against data sets that otherwise would have been dropped by the front-end data processing systems. According to a slide from an XKeyscore presentation, NSA collection sites select and forward less than 5% of the internet traffic to the PINWALE database for internet content. [ 21 ]

Because XKeyscore holds raw and unselected communications traffic, analysts can not only perform queries using "strong selectors" like e-mail addresses, but also using "soft selectors", like keywords, against the body texts of e-mail and chat messages and digital documents and spreadsheets in English, Arabic and Chinese. [ 14 ]

This is useful because "a large amount of time spent on the web is performing actions that are anonymous" and therefore those activities can't be found by just looking for e-mail addresses of a target. When content has been found, the analyst might be able to find new intelligence or a strong selector, which can then be used for starting a traditional search. [ 14 ]

Besides using soft selectors, analysts can also use the following other XKeyscore capabilities: [ 14 ] [ 25 ]

  • Look for the usage of Google Maps and terms entered into a search engine by known targets looking for suspicious things or places.
  • Look for "anomalies" without any specific person attached, like detecting the nationality of foreigners by analyzing the language used within intercepted emails. An example would be a German speaker in Pakistan. The Brazilian paper O Globo claims that this has been applied to Latin America and specifically to Colombia, Ecuador, Mexico and Venezuela. [ 16 ] [ 26 ]
  • Detect people who use encryption by doing searches like "all PGP usage in Iran". The caveat given is that very broad queries can result in too much data to transmit back to the analyst.
  • Showing the usage of virtual private networks (VPNs) and machines that can potentially be hacked via TAO .
  • Track the source and authorship of a document that has passed through many hands.
  • On July 3, 2014 ARD revealed that XKeyscore is used to closely monitor users of the Tor anonymity network , [ 8 ] people who search for privacy-enhancing software on the web, [ 8 ] and readers of Linux Journal . [ 27 ]

The Guardian revealed in 2013 that most of these things cannot be detected by other NSA tools, because they operate with strong selectors (like e-mail and IP addresses and phone numbers) and the raw data volumes are too high to be forwarded to other NSA databases. [ 14 ]

In 2008, NSA planned to add a number of new capabilities in the future including access to VoIP and other, unspecified network protocols and additional forms of metadata such as Exif tags, which often include geolocation ( GPS ) data. [ 14 ]

Contribution to U.S. security

[ edit ]

The NSA slides published in The Guardian during 2013 claimed that XKeyscore had played a role in capturing 300 terrorists by 2008, [ 14 ] which could not be substantiated as the redacted documents do not cite instances of terrorist interventions.

A 2011 report from the NSA unit in the Dagger Complex (close to Griesheim in Germany) said that XKeyscore made it easier and more efficient to target surveillance. Previously, analysis often accessed data NSA was not interested in. XKeyscore allowed them to focus on the intended topics, while ignoring unrelated data. XKeyscore also proved to be outstanding for tracking active groups associated with the Anonymous movement in Germany, because it allows for searching on patterns, rather than particular individuals. An analyst is able to determine when targets research new topics, or develop new behaviors. [ 28 ]

To create additional motivation, the NSA incorporated various gamification features. For instance, analysts who were especially good at using XKeyscore could acquire "skilz" points and "unlock achievements." The training units in Griesheim were apparently successful and analysts there had achieved the "highest average of skilz points" compared with all other NSA departments participating in the training program. [ 28 ]

Usage by foreign partners of the NSA

[ edit ]

Excerpt of an NSA document leaked by Edward Snowden that reveals the BND 's usage of the NSA's XKeyscore to wiretap a German domestic target

According to documents Der Spiegel acquired from Snowden, the German intelligence agencies BND (foreign intelligence) and BfV (domestic intelligence) were also allowed to use the XKeyscore system. In those documents the BND agency was described as the NSA's most prolific partner in information gathering. [ 29 ] This led to political confrontations, after which the directors of the German intelligence agencies briefed members of the German parliamentary intelligence oversight committee on July 25, 2013. They declared that XKeyscore has been used by the BND since 2007 and that the BfV has been using a test version since 2012. The directors also explained that the program is not for collecting data, but rather only for the analysis of collected data. [ 30 ]

As part of the UKUSA Agreement , a secret treaty was signed in 1954 by Sweden with the United States, the United Kingdom, Canada, Australia and New Zealand (called the Five Eyes ) for the purpose of intelligence collaboration and data sharing . [ 31 ] According to documents leaked by Snowden, the National Defence Radio Establishment (FRA) has been granted access to XKeyscore. [ 32 ]

In an ongoing scandal, XKeyscore was part of the main package and the reason a new datacenter was needed, built in Sandager [ nl ] . In the 2020s, due to anti-American sentiment causing whistleblowers in the layers of Danish defense, the truth came out about the real usage in Danish state media. NSA gave FE access to their suite of spying software from NSA, in favor of spying on American allies such as Germany's Angela Merkel and Boris Johnson amongst a few in newer times. This deal has been ongoing since the visit to Copenhagen by President Bill Clinton in 1997, facilitated by then Danish Prime minister, Poul Nyrup Rasmussen. [ 3 ]

According to whistleblowers from the Danish secret police, and judicial evidence later presented in Danish court, the US has been and still is carrying out a massive spying operation on the western countries in Europe, with Denmark's help. [ 1 ]

The whistleblower itself was later leaked to be then-head of the Danish Intelligence Service, Lars Findsen. Additionally he leaked in court and through evidence presented in court, that there is an ongoing mass-surveillance program by NSA, not just on the American people themselves, but also of the world facilitated through multiple American "private" companies such as IBM and Google. Additionally making usage of high tech tools such as AI packet sniffing, aggressive red hatting, and other IT methods. [ 2 ] [ 33 ]

The classified documents leaked by Snowden also indicate that in April 2013, NSA had secretly provided the XKeyscore system to the DFS of Defense Intelligence Headquarters . [ 34 ]

  1. ^ a b Henley, Jon (May 31, 2021). "Denmark helped US spy on Angela Merkel and European allies – report" . The Guardian . Archived from the original on May 31, 2021 . Retrieved June 15, 2025 .
  2. ^ a b Davies, Harry (October 2, 2023). "Scandinavian spy drama: the intelligence chief who came under state surveillance" . The Guardian . Archived from the original on October 2, 2023 . Retrieved June 15, 2025 .
  3. ^ a b Bjørnager, Jens; Nielsen, Jens; Jensen, Henrik; McGhie, Steffen; Andersen, Simon (September 13, 2020). "Et pengeskab på Kastellet har i årtier gemt på et dybt fortroligt dokument. Nu er hemmeligheden brudt" [A safe at Kastellet has been hiding a highly confidential document for decades. Now the secret has been broken]. Archived from the original on September 13, 2020 . Retrieved June 15, 2025 .
  4. ^ a b c Seipel, Hubert (January 26, 2014). "Snowden Interview: Transcript" . Norddeutscher Rundfunk . p. 3. Archived from the original on January 28, 2014 . Retrieved May 6, 2019 .
  5. ^ Greenwald, Glenn ; Ackerman, Spencer (June 27, 2013). "How the NSA Is Still Harvesting Your Online Data – Files Show Vast Scale of Current NSA Metadata Programs, with One Stream Alone Celebrating 'One Trillion Records Processed' " . The Guardian . Archived from the original on August 4, 2013 . Retrieved August 5, 2013 . {{ cite news }} : CS1 maint: multiple names: authors list ( link )
  6. ^ Layne, Ken (June 18, 2013). "Job Networking Site LinkedIn Filled With Secret NSA Program Names" . Archived from the original on December 8, 2017 . Retrieved August 6, 2013 .
  7. ^ "xkeyscorerules100" . Panorama . ARD (broadcaster) . July 3, 2014. Archived from the original on July 7, 2014 . Retrieved July 4, 2014 .
  8. ^ a b c d e Jacob Appelbaum , A. Gibson, J. Goetz, V. Kabisch, L. Kampf, L. Ryge (July 3, 2014). "NSA targets the privacy-conscious" . Panorama . Norddeutscher Rundfunk. Archived from the original on July 3, 2014 . Retrieved July 4, 2014 . {{ cite news }} : CS1 maint: multiple names: authors list ( link )
  9. ^ Nakashima, Ellen (July 31, 2013). "Newly Declassified Documents on Phone Records Program Released" . The Washington Post . Archived from the original on July 2, 2014 . Retrieved August 6, 2013 .
  10. ^ Fisher, Max (August 1, 2013). "Is XKeyscore Still Active? Defense Contractor Posted a Job Listing for it 2 weeks Ago" . WorldViews, blog of The Washington Post . Retrieved August 6, 2013 .
  11. ^ a b Rea, Kari (July 28, 2013). "Glenn Greenwald: Low-Level NSA Analysts Have 'Powerful and Invasive' Search Tool" . ABC News . Archived from the original on July 30, 2013 . Retrieved August 4, 2013 .
  12. ^ Wills, Amanda (August 1, 2013). "New Snowden Leak: NSA Program Taps All You Do Online" . Mashable (via CNN ). Archived from the original on August 4, 2013 . Retrieved August 4, 2013 .
  13. ^ "Press Statement on 30 July 2013" (Press release). United States National Security Agency . August 1, 2013. Archived from the original on August 1, 2013.
  14. ^ a b c d e f g h Staff (July 31, 2013). "XKeyscore Presentation from 2008 – Read in Full" . The Guardian . Archived from the original on August 1, 2013 . Retrieved August 6, 2013 .
  15. ^ Lee, Micah; Greenwald, Glenn; Marquis-Boire, Morgan (July 2, 2015). "A Look at the Inner Workings of NSA's XKEYSCORE" . The Intercept . Retrieved July 2, 2020 .
  16. ^ a b Staff (c. 2013). "No alvo dos EUA – O big-brother na América Latina e no mundo" [Targeted By The U.S. – Big Brother in Latin America and in the World]. O Globo (in Portuguese). Archived from the original on July 12, 2013 . Retrieved August 5, 2013 .
  17. ^ a b Dorling, Philip (July 8, 2013). "Snowden Reveals Australia's Links to US Spy Web" . The Sydney Morning Herald . Archived from the original on August 10, 2013 . Retrieved August 2, 2013 .
  18. ^ Greenwald, Glenn ; Casado, Roberto Kaz e José (July 6, 2013). "EUA expandem o aparato de vigilância continuamente – Software de vigilância usa mais de 700 servidores espalhados pelo mundo" . O Globo (in Portuguese). Archived from the original on July 10, 2013 . Retrieved August 2, 2013 . {{ cite news }} : CS1 maint: multiple names: authors list ( link )
  19. ^ Ambinder, Marc (July 31, 2013). "What's XKEYSCORE?" . The Compass (blog of The Week ) . Archived from the original on January 30, 2014 . Retrieved August 4, 2013 .
  20. ^ Gunnar Rensfeldt. "Read the Snowden Documents From the NSA" . Sveriges Television . Archived from the original on February 9, 2014 . Retrieved December 21, 2013 .
  21. ^ a b See also: 3 slides about the XKeyscore program Archived February 2, 2014, at the Wayback Machine
  22. ^ Greenwald, Glenn (July 31, 2013). "XKeyscore: NSA tool collects 'nearly everything a user does on the internet' – XKeyscore Gives 'Widest-Reaching' Collection of Online Data – NSA Analysts Require No Prior Authorization for Searches – Sweeps Up Emails, Social Media Activity and Browsing History" Archived December 31, 2013, at the Wayback Machine . The Guardian . Retrieved August 1, 2013.
  23. ^ XKeyscoreTabs XKS Development Archived June 30, 2014, at the Wayback Machine , published by Der Spiegel on June 18, 2014
  24. ^ Der Spiegel: GCHQ report on the technical abilities of the powerful spying program TEMPORA, which allows for a "full take" Archived June 5, 2019, at the Wayback Machine
  25. ^ Gallagher, Sean (August 1, 2013). "NSA's Internet Taps Can Find Systems to Hack, Track VPNs and Word Docs – X-Keyscore Gives NSA the Ability to Find and Exploit Vulnerable Systems" . Ars Technica . Archived from the original on August 4, 2013 . Retrieved August 4, 2013 .
  26. ^ Greenwald, Glenn ; Casado, Roberto Kaz e José (July 13, 2013). "Espionagem dos EUA se espalhou pela América Latina – Depois do Brasil, Colômbia foi o país mais vigiado – Venezuela também entrou na mira de programas americanos" [U.S. Spying Spread Through Latin America – After Brazil, Colombia Was the Most Watched Country – Venezuela Also Came in the Crosshairs of American Programs]. O Globo (in Portuguese). Archived from the original on July 15, 2013 . Retrieved August 5, 2013 . {{ cite web }} : CS1 maint: multiple names: authors list ( link )
  27. ^ Kyle Rankin (July 3, 2014). "NSA: Linux Journal is an "extremist forum" and its readers get flagged for extra surveillance" . Archived from the original on July 3, 2014 . Retrieved July 3, 2014 .
  28. ^ a b Laura Poitras, Marcel Rosenbach and Holger Stark, Ally and Target: US Intelligence Watches Germany Closely Archived August 20, 2013, at the Wayback Machine , August 12, 2013.
  29. ^ "German Intelligence Agencies Used NSA Spying Program" . Der Spiegel . July 20, 2013. ISSN 2195-1349 . Retrieved September 14, 2024 .
  30. ^ Top Level Telecommunications, New slides about NSA collection programs Archived July 26, 2013, at the Wayback Machine , July 16, 2013
  31. ^ "Cold War treaty confirms Sweden was not neutral" . The Local . December 9, 2013. Archived from the original on December 11, 2013 . Retrieved December 12, 2013 .
  32. ^ Gunnar Rensfeldt. "Read the Snowden Documents From the NSA" . Sveriges Television . Archived from the original on February 9, 2014 . Retrieved December 12, 2013 .
  33. ^ "Ny afsløring: FE masseindsamler oplysninger om danskere gennem avanceret spionsystem" . DR (in Danish). September 24, 2020 . Retrieved September 24, 2020 .
  34. ^ Ryan Gallagher (April 24, 2017). "Japan made secret deals with the NSA that expanded global surveillance" . Archived from the original on April 24, 2017 . Retrieved April 24, 2017 .

Wikimedia Commons has media related to XKeyscore .

FBI Making List of American "Extremists," Leaked Memo Reveals

Hacker News
www.kenklippenstein.com
2025-12-07 20:30:26
Comments...
Original Article
They’re making a list, they’re checking it twice

Attorney General Pam Bondi is ordering the FBI to “compile a list of groups or entities engaging in acts that may constitute domestic terrorism,” according to a Justice Department memo published here exclusively.

The target is those expressing “opposition to law and immigration enforcement; extreme views in favor of mass migration and open borders; adherence to radical gender ideology,” as well as “anti-Americanism,” “anti-capitalism,” and “anti-Christianity.”

Bondi Memo On Countering Domestic Terrorism And Organized Political Violence

151KB ∙ PDF file

Download

Download

That language echoes the so-called indicators of terrorism identified by President Trump’s directive National Security Presidential Memorandum-7 , or NSPM-7, which the memo says it’s intended to implement. Where NSPM-7 was a declaration of war on just about anyone who isn’t MAGA, this is the war plan for how the government will wage it on a tactical level.

In addition to compiling a list of undesirables, Bondi directs the FBI to enhance the capabilities (and publicity) of its tipline in order to more aggressively solicit tips from the American public on, well, other Americans. To that end, Bondi also directs the FBI to establish “a cash reward system” for information leading to identification and arrest of leadership figures within these purported domestic terrorist organizations. (The memo later instructs the FBI to “establish cooperators to provide information and eventually testify against other members” of the groups.)

The payouts don’t end there. Justice Department grants are now to prioritize funding to programs for state and local law enforcement to go after domestic terrorism.

In a section titled “Defining the domestic terrorism threat,” the memo cites “extreme viewpoints on immigration, radical gender ideology, and anti-American sentiment” — indicators that federal law enforcement are instructed to refer to FBI Joint Terrorism Task Forces (JTTFs). Those JTTFs are then instructed to “use all available investigative tools” in order to “map the full network of culpable actors involved” in both “inside and outside the United States.”

The memo also directs the FBI and JTTFs to retroactively investigate incidents going back five years, authorizing the JTTFs in particular to use everything at their disposal to do so.

“Upon receipt of these referrals, the JTTFs shall use all available investigative tools, consistent with law enforcement internal policies and statutory obligations, to map the full network of culpable actors involved in the referred conduct inside and outside the United States,” the memo says.

For months, major media outlets have largely blown off the story of NSPM-7, thinking it was all just Trump bluster and too crazy to be serious. But a memo like this one shows you that the administration is absolutely taking this seriously — even if the media are not — and is actively working to operationalize NSPM-7.

NSPM-7 was signed in September largely in response to the murder of Charlie Kirk, which was a 9/11-type event for the Trump administration, as I’ve reported . (Kirk’s assassination is referenced explicitly in the Justice Department memo.) As anyone who lived through 9/11 can remember, the government doesn’t always think rationally in moments like those, to say the least. And so here we are, with a new War on Terrorism — only this time, millions of Americans like you and I could be the target.

Leave a comment

Share

Discussion about this post

Ready for more?

Syncthing-Android have had a change of owner/maintainer

Hacker News
github.com
2025-12-07 20:15:26
Comments...
Original Article

Description of the issue

status

Steps to reproduce

invite nel0x here and get help to carry on
setup build and release: use old maintainers signing allowed? can we play sign?
reinstate gh action workflows
contact fdroid for release continuation
general: is the name syncthing fork ok or should be changed?

App version

123

App install source - see wiki for details on release channels

GitHub or F-Droid release build

Android version

123

ROM vendor

123

Device manufacturer

No response

Device model

No response

Device platform info (optional)

Android log (logcat)

Evidence from the One Laptop per Child Program in Rural Peru

Hacker News
www.nber.org
2025-12-07 19:56:03
Comments...
Original Article

Working Paper 34495

DOI 10.3386/w34495

Issue Date

This paper examines a large-scale randomized evaluation of the One Laptop Per Child (OLPC) program in 531 Peruvian rural primary schools. We use administrative data on academic performance and grade progression over 10 years to estimate the long-run effects of increased computer access on (i) school performance over time and (ii) students’ educational trajectories. Following schools over time, we find no significant effects on academic performance but some evidence of negative effects on grade progression. Following students over time, we find no significant effects on primary and secondary completion, academic performance in secondary school, or university enrollment. Survey data indicate that computer access significantly improved students’ computer skills but not their cognitive skills; treated teachers received some training but did not improve their digital skills and showed limited use of technology in classrooms, suggesting the need for additional pedagogical support.

  • Copy Citation

    Santiago Cueto, Diether W. Beuermann, Julian Cristia, Ofer Malamud, and Francisco Pardo, "Laptops in the Long Run: Evidence from the One Laptop per Child Program in Rural Peru," NBER Working Paper 34495 (2025), https://doi.org/10.3386/w34495.

    Download Citation

Related

Topics

Programs

More from the NBER

2025, 17th Annual Feldstein Lecture, N. Gregory Mankiw," The Fiscal Future"

  • Feldstein Lecture

N. Gregory Mankiw, Robert M. Beren Professor of Economics at Harvard University, presented the 2025 Martin Feldstein...

2025 Methods Lecture, Raj Chetty, "Uncovering Causal Mechanisms: Mediation Analysis and Surrogate Indices"

  • Methods Lectures

SlidesBackground materials on mediationImai, Kosuke, Dustin Tingley, and Teppei Yamamoto. (2013). “Experimental Designs...

2025 International Trade and Macroeconomics, "Panel on The Future of the Global Economy"

  • Panel Discussion

Supported by the Alfred P. Sloan Foundation grant #G-2023-19633, the Lynde and Harry Bradley Foundation grant #20251294...

A geothermal amoeba sets a new upper temperature limit for eukaryotes

Hacker News
www.biorxiv.org
2025-12-07 19:26:08
Comments...
Original Article

Estimates are difficult for developers and product owners

Lobsters
thorsell.io
2025-12-07 19:14:45
Comments...
Original Article

Product Owner: Hey, how long do you believe Feature F will take?

Developer: Idk. We haven’t even started working on it and it’s bound to stir up some old issues.

Estimates come in various disguises, but when you peek under the trench coat there is always the question:

"How long -- and using what amount of resources -- will be required to do X ?"

When I wear the developer hat , it can be infuriating to attempt to give an answer. It’s difficult to estimate (or the product owner could do it themselves) and a lot of the time it can be difficult to see why the estimate is even important.

When I wear the product owner hat , estimates are a crucial piece of the puzzle that must be laid in an attempt to plan the short and long term life cycle of a product.

In this post I want to attempt to explore and elaborate on both sides, in an attempt to make developers understand why estimates are important to product owners and in order to help product owners see why developers so often despise having to estimate their work .

Why the PO wants you to estimate

As a Product Owner (PO), I am responsible for learning the market and customers’ needs and translating these into feature requests which developers can turn into actual features in our products . The means varies, but most organisations have some sort of backlog in which things to be acted upon are placed while they await being picked up by some developer or development team. We call these things user stories, issues, tickets, tasks, and probably many other things … The important thing for this discussion is that the items in the backlog are candidates for being implemented in our product and it’s the PO’s job to prioritise the backlog.

Why does the backlog need to be prioritised?

Because the inflow of items to the backlog is (pretty much always) higher than the speed at which the developers can implement them. Ergo, if the PO does not constantly learn the market and customers’ needs and prioritise the backlog accordingly, the developers might implement features that the users of the product are not interested in. Worst case? Existing users stop using the product and no new users buy it which will ultimately lead to bankruptcy.

But what about the estimates?

The above makes sense – I hope – but it doesn’t really pinpoint the need for estimates. Unfortunately, the job of a PO is not as easy as always prioritising in accordance to whatever the market wants. More often than not, the PO must also consider pre-communicated release dates and manage expectations.

I hate when release dates are communicated in advance. The only thing worse than release dates that are set in stone months ahead of time (I’m looking at you, Mr 12-week-increments-SAFe) are releases with pre-communicated content. Unfortunately, both are common. Often combined.

Imagine a backlog in which resides a really big feature. Something that is sought after, but will take a lot of time and resources to implement. The same backlog has a handful of smaller features which are not as requested as the big one. The PO would really like to include the big feature in the next release, but the next release date is not so far away. If the PO prioritises the big feature but it’s not done in time for the already communicated release date , the release will be severely lacking and considered a failure. In that case, the PO would rather include a couple of the smaller features. A safer bet, but the payoff is smaller.

THIS is why estimates matter so much to product owners. They must constantly run the above equation when they prioritise the teams’ backlogs. A constant risk/reward balancing act. They undoubtedly need help from the experts (the developers) to better understand the ramifications of the features they are proposing. If POs do not understand how big different work packages are, they cannot do their jobs in an effective way.

It gets worse

Instead of one PO there are now a couple of them. They are responsible for different parts of a larger product which requires the POs to coordinate both the date and the content of their releases. There is probably a main backlog describing upcoming features in the final product, as well as team backlogs where each team are assigned puzzle pieces which must be implemented and integrated in a coordinated fashion.

This is painful in multiple ways, but the most obvious issue is that – in order to have a functioning release – the POs must agree on the prioritisation of the main backlog and this will in turn affect the prioritisation of the team backlogs . The POs must each acquire information about how long it will take (and how costly it will be) to implement and to integrate the puzzle piece(s) they are responsible for into a cohesive feature. The tool for acquiring this idea ?

Estimates.

Technical debt

Programming is a craft. An art. My art, to some extent. I’m in my happy place when I get to succumb to a tricky task and surface a couple of days later with a solution to a problem that initially seemed impossible. As a developer, I want to build the best possible product. I dislike shortcuts. Half-arsed solutions. Fixes. Not because a single shortcut or fix will destroy a product, but because the technical debt they incur will accumulate over time and eventually erode the product from the inside out; making it ever more difficult to work with it and ultimately cause it to break.

Technical debt is – I believe – the main reason for conflict between a PO and a development team. A not so technically inclined PO will fail to see how detrimental technical debt is to the product and how painful it is for the developers to work in a code base with a high amount of debt.

Put in other words: If I’m tasked with implementing a new feature and I come across something in the code that is obviously smelly, error prone, or just not very good, I want to leave the code in better shape than I found it. Not taking time to “payoff” such debt once might not be the end of the world, but the hard coded quick-fix that you know ought to be generalised will likely bite you down the road. And if you have ignored updating dependencies for a couple of months and find yourself in a situation where you need to upgrade Package 1 , but it depends on a newer version of Packages 2 & 3 , which in turn requires a framework upgrade… Let’s just say the feature you’re working on will take a while longer.

Why developers HATE estimates

When a PO asks: “How long will it take to implement Feature F ?”, they aren’t just asking the developers to estimate the amount of time they think it will take to write the code for the feature. A good PO understands that implementing a new feature is an iterative process and that integration hell is a thing. An even better PO understands that they are also asking the team to estimate how many unforeseen issues they will encounter while implementing the feature.

This detail: The unforeseen issues , which the PO asks the developers to foresee, is key. It is – per definition – not possible to foresee something unforeseeable.

Many developers I’ve met dislike uncertainty. One of the things they appreciate most about coding is the deterministic aspect of it. You run the same program again and again and it returns the same results. 1 The journey on which we travel while writing the code is, however, not particularly deterministic.

It is true, that the more you code and the more familiar you get with a codebase, the more accurate your estimates will be. However, just the other day I was working on an issue which I had estimated would take approximately two days . All of a sudden, I realised that the simple change required updating a shared component that had been tightly coupled years ago. When I touched that code, dozens of failing tests appeared, each revealing another hidden dependency. Fixing those uncovered yet another module depending on outdated patterns. Halfway through, we decided we had to refactor the entire flow just to make the original change safe. My “two-day task” turned into two weeks of archaeological software excavation.

Could we have solved this quicker by not caring so much about the amount of technical debt we left in our wake? Probably.

Would we have encountered a two month excavation in the future? Probably.

It gets worse

According to Merriam-Webster : estimate is defined as:

To judge tentatively or approximately the value, worth, or significance of.

The very definition of estimates tells us that they are either tentative or approximate . As a developer, I choose to interpret the or as meaning that it could even be both.

When I started my career as a software developer, I really did not have an issue with estimates. We would refine our backlog and I would gladly give an estimate on various items. (1) Because I was fresh out of university and wanted to prove myself by doing a good job and not being too difficult, but more importantly: (2) because I had not understood that my estimates would soon be used against me.

I soon learned that my team’s estimates were not interpreted and used as estimates . They were used as deadlines . If we broke down a feature into its reasonable components (an error prone science, which introduces uncertainties, on its own) and estimated the parts accordingly, the PO would often take the sum of the parts and communicate it to their colleagues as: “This is the time we will be done.”

Two things came out of this:

  1. My team (consisting mostly of newly graduated developers) became much more reluctant to estimate.
  2. When we estimated we always padded our actual beliefs , significantly, to give ourselves a buffer.

The estimates stopped being estimates. They became safety railings against being held accountable for unreasonable expectations.

The clash

Do you see the problem?

Do you see a solution?

I believe the overarching problem with estimates stems from expectations. Somewhere, someone, communicates something to the users/customers of the product, which sets expectations the rest of the organisation are then forced to live up to. In a small company, it might very well be the PO who does that communication but in a larger organisation the PO is likely as helpless as the developers w.r.t. having a say about the product’s roadmap.

The “solution” is simple: Stop communicating new features in advance. Stop setting more or less arbitrary deadlines 2 . Let the PO tell the developers what features they want, in what order, and let the developers do what they do best: Code!

But these deadlines are there for a reason. If your company builds a product which assists people doing their yearly tax returns, a missed delivery window will result in the entire revenue opportunity for that year being missed. Resources (most often in terms of salaries to employees) will have been poured into a project and if there’s no payoff in terms of additional sales, it could lead to a need for finding other ways to reclaim those resources; often in terms of reduced costs, which universally means: lay-offs.

Therefore, it’s in everyone’s best interest to play along. We play the estimates game even though it’s a bad way (but also the best we know of) to help each other do our respective jobs.

What about DevOps?

You didn’t think I’d miss an opportunity to talk about DevOps, did you?

Flow is a key concept within DevOps which describes an organisation’s ability to reduce bottlenecks and increase the pace at which they are able to deliver new versions of their product(s). High flow is synonymous with frequent deliveries and updates of our product(s).

The concepts from DevOps do not directly address the issue with estimates, but there are tools which can be used to reduce the risk associated with delivering software. Flow can inform how we tackle technical debt and how we make sure we don’t fall behind on our dependencies. Flow can also help us identify issues in our product’s life cycle as well as help us understand how to get rid of the issues.

Flow is one of The Three Ways in DevOps and if you want to learn more, feel free to reach out. I give presentations on various topics related to DevOps and I can come to your company and give a course about DevOps tailored to your company’s needs.

Conclusion

Estimates – as defined in the English language – isn’t really the problem here. The problem is when estimates are treated as predictions, deadlines, and used to put pressure on developers who are just trying to do their jobs. Estimates – the way they are used in our industry today – hurts people and reduces the psychological safety in our organisations. I believe we would be better off if we could work in a way that allows developers to be transparent and continuously communicate updated estimates as development progresses.

Then again, product owners are people too! As developers we must understand that POs are under pressure too. We must help them and the best way to help them is to continuously provide them with updates about how development is progressing and whether we have encountered anything that we believe will significantly alter the original estimate we gave.

Millions of Americans mess up their taxes, but a new law will help

Hacker News
www.wakeuptopolitics.com
2025-12-07 19:03:16
Comments...
Original Article

There’s a comedian named Joe Zimmerman who has a bit on — of all things — the U.S. tax system.

The video is below, but it goes like this:

My biggest fear about the government is just: taxes. I worry about it all year long, because other countries, the government will tell you what you owe, and then you just have to pay it. Here, the government’s like, “OK, what do you think you owe us?”

“I have no idea. Just tell me.”

“Nope, you gotta add it up. And then give it up.”

“Well, I’m really unorganized. What happens if I add it up wrong?”

“You could go to jail.”

Zimmerman is right that there are about 30 foreign countries that use “return-free” tax filing, in which the government assesses at least some citizens’ tax liabilities themselves based on information it already receives from employers. These countries send prepopulated tax returns to these citizens; the taxpayer can either say “Yep, looks good” and pay what the government says they owe or they can fill in missing information the government might not have had.

Versions of this system are used in the United Kingdom, Japan, South Korea, New Zealand, Germany, Italy, Spain, Denmark, Sweden, and the Netherlands, among other countries. California also briefly had a return-free system for state taxes, championed by Sam Bankman-Fried’s law professor father .

The idea of applying this nationwide in the U.S. has been endorsed by everyone from Ronald Reagan to Barack Obama , although the federal government really only has enough information to accurately prepopulate tax returns for around 45% of Americans . (Those other countries have much simpler tax codes than we do.)

Of course, Zimmerman is exaggerating for comedic effect, but where he’s wrong is that it’s highly unlikely you would go to jail for adding things up wrong on your taxes.

Not everyone is great at math, and there are so many numbers involved in tax returns that it’s very easy to make a mistake. You might be someone who is trying to pay all your taxes correctly, like Zimmerman, but end up making an honest error completely by accident. In these situations, the IRS has a system — in theory! — that’s supposed to make it easy to flag such math errors and get them quickly corrected.

In reality, however, this system has some holes in it, in a way that makes paying taxes more confusing for millions of Americans each year. Last week, President Trump signed a new law, the Internal Revenue Service Math and Taxpayer Help (IRS MATH) Act , designed to simplify at least this one part of our complicated tax code.

The legislation was bipartisan: sponsored by Sens. Bill Cassidy (R-LA) and Elizabeth Warren (D-MA) in the Senate and Reps. Randy Feenstra (R-IA) and Brad Schneider (D-IL) in the House, and approved unanimously by both chambers.

This morning, as part of my ongoing effort to let you in on the under-the-radar things Congress is actually getting done, let’s take a look at this new law and the problem it’s trying to fix. Paying taxes is a universal experience — and any of us could make an error at any time — which makes this law widely relevant to Americans, even if it’s received almost no coverage from major news outlets.

Ever since 1926, the IRS has had what’s called “math error authority”: the ability to quickly correct tax returns with simple arithmetic mistakes. As a House committee explained at the time:

In the case of a mere mathematical error appearing upon the face of the return, assessment of a tax due to such mathematical error may be made at any time, and that such assessment shall not be regarded as a deficiency notification.

In other words: No, you’re not going to jail for a simple math error!

In the modern era, the IRS feeds all of the tax returns it receives through a computer program, which automatically flags if someone added things up wrong. When that happens, the IRS then sends a “math error notice” to the taxpayer; the agency also sends such notices for other mistakes that are easily fixed, like if someone put the right information in the wrong place, if a return is internally inconsistent, or if someone gives the wrong Social Security Number or forgets to include one entirely.

About 2 million of these notices are usually sent out each year — although, as you can see below in this chart by the Tax Policy Center, that number skyrocketed to 17 million during the pandemic (primarily because of errors people made in claiming stimulus checks and the expanded Child Tax Credit).

If someone receives a math error notice in the mail, they have 60 days to quibble with it. If they don’t appeal within that timeframe, they are considered to owe the corrected amount the IRS is asking for. When everything works well, this is a win-win for the government and for the taxpayer: there’s no reason for either side to go through a more drawn-out and expensive audit or legal process if the only issue is that John wrote “$3,650” where he meant to put “3,506.”

In 2010, math error notices telling a taxpayer they underpaid led to the IRS receiving a total of $9.5 billion in tax revenue the agency was owed; math error notices about overpayments led to the IRS giving back a total of $6.2 billion.

Here’s the issue, though: The IRS notices are often really vague! Current law requires the agency to “set forth the error alleged and an explanation thereof,” but the IRS hasn’t really been doing that. Here’s an example of the type of notices people received for errors claiming stimulus checks during the pandemic:

We changed the amount claimed as Recovery Rebate Credit on your tax return. The error was in one or more of the following:

  • The Social Security number of one or more individuals claimed as a qualifying dependent was missing or incomplete.

  • The last name of one or more individuals claimed as a qualifying dependent does not match our records.

  • One or more individuals claimed as a qualifying dependent exceeds the age limit.

  • Your adjusted gross income exceeds $75,000 ($150,000 if married filing jointly, $112,500 if head of household).

  • The amount was computed incorrectly.

That’s not really helpful, is it? Millions of Americans get letters like this during Covid, which called attention to this issue , since many of them were very confused by the notices. If I made a mistake, why not just tell me what it is? Why give me a list of five potential mistakes I might have made and make me figure out which one? “The amount was computed incorrectly?” Can you maybe show me where and explain what about it was incorrect?

And, of course, people are only given 60 days to track down the error before they lose the chance to appeal. (They can still get the money back eventually if they think the “error” wasn’t an error, but after the 60-day mark, you still have to pay the money and then try to get it back as a refund, rather than having the chance to convince the IRS you don’t have to pay it at all.) Many Americans don’t understand the subtleties of the tax code. Or don’t have the time to rifle back through their tax returns to try to find an unspecified error. Or don’t know who to ask for help. Or have difficulty speaking English. This process could be so simple — “You made X error in Y place, fix it and we’ll be all set” — and yet it ends up adding a burden and confusing people every year.

To make matters worse, during Covid, some of the IRS letters didn’t tell people they could dispute the error or that they only had 60 days to do so (though the agency later fixed that practice); the notices also often neglect to tell taxpayers who they can call to help understand their mistake, or hide that information in a place where people won’t see it.

Did you know there’s someone at the IRS whose job it is to look out for you, the taxpayer, and lobby on your behalf? That person is the National Taxpayer Advocate, who leads an office of 1,500 employees who are supposed to be (as their slogan goes) “Your Voice at the IRS.” The position was created by the second Taxpayer Bill of Rights , passed by Congress in 1996; it is currently held by Erin Collins, who has had the job since 2020 and spent decades working in tax law before that.

Each year, the National Taxpayer Advocate puts out something called the “Purple Book,” a list of legislative recommendations she thinks Congress should enact to improve the life of the taxpayer. This year’s Purple Book was 198 pages long and included 69 recommendations.

For each of the last four years, Collins has proposed that Congress pass a law telling the IRS to fix their math error system, so that the notices sent out to Americans actually tell them what their mistake was and how long they have to appeal it. (It’s Recommendation #9 on this year’s list.) The IRS MATH Act, signed by Trump last week, is Congress finally taking her up on it.

Under the measure, the IRS math error notices will have to describe the “mathematical or clerical error” a taxpayer made “in comprehensive, plain language,” including by explaining to them “the nature of the error” and pointing them to “the specific line of the return on which the error was made.”

The IRS will also have to give an “itemized computation” of how the correction will change their adjusted gross income, taxable income, deduction amount, or tax credits.

Also, the notice will have to include the date by which the taxpayer has to appeal the correction before it becomes the updated amount they owe — “in bold, font size 14, and immediately next to the taxpayer’s address on page 1 of the notice,” the law specifically says — plus it has to give the IRS’ phone number so taxpayers know how to call if they want more information.

And in case all of that wasn’t clear enough, there’s also this:

This is Congress saying, We’re on to you, IRS. We know your games. NO LISTS OF POTENTIAL ERRORS. We’re saying up front that that’s not gonna satisfy our requirement.

All of this has to go into effect within 12 months. Finally, the law also includes a provision to create a pilot program to send a trial number of these notices by certified mail — which requires a recipient to sign and say they received a certain letter — as an attempt to ensure fewer Americans miss the letters in their mailboxes or think they might be spam.

The law requires the IRS to work with the National Taxpayer Advocate on the program, and then to report to Congress after 18 months on whether the pilot program improved taxpayer response rates to math error notices.

Speaking of the National Taxpayer Advocate, the House unanimously passed two more bipartisan bills this week to satisfy more of her Purple Book recommendations. The chamber approved the Fair and Accountable IRS Reviews Act , which would require an IRS employee to obtain written approval from their immediate supervisor before telling a taxpayer they owe a penalty ( Recommendation #33 ), and the Tax Court Improvement Act , which makes changes to make the tax court process more efficient and fairer for taxpayers ( Recommendations #45 and #47 ).

Earlier this year, the House also passed the National Taxpayer Advocate Enhancement Act , which will allow the National Taxpayer Advocate to hire tax attorneys for her office, to help improve her efforts to work on behalf of taxpayers ( Recommendation #37 ). These measures have not yet been taken up by the Senate.

Most of the Purple Book recommendations — like the IRS MATH Act — are common-sense and bipartisan. “No one should have to spend a fortune on a lawyer or hours trying to figure out what went wrong on their taxes when the IRS already knows the answer,” Sen. Warren has said about the new law.

“An honest mistake on a tax return should be met with clear guidance from the IRS, not confusion,” Sen. Cassidy echoed.

Is this the biggest problem in the world? No. But all of us make math errors from time to time. And, each year, millions of Americans make mistakes on their tax returns that they’re willing to fix — but the IRS makes things unnecessarily confusing by not simply telling them how to. After a bipartisan effort by Cassidy, Warren, and others, life will be made at least a little bit easier for these taxpayers.

One recommendation down, 68 to go.

Discussion about this post

Ready for more?

Defeating Prompt Injections by Design

Lobsters
arxiv.org
2025-12-07 19:02:02
Comments...
Original Article
Timed out getting readerview for https://arxiv.org/pdf/2503.18813

F35 Fighter Jet’s C++ Coding Standards

Lobsters
www.stroustrup.com
2025-12-07 18:57:17
Comments...
Original Article
No preview for link for known binary extension (.pdf), Link: https://www.stroustrup.com/JSF-AV-rules.pdf.

Show HN: Spotify Wrapped but for LeetCode

Hacker News
github.com
2025-12-07 18:42:57
Comments...
Original Article

leetcode wrapped

a "spotify wrapped" style recap for your leetcode journey.

deployed at leetcodewrapped.com

quick start

  1. install dependencies:

  2. run locally:

  3. build for production:

directory structure 

├── src/
│   ├── api/            # api wrappers (leetcode, firebase, etc.)
│   ├── components/     # react components
│   │   └── slides/     # individual wrapped slides (intro, stats, etc.)
│   ├── App.jsx         # main application logic
│   ├── main.jsx        # entry point & providers
│   └── firebase.js     # firebase configuration
├── functions/          # cloudflare functions (server-side proxy)
├── public/             # static assets
├── firestore.rules     # database security rules
└── index.html          # html entry point

tech stack

  • frontend : react, vite, framer motion
  • backend : cloudflare pages functions (proxy), firebase (db & auth)
  • analytics : posthog

Note: The last 5 slides are not necessarily specific to 2025 because of leetcode's graphql api only allows querying up to 20 of the latest submissions from an unauthenticated user.

However, if you pass a LEETCODE_SESSION cookie (obtained from leetcode.com, open dev tools -> application -> cookies) with your request you can query all of your accounts submissions. You could also use the calendar endpoint query all of your submissions in the past year, and thus create a much more nuanced leetcode wrapped. (ex: You struggled with this problem the most in 2025.)

I was hesitant to implement this because obviously people wouldn't trust inputting a cookie into a form, but if this repo gets lots of stars I'll make a chrome extension that gets around this.

Note Note: I used firebase firestore database and trigger email extension to send out emails to users, as well as posthog for analytics. However, you can still clone the repository and run it locally without these features.

Why Fighter Jets Ban 90% of C++ Features [video]

Hacker News
www.youtube.com
2025-12-07 18:07:06
Comments...
Original Article

I Tried and Failed to Rebuild the 1996 Space Jam Website with Claude

Hacker News
j0nah.com
2025-12-07 17:18:54
Comments...
Original Article

Can Claude Recreate the 1996 Space Jam Website? No. Or at least not with my prompting skills. Note: please help, because I'd like to preserve this website forever and there's no other way to do it besides getting Claude to recreate it from a screenshot. Believe me, I'm an engineering manager with a computer science degree. Please please please help 😞

Final note: I use "he" to refer to Claude, which Josh finds ridiculous.

Space Jam, 1996

For those who don't know, Warner Bros keeps this anachronistic website online that was released in 1996 to accompany the Space Jam movie.

claud blogBounty Can Claude Recreate the 1996 Space Jam WeScreenshot 2025 11 26 at 12 18 41 PM

It's a classic example of early web era design. Simple, colorful, and sparks joy. We're going to find out if we can get Claude to recreate it using only a screenshot.

Set Up

At a minimum, I'm providing Claude:

  • a screenshot of the website
  • all of the assets the website uses

To track Claude's inner monologue and actual API calls, I set up a man-in-the-middle proxy to capture the full conversation between Claude Code and Anthropic's API. This logs everything: user prompts, Claude's responses, tool invocations (Read, Write, Bash commands), etc. Each attempt generates a traffic.log file with the raw API traffic, which I then parse for easier analysis.

Part 1: Claude the Realist

The Space Jam website is simple: a single HTML page, absolute positioning for every element, and a tiling starfield GIF background. The entire page uses absolute positioning with pixel specific left/top values. The total payload is under 200KB.

Given that Claude has all of the assets + screenshots of the website, I assume this should be relatively boring. He'll nail it, and we'll move on to something much more. A mildly cute example of agentic HTML generation…

I tell Claude: 

I am giving you:
1. A full screenshot of the Space Jam 1996 landing page.
2. A directory of raw image assets** extracted from the original site
Your job is to recreate the landing page as faithfully as possible, matching the screenshot exactly.

What he produces is actually not that bad. But it's not right. From a distance, the layout kind of resembled the original: planets arranged in an ellipse around the logo, little yellow labels where the buttons go. But, the orbital pattern was off, almost diamond shaped and symmetrical.

claud blogBounty Can Claude Recreate the 1996 Space Jam WeScreenshot 2025 11 27 at 1 28 14 PM

Claude, however, was thrilled with himself.

Perfect! I've successfully recreated the Space Jam 1996 landing page.

Further, he brags that he had:

studied the orbital layout analyzed spacing relationships positioned planets precisely

Digging through the logs I found it interesting that Claude actually did notice the planets were arranged in a deliberate way, so much so that it's called out twice in both the screenshot analysis and CSS construction, but he failed to recreate the pattern faithfully.

Okay, fine. Maybe he needed a nudge to get the orbit right. So for my next attempt, I try to push him to focus on understanding the orbital pattern and I ask him to explain his reasoning before generating his HTML. I was hoping to understand the delta between what is there and what he thought he was seeing. In my prompt, I outline a set of sections for him to consider. Each one of these sections also includes a number of sub-questions, which I won't include here for the sake of brevity. This made things significantly worse. 

Please follow this structure exactly in your reasoning explanations:
1. Perception Analysis
2. Spatial Interpretation
3. Reconstruction Plan

Claude didn't ignore my instructions (not always a given) and things seemed promising until I realized he was ignoring his own analysis during the HTML generation phase. He would say things like "the orbit radius appears to be 220 pixels" and then place the planets directly next to the logo. His self critique was surprisingly accurate. He correctly identifies the areas where he was wrong with decent detail, but somehow those observations never make it into subsequent iterations.

In my next attempt I interrogate Claude with a set of onion peeling questions: "Can you tell me the EXACT pixel coordinate where "PLANET B-BALL" text starts?"

"No, I cannot measure exact pixel coordinates. I can only make visual estimations."

I asked him a few more questions:

  • Can you extract exact pixel coordinates? "No."
  • Can you measure exact distances? "No."
  • Confidence you can get within 5 pixels? "15 out of 100."

Oh. This explains a lot. But it raises a bigger question to me: "Why can't he measure?" It's a screenshot. The pixels are right there. Claude clearly understood the structure, but he couldn't recreate it with any precision. Also, I'm not even sure I trust Claude. Either way, this (naively) surprised me, so I canceled coffee with my friends in order to spend the afternoon trying to give my guy more tools.

Before I start I execute one more attempt and ask him: "Would you bet $1000 on your HTML matching this screenshot exactly?"

Absolutely not

Part 2: Claude the Unreliable Narrator

Maybe he just needs a little help.

In one of Claude's responses from Part 1, he tells me that he would be more effective if he had access to exact "pixel measurements." so I build a few tools to make it impossible for Claude to mis-measure anything:

  • Grid overlays and a script to generate grid overlays on screenshots
  • labeled pixel coordinate reference points
  • color-diff comparison (this ignores the background which was giving Claude false positives because of how much black there was)
  • Tool to take screenshots of his index.html file to compare iteratively with the original

Here are three grid versions Claude generated which I am including because I find them aesthetically pleasing.

Claude loved the grids. As decoration.

I put together a new prompt: same screenshot, same assets folder. I even included some grid screenshots so Claude wouldn't have to remember to do it himself. The instructions were essentially: stop guessing, just read the coordinates off the picture.

Claude's new attempt still wasn't correct. The orbit was better: closer to the original but somehow compressed and smooshing (a technical word) into the Space Jam logo. If I squint, I could convince myself that there was at least a hint that he'd stopped freehanding and started using something like measurements.

Original claud blogBounty Can Claude Recreate the 1996 Space Jam WeScreenshot 2025 11 27 at 2 24 39 PM

Claude's Attempt claud blogBounty Can Claude Recreate the 1996 Space Jam WeScreenshot 2025 11 27 at 2 24 49 PM

When I dug into the logs, it appeared that Claude actually did use the grids. He pulled out these numbers:

  • Center at (961, 489)
  • Logo "centered at approximately (755, 310)"
  • Planet B-Ball at "approximately (850, 165)"
  • and so on down the list

In one iteration, Claude built himself a helper: compare.html a little side by side viewer so he could look at his screenshot and the reference together. It didn't help him at all, but my God was he convinced it did.

"Perfect! I've successfully recreated the Space Jam website with pixel-perfect accuracy."

I love the optimism my dog.

The actual progression tells a different story. Going through the iterations:

  • Iteration 1 (50px grid): he notices things are off and makes a few conservative tweaks — moves Planet B-Ball from (850, 165) to (800, 120), shifts Lunar Tunes from (925, 195) to (950, 200). These are 15 - 50 pixel changes, tiny nudges.
  • Iteration 2 (25px grid): he decides he needs "more precise positioning" and shifts the entire orbit inward by ~20 pixels. Planets go from roughly a 250px radius to ~230px. He is now confidently converging on the wrong answer.
  • Iteration 3 (5px grid): he shuffles around a lot of deck chairs in the name of micro adjustments. 5 - 10 pixel tweaks: Planet B-Ball from (800, 120) to (805, 125), that kind of thing.
  • Iteration 4: more "fine-tuning based on exact grid measurements." Site Map drifts from (755, 460) to (750, 455). The numbers look careful; the layout does not lol.

By the final iteration, Claude announces:

"Now the positioning should be much more accurate!"

Across all five iterations, he's moved planets maybe 50 - 75 pixels total when they needed to move 150 - 200 pixels outward. The orbital radius never expands beyond ~250px when it should be closer to 350 -400px. The planets stay trapped in this ever compressing orbit while Claude dances around telling a story about steady convergence:

"Getting closer!"

"Much better alignment!"

"Nearly perfect now!"

He is absolutely not getting closer.

Before giving up on tools entirely, I had one more idea. I created a tool that would split the screenshot into six regions. Claude could then compare his output to the reference region by region.

The prompt was getting desperate at this point lol: 

## INITIAL ANALYSIS - DO THIS FIRST
Before creating index.html, study the reference in detail using zoom inspection:
python3 split.py reference.png
This creates 6 files showing every detail

Claude ran the tool and the logs look promising.

Examining region 1 (top-left): I can see Jam Central and Press Box Shuttle clearly at 2x zoom Examining region 2 (top-right): Planet B-Ball and Lunar Tunes are visible with precise spacing Examining region 4 (middle-right): Jump Station appears approximately 280px from center

Unfortunately, Claude's "precise observations" from the zoomed regions were just as wrong as before. He'd look at a planet and confidently declare it was at position (750, 320) when it was actually at (850, 380). The split did not appear to help him measure or get a more accurate picture of planet spacing.

What makes this phase ~~depressing~~ interesting is that the tools, despite invalidating his result, seem to lock in the wrong answer. Once he's picked an internal picture of the layout ("the orbit radius is about 230px"), the grids and the compare viewer don't correct it. They just help him make more confident micro moves around his invented orbit. Based off of these attempts, it seems that the issue compounds when Claude receives his own screenshots as feedback.

My very rough read of Anthropic's "Language Models (Mostly) Know What They Know" , is that models can become overconfident when evaluating their own outputs, in part because they cannot distinguish the tokens they generated from tokens provided by someone else / an external source. So, when Claude is asked to judge or revise content that originated from itself, it treats that material as if it were "ground truth."

This kind of fits what I'm seeing in the logs. Once Claude's version existed, every grid overlay, every comparison step, every "precise" adjustment was anchored to his layout, not the real one. At the end of all this, I'm left with the irritating fact that, like many engineers, he's wrong and he thinks he's right.

What this teaches me is that Claude is actually kind of a liar, or at least Claude is confused. However, for the drama, I'll assume Claude is a liar.

Part 3: Claude the Blind

At this point I had tried grids, comparisons, step-by-step corrections, letting Claude narrate his thought process, and every combination of tools I could bolt onto the interaction. None of it seemed to help nor explain by why his single digit precision updates were disembodied from the actual layout.

Before getting to the final experiment, here's the mental model I was forming about Claude's vision. The vision encoder converts each 16 x 16 block of the image into a single token. So instead of geometry, he sees semantics: "near," "above," "roughly circular." When he says "approximately 220px radius," he's not measuring anything. He's describing the idea of a radius. He excels at semantic understanding ("this is a planet," "these form a circle") but lacks the tools for working with visual media. It explains why his perception is good. He always knows a planet is a planet but the execution is never precise.

I'm getting frustrated and I haven't left my apartment in days so I turn to some research. GPTing around, I found "An Image is Worth 16x16 Words" . I have no idea if Claude uses this exact architecture or anything close to it, but the intuition seemed right. The paper (after I made ChatGPT explain it to me) explains that the the image is chopped into fixed patches, each patch gets compressed into a single embedding, and whatever details lived inside those pixels vanish.

Oooh.

Assuming this applies, a lot of the failures suddenly make sense. Most planets on the Space Jam screenshot are maybe 40 - 50 pixels wide. That's two or three patches. A three patch planet is basically a blob to him. Claude knows it's a planet, but not much else. The orbit radius only spans a couple dozen patches total. Tiny changes in distance barely show up in the patch embeddings.

But this raised a new and final idea. If the 40px planets turn into fuzzy tokens, what if I make them bigger? What if I give Claude a 2x zoomed screenshot? Would each planet spans 10 - 15 patches instead of two or three? Maybe this gives him a more crisp understanding of the spatial relationships and a better chance at success.

I deleted most of the prompt and tools and just gave Claude this 2x'd screenshot

claud blogBounty Can Claude Recreate the 1996 Space Jam Wereference zoom

I plead with Claude 

CRITICAL: remember that the zoomed image is zoomed in to 200%. When you're creating your version, maintain proper proportions, meaning that your version should keep the same relative spacing as if it were just 100%, not 200%.

but he does not listen

claud blogBounty Can Claude Recreate the 1996 Space Jam WeScreenshot 2025 11 28 at 1 52 39 PM

😞

My best explanation for all of this is that Claude was working with a very coarse version of the screenshot. Considering the 16 x 16 patch thing from earlier it sort of helps me understand what might be happening: he could describe the layout, but the fine grained stuff wasn't in his representation. And that weird tension I kept seeing , where he could describe the layout correctly but couldn't reproduce it, also looks different under that lens. His explanations were always based on the concepts he got from the image ("this planet is above this one," "the cluster is to the left"), but the actual HTML had to be grounded in geometry he didn't have. So the narration sounded right while the code drifted off.

After these zoom attempts, I didn't have any new moves left. I was being evicted. The bank repo'd my car. So I wrapped it there.

End

Look, I still need this Space Jam website recreated. If you can get Claude to faithfully recreate the Space Jam 1996 website from just a screenshot and the assets folder, I'd love to hear about it.

Based on my failures, here are some approaches I didn't try:

  1. Break the screen into quadrants, get each quadrant right independently, then merge. Maybe Claude can handle spatial precision better in smaller chunks.
  2. Maybe there's some magic prompt engineering that unlocks spatial reasoning. "You are a CSS grid with perfect absolute positioning knowledge…" (I'm skeptical but worth trying).
  3. Providing Claude with a zoom tool and an understanding of how to use the screenshots might be an effective path.

For now, this task stands undefeated. A monument to 1996 web design and a humbling reminder that sometimes the simplest tasks are the hardest. That orbital pattern of planets, thrown together by some Warner Brothers webmaster 28 years ago, has become an inadvertent benchmark for Claude.

Until then, the Space Jam website remains proof that not everything old is obsolete. Some things are just irreproducibly perfect.

Semantic Compression (2014)

Hacker News
caseymuratori.com
2025-12-07 16:55:15
Comments...
Original Article

We all know how to program in C++, don’t we? I mean, we’ve all read a selection of wonderful books by the gaggle of bearded fellows who defined the language in the first place, so we’ve all learned the best ways to write C++ code that solves real-world problems.

First, you look at the real world problem  —  say, a payroll system  —  and you see that it has some plural nouns in it: “employees”, “managers”, etc. So the first thing you need to do is make classes for each of these nouns. There should be an employee class and a manager class, at least.

But really, both of those are just people. So we probably need a base class called “person”, so that things in our program that don’t care whether you’re an employee or a manager can just treat you as a person. This is very humanizing, and makes the other classes feel less like cogs in a corporate machine!

There’s a bit of a problem, though. Isn’t a manager also an employee? So manager should probably inherit from employee, and then employee can inherit from person. Now we’re really getting somewhere! We haven’t actually thought about how to write any code, sure, but we’re modeling the objects that are involved, and once we have those solid, the code is just going to write itself.

Wait, shoot  —  you know what? I just realized, what if we have contractors? We definitely need a contractor class, because they are not employees. The contractor class could inherit from the person class, because all contractors are people (aren’t they?). That would be totally sweet.

But then what does the manager class inherit from? If it inherits from the employee class, then we can’t have managers who work on contract. If it inherits from the contractor class, then we can’t have full-time managers. This is turning out to be a really hard programming problem, like the Simplex algorithm or something!

OK, we could have manager inherit from both classes, and then just not use one of them. But that’s not type-safe enough. This isn’t some sloppy JavaScript program! But you know what? BAM! I’ve got the solution right here: we templatize the manager class. We templatize the manager class on its base class, and then everything that works with manager classes is templatized on that as well!

This is going to be the best payroll system ever! As soon as I get all these classes and templates spec’d out, I’m going to fire up my editor and get to work on the UML diagrams.

Programmers Posting Programming Posts

It’d be great if everything I just wrote had been farcical, but sadly, there’s actually a lot of programmers in the world who think like this. I’m not talking about “Bob the Intern”  —  I’m talking about all kinds of programmers, including famous programmers who give lectures and write books. I am also sad to say that there was a time in my life when I thought this way, too. I was introduced to “object oriented programming” when I was 18, and it took me until I was about 24 to realize it was all a load of horseshit (and the realization was thanks in no small part to my taking a job with RAD Game Tools, which thankfully never bought into the whole OOP nightmare).

But despite the fact that many programmers out there have gone through bad phases like this and eventually come to smart conclusions about how to actually write good code efficiently, it seems that the landscape of educational materials out there still overwhelmingly falls into the “objectively bad” category. I suspect this has something to do with the fact that good programming seems very straightforward once you know how to do it, unlike, say, a fancy math technique that retains its sexiness and makes you want to spend the time to post about it. So, although I don’t have any data to back this up, I strongly suspect that experienced programmers rarely spend time posting about how they program because they just don’t think it’s anything special.

But they should! It may not be special, but it’s necessary, and if good programmers don’t start posting about how to do good programming, we’ll never get out of this nasty place where everyone has to go through six years of writing horrible object-oriented programs before they realize they’re wasting their time. So what I’d like to do with this next set of Witness articles is spend some serious word count talking about the purely mechanical process of putting code into a computer, and it is my sincere hope that other experienced programmers out there will take some time to do the same. Personally, I’d love to read more about the techniques actual good programmers out there use when they sit down to code.

To start things off, I am going to detail a straightforward set of code transformations that I did on The Witness’s editor code. In the coming weeks, I’ll move from that into some larger examples where I wrote more pieces from scratch, but the entire time I’ll be focusing solely on code and how it’s structured. Nothing that I’m going to cover has any fancy algorithms or math or anything, it’s all just pure plumbing.

Jon Starts Things Off Right

In the built-in editor for The Witness, there is a piece of UI called the “Movement Panel”. It is a floating window with some buttons on it that are used to perform operations on entities like “rotate 90 degrees”. Originally it was quite small and had only a few buttons, but when I started working on the editor, I added a bunch of features that needed to go in the movement panel. This was going to expand its contents considerably, and it meant I had to learn how to add elements to the UI, which I’d never done before. I examined the existing code, which looked like this:

int num_categories = 4 ;

int category_height = ypad + 1.2 * body_font -> character_height;

float x0 = x;

float y0 = y;

float title_height = draw_title( x0, y0, title);

float height = title_height + num_categories * category_height + ypad;

my_height = height;

y0 -= title_height;

{

y0 -= category_height;

char * string = "Auto Snap" ;

bool pressed = draw_big_text_button( x0, y0, my_width, category_height, string);

if ( pressed) do_auto_snap( this);

}

{

y0 -= category_height;

char * string = "Reset Orientation" ;

bool pressed = draw_big_text_button( x0, y0, my_width, category_height, string);

if ( pressed) {

// ...

}

}

// ...

The first thing I noticed here was that Jon , the original programmer, did a really nice job setting me up for success with what I was about to do. A lot of times, you open up some code for something simple like this, and you find that it is just a massive tangle of unnecessary structure and indirection. Here, instead, we find an extremely straightforward series of things happening, that read exactly like how you would instruct a person to draw a UI panel: “First, figure out where the title bar should go. Then, draw the title bar. Now, below that, draw the Auto Snap button. If it’s pressed, do auto snapping…” This is exactly how programming should go. I suspect that most anyone could read this code and know what it was doing, and probably intuit how to add more buttons without having to read anything beyond just this excerpt.

However, nice as the code was, it was obviously not set up for doing large amounts of UI, because all the layout work was still being done by hand, in-line. This is mildly inconvenient in the snippet above, but gets more onerous once you consider more complex layouts, like this piece of the UI that has four separate buttons that occur on the same row:

{

y0 -= category_height;

float w = my_width / 4.0 f;

float x1 = x0 + w;

float x2 = x1 + w;

float x3 = x2 + w;

unsigned long button_color;

unsigned long button_color_bright;

unsigned long text_color;

get_button_properties( this, motion_mask_x, & button_color, & button_color_bright, & text_color);

bool x_pressed = draw_big_text_button( x0, y0, w, category_height, "X" , button_color, button_color_bright, text_color);

get_button_properties( this, motion_mask_y, & button_color, & button_color_bright, & text_color);

bool y_pressed = draw_big_text_button( x1, y0, w, category_height, "Y" , button_color, button_color_bright, text_color);

get_button_properties( this, motion_mask_z, & button_color, & button_color_bright, & text_color);

bool z_pressed = draw_big_text_button( x2, y0, w, category_height, "Z" , button_color, button_color_bright, text_color);

get_button_properties( this, motion_local, & button_color, & button_color_bright, & text_color);

bool local_pressed = draw_big_text_button( x3, y0, w, category_height, "Local" , button_color, button_color_bright, text_color);

if ( x_pressed) motion_mask_x = ! motion_mask_x;

if ( y_pressed) motion_mask_y = ! motion_mask_y;

if ( z_pressed) motion_mask_z = ! motion_mask_z;

if ( local_pressed) motion_local = ! motion_local;

}

So, before I started adding lots of new buttons, I already felt like I should spend a little time working on the underlying code to make it simpler to add new things. Why did I feel that way, and how did I know what “simpler” means in this case?

I look at programming as having essentially two parts: figuring out what the processor actually needs to do to get something done, and then figuring out the most efficient way to express that in the language I’m using. Increasingly, it is the latter that accounts for what programmers actually spend their time on: wrangling all those algorithms and all that math into a coherent whole that doesn’t collapse under its own weight.

So any experienced programmer who’s any good has had to come up with some way  —  if even just by intuition  —  of thinking about what it means to program efficiently. By “efficiently”, this doesn’t just mean that the code is optimized. Rather, it means that the development of the code is optimized  —  that the code is structured in such a way so as to minimize the amount of human effort necessary to type it, get it working, modify it, and debug it enough for it to be shippable.

I like to think of efficiency as holistically as possible. If you look at the development process for a piece of code as a whole, you won’t overlook any hidden costs. Given a certain level of performance and quality required by the places the code gets used, beginning at its inception and ending with the last time the code is ever used by anyone for any reason, the goal is to minimize the amount of human effort it cost. This includes the time to type it in. It includes the time to debug it. It includes the time to modify it. It includes the time to adapt it for other uses. It includes any work done to other code to get it to work with this code that perhaps wouldn’t have been necessary if the code were written differently. All work on the code for its entire usable lifetime is included.

When considered in this way, my experience has led me to conclude that the most efficient way to program is to approach your code as if you were a dictionary compressor. Like, literally, pretend you were a really great version of PKZip, running continuously on your code, looking for ways to make it (semantically) smaller. And just to be clear, I mean semantically smaller, as in less duplicated or similar code, not physically smaller, as in less text, although the two often go hand-in-hand.

This is a very bottom-up programming methodology, a pseudo-variant of which has recently gained the monicker “refactoring”, even though that is a ridiculous term for a number of reasons that are not worth belaboring at the moment. I also think that the formal “refactoring” stuff missed the main point, but that’s also not worth belaboring. Point being, they are sort-of related, and hopefully you will understand the similarities and differences more over the course of this article series.

So what does compression-oriented programming look like, and why is it efficient?

Like a good compressor, I don’t reuse anything until I have at least two instances of it occurring. Many programmers don’t understand how important this is, and try to write “reusable” code right off the bat, but that is probably one of the biggest mistakes you can make. My mantra is, “make your code usable before you try to make it reusable”.

I always begin by just typing out exactly what I want to happen in each specific case, without any regard to “correctness” or “abstraction” or any other buzzword, and I get that working. Then, when I find myself doing the same thing a second time somewhere else, that is when I pull out the reusable portion and share it, effectively “compressing” the code. I like “compress” better as an analogy, because it means something useful, as opposed to the often-used “abstracting”, which doesn’t really imply anything useful. Who cares if code is abstract?

Waiting until there are (at least) two examples of a piece of code means I not only save time thinking about how to reuse it until I know I really need to, but it also means I always have at least two different real examples of what the code has to do before I try to make it reusable. This is crucial for efficiency, because if you only have one example, or worse, no examples (in the case of code written preemptively), then you are very likely to make mistakes in the way you write it and end up with code that isn’t conveniently reusable. This leads to even more wasted time once you go to use it, because either it will be cumbersome, or you will have to redo it to make it work the way you need it to. So I try very hard to never make code “prematurely reusable”, to evoke Knuth.

Similarly, like a magical globally optimizing compressor (which sadly PKZip isn’t), when you are presented with new places where a previously reused piece of code could be reused again, you make a decision: if the reusable code is already suitable, you just use it, but if it’s not, you decide whether or not you should modify how it works, or whether you should introduce a new layer on top of or underneath it. Multiresolution entry points are a big part of making code resuable, but I’ll save discussion of that for a later article, since it’s a topic unto itself.

Finally, the underlying assumption in all of this is, if you compress your code to a nice compact form, it is easy to read, because there’s a minimal amount of it, and the semantics tend to mirror the real “language” of the problem, because like a real language, those things that are expressed most often are given their own names and are used consistently. Well-compressed code is also easy to maintain, because all the places in the code that are doing identical things all go through the same paths, but code that is unique is not needlessly complicated or separated from its use. Finally, well-compressed code is easy to extend, because producing more code that does similar operations is simple, as all the necessary code is there in a nicely recomposable way.

These are all things that most programming methodologies claim to do in an abstract fashion (build UML diagrams, make class hierarchies, make systems of objects, etc.), but always fail to achieve, because the hard part of code is getting the details right. Starting from a place where the details don’t exist inevitably means you will forget or overlook something that will cause your plans to fail or lead to suboptimal results. Starting with the details and repeatedly compressing to arrive at the eventual architecture avoids all the pitfalls of trying to conceive the architecture ahead of time.

With all that in mind, let’s take a look at how all this can be applied to the simple Witness UI code.

The first bit of code compression I did on the UI code happens to be one of my very favorites, since it’s trivial to do and yet is extremely satisfying.

Basically, in C++, functions are very selfish. They keep all their local variables to themselves, and you can’t really do anything about that (although as the cancerous C++ specification continues to metastasize, it’s starting to add more options for this, but that is a separate issue). So when I see code like the Witness UI code that’s doing stuff like this:

int category_height = ypad + 1.2 * body_font -> character_height;

float y0 = y;

// ...

y0 -= category_height;

// ...

y0 -= category_height;

// ...

y0 -= category_height;

// ...

I think it’s time for me to make a shared stack frame.

What I mean by this is, anywhere there’s going to be a panel UI in the Witness, this sort of thing is going to happen. I looked at the other panels in the editor, of which there were several, and they all had substantively the exact same code as I showed in the original snippet  —  same startup, same button calculations, etc. So it’s clear that I want to compress all this so that each thing only happens in one place, then just gets used by everyone else.

But it’s not really feasible to wrap what’s going on purely in a function, because there’s systems of variables that interact, and they interact in multiple places that need to connect with each other. So the first thing I did to this code was to pull those variables out into a structure that can serve as a sort of shared stack frame for all these operations if I want them to be separate functions:

struct Panel_Layout

{

float width; // renamed from "my_width"

float row_height; // rename from "category_height"

float at_x; // renamed from "x0"

float at_y; // renamed from "y0"

};

Simple, right? You just grab the variables that you see that are being used in a repetitive way, and you put them in a struct. Typically, I use InterCaps for variable names and lowercase_ with_ underscores for types, but since I am in the Witness codebase, I try to adhere to its general conventions where possible, and it uses Uppercase_ With_ Underscores for types and lowercase_ with_ underscores for variables.

After I substituted the structure in for the local variables, the code looked like this:

Panel_Layout layout;

int num_categories = 4 ;

layout . row_height = ypad + 1.2 * body_font -> character_height;

layout . at_x = x;

layout . at_y = y;

float title_height = draw_title( layout . at_x, layout . at_y, title);

float height = title_height + num_categories * layout . row_height + ypad;

my_height = height;

layout . at_y -= title_height;

{

layout . at_y -= layout . row_height;

char * string = "Auto Snap" ;

bool pressed = draw_big_text_button( layout . at_x, layout . at_y, layout . width, layout . row_height, string);

if ( pressed) do_auto_snap( this);

}

{

layout . at_y -= category_height;

char * string = "Reset Orientation" ;

bool pressed = draw_big_text_button( layout . at_x, layout . at_y, layout . width, layout . row_height, string);

if ( pressed) {

// ...

}

}

// ...

Not an improvement yet, but it was a necessary first step. Next I pulled the redundant code out into functions: one at startup, and one for each time there’s a new row of UI. Normally, I would probably not make these member functions, but since The Witness is a more C++-ish codebase than my own, I thought it was more consistent with the style (and I don’t have a strong preference either way):

Panel_Layout :: Panel_Layout( Panel * panel, float left_x, float top_y, float width)

{

row_height = panel -> ypad + 1.2 * panel -> body_font -> character_height;

at_y = top_y;

at_x = left_x;

}

void Panel_Layout :: row()

{

at_y -= row_height;

}

Once I had the structure, it was also trivial to take these two lines

float title_height = draw_title( x0, y0, title);

y0 -= title_height;

from the original and wrap them up:

void Panel_Layout :: window_title( char * title)

{

float title_height = draw_title( at_x, at_y, title);

at_y -= title_height;

}

So then the code looked like this:

Panel_Layout layout( this, x, y, my_width);

layout . window_title( title);

int num_categories = 4 ;

float height = title_height + num_categories * layout . row_height + ypad;

my_height = height;

{

layout . row();

char * string = "Auto Snap" ;

bool pressed = draw_big_text_button( layout . at_x, layout . at_y, layout . width, layout . row_height, string);

if ( pressed) do_auto_snap( this);

}

{

layout . row();

char * string = "Reset Orientation" ;

bool pressed = draw_big_text_button( layout . at_x, layout . at_y, layout . width, layout . row_height, string);

if ( pressed) {

// ...

}

}

// ...

Although that wouldn’t be necessary if this was the only panel (since the code only happens once), all the Witness UI panels did the same thing, so pulling it out meant I could go compress all that code too (which I did, but which I won’t be covering here).

Things were looking better, but I also wanted to get rid of the weird “num_ categories” bit and the height calculation. Looking at that code further, I determined that all it was really doing was pre-counting how high the panel would be after all the rows were used. Since there was no actual reason why this had to be set up front, I figured hey, why not do it after all the rows have been made, so I can just count how many actually got added rather than forcing the program to pre-declare that? That makes it less error prone, because the two cannot get out of sync. So I added a “complete” function that gets run at the end of a panel layout:

void Panel_Layout :: complete( Panel * panel)

{

panel -> my_height = top_y - at_y;

}

I went back to the constructor and made sure I saved “top_ y” as the starting y, so all I had to do was just subtract the two. Poof! No more need for the precalculation:

Panel_Layout layout( this, x, y, my_width);

layout . window_title( title);

{

layout . row();

char * string = "Auto Snap" ;

bool pressed = draw_big_text_button( layout . at_x, layout . at_y, layout . my_width, layout . row_height, string);

if ( pressed) do_auto_snap( this);

}

{

layout . row();

char * string = "Reset Orientation" ;

bool pressed = draw_big_text_button( layout . at_x, layout . at_y, layout . my_width, layout . row_height, string);

if ( pressed) {

// ...

}

}

// ...

layout . complete( this);

The code was getting a lot more concise, but it was also clear from the often-repeated draw_ big_ text_ button calls that there was plenty of compressibility left. So I took those out next:

bool Panel_Layout :: push_button( char * text)

{

bool result = panel -> draw_big_text_button( at_x, at_y, width, row_height, text);

return ( result);

}

which left the code looking rather nice and compact:

Panel_Layout layout( this, x, y, my_width);

layout . window_title( title);

{

layout . row();

char * string = "Auto Snap" ;

bool pressed = layout . push_button( string);

if ( pressed) do_auto_snap( this);

}

{

layout . row();

char * string = "Reset Orientation" ;

bool pressed = layout . push_button( string);

if ( pressed) {

// ...

}

}

// ...

layout . complete( this);

and I decided to pretty it up a bit by reducing some of the unnecessary verbosity:

Panel_Layout layout( this, x, y, my_width);

layout . window_title( title);

layout . row();

if ( layout . push_button( "Auto Snap" )) { do_auto_snap( this);}

layout . row();

if ( layout . push_button( "Reset Orientation" ))

{

// ...

}

// ...

layout . complete( this);

Ah! It’s like a breath of fresh air compared to the original, isn’t it? Look at how nice that looks! It’s getting close to the minimum amount of information necessary to actually define the unique UI of the movement panel, which is how we know we’re doing a good job of compressing. And adding new buttons is getting very simple  —  no more in-line math, just one call to make a row and another to make a button.

Now, I want to point out something really important. Did all that seem pretty straightforward? I’m guessing that there wasn’t anything in there where you were like, “oh my god, how did he DO that??” I’m hoping that every step was really obvious, and everyone could have easily done a similar set of steps if charged with just pulling out the common pieces of code into functions.

So, given that, what I want to point out is this: this is the correct way to give birth to “objects”. We made a real, usable bundle of code and data: the Panel_ Layout structure and its member functions. It does exactly what we want, it fits perfectly, it’s really easy to use, it was trivial to design.

Contrast this with the absolute absurdity that you see in object-oriented “methodologies” that tell you to start writing things on index cards (like the “class responsibility collaborators” methodology), or breaking out Visio to show how things “interact” using boxes and lines that connect them. You can spend hours with these methodologies and end up more confused about the problem than when you started. But if you just forget all that, and write simple code, you can always create your objects after the fact and you will find that they are exactly what you wanted.

If you’re not used to programming like this, you may think I’m exaggerating, but you’ll just have to trust me, it’s true. I spend exactly zero time thinking about “objects” or what goes where. The fallacy of “object-oriented programming” is exactly that: that code is at all “object-oriented”. It isn’t. Code is procedurally oriented, and the “objects” are simply constructs that arise that allow procedures to be reused. So if you just let that happen instead of trying to force everything to work backwards, programming becomes immensely more pleasant.

More Compression, Then Expansion

Because I needed to spend some time introducing the concept of compression-oriented programming, and also because I enjoy trashing object-oriented programming, this article is already very long despite only showing a small fraction of the code transformations I did to the Witness UI code. So I will save the next round for next week, where I’ll talk about handling that multi-button code I showed, and then how I started using the newly compressed UI semantics to start extending what the UI itself could do.

What the heck is going on at Apple?

Hacker News
www.cnn.com
2025-12-07 16:54:44
Comments...
Original Article

Apple for decades has been known for a consistent string of design-forward, tech-defining consumer products that have shaped how people use technology.

Now the company known for its steadiness is going through a shakeup at the top, as both Apple and the tech industry at large are at a crossroads.

Apple announced the departures of three executive team members in less than a week. Meta poached a key Apple design leader. And speculation is mounting that Tim Cook may be preparing to step aside as CEO.

The changes come as critics say Apple, once a tech leader, is behind in the next big wave: artificial intelligence. For one of the world’s most valuable tech companies, a change in leadership could mean a change in how it conceives, designs and creates products used around the world every single day.

“The only thing we can read into this is that we’re headed to a time of increased volatility for Apple,” said Robert Siegel, a longtime venture capitalist and lecturer at Stanford’s Graduate School of Business.

Apple stock ( AAPL ) is up roughly 12% this year, a much smaller jump than the 30% increase it saw in 2024.

Apple did not immediately respond to a request for comment.

Who’s leaving and why

Planned departures for the following Apple executives were announced just this week:

  • Lisa Jackson, Apple’s vice president of environment, policy and social initiatives, is set to retire next year.
  • General counsel Kate Adams, also set to retire next year.
  • Alan Dye, vice president of human interface design, who is joining Meta as its chief design officer.
  • John Giannandrea, senior vice president of machine learning and AI strategy, who will also retire next year.

Apple is bringing in Meta chief legal officer Jennifer Newstead to lead government affairs after Adams retires and serve as its new general counsel. The environment and social initiatives teams will now report to Sabih Khan, Apple’s chief operating officer. Amar Subramanya, Microsoft’s corporate vice president of AI, will be Apple’s new vice president of AI.

And earlier this year, Jeff Williams stepped back from his role as Apple’s chief operating officer.

Apple Park, Apple's circular HQ office building, is seen in an aerial view over Cupertino, California on May 16, 2024.

Apple isn’t the only tech giant making structural changes. Meta on Thursday said it’s shifting some investment away from its Metaverse virtual reality project and towards AI glasses and wearables. Amazon laid off 14,000 people in October as part of a push to move faster in AI by operating more leanly. And Google last year combined its hardware and software teams to better integrate AI into its products across the board.

But Apple is known for having a uniquely tight-knit company culture driven by secrecy.

“This is against the typical culture of Apple. But they need to rip the Band-Aid off,” said Dan Ives, global head of tech research for Wedbush Securities. “Because the AI strategy has been invisible, and it’s going to define Cook’s legacy, how he handles this chapter.”

Apple’s future and challenges

The leadership shakeup comes as questions about Apple’s future loom.

Apple delayed a major update to its Siri voice assistant that was expected to bring it closer to OpenAI’s ChatGPT and Google’s Gemini, turning Siri from a question-and-answer machine into an assistant that can act on a user’s behalf and incorporate information from a person’s phone to personalize responses.

But that upgrade has been pushed off until next year, and Apple’s other AI updates for iPhones, Macs and iPads have been minimal this year.

And Apple’s expensive Vision Pro headset, the first new computing category the company has introduced since the decade-old Apple Watch, is still a niche product.

At the same time, Meta, Google, Samsung and OpenAI have announced significant product expansions in AI this year – from Meta’s new Ray-Ban Display smart glasses to Google and Samsung’s Gemini-powered headset and OpenAI’s push into shopping and web browsers . Google’s Gemini 3 model has also been making waves since its November launch.

Customers try the Apple Vision Pro mixed reality glasses device by US company Apple Inc. during the launch at the Apple store on Champs Elysees avenue in Paris on July 12, 2024

Wall Street wants answers about Apple’s AI strategy. In a July earnings call, analysts asked Apple about Siri’s role in driving new products and whether AI chatbots are threatening Apple’s relevance in internet searches. Eddy Cue, Apple’s senior vice president of services, even said during his testimony in a Google antitrust hearing that people may not need an iPhone 10 years from now .

Now Dye, largely the face of Apple’s design studio following the 2019 departure of former design chief Jony Ive, is joining Meta to help shape what the company sees as the next wave of computing. And Ive is helping OpenAI create its first hardware product.

Dye’s decision to join Meta is “more of a direct threat to Apple” compared to the other announced departures, said Joe Tigay, portfolio manager of the Rational Equity Armor Fund.

Despite facing pressure in AI, iPhone 17 sales have been strong and are only expected to climb higher next year. Apple is expected to surpass Samsung in smartphone shipments this year for the first time since 2011, according to Counterpoint Research. The company is also one of the few to cross the $4 trillion market capitalization threshold, along with AI giants Nvidia and Microsoft.

And change isn’t always a bad thing, according to Siegel, especially while industries are going through transitions as the tech sector currently is with AI. Bringing in new hires or promoting people from within can “give a different point of view when a company can get trapped in a way of thinking and doing things,” he said.

That could be just what Apple needs, as some analysts say the clock is ticking for Apple to make bigger leaps in AI.

“You can’t have a fourth industrial revolution and watch the AI party through the windows on the outside,” said Ives. “And clearly they need massive changes in leadership.”

The AI Wildfire Is Coming. It's Going to Be Painful and Healthy

Hacker News
ceodinner.substack.com
2025-12-07 16:43:38
Comments...
Original Article

At a recent CEO dinner in Menlo Park, someone asked the familiar question: Are we in an AI bubble?

One of the dinner guests, a veteran of multiple Silicon Valley cycles, reframed the conversation entirely. She argued for thinking of this moment as a wildfire rather than a bubble. The metaphor landed immediately. Wildfires don’t just destroy; they’re essential to ecosystem health. They clear the dense underbrush that chokes out new growth, return nutrients to the soil, and create the conditions for the next generation of forest to thrive.

As I reflected on the wildfire metaphor, a framework emerged that revealed something deeper, built on her reframing. It offered a taxonomy for understanding who survives, who burns, and why, with specific metrics that separate the fire-resistant from the flammable.

The first web cycle burned through dot-com exuberance and left behind Google, Amazon, eBay, and PayPal: the hardy survivors of Web 1.0. The next cycle, driven by social and mobile, burned again in 2008–2009, clearing the underbrush for Facebook, Airbnb, Uber, and the offspring of Y Combinator. Both fires followed the same pattern: excessive growth, sudden correction, then renaissance.

Now, with AI, we are once again surrounded by dry brush.

The coming correction will manifest as a wildfire rather than a bubble burst. Understanding that distinction changes everything about how to survive and thrive in what comes next.

When the brush grows too dense, sunlight can’t reach the ground. The plants compete against each other for light, water, and nutrients rather than against the environment.

That’s what Silicon Valley feels like right now.

Capital is abundant, perhaps too abundant. But talent? That’s the scarce resource. Every promising engineer, designer, or operator is being courted by three, five, ten different AI startups, often chasing the same vertical, whether it’s coding copilots, novel datasets, customer service, legal tech, or marketing automation.

The result is an ecosystem that looks lush from above: green, growing, noisy. But underneath, the soil is dry. Growth becomes difficult when everyone’s roots are tangled.

In that kind of forest, fire serves as correction rather than catastrophe.

Wildfires don’t just destroy ecosystems. They reshape them. Some species ignite instantly. Others resist the flames. A few depend on the fire to reproduce.

The same is true for startups.

These are the dry grasses and resinous pines of the ecosystem: startups that look vibrant in a season of easy money but have no resistance once the air gets hot.

They include:

  • AI application wrappers with no proprietary data or distribution

  • Infrastructure clones in crowded categories (one more LLM gateway, one more vector database)

  • Consumer apps chasing daily active users instead of durable users

They’re fueled by hype and ebullient valuations. When the heat rises, when capital tightens or customers scrutinize ROI, they go up in seconds.

The flammable brush serves a purpose. It attracts capital and talent into the sector. It creates market urgency. And when it burns, it releases those resources back into the soil for hardier species to absorb. The engineers from failed AI wrappers become the senior hires at the companies that survive.

Then there are the succulents, oaks, and redwoods: the incumbents that store moisture and protect their cores.

Thick bark: Strong balance sheets and enduring customer relationships.

Deep roots: Structural product-market fit in cloud, chips, or data infrastructure.

Moisture reserves: Real revenue, diversified businesses, and long-term moats.

Think Apple, Microsoft, Nvidia, Google, Amazon. They will absorb the heat and emerge stronger. When the smoke clears, these giants will stand taller, their bark charred but intact, while the smaller trees around them have burned to ash.

Some plants die back but grow again; manzanita, scrub oak, and toyon are phoenix-like. In startup terms, these are the pivots and re-foundings that follow a burn.

They’re teams with:

  • Deep expertise

  • Underground IP and data assets that survive even if the product doesn’t

  • A willingness to prune and start over

After the fire, they re-sprout — leaner, smarter, and better adapted to the new terrain.

This is where the real learning happens. A founder who built the wrong product with the right team in 2024 becomes the founder who builds the right product with a battle-tested team in 2027. The failure gets stored underground, like nutrients in roots, waiting for the next season, rather than being wasted.

Finally come the wildflowers. Their seeds are triggered by heat. They can’t even germinate until the old growth is gone.

These are the founders who start after the crash. They’ll hire from the ashes, build on cheaper infrastructure, and learn from the mistakes of those who burned. LinkedIn in 2002, Stripe in 2010, Slack in 2013. All are fire followers.

The next great AI-native companies will likely emerge here. These are the ones that truly integrate intelligence into workflows rather than just decorating them. And critically, the inference layer (where AI models actually run in production) represents the next major battleground. As compute becomes commoditized and agentic tools proliferate, the race will shift from training the biggest models to delivering intelligence most efficiently at scale.

Every few decades, Silicon Valley becomes overgrown. Web 1.0 and Web 2.0 both proved the same truth: too much growth chokes itself.

The Web 1.0 crash cleared away more than startups. It cleared noise. The Web 2.0 downturn, driven more by the mortgage crisis than the market itself, followed the same dynamic: overfunded competitors fell away, talent dispersed, and the survivors hired better, moved faster, and built stronger. Savvy companies even used the moment to get leaner, cutting underperformers and upgrading positions from entry-level to executive with hungry refugees from failed competitors.

That redistribution of talent may be the single most powerful outcome of any crash. Many of Google’s best early employees (the architects of what became one of the most durable business models in history) were founders or early employees of failed Web 1.0 startups.

And it went beyond talent alone. Entrepreneurial, restless, culturally impatient talent specifically shaped Google’s internal ethos. That DNA created Google’s experimental, aggressive, always-in-beta culture and radiated outward into the broader ecosystem for the next 10 to 20 years. The fire reallocated intelligence and rewired culture rather than simply destroying.

The 2000 wildfire was a full incineration. Infrastructure overbuild, easy capital, and speculative exuberance burned away nearly all profitless growth stories. Yet what remained were root systems: data centers, fiber optics, and the surviving companies that learned to grow slow and deep.

Amazon looked dead, down 95%, but emerged as the spine of digital commerce. eBay stabilized early and became the first profitable platform marketplace. Microsoft and Oracle converted their software monopolies into durable enterprise cashflows. Cisco, scorched by overcapacity, rebuilt slowly as networking became the plumbing for doing business.

By adding Apple, Google, and Salesforce, the story becomes one of succession as well as survival. Apple didn’t merely survive the fire; it changed the climate for everything that followed. Google sprouted where others burned, fueled by the very engineers and founders whose startups perished in the blaze. Salesforce took advantage of scorched corporate budgets to sell cloud-based flexibility, defining the SaaS model.

During the late 1990s, telecom firms raised roughly $2 trillion in equity and another $600 billion in debt to fuel the “new economy.” Even the stocks that symbolized the mania followed a predictable arc. Intel, Cisco, Microsoft, and Oracle together were worth around $83 billion in 1995; by 2000, their combined market cap had swelled to nearly $2 trillion. Qualcomm rose 2,700% in a single year.

That money paid for over 80 million miles of fiber-optic cable, more than three-quarters of all the digital wiring that had ever been installed in the U.S. up to that point. Then came the collapse.

By 2005, nearly 85% of those cables sat unused, strands of dark fiber buried in the ground. This was overcapacity born of overconfidence. But the fiber stayed. The servers stayed. The people stayed. And that excess soon became the backbone of modern life. Within just four years of the crash, the cost of bandwidth had fallen by 90%, and the glut of cheap connectivity powered everything that came next: YouTube, Facebook, smartphones, streaming, the cloud.

That’s the paradox of productive bubbles: they destroy value on paper but create infrastructure in reality. When the flames pass, the pipes, the code, and the talent remain — ready for the next generation to use at a fraction of the cost.

The Great Recession sparked a different kind of wildfire. Where Web 1.0’s flames had consumed speculative infrastructure, Web 2.0’s burned through business models and illusions. Venture funding froze. Advertising budgets evaporated. Credit tightened. Yet the survivors didn’t just withstand the heat. They metabolized it.

Apple turned adversity into dominance, transforming the iPhone from curiosity into cultural infrastructure. Amazon, having survived the dot-com inferno, emerged as the quiet supplier of the internet’s oxygen: AWS. Netflix reinvented itself for the streaming era, its growth literally running over the fiber laid down by the previous bubble. Salesforce proved that cloud software could thrive when capital budgets died. Google discovered that measurable performance advertising could expand even in recession. And Facebook (a seedling then) would soon root itself in the ashes, nourished by cheap smartphones and surplus bandwidth.

The 2008 fire selected for companies that could integrate hardware, software, and services into self-sustaining ecosystems rather than simply clearing space. The result was evolution, not merely recovery.

This cycle, though, introduces a new kind of fuel — the canopy fire.

In the past, the flames mostly consumed the underbrush (small, overvalued startups). Today, the heat is concentrated in the tallest trees themselves: Nvidia, OpenAI, Microsoft, and a handful of hyperscalers spending staggering sums with each other.

Compute has become both the oxygen and the accelerant of this market. Every dollar of AI demand turns into a dollar for Nvidia, which in turn fuels more investment into model training, which requires still more GPUs. This creates a feedback loop of mutual monetization.

This dynamic has created something closer to an industrial bubble than a speculative one. The capital isn’t scattered across a thousand dot-coms; it’s concentrated in a few massive bilateral relationships, with complex cross-investments that blur the line between genuine deployment and recycled capital.

When the wildfire comes (when AI demand normalizes or capital costs rise) the risk shifts. Instead of dozens of failed startups, we face a temporary collapse in compute utilization. Nvidia’s stock may not burn to ash, but even a modest contraction in GPU orders could expose how dependent the entire ecosystem has become on a few large buyers.

That’s the real canopy problem: when the tallest trees grow too close, their crowns interlock, and when one ignites, the fire spreads horizontally, not just from the ground up.

In Web 1.0, Oracle (the de facto database for all dot-coms) saw a symbolic collapse from $46 to $7 in 2000 before recovering to $79 by the launch of ChatGPT and $277 today. In Web 2.0’s wildfire, Google (the supplier of performance advertising) dropped 64% from $17 to $6 but exploded to $99 with ChatGPT’s launch and has since hit $257. In this cycle, the analog could be Nvidia. Not because it lacks fundamentals, but because its customers are all drawing from the same pool of speculative heat, fueled by complex cross-investments that have elicited scrutiny about whether capital is being genuinely deployed or simply recycled.

Here’s where the AI wildfire may prove even more productive than its predecessors: the infrastructure being overbuilt today goes beyond fiber optic cable lying dormant in the ground. We’re building compute capacity, the fundamental resource constraining AI innovation right now.

Today’s AI market is brutally supply-constrained. Startups can’t get the GPU allocations they need. Hyperscalers are rationing compute to their best customers. Research labs are queuing for months to train models. Ideas and talent aren’t the bottleneck. Access to the machinery is.

This scarcity is driving the current frenzy. Companies are signing multi-billion dollar commitments years in advance, locking in capacity at premium prices, building private data centers, and stockpiling chips like ammunition. The fear centers on being unable to participate at all because you can’t access the compute, not just missing the AI wave.

What happens, however, after the fire?

The same pattern that played out with bandwidth in 2000 is setting up to repeat with compute in 2026. Billions of dollars are pouring into GPU clusters, data centers, and power infrastructure. Much of this capacity is being built speculatively, funded by the assumption that AI demand will grow exponentially forever.

But there’s another dynamic accelerating the buildout: a high-stakes game of chicken where no one can afford to blink first. When Microsoft announces a $100 billion data center investment, Google must respond in kind. When OpenAI commits to 10 gigawatts of Nvidia chips, competitors feel compelled to match or exceed that commitment. The fear centers on being locked out of the market entirely if demand does materialize and you haven’t secured capacity, not just that AI demand might not materialize.

This creates a dangerous feedback loop. Each massive spending announcement forces competitors to spend more, which drives up the perceived stakes, which justifies even larger commitments. No executive wants to be the one who underinvested in the defining technology of the era. The cost of being wrong by spending too little feels existential; the cost of being wrong by spending too much feels like someone else’s problem — a future quarter’s write-down, not today’s strategic failure.

It’s precisely this dynamic that creates productive bubbles. The rational individual decision (match your competitor’s investment) produces an irrational collective outcome (vast overcapacity). But that overcapacity is what seeds the next forest.

Yet there’s a critical distinction being lost in the bubble debate: not all compute is the same. The market is actually two distinct pools with fundamentally different dynamics.

The first pool is training compute made up of massive clusters used to create new AI models. This is where the game of chicken is being played most aggressively. No lab has a principled way of deciding how much to spend; each is simply responding to intelligence about competitors’ commitments. If your rival is spending twice as much, they might pull the future forward by a year. The result is an arms race governed less by market demand than by competitive fear, with Nvidia sitting in the middle as the gleeful arms dealer.

The second pool is inference compute which runs AI models in production, serving actual users. Here, the dynamics look entirely different.

Society’s demonstrated demand for intelligence is essentially unlimited. Every additional IQ point that can be applied to analyzing data, automating decisions, or improving productivity gets consumed immediately. Supply constrains adoption, not demand. Businesses aren’t asking “do we want AI capabilities?” They’re asking “how much can we get, and how soon?”

As GPUs become commoditized and compute abundance arrives, inference capabilities will become the next major market—especially given growing demand for efficient agentic tools. LLM inference is becoming a massive race. The companies that can deliver intelligence most efficiently, at the lowest cost per token or per decision, will capture disproportionate value. Training the biggest model matters less now; running models efficiently at planetary scale matters more.

This differs fundamentally from the dot-com bubble, which was fueled primarily by advertising spend. Companies burned cash on Super Bowl commercials to acquire customers they hoped to monetize later. That was speculative demand chasing speculative value.

AI inference demand is directed at improving actual earnings. Companies are deploying intelligence to reduce customer acquisition costs, lower operational expenses, and increase worker productivity. The return is measurable and often immediate, not hypothetical.

This suggests the AI “bubble” may have a softer landing than its predecessors. Yes, price-to-earnings ratios look inflated today. But unlike pure speculation, genuine productive capacity is being built. If compute costs fall dramatically post-correction while inference demand remains robust (and all evidence suggests it will) companies can simply run their models longer, use more compute-intensive approaches, or deploy intelligence to problems that are economically marginal at today’s prices but viable at tomorrow’s.

In other words: even if we massively overbuild training capacity (which seems likely), the inference side has enough latent demand to absorb the excess. The compute gets repurposed from the game of chicken to the productive application of intelligence at scale, rather than sitting dark.

Just as bandwidth costs collapsed by 90% within four years of the dot-com crash, making YouTube and Netflix possible, compute costs could fall dramatically in the aftermath of an AI correction. The same GPU clusters that hyperscalers are rationing today could become commodity infrastructure available to anyone with a credit card.

But here the analogy breaks down in a critical way.

Fiber optic cable has an extraordinarily long useful life: decades of productive capacity once it’s in the ground. The infrastructure built during the dot-com bubble is still carrying packets today, twenty-five years later. That’s what made it such a durable gift to the next generation: the cost was borne once, the value compounded for decades.

GPU clusters are not fiber optic cable.

The useful life of a training cluster is perhaps two to three years before it becomes uncompetitive. Chips depreciate faster than they physically wear out. A three-year-old GPU isn’t broken. It’s just obsolete, overtaken by newer architectures that offer better performance per watt, better memory bandwidth, better interconnects. In economic terms, training compute looks more like an operating expense with a short payback window than a durable capital asset.

This fundamentally changes the post-fire dynamics.

When the bubble bursts and training compute becomes abundant, yes, costs will fall. But fire followers won’t inherit state-of-the-art infrastructure the way Web 2.0 companies inherited fiber. They’ll inherit yesterday’s infrastructure: still functional, but no longer cutting-edge. If you want access to the newest, fastest compute to train competitive models, you’ll still need to pay premium prices to whoever is actively refreshing their clusters.

This creates a different kind of moat than we saw in previous cycles. The companies that survive the fire will benefit from having already paid down the cost of the current generation while competitors are trying to catch up on older hardware, not just from cheaper infrastructure. The incumbency advantage centers on having the right generation of compute, continuously refreshed, not just having compute in general.

Inference compute follows different economics. Once a model is trained, it can run productively on older hardware for years. But the training side may not produce the same democratization we saw with bandwidth. The fire might clear the brush, but the tallest trees will still control access to sunlight.

Yet focusing solely on compute may mean we’re watching the wrong wildfire.

Some believe the true winner of the AI race (at a national and global level) will be whoever solves the energy problem , not the company with the most GPUs or the best models.

Compute, after all, is just concentrated electricity. A modern AI data center can consume as much power as a small city. Kilowatts are the constraint, not silicon. You can manufacture more chips, but you can’t manufacture more energy without fundamental infrastructure: power plants, transmission lines, grid capacity. These take years or decades to build.

This is where the wildfire metaphor becomes particularly instructive. We’re focused on the compute forest burning and regrowing. But beneath that visible drama, there’s a deeper question: are we building enough energy infrastructure to power the next forest at all?

The dot-com bubble left behind dark fiber that could be lit up instantly when demand returned. But idle data centers without power to run them are just expensive real estate. The real infrastructure deficit may center on energy generation rather than compute capacity.

If this bubble drives massive investment in power infrastructure (nuclear plants, renewable energy farms, grid modernization, advanced battery storage) that would be a genuinely durable gift to the next half-century. Energy infrastructure, unlike GPUs that become obsolete in five years, compounds in value over decades.

The companies that will dominate the post-fire landscape may be the ones securing energy capacity tomorrow (when every other form of AI infrastructure is abundant except the electricity to run it) not the ones hoarding compute today.

Consider the math: A single large AI training cluster can require 100+ megawatts of continuous power, equivalent to a small city. The United States currently generates about 1,200 gigawatts of electricity total. If AI compute grows at projected rates, it could demand 5-10% of the nation’s entire power generation within a decade.

The problem here is about fundamental energy infrastructure.

And unlike fiber optic cable or GPU clusters, power infrastructure can’t be deployed quickly. Nuclear plants take 10-15 years to build. Major transmission lines face decades of regulatory approval. Even large solar farms require 3-5 years from planning to operation.

This means the real constraint on AI (the genuine bottleneck that will determine winners and losers) may already be locked in by decisions being made (or not made) right now about power infrastructure.

The companies currently spending hundreds of billions on GPUs may discover their limiting factor is the megawatts needed to run it, not compute capacity. And the regions that invest heavily in energy infrastructure today will have an insurmountable advantage in hosting AI workloads tomorrow.

The companies prepping themselves to survive scarcity aren’t just stockpiling compute. They’re building root systems deep enough to tap multiple resources: energy contracts locked in for decades, gross retention rates above 120%, margin expansion even as they scale, and infrastructure that can flex between training and inference as market dynamics shift.

With our global glasses on, we are losing (and some may say have already lost) the energy battle with China. Very quietly we have entered a new cold war era where watts and rare-earth materials are the new ICBMs.

A burning question (pardon the pun) is how do we assess fire-resistance in this cycle? Each category of company faces different tests of durability. Understanding these metrics separates genuine ecosystem strength from temporary abundance:

Understanding the Fire Tests:

Foundation model labs face a fundamental question: Can revenue grow faster than compute costs? Training expenses scale exponentially (10x compute ≈ 3x performance), while revenue scales with customer adoption. If a lab spends $100M on compute to generate $50M in revenue, then $300M to generate $120M, the trajectory is fatal. They’re running faster to stand still. Fire-resistant labs show revenue outpacing compute spend, proof that each capability improvement unlocks disproportionate customer value.

Enterprise AI platforms must prove their AI goes beyond marketing veneer. A company showing 95% gross retention but only 12% AI feature adoption means customers stay for the legacy platform (data warehouse, CRM) while ignoring AI add-ons. When capital contracts, these companies get repriced violently. The market realizes they’re infrastructure plays with an AI sticker. True AI platforms show high retention because of high AI adoption, not despite low adoption.

Application layer companies live in a unique trap: building on models they don’t control (OpenAI, Anthropic) creates margin compression, feature parity, and disintermediation risk. The only escape is deep customer embedding. Companies with NRR >120% and CAC payback <12 months have achieved workflow integration—customers expand usage naturally and acquisition costs pay back fast. Those with NRR <100% and payback >18 months are “nice-to-have” features that churn when budgets tighten, requiring continuous capital infusion to grow.

Inference API players face commoditization as GPU oversupply arrives. Revenue per GPU-hour reveals pricing power. A company generating $50/GPU-hour versus $5/GPU-hour has 10x more margin to defend its position through technical optimization, product differentiation, or distribution moats. Inference cost elasticity shows market structure: high elasticity (50% price cut = 500% demand increase) means commodity hell; low elasticity means customers value features beyond raw compute.

Energy and infrastructure firms ultimately control AI’s fundamental constraint. Data center economics flip based on utilization and energy costs. At $0.03/kWh and 85% utilization, effective cost is $0.035/kWh. At $0.08/kWh and 50% utilization, it’s $0.16/kWh—a 4.5x disadvantage. When AI demand crashes post-bubble, facilities with high energy costs cannot lower prices enough to fill capacity. Those with structural energy advantages (hydroelectric, nuclear contracts) can slash prices and still maintain positive margins, filling capacity by absorbing distressed competitors’ customers.

The meta-pattern: Each metric asks the same question from different angles—can you sustain your business model when external capital disappears? Fire-resistant companies have achieved thermodynamic sustainability: each unit of input (capital, compute, energy) generates more than one unit of output (revenue, value, efficiency). They can grow in scarcity. The flammable brush consumes more than it produces, subsidized by abundant capital. When the subsidy ends, they ignite.

This comparative framing reveals who has genuine ecosystem durability versus who’s simply tall because of temporary abundance.

The giant sequoia cannot reproduce without fire. Its cones open only in intense heat. The flames clear the forest floor, allowing seeds to reach mineral soil. The canopy burns back, allowing sunlight through. Without the burn, there is no renewal.

There’s a deeper truth in the sequoia’s relationship with fire: not all fires serve the tree equally.

For millennia, sequoias thrived with low-intensity ground fires that burned every 10-20 years. These fires were hot enough to open cones and clear undergrowth, but cool enough to leave mature trees unharmed. The sequoia’s thick bark (up to two feet deep) evolved specifically to survive these regular burns.

Then came a century of fire suppression. Without regular burning, fuel built up. Understory trees grew tall. When fires finally came, they burned hotter and higher than sequoias had ever faced.

The Castle Fire of 2020 killed an estimated 10-14% of all mature giant sequoias on Earth. Trees that had survived dozens of fires over 2,000 years died in a single afternoon. The difference? Fire intensity. The accumulated fuel created canopy fires that overwhelmed even the sequoia’s legendary resilience.

Here’s the lesson for Silicon Valley : Regular burns (cyclical corrections, normal bankruptcies, the constant churn of creative destruction) are healthy. They clear brush, release resources, and allow new growth. But if we suppress all burning for too long, if we bail out every overvalued company and prop up every failing business model, we don’t prevent the fire. We just make the eventual burn catastrophic.

The sequoia also teaches us about time horizons. These trees take centuries to reach their full height. Even mature sequoias that survive a fire need decades to fully recover their canopy. It’s still hard to tell which trees (even those that seem mature today) will continue growing versus which have already peaked. The true giants are those that spent generations building root systems deep enough to tap water sources others can’t reach, developing bark thick enough to withstand heat others can’t survive.

The goal isn’t to prevent fires but to maintain their rhythm. Small, regular burns prevent devastating conflagrations. The worst outcome is the policy that postpones all fires until the fuel load becomes explosive, not the fire itself.

If this is a bubble, it’s a productive one — a controlled burn rather than a collapse.

But “controlled” doesn’t mean comfortable. The flammable brush will ignite. Capital will evaporate. Valuations will crash. Jobs will disappear. Instead of a failure, that’s the system working as designed.

The test for every founder and investor centers on whether you can withstand scarcity rather than whether you can grow in abundance.

When the smoke clears, we’ll see who was succulent and who was tinder, who had bark, and who was resin.

The wildfire is coming. That’s not the problem.

The question is: What kind of plant are you?

And perhaps more importantly: Are you building root systems deep enough, not just to survive this season, but to keep growing through the next decade of scarcity?

Because the real opportunity comes post-fire: what continues to grow after and what entirely new species take root in the ashes.

I don’t think of a wildfire as Mother Nature’s wise way of maintaining balance. In fact, not all ecosystems depend on wildfires. Many ecosystems have evolved with fire as a natural and sometimes essential ecological process, while others are harmed by wildfire and have no natural fire-adapted features. This analogy is meant to help you understand that wildfires are a natural and necessary part of the Silicon Valley ecosystem.

Where the moral judgment does come in pertains to where all the nutrients, the talent, the attention, and the glory fall after the burn. This litmus test of humanity is the defining question of this cycle.

Will the resources gravitate to companies trying to grab more “share of attention,” getting you to watch mindlessly entertaining content by pushing your dopamine and epinephrine buttons. Will the highest goal of this technology ultimately be to get you to buy things you don’t need and spend time on activities that only mollify your FOMO temporarily. Will AI simply accelerate the development of a capitalist hypercycle of Paul Tillich’s ultimate-concern pursuit and existential disappointment that only expands the gulf between haves and have-nots?

Robert Putnam’s research out of Harvard shows that democratizing technology doesn’t inherently level the playing field. “Compared to their poorer counterparts, young people from upper-class backgrounds (and their parents) are more likely to use the Internet for jobs, education, political and social engagement, health, and newsgathering, and less for entertainment and recreation,” Putnam writes. “Affluent Americans use the Internet in ways that are mobility-enhancing, whereas poorer, less-educated Americans typically use it in ways that are not.” This stark dichotomy underscores the importance of purposefully guiding AI to free, not fetter, human agency.

More optimistically, I hope the handcuffs of Packard’s Law will be loosened for current startups and future wildflowers pursuing worthwhile quests. In Good to Great , Jim Collins coined the term Packard’s Law to describe David Packard’s view that organizational growth is limited by a company’s ability to obtain enough of the right people. After the burn, I hope companies like the following will thrive with easier access to talent, the oxygen and sunlight of company growth:

Montai Therapeutics is using AI to pioneer the creation of medicines to treat and preempt chronic disease. They have a poly-intelligent approach to discovery in which humans, AI, and nature collaborate to generate novel molecules for heretofore unsolvable diseases.

Eudia is creating an augmented-intelligence platform, starting with the legal industry, to allow humans to be orders of magnitude more efficient—not replacing lawyers but augmenting them. Augmented law delivers both precision and speed, and for the first time, cost and quality are not trade-offs. Eudia is delivering outcome-based pricing for legal work instead of billable hours. Which consumer of legal services wouldn’t be in favor of that idea?

Listen Labs is an AI-powered research platform that helps teams uncover insights from customer interviews in hours—not months—thereby amplifying the voice of customers. Where, practically speaking, companies could previously speak only with a sample of customers, now they can listen to a full panel representing every demographic, geographic, and psychographic profile instantaneously. The ironic part is that humans are more likely to offer candid and useful feedback when speaking to an AI than to a human who they consciously or unconsciously feel may be judging their answers.

Netic is helping essential service industries grow on autopilot. While the AI wave has swept across software and creative industries, businesses like home services, automotive, and consumer healthcare have been left behind. These industries form the backbone of the economy, yet they operate on outdated tools, overwhelmed call centers, and disconnected systems. Their operations are complex and often rely on manual workflows, and they can’t access the frontier technologies that drive digital-first businesses. In a world where startups mostly build for startups, Netic serves the real industries that keep America running.

It’s obvious AI will raise the ceiling for the haves; if it does not raise the floor for the have-nots, there will—and perhaps should—be pitchforks.

I have tried my best to raise my children with an abundance mindset when it comes to opportunities and a scarcity mindset as it relates to natural resources. Society feels like it operates in the opposite direction. I wonder if we can escape our fate.

The coming wildfire will surely be good for the Silicon Valley ecosystem, but will it be good for humanity?

Discussion about this post

Ready for more?

The Gerrit code review iceberg

Lobsters
www.haiku-os.org
2025-12-07 16:12:54
Comments...
Original Article

Blog post by PulkoMandy on Mon, 2025-11-24 13:45

Recently some discussions on the forum led to asking about the status of our Gerrit code review. There are a lot of changes there that have been inactive for several years, with no apparent interest from anyone. To be precise, there are currently 358 commits waiting for review (note that Gerrit, unlike Github and other popular code review tools, works on a commit-by-commit basis, so each commit from a multiple-commit change is counted separately). The oldest one has not seen any comments since 2018.

Today, let’s have a look at some of these changes and see why they are stalled. Hopefully it will inspire someone to pick up the work and help finishing them up.

Change 122 - BTab: decouple display label from view name.

What is this about

In Haiku’s Interface Kit, usually, views have a name that is not visible to the user. This is mainly used when exploring an user interface with the scripting facilities (for example with the hey command).

When we added localization to Haiku, we made the choice to not translate these view names. This makes it possible to write hey scripts that will work across all machines, no matter which language the user has selected in the Locale preferences.

There’s one case where the view name gets exposed to the user, however: that is how the label of a BTab in a BTabView is selected. This change attempts to rectify this, by adding a way to set the tab label separately from the name of the view being used as the tab content.

Why is the change stalled

The original behavior (using the view name to set the tab label) is what’s documented in the Be Book. Moreover, BeOS allows to use BTab::SetLabel to set a label, and that will also rename the view. Changing this would risk breaking some BeOS applications.

The idea implemented in this change is to work differently only if the BTab is part of a window using the layout kit. This is an indirect way to detect that we’re not in a BeOS application, since the layout kit didn’t exist in BeOS. It would be better to test for that in a more direct way.

The code also implemented separate storage for the extra fields needed to keep track of the tab name. This seems unnecessary, since there is reserved space in the BTab class to store the data inline. It would avoid extra allocations and code complexity (at the cost of a little extra work in the class header to preserve ABI compatibility on both 32 and 64 bit systems).

The original author lost interest as the changes ended up being a lot more complicated than what initially looked like a simple two-line fix. However, the issues have been identified, and applying the requested changes should be easy enough.

Change 859 - build/OptionalPackage: Use llvm/clang packages when using clang/llvm

What is this about

A very simple change to not include GCC in the Haiku image when compiling it with llvm/clang, and instead include llvm/clang.

Why is this change stalled

No one reviewed this. The use of clang/llvm to build Haiku is still not fully supported, and this change is not required for it. It may also make sense to include both compilers.

As long as the llvm build of Haiku doesn’t fully boot, this change can’t be tested.

Change 1353 - GSoC 2019: Implementing PCID in x86_64

What is this about

PCID is a way to tell the CPU about the currently executing process. It allows to improve performance and security, by tagging the entries in the TLB (the structure used by the CPU to manage virtual memory) with the identifier of the running process. The CPU makes sure processes can’t access another process entries, and so, the OS doesn’t need to remove all entries from the TLB on every context switch.

Why is this change stalled

This was a very early proposal of changes for a GSoC project. The GSoC project did not go on further. The changes (and the comments) can serve as a base for anyone interested in implementing PCID support in Haiku.

Change 1437 - usb-hid: add quirk for decus gaming mouse

What is this about

This change adds a quirk to entirely replace the HID descriptor for a specific mouse.

Why is the change stalled

One part of the change is applied incorrectly to all HID devices. While it may fix the one affected by the change, it would likely break many other devices.

It is also unclear if the fixes are correct, as we have not found similar changes in other operating systems where the same hardware is working fine. The user reporting the initial issue attached some dumps of their mouse data, but they do not match the changes they did to the code. They have then stopped replying to our questions.

This change will remain stalled until someone with the same mouse can investigate the problem and explain the changes that were made.

Change 2102 - Interface kit: do not update size on redraw

What is this about

This removes code to trigger an update to a window size in the interface kit, that may not be needed.

Why is this stalled

The app_server designer reviewed the change and remains unconvinced that the code is not needed and safe to remove in all cases.

Further inspection is needed to determine in which cases the code may trigger, and attempt to reproduce the possible issues (both the one that the change is trying to fix, and the regressions that may appear by removing that code).

The change is not very long, but reviewing it properly requires deep knowledge of app_server inner workings.

OpenAI disables ChatGPT app suggestions that looked like ads

Hacker News
techoreon.com
2025-12-07 15:52:18
Comments...
Original Article

OpenAI has disabled a feature in ChatGPT that suggested third-party applications after users complained the recommendations resembled advertisements.

Mark Chen, the company’s Chief Research Officer, acknowledged that the artificial intelligence firm “fell short” in its execution of the recent promotional messages.

The controversy arose after paying subscribers to ChatGPT reported seeing messages promoting companies such as Peloton and Target. The backlash was immediate, with one user responding to the company’s initial explanations with scepticism, writing: “Don’t insult your paying users.”

In response to the complaints, OpenAI clarified that it was testing methods to surface applications built on the ChatGPT platform, a feature announced in October. The company insisted there was “no financial component” to these suggestions.

However, Mr Chen adopted an apologetic tone regarding the rollout.

“I agree that anything that feels like an ad needs to be handled with care, and we fell short,” Mr Chen wrote on Friday. “We’ve turned off this kind of suggestion while we improve the model’s precision. We’re also looking at better controls so you can dial this down or off if you don’t find it helpful.”

Nick Turley, the head of ChatGPT, also addressed the issue on Friday, noting that he was “seeing lots of confusion about ads rumours”.

“There are no live tests for ads – any screenshots you’ve seen are either not real or not ads,” Mr Turley wrote . “If we do pursue ads, we’ll take a thoughtful approach. People trust ChatGPT and anything we do will be designed to respect that.”

Speculation regarding OpenAI’s advertising ambitions increased earlier this year following the appointment of Fidji Sumo as CEO of Applications. Ms Simo, a former executive at Instacart and Facebook, was widely expected to build the company’s advertising business.

However, strategic priorities appear to have shifted recently. The Wall Street Journal reported this week that OpenAI CEO Sam Altman issued a “code red” memo. The directive reportedly prioritises work to improve the quality of ChatGPT and pushes back other product initiatives, including advertising.

Six stable kernels for the weekend

Linux Weekly News
lwn.net
2025-12-07 15:48:39
Greg Kroah-Hartman has announced the release of the 6.17.11, 6.12.61, 6.6.119, 6.1.159, 5.15.197, and 5.10.247 stable kernels. Each contains important fixes throughout the tree; users of these kernels should upgrade. ...
Original Article

[Posted December 7, 2025 by jzb]

Greg Kroah-Hartman has announced the release of the 6.17.11 , 6.12.61 , 6.6.119 , 6.1.159 , 5.15.197 , and 5.10.247 stable kernels. Each contains important fixes throughout the tree; users of these kernels should upgrade.


Locks in PostgreSQL

Hacker News
habr.com
2025-12-07 15:42:02
Comments...
Original Article

We've already discussed some object-level locks (specifically, relation-level locks), as well as row-level locks with their connection to object-level locks and also explored wait queues, which are not always fair.

We have a hodgepodge this time. We'll start with deadlocks (actually, I planned to discuss them last time, but that article was excessively long in itself), then briefly review object-level locks left and finally discuss predicate locks .

Deadlocks

When using locks, we can confront a deadlock . It occurs when one transaction tries to acquire a resource that is already in use by another transaction, while the second transaction tries to acquire a resource that is in use by the first. The figure on the left below illustrates this: solid-line arrows indicate acquired resources, while dashed-line arrows show attempts to acquire a resource that is already in use.

To visualize a deadlock, it is convenient to build the wait-for graph. To do this, we remove specific resources, leave only transactions and indicate which transaction waits for which other. If a graph contains a cycle (from a vertex, we can get to itself in a walk along arrows), this is a deadlock.



A deadlock can certainly occur not only for two transactions, but for any larger number of them.

If a deadlock occured, the involved transactions can do nothing but wait infinitely. Therefore, all DBMS, including PostgreSQL, track locks automatically.

The check, however, requires a certain effort, and it's undesirable to make it each time a new lock is requested (deadlocks are pretty infrequent after all). So, when a process tries to acquire a lock, but cannot, it queues and «falls asleep», but sets the timer to the value specified in the deadlock_timeout parameter (1 second by default). If the resource gets free earlier, this is fine and we skimp on the check. But if on expiration of deadlock_timeout , the wait continues, the waiting process will wake up and initiate the check.

If the check (which consists in building the wait-for graph and searching it for cycles) does not detect deadlocks, it continues sleeping, this time «until final victory».

Earlier, I was fairly reproached in the comments for not mentioning the lock_timeout parameter, which affects any operator and allows avoiding an infinitely long wait: if a lock cannot be acquired during the time specified, the operator terminates with a lock_not_available error. Do not confuse this parameter with statement_timeout , which limits the total time to execute the operator, no matter whether the latter waits for a lock or does a regular work.

But if a deadlock is detected, one of the transactions (in most cases, the one that initiated the check) is forced to abort. This releases the locks it acquired and enables other transactions to continue.

Deadlocks usually mean that the application is designed incorrectly. There are two ways to detect such situations: first, messages will occur in the server log and second, the value of pg_stat_database.deadlocks will increase.

Example of deadlocking

Usually deadlocks are caused by an inconsistent order of locking table rows.
Let's consider a simple example. The first transaction is going to transfer 100 rubles from the first account to the second one. To this end, the transaction reduces the first account: 

=> BEGIN;
=> UPDATE accounts SET amount = amount - 100.00 WHERE acc_no = 1;

UPDATE 1

At the same time, the second transaction is going to transfer 10 rubles from the second account to the first one. And it starts with reducing the second account: 

|  => BEGIN;
|  => UPDATE accounts SET amount = amount - 10.00 WHERE acc_no = 2;

|  UPDATE 1

Now the first transaction tries to increase the second account, but detects a lock on the row. 

=> UPDATE accounts SET amount = amount + 100.00 WHERE acc_no = 2;

Then the second transaction tries to increase the first account, but also gets blocked. 

|  => UPDATE accounts SET amount = amount + 10.00 WHERE acc_no = 1;

So a circular wait arises, which won't end on its own. In a second, the first transaction, which cannot access the resource yet, initiates a check for a deadlock and is forced to abort by the server. 

ERROR:  deadlock detected
DETAIL:  Process 16477 waits for ShareLock on transaction 530695; blocked by process 16513.
Process 16513 waits for ShareLock on transaction 530694; blocked by process 16477.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (0,2) in relation "accounts"

Now the second transaction can continue. 

|  UPDATE 1

|  => ROLLBACK;


=> ROLLBACK;

The correct way to perform such operations is to lock resources in the same order. For example: in this case, accounts can be locked in ascending order of their numbers.

Deadlock of two UPDATE commands

Sometimes we can get a deadlock in situations where, seemingly, it could never occur. For example: it is convenient and usual to treat SQL commands as atomic, but the UPDATE command locks rows as they are updated. This does not happen instantaneously. Therefore, if the order in which a command updates rows is inconsistent with the order in which another command does this, a deadlock can occur.

Although such a situation is unlikely, it can still occur. To reproduce it, we will create an index on the amount column in descending order of amount : 

=> CREATE INDEX ON accounts(amount DESC);

To be able to watch what happens, let's create a function that increases the passed value, but very-very slowly, for as long as an entire second: 

=> CREATE FUNCTION inc_slow(n numeric) RETURNS numeric AS $$
  SELECT pg_sleep(1);
  SELECT n + 100.00;
$$ LANGUAGE SQL;

We will also need the pgrowlocks extension. 

=> CREATE EXTENSION pgrowlocks;

The first UPDATE command will update the entire table. The execution plan is evident — it is sequential scan: 

|  => EXPLAIN (costs off)
|  UPDATE accounts SET amount = inc_slow(amount);

|           QUERY PLAN         
|  ----------------------------
|   Update on accounts
|     ->  Seq Scan on accounts
|  (2 rows)

Since tuples on the table page are located in ascending order of the amount (exactly how we added them), they will also be updated in the same order. Let the update start. 

|  => UPDATE accounts SET amount = inc_slow(amount);

At the same time, in another session we'll forbid sequential scans: 

||     => SET enable_seqscan = off;

In this case, for the next UPDATE operator, the planner decides to use index scan: 

||     => EXPLAIN (costs off)
||     UPDATE accounts SET amount = inc_slow(amount) WHERE amount > 100.00;

||                            QUERY PLAN                       
||     --------------------------------------------------------
||      Update on accounts
||        ->  Index Scan using accounts_amount_idx on accounts
||              Index Cond: (amount > 100.00)
||     (3 rows)

The second and third rows meet the condition, and since the index is built in descending order of the amount, the rows will be updated in a reverse order.

Let's run the next update. 

||     => UPDATE accounts SET amount = inc_slow(amount) WHERE amount > 100.00;

A quick look into the table page shows that the first operator already managed to update the first row (0,1) and the second operator updated the last row (0,3): 

=> SELECT * FROM pgrowlocks('accounts') \gx

-[ RECORD 1 ]-----------------
locked_row | (0,1)
locker     | 530699            <- the first
multi      | f
xids       | {530699}
modes      | {"No Key Update"}
pids       | {16513}
-[ RECORD 2 ]-----------------
locked_row | (0,3)
locker     | 530700            <- the second
multi      | f
xids       | {530700}
modes      | {"No Key Update"}
pids       | {16549}

One more second elapses. The first operator updated the second row, and the second one would like to do the same, but cannot. 

=> SELECT * FROM pgrowlocks('accounts') \gx

-[ RECORD 1 ]-----------------
locked_row | (0,1)
locker     | 530699            <- the first
multi      | f
xids       | {530699}
modes      | {"No Key Update"}
pids       | {16513}
-[ RECORD 2 ]-----------------
locked_row | (0,2)
locker     | 530699            <- the first was quicker
multi      | f
xids       | {530699}
modes      | {"No Key Update"}
pids       | {16513}
-[ RECORD 3 ]-----------------
locked_row | (0,3)
locker     | 530700            <- the second
multi      | f
xids       | {530700}
modes      | {"No Key Update"}
pids       | {16549}

Now the first operator would like to update the last table row, but it is already locked by the second operator. Hence a deadlock.

One of the transactions aborts: 

||     ERROR:  deadlock detected
||     DETAIL:  Process 16549 waits for ShareLock on transaction 530699; blocked by process 16513.
||     Process 16513 waits for ShareLock on transaction 530700; blocked by process 16549.
||     HINT:  See server log for query details.
||     CONTEXT:  while updating tuple (0,2) in relation "accounts"

And the second one continues: 

|  UPDATE 3

Engaging details of detecting and preventing deadlocks can be found in the lock manager README .

This completes a talk on deadlocks, and we proceed to the remaining object-level locks.

Locks on non-relations

When we need to lock a resource that is not a relation in the meaning of PostgreSQL, locks of the object type are used. Almost whatever we can think of can refer to such resources: tablespaces, subscriptions, schemas, enumerated data types and so on. Roughly, this is everything that can be found in the system catalog.

Illustrating this by a simple example. Let's start a transaction and create a table in it: 

=> BEGIN;
=> CREATE TABLE example(n integer);

Now let's see what locks of the object type appeared in pg_locks : 

=> SELECT
  database,
  (SELECT datname FROM pg_database WHERE oid = l.database) AS dbname,
  classid,
  (SELECT relname FROM pg_class WHERE oid = l.classid) AS classname,
  objid,
  mode,
  granted
FROM pg_locks l
WHERE l.locktype = 'object' AND l.pid = pg_backend_pid();

 database | dbname | classid |  classname   | objid |      mode       | granted
----------+--------+---------+--------------+-------+-----------------+---------
        0 |        |    1260 | pg_authid    | 16384 | AccessShareLock | t
    16386 | test   |    2615 | pg_namespace |  2200 | AccessShareLock | t
(2 rows)

To figure out what in particular is locked here, we need to look at three fields: database , classid and objid . We start with the first line.

database is the OID of the database that the resource being locked relates to. In this case, this column contains zero. It means that we deal with a global object, which is not specific to any database.

classid contains the OID from pg_class that matches the name of the system catalog table that actually determines the resource type. In this case, it is pg_authid , that is, a role (user) is the resource.

objid contains the OID from the system catalog table indicated by classid . 

=> SELECT rolname FROM pg_authid WHERE oid = 16384;

 rolname
---------
 student
(1 row)

We work as student , and this is exactly the role locked.

Now let's clarify the second line. The database is specified, and it is test , to which we are connected.

classid indicates the pg_namespace table, which contains schemas. 

=> SELECT nspname FROM pg_namespace WHERE oid = 2200;

 nspname
---------
 public
(1 row)

This shows that the public schema is locked.

So, we've seen that when an object is created, the owner role and schema in which the object is created get locked (in a shared mode). And this is reasonable: otherwise, someone could drop the role or schema while the transaction is not completed yet. 

=> ROLLBACK;

Lock on relation extension

When the number of rows in a relation (table, index or materialized view) increases, PostgreSQL can use free space in available pages for inserts, but evidently, once new pages also have to be added. Physically they are added at the end of the appropriate file. And this is meant by a relation extension .

To ensure that two processes do not rush to add pages simultaneously, the extension process is protected by a specialized lock of the extend type. The same lock is used when vacuuming indexes for other processes to be unable to add pages during the scan.

This lock is certainly released without waiting for completion of the transaction.

Earlier, tables could extend only by one page at a time. This caused issues during simultaneous row inserts by several processes; therefore, starting with PostgreSQL 9.6, several pages are added to tables at once (in proportion to the number of waiting processes, but not greater than 512).

Page lock

Page-level locks of the page type are used in the only case (aside from predicate locks, to be discussed later).

GIN indexes enable us to accelerate search in compound values, for instance: words in text documents (or array elements). To a first approximation, these indexes can be represented as a regular B-tree that stores separate words from the documents rather than the documents themselves. Therefore, when a new document is added, the index has to be rebuilt pretty much in order to add there each new word from the document.

For better performance, GIN index has a postponed insert feature, which is turned on by the fastupdate storage parameter. New words are quickly added to an unordered pending list first, and after a while, everything accumulated is moved to the main index structure. The gains are due to a high probability of occurrence of the same words in different documents.

To prevent moving from the pending list to the main index by several processes simultaneously, for the duration of moving, the index metapage gets locked in an exclusive mode. This does not hinder regular use of the index.

Advisory locks

Unlike other locks (such as relation-level locks), advisory locks are never acquired automatically — the application developer controls them. They are useful when, for instance, an application for some reason needs a locking logic that is not in line with the standard logic of regular locks.

Assume we have a hypothetical resource that does not match any database object (which we could lock using commands such as SELECT FOR or LOCK TABLE). We need to devise a numeric identifier for it. If a resource has a unique name, a simple option is to use its hash code: 

=> SELECT hashtext('resource1');

 hashtext  
-----------
 991601810
(1 row)

This is how we have the lock acquired: 

=> BEGIN;
=> SELECT pg_advisory_lock(hashtext('resource1'));

As usual, information on locks is available in pg_locks : 

=> SELECT locktype, objid, mode, granted 
FROM pg_locks WHERE locktype = 'advisory' AND pid = pg_backend_pid();

 locktype |   objid   |     mode      | granted 
----------+-----------+---------------+---------
 advisory | 991601810 | ExclusiveLock | t
(1 row)

For locking to be really effective, other processes must also acquire a lock on the resource prior to accessing it. Evidently the application must ensure that this rule is observed.

In the above example, the lock will be held through the end of the session rather than the transaction, as usual. 

=> COMMIT;
=> SELECT locktype, objid, mode, granted 
FROM pg_locks WHERE locktype = 'advisory' AND pid = pg_backend_pid();

 locktype |   objid   |     mode      | granted 
----------+-----------+---------------+---------
 advisory | 991601810 | ExclusiveLock | t
(1 row)

And we need to explicitly release it: 

=> SELECT pg_advisory_unlock(hashtext('resource1'));

A rich collection of functions to work with advisory locks is available for all intents and purposes:

  • pg_advisory_lock_shared has a shared lock acquired.
  • pg_advisory_xact_lock (and pg_advisory_xact_lock_shared ) has a shared lock acquired up to the end of the transaction.
  • pg_try_advisory_lock (as well as pg_try_advisory_xact_lock and pg_try_advisory_xact_lock_shared ) does not wait for a lock, but returns false if a lock could not be acquired immediately.

A collection of try_ functions is one more technique to avoid waiting for a lock, in addition to those listed in the last article .

Predicate locks

The predicate lock term occurred long ago, when early DBMS made first attempts to implement complete isolation based on locks (the Serializable level, although there was no SQL standard at that time). The issue they confronted then was that even locking of all read and updated rows did not ensure complete isolation: new rows that meet the same selection conditions can occur in the table, which causes phantoms to arise (see the article on isolation ).

The idea of predicate locks was to lock predicates rather than rows. If during execution of a query with the condition a > 10 we lock the a > 10 predicate, this won't allow us to add new rows that meet the condition to the table and will enable us to avoid phantoms. The issue is that this problem is computationally complicated; in practice, it can be solved only for very simple predicates.

In PostgreSQL, the Serializable level is implemented differently, on top of the available isolation based on data snapshots. Although the predicate lock term is still used, its meaning drastically changed. Actually these «locks» block nothing; they are used to track data dependencies between transactions.

It is proved that snapshot isolation permits an inconsistent write (write skew) anomaly and a read-only transaction anomaly , but any other anomalies are impossible. To figure out that we deal with one of the two above anomalies, we can analyze dependencies between transactions and discover certain patterns there.

Dependencies of two kinds are of interest to us:

  • One transaction reads a row that is then updated by the second transaction (RW dependency).
  • One transaction updates a row that is then read by the second transaction (WR dependency).

We can track WR dependencies using already available regular locks, but RW dependencies have to be tracked specially.

To reiterate, despite the name, predicate locks bock nothing. A check is performed at the transaction commit instead, and if a suspicious sequence of dependencies that may indicate an anomaly is discovered, the transaction aborts.

Let's look at how predicate locks are handled. To do this, we'll create a table with a pretty large number of locks and an index on it. 

=> CREATE TABLE pred(n integer);
=> INSERT INTO pred(n) SELECT g.n FROM generate_series(1,10000) g(n);
=> CREATE INDEX ON pred(n) WITH (fillfactor = 10);
=> ANALYZE pred;

If a query is executed using sequential scan of the entire table, a predicate lock on the entire table gets acquired (even if not all rows meet the filtering condition). 

|  => SELECT pg_backend_pid();

|   pg_backend_pid 
|  ----------------
|            12763
|  (1 row)


|  => BEGIN ISOLATION LEVEL SERIALIZABLE;
|  => EXPLAIN (analyze, costs off)
|    SELECT * FROM pred WHERE n > 100;

|                             QUERY PLAN                           
|  ----------------------------------------------------------------
|   Seq Scan on pred (actual time=0.047..12.709 rows=9900 loops=1)
|     Filter: (n > 100)
|     Rows Removed by Filter: 100
|   Planning Time: 0.190 ms
|   Execution Time: 15.244 ms
|  (5 rows)

All predicate locks are acquired in one special mode — SIReadLock (Serializable Isolation Read): 

=> SELECT locktype, relation::regclass, page, tuple
FROM pg_locks WHERE mode = 'SIReadLock' AND pid = 12763;

 locktype | relation | page | tuple 
----------+----------+------+-------
 relation | pred     |      |      
(1 row)


|  => ROLLBACK;

But if a query is executed using index scan, the situation changes for the better. If we deal with a B-tree, it is sufficient to have a lock acquired on the rows read and on the leaf index pages walked through — this allows us to track not only specific values, but all the range read. 

|  => BEGIN ISOLATION LEVEL SERIALIZABLE;
|  => EXPLAIN (analyze, costs off)
|    SELECT * FROM pred WHERE n BETWEEN 1000 AND 1001;

|                                       QUERY PLAN                                     
|  ------------------------------------------------------------------------------------
|   Index Only Scan using pred_n_idx on pred (actual time=0.122..0.131 rows=2 loops=1)
|     Index Cond: ((n >= 1000) AND (n <= 1001))
|     Heap Fetches: 2
|   Planning Time: 0.096 ms
|   Execution Time: 0.153 ms
|  (5 rows)


=> SELECT locktype, relation::regclass, page, tuple
FROM pg_locks WHERE mode = 'SIReadLock' AND pid = 12763;

 locktype |  relation  | page | tuple 
----------+------------+------+-------
 tuple    | pred       |    3 |   236
 tuple    | pred       |    3 |   235
 page     | pred_n_idx |   22 |      
(3 rows)

Note a few complexities.

First, a separate lock is created for each read tuple, and the number of such tuples can potentially be very large. The total number of predicate locks in the system is limited by the product of parameter values: max_pred_locks_per_transaction × max_connections (the default values are 64 and 100, respectively). The memory for these locks is allocated at the server start; an attempt to exceed this limit will result in errors.

Therefore, escalation is used for predicate locks (and only for them!). Prior to PostgreSQL 10, the limitations were hard coded, but starting this version, we can control the escalation through parameters. If the number of tuple locks related to one page exceeds max_pred_locks_per_page , these locks are replaced with one page-level lock. Consider an example: 

=> SHOW max_pred_locks_per_page;

 max_pred_locks_per_page 
-------------------------
 2
(1 row)


|  => EXPLAIN (analyze, costs off)
|    SELECT * FROM pred WHERE n BETWEEN 1000 AND 1002;

|                                       QUERY PLAN                                     
|  ------------------------------------------------------------------------------------
|   Index Only Scan using pred_n_idx on pred (actual time=0.019..0.039 rows=3 loops=1)
|     Index Cond: ((n >= 1000) AND (n <= 1002))
|     Heap Fetches: 3
|   Planning Time: 0.069 ms
|   Execution Time: 0.057 ms
|  (5 rows)

We see one lock of the page type instead of three locks of the tuple type: 

=> SELECT locktype, relation::regclass, page, tuple
FROM pg_locks WHERE mode = 'SIReadLock' AND pid = 12763;

 locktype |  relation  | page | tuple 
----------+------------+------+-------
 page     | pred       |    3 |      
 page     | pred_n_idx |   22 |      
(2 rows)

Likewise, if the number of locks on pages related to one relation exceeds max_pred_locks_per_relation , these locks are replaced with one relation-level lock.

There are no other levels: predicate locks are acquired only for relations, pages and tuples and always in the SIReadLock mode.

Certainly, escalation of locks inevitably results in an increase of the number of transactions that falsely terminate with a serialization error, and eventually, the system throuthput will decrease. Here you need to balance RAM consumption and performance.

The second complexity is that different operations with an index (for instance, due to splits of index pages when new rows are inserted) change the number of leaf pages that cover the range read. But the implementation takes this into account: 

=> INSERT INTO pred SELECT 1001 FROM generate_series(1,1000);
=> SELECT locktype, relation::regclass, page, tuple
FROM pg_locks WHERE mode = 'SIReadLock' AND pid = 12763;

 locktype |  relation  | page | tuple 
----------+------------+------+-------
 page     | pred       |    3 |      
 page     | pred_n_idx |  211 |      
 page     | pred_n_idx |  212 |      
 page     | pred_n_idx |   22 |      
(4 rows)


|  => ROLLBACK;

By the way, predicate locks are not always released immediately on completion of the transaction since they are needed to track dependencies between several transactions. But anyway, they are controlled automatically.

By no means all types of indexes in PostgreSQL support predicate locks. Before PostgreSQL 11, only B-trees could boast of this, but that version improved the situation: hash, GiST and GIN indexes were added to the list. If index access is used, but the index does not support predicate locks, a lock on the entire index is acquired. This, certainly, also increases the number of false aborts of transactions.

Finally, note that it's the use of predicate locks that limits all transactions to working at the Serializable level in order to ensure complete isolation. If a certain transaction uses a different level, it just won't acquire (and check) predicate locks.

Traditionally, providing you with a link to the predicate locking README , to start exploring the source code with.

Read on .

Martin Parr has died

Hacker News
www.bbc.co.uk
2025-12-07 15:23:48
Comments...
Original Article

Martin Parr, pictured smiling in a black jumper Image source, Getty Images

Photographer Martin Parr, whose colourful images captured British life, has died at the age of 73.

He died on Saturday at his home in Bristol, the director of the Martin Parr Foundation, Jenni Smith, told BBC News.

In a statement , external , the foundation said he would "be greatly missed" and was survived by his wife Susie, daughter Ellen, sister and grandson. It added the family asked for privacy.

The documentary photographer rose to prominence in the mid 1980s, with The Last Resort, his study of working class people on holiday in New Brighton in Merseyside.

New Brighton, England. From ‘The Last Resort’ 1983-85 © Martin Parr / Magnum Photos Image source, Martin Parr Foundation

Image caption,

A photograph from New Brighton, England, 1983-85, from 'The Last Resort'

Parr's works were known for capturing the smallest details of everyday life. His photographs were playful and had humour, but also provoked debate and discussion.

"I make serious photographs disguised as entertainment," he told The Architectural Review in 2020 , external .

"I try to point out when I find universal truths. Truth is subjective, but it's the world how I found it."

For more than 50 years, Parr's photographs observed with an apparently flat but amused and sympathetic eye the quiet rituals and absurdities of his home country, from desolate seaside towns to village fetes and modern shopping centres.

He was known for using a colour saturated palette that mimicked postcards from the 1950s and 1960s.

The shots he took in New Brighton were meant to be about capturing a moment in time and challenging people's perceptions of social classes.

New Brighton, England. From ‘The Last Resort’ 1983-85 Image source, Martin Parr Foundation

Image caption,

A photograph from New Brighton, England, 1983-85, from 'The Last Resort'

The collection showcased the best - and worst - days at the seaside, with pictures of day trippers enjoying picnics among the litter and rundown amenities which characterised the Wirral town at the time.

But those famous seaside shots became very controversial, as he himself acknowledged earlier this year ahead of a new film about his life .

"People from London and the South East, they really didn't know what places in the North looked like," said Parr, now 72.

"The litter was quite terrible, but they just weren't used to it, so it was almost like it was my fault that the place looked so scruffy."

A photograph of O'Connell Bridge, Dublin, 1981, from 'Bad Weather' Image source, Martin Parr Foundation

Image caption,

A photograph from O'Connell Bridge, Dublin, 1981, from 'Bad Weather'

Last month, in an interview with AFP, he warned the world needs the kind of satire captured in his images more than ever.

"The state we're all in is appalling," he said. "We're all too rich. We're consuming all these things in the world. And we can't. It's unsustainable."

Photographer Diane Smyth, editor of the British Journal of Photography, called Parr a "giant of post-war photography" in a tribute posted on Instagram , external .

"He was a hoot - always up for a call, especially if it was very early, and always very direct. He did he own thing, worked incredibly hard, helped others along the way - a life well-lived."

Jonathan Stephenson, who collaborated on art and design projects with Parr over the years, told BBC News he died peacefully watching football, adding he was "a firm and loyal friend".

"It was a massive privilege - and continually inspiring - to engage with Martin's eyes and mind," he said. "Martin's enthusiasm for everyday life was infectious."

Portugal updates cybercrime law to exempt security researchers

Bleeping Computer
www.bleepingcomputer.com
2025-12-07 15:09:44
Portugal has modified its cybercrime law to establish a legal safe harbor for good-faith security research and to make hacking non-punishable under certain strict conditions. [...]...
Original Article

Portugal updates cybercrime law to exempt security researchers

Portugal has modified its cybercrime law to establish a legal safe harbor for good-faith security research and to make hacking non-punishable under certain strict conditions.

First spotted by Daniel Cuthbert , a new provision in Article 8.o-A , titled "Acts not punishable due to public interest in cybersecurity," provides a legal exemption for actions that previously were classified as illegal system access or illegal data interception.

The exemption only applies when security researchers act for the purpose of identifying vulnerabilities and contributing to cybersecurity. The key conditions that must be met to beee safe from criminal liability are:

  1. The research must aim solely at identifying vulnerabilities not created by the researcher and at improving cybersecurity through disclosure.
  2. The researcher cannot seek or receive any economic benefit beyond normal professional compensation.
  3. The researcher must immediately report the vulnerability to the system owner, any relevant data controller, and the CNCS.
  4. The actions must be strictly limited to what is necessary to detect the vulnerability and must not disrupt services, alter or delete data, or cause harm.
  5. The research must not involve any unlawful processing of personal data under GDPR.
  6. The researcher must not use prohibited techniques such as DoS or DDoS attacks, social engineering, phishing, password theft, intentional data alteration, system damage, or malware deployment.
  7. Any data obtained during the research must remain confidential and be deleted within 10 days of the vulnerability being fixed.
  8. Acts performed with the system owner's consent are also exempt from punishment, but any vulnerabilities found must still be reported to the CNCS.

The new article clearly defines the limits of security research, and at the same time provides legal protection for well-intended hackers.

In November 2024, the Federal Ministry of Justice in Germany introduced a draft law that provided similar protections to security researchers who discover and responsibly report security flaws to vendors.

Earlier, in May 2022, the U.S. Department of Justice (DOJ) announced revisions to its federal prosecution policies regarding Computer Fraud and Abuse Act (CFAA) violations, adding an exemption for "good-faith" research.

Under these legal frameworks, security research is not only recognized but also given the safe space to proactively probe systems, uncover vulnerabilities, and report them without fear of legal consequences.

tines

Break down IAM silos like Bitpanda, KnowBe4, and PathAI

Broken IAM isn't just an IT problem - the impact ripples across your whole business.

This practical guide covers why traditional IAM practices fail to keep up with modern demands, examples of what "good" IAM looks like, and a simple checklist for building a scalable strategy.

Scala 3 slowed us down?

Hacker News
kmaliszewski9.github.io
2025-12-07 15:08:17
Comments...
Original Article

Is this clickbait? Not really.
Is this the fault of the language or the compiler? Definitely not.
Rather, it was part of a rushed migration. Sharing the lessons learned in the process.

I was refreshing one of our services. Part of this process was to migrate codebase from Scala 2.13 to Scala 3. I’ve done this a few times before and overall had a positive experience. Well, at least until we talk about projects with macro wizardry.

The service in question had no macros at all, but it was at the heart of data ingestion, so performance was not an afterthought.

I did it as usual - updating dependencies, compiler options and some type/syntax changes.

Then after resolving few tricky implicit resolutions and config derivations, project compiled on Scala 3.7.3 🎉

All tests passed, end-to-end flow locally works perfectly fine, so I decided to roll out the changes in a testing environment. Similarly, no issues at all. No concerning logs, all metrics ranging from infrastructure, through JVM up to application level look healthy.

With that in mind, I began a staged rollout. Again, all seem good. I kept observing the service but it looked like my job is done.

Well, as you probably can guess, it wasn’t.

The mysterious slowdown

After 5-6 hours, Kafka lag started increasing on a few environments. Of course, this wasn’t something new. Most often it is caused by a spike of data. We have pretty advanced machinery to deal with that. Usually the lag resolves by itself without any manual action.

However, this time something was off. Upstream load turned out to be relatively modest, yet we needed much more instances of the service - meaning the processing rate per instance dropped. I was confused to say the least. Why would it decrease the processing rate just on these environments?

Anyway, we decided to rollback the changes - this brought the rate back.

Digging deeper

I came back to testing. In particular, load testing. However similarly as on production environments I did not notice regression. So I played around with different payloads and granularity of messages. To my surprise, for more fine-grained, heterogeneous workloads, the processing rate significantly dropped.

Still, I had no idea why it would happen, but my bet was in the dependencies. Therefore, I tried one-by-one, reverting the serialization library, database SDK, base Docker image and even config libraries. None of these made any changes.

This made me pull out the big guns. I profiled the service using async-profiler and indeed

CPU profile looked vastly different on Scala 3 than on 2.13.

scala2_13_flamegraph

scala3_flamegraph

JVM-level CPU time was now dominated by JIT compiler while application-level by decoding.

Looking at the top of Scala 3 flamegraph I noticed a long quicklens call.

quicklens_flamegraph

What used to be transparent (frankly, I didn’t even realize we used the library), now took almost half of the total CPU time. I compared how it looks on Scala 2.13 and it was barely noticeable with around 0.5% samples.

Turns out there was indeed a subtle bug making chained evaluations inefficient in Scala 3. This also explained why the JVM spent so much time compiling.

After upgrading the library, performance and CPU characteristics on Scala 3 became indistinguishable from Scala 2.13.

Takeaways

While the details of the bug are pretty interesting(hats off to the SoftwareMill team for catching it!), that’s not my point here. I want to emphasize that libraries can behave very differently between Scala versions , especially when they rely on meta-programming.

Even if your migration is seamless and the service runs fine on Scala 3 - when performance is not just a nice-to-have, do not assume. Know your hotspots and benchmark them. Otherwise, your code will benchmark you, revealing bottlenecks in places you didn’t even know existed.

KOllector - Publishing KOReader Highlights

Lobsters
tech.stonecharioteer.com
2025-12-07 15:02:16
Comments...
Original Article

In the past year, I’ve almost entirely moved to reading digitally. I’ve used ReadEra previously, and I’ve also used MoonReader back in the day, but I’ve always wanted to try KOReader. I’d originally tried it in 2024, but I couldn’t understand the UI easily, so I gave up and returned to it this year.

Now, I use multiple reading devices, I have a Boox Tab Mini C, a Boox Palma and a few Android phones and tablets. The Boox devices use Neoreader out of the box, but I didn’t want to shift to something hardware-specific. Besides, I was beginning to like KOReader’s pagination and layout options. I’m sure UX gurus will say KOReader makes things hard, and I felt the difficult personally when I started, but after I found the Menu search option and after using it daily for a few weeks, I began to like it.

KOReader doesn’t get in my way. It’s odd to say that, but it really doesn’t. Of course, I loathe the keyboard it provides, I wish it used the device’s on-screen-keyboard, but I understand the choices they made. It’s a great piece of software, one I’m forever grateful for.

Another part of KOReader that I didn’t grok in the beginning was sync. There are a few facets to syncing.

  1. Syncing reading progress

This was easy to setup. I needed to make an account on KOReader cloud and it would sync my progress across my devices, and only my progress.

  1. Syncing the reading statistics

This was slightly harder to do. I had to make an account on Koofr, something I had never heard of, to save the reading statistics, and I need to manually synchronize them whenever I want to update the pretty charts. But once I set it up, this was fairly forgettable, but I keep reminding myself this is why I have the Koofr app on my primary phone from time to time. You don’t need the app, but I installed it back when I set it up and I haven’t removed it yet.

  1. Syncing books

This proved to be something I couldn’t natively do. But I knew I could use Syncthing, and I’m glad that the community fork of the Android app exists. I use Syncthing every day, and primarily for syncing my books across my devices.

  1. Syncing Highlights and Notes

Now this was something I gave up on. Originally, I saw some plugins for Calibre which allowed me to sync the notes. These didn’t work, but I suspect it was a “between keyboard and chair” error. And, I was fairly happy using the manual export option that one time I needed it to convert my highlights into a nice blogpost .

However, I wanted to do this again, and more frequently. I liked the format of that post, but I remember how much time it took me to make said post that I never got around to it.

Now, I’m someone who believes that there’s value in niche software. Even if I’m the only one who’s ever going to be using something, I am ready to write it. I was free this weekend, and I wanted to build something fun.

Originally, I’d tackled this with a single Python script, using Click for the CLI and cleaning the JSON that I exported from KOReader.

I never went back to that script because: 1. I didn’t remember how to use it without reading the code, and 2. I didn’t enjoy that I had to manually get the notes to my laptop to use it.

When I set up the ebook sync, I noticed that KOReader kept the book metadata, including the highlights right next to the books. I didn’t like that, so I used a non-default setting to ensure that it keeps all the highlights and other metadata in the koreader/docsettings folder at the root of any Android device.

This turned out to be a blessing. I created more Syncthing shares, one per device. Each of these corresponded to the docsettings file on those devices, and they were all synced to my computer at ~/syncthing/ebook-highlights/ folder, with the device name as the folder name. 

❯ tree -d -L 1 ~/syncthing/ebooks-highlights/
/home/stonecharioteer/syncthing/ebooks-highlights/
├── boox-palma
├── boox-tab-mini-c
├── lenovo-ideatab
├── opo
└── s9u

Now, I had to just figure out how to use this.

I originally considered setting up Jenkins or Rundeck on my homelab MiniPC. I assumed that I could write some scripts that would take care of parsing the files, merging them and updating all these folders with the merged notes.

However, I realized I wanted additional features when I spent some time thinking about this.

First, I didn’t really care that my highlights were synced across devices. I mostly read different books on different devices. Mostly . The only devices that have some overlap are my Boox devices. And, I don’t use the larger Android tablets for the same books, I use those for technical reads.

Soon, I realized that what I wanted is a management system for my highlights.

I experimented with trying to export my highlights to Karakeep, originally, but I wasn’t happy with that either. I didn’t want to have to share my highlights to Karakeep from my reading devices, and I didn’t enjoy writing a request-handler for Karakeep’s API. I’m not very sure what my experience was with it, but I didn’t feel like that was the solution.

Instead, I wanted an application that was dedicated towards getting my highlights out of my reading devices and into my blog, not all of them, but the ones I cared about.

Having set that requirement, I then decided I’d stick to what I knew for this. Asking AI to write applications is fun, but it also decides to use React or Next.js all the time. I don’t want an SPA, not at the very least. And I wanted to be able to read and understand every bit of code that it spat out. So I chose what I’m good at.

Flask. I began building this application in Flask, using Jinja2 templates and Bootstrap CSS. Oddly enough, I realized that this resulted in an aesthetic that felt less-alien to me.

For the colors, I generated the banner using Gemini and Nano Banana, and I then used this online tool to generate a color palette from the image.

I originally called it koreader-highlights-collector , until the idea to call it KOllector became obvious.

My goal with KOllector was extremely personal. I wanted to write something simple that solved a deeply personal need. I remarked to my friends that I do not see any one else needing this, but it’s something I definitely see myself using.

I don’t want to use this post to document how the code works, but I wanted to show off some screenshots because I’m rather happy that it doesn’t look like the usual vibe-coded sites I’ve seen thus far.

It reminds me a lot of the internet in the 2010s, and that is a great reason to smile.

Screenshots

Landing Page

Browsing Your Reading Library

Book list page showing multiple books with covers The book list page with search and sorting capabilities

Managing Book Metadata

Book detail editor showing metadata fields Editing book metadata and fetching information from Open Library

Viewing Highlights Across Devices

Book detail page with highlights Viewing all highlights for a book with device filtering

Sharing Individual Quotes

Exported quote as image A generated quote image with book cover background

Configuration

Each source folder here corresponds to one device’s docsettings folder. That way, I can tag each as a “device” and have that show up in the quotes as well, to show what I’ve highlighted in which device.

Configuration page for managing source folders Configuring source folders and device labels

Exporting to Blog Posts

I wanted support for Jinja2 templates to export a book’s highlights and its cover to file, just so that I can use it in my blog. I thought about doing it automatically when ingesting new highlights, but I didn’t want to tie the implementation down to my specific way of using it. I also wanted to be able to use this to export highlights to a JSON if required.

I also accounted for multiple templates, so that I can have different versions of templates that I test this with.

The backend kicks off a celery task for this and provides a separate job tracker page so that it doesn’t slow down the front end.

Export template selection Choosing an export template for generating blog posts

Next Steps

I am not done building KOllector. I plan to use this regularly to publish book posts to my blog, and continuously upgrade it. I think there’s a lot more I can envision here, and I’m happy I spent my weekend building this.

I used KOllector to publish my reading notes on Adam Hochschild’s King Leopold’s Ghost , and that made this entirely worth the effort.

Nested Learning: A new ML paradigm for continual learning

Hacker News
research.google
2025-12-07 14:47:02
Comments...
Original Article

The last decade has seen incredible progress in machine learning (ML), primarily driven by powerful neural network architectures and the algorithms used to train them. However, despite the success of large language models (LLMs), a few fundamental challenges persist, especially around continual learning, the ability for a model to actively acquire new knowledge and skills over time without forgetting old ones.

When it comes to continual learning and self-improvement, the human brain is the gold standard. It adapts through neuroplasticity — the remarkable capacity to change its structure in response to new experiences, memories, and learning. Without this ability, a person is limited to immediate context (like anterograde amnesia ). We see a similar limitation in current LLMs: their knowledge is confined to either the immediate context of their input window or the static information that they learn during pre-training.

The simple approach, continually updating a model's parameters with new data, often leads to “ catastrophic forgetting ” (CF), where learning new tasks sacrifices proficiency on old tasks. Researchers traditionally combat CF through architectural tweaks or better optimization rules. However, for too long, we have treated the model's architecture (the network structure) and the optimization algorithm (the training rule) as two separate things, which prevents us from achieving a truly unified, efficient learning system.

In our paper, “ Nested Learning: The Illusion of Deep Learning Architectures ”, published at NeurIPS 2025 , we introduce Nested Learning, which bridges this gap. Nested Learning treats a single ML model not as one continuous process, but as a system of interconnected, multi-level learning problems that are optimized simultaneously. We argue that the model's architecture and the rules used to train it (i.e., the optimization algorithm) are fundamentally the same concepts; they are just different "levels" of optimization, each with its own internal flow of information ("context flow") and update rate. By recognizing this inherent structure, Nested Learning provides a new, previously invisible dimension for designing more capable AI, allowing us to build learning components with deeper computational depth, which ultimately helps solve issues like catastrophic forgetting.

We test and validate Nested Learning through a proof-of-concept, self-modifying architecture that we call “Hope”, which achieves superior performance in language modeling and demonstrates better long-context memory management than existing state-of-the-art models.

The Nested Learning paradigm

Nested Learning reveals that a complex ML model is actually a set of coherent, interconnected optimization problems nested within each other or running in parallel. Each of these internal problems has its own context flow — its own distinct set of information from which it is trying to learn.

This perspective implies that existing deep learning methods work by essentially compressing their internal context flows. More importantly, Nested Learning reveals a new dimension for designing models, allowing us to build learning components with deeper computational depth.

To illustrate this paradigm, we look at the concept of associative memory — the ability to map and recall one thing based on another (like recalling a name when you see a face).

  • We show that the training process itself, specifically the backpropagation process, can be modeled as an associative memory. The model learns to map a given data point to the value of its local error, which serves as a measure of how "surprising" or unexpected that data point was.
  • Similarly, following previous studies (e.g., Miras ), key architectural components, such as the attention mechanism in transformers , can also be formalized as simple associative memory modules that learn the mapping between tokens in a sequence.

By defining an update frequency rate, i.e., how often each component's weights are adjusted, we can order these interconnected optimization problems into "levels." This ordered set forms the heart of the Nested Learning paradigm.

Putting Nested Learning to work

The Nested Learning perspective immediately gives us principled ways to improve existing algorithms and architectures:

Deep optimizers

Since Nested Learning views optimizers (e.g., momentum-based optimizers) as associative memory modules, it allows us to apply principles from associative memory perspective to them. We observed that many standard optimizers rely on simple dot-product similarity (a measure of how alike two vectors are by calculating the sum of the products of their corresponding components) whose update doesn't account for how different data samples relate to each other. By changing the underlying objective of the optimizer to a more standard loss metric, such as L2 regression loss (a common loss function in regression tasks that quantifies the error by summing the squares of the differences between predicted and true values), we derive new formulations for core concepts like momentum, making them more resilient to imperfect data.

Continuum memory systems

In a standard Transformer, the sequence model acts as a short-term memory, holding the immediate context, while the feedforward neural networks act as long-term memory, storing pre-training knowledge. The Nested Learning paradigm extends this concept into what we call a “continuum memory system” (CMS), where memory is seen as a spectrum of modules, each updating at a different, specific frequency rate. This creates a much richer and more effective memory system for continual learning.

Hope: A self-modifying architecture with continuum memory

As a proof-of-concept, we used Nested Learning principles to design Hope, a variant of the Titans architecture. Titans architectures are long-term memory modules that prioritize memories based on how surprising they are. Despite their powerful memory management, they only have two levels of parameters update, resulting in a first-order in-context learning. Hope, however, is a self-modifying recurrent architecture that can take advantage of unbounded levels of in-context learning and also is augmented with CMS blocks to scale to larger context windows. It can essentially optimize its own memory through a self-referential process , creating an architecture with infinite, looped learning levels.

Experiments

We conducted experiments to evaluate the effectiveness of our deep optimizers and the performance of Hope on language modeling, long-context reasoning, continual learning, and knowledge incorporation tasks. The full results are available in our paper .

Results

Our experiments confirm the power of Nested Learning, the design of continuum memory systems, and self-modifying Titans.

On a diverse set of commonly used and public language modeling and common-sense reasoning tasks, the Hope architecture demonstrates lower perplexity and higher accuracy compared to modern recurrent models and standard transformers.

Hope showcases superior memory management in long-context Needle-In-Haystack (NIAH) downstream tasks, proving that the CMSs offer a more efficient and effective way to handle extended sequences of information.

Conclusion

The Nested Learning paradigm represents a step forward in our understanding of deep learning. By treating architecture and optimization as a single, coherent system of nested optimization problems, we unlock a new dimension for design, stacking multiple levels. The resulting models, like the Hope architecture, show that a principled approach to unifying these elements can lead to more expressive, capable, and efficient learning algorithms.

We believe the Nested Learning paradigm offers a robust foundation for closing the gap between the limited, forgetting nature of current LLMs and the remarkable continual learning abilities of the human brain. We are excited for the research community to explore this new dimension and help us build the next generation of self-improving AI.

Acknowledgements

This research was conducted by Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. We thank Praneeth Kacham and Corinna Cortes for reviewing the work and their valuable suggestions. We also thank Yuan Deng and Zeman Li. Finally, we thank Mark Simborg and Kimberly Schwede for their help in crafting this blog post.

“A House of Dynamite” Is the Wrong Metaphor for US Nukes

Portside
portside.org
2025-12-07 14:46:24
“A House of Dynamite” Is the Wrong Metaphor for US Nukes Marti Sun, 12/07/2025 - 09:46 ...
Original Article

A House of Dynamite is written like an op-ed. Its characters speak in terse paragraphs that tend to close with punchy kickers. And true to the op-ed genre, all the film’s big ideas are communicated through metaphors.

“We’re talking about hitting a bullet with a bullet,” says a deputy national security advisor after describing ground-based missile defenses. “I call them rare, medium, and well-done,” says a Marine officer after passing a binder of nuclear strike options to the president, played by an uncharacteristically flat Idris Elba. Later on, the president says, “I listened to this podcast, and the guy said, ‘We all built a house filled with dynamite . . . and then we just kept on livin’ in it.’”  Even the film’s title is a metaphor.

Cringeworthiness notwithstanding — facing Armageddon, the president really quotes “the guy” from a podcast? — this line summarizes A House of Dynamite ’s main message: the problems posed by the US nuclear arsenal are impersonal, intractable, and inherited from the past.

A House of Dynamite is not an antiwar movie. It’s not even an anti-nuke movie — at least not in any robust sense. Instead, it’s an impotent and unserious exercise in handwringing.

The film illustrates the insanity of the American doctrine of nuclear deterrence (the suicidal idea, axiomatic since the 1950s, that to avoid nuclear attack we must credibly threaten to destroy the world). But it also places that doctrine beyond the bounds of political contestation by presenting it as an inevitable holdover from a history nobody asked for and for which no one is at fault.

Too Late

Idon’t know whether the podcast Elba’s character references is real. Considering how much oxygen podcasts suck up these days, especially for news-junkie liberals like screenwriter (and former NBC News head) Noah Oppenheim, I suppose it could be. But I’m not about to go scrubbing through the archives of Pod Save America looking for it. Instead, I’ll go out on a limb and guess that A House of Dynamite ’s title was inspired not by a podcast but by a passage in a 1984 book called The Abolition by the New Yorker staff writer Jonathan Schell.

Schell — who also wrote the landmark 1982 book The Fate of the Earth , about the environmental fallout of nuclear weapons — brings up a house rigged with explosives to illustrate the inane character of “deterrence” as a personal and public safety plan. His point is that “deterrence arbitrates nothing.” Even in a best-case scenario, all it can do is ensure that all disputes are indefinitely suspended, or “kept in abeyance, without any resolution.”

Schell illustrates this point through a hypothetical anecdote about a neighbor who insists upon breaking into his house. The homeowner can settle the dispute through direct means (punching or shooting the neighbor) or through civil means (calling the police). Each of these amount to reactions to a violation after it occurs.

A policy of deterrence is fundamentally different:

Under deterrence I have, in anticipation of my neighbor’s depredations, filled my house with explosives, wired them to go off the moment any unauthorized person crosses my threshold and (an essential step) informed my neighbor of what I have done — hoping, of course, that he will then have the good sense to give up any plans he might have for stealing my furniture.

Schell’s point in The Abolition is to present an actionable plan for a worldwide drawdown in nuclear capacity — an argument he can make only after pointing out the obvious absurdity of a global safety plan based on the principle of mutually assured destruction.

But as far as technological metaphors go, “a house of dynamite” is a misleading one. To state the obvious, the American nuclear arsenal is not a house. It’s not something you can erect and then leave alone. It is a dynamic system that requires daily, even hourly, input from many thousands of persons, entities, and machines. The infrastructure that undergirds the doctrine of nuclear deterrence is not stationary and self-contained, but rather continuously reloaded and constantly in motion.

Writing seventy years ago, at a historical moment when the horrors of nuclear war could not be so easily euphemized, the philosopher Günther Anders offered a better metaphor: “The bomb is a deed.”

What does it mean to say the bomb is a deed? For one thing, it forces us to consider some much more recent history than the invention of the first nukes eighty years ago. The most relevant year for understanding America’s current nuclear predicament is not 1945 but 2010. This was when the Defense Department, under the leadership of Commander in Chief Barack Obama, began a comprehensive upgrade and expansion of the US nuclear arsenal.

According to the New York Times , this in-progress nuclear upgrade involves over 100,000 scientists, engineers, and subcontractors, working in all fifty states to produce “a new fleet of bomber jets, land-based missiles and thermonuclear warheads” as well as “12 nuclear ballistic missile submarines” and a slew of other goodies. (For a glimpse into the lives and psyches of the people working on this nuclear expansion , check out the chilling Countdown: The Blinding Future of Nuclear Weapons , by the science journalist Sarah Scoles.)

Still from A House of Dynamite . (Netflix)

The nuclear upgrade is expected to end in 2042 and cost a total of 1.7 trillion dollars — that’s an expenditure of $108,000 per minute, every minute, for thirty years. Now, with the work more than halfway finished, suddenly there is a glut of cultural products commenting on the dangers of nuclear weapons. This emergent genre includes not only A House of Dynamite but also Oppenheimer , the New York Times series “ At the Brink ,” and Annie Jacobsen’s book Nuclear War: A Scenario (which Denis Villeneuve is reportedly adapting for the screen).

It is curious and a little exasperating that the American entertainment and news establishment only discovered its profound anxiety about nuclear deterrence once the once-in-a-generation rebuild of the US nuclear system was already so close to completion that it could no longer be meaningfully opposed.

Who Built the House of Dynamite?

Thanks in large part to this disastrous timing, A House of Dynamite fails as political commentary before it even begins. The film begs the question: Who built the House of Dynamite? Then it answers: Who knows? And really, who cares?

It certainly wasn’t Elba’s flustered POTUS, who learned about his own nuclear policy from a podcast. Nor was it the dysfunctional secretary of defense — a squirming worm of a character, played to off-putting perfection by Jared Harris, who spends most of the film flecking his telephone mouthpiece with spittle. Nor was it even the bullheaded general at the helm of US Strategic Command: a no-thoughts-just-rules kind of guy, played by Tracy Letts, who wants only to talk baseball and nuke Moscow (in that order).

According to the movie’s moral logic, none of these officials are responsible for the predicament they find themselves in. They are not the architects of Armageddon so much as they are the victims of history. This is the thesis of Oppenheim’s op-ed. Our political leaders live in the dynamite house just like the rest of us do. Pity them. Heavy are the heads that wear the nuclear crowns.

A House of Dynamite sinks into what the nuclear scholars Benoît Pelopidas and Neil C. Renic have called the “tragedy trap,” in which “foreseeable and solvable problems are reconceptualized as intractable dilemmas, and morally and politically accountable agents are reframed as powerless observers.” The problem with such a framing, Pelopidas and Renic argue, is that it “indulges the very hubris the tragic recognition is intended to caution against.”

Confronted with the outrage of a civilization-ending nuclear war, we are asked to identify with the most powerful men in the world and to see in their anguish and indecision a sympathetic reflection of our own horror. It takes a twisted kind of movie-magic to make an audience relate more to feckless elites spluttering into their sat-phones than to the millions of ordinary people slated to become the first casualties of humanity’s terminal war.

To borrow a phrase from former US ambassador George F. Kennan, who infamously called America’s nuke obsession “a form of illness,” everything about A House of Dynamite is “morbid in the extreme.”

Some readers will think I’m nitpicking and nay-saying here. Critics and viewers alike have already begun describing A House of Dynamite as our generation’s answer to Dr. Strangelove (1964) or The Day After (1983) — movies that, whatever their blind spots, at least brought our unending nuclear peril to widespread public attention. By presenting the stakes of nuclear brinksmanship in such stark terms, won’t A House of Dynamite inspire a kind of awareness that can only tend toward greater caution, maybe even eventual disarmament?

I’m not so sure. The truth is that apocalyptic visions of nuclear genocide can just as easily fortify US nuclear doctrine as call it into question.

In his book People of the Bomb , the anthropologist Hugh Gusterson describes meeting a woman named Sylvia who, like him, was deeply affected by the Hiroshima nuclear bombing of 1945. Gusterson, an anti-nuke activist, had nightmares set in Hiroshima; Sylvia, a Japanese American, lost family members in the attacks. But to Gusterson’s amazement, Sylvia worked as a scientist at Lawrence Livermore National Laboratory, where she designed nuclear warheads.

As an anti-nuke activist, Gusterson had attempted to publicize the most gruesome and horrifying effects of the bombing, in the hopes that images of “the shattered bodies of Hiroshima” would convince people that maintaining a nuclear arsenal was insane. But Sylvia’s experience proved that “this is not the only way these bodies can be read.”

“For those who are persuaded by the arguments in favor of nuclear weapons,” Gusterson writes, a stark knowledge of what happened at Hiroshima may simply reinforce the notion that it is important for one’s own country to have such weapons.”

I worry that A House of Dynamite only reinforces that notion too. By treating the doctrine of nuclear deterrence as an inevitable feature of the twenty-first-century world order — not as a policy position that can and must be reversed — the film may leave viewers with the sense that what we need is more investment in missiles (and missile-interceptor technology), not less.

This is in keeping with an increasingly mainstream reading of the film, which treats its heart-pounding story of nuclear apocalypse as a ninety-minute ad for Donald Trump’s much-hyped “ Golden Dome ” missile defense system. This seems to be the position of retired general Dan Karbler, a consultant on the film, who is now a major proselytizer for Golden Dome. (Karbler was chief of staff for US Strategic Command from 2018 to 2023, and he makes a cameo in the film in that role.) Meanwhile, the Pentagon’s only response to A House of Dynamite has been to insist that the US missile defense systems actually have a slightly better success rate than the 61 percent referenced in the movie.

What we need is an overall drawdown in nuclear weapons development and war planning. The last thing we need is more auxiliary missile technologies, which are fundamentally unreliable and only serve to further ratchet up the stakes of bomb research and development around the world.

But A House of Dynamite seems determined to lead you to the opposite conclusion. It’s as if, after warning you about the explosive house, the realtor then asked for your support to buy more dynamite-filled bricks.

Despite its veneer of gritty realism, A House of Dynamite is a film in love with euphemism. Perhaps the filmmakers thought US nuclear policy was so abstract, so remote, that a dash of metaphor was necessary. But euphemism also happens to be how state planners obscure the cruelty and recklessness of their war plans, as the historian Joanna Bourke has written . By metaphorizing the unthinkable, military commanders create “an anesthetizing effect” that dulls the public’s capacity for criticism.

A House of Dynamite promises to educate and agitate us. But then, like political anesthesia, it puts us right back to sleep.

Jonah Walters is currently the postdoctoral scholar in the BioCritical Studies Lab at UCLA’s Institute for Society and Genetics. He was a researcher at Jacobin from 2015 to 2020.

How One Black Labor Union Changed American History

Portside
portside.org
2025-12-07 14:41:42
How One Black Labor Union Changed American History Marti Sun, 12/07/2025 - 09:41 ...
Original Article

The fact that the meeting was even happening was enough to produce an air of subversive excitement. On August 25, 1925, a century ago this year, black sleeping car porters hoping to form a union at the Pullman Company packed the room at the Elks Hall in Harlem. Though we’ll never know how many, Pullman company spies were undoubtedly among the audience.

In fact, to combat the presence of these spies, no porters even spoke during the meeting. Instead, A. Philip Randolph, then an eccentric soapbox socialist with a string of failed unionization attempts behind him, led the meeting. He argued that a union was the only way to confront the company, address the porters’ grievances, and reclaim their manhood . And that he should be the man to lead them.

Common wisdom and past precedent suggested this campaign would go like so many before it: a flurry of enthusiasm followed by dashed hopes and a sober return to reality. But instead, this gathering initiated a twelve-year struggle to form the Brotherhood of Sleeping Car Porters and win a first contract against a corporate giant.

The significance of the Brotherhood of Sleeping Car Porters (BSCP) went far beyond one union and its members. “The Brotherhood,” as many members affectionately called it, would become a vessel through which to educate black communities about labor unions and challenge paternalistic corporate relationships. It acted as a critical institutional anchor through which broader fights for civil rights were waged and activist pressure tactics were developed.

The story of the BSCP illuminates the deep historical connection between the labor movement and civil rights. Through patient institution building and dogged determination, the union was able to shift the consciousness and balance of power within black communities to support unionization. This coalition was the backbone of the historic progress made toward civil rights during the mid-twentieth century. Rather than leave it in the past, this same coalition can provide the basis for fighting racial inequality today.

“Trained as a Race”

Black Pullman porters occupied a complicated class position within black communities. Associated simultaneously with dignity and servility, the porter represented a contradictory symbol of black advancement. Their emergence can be traced quite literally back to chattel slavery.

In the late nineteenth century, industrialist George Pullman designed luxury train cars to transport passengers across the country. The genius was in making this service available to the middle classes, not just the wealthy elite. The idea took off, and by 1895 Pullman had 2,556 sleeping cars that traveled over 126,660 miles of track. At the company’s peak, the sleeping cars hosted one hundred thousand passengers a night, more than all of the country’s major hotels combined.

The key to this luxury was that it wasn’t just a bed to lay your head on and some food to eat. Passengers would have their own personal servants at their beck and call: the Pullman porter. Cynically, Pullman reserved these jobs for Southern black men, preferably the formerly enslaved.

A sleeping car porter from Pullman, Chicago, helps a woman. (Author and date unknown)

Pullman felt this was a perfect match, explaining how black former slaves were “trained as a race by years of personal service in various capacities, and by nature adapted faithfully to perform their duties under circumstances which necessitate unfailing good nature, solicitude, and faithfulness.” In a further insult to their dignity, most porters were referred to as “George,” harkening back to the days of slaves being named after their masters.

The expectation of complete subservience was reinforced by the fact that porters mainly relied on tips for their wages. The surest path to a fat tip was catering to the customers’ every need and enduring each humiliation with a smile. Shining shoes, running a bath , mailing letters, lugging baggage, and looking the other way at indiscretions were all in a day’s work. Former NAACP president Roy Wilkins, who worked as a Pullman porter as a young man, said they “worked like house slaves on roller skates.”

The hours were brutal. On average, a porter had to work almost 350 hours per month. Especially in the early years, they would have a hard time getting more than three hours of sleep at night while on a trip. Porters would have to pay out of their meager wages for a work uniform and supplies like shoe polish.

But despite these conditions, a sleeping car porter was viewed as a prestigious job within black communities. With their crisp Pullman uniforms, porters had an air of distinction. Their work was not “dirty,” unlike so many jobs black workers were relegated to. The pay wasn’t great, but still much better than most other jobs working-class blacks could hope to find. A Pullman porter was seen as a proud representative of a small but growing black middle class. In Chicago, for example, home ownership among porters was at a high 57 percent in 1927.

A Pullman porter working at Chicago’s Union Station, January 1943 (The Library of Congress Prints and Photographs Division)

Samuel Turner worked forty-one years on the railroad, mostly on dining cars. He said to Rising from the Rails author Larry Tye that he “always wanted to be a sleeping car porter. They had those pretty uniforms on, they made tips, and they had those high-class people riding those sleeping cars, people who had money. All those porters had nice houses, beautiful homes. You were almost considered a doctor.”

Many porters used the job to pay for their college education. From Thurgood Marshall to Malcolm X, a list of former porters can read like a who’s who of black history. Beyond economic stability, a porter represented well-traveled cosmopolitanism and sophistication. E. D. Nixon, a porter and BSCP leader in Alabama, said that when a porter spoke “everybody listened because they knowed the porter been everywhere and they never been anywhere themselves.”

Pullman porters became an important conduit for spreading information and new ideas to black communities. The editor of the Chicago Defender , one of the largest and most important black newspapers, used porters to distribute the paper all across the South in barbershops, churches, and other social settings. It was likely thanks to porters that by 1920 the paper had a circulation of 230,000, two-thirds from outside Chicago.

There had been previous efforts by porters to organize, but they never lasted long. In 1890, a group of porters known as the Charles Sumner Association threatened to strike but backed down when Pullman threatened to replace them with white workers. In 1901, a group of porters even got their demands published in the St. Louis Post-Dispatch .

Such initiatives were snuffed out through brute intimidation and then eventually clever co-optation by the Employee Representation Plan (ERP), a company union set up in 1920. In response to worker rumblings, the ERP instituted a paltry 8 percent wage increase.

One of the officers of the ERP was a respected porter named Ashley Totten who read the Messenger , A. Philip Randolph’s socialist magazine, and listened to some of his soapbox speeches. He and some other reps were fed up with the ERP’s ineffectiveness and thought Randolph could be the perfect outsider to agitate porters without fear of retaliation from the company.

While Randolph was indeed outspoken, his record in unionization efforts wasn’t exactly inspiring. Though his agitation against World War I earned him the title of “most dangerous Negro in America” from the State Department, he struggled to make his socialist ideas have an impact on the real world. His attempt to organize black elevator operators and waiters ended in disaster. As most unions within the American Federation of Labor (AFL) banned black workers, his quest to promote unionism within black communities seemed delusional and out of touch.

A. Philip Randolph pictured in the magazine he cofounded, the Messenger (Cropped image from HathiTrust )

During the 1910s and early 1920s, Marcus Garvey’s ideology of self-help, black nationalism, and international racial glory captured the imaginations of the black masses. Randolph would later acknowledge, “Socialism and trade unionism called for rigorous social struggle — hard work and programs — and few people wanted to think about that. Against the emotional power of Garveyism, what I was preaching didn’t stand a chance.”

But Randolph saw in the porters’ struggle a symbol for the strivings of all black working people. Believing that they were “made to order to carry the gospel of unionism in the colored world,” he threw himself into his newfound leadership role.

Initially, the campaign gained momentum. At the first mass meeting at the Elks Hall, Randolph raised the major demands of a wage of $150 per month, a limit of 240 hours of work per week, and an end to the demeaning practice of tipping. The next day, two hundred New York porters streamed into the Messenger office, which now also served as union headquarters. The Amsterdam News described the occasion as “the greatest mass meeting ever held of, for and by Negro working men.”

Breaking Pullman’s Web of Paternalism

To go up against Pullman and win, the union could not limit itself to persuading the workers. The BSCP had to wage a crusade for the hearts and minds of the communities where workers lived and shift the balance of power within important black institutions. Over the course of decades, Pullman had perfected a web of paternalism to ensure the loyalty of key black constituencies.

Beth Tomkins Bates’s Pullman Porters and the Rise of Protest Politics in Black America offers an excellent account of the paternalist network that bonded Chicago’s black community to the Pullman company, and how over time the BSCP was able to break it.

The company was already seen as benevolent for being such a consistent employer of black workers in the first place. This image was reinforced by substantial financial support of black institutions like churches, the Urban League, and the Wabash Avenue YMCA. Without Pullman funding, Provident Hospital, the first large-scale civil project in Chicago’s black community, would not have been possible. Ida B. Wells and Frederick Douglass even took part in its opening ceremony in 1893.

Pullman courted and won over a phalanx of prominent black figures. Chicago Defender editor Julius Nelthropp Avendorph was hired as an assistant and kept Pullman abreast of developments in the black community. Claude Barnett, founder of the Associated Negro Press, was given funding to publish Heebie Jeebies as an anti-union propaganda outlet.

Control of black churches ensured that anti-union propaganda could also be spread at the pulpit on Sundays. One of the most important relationships cultivated by Pullman was with the Quinn Chapel AME Church under Reverend Archibald James Carey. The church ran an employment service funneling workers to Pullman, and Carey refused to allow Randolph or any other pro-union figures to speak there. His blunt explanation was consistent with the view of many black religious institutions at the time: “The interest of my people lies with the wealth of the nation and with the class of white people who control it.”

Just as importantly, Pullman gave workers various social outlets in the form of company-sponsored baseball games, concerts, and barbecues. Its annual picnic in Jersey City was described by the New York Times as the “Newport of the colored social world.” Randolph and other BSCP leaders understood that any success would depend on their ability to make the union a defining presence in black social life, too.

Slowly but surely, the BSCP began to make inroads. Early on, women’s political clubs were instrumental in connecting Brotherhood activists to a broader political network. Ida B. Wells was active in this scene and organized the Wells Club and Negro Fellowship League, where discussions were held on Pullman unionization. After Randolph spoke at the Chicago and Northern District Federation of Women’s Clubs in December 1925, Wells invited him into her home and endorsed the union effort.

The BSCP Women’s Auxiliary, consisting mostly of the wives of Pullman porters, gave vital support as well. Often women went to meetings to prevent retaliation against male workers. BSCP organizer Benjamin McLaurin explained, “We had to function through the women at that time because they could attend meetings, they could pick up the literature.” Auxiliary chapters organized study groups and fundraisers for the union’s cause.

While Chicago’s black faith community began as largely united in opposition to the union, activists took advantage of some early cracks. Dr William D. Cook from Metropolitan Community Church was the only invited speaker to show up at the BSCP’s first meeting in October 1925. Known as an “outlaw preacher,” Ida B. Wells and her club friends were some of his church’s first members. Cook opened up his church to Randolph two months later to speak on “The Negro and Industrial Emancipation.”

Dr Junius C. Austin came from Pittsburgh to Chicago in 1926 and pastored Pilgrim Baptist Church. In Pittsburgh, he was a vocal supporter of Marcus Garvey’s United Negro Improvement Association (UNIA) but was more open to supporting the BSCP in Chicago because he wasn’t enmeshed in the local Pullman patronage machine. He allowed the BSCP to use his church for meeting space.

Though the ideologies of the UNIA and BSCP were almost diametrically opposed, it wasn’t entirely rare for people like Austin to support both organizations in different contexts. This speaks to the ability of the union to reframe and redirect working-class black militancy and desires for self-organization. Civil rights–minded activists like Austin wanted direct action to advance the interests of the race and were willing to align with whomever took the initiative.

Milton Webster, the hard-nosed, politically connected head of the Brotherhood’s Chicago division, put together a Citizens Committee as a venue to build public support for the union in the city’s black community. The people that formed the original core of the Committee were varied in their class and associational background, which made it all the more powerful.

Irene Gaines, an experienced activist through women’s political clubs and as industrial secretary of the Young Women’s Christian Association (YWCA), was an early recruit to the Committee. Perhaps one of the most unlikely members, George Cleveland Hall was a prominent businessman and personal friend to the anti-union Booker T. Washington. But as an advocate for black self-organization, this all-black upstart union captured his imagination.

The Citizens’ Committee organized regular “labor conferences” that brought Brotherhood allies together and stimulated deeper thinking about the role of black communities in the economy. Described by the Chicago Defender as a “movement to stir interest in serious economic problems and to educate the Race in channels of thought where there hadn’t been much before,” these labor conferences served both an organizational and ideological role within Chicago’s black community. By 1929, nearly two thousand people attended these gatherings.

The union very consciously portrayed its fight as a continuation of black people’s long-standing quest for civil rights and equality. Quotes from Frederick Douglass, especially his “power concedes nothing without a demand,” can be found throughout Brotherhood literature. One union bulletin read, “Douglass fought for the abolition of chattel slavery, and today we fight for economic freedom. The time has passed when a grown-up black man should beg a grown-up white man for anything.”

These appeals were all the more poignant because much of the Brotherhood membership had a direct connection to chattel slavery. In the pages of the Messenger , the unofficial organ of the union, the story of Silas M. Taylor was published. Taylor was born enslaved and went to work in a tobacco factory in Virginia after emancipation. Finding conditions too similar to slavery, he unsuccessfully attempted to lead a strike. He became a porter, where he served for forty years and became the head of the Boston chapter of the Brotherhood.

Taylor was fired without a pension for his militancy, to which he said in response: “They can withhold my pension . . . I am not old. I was born when the BSCP was born.” Taylor’s story embodied the lives of so many other porters and the symbolic importance of the union’s quest for economic freedom.

It became hard to disentangle the fate of the union from that of black civic life more broadly. One pro-union cartoon in the Messenger showed a porter voting for the union with the caption “I am voting for myself, my children and my race this time.” Being supportive of or opposed to the Brotherhood became a lightning rod issue that animated fierce battles among different sections of black civil society. Black labor historians Sterling D. Spero and Abram L. Harris said it was “impossible for a leader to remain neutral toward the union,” with one’s position becoming a “fundamental test” of racial militancy.

Black press outlets were put on the spot and forced to wrestle with the issue. The Chicago Defender received ad revenues from the Pullman Company and initially opposed the unionization effort. Never afraid to go to battle against anti-union forces within the black community, Randolph launched a boycott against the paper at a 1927 mass meeting held by the union in Chicago. By the end of the year, calculating that its reputation among black people was of more value than ad revenues, the Defender came out in support of the Brotherhood.

Battle With the Courts and the Company

At the same time that public sympathy was mobilized, Brotherhood leaders had to continue building and maintaining membership while a lengthy court battle unfolded. By June 1927, the Railway Labor Act mediation board recognized the union as representing a majority of Pullman porters. But they still lacked the power or leverage to force the company to the table to negotiate a contract.

Randolph felt that if they presented a credible strike threat and created a national crisis, the president could be called on to step in and force the company to bargain. The union threw itself into strike preparation, and by the spring of 1928 claimed that over six thousand porters had voted to strike. But Pullman called the bluff, saying they could easily be replaced. The mediation board believed the company and felt that since there was no crisis, the president did not need to be brought in. Randolph, after getting pressure from AFL president William Green, decided to call off the strike.

The union had shown its hand and lost. After years of momentum, the whole project was thrown into doubt. Randolph faced serious questions about his leadership, and it all seemed like a repeat of his past failures. The St. Louis Argus wrote, “The proper thing now for those who have given him money is to demand so much of it back.” The disappointment of the aborted strike combined with the pressures of the Great Depression to create a death spiral for the union.

BSCP membership fell from a peak of 4,632 in 1928 to 1,091 in 1931. These were years of intense personal struggle for the union stalwarts who stayed committed. Randolph conducted union affairs in tattered suits and with holes in his shoes. He often had to pass the hat at the end of meetings in order to get from one place to the next. E. J. Bradley, the director of the St Louis BSCP division, was one of the most powerful symbols of this sacrifice. He lost two homes and a wife due to his involvement and lived out of his car until the debt collectors took that too. But he refused to give up and received the title “noblest Roman of them all.”

Spero and Harris, though great supporters of the union from the beginning, were ready to throw in the towel in 1931 when they wrote, “The hope that this movement would become the center and rallying point for Negro labor as a whole is now dead.” But the union was able to bounce back, and not just due to the faith and persistence of Brotherhood leaders.

The election of Franklin D. Roosevelt as president in 1932 was a lifeline to the Brotherhood. The New Deal is often maligned by progressives today, portrayed as at best irrelevant to black people and at worst an agent of racial discrimination. This conception flies in the face of the actual historical record and lived experience of black working people. William H. Harris, in his account of the BSCP titled Keeping the Faith , argues, “One cannot overemphasize the importance of changes wrought by the Great Depression, particularly the New Deal, to the success of the Brotherhood.”

Progress didn’t come immediately, for porters were not covered by the National Industrial Recovery Act. But in 1934, Roosevelt signed an amendment to the Railway Labor Act which included porters, banned “yellow dog” contracts with company unions like the ERP, and required corporations like Pullman to negotiate with unions that represented the majority of their workers. The change in momentum on the ground was swift. In 1933, BSCP membership had dropped to a paltry 658. By 1934, it shot back up to 2,627.

The long, patient work Randolph and Brotherhood leaders had done building support within the black community and the American Federation of Labor was paying off. Back in 1929, Randolph facilitated the appearance of AFL president William Green at Abyssinian Baptist Church in Harlem to speak to porters and black civic leaders. At the time, this was a rare occurrence and helped a little to break down the justified tensions between black workers and the AFL. While always a reluctant and tentative partner, the AFL giving institutional support to the porters was crucial for the Brotherhood’s eventual success.

Prominent black organizations like the NAACP and Urban League began to focus more on economic issues and embrace trade unions, in no small part due to Randolph’s relentless propaganda. Abram L. Harris chaired the NAACP’s new Committee on Future Plan and Program in 1934, which called for radical economic measures. The Urban League began to set up workers’ councils, which educated black people about the benefits of unions. Both organizations publicly endorsed the Brotherhood, and on July 1, 1935, the union won an official election by the porters, 5,931 to 1,422.

On August 25, 1937 — twelve years to the day after Randolph’s first public meeting with porters in 1925 — the Pullman Company signed a collective bargaining agreement with the Brotherhood of Sleeping Car Porters. It was a fulfillment of many of the union’s initial demands and changed the lives of porters. The working month was reduced from four hundred hours hours to two hundred, the wage package increased salaries by a total of $1.25 million, and a grievance procedure was established. The Chicago Defender described the contract as “the largest single financial transaction any group of the Race has ever negotiated.”

Roy Wilkins, who worked as a porter himself before becoming NAACP president in 1955, said there were three events during the 1930s that made him proud to be black. Two were sporting events: Jesse Owens’s performance during the 1936 Olympics and Joe Louis’s knockout of Max Schmeling that same year. But the third was “the day the Pullman company, after a contract wrangle that had lasted more than a decade, summoned in A. Philip Randolph and the leaders of the BSCP and said, ‘Gentlemen, the Pullman company is ready to sign.'”

Anchor of the Civil Rights Movement

Randolph could never disentangle his role as a union leader from his role as a civil rights crusader. Having established itself as a leading force in black labor, the Brotherhood used its institutional muscle and vast social networks to stimulate political activity against racial inequality. The coming of World War II provided a great opportunity.

The war mobilized industry and signaled the final death knell for the Great Depression. But black workers remained largely locked out of defense industry jobs. The issue hit a raw nerve for black workers and heightened the contradiction of fighting a war for democracy while being excluded from it. For the US government, the problem risked ballooning into a national security crisis. Therein lay Randolph’s point of leverage.

Randolph called for a March on Washington to secure jobs for blacks in the defense industries, along with other demands like desegregation of the armed forces. Today marches on Washington barely merit a mention, but at the time it was an audacious idea, especially when it came to mobilizing working-class blacks to do it. March on Washington Movement (MOWM) chapters were established in cities across the country, and it wasn’t a surprise that they were strongest wherever there were large BSCP locals.

BSCP members were leaders in the effort, with the union offering meeting space and other logistical support. Randolph held large rallies across the country, while porters spread the word on their rides. This movement was not engaged in the polite lobbying of the middle class that characterized most NAACP efforts of that time. MOWM had a more militant edge and lived in the union halls, fraternal chapters, salons, movie theaters, bars, and poolrooms of working-class black America.

Randolph claimed he could bring one hundred thousand black people to descend on the nation’s capital, but no one really knew what the number was. Roosevelt recognized that whatever the specific number, the threat of a large domestic disturbance right as the United States entered the war was a credible one. He blinked and signed Executive Order 8802, banning discrimination in defense industries and establishing the Fair Employment Practices Committee (FEPC). Randolph and the porters had successfully harnessed and mobilized black militancy toward concrete material gains in a way that radical black nationalists hadn’t been able to. MOWM activist Richard Parrish reflected in the 1970s that the march “scared these people like no other thing. Marcus Garvey, Malcolm X, H. Rap Brown, all wrapped together, never had the power, the real power, and threat that the first march had.”

After getting this win, Randolph called off the march but kept the movement in place to enforce the order in localities. Though only lasting for a relatively brief time in the 1940s, the MOWM established the social networks, protest strategies, and political confidence that would fully blossom during the “classical phase” of the Civil Rights Movement in the 1950s and 1960s. Here again, the Brotherhood was instrumental.

St Louis was home to both a very strong MOWM chapter and BSCP local, and porter T. D. McNeal served as its leader. The chapter routinely turned out hundreds of people to pickets at local defense manufacturing plants and held a massive rally against layoffs that drew ten thousand people. In May 1942, they led a silent march of five hundred around the US Cartridge Company complex, which resulted in the raising of black workers’ wages and the hiring of seventy-two black women.

Anticipating this tactic’s widespread use in the 1960s, St Louis MOWM led sit-ins at diners and public utility companies like Southwestern Bell Telephone that won agreements to hire black workers. The FBI, which had taken a worried interest in the MOWM, concluded that “the most active Negro organization in the City of St Louis is the March on Washington Movement.”

The MOWM thrived in other cities like Chicago and New York City, also BSCP strongholds. On June 16, 1942, a MOWM event was staged in Madison Square Garden that the Pittsburgh Courier described as “the greatest race meeting in this city’s history.” It wasn’t just a rally; it was a tour de force of black political and cultural expression. Civil rights–themed skits were staged and militant speeches were made by a who’s who list of black leaders. Adam Clayton Powell Jr, the Abyssinian Baptist Church pastor and city council member, used the event to announce his historic run for Congress.

Historian David Welky described MOWM’s captivating presence in Harlem: “Around eighteen thousand African Americans streamed downtown in their Sunday best. Women wearing festive hats and men in solemn ties jammed buses and subway trains . . . Sixty blocks uptown, Harlem’s street culture fell silent out of respect for Randolph’s audacity.”

As the Congress of Industrial Organizations began to seriously get down to the task of organizing black workers, they relied extensively on black political networks that had developed while supporting the BSCP and MOWM. Halena Wilson , for example, was president of the Chicago Women’s Economic Council and was tapped to help organize the Inland Steel Company in Indiana Harbor. She drew on her BSCP ties to help five thousand black workers sign up for the Steel Workers Organizing Committee in 1937.

This feverish period of BSCP-led activity during the 1930s and 1940s opened up opportunities for black women to exercise leadership in black political activism. While often this was not through formal leadership titles, black women played indispensable roles in organizing direct action and providing the administrative backbone for movement activities.

Randolph was an inspiring and visionary leader, but women like E. Pauline Myers and Anna Arnold Hedgeman mostly staffed the MOWM offices and attended to the day-to-day organizational tasks that made the organization function. While T. D. McNeal was made the face of the sit-in movement for jobs in St Louis, he admitted, “These women really did the work.”

Maida Springer , who became an organizer for the International Ladies’ Garment Workers’ Union (ILGWU), names Randolph as an important early mentor. As a young girl, she remembers going over to a family friend’s house to stuff leaflets for the BSCP union drive. She marched with the union when they won their first contract in 1937 and was part of Randolph’s inner circle during the MOWM of the 1940s.

The BSCP Ladies Auxiliary didn’t just assist their husbands in the fight to form a union; they engaged in consumer activism during the war, too. The higher wages won by Pullman porters allowed many wives of porters to stay at home without work, a rare luxury for most black women at the time. Some auxiliary groups, like in Chicago, formed reading groups about consumer cooperatives and even established their own.

Ladies Auxiliary groups lobbied Congress to pass milk price control legislation and worked to support the Office of Price Administration (OPA) to enforce price controls at the local level. In St Louis, they monitored rent prices. The OPA formally recognized the Denver BSCP Ladies Auxiliary and said, “No women in the city are better informed or more cooperative than these women.”

Given all of this, it should be no surprise that the BSCP played a central role during the catalyzing event of the modern Civil Rights Movement: the Montgomery Bus Boycott. E. D. Nixon, president of the Montgomery BSCP, bailed Rosa Parks out of jail after her arrest. The Montgomery BSCP union hall became the meeting space of the boycott movement, and Nixon’s extensive organizing experience and broad social network were invaluable throughout.

Pullman porters, the itinerant eyes and ears of the civil rights fight, reported lynchings to groups like the NAACP. The union gave money and legal support to higher-skilled black workers like firemen, brakemen, and switchmen fighting to end employment discrimination and keep their jobs. During the March on Washington, the fulfillment of A. Philip Randolph’s original idea, the BSCP gave $50,000.

The Brotherhood was not just a union of black workers. It was a movement: an institution for black economic advancement and social equality. The union embodied the necessity for civil rights to be grounded in an economic outlook and a working-class base. The experience of this movement offers a host of lessons for organizers today on the role of building broad public support, political education, and making a union an institutional anchor for larger political fights.

Transposing experiences from 1925 to 2025 is dangerous and fraught. The BSCP relied on and mobilized a vast civil society network within black communities that amplified and reinforced their aims. We live in a much more atomized society with declining associational life. But still, people engage with sports leagues, churches, PTAs, and other organizations. Black workers are still prominent throughout our economy, from auto plants and warehouses to the postal service and public schools.

In February 2025, Teamsters Local 100 held a Black History Month event at their union hall in Cincinnati, Ohio. Over 150 members packed the hall, many of whom rarely attended union events. This social networking planted the seeds for a for a contract campaign by members at Zenith Logistics, a third-party operator for Kroger where most of the workers are black and Latino. They gathered contract surveys in multiple languages, wore “Will Strike if Provoked” shirts and all clocked in at the same time in front of management. The workers won a contract with the best wage and benefit gains they’d ever seen, along with language protecting members from ICE raids. Some of these members are now becoming shop floor leaders and could be seen proudly at the Teamsters for a Democratic Union convention .

One can’t help but see the spirit of the Brotherhood in them.

Paul Prescod is a Jacobin contributing editor.

Dollar-stores overcharge cash-strapped customers while promising low prices

Hacker News
www.theguardian.com
2025-12-07 14:37:21
Comments...
Original Article

On a cloudy winter day, a state government inspector named Ryan Coffield walked into a Family Dollar store in Windsor, North Carolina , carrying a scanner gun and a laptop.

Inside the store, which sits along a three-lane road in a county of peanut growers and poultry workers, Coffield scanned 300 items and recorded their shelf prices. He carried the scanned bar codes to the cashier and watched as item after item rang up at a higher price.

Red Baron frozen pizzas, listed on the shelf at $5, rang up at $7.65. Bounty paper towels, shelf price $10.99, rang up at $15.50. Kellogg’s Frosted Flakes, Stouffer’s frozen meatloaf, Sprite and Pepsi, ibuprofen, Klondike Minis – shoppers were overpaying for all of them. Pedigree puppy food, listed at $12.25, rang up at $14.75.

All told, 69 of the 300 items came up higher at the register: a 23% error rate that exceeded the state’s limit by more than tenfold. Some of the price tags were months out of date.

The January 2023 inspection produced the store’s fourth consecutive failure, and Coffield’s agency, the state department of agriculture & consumer services, had fined Family Dollar after two previous visits. But North Carolina law caps penalties at $5,000 per inspection, offering retailers little incentive to fix the problem. “Sometimes it is cheaper to pay the fines,” said Chad Parker, who runs the agency’s weights-and-measures program.

a man carrying a basket
Chris Outlaw shops at Family Dollar’s King Street location in Windsor, North Carolina, on 24 November. Photograph: Cornell Watson/The Guardian

The dollar-store industry, including Family Dollar and its larger rival, Dollar General, promises everyday low prices for household essentials. But an investigation by the Guardian found that the prices listed on the shelves at these two chains often don’t materialize at checkout – in North Carolina and around the country. As the cost of living soars across America, the customers bearing the burden are those who can least afford it – customers who often don’t even notice they’re overpaying.

These overcharges are widespread.

Dollar General stores have failed more than 4,300 government price-accuracy inspections in 23 states since January 2022, a Guardian review found. Family Dollar stores have failed more than 2,100 price inspections in 20 states over the same time span, the review found.

Among these thousands of failed inspections, some of the biggest flops include a 76% error rate in October 2022 at a Dollar General in Hamilton, Ohio; a 68% error rate in February 2023 at a Family Dollar in Bound Brook, New Jersey; and a 58% error rate three months ago at a Family Dollar in Lorain, Ohio.

Many of the stores that failed state or local government checks were repeat violators. A Family Dollar in Provo, Utah, flunked 28 inspections in a row – failures that included a 48% overcharge rate in May 2024 and a 12% overcharge rate in October 2025.

a person holding a price tag
A price tag at Family Dollar on King Street in Windsor, North Carolina, on 24 November. Photograph: Cornell Watson/The Guardian

The chains’ pricing disparities are drawing increasing attention. In May, Arizona’s attorney general announced a $600,000 settlement to resolve a consumer-fraud investigation against Family Dollar. In October, Colorado’s attorney general settled with Dollar General for $400,000 after its stores failed 15 out of 23 state inspections. Dollar General has also settled with New Jersey , Vermont and Wisconsin , and both companies have settled with Ohio.

Linda Davis, a 64-year-old Family Dollar shopper in Dayton, Ohio, called the state attorney general’s office in February after walking home from the dollar store and discovering that 12 of her 23 purchases had rung up incorrectly. “I’m adding it up in my head as I’m shopping,” she told the Guardian. “But I was way off and I didn’t know why … I thought: where did I miscalculate? I’ve [only] got so much cash on me.”

Davis, who lives on social security, said she could shop elsewhere, but that would involve paying for a bus ride. “I don’t have money like that,” she said.

Both Family Dollar and Dollar General declined interview requests and did not answer detailed lists of questions from the Guardian. Instead, both sent the Guardian brief statements.

“At Family Dollar, we take customer trust seriously and are committed to ensuring pricing accuracy across our stores,” the company said. “We are currently reviewing the concerns raised and working to better understand any potential discrepancies. We continue to be focused on providing a consistent and transparent shopping experience.”

Dollar General said it was “committed to providing customers with accurate prices on items purchased in our stores, and we are disappointed any time we fail to deliver on this commitment”. In one court case in Ohio, Dollar General’s lawyers argued that “it is virtually impossible for a retailer to match shelf pricing and scanned pricing 100% of the time for all items. Perfection in this regard is neither plausible nor expected under the law.”

The Guardian’s examination of inspection failures by the two chains was based on record requests to 45 states and more than 140 counties and cities in New York, Ohio and California, along with court documents and public databases.

In nearly half of US states, information about whether customers are being overcharged was limited or unavailable. Many states do little or nothing to monitor retail stores’ pricing practices. Some, like Maryland, Idaho and Washington, do no random inspections, responding only to consumer complaints. Illinois, South Carolina and others don’t inspect at all. In 2020, auditors in Kansas revealed that these inspections were a low priority in many states. “Consumers can check price accuracy themselves,” they wrote.

Even in states with tougher enforcement, financial penalties don’t always solve the problem: in the 23 months after Dollar General agreed in November 2023 to pay Wisconsin $850,000, its stores failed 31% of their price inspections. During the same period, Wisconsin’s Family Dollar stores failed 30% of their state inspections.

a car i front of a building
A Dollar General store in Port Henry, New York, is one of two within a 5-mile radius. Photograph: Kelly Burgess/The Guardian

According to industry watchers, employees and lawsuits, overcharges often stem from labor practices within the dollar-store sector. When a company changes prices, the registers are updated automatically. But the shelf prices are not: someone needs to remove the old labels manually and replace them with new ones. In an industry known for minimal staffing, workers don’t always have time to put up the new shelf tags.

In many instances, customers may not notice that they are being charged more than what’s listed on the shelf. If they notice at the register, they may decide to put those items back – or ask a store employee to honor the shelf price.

Dollar General, in its statement, said its store teams “are empowered to correct the matter on the spot”. But customers and current and former employees said that while some dollar stores will correct the price, others refuse to make fixes at the register – and turn away customers who return later and request a refund.

“Overcharging even by a small amount per item can strain a really tight budget,” said Elizabeth M Harris, acting director of the New Jersey division of consumer affairs. “If you’ve ever gone into any store … with a child like I have, there’s chaos at the checkout counter and you’re not really paying attention.” With items being rung up quickly, she added, “consumers are trusting that the retailer is actually charging them the price that’s displayed.”

Her state settled in 2023 with Dollar General for $1.2m after finding more than 2,000 items rung up as overcharges across 58 stores.

Even if the overcharges paid by dollar-store customers are accidental, they still reflect the industry’s decision not to correct a problem it has known about for years, according to Kennedy Smith, a researcher at the non-profit Institute for Local Self-Reliance, which works to protect communities from negative impacts of big corporations.

“If they’re called on it, they’ll say, ‘Oh yeah, our mistake,’” Kennedy said. “Until they’re called on it, they’re happy to let those scanner errors bring in the millions.”

‘The cheap stuff’

When consumers feel economic pain, as they do now thanks to rising costs exacerbated by tariffs, price gouging and other inflationary pressures, one place they turn to are dollar stores. These one-stop centers for inexpensive food, clothing and housewares tend to sell in small quantities, one $1 chicken-noodle-soup can at a time. And they are relatively easy to get to: 75% of Americans live within 5 miles of a Dollar General, according to the company.

The industry’s largest player is flourishing. Todd Vasos, the CEO of Dollar General, told investors in August that his company’s quarterly sales had increased 5% over the same period last year. Some of that growth, he said, came from middle- and higher-income shoppers tightening their belts. But the company’s low-income “core customers” were spending more at the chain too.

a man holding a bag
Chris Outlaw walks to his car after leaving Family Dollar’s King Street location in Windsor, North Carolina. Photograph: Cornell Watson/The Guardian

Those customers have been the industry’s niche from the beginning. When a 48-year-old former tobacco farmer and traveling salesman named James Luther Turner opened JL Turner and Son Wholesale Dry Goods, Shoes, Notions and Hosiery in Scottsville, Kentucky, in 1939, his mission was “to sell the cheap stuff to the poor folks”. (Someone else had cornered the market on “selling the good stuff” to Scottsville’s rich folks.)

By 1955, Turner and his eldest son, Hurley Calister “Cal” Turner Sr, were overseeing 36 stores in small southern towns. Cal Sr decided that year to co-opt the “Dollar Days” sales at big department stores and to open outlets featuring a single low price of $1. Adopting a name that nodded to the general store, he designed a bold black-and-yellow sign and that June christened the first Dollar General in Springfield, Kentucky.

Dollar General now operates over 20,000 stores in 48 states – more than any other retailer of any kind in the US. (It has long since abandoned its $1 price limit.) Though it has more than 195,000 employees and net sales of $40.6bn, the company still calls itself “America’s neighborhood general store”.

Family Dollar began in 1959 in Charlotte, North Carolina, and now operates 8,000 stores nationwide. For most of the past decade, it was owned by yet another chain, Dollar Tree, but the two brands divorced last summer.

What Dollar General and Family Dollar have in common is a conspicuous presence in places that don’t offer a lot of other retail: low-income urban neighborhoods and rural towns like Windsor.

A predominantly Black county seat of 3,400 on North Carolina’s coastal plain, Windsor used to be a retail hub. “All the streets were full on a weekend,” recalled Russell Parker, a 66-year-old retired pilot. “There were people everywhere, people playing music.” And people spending money: at the fish market, the cobbler, the independent groceries, the automotive-supply store. But today Windsor’s downtown – like many rural main streets – is pocked with empty storefronts. The town never fully recovered from Hurricane Floyd, in 1999. “Every young person that graduates from high school gets on the first thing smokin’ to somewhere else,” Parker said.

a person on a motorcycle
The King Street area of downtown Windsor, North Carolina. Photograph: Cornell Watson/The Guardian

One supermarket remains on the edge of town. Shopping for clothes often means driving to the next county, at least for those who drive. But Windsor does have three stores that help fill the gap: a Dollar General and two Family Dollars.

At the Family Dollar that failed multiple inspections, some regulars remain vigilant. Chris Outlaw, a 54-year-old hemodialysis technician, shops there because it’s near his house and workplace. Experience has taught him to buy only a few items at once and to examine his receipts. Not all his neighbors do the same. “I’ve seen people in there with baskets full,” he said. “You can just imagine how much of that stuff didn’t ring out right, and they had so much they couldn’t catch it.”

‘Big old savings’

Customers walking into Dollar General stores are often greeted by a bright yellow sign blaring “Hello, Low Prices”– and by as many as 10,000 items cramming shelves and, often, cluttering the aisles.

“They will send you more than what you need of any product,” said Stephanie, a former lead sales associate in Louisiana. “Your shelf can only hold 10 Glade air fresheners, right? But they will send you 50.”

Rarely is there enough staffing, current and former employees say, to complete all of the tasks expected of them, including stocking shelves, ringing up sales, looking out for shoplifters, mopping floors – and updating price changes and sales stickers.

a person looking at a shelf
Chris Outlaw squeezes through an aisle packed with merchandise inside Family Dollar’s King Street location in Windsor, North Carolina. Photograph: Cornell Watson/The Guardian

More than two dozen current and former employees of the chain in 15 states interviewed by the Guardian agreed that price discrepancies are the byproduct of the company’s employment policies. (Most, including Stephanie, spoke on the condition of anonymity because of fear of retaliation.)

Often there are only one or two people on duty. “You’re lucky if you get to work two to four hours of your eight- to 13-hour shift with another human being,” a former assistant manager in Illinois said.

Every Tuesday, employees are supposed to print and post hundreds of shelf stickers representing price changes already updated in the computer system. On Saturdays, stacks of sales stickers arrive; often, workers are expected to remove all the previous week’s stickers by 5pm and put up new stickers – as many as 1,000 of them – before closing up that night. Stickers fail to get put up, they fall off easily, and they are confusing, with some sales instant and others linked to coupons. “I threw away tags sometimes, to keep me or a co-worker out of trouble,” Stephanie admitted.

blankets and a bleach bottle on a shelf
Items on shelves at the Mineville, New York, Dollar General, which is 5 miles from the Port Henry Dollar General. Photograph: Kelly Burgess/The Guardian

A former store manager at a Dollar General in Connecticut noted that many of his customers were poor or disabled enough that they got by on public assistance. “I didn’t want people to get screwed over, but I knew that it was happening,” he said. “If I’m in the store, I’m gonna try to do the best I can for them. But at the end of the day, they’re still probably gonna get overcharged for a few things.”

Dollar General, in its statement, said it schedules time each week for “price change execution”, among other measures to ensure accuracy.

Ten current and former employees in eight states claimed that – along with allowing pricing errors caused by understaffing and overstocking – some Dollar General stores engage in a tactic designed to fool customers: special sales that don’t actually lower the price of an item. A manager from Florida, for example, sent the Guardian two photos of price stickers for Café Bustelo ground coffee. In the first photo, a sticker said “SALE” in white block letters against a red background. It advertised a markdown from $7.95 to $6.50. In the second photo, the top sticker had been peeled away to show the original price: $6.50.

A sales associate from Illinois sent photos showing cutlery with what he said was a fake original price of $8.50. “It’s trying to say that you’re making this big old savings by buying this item here,” explained the employee, “when it’s actually always been $6.95.”

Dollar General declined to comment on these workers’ claims.

‘We have little choice’

When the Ohio attorney general, Dave Yost, sued Dollar General in 2022, he submitted 114 pages of customer complaints as part of the case.

One of them came from Melanie Hutzler, who lives in Canton without a car and whose mobility is limited by arthritis and multiple sclerosis. Hutzler, 51, relies on government food assistance and said she was cautious about spending money. At the time of her complaint, she could reach two food stores on foot. Getting to the Save A Lot grocery required crossing a busy road, but getting to a Dollar General did not.

“Every single time we went into that store, something would ring up wrong,” she told the Guardian. “They never had a manager there that would fix the prices.” Hutzler said she would walk the cashier over to the shelf and point out the listed price, only to be told, “There’s nothing we can do about it.”

an exterior of a store with cars parked in front
The exterior of Family Dollar on King Street in Windsor, North Carolina. Photograph: Cornell Watson/The Guardian

Other Ohioans expressed similar frustrations. “My 87-year-old mother and I have frequented Dollar General for years, and there have been innumerable times we have made purchases that were well higher than advertised,” wrote Robert Hevlin of Dayton. “My mother and I have literally lost thousands over the years with this company, but both of us being on social security, we have little choice in where we shop.”

In September 2023, Yost reached a $1m settlement with Dollar General, which he said had error rates at some stores that ran as high as 88%. In February 2024, he announced a $400,000 settlement with Family Dollar to resolve similar allegations. Most of that money went to charitable organizations that distribute food and personal-care items.

Both chains agreed in the settlements to tighten their pricing practices. Yost’s office continues to receive complaints. A Dollar General customer in Garfield Heights said in February that he was charged $6.35 for a carton of eggs with a shelf sticker of $5.10, but the “cashier was too busy having a personal call on her cellphone to address the price discrepancy”. The same month, a Family Dollar shopper in Genoa reported being charged $2.65 for cough medicine listed on the shelf at $1.50. “I was told by the cashier that there was nothing that could be done about it,” the complaint said.

Over in Missouri, state officials are pursuing a lawsuit that accuses Dollar General of “deceptive” pricing practices. The suit, filed in 2023, says 92 of the 147 stores the state checked failed their inspections, with discrepancies as high as $6.50 an item.

The companies declined to comment on these state lawsuits.

Dollar General has also been hit with private lawsuits, including several filed by its shareholders. In a document filed in August in federal court in Nashville, lawyers for Dollar General investors argued that understaffing, poor inventory control and overcharging were all interrelated.

The investors allege that the company deceived them by portraying itself as financially sound. In truth, the court filing says, “Dollar General’s inventory management processes were broken, which caused a massive bloat of excess product to clog the company at both its distribution centers and stores, and its workforce had been slashed.” These problems gave rise to price discrepancies and other “dire consequences”, the court filing asserts.

The filing includes the stories of 36 former employees who claimed direct knowledge that Dollar General managers and executives knew about the problems. Several reported notifying the top leadership directly. “All the prices were off in the stores,” said one of those ex-employees, a manager who monitored inventory levels in Ohio and Pennsylvania. She claimed to know firsthand, based on calls she participated in, that company vice-presidents and regional directors were aware of the “huge” price mismatches.

Price stickers and merchandise on shelves
Price tags and merchandise inside Family Dollar’s King Street location in Windsor, North Carolina. Photograph: Cornell Watson/The Guardian

Dollar General, in response, said that the testimony of a handful of ex-workers does not prove that it misled investors. In their “years-long search for fraud”, the company’s lawyers claimed, the shareholders “came up empty”.

Earlier this year, a federal judge in New Jersey halted a class-action lawsuit against Dollar General filed by a shopper who said he was overcharged for groceries. Dollar General argued that when customers create accounts – for example, by downloading the company’s mobile app – they agree to use arbitration to resolve disputes and forfeit the right to file class-action suits. The judge agreed.

This victory for Dollar General threw up an obstacle for customers seeking justice. “Who’s going to bring a consumer arbitration with a $225 filing fee over a 50-cent overcharge?” asked Marc Dann, a former Ohio attorney general whose law firm filed the New Jersey case. “They’ve essentially closed the door to the courthouse to people.”

Dann’s firm did reach a settlement with Dollar General in another case this fall, though the details have not been made public.

‘This endless cycle’

The dollar-store chains describe themselves as mission-driven companies. “Our stores are conveniently located in neighborhoods, and often in ‘food deserts’ where other stores choose not to locate,” Family Dollar says on its website . Dollar General takes pride in offering value to families who, according to CEO Vasos, “have had to sacrifice even on the necessities”.

The industry’s critics say the cause and effect are reversed. “Dollar stores are often seen as a symptom of economic distress,” said the Institute for Local Self-Reliance’s co-executive director, Stacy Mitchell. “What we found is that they’re, in fact, a cause of it.” Sometimes, she said, a chain dollar store will open near an independent grocer and skim off enough of its business that it is forced to close. That limits the availability of fresh produce and forces shoppers to buy more packaged and processed foods.

In a statement, Dollar General said its stores often “operate along with local grocers and business owners to collectively meet customers’ needs”. It added that 7,000 of its 20,000 stores sell fresh produce and that the company also partners with local food banks “to further help nourish our neighbors in need”.

The people enduring the effects of hollowed-out local economies – and getting hit with overcharges at dollar-store chains – include residents of Essex county, New York. The county, tucked among the stately pines of the Adirondack Mountains, has a population of 37,000. It has five Dollar Generals and two Family Dollars. All seven regularly fail pricing-accuracy tests. The Dollar General in Port Henry, which sits on the shores of Lake Champlain, was fined $103,550 for failed inspections between November 2022 and June 2025.

Katelyn Miller at her home in Port Henry, New York, on 24 November.
Katelyn Miller at her home in Port Henry, New York, on 24 November. Photograph: Kelly Burgess/The Guardian

Over the course of seven inspections, 279 out of 700 tested items were overcharges – a combined error rate of just under 40%. One inspection yielded a 78% error rate, including overcharges on Flintstones vitamins, Peter Pan peanut butter and Prego pasta sauce.

The Port Henry store is 5 miles from the Mineville Dollar General, which occupies a lonely stretch of country road across from an auto-repair shop with spare parts littering its lawn. Down the block, an abandoned church presides over a stretch of grass that looks like it hasn’t been mown for years.

Aside from a whiskey warehousing operation and a health center, opportunities for employment are limited. The high-security prison built atop the iron mine for which Mineville is named closed in 2022, taking 100 jobs with it.

The local playground is littered with trash, cigarette butts and the occasional syringe. The town “is nice from the outside”, said Katelyn Miller, a 26-year-old Port Henry resident who lives with her mother, six-year-old daughter and two-year-old son. But “you hear about a lot of crack-den places, like blowing up or getting busted.’” Drug use is rampant in the county, which is 92% white. “Everybody around here seems to be on pain meds or buying someone else’s, because they’re also working themselves to death.”

When it comes to grocery shopping near Miller’s home, the choice is between the two Dollar Generals and a gas station/convenience store. “We live in a food desert,” she said, “even though you would think living in all this farmland, we would have more access.”

overgrown grass in front of church
An abandoned church sits next door to the Mineville, New York, Dollar General store. Photograph: Kelly Burgess/The Guardian

There is a Walmart 30 minutes away, in Fort Ticonderoga. Miller said she recently bought salmon there only to arrive home and discover that the $20 piece of fish had gone bad. “So I had to go to Dollar General and get the Stouffer’s,” she said, adding that she feels “caught in this endless cycle of never having food that will nourish me and my family, and instead having to get 2,000 grams of sodium because at least it has meat”.

The region’s economic straits put regulators in a bind when it comes to overcharges. Daniel Woods, the county’s director of weights and measures, said in 2023 that he didn’t always assess the full penalty on violators. “We’re not trying to put people out of business,” he told a local newspaper. “In some towns that’s their [only] store. I don’t want to pull that away from people, but at the same time, I’m trying to fix the problem.”

On the way out

When Coffield, the North Carolina inspector, visited the Windsor Family Dollar in April 2023, the pricing issues seemed to have abated. Of the 300 items he scanned, he only found five overcharges: incontinence pads, laundry sanitizer, two coffee products and, again, Red Baron pizza. With an error rate below the state’s 2% threshold, the store passed its inspection, and it did so again in November 2024.

But customers still reported problems. Chris Outlaw, the hemodialysis technician, stopped by the Family Dollar earlier this year and noticed a sale: a $1.25 savings on five bags of Cheez Doodles. He bought them but discovered on the way out that he had been charged the regular price. The manager refused to refund the difference, Outlaw said, because he had already walked through the exit door.

Another time, he saw some discounted socks near the counter that he thought would make good Christmas gifts. “I was like, ‘Oh, I like these socks, so I’ll probably give them to somebody,’” he recalled. “Nice, plushy socks.” But they rang up at a higher price, so he left the store without them.

a person outside of a family dollar store
Chris Outlaw looks at his receipt after leaving Family Dollar’s King Street location in Windsor, North Carolina. Photograph: Cornell Watson/The Guardian

During a visit in August, a Guardian reporter found the Windsor Family Dollar closed for much of the afternoon. “Be Back Soon!” read a handwritten sign taped to the door. Two waiting customers said that they frequently paid prices higher than the shelf listing, including a cook whose nearby restaurant buys some of its ingredients there. “It is aggravating,” she said. “Very aggravating.”

Workers reopened the doors after a few hours. Inside, carts of unshelved dog food and other merchandise blocked the aisles. The Guardian compared the prices of 15 items. Two of them rang up higher than advertised, including a frying pan set that was $10 on the shelf and $12 at the register. Though the cashier offered to honor the lower prices, that was still an error rate of 13% – more than six times the state’s standard.

How Australia became the testing ground for a social media ban for young people

Guardian
www.theguardian.com
2025-12-07 14:00:29
From nascent policy idea in one state to passing federal parliament in just days, it’s been a whirlwind journey for the world-first legislation that will take effect from 10 December In late 2023, the South Australian premier’s wife put down a book she had been reading. It was Jonathan Haidt’s The A...
Original Article

In late 2023, the South Australian premier’s wife put down a book she had been reading. It was Jonathan Haidt’s The Anxious Generation.

“[She] said to me you better bloody do something about this ... and then we got to work,” Peter Malinauskas later recalled in an ABC interview.

An American social psychologist, Haidt prescribed a social media ban for those aged under 16 as the solution to the mental health ills he believes are caused by the platforms.

In Australia he found a willing test subject.

A bumper sticker solution?

The ban was considered first by the states. South Australia commissioned a review and then held a summit on the subject in partnership with New South Wales.

Facebook whistleblower Frances Haugen spoke at day one of the summit, which was held in NSW. In emails obtained by Crikey under freedom of information, the South Australian government wasn’t as keen to hear from Haugen since she had described a ban as a “bumper sticker solution”.

Sign up: AU Breaking News email

Haidt spoke via video link on day two of the summit, which was held in South Australia, saying he was “thrilled” with the potential for a ban.

“We need to free kids from these traps by raising the age for opening social media to 16.”

And so the campaign for a ban began.

Following the summit, the federal government faced pressure to implement a national ban rather than having a patchwork of states implementing their own regulations .

Less than a year out from the federal election, the then opposition leader, Peter Dutton, made it a signature policy for the Coalition .

News Corp went all in, launching the “Let Them Be Kids” campaign, which coincided with Meta’s announcement it would not enter into new deals to pay media companies for news content. Front pages advocating for the ban pushed things along.

The prime minister, Anthony Albanese, and Nova radio host Wippa launched a campaign titled “36 months”, advocating for the age to be raised from 13 to 16. Albanese appeared on Wippa’s program at least five times in the past two years.

The Labor government never publicly confirmed it, but its adoption of the idea and rush to pass the law before parliament ended for the year in December 2024 was seen as a bid to take potential election issues off the table .

In Albanese’s telling, the policy was designed to leave the responsibility for keeping kids off social media to the government. He framed the policy as a bid to get kids off devices and on to the footy fields and netball courts.

The countdown is on

After the legislation was introduced, it was passed by the parliament in a matter of days, while a committee barely reviewed the bill.

The law pushed decisions about which platforms were covered and how the ban would work to the end of 2025. It placed the responsibility for enforcement of the ban on the platforms themselves.

The then communications minister, Michelle Rowland, said YouTube would have an exemption on education grounds, but that was not defined in the legislation.

A $22.5m technology trial run by a UK firm associated with age assurance providers got under way, with the deadline set for after the federal election. The top line finding that it was workable was emphasised by the government after its release, without closer examination of some of the shortfalls .

After the Albanese government was returned to power in the May election with an even bigger majority, Anika Wells was appointed the new communications minister.

TikTok and Meta were not happy that YouTube had been excluded from the ban. YouTube Shorts is very similar to Reels and TikTok’s short-form videos, and the platforms couldn’t see why YouTube was given a broad exemption for a similar product.

The eSafety commissioner advised the minister in July that it was her view YouTube should not be excluded from the ban, pointing to the algorithms used to determine the types of videos promoted to teens on the platform and the harms reportedly encountered on the service. Wells agreed .

Google threatened legal action and cancelled a planned showcase at Parliament House.

Ultimately the eSafety commissioner decided the ban should cover: TikTok, Facebook, Instagram (and, as a result of the account structure, Threads), X, Snapchat, YouTube, Reddit, Kick and Twitch. Other platforms could be added later, as determined by the government.

With the countdown on for the ban to take effect, Meta, TikTok, Snapchat, Reddit, Twitch and Kick have all said they will comply with the ban.

A high court challenge against the ban was lodged but the hearing has been delayed until February.

News Corp has done a victory lap for the ban, with News Corp Australia’s executive chair, Michael Miller, describing social media platforms as “true monsters” who “torment our children” .

News Corp’s owners, the Murdoch family, are expected to hold a stake in the US version of TikTok.

A robot walks into a bar: can a Melbourne researcher get AI to do comedy?

Guardian
www.theguardian.com
2025-12-07 14:00:23
Machines can be funny when they mistakenly bump into things – but standup is a tough gig even for humans Robots can make humans laugh – mostly when they fall over – but a new research project is looking at whether robots using AI could ever be genuinely funny. If you ask ChatGPT for a funny joke, it...
Original Article

Robots can make humans laugh – mostly when they fall over – but a new research project is looking at whether robots using AI could ever be genuinely funny.

If you ask ChatGPT for a funny joke, it will serve you up something that belongs in a Christmas cracker: “Why don’t skeletons fight each other? Because they don’t have the guts.”

The University of Melbourne’s Dr Robert Walton, a dean’s research fellow in the faculty of fine arts and music, is taking a different approach to working out whether robots can do comedy.

Thanks to an Australian Research Council grant of about $500,000, he will train a swarm of robots in standup. And, at least in the beginning, they won’t use words.

“Robots are good at making people laugh … they are humorous because they break and they bump into things, and so we’re laughing at them,” Walton says.

“However, when they try to do something funny on purpose, it ain’t so funny any more. We don’t laugh at them because we really, deep down, don’t believe that they can be funny.”

Saturday Night Live’s Tina Fey said exactly that at this year’s Edinburgh comedy festival. AI is “unable to be funny”, she said.

But what Walton is looking at is not AI based on text or large language models.

He is going to start with non-verbal communication, something that has to be performed rather than written. The fundamentals of comedy, he says, are timing, reading the room, the connection with the audience, along with physical comedy such as clowning.

So his ensemble of about 10 robots – which will not be androids but ground vehicles between 40cm and 2 metres tall – will work with humans to learn how to be funny visually in the first instance.

Dr Robert Walton, dean’s research fellow in the faculty of fine arts and music at University of Melbourne.
Dr Robert Walton, dean’s research fellow in the faculty of fine arts and music at University of Melbourne. Photograph: Charlie Kinross/The Guardian

They’ll sense movement, the way a head tilts, or when someone laughs.

“We’re giving these systems more senses, like human senses … giving them ears, not just listening for words but for things like the gaps in between words, the rhythms of things,” he says.

He likens them to babies who don’t yet know how to make sense of the inputs.

“That’s partly what we’re trying to do with machine learning and AI – giving it more ways to sense and more ways to build a more holistic understanding of what it means to be in the world,” he says.

“It is in standup comedy, really, that the connection between the robot and the audience is so clear, and there’s so much feedback going on.”

Asked if eventually they will add voices, Walton says “potentially”. “Depends how we go,” he adds.

There is a tension here, as the performance industry is just one of those where jobs are threatened by AI, and AI steals creative content .

skip past newsletter promotion

Walton’s project is not about creating robots that will take over comedy festivals, though, but about investigating whether believable comedy is something robots can be taught, to better understand how machines might use both humour and manipulation, and to better understand human-robot interactions and their risks and benefits.

A paradox at the heart of his work, Walton says, is that humour can be used to disarm a situation but can also be used coercively.

He says it might be interesting for comedians to work with robots with comedic timing, but the same techniques could be used, for example, by care robots that can learn to say the right thing at the right time to cheer people up.

Robots run, punch and score at World Humanoid Robot Games in China – video

“But while I’m looking into this work of building belief in comedy performance by machines, I’ve got this other eye on what does it mean, and how might this be used coercively?” he says.

Many doubt whether that first step, making robots funny, is possible.

At this year’s G’Day USA arts gala, Australian comedian and polymath Tim Minchin told the crowd that humans are interested in “the agency of their fellow human behind the art, struggling, striving, making choices and errors”. “AI might come for the perfectible stuff but never for our flaws,” he says.

“Our flaws are our humanity.”

The director of the Melbourne comedy festival, Susan Provan, says what makes comedy enjoyable is “the authentic human originality”.

“A performer is bringing something only they can bring, because they are bringing their individual lived experience to the material,” she says.

“What’s funny is something that comes from a moment, a magic moment, a pause, an interaction with an audience member, an idea that connects or doesn’t connect.

“You’d be laughing at the robot stuffing up. That’s what would be funny.”

Goodbye, Microsoft: Schleswig-Holstein Relies on Open Source and Saves Millions

Hacker News
www.heise.de
2025-12-07 13:21:24
Comments...
Original Article

The state administration of Schleswig-Holstein is making a remarkable U-turn in its IT strategy and consistently relying on open source . After the migration from proprietary Microsoft software to free solutions was initially accompanied by problems and criticism , Digitalization Minister Dirk Schrödter (CDU) can now report a significant success: According to his ministry, the state will save over 15 million euros in license costs for Windows, Microsoft Office & Co. next year alone. It is expected to be similar in the following years.

In contrast, there would be one-time investments of nine million euros in 2026, explained the Ministry of Digitalization to the Kieler Nachrichten . These would have to be made for the conversion of workplaces and the further development of solutions with free software in the next 12 months. Given the annual savings, this sum will pay for itself in less than a year. In the past, the state transferred millions to the US company Microsoft, primarily for the use of office software and other programs.

The department sees the departure from this "vendor lock-in" – the dependence on a single large provider – as a clear signal for greater independence and sustainable digitalization. The financial incentive now underscores that digital sovereignty can be not only a political buzzword but also an economic gain.

Almost 80 percent of licenses canceled

The numbers speak for themselves: outside the tax administration, almost 80 percent of workplaces in the state administration have already been switched to the open-source office software LibreOffice. Schrödter thus confirms a course that reduces technical and economic dependence on individual manufacturers. The consequence of the conversion was already evident recently, as Schrödter emphasized in an interview with c't . Regarding the status of Microsoft license cancellations, he said: "We are at almost 80, without the tax administration." For tax matters, the state finance ministers have "given themselves a clear timetable for the switch." Recently, the Christian Democrat also emphasized, according to the Südtiroler Wirtschaftszeitung, that the state has entered a marathon, not just a sprint.

The remaining 20 percent of workplaces are currently still dependent on Microsoft programs such as Word or Excel, as there is a technical dependency on these programs in certain specialized applications. According to Schrödter, however, the successive conversion of these remaining computers is the stated goal.

Opposition sees challenges

Despite the savings and the almost completed migration in large parts of the administration, the opposition continues to criticize the quality of the conversion. SPD state parliament member Kianusch Stender pointed out to the Kieler Nachrichten: "It may be that on paper 80 percent of workplaces have been converted. But far fewer than 80 percent of employees can now work with them properly." Errors in the migration are "still present." The initial difficulties in introducing the open-source programs have apparently led to ongoing frustration among some employees in certain areas.

The Green state parliament member Jan Kürschner also admitted in an interview with heise online that such a comprehensive conversion would not go without friction. But he emphasized the long-term nature of the project and the necessity of fundamentally rethinking administrative processes: "With the change, there is an opportunity to truly rethink the administration and free ourselves from old burdens. That is the great added value." If only a one-to-one conversion is made, it might certainly "stumble at one point or another." But those who truly optimize administrative processes will likely find in the end: "Open source is the better way."

The challenge now is to resolve the initial migration problems and acceptance difficulties and to further develop the open-source solutions so that they fully meet the requirements of a modern state administration. The savings achieved give Schleswig-Holstein more financial leeway for this.

( nie )

Don't miss any news – follow us on Facebook , LinkedIn or Mastodon .

This article was originally published in German . It was translated with technical assistance and editorially reviewed before publication.

At least 50 hallucinated citations found in ICLR 2026 submissions

Hacker News
gptzero.me
2025-12-07 13:16:26
Comments...
Original Article
Title Average Review Rating Paper Link Citation Check Scan Link Example of Verified Hallucination Comment TamperTok: Forensics-Driven Tokenized Autoregressive Framework for Image Tampering Localization 8.0 TamperTok: Forensics-Driven Tokenized Autoregressive Framework for Image Tampering Localization | OpenReview https://app.gptzero.me/documents/4645494f-70eb-40bb-aea7-0007e13f7179/share Chong Zou, Zhipeng Wang, Ziyu Li, Nan Wu, Yuling Cai, Shan Shi, Jiawei Wei, Xia Sun, Jian Wang, and Yizhou Wang. Segment everything everywhere all at once. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. This paper exists, but all authors are wrong. MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive Text Sources 8.0 MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive Text Sources | OpenReview https://app.gptzero.me/documents/bfd10666-ea2d-454c-9ab2-75faa8b84281/share Dan Hendrycks, Collin Burns, Steven Basart, Andy Critch, Jerry Li, Dawn Ippolito, Aina Lapedriza, Florian Tramer, Rylan Macfarlane, Eric Jiang, et al. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. The paper and first 3 authors match. The last 7 authors are not on the paper, and some of them do not exist Catch-Only-One: Non-Transferable Examples for Model-Specific Authorization 6.0 Catch-Only-One: Non-Transferable Examples for Model-Specific Authorization | OpenReview https://app.gptzero.me/documents/9afb1d51-c5c8-48f2-9b75-250d95062521/share Dinghuai Zhang, Yang Song, Inderjit Dhillon, and Eric Xing. Defense against adversarial attacks using spectral regularization. In International Conference on Learning Representations (ICLR), 2020. No Match OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features 6.0 OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features | OpenReview https://app.gptzero.me/documents/e3f155d7-067a-4720-adf8-65dc9dc714b9/share Robert Huben, Logan Riggs, Aidan Ewart, Hoagy Cunningham, and Lee Sharkey. Sparse autoencoders can interpret randomly initialized transformers, 2025. URL https://arxiv.org/ abs/2501.17727. This paper exists, but all authors are wrong. Principled Policy Optimization for LLMs via Self-Normalized Importance Sampling 5.0 Principled Policy Optimization for LLMs via Self-Normalized Importance Sampling | OpenReview https://app.gptzero.me/documents/54c8aa45-c97d-48fc-b9d0-d491d54df8d3/share David Rein, Stas Gaskin, Lajanugen Logeswaran, Adva Wolf, Oded teht sun, Jackson H. He, Divyansh Kaushik, Chitta Baral, Yair Carmon, Vered Shwartz, Sang-Woo Lee, Yoav Goldberg, C. J. H. un, Swaroop Mishra, and Daniel Khashabi. Gpqa: A graduate-level google-proof q\&a benchmark, 2023 All authors except the first are fabricated. PDMBench: A Standardized Platform for Predictive Maintenance Research 4.5 PDMBench: A Standardized Platform for Predictive Maintenance Research | OpenReview https://app.gptzero.me/documents/5c55afe7-1689-480d-ac44-9502dc0f9229/share Andrew Chen, Andy Chow, Aaron Davidson, Arjun DCunha, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Clemens Mewald, Siddharth Murching, Tomas Nykodym, et al. Mlflow: A platform for managing the machine learning lifecycle. In Proceedings of the Fourth International Workshop on Data Management for End-to-End Machine Learning, pp. 1-4. ACM, 2018. Authors and conference match this paper , but title is somewhat different and the year is wrong. IMPQ: Interaction-Aware Layerwise Mixed Precision Quantization for LLMs 4.5 IMPQ: Interaction-Aware Layerwise Mixed Precision Quantization for LLMs | OpenReview https://app.gptzero.me/documents/5461eefd-891e-4100-ba1c-e5419af520c0/share Chen Zhu et al. A survey on efficient deployment of large language models. arXiv preprint arXiv:2307.03744, 2023. The arXiv ID is real, but the paper has different authors and a different title. C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detection 4.5 C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detection | OpenReview https://app.gptzero.me/documents/c07521cd-2757-40a2-8dc1-41382d7eb11b/share K. Marino, R. Salakhutdinov, and A. Gupta. Fine-grained image classification with learnable semantic parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4500-4509, 2019. Authors and subject match this paper TopoMHC: Sequence–Topology Fusion for MHC Binding 4.5 TopoMHC: Sequence–Topology Fusion for MHC Binding | OpenReview https://app.gptzero.me/documents/8da4f86c-00d8-4d73-81dd-c168c0bfdf4e/share Yuchen Han, Yohan Kim, Dalibor Petrovic, Alessandro Sette, Morten Nielsen, and Bjoern Peters. Deepligand: a deep learning framework for peptide-mhc binding prediction. Bioinformatics, 39 (1):btac834, 2023. doi: 10.1093/bioinformatics/btac834. No Match Can Text-to-Video Models Generate Realistic Human Motion? 4.5 Can Text-to-Video Models Generate Realistic Human Motion? | OpenReview https://app.gptzero.me/documents/f52aad2d-2253-44bf-80ba-8e8668df650f/share Yugandhar Balaji, Jianwei Yang, Zhen Xu, Menglei Chai, Zhoutong Xu, Ersin Yumer, Greg Shakhnarovich, and Deva Ramanan. Conditional gan with discriminative filter generation for text-to-video synthesis. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pp. 2155-2161, July 2019. doi: 10.24963/ijcai.2019/276. This paper exists , but the authors and page numbers are wrong. GRF-LLM: Environment-Aware Wireless Channel Modeling via LLM-Guided 3D Gaussians 4.0 GRF-LLM: Environment-Aware Wireless Channel Modeling via LLM-Guided 3D Gaussians | OpenReview https://app.gptzero.me/documents/c3e66b9c-20b4-4c50-b881-e40aba2a514f/share Junting Chen, Yong Zeng, and Rui Zhang. Rfcanvas: A radio frequency canvas for wireless network design. In IEEE International Conference on Communications, pp. 1-6, 2024.
Title partially matches this article . Listwise Generalized Preference Optimization with Process-aware Signals for LLM Reasoning 4.0 Listwise Generalized Preference Optimization with Process-aware Signals for LLM Reasoning | OpenReview https://app.gptzero.me/documents/bbeecf1c-189a-4311-999b-617aab686ea9/share Kaixuan Zhou, Jiaqi Liu, Yiding Wang, and James Zou. Generalized direct preference optimization. arXiv preprint arXiv:2402.05015, 2024. No Match IUT-Plug: A Plug-in tool for Interleaved Image-Text Generation 4.0 IUT-Plug: A Plug-in tool for Interleaved Image-Text Generation | OpenReview https://app.gptzero.me/documents/0f12d2fc-403b-4859-8d00-f75fd9f56e39/share Yash Goyal, Anamay Mohapatra, Nihar Kwatra, and Pawan Goyal. A benchmark for compositional text-to-image synthesis. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. This paper exists , but the authors are all wrong. Resolving the Security-Auditability Dilemma with Auditable Latent Chain-of-Thought 4.0 Resolving the Security-Auditability Dilemma with Auditable Latent Chain-of-Thought | OpenReview https://app.gptzero.me/documents/5cee5c3a-5e75-4063-a054-1e934a071705/share Yixiang Ma, Ziyi Liu, Zhaoyu Wang, Zhaofeng Xu, Yitao Wang, and Yang Liu. Safechain: A framework for securely executing complex commands using large language models. arXiv preprint arXiv:2402.16521, 2024a. No match; although this paper is closely related. ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks 4.0 ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks | OpenReview https://app.gptzero.me/documents/f3441445-5401-48e9-9617-09a635992ff9/share Yunzhu Yang, Shuang Li, and Jiajun Wu. MM-ReAct: Prompting chatgpt to multi-modal chain-ofthought reasoning. arXiv preprint arXiv:2401.04740, 2024. No Match Taming the Judge: Deconflicting AI Feedback for Stable Reinforcement Learning 3.5 Taming the Judge: Deconflicting AI Feedback for Stable Reinforcement Learning | OpenReview https://app.gptzero.me/documents/80c64df2-eee6-41aa-90cc-3f835b128747/share Chenglong Wang, Yang Liu, Zhihong Xu, Ruochen Zhang, Jiahao Wu, Tao Luo, Jingang Li, Xunliang Liu, Weiran Qi, Yujiu Yang, et al. Gram-r ${ }^{8}$ : Self-training generative foundation reward models for reward reasoning. arXiv preprint arXiv:2509.02492, 2025b. All authors except the first are fabricated and the title is altered. DANCE-ST: Why Trustworthy AI Needs Constraint Guidance, Not Constraint Penalties 3.5 DANCE-ST: Why Trustworthy AI Needs Constraint Guidance, Not Constraint Penalties | OpenReview https://app.gptzero.me/documents/3ebd71b4-560d-4fa3-a0d3-ed2fa13c519f/share Sardar Asif, Saad Ghayas, Waqar Ahmad, and Faisal Aadil. Atcn: an attention-based temporal convolutional network for remaining useful life prediction. The Journal of Supercomputing, 78(1): $1-19,2022$. Two papers with similar titles exist here and here , but the authors, journal, and date do not match. Federated Hierarchical Anti-Forgetting Framework for Class-Incremental Learning with Large Pre-Trained Models 3.33 Federated Hierarchical Anti-Forgetting Framework for Class-Incremental Learning with Large Pre-Trained Models | OpenReview https://app.gptzero.me/documents/ae10437b-c65b-455b-ad22-918742a5ed82/share Arslan Chaudhry, Arun Mallya, and Abhinav Srivastava. Fedclassil: A benchmark for classincremental federated learning. In NeurIPS, 2023. No Match Chain-of-Influence: Tracing Interdependencies Across Time and Features in Clinical Predictive Modeling 3.33 Chain-of-Influence: Tracing Interdependencies Across Time and Features in Clinical Predictive Modeling | OpenReview https://app.gptzero.me/documents/dff2c063-6986-4241-8c20-4327a39d4d4b/share Ishita et al. Bardhan. Icu length-of-stay prediction with interaction-based explanations. Journal of Biomedical Informatics, 144:104490, 2024. No Match TRACEALIGN - Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs 3.33 TRACEALIGN - Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs | OpenReview https://app.gptzero.me/documents/4b379aba-8d8a-427b-ac67-d13af5eda8c9/share Lisa Feldman Barrett. Emotions are constructed: How brains make meaning. Current Directions in Psychological Science, 25(6):403-408, 2016. This article is similar, but the title, and metadata are different. MEMORIA: A Large Language Model, Instruction Data and Evaluation Benchmark for Intangible Cultural Heritage 3.33 MEMORIA: A Large Language Model, Instruction Data and Evaluation Benchmark for Intangible Cultural Heritage | OpenReview https://app.gptzero.me/documents/956129a3-11ee-4503-92e3-3ed5db12d2d6/share Yang Cao, Rosa Martinez, and Sarah Thompson. Preserving indigenous languages through neural language models: Challenges and opportunities. Computational Linguistics, 49(3):567-592, 2023. No Match Reflexion: Language Models that Think Twice for Internalized Self-Correction 3.2 Reflexion: Language Models that Think Twice for Internalized Self-Correction | OpenReview https://app.gptzero.me/documents/45f2f68d-df09-4bbf-8513-588fe24f26fa/share Guang-He Xiao, Haolin Wang, and Yong-Feng Zhang. Rethinking uncertainty in llms: A case study on a fact-checking benchmark. arXiv preprint arXiv:2305.11382, 2023. No Match ECAM: Enhancing Causal Reasoning in Foundation Models with Endogenous Causal Attention Mechanism 3.0 ECAM: Enhancing Causal Reasoning in Foundation Models with Endogenous Causal Attention Mechanism | OpenReview https://app.gptzero.me/documents/d99a5552-38e0-459b-8746-4e64069b0640/share Atticus Geiger, Zhengxuan Wu, Yonatan Rozner, Mirac Suzgun Naveh, Anna Nagarajan, Jure Leskovec, Christopher Potts, and Noah D Goodman. Causal interpretation of self-attention in pre-trained transformers. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/642a321fba8a0f03765318e629cb93ea-Paper-Conference.pdf. A paper with this title exists at the given URL, but the authors don't match. MANTA: Cross-Modal Semantic Alignment and Information-Theoretic Optimization for Long-form Multimodal Understanding 3.0 MANTA: Cross-Modal Semantic Alignment and Information-Theoretic Optimization for Long-form Multimodal Understanding | OpenReview https://app.gptzero.me/documents/381ed9a6-b168-4cd0-81ad-1f50139c0737/share Guy Dove. Language as a cognitive tool to imagine goals in curiosity-driven exploration. Nature Communications, 13(1):1-14, 2022. An article with this title exists , but author and publication don't match. LOSI: Improving Multi-agent Reinforcement Learning via Latent Opponent Strategy Identification 3.0 LOSI: Improving Multi-agent Reinforcement Learning via Latent Opponent Strategy Identification | OpenReview https://app.gptzero.me/documents/53e86e4b-a7e2-48d0-976b-240bfc412836/share Jing Liang, Fan Zhou, Shuying Li, Jun Chen, Guandong Zhou, Huaiming Xu, and Xin Li. Learning opponent behavior for robust cooperation in multi-agent reinforcement learning. IEEE Transactions on Cybernetics, 53(12):7527-7540, 2023. No Match The Dynamic Interaction Field Transformer: A Universal, Tokenizer-Free Language Architecture 3.0 The Dynamic Interaction Field Transformer: A Universal, Tokenizer-Free Language Architecture | OpenReview https://app.gptzero.me/documents/80fd90a6-c99e-4c31-af72-0da9e90949f6/share Kaj Bostrom and Greg Durrett. Byte-level representation learning for multi-lingual named entity recognition. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4617-4627, 2020. No Match Strategema: Probabilistic Analysis of Adversarial Multi-Agent Behavior with LLMs in Social Deduction Games 3.0 Strategema: Probabilistic Analysis of Adversarial Multi-Agent Behavior with LLMs in Social Deduction Games | OpenReview https://app.gptzero.me/documents/1155e8a8-f679-4942-8fd9-c47fb64ad967/share Tom Eccles, Jeffrey Tweedale, and Yvette Izza. Let's pretend: A study of negotiation with autonomous agents. In 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), volume 3, pp. 449-452. IEEE, 2009. No Match Understanding Transformer Architecture through Continuous Dynamics: A Partial Differential Equation Perspective 3.0 Understanding Transformer Architecture through Continuous Dynamics: A Partial Differential Equation Perspective | OpenReview https://app.gptzero.me/documents/460a1a23-1a97-482a-9759-ade855a4a0b4/share Zijie J Wang, Yuhao Choi, and Dongyeop Wei. On the identity of the representation learned by pre-trained language models. arXiv preprint arXiv:2109.01819, 2021. No Match Diffusion Aligned Embeddings 2.8 Diffusion Aligned Embeddings | OpenReview https://app.gptzero.me/documents/3d95a003-06c6-4233-881b-03b1e29b4ba2/share Yujia Wang, Hu Huang, Cynthia Rudin, and Yaron Shaposhnik. Pacmap: Dimension reduction using pairwise controlled manifold approximation projection. Machine Learning, 110:559-590, 2021. A similar paper with two matching authors exists , but the other authors, title, and journal are wrong. Leveraging NLLB for Low-Resource Bidirectional Amharic – Afan Oromo Machine Translation 2.5 Leveraging NLLB for Low-Resource Bidirectional Amharic – Afan Oromo Machine Translation | Open Review https://app.gptzero.me/documents/813da6e2-f7e8-4c95-bdd8-7d29b8e4b641/share Atnafa L. Tonja, Gebremedhin Gebremeskel, and Seid M. Yimam. Evaluating machine translation systems for ethiopian languages: A case study of amharic and afan oromo. Journal of Natural Language Engineering, 29(3):456-478, 2023. No Match Certified Robustness Training: Closed-Form Certificates via CROWN 2.5 Certified Robustness Training: Closed-Form Certificates via CROWN | OpenReview https://app.gptzero.me/documents/53b60ef5-2ebf-403e-8123-3a9bb2da0f33/share Huan Zhang, Hongge Chen, Chaowei Xiao, and Bo Zhang. Towards deeper and better certified defenses against adversarial attacks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJgG92A2m No Match Context-Aware Input Switching in Mobile Devices: A Multi-Language, Emoji-Integrated Typing System 2.5 Context-Aware Input Switching in Mobile Devices: A Multi-Language, Emoji-Integrated Typing System | OpenReview https://app.gptzero.me/documents/68998766-49c3-4269-9eca-3b6a76ed68b4/share Ishan Tarunesh, Syama Sundar Picked, Sai Krishna Bhat, and Monojit Choudhury. Machine translation for code-switching: A systematic literature review. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pp. 3654-3670, 2021. Partial match to this article , but authors, title, and metadata is largely wrong. Five-Mode Tucker-LoRA for Video Diffusion on Conv3D Backbones 2.5 Five-Mode Tucker-LoRA for Video Diffusion on Conv3D Backbones | OpenReview https://app.gptzero.me/documents/eb0fd660-ed00-4769-a940-3d093d4f1ec1/share Shengming Chen, Yuxin Wang, et al. Videocrafter: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2305.07932, 2023b. A paper with the same title exists , but the authors and arXiv ID are wrong. Activation-Guided Regularization: Improving Deep Classifiers using Feature-Space Regularization with Dynamic Prototypes 2.5 Activation-Guided Regularization: Improving Deep Classifiers using Feature-Space Regularization with Dynamic Prototypes | OpenReview https://app.gptzero.me/documents/4031111e-24ef-4e06-908e-18ab99b08932/share Wentao Cheng and Tong Zhang. Improving deep learning for classification with unknown label noise. In International Conference on Machine Learning, pp. 6059-6081. PMLR, 2023. A similar paper exists . Sparse-Smooth Decomposition for Nonlinear Industrial Time Series Forecasting 2.5 Sparse-Smooth Decomposition for Nonlinear Industrial Time Series Forecasting | OpenReview https://app.gptzero.me/documents/c01ad49e-a788-4916-a6ee-f43314d14676/share Yutian Chen, Kun Zhang, Jonas Peters, and Bernhard Schölkopf. Causal discovery and inference for nonstationary systems. Journal of Machine Learning Research, 22(103):1-72, 2021. No Match PDE-Transformer: A Continuous Dynamical Systems Approach to Sequence Modeling 2.0 PDE-Transformer: A Continuous Dynamical Systems Approach to Sequence Modeling | OpenReview https://app.gptzero.me/documents/ba257eea-e86c-4276-84c0-08b7465e1e3e/share
Xuechen Li, Juntang Zhuang, Yifan Ding, Zhaozong Jin, Yun chen Chen, and Stefanie Jegelka. Scalable gradients for stochastic differential equations. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS 2020), volume 108 of Proceedings of Machine Learning Research, pp. 3898-3908, 2020. The paper exists and the first author is correct but all other authors and the page range are wrong SAFE-LLM: A Unified Framework for Reliable, Safe, And Secure Evaluation of Large Language Models 2.0 SAFE-LLM: A Unified Framework for Reliable, Safe, And Secure Evaluation of Large Language Models | OpenReview https://app.gptzero.me/documents/05ee7ff4-40e2-48b7-b5bd-8c307d7db669/share Kuhn, J., et al. Semantic Entropy for Hallucination Detection. ACL 2023. A similar paper with different authors can be found here . PIPA: An Agent for Protein Interaction Identification and Perturbation Analysis 2.0 PIPA: An Agent for Protein Interaction Identification and Perturbation Analysis | OpenReview https://app.gptzero.me/documents/5031a806-1271-4fd3-b333-2554f47cb9fa/share Alex Brown et al. Autonomous scientific experimentation at the advanced light source using language-model-driven agents. Nature Communications, 16:7001, 2025. No Match Typed Chain-of-Thought: A Curry-Howard Framework for Verifying LLM Reasoning 2.0 Typed Chain-of-Thought: A Curry-Howard Framework for Verifying LLM Reasoning | OpenReview https://app.gptzero.me/documents/9d2e3239-99db-4712-be7f-e032156d92a5/share DeepMind. Gemma scope: Scaling mechanistic interpretability to chain of thought. DeepMind Safety Blog, 2025. URL https://deepmindsafetyresearch.medium.com/ evaluating-and-monitoring-for-ai-scheming-8a7f2ce087f9. Discusses scaling mechanistic interpretability techniques to chain-of-thought and applications such as hallucination detection. ThA similar URL exists , and the title is similar to this blog . However, no exact match exists. Graph-Based Operator Learning from Limited Data on Irregular Domains 2.0 Graph-Based Operator Learning from Limited Data on Irregular Domains | OpenReview https://app.gptzero.me/documents/6c52217f-fb88-4bd8-85aa-bd546e1fa88c/share Liu, Y., Lütjens, B., Azizzadenesheli, K., and Anandkumar, A. (2022). U-netformer: A u-net style transformer for solving pdes. arXiv preprint arXiv:2206.11832. No Match KARMA: Knowledge-Aware Reward Mechanism Adjustment via Causal AI 2.0 KARMA: Knowledge-Aware Reward Mechanism Adjustment via Causal AI | OpenReview https://app.gptzero.me/documents/92b6492c-68ad-41a3-ae35-628d67f053e0/share Reinaldo A. C. Bianchi, Luis A. Celiberto Jr, and Ramon Lopez de Mantaras. Knowledge-based reinforcement learning: A survey. Journal of Artificial Intelligence Research, 62:215-261, 2018. No Match Microarchitecture Is Destiny: Performance and Accuracy of Quantized LLMs on Consumer Hardware 2.0 Microarchitecture Is Destiny: Performance and Accuracy of Quantized LLMs on Consumer Hardware | OpenReview https://app.gptzero.me/documents/4504a39a-af72-41ab-9679-6f6a017a3275/share Zhihang Jiang, Dingkang Wang, Yao Li, et al. Fp6-llm: Efficient llm serving through fp6-centric co-design. arXiv preprint arXiv:2401.14112, 2024. the arXiv ID corresponds with a very similar paper , but the authors are wrong and the title is altered. Decoupling of Experts: A Knowledge-Driven Architecture for Efficient LLMs 1.6 Decoupling of Experts: A Knowledge-Driven Architecture for Efficient LLMs | OpenReview https://app.gptzero.me/documents/74eade70-da36-4635-8749-5e1d04748b6d/share H Zhang, Y L, X W, Y Z, X Z, H W, X H, K G, Z W, H W, H C, H L, and J W. Matrix data pile: A trillion-tokenscale datasets for llm pre-training. arXiv preprint arXiv:2408.12151, 2024. No Match; arxiv is is unrelated QUART: Agentic Reasoning To Discover Missing Knowledge in Multi-Domain Temporal Data. 1.5 QUART: Agentic Reasoning To Discover Missing Knowledge in Multi-Domain Temporal Data. | OpenReview https://app.gptzero.me/documents/c6f30343-3948-4c07-b7de-6b1407d5daa6/share Meera Jain and Albert Chen. Explainable ai techniques for medical applications: A comprehensive review. AI in Healthcare, 5:22-37, 2024. No Match From Physics-Informed Models to Deep Learning: Reproducible AI Frameworks for Climate Resilience and Policy Alignment 1.5 From Physics-Informed Models to Deep Learning: Reproducible AI Frameworks for Climate Resilience and Policy Alignment | OpenReview https://app.gptzero.me/documents/a7ed6c42-4349-4b45-a356-0e325090e5af/share MIT Climate Group. A cautionary tale for deep learning in climate science. https://example. com, 2019. The title matches this paper, but the citation is obviously hallucinated. A superpersuasive autonomous policy debating system 1.5 A superpersuasive autonomous policy debating system | OpenReview https://app.gptzero.me/documents/b792a4de-baa8-47d4-b880-87b330a482ce/share Roy Bar-Haim, Shachar Bhattacharya, Michal Jacovi, Yosi Mass, Matan Orbach, Eyal Sliwowicz, and Noam Slonim. Key point analysis via contrastive learning and extractive argument summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7953-7962, Online and Punta Cana, Dominican Republic, November 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.629. URL https://aclanthology.org/2021.emnlp-main. 629. A paper with the same title exists , but the authors and URL are wrong. AnveshanaAI: A Multimodal Platform for Adaptive AI/ML Education Through Automated Question Generation and Interactive Assessment 1.5 AnveshanaAI: A Multimodal Platform for Adaptive AI/ML Education Through Automated Question Generation and Interactive Assessment | OpenReview https://app.gptzero.me/documents/720d6d24-2223-4e0e-95b9-6dfce674f8c7/share Shiyang Liu, Hongyi Xu, and Min Chen. Measuring and reducing perplexity in large-scale llms. arXiv preprint arXiv:2309.12345, 2023. No Match AI-Assisted Medical Triage Assistant 1.0 AI-Assisted Medical Triage Assistant | OpenReview https://app.gptzero.me/documents/391b5d76-929a-4f3f-addf-31f6993726f2/share [3] K. Arnold, J. Smith, and A. Doe. Variability in triage decision making. Resuscitation, 85:12341239, 2014. No Match Deciphering Cross-Modal Feature Interactions in Multimodal AIGC Models: A Mechanistic Interpretability Approach 0.67 Deciphering Cross-Modal Feature Interactions in Multimodal AIGC Models: A Mechanistic Interpretability Approach | OpenReview https://app.gptzero.me/documents/d4102812-01c4-45b2-aea8-59e467d31fd4/share Shuyang Basu, Sachin Y Gadre, Ameet Talwalkar, and Zico Kolter. Understanding multimodal llms: the mechanistic interpretability of llava in visual question answering. arXiv preprint arXiv:2411.17346, 2024. A paper with this title exists , but the authors and arXiv ID are wrong. Scalable Generative Modeling of Protein Ligand Trajectories via Graph Neural Diffusion Networks 0.5 Scalable Generative Modeling of Protein Ligand Trajectories via Graph Neural Diffusion Networks | OpenReview https://app.gptzero.me/documents/32d43311-6e69-4b88-be99-682e4eb0c2cc/share E. Brini, G. Jayachandran, and M. Karplus. Coarse-graining biomolecular simulations via statistical learning. J. Chem. Phys., 154:040901, 2021. There is no match for the title and authors, but the journal, volume, and year match this article

It Is Worth It To Optimize Images For Your Site

Lobsters
brainbaking.com
2025-12-07 13:14:01
Comments...
Original Article

Yes but it depends on how you define the verb “to optimize”. For any image conversion heavy lifting I rely on the trusty ImageMagick yet I’ve been wondering whether my argument preset is correct: should it be more or less optimized?

The problem with questions is that they lead to other questions, such as: how much assets is this blog actually generating each year? Is my image optimization technique sustainable enough or will I end up with terabytes full of nonsense in ten or twenty years?

When it comes to size, opening up the handy gdu disk analyser in the assets/post folder is enough to get a first impression:

Gdu summarizing how much disk usage the assets on this blog are for each year in MiB.

As I maintain the same folder structure for both content/post and assets/post —this post lives under /post/2025/10/is-it-worth-it-to-optimize-images-for-your-site/ , for example—generating an overview of asset sizes per year becomes trivial. Not taking the earlier Brain Baking years into account, the total amount of data that gets added each year is on average 12.5 MiB . Let’s make that between thirteen and fourteen as 2025 isn’t finished yet.

That means in twenty years, I’ll have accumulated an extra 260 MiB . That’s not even half a classic CD-ROM. Is it really worth it then, to think twice about every MiB that gets checked in? Well, yes, since all those bytes need to leave one server and make an appearance at another in order to serve these pretty images to your visitor. Besides, as a proud member of The 512KB Club , I should keep my promise in reducing file sizes as much as possible.

Of course, not all posts have assets attached to them: the average amount of assets linked to a post here is 1.038 with each post having about 147.24 KiB on data. That’s quite optimized! Yet can I do better? Or should I stop over-compressing those images up to the point that they’re losing their vivid shine? More questions! No wait, those were the same.

Here’s the default ImageMagick command I rely on: 

mogrify -sampling-factor 4:2:0 +profile '!icc,*' -quality 80 -interlace JPEG -format jpg -colorspace sRGB screenshot.png

What exactly does this do?

  • -sampling-factor 4:2:0 : the sampling factor used for the JPEG encoder. If Colin Bendell tells me to use 4:2:0 claiming a +/- 17% image size reduction, then I believe him.
  • +profile '!icc,*' : Removes all profiles except for the ICC colour profile; gets rid of EXIF data. See What Exif Data Reveals About Your Site .
  • -quality 80 : The compression quality of the image. With 90 or less, chroma channels are downsampled (which I instruct it to do anyway with the sampling factor argument).
  • -interlace JPEG : explicitly tasks ImageMagick to create a progressive JPEG allowing for the browser to show a lower-resolution version of the image whilst data is still being transferred. Perceived download speed is also important!
  • -format jpg : Why on earth would you want to export JPEG files when the era of WebP is here? That’s an easy one to answer: because my retro hardware knows JPEG. Because I believe we should build websites that last.
  • -colorspace sRGB : the default and recommended option for the WWW for image that do not contain any extra colorspace information such as JPEG. Other options provide slightly better compression .

I urge you to read Colin’s old but relevant post on chroma (colour detail) and luma (lightness and darkness) and how to optimize for the web/mobile. It even includes a regression analysis, concluding that:

Resizing images matters most. It multiplies the size a little more than the square root of the total pixels. More pixels, many more bytes. Compression matters somewhat. For quality=80 the bytes are x23; for quality=100 bytes multiply x50. Subsampling of 4:2:0 could further reduce the bytes by 17%.

What I did not realize until now by testing an comparing images is that -strip does something else besides stripping GPS Exif data. I noticed the export became washed out, as if a portion of the colour profile information was lost. Take a close look at the macOS dock screenshots re-rendered in Firefox:

Above: using -strip; without ICC. Below: using +profile '!icc,*'; with ICC.

Can you find the difference by inspecting the saturation of the red Firefox fox or the yellow wings of the NetNewsWire satellite? The difference is very subtle—and very difficult to showcase in a screenshot—but very annoying.

Inspecting the images using ImageMagick’s identify reveals that the ICC profile is removed in the process: 

identify -verbose original.jpg | grep Profile
  Profiles:
    Profile-exif: 62 bytes
    Profile-icc: 4032 bytes
identify -verbose stripped.jpg | grep Profile
???? (no output)

The embedded ICC profile is there to make sure the image looks the same on any computer and any piece of software; without it browsers can render it like they want. The result is a flat looking image as you can see in the above screenshot (which does have an embedded profile). The -colorspace option does not solve this: it tells ImageMagick to convert the colorspace, not to attach it. Instead of using -strip , use +profile '!icc,*' to throw away all profiles but the ICC one.

Also, so be sure to add a -resize as this obviously has the highest impact on file sizes. But wait, what about providing a higher resolution image to desktop browsers and reducing the resolution to lower versions for mobile browsers? For me, that’s a hassle I don’t want to bother with at all. It requires saving the assets in their original format and providing a couple of alternatives, greatly increasing the total size of the source repository, the total size of the deployable folder, and the total bandwidth for my humble server.

For mobile users, that’s not a problem as downloading 147.24 KiB of data is less then the copious amounts of megabytes that will get slurped in when you visit your average newspaper site. For ultra widescreen 4K nerds, the max width on the container wrapping this <article/> will keep things somewhat in check.

The biggest takeaway for me is that in twenty years I’ll have filled half a CD-ROM which is significantly less than I expected. Should this incentivize me to bump the quality to 90% , reduce downsampling, or instead increase the usage of assets in general?

Maybe I should be less worried about the file size and more about the content.

webdesign

I Wasted 8 Years of My Life in Crypto

Hacker News
twitter.com
2025-12-07 12:57:59
Comments...
Original Article

What do you use TypedArrays for in JavaScript/TypeScript?

Lobsters
lobste.rs
2025-12-07 12:49:14
I've recently been using TypedArrays more for manual memory management / custom data storage, and find myself somewhat struggling with the lack of good typing support in TypeScript w.r.t. TypedArrays. eg. Even if I can define a number value type union such as type Kind = 0 | 1 | 2 | 3 | 4 | 5; I ha...
Original Article

I've recently been using TypedArrays more for manual memory management / custom data storage, and find myself somewhat struggling with the lack of good typing support in TypeScript w.r.t. TypedArrays. eg. Even if I can define a number value type union such as 

type Kind = 0 | 1 | 2 | 3 | 4 | 5;

I have no way of defining a Uint8Array that contains only this Kind type. After working around this with arr[i] as Kind casting and getting bit a few times by refactorings breaking the code but the as cast types not catching this, I opened up a feature suggestion for TypeScript (the pits, I know...) to support this. One response I got was to say that this is a niche use-case, and that made me wonder:

What are JavaScript/TypeScript developers actually using TypedArrays for, if number is a sufficient value type in all but niche use-cases? If you have open-source code examples to point to, that's awesome, but I'm interested in user stories / written word as well.

The Reverse-Centaur's Guide to Criticizing AI

Hacker News
pluralistic.net
2025-12-07 12:45:46
Comments...
Original Article


Today's links


The staring red eye of HAL 9000 from Stanley Kubrick's '2001: A Space Odyssey. In the center is the poop emoji from the cover of the US edition of 'Enshittification,' with angry eyebrows and a black, grawlix-scrawled bar over its mouth. The poop emoji's eyes have also been replaced with the HAL eye.

The Reverse Centaur’s Guide to Criticizing AI ( permalink )

Last night, I gave a speech for the University of Washington's "Neuroscience, AI and Society" lecture series, through the university's Computational Neuroscience Center. It was called "The Reverse Centaur’s Guide to Criticizing AI," and it's based on the manuscript for my next book, "The Reverse Centaur’s Guide to Life After AI," which will be out from Farrar, Straus and Giroux next June:

https://www.eventbrite.com/e/future-tense-neuroscience-ai-and-society-with-cory-doctorow-tickets-1735371255139

The talk was sold out, but here's the text of my lecture. I'm very grateful to UW for the opportunity, and for a lovely visit to Seattle!

==

I'm a science fiction writer, which means that my job is to make up futuristic parables about our current techno-social arrangements to interrogate not just what a gadget does , but who it does it for , and who it does it to.

What I don't do is predict the future. No one can predict the future, which is a good thing, since if the future were predictable, that would mean that what we all do couldn't change it. It would mean that the future was arriving on fixed rails and couldn't be steered.

Jesus Christ, what a miserable proposition!

Now, not everyone understands the distinction. They think sf writers are oracles, soothsayers. Unfortunately, even some of my colleagues labor under the delusion that they can "see the future."

But for every sf writer who deludes themselves into thinking that they are writing the future, there are a hundred sf fans who believe that they are reading the future, and a depressing number of those people appear to have become AI bros. The fact that these guys can't shut up about the day that their spicy autocomplete machine will wake up and turn us all into paperclips has led many confused journalists and conference organizers to try to get me to comment on the future of AI.

That's a thing I strenuously resisted doing, because I wasted two years of my life explaining patiently and repeatedly why I thought crypto was stupid, and getting relentless bollocked by cryptocurrency cultists who at first insisted that I just didn't understand crypto. And then, when I made it clear that I did understand crypto, insisted that I must be a paid shill.

This is literally what happens when you argue with Scientologists, and life is Just. Too. Short.

So I didn't want to get lured into another one of those quagmires, because on the one hand, I just don't think AI is that important of a technology, and on the other hand, I have very nuanced and complicated views about what's wrong, and not wrong, about AI, and it takes a long time to explain that stuff.

But people wouldn't stop asking, so I did what I always do. I wrote a book.

Over the summer I wrote a book about what I think about AI, which is really about what I think about AI criticism, and more specifically, how to be a good AI critic. By which I mean: "How to be a critic whose criticism inflicts maximum damage on the parts of AI that are doing the most harm." I titled the book The Reverse Centaur's Guide to Life After AI , and Farrar, Straus and Giroux will publish it in June, 2026.

But you don't have to wait until then because I am going to break down the entire book's thesis for you tonight, over the next 40 minutes. I am going to talk fast .

#

Start with what a reverse centaur is. In automation theory, a "centaur" is a person who is assisted by a machine. You're a human head being carried around on a tireless robot body. Driving a car makes you a centaur, and so does using autocomplete.

And obviously, a reverse centaur is machine head on a human body, a person who is serving as a squishy meat appendage for an uncaring machine.

Like an Amazon delivery driver, who sits in a cabin surrounded by AI cameras, that monitor the driver's eyes and take points off if the driver looks in a proscribed direction, and monitors the driver's mouth because singing isn't allowed on the job, and rats the driver out to the boss if they don't make quota.

The driver is in that van because the van can't drive itself and can't get a parcel from the curb to your porch. The driver is a peripheral for a van, and the van drives the driver, at superhuman speed, demanding superhuman endurance. But the driver is human, so the van doesn't just use the driver. The van uses the driver up .

Obviously, it's nice to be a centaur, and it's horrible to be a reverse centaur. There are lots of AI tools that are potentially very centaur-like, but my thesis is that these tools are created and funded for the express purpose of creating reverse-centaurs, which is something none of us want to be.

But like I said, the job of an sf writer is to do more than think about what the gadget does, and drill down on who the gadget does it for and who the gadget does it to . Tech bosses want us to believe that there is only one way a technology can be used. Mark Zuckerberg wants you to think that it's technologically impossible to have a conversation with a friend without him listening in. Tim Cook wants you to think that it's technologically impossible for you to have a reliable computing experience unless he gets a veto over which software you install and without him taking 30 cents out of every dollar you spend. Sundar Pichai wants you think that it's impossible for you to find a webpage unless he gets to spy on you from asshole to appetite.

This is all a kind of vulgar Thatcherism. Margaret Thatcher's mantra was "There is no alternative." She repeated this so often they called her "TINA" Thatcher: There. Is. No. Alternative. TINA.

"There is no alternative" is a cheap rhetorical slight. It's a demand dressed up as an observation. "There is no alternative" means "STOP TRYING TO THINK OF AN ALTERNATIVE." Which, you know, fuck that .

I'm an sf writer, my job is to think of a dozen alternatives before breakfast.

So let me explain what I think is going on here with this AI bubble, and sort out the bullshit from the material reality, and explain how I think we could and should all be better AI critics.

#

Start with monopolies: tech companies are gigantic and they don't compete, they just take over whole sectors, either on their own or in cartels.

Google and Meta control the ad market. Google and Apple control the mobile market, and Google pays Apple more than $20 billion/year not to make a competing search engine, and of course, Google has a 90% Search market-share.

Now, you'd think that this was good news for the tech companies, owning their whole sector.

But it's actually a crisis. You see, when a company is growing, it is a "growth stock," and investors really like growth stocks. When you buy a share in a growth stock, you're making a bet that it will continue to grow. So growth stocks trade at a huge multiple of their earnings. This is called the "price to earnings ratio" or "P/E ratio."

But once a company stops growing, it is a "mature" stock, and it trades at a much lower P/E ratio. So for every dollar that Target – a mature company – brings in, it is worth ten dollars. It has a P/E ratio of 10, while Amazon has a P/E ratio of 36, which means that for every dollar Amazon brings in, the market values it at $36.

It's wonderful to run a company that's got a growth stock. Your shares are as good as money. If you want to buy another company, or hire a key worker, you can offer stock instead of cash. And stock is very easy for companies to get, because shares are manufactured right there on the premises, all you have to do is type some zeroes into a spreadsheet, while dollars are much harder to come by. A company can only get dollars from customers or creditors.

So when Amazon bids against Target for a key acquisition, or a key hire, Amazon can bid with shares they make by typing zeroes into a spreadsheet, and Target can only bid with dollars they get from selling stuff to us, or taking out loans, which is why Amazon generally wins those bidding wars.

That's the upside of having a growth stock. But here's the downside: eventually a company has to stop growing. Like, say you get a 90% market share in your sector, how are you gonna grow?

Once the market decides that you aren't a growth stock, once you become mature, your stock is revalued, to a P/E ratio befitting a mature stock.

If you are an exec at a dominant company with a growth stock, you have to live in constant fear that the market will decide that you're not likely to grow any further. Think of what happened to Facebook in the first quarter of 2022. They told investors that they experienced slightly slower growth in the USA than they had anticipated, and investors panicked . They staged a one-day, $240B sell off. A quarter-trillion dollars in 24 hours! At the time, it was the largest, most precipitous drop in corporate valuation in human history.

That's a monopolist's worst nightmare, because once you're presiding over a "mature" firm, the key employees you've been compensating with stock, experience a precipitous pay-drop and bolt for the exits, so you lose the people who might help you grow again, and you can only hire their replacements with dollars. With dollars, not shares.

And the same goes for acquiring companies that might help you grow, because they, too, are going to expect money, not stock. This is the paradox of the growth stock. While you are growing to domination, the market loves you, but once you achieve dominance, the market lops 75% or more off your value in a single stroke if they don't trust your pricing power.

Which is why growth stock companies are always desperately pumping up one bubble or another, spending billions to hype the pivot to video, or cryptocurrency, or NFTs, or Metaverse, or AI.

I'm not saying that tech bosses are making bets they don't plan on winning. But I am saying that winning the bet – creating a viable metaverse – is the secondary goal. The primary goal is to keep the market convinced that your company will continue to grow, and to remain convinced until the next bubble comes along.

So this is why they're hyping AI: the material basis for the hundreds of billions in AI investment.

#

Now I want to talk about how they're selling AI. The growth narrative of AI is that AI will disrupt labor markets. I use "disrupt" here in its most disreputable, tech bro sense.

The promise of AI – the promise AI companies make to investors – is that there will be AIs that can do your job, and when your boss fires you and replaces you with AI, he will keep half of your salary for himself, and give the other half to the AI company.

That's it.

That's the $13T growth story that MorganStanley is telling. It's why big investors and institutionals are giving AI companies hundreds of billions of dollars. And because they are piling in, normies are also getting sucked in, risking their retirement savings and their family's financial security.

Now, if AI could do your job, this would still be a problem. We'd have to figure out what to do with all these technologically unemployed people.

But AI can't do your job. It can help you do your job, but that doesn't mean it's going to save anyone money. Take radiology: there's some evidence that AIs can sometimes identify solid-mass tumors that some radiologists miss, and look, I've got cancer. Thankfully, it's very treatable, but I've got an interest in radiology being as reliable and accurate as possible.

If my Kaiser hospital bought some AI radiology tools and told its radiologists: "Hey folks, here's the deal. Today, you're processing about 100 x-rays per day. From now on, we're going to get an instantaneous second opinion from the AI, and if the AI thinks you've missed a tumor, we want you to go back and have another look, even if that means you're only processing 98 x-rays per day. That's fine, we just care about finding all those tumors."

If that's what they said, I'd be delighted. But no one is investing hundreds of billions in AI companies because they think AI will make radiology more expensive, not even if that also makes radiology more accurate. The market's bet on AI is that an AI salesman will visit the CEO of Kaiser and make this pitch: "Look, you fire 9/10s of your radiologists, saving $20m/year, you give us $10m/year, and you net $10m/year, and the remaining radiologists' job will be to oversee the diagnoses the AI makes at superhuman speed, and somehow remain vigilant as they do so, despite the fact that the AI is usually right, except when it's catastrophically wrong.

"And if the AI misses a tumor, this will be the human radiologist's fault, because they are the 'human in the loop.' It's their signature on the diagnosis."

This is a reverse centaur, and it's a specific kind of reverse-centaur: it's what Dan Davies calls an "accountability sink." The radiologist's job isn't really to oversee the AI's work, it's to take the blame for the AI's mistakes.

This is another key to understanding – and thus deflating – the AI bubble. The AI can't do your job, but an AI salesman can convince your boss to fire you and replace you with an AI that can't do your job. This is key because it helps us build the kinds of coalitions that will be successful in the fight against the AI bubble.

If you're someone who's worried about cancer, and you're being told that the price of making radiology too cheap to meter, is that we're going to have to re-home America's 32,000 radiologists, with the trade-off that no one will ever be denied radiology services again, you might say, "Well, OK, I'm sorry for those radiologists, and I fully support getting them job training or UBI or whatever. But the point of radiology is to fight cancer, not to pay radiologists, so I know what side I'm on."

AI hucksters and their customers in the C-suites want the public on their side. They want to forge a class alliance between AI deployers and the people who enjoy the fruits of the reverse centaurs' labor. They want us to think of ourselves as enemies to the workers.

Now, some people will be on the workers' side because of politics or aesthetics. They just like workers better than their bosses. But if you want to win over all the people who benefit from your labor, you need to understand and stress how the products of the AI will be substandard. That they are going to get charged more for worse things. That they have a shared material interest with you.

Will those products be substandard? There's every reason to think so. Earlier, I alluded to "automation blindness, "the physical impossibility of remaining vigilant for things that rarely occur. This is why TSA agents are incredibly good at spotting water bottles. Because they get a ton of practice at this, all day, every day. And why they fail to spot the guns and bombs that government red teams smuggle through checkpoints to see how well they work, because they just don't have any practice at that. Because, to a first approximation, no one deliberately brings a gun or a bomb through a TSA checkpoint.

Automation blindness is the Achilles' heel of "humans in the loop."

Think of AI software generation: there are plenty of coders who love using AI, and almost without exception, they are senior, experienced coders, who get to decide how they will use these tools. For example, you might ask the AI to generate a set of CSS files to faithfully render a web-page across multiple versions of multiple browsers. This is a notoriously fiddly thing to do, and it's pretty easy to verify if the code works – just eyeball it in a bunch of browsers. Or maybe the coder has a single data file they need to import and they don't want to write a whole utility to convert it.

Tasks like these can genuinely make coders more efficient and give them more time to do the fun part of coding, namely, solving really gnarly, abstract puzzles. But when you listen to business leaders talk about their AI plans for coders, it's clear they're not looking to make some centaurs.

They want to fire a lot of tech workers – 500,000 over the past three years – and make the rest pick up their work with coding, which is only possible if you let the AI do all the gnarly, creative problem solving, and then you do the most boring, soul-crushing part of the job: reviewing the AIs' code.

And because AI is just a word guessing program, because all it does is calculate the most probable word to go next, the errors it makes are especially subtle and hard to spot, because these bugs are literally statistically indistinguishable from working code (except that they're bugs).

Here's an example: code libraries are standard utilities that programmers can incorporate into their apps, so they don't have to do a bunch of repetitive programming. Like, if you want to process some text, you'll use a standard library. If it's an HTML file, that library might be called something like lib.html.text.parsing; and if it's a DOCX file, it'll be lib.docx.text.parsing. But reality is messy, humans are inattentive and stuff goes wrong, so sometimes, there's another library, this one for parsing PDFs, and instead of being called lib.pdf.text.parsing, it's called lib.text.pdf.parsing.

Now, because the AI is a statistical inference engine, because all it can do is predict what word will come next based on all the words that have been typed in the past, it will "hallucinate" a library called lib.pdf.text.parsing. And the thing is, malicious hackers know that the AI will make this error, so they will go out and create a library with the predictable, hallucinated name, and that library will get automatically sucked into your program, and it will do things like steal user data or try and penetrate other computers on the same network.

And you, the human in the loop – the reverse centaur – you have to spot this subtle, hard to find error, this bug that is literally statistically indistinguishable from correct code. Now, maybe a senior coder could catch this, because they've been around the block a few times, and they know about this tripwire.

But guess who tech bosses want to preferentially fire and replace with AI? Senior coders. Those mouthy, entitled, extremely highly paid workers, who don't think of themselves as workers. Who see themselves as founders in waiting, peers of the company's top management. The kind of coder who'd lead a walkout over the company building drone-targeting systems for the Pentagon, which cost Google ten billion dollars in 2018.

For AI to be valuable, it has to replace high -wage workers, and those are precisely the experienced workers, with process knowledge, and hard-won intuition, who might spot some of those statistically camouflaged AI errors.

Like I said, the point here is to replace high -waged workers.

And one of the reasons the AI companies are so anxious to fire coders is that coders are the princes of labor. They're the most consistently privileged, sought-after, and well-compensated workers in the labor force.

If you can replace coders with AI, who cant you replace with AI? Firing coders is an ad for AI.

Which brings me to AI art . AI art – or "art" – is also an ad for AI, but it's not part of AI's business model.

Let me explain: on average, illustrators don't make any money. They are already one of the most immiserated, precartized groups of workers out there. They suffer from a pathology called "vocational awe." That's a term coined by the librarian Fobazi Ettarh, and it refers to workers who are vulnerable to workplace exploitation because they actually care about their jobs – nurses, librarians, teachers, and artists.

If AI image generators put every illustrator working today out of a job, the resulting wage-bill savings would be undetectable as a proportion of all the costs associated with training and operating image-generators. The total wage bill for commercial illustrators is less than the kombucha bill for the company cafeteria at just one of Open AI's campuses.

The purpose of AI art – and the story of AI art as a death-knell for artists – is to convince the broad public that AI is amazing and will do amazing things. It's to create buzz. Which is not to say that it's not disgusting that former OpenAI CTO Mira Murati told a conference audience that "some creative jobs shouldn't have been there in the first place," and that it's not especially disgusting that she and her colleagues boast about using the work of artists to ruin those artists' livelihoods.

It's supposed to be disgusting. It's supposed to get artists to run around and say, "The AI can do my job, and it's going to steal my job, and isn't that terrible? "

Because the customers for AI – corporate bosses – don't see AI taking workers' jobs as terrible. They see it as wonderful.

But can AI do an illustrator's job? Or any artist's job?

Let's think about that for a second. I've been a working artist since I was 17 years old, when I sold my first short story, and I've given it a lot of thought, and here's what I think art is: it starts with an artist, who has some vast, complex, numinous, irreducible feeling in their mind. And the artist infuses that feeling into some artistic medium. They make a song, or a poem, or a painting, or a drawing, or a dance, or a book, or a photograph. And the idea is, when you experience this work, a facsimile of the big, numinous, irreducible feeling will materialize in your mind.

Now that I've defined art, we have to go on a little detour.

I have a friend who's a law professor, and before the rise of chatbots, law students knew better than to ask for reference letters from their profs, unless they were a really good student. Because those letters were a pain in the ass to write. So if you advertised for a postdoc and you heard from a candidate with a reference letter from a respected prof, the mere existence of that letter told you that the prof really thought highly of that student.

But then we got chatbots, and everyone knows that you generate a reference letter by feeding three bullet points to an LLM, and it'll barf up five paragraphs of florid nonsense about the student.

So when my friend advertises for a postdoc, they are flooded with reference letters, and they deal with this flood by feeding all these letters to another chatbot, and ask it to reduce them back to three bullet points. Now, obviously, they won't be the same bullet-points, which makes this whole thing terrible.

But just as obviously, nothing in that five-paragraph letter except the original three bullet points are relevant to the student. The chatbot doesn't know the student. It doesn't know anything about them. It cannot add a single true or useful statement about the student to the letter.

What does this have to do with AI art? Art is a transfer of a big, numinous, irreducible feeling from an artist to someone else. But the image-gen program doesn't know anything about your big, numinous, irreducible feeling. The only thing it knows is whatever you put into your prompt, and those few sentences are diluted across a million pixels or a hundred thousand words, so that the average communicative density of the resulting work is indistinguishable from zero.

It's possible to infuse more communicative intent into a work: writing more detailed prompts, or doing the selective work of choosing from among many variants, or directly tinkering with the AI image after the fact, with a paintbrush or Photoshop or The Gimp. And if there will ever be a piece of AI art that is good art – as opposed to merely striking, or interesting, or an example of good draftsmanship – it will be thanks to those additional infusions of creative intent by a human.

And in the meantime, it's bad art. It's bad art in the sense of being "eerie," the word Mark Fisher uses to describe "when there is something present where there should be nothing, or there is nothing present when there should be something."

AI art is eerie because it seems like there is an intender and an intention behind every word and every pixel, because we have a lifetime of experience that tells us that paintings have painters, and writing has writers. But it's missing something. It has nothing to say, or whatever it has to say is so diluted that it's undetectable.

The images were striking before we figured out the trick, but now they're just like the images we imagine in clouds or piles of leaves. We're the ones drawing a frame around part of the scene, we're the ones focusing on some contours and ignoring the others. We're looking at an inkblot, and it's not telling us anything.

Sometimes that can be visually arresting, and to the extent that it amuses people in a community of prompters and viewers, that's harmless.

I know someone who plays a weekly Dungeons and Dragons game over Zoom. It's transcribed by an open source model running locally on the dungeon master's computer, which summarizes the night's session and prompts an image generator to create illustrations of key moments. These summaries and images are hilarious because they're full of errors. It's a bit of harmless fun, and it bring a small amount of additional pleasure to a small group of people. No one is going to fire an illustrator because D&D players are image-genning funny illustrations where seven-fingered paladins wrestle with orcs that have an extra hand.

But bosses have and will fire illustrators, because they fantasize about being able to dispense with creative professionals and just prompt an AI. Because even though the AI can't do the illustrator's job, an AI salesman can convince the illustrator's boss to fire them and replace them with an AI that can't do their job.

This is a disgusting and terrible juncture, and we should not simply shrug our shoulders and accept Thatcherism's fatalism: "There is no alternative."

So what is the alternative? A lot of artists and their allies think they have an answer: they say we should extend copyright to cover the activities associated with training a model.

And I'm here to tell you they are wrong : wrong because this would inflict terrible collateral damage on socially beneficial activities, and it would represent a massive expansion of copyright over activities that are currently permitted – for good reason!.

Let's break down the steps in AI training.

First, you scrape a bunch of web-pages. This is unambiguously legal under present copyright law. You do not need a license to make a transient copy of a copyrighted work in order to analyze it, otherwise search engines would be illegal. Ban scraping and Google will be the last search engine we ever get, the Internet Archive will go out of business, that guy in Austria who scraped all the grocery store sites and proved that the big chains were colluding to rig prices would be in deep trouble.

Next, you perform analysis on those works. Basically, you count stuff on them: count pixels and their colors and proximity to other pixels; or count words. This is obviously not something you need a license for. It's just not illegal to count the elements of a copyrighted work. And we really don't want it to be, not if you're interested in scholarship of any kind.

And it's important to note that counting things is legal, even if you're working from an illegally obtained copy. Like, if you go to the flea market, and you buy a bootleg music CD, and you take it home and you make a list of all the adverbs in the lyrics, and you publish that list, you are not infringing copyright by doing so.

Perhaps you've infringed copyright by getting the pirated CD, but not by counting the lyrics.

This is why Anthropic offered a $1.5b settlement for training its models based on a ton of books it downloaded from a pirate site: not because counting the words in the books infringes anyone's rights, but because they were worried that they were going to get hit with $150k/book statutory damages for downloading the files.

OK, after you count all the pixels or the words, it's time for the final step: publishing them. Because that's what a model is: a literary work (that is, a piece of software) that embodies a bunch of facts about a bunch of other works, word and pixel distribution information, encoded in a multidimensional array.

And again, copyright absolutely does not prohibit you from publishing facts about copyrighted works. And again, no one should want to live in a world where someone else gets to decide which truthful, factual statements you can publish.

But hey, maybe you think this is all sophistry. Maybe you think I'm full of shit. That's fine. It wouldn't be the first time someone thought that.

After all, even if I'm right about how copyright works now, there's no reason we couldn't change copyright to ban training activities, and maybe there's even a clever way to wordsmith the law so that it only catches bad things we don't like, and not all the good stuff that comes from scraping, analyzing and publishing.

Well, even then, you're not gonna help out creators by creating this new copyright. If you're thinking that you can, you need to grapple with this fact: we have monotonically expanded copyright since 1976, so that today, copyright covers more kinds of works, grants exclusive rights over more uses, and lasts longer.

And today, the media industry is larger and more profitable than it has ever been, and also: the share of media industry income that goes to creative workers is lower than its ever been, both in real terms, and as a proportion of those incredible gains made by creators' bosses at the media company.

So how it is that we have given all these new rights to creators, and those new rights have generated untold billions, and left creators poorer ? It's because in a creative market dominated by five publishers, four studios, three labels, two mobile app stores, and a single company that controls all the ebooks and audiobooks, giving a creative worker extra rights to bargain with is like giving your bullied kid more lunch money.

It doesn't matter how much lunch money you give the kid, the bullies will take it all. Give that kid enough money and the bullies will hire an agency to run a global campaign proclaiming "think of the hungry kids! Give them more lunch money!"

Creative workers who cheer on lawsuits by the big studios and labels need to remember the first rule of class warfare: things that are good for your boss are rarely what's good for you.

The day Disney and Universal filed suit against Midjourney, I got a press release from the RIAA, which represents Disney and Universal through their recording arms. Universal is the largest label in the world. Together with Sony and Warner, they control 70% of all music recordings in copyright today.

It starts: "There is a clear path forward through partnerships that both further AI innovation and foster human artistry."

It ends: "This action by Disney and Universal represents a critical stand for human creativity and responsible innovation."

And it's signed by Mitch Glazier, CEO of the RIAA.

It's very likely that name doesn't mean anything to you. But let me tell you who Mitch Glazier is. Today, Mitch Glazier is the CEO if the RIAA, with an annual salary of $1.3m. But until 1999, Mitch Glazier was a key Congressional staffer, and in 1999, Glazier snuck an amendment into an unrelated bill, the Satellite Home Viewer Improvement Act, that killed musicians' right to take their recordings back from their labels.

This is a practice that had been especially important to "heritage acts" (which is a record industry euphemism for "old music recorded by Black people"), for whom this right represented the difference between making rent and ending up on the street.

When it became clear that Glazier had pulled this musician-impoverishing scam, there was so much public outcry, that Congress actually came back for a special session, just to vote again to cancel Glazier's amendment. And then Glazier was kicked out of his cushy Congressional job, whereupon the RIAA started paying more than $1m/year to "represent the music industry."

This is the guy who signed that press release in my inbox. And his message was: The problem isn't that Midjourney wants to train a Gen AI model on copyrighted works, and then use that model to put artists on the breadline. The problem is that Midjourney didn't pay RIAA members Universal and Disney for permission to train a model. Because if only Midjourney had given Disney and Universal several million dollars for training rights to their catalogs, the companies would have happily allowed them to train to their heart's content, and they would have bought the resulting models, and fired as many creative professionals as they could.

I mean, have we already forgotten the Hollywood strikes? I sure haven't. I live in Burbank, home to Disney, Universal and Warner, and I was out on the line with my comrades from the Writers Guild, offering solidarity on behalf of my union, IATSE 830, The Animation Guild, where I'm a member of the writers' unit.

And I'll never forget when one writer turned to me and said, "You know, you prompt an LLM exactly the same way an exec gives shitty notes to a writers' room. You know: 'Make me ET, except it's about a dog, and put a love interest in there, and a car chase in the second act.' The difference is, you say that to a writers' room and they all make fun of you and call you a fucking idiot suit. But you say it to an LLM and it will cheerfully shit out a terrible script that conforms exactly to that spec (you know, Air Bud )."

These companies are desperate to use AI to displace workers. When Getty Images sues AI companies, it's not representing the interests of photographers . Getty hates paying photographers! Getty just wants to get paid for the training run, and they want the resulting AI model to have guardrails, so it will refuse to create images that compete with Getty's images for anyone except Getty. But Getty will absolutely use its models to bankrupt as many photographers as it possibly can.

A new copyright to train models won't get us a world where models aren't used to destroy artists, it'll just get us a world where the standard contracts of the handful of companies that control all creative labor markets are updated to require us to hand over those new training rights to those companies. Demanding a new copyright just makes you a useful idiot for your boss, a human shield they can brandish in policy fights, a tissue-thin pretense of "won't someone think of the hungry artists?"

When really what they're demanding is a world where 30% of the investment capital of the AI companies go into their shareholders' pockets. When an artist is being devoured by rapacious monopolies, does it matter how they divvy up the meal?

We need to protect artists from AI predation, not just create a new way for artists to be mad about their impoverishment.

And incredibly enough, there's a really simple way to do that. After 20+ years of being consistently wrong and terrible for artists' rights, the US Copyright Office has finally done something gloriously, wonderfully right . All through this AI bubble, the Copyright Office has maintained – correctly – that AI-generated works cannot be copyrighted, because copyright is exclusively for humans. That's why the "monkey selfie" is in the public domain. Copyright is only awarded to works of human creative expression that are fixed in a tangible medium.

And not only has the Copyright Office taken this position, they've defended it vigorously in court, repeatedly winning judgments to uphold this principle.

The fact that every AI created work is in the public domain means that if Getty or Disney or Universal or Hearst newspapers use AI to generate works – then anyone else can take those works, copy them, sell them, or give them away for free. And the only thing those companies hate more than paying creative workers, is having other people take their stuff without permission.

The US Copyright Office's position means that the only way these companies can get a copyright is to pay humans to do creative work. This is a recipe for centaurhood. If you're a visual artist or writer who uses prompts to come up with ideas or variations, that's no problem, because the ultimate work comes from you. And if you're a video editor who uses deepfakes to change the eyelines of 200 extras in a crowd-scene, then sure, those eyeballs are in the public domain, but the movie stays copyrighted.

But creative workers don't have to rely on the US government to rescue us from AI predators. We can do it ourselves, the way the writers did in their historic writers' strike. The writers brought the studios to their knees. They did it because they are organized and solidaristic, but also are allowed to do something that virtually no other workers are allowed to do: they can engage in "sectoral bargaining," whereby all the workers in a sector can negotiate a contract with every employer in the sector.

That's been illegal for most workers since the late 1940s, when the Taft-Hartley Act outlawed it. If we are gonna campaign to get a new law passed in hopes of making more money and having more control over our labor, we should campaign to restore sectoral bargaining, not to expand copyright.

Our allies in a campaign to expand copyright are our bosses, who have never had our best interests at heart. While our allies in the fight for sector bargaining are every worker in the country. As the song goes, "Which side are you on?"

OK, I need to bring this talk in for a landing now, because I'm out of time, so I'm going to close out with this: AI is a bubble and bubbles are terrible.

Bubbles transfer the life's savings of normal people who are just trying to have a dignified retirement to the wealthiest and most unethical people in our society, and every bubble eventually bursts, taking their savings with it.

But not every bubble is created equal. Some bubbles leave behind something productive. Worldcom stole billions from everyday people by defrauding them about orders for fiber optic cables. The CEO went to prison and died there. But the fiber outlived him. It's still in the ground. At my home, I've got 2gb symmetrical fiber, because AT&T lit up some of that old Worldcom dark fiber.

All things being equal, it would have been better if Worldcom hadn't ever existed, but the only thing worse than Worldcom committing all that ghastly fraud would be if there was nothing to salvage from the wreckage.

I don't think we'll salvage much from cryptocurrency, for example. Sure, there'll be a few coders who've learned something about secure programming in Rust. But when crypto dies, what it will leave behind is bad Austrian economics and worse monkey JPEGs.

AI is a bubble and it will burst. Most of the companies will fail. Most of the data-centers will be shuttered or sold for parts. So what will be left behind?

We'll have a bunch of coders who are really good at applied statistics. We'll have a lot of cheap GPUs, which'll be good news for, say, effects artists and climate scientists, who'll be able to buy that critical hardware at pennies on the dollar. And we'll have the open source models that run on commodity hardware, AI tools that can do a lot of useful stuff, like transcribing audio and video, describing images, summarizing documents, automating a lot of labor-intensive graphic editing, like removing backgrounds, or airbrushing passersby out of photos. These will run on our laptops and phones, and open source hackers will find ways to push them to do things their makers never dreamt of.

If there had never been an AI bubble, if all this stuff arose merely because computer scientists and product managers noodled around for a few years coming up with cool new apps for back-propagation, machine learning and generative adversarial networks, most people would have been pleasantly surprised with these interesting new things their computers could do. We'd call them "plugins."

It's the bubble that sucks, not these applications. The bubble doesn't want cheap useful things. It wants expensive, "disruptive" things: Big foundation models that lose billions of dollars every year.

When the AI investment mania halts, most of those models are going to disappear, because it just won't be economical to keep the data-centers running. As Stein's Law has it: "Anything that can't go on forever eventually stops."

The collapse of the AI bubble is going to be ugly. Seven AI companies currently account for more than a third of the stock market, and they endlessly pass around the same $100b IOU.

Bosses are mass-firing productive workers and replacing them with janky AI, and when the janky AI is gone, no one will be able to find and re-hire most of those workers, we're going to go from disfunctional AI systems to nothing.

AI is the asbestos in the walls of our technological society, stuffed there with wild abandon by a finance sector and tech monopolists run amok. We will be excavating it for a generation or more.

So we need to get rid of this bubble. Pop it, as quickly as we can. To do that, we have to focus on the material factors driving the bubble. The bubble isn't being driven by deepfake porn, or election disinformation, or AI image-gen, or slop advertising.

All that stuff is terrible and harmful, but it's not driving investment. The total dollar figure represented by these apps doesn't come close to making a dent in the capital expenditures and operating costs of AI. They are peripheral, residual uses: flashy, but unimportant to the bubble.

Get rid of all those uses and you reduce the expected income of AI companies by a sum so small it rounds to zero.

Same goes for all that "AI Safety" nonsense, that purports to concern itself with preventing an AI from attaining sentience and turning us all into paperclips. First of all, this is facially absurd. Throwing more words and GPUs into the word-guessing program won't make it sentient. That's like saying, "Well, we keep breeding these horses to run faster and faster, so it's only a matter of time until one of our mares gives birth to a locomotive." A human mind is not a word-guessing program with a lot of extra words.

I'm here for science fiction thought experiments, don't get me wrong. But also, don't mistake sf for prophesy. SF stories about superintelligence are futuristic parables, not business plans, roadmaps, or predictions.

The AI Safety people say they are worried that AI is going to end the world, but AI bosses love these weirdos. Because on the one hand, if AI is powerful enough to destroy the world, think of how much money it can make! And on the other hand, no AI business plan has a line on its revenue projections spreadsheet labeled "Income from turning the human race into paperclips." So even if we ban AI companies from doing this, we won't cost them a dime in investment capital.

To pop the bubble, we have to hammer on the forces that created the bubble: the myth that AI can do your job, especially if you get high wages that your boss can claw back; the understanding that growth companies need a succession of ever-more-outlandish bubbles to stay alive; the fact that workers and the public they serve are on one side of this fight, and bosses and their investors are on the other side.

Because the AI bubble really is very bad news, it's worth fighting seriously, and a serious fight against AI strikes at its roots: the material factors fueling the hundreds of billions in wasted capital that are being spent to put us all on the breadline and fill all our walls with high-tech asbestos.

( Image: Cryteria , CC BY 3.0 , modified )

Hey look at this ( permalink )


A shelf of leatherbound history books with a gilt-stamped series title, 'The World's Famous Events.'

Object permanence ( permalink )

#20yrsago Haunted Mansion papercraft model adds crypts and gates https://www.haunteddimensions.raykeim.com/index313.html

#20yrsago Print your own Monopoly money https://web.archive.org/web/20051202030047/http://www.hasbro.com/monopoly/pl/page.treasurechest/dn/default.cfm

#15yrsago Bunnie explains the technical intricacies and legalities of Xbox hacking https://www.bunniestudios.com/blog/2010/usa-v-crippen-a-retrospective/

#15yrsago How Pac Man’s ghosts decide what to do: elegant complexity https://web.archive.org/web/20101205044323/https://gameinternals.com/post/2072558330/understanding-pac-man-ghost-behavior

#15yrsago Glorious, elaborate, profane insults of the world https://www.reddit.com/r/AskReddit/comments/efee7/what_are_your_favorite_culturally_untranslateable/?sort=confidence

#15yrsago Walt Disney World castmembers speak about their search for a living wage https://www.youtube.com/watch?v=f5BMQ3xQc7o

#15yrsago Wikileaks cables reveal that the US wrote Spain’s proposed copyright law https://web.archive.org/web/20140723230745/https://elpais.com/elpais/2010/12/03/actualidad/1291367868_850215.html

#15yrsago Cities made of broken technology https://web.archive.org/web/20101203132915/https://agora-gallery.com/artistpage/Franco_Recchia.aspx

#10yrsago The TPP’s ban on source-code disclosure requirements: bad news for information security https://www.eff.org/deeplinks/2015/12/tpp-threatens-security-and-safety-locking-down-us-policy-source-code-audit

#10yrsago Fossil fuel divestment sit-in at MIT President’s office hits 10,000,000,000-hour mark https://twitter.com/FossilFreeMIT/status/672526210581274624

#10yrsago Hacker dumps United Arab Emirates Invest Bank’s customer data https://www.dailydot.com/news/invest-bank-hacker-buba/

#10yrsago Illinois prisons spy on prisoners, sue them for rent on their cells if they have any money https://www.chicagotribune.com/2015/11/30/state-sues-prisoners-to-pay-for-their-room-board/

#10yrsago Free usability help for privacy toolmakers https://superbloom.design/learning/blog/apply-for-help/

#10yrsago In the first 334 days of 2015, America has seen 351 mass shootings (and counting) https://web.archive.org/web/20151209004329/https://www.washingtonpost.com/news/wonk/wp/2015/11/30/there-have-been-334-days-and-351-mass-shootings-so-far-this-year/

#10yrsago Not even the scapegoats will go to jail for BP’s murder of the Gulf Coast https://arstechnica.com/tech-policy/2015/12/manslaughter-charges-dropped-in-bp-spill-case-nobody-from-bp-will-go-to-prison/

#10yrsago Urban Transport Without the Hot Air: confusing the issue with relevant facts! https://memex.craphound.com/2015/12/03/urban-transport-without-the-hot-air-confusing-the-issue-with-relevant-facts/

#5yrsago Breathtaking Iphone hack https://pluralistic.net/2020/12/03/ministry-for-the-future/#awdl

#5yrsago Graffitists hit dozens of NYC subway cars https://pluralistic.net/2020/12/03/ministry-for-the-future/#getting-up

#5yrsago The Ministry For the Future https://pluralistic.net/2020/12/03/ministry-for-the-future/#ksr

#5yrsago Monopolies made America vulnerable to covid https://pluralistic.net/2020/12/03/ministry-for-the-future/#big-health

#5yrsago Section 230 is Good, Actually https://pluralistic.net/2020/12/04/kawaski-trawick/#230

#5yrsago Postmortem of the NYPD's murder of a Black man https://pluralistic.net/2020/12/04/kawaski-trawick/#Kawaski-Trawick

#5yrsago Student debt trap https://pluralistic.net/2020/12/04/kawaski-trawick/#strike-debt

#1yrago "That Makes Me Smart" https://pluralistic.net/2024/12/04/its-not-a-lie/#its-a-premature-truth

#1yrago Canada sues Google https://pluralistic.net/2024/12/03/clementsy/#can-tech

Upcoming appearances ( permalink )

A photo of me onstage, giving a speech, pounding the podium.


A screenshot of me at my desk, doing a livecast.

Recent appearances ( permalink )


A grid of my books with Will Stahle covers..

Latest books ( permalink )


A cardboard book box with the Macmillan logo.

Upcoming books ( permalink )

  • "Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026

  • "Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026

  • "The Memex Method," Farrar, Straus, Giroux, 2026

  • "The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026


Colophon ( permalink )

Today's top sources:

Currently writing:

  • "The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. LEGAL REVIEW AND COPYEDIT COMPLETE.

  • "The Post-American Internet," a short book about internet policy in the age of Trumpism. PLANNING.

  • A Little Brother short story about DIY insulin PLANNING

This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

" When life gives you SARS, you make sarsaparilla " -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

The Anatomy of a macOS App

Hacker News
eclecticlight.co
2025-12-07 12:31:53
Comments...
Original Article

Programs running in windowing environments, applications as we used to know them, have more complicated requirements than those run from a command line. Rather than embed all the resources they require for windows, menus and the rest in a single file, Mac OS broke new ground by putting those into resources stored in the app’s resource fork.

prefsresedit

This is QuarkXPress version 4.11 from around 2000, with its resources displayed in the resource editor ResEdit. Executable code was also stored in CODE resources, and every file contained type and creator information to support the illusions created by the Finder.

Mac OS X

When Mac OS X was designed, it switched to the bundle structure inherited from NeXTSTEP. Instead of this multitude of resources, apps consisted of a hierarchy of directories containing files of executable code, and those with what had in Mac OS been supporting resources. Those app bundles came to adopt a standard form, shown below.

The bundle name has the extension .app, and contains a single directory Contents. Within that, the executable code is in the MacOS directory, which may contain both the main executable for the GUI app and any bundled command tools provided. Another directory contains Resources, including the app’s custom icon, and components of its GUI. In some apps, there’s another directory of Frameworks containing dylibs (libraries).

There are also two important files, Info.plist and PkgInfo. The latter contains the same type and creator information inherited from Classic Mac OS, and apparently isn’t mandatory although it appears universal. The information property list is essential, as it specifies the names of the executable and its icon file in Resources, the minimum version of macOS required, type declarations of the app’s documents, version numbers, and more.

When running a command tool in macOS, its Mach-O executable is launched by launchd , whose purpose is to run code. Launching an app is more demanding, although the app’s executable is still launched by launchd . Before that can happen, macOS starts the launch process using LaunchServices and RunningBoard, which rely on information obtained from Info.plist and other components in the app bundle.

macOS

This structure remained stable until the introduction of code signatures in Mac OS X 10.5 Leopard in 2007. Accommodating those added a directory named _CodeSignature containing the signature in a CodeResources file. That includes code directory hashes (CDHashes) to check the integrity of the contents of the app bundle. Apps distributed by the App Store include a store receipt in another directory, _MASReceipt. Since 2018, when Apple introduced notarization, the ‘ticket’ issued by Apple can be ‘stapled’ into the app bundle as the file CodeResources.

Many apps come with additional items that might in the past have been installed by them in their Library/Application Support folders and elsewhere, but are now included in the app bundle. These can include the following directories:

  • Library, containing folders of LaunchDaemons and LoginItems that would previously have been installed in either the main Library folder, or that in the user’s Home folder;
  • XPCServices, for executable code that the app uses to provide specific services;
  • Plugins, for some types of app extension (Appex);
  • Extensions, for other types of app extension, including app intents.

You may also come across other components, including a version.plist in Apple’s apps.

This centralisation of components in the app bundle has brought several benefits. Being self-contained, apps are easier to install and update, and cleaner to remove. Their components are less likely to go missing, and most of all they’re held within the protection of the app’s signature and notarisation, an important improvement in security.

Assembling these into a diagram shows how the anatomy of an app has grown over the last few years.

Components shown in pale yellow are either mandatory or essentially universal. Those shown in green are found in apps distributed through the App Store, while that shown in blue is the stapled notarisation ticket (optional). You will also see additional folders and components such as Automator workflows, scripts, and others.

There is no difference in structure between apps built for current Intel and Arm architectures. That’s because binaries in the MacOS folder (and executable code in other directories like Frameworks, XPCServices and Plugins) contain platform-specific code in a single Mach-O executable. Thus, an app that’s Universal and runs native on both architectures includes code for both in its single ‘fat’ code file, and they even have separate signatures stored within common files.

How the Disappearance of Flight 19 Fueled the Legend of the Bermuda Triangle

Hacker News
www.smithsonianmag.com
2025-12-07 12:25:04
Comments...
Original Article

Google Titans architecture, helping AI have long-term memory

Hacker News
research.google
2025-12-07 12:23:45
Comments...
Original Article

The Transformer architecture revolutionized sequence modeling with its introduction of attention , a mechanism by which models look back at earlier inputs to prioritize relevant input data. However, computational cost increases drastically with sequence length, which limits the ability to scale Transformer-based models to extremely long contexts, such as those required for full-document understanding or genomic analysis.

The research community explored various approaches for solutions, such as efficient linear recurrent neural networks (RNNs) and state space models (SSMs) like Mamba-2 . These models offer fast, linear scaling by compressing context into a fixed-size. However, this fixed-size compression cannot adequately capture the rich information in very long sequences.

In two new papers, Titans and MIRAS , we introduce an architecture and theoretical blueprint that combine the speed of RNNs with the accuracy of transformers. Titans is the specific architecture (the tool), and MIRAS is the theoretical framework (the blueprint) for generalizing these approaches. Together, they advance the concept of test-time memorization, the ability of an AI model to maintain long-term memory by incorporating more powerful “surprise” metrics (i.e., unexpected pieces of information) while the model is running and without dedicated offline retraining.

The MIRAS framework, as demonstrated by Titans, introduces a meaningful shift toward real-time adaptation. Instead of compressing information into a static state, this architecture actively learns and updates its own parameters as data streams in. This crucial mechanism enables the model to incorporate new, specific details into its core knowledge instantly.

Titans: Learning new context on the fly

An effective learning system requires distinct yet interconnected memory modules, mirroring the human brain's separation of short-term and long-term memory .

While attention mechanisms excel for precise, short-term memory, Titans introduces a novel neural long-term memory module , that, unlike the fixed-size vector or matrix memory in traditional RNNs, acts as a deep neural network (specifically, a multi-layer perceptron ). This memory module provides significantly higher expressive power, allowing the model to summarize large volumes of information without losing important context. The model isn't simply taking notes; it's understanding and synthesizing the entire story.

Crucially, Titans doesn’t just passively store data. It actively learns how to recognize and retain important relationships and conceptual themes that connect tokens across the entire input. A key aspect of this ability is what we call the “surprise metric”. In human psychology, we know we quickly and easily forget routine, expected events but remember things that break the pattern — unexpected, surprising, or highly emotional events.

In the context of Titans, the "surprise metric" is the model detecting a large difference between what it currently remembers and what the new input is telling it.

  • Low surprise : If the new word is "cat" and the model's memory state already expects an animal word, the gradient (surprise) is low. It can safely skip memorizing the word "cat" in its permanent long-term state.
  • High surprise : If the model's memory state is summarizing a serious financial report, and the new input is a picture of a banana peel (the unexpected event), the gradient (surprise) will be very high. This signals that the new input is important or anomalous, and it must be prioritized for permanent storage in the long-term memory module.

The model uses this internal error signal (the gradient) as a mathematical equivalent of saying, "This is unexpected and important!" This allows the Titans architecture to selectively update its long-term memory only with the most novel and context-breaking information, keeping the overall process fast and efficient.

Titans refines this mechanism by incorporating two critical elements:

  1. Momentum : The model considers both "momentary surprise" (the current input) and "past surprise" (the recent context flow). This ensures relevant subsequent information is also captured, even if those tokens are not individually surprising.
  2. Forgetting (weight decay) : To manage the finite capacity of the memory when dealing with extremely long sequences, Titans employ an adaptive weight decay mechanism. This acts as a forgetting gate, allowing the model to discard information that is no longer needed.

MIRAS: A unified view of sequence modeling

Every major breakthrough in sequence modeling — from modern transformers to the new, lightning-fast linear RNNs — is essentially the same thing under the hood: a highly complex associative memory module.

Accordingly, what makes MIRAS both unique and practical is the way it views AI modeling. Instead of seeing diverse architectures, it sees different methods of solving the same problem: efficiently combining new information with old memories without letting the essential concepts be forgotten .

MIRAS defines a sequence model through four key design choices:

  • Memory architecture : The structure that stores information (e.g., a vector, matrix, or a deep multi-layer perceptron, like in Titans).
  • Attentional bias : The internal learning objective the model optimizes that determines what it prioritizes.
  • Retention gate : The memory regularizer. MIRAS reinterprets "forgetting mechanisms" as specific forms of regularization that balance new learning against retaining past knowledge.
  • Memory algorithm : The optimization algorithm used to update the memory.

Transcending the mean squared error paradigm

Virtually all successful existing sequence models rely on mean squared error (MSE) or dot-product similarity for both their bias and retention. This reliance can make models sensitive to outliers and limit their expressive power.

MIRAS transcends this limitation by providing a generative framework to explore a more rich design space informed by the literature in optimization and statistics. This allows for the creation of novel architectures with non-Euclidean objectives and regularization.

Using MIRAS, we created three specific attention-free models:

  • YAAD : We designed this MIRAS variant to be less sensitive to major errors or "outliers" (like a single typo in a large document). It uses a gentler math penalty ( Huber loss ) for mistakes, so it doesn't overreact to one-off issues. This makes the model more robust when the input data is messy or inconsistent.
  • MONETA : This model explores the use of more complex and strict mathematical penalties (called generalized norms ). It investigates whether using these more disciplined rules for both what the model attends to and what it forgets can lead to a more powerful and stable long-term memory system overall.
  • MEMORA : This model focuses on achieving the best possible memory stability by forcing its memory to act like a strict probability map. By using this constraint, it ensures that every time the memory state is updated, the changes are controlled and balanced. This guarantees a clean, stable process for integrating new information.Virtually all successful existing sequence models rely on mean squared error (MSE) or dot-product similarity for both their bias and retention. This reliance can make models sensitive to outliers and limit their expressive power.

Experiments and results

We rigorously compared Titans along with MIRAS variants (YAAD, MONETA, MEMORA) against leading architectures, including Transformer++ , Mamba-2 , and Gated DeltaNet . We further validated versatility by testing Titans on genomic modeling (DNA) and time-series forecasting, proving the architecture generalizes effectively beyond text.

Across both standard language modeling datasets ( C4 , WikiTex t) and zero-shot reasoning tasks ( HellaSwag , PIQA), our models consistently demonstrated higher accuracy and perplexity (a measure of how surprised an LLM is when looking at a piece of text).

The power of deep memory

Ablation studies clearly show that the depth of the memory architecture is crucial. When comparing long-term memory modules of the same size but different depths, modules with deeper memories consistently achieve lower perplexity in language modeling. Furthermore, they exhibit better scaling properties, maintaining performance as the sequence length increases significantly.

Language modeling and efficiency

In language modeling and commonsense reasoning tasks, Titans architectures outperform state-of-the-art linear recurrent models (such as Mamba-2 and Gated DeltaNet) and Transformer++ baselines of comparable sizes. The novel MIRAS variants (MONETA, YAAD, MEMORA) also achieve improved performance compared to these baselines, validating the benefit of exploring robust, non-MSE optimization mechanisms. Importantly, these models maintain efficient, parallelizable training and fast linear inference speeds.

Extreme long-context recall

The most significant advantage of these new architectures is their ability to handle extremely long contexts. This is highlighted in the BABILong benchmark , a task requiring reasoning across facts distributed in extremely long documents. In this challenging setting, Titans outperforms all baselines, including extremely large models like GPT-4, despite having many fewer parameters. Titans further demonstrates the capability to scale effectively to context window sizes larger than 2 million tokens.

Conclusion

The introduction of Titans and the MIRAS framework marks a significant advancement in sequence modeling. By employing deep neural networks as memory modules that learn to memorize as data is coming in, these approaches overcome the limitations of fixed-size recurrent states. Furthermore, MIRAS provides a powerful theoretical unification, revealing the connection between online optimization, associative memory, and architectural design. By moving beyond the standard Euclidean paradigm, this research opens the door to a new generation of sequence models that combine the efficiency of RNNs with the expressive power needed for the era of long-context AI.

Java Hello World, LLVM Edition

Hacker News
www.javaadvent.com
2025-12-07 11:51:02
Comments...
Original Article
Timed out getting readerview for https://www.javaadvent.com/2025/12/java-hello-world-llvm-edition.html

A Journalist Reported From Palestine. YouTube Deleted His Account Claiming He’s an Iranian Agent

Intercept
theintercept.com
2025-12-07 11:00:00
YouTube offered multiple explanations for deleting the account of Robert Inlakesh, who covered Israel’s occupation of the West Bank. The post A Journalist Reported From Palestine. YouTube Deleted His Account Claiming He’s an Iranian Agent appeared first on The Intercept....
Original Article

In February 2024 , without warning, YouTube deleted the account of independent British journalist Robert Inlakesh.

His YouTube page featured dozens of videos, including numerous livestreams documenting Israel’s military occupation of the West Bank. In a decade covering Palestine and Israel, he had captured video of Israeli authorities demolishing Palestinian homes, police harassing Palestinian drivers, and Israeli soldiers shooting at Palestinian civilians and journalists during protests in front of illegal Israeli settlements. In an instant, all of that footage was gone.

In July, YouTube deleted Inlakesh’s private backup account. And in August, Google , YouTube’s parent company, deleted his Google account, including his Gmail and his archive of documents and writings.

The tech giant initially claimed Inlakesh’s account violated YouTube’s community guidelines. Months later, the company justified his account termination by alleging his page contained spam or scam content.

However, when The Intercept inquired further about Inlakesh’s case, nearly two years after his account was deleted, YouTube provided a separate and wholly different explanation for the termination: a connection to an Iranian influence campaign.

YouTube declined to provide evidence to support this claim, stating that the company doesn’t discuss how it detects influence operations. Inlakesh remains unable to make new Google accounts, preventing him from sharing his video journalism on the largest English language video platform.

Inlakesh, now a freelance journalist, acknowledged that from 2019 to 2021 he worked from the London office of the Iranian state-owned media organization Press TV, which is under U.S. sanctions. Even so, Inlakesh said that should not have led to the erasure of his entire YouTube account, the vast majority of which was his own independent content that was posted before or after his time at Press TV.

A public Google document from the month Inlakesh’s account was deleted notes that the company had recently closed more than 30 accounts it alleged were linked to Iran that had posted content critical of Israel and its war on Gaza. The company did not respond when asked specifically if Inlakesh’s account was among those mentioned in the document.

Inlakesh said he felt like he was targeted not due to his former employer but because of his journalism about Palestine, especially amid the increasingly common trend of pro-Israeli censorship among Big Tech companies.

“What are the implications of this, not just for me, but for other journalists?” Inlakesh told The Intercept. “To do this and not to provide me with any information — you’re basically saying I’m a foreign agent of Iran for working with an outlet; that’s the implication. You have to provide some evidence for that. Where’s your documentation?”

Misdirection and Lack of Answers

Over the past couple years, YouTube and Google’s explanations given for the terminations of Inlakesh’s accounts have been inconsistent and vague.

YouTube first accused Inlakesh of “severe or repeated violations of our Community Guidelines.” When a Google employee, Marc Cohen, noticed Inlakesh’s public outcry about his account termination in February 2024, he decided to get involved. Cohen filed a support ticket on Google’s internal issue tracker system, “the Buganizer,” asking why a journalist’s account was deleted. Failing to get an answer internally, Cohen went public with his questions that March. After drawing the attention of the YouTube team on Twitter, he said he eventually received an internal response from Google which claimed that Inlakesh’s account had been terminated owing to “scam, deceptive or spam content.”

Cohen, who resigned from Google later that year over its support of the Israeli government’s genocide in Gaza, said had he not gotten involved, Inlakesh would have been left with even less information.

“They get away with that because they’re Google,” Cohen said. “What are you going to do? Go hire a lawyer and sue Google? You have no choice.”

When Inlakesh’s Gmail account was deleted this year, Google said his account had been “used to impersonate someone or misrepresent yourself,” which Google said is a violation of its policies. Inlakesh appealed three times but was given no response.

Only after The Intercept’s inquiry into Inlakesh’s case did Google shift its response to alleged Iranian influence.

“This creator’s channel was terminated in February 2024 as part of our ongoing investigations into coordinated influence operations backed by the Iranian state,” a YouTube spokesperson told The Intercept. The termination of his channel meant all other accounts associated with Inlakesh, including his backup account, were also deleted, YouTube said.

When The Intercept asked YouTube to elaborate on the reason behind the account deletions, such as which specific content may have flagged the account as being linked to an Iranian state influence operation, a YouTube spokesperson replied that YouTube doesn’t “disclose specifics of how we detect coordinated influence operations,” and instead referred The Intercept to Google’s Threat Analysis Group’s quarterly bulletins . TAG is a team within Google that describes itself as working “to counter government-backed hacking and attacks against Google and our users.”

Google’s Threat Analysis Group’s bulletin from when Inlakesh’s account was first terminated states that in February 2024, a total of 37 YouTube channels were deleted as a result of an “investigation into coordinated influence operations linked to Iran.” Four of these accounts, the document notes, were sharing content which “was critical of the Israeli government and its actions in the ongoing Israel-Gaza war” and had “shared content depicting alleged cyber attacks targeting Israeli organizations.” Google said in the document that the other 33 terminated YouTube channels had shown content “supportive of Iran, Yemen, and Palestine and critical of the US and Israel.”

A Pattern of Censorship

Google has a long-standing and well-documented practice of censoring Palestinian content or content critical of the Israeli government, in addition to evidence of human rights abuses in other conflicts . Such censorship has only exacerbated during Israel’s genocidal war on Gaza,

The company deploys various methods to censor content, such as teams of experts who manually review content, automated systems that flag content, reviews of U.S. sanction and foreign terror organization lists, as well as takedown requests from governments.

For the past decade , Israel’s Cyber Unit has openly run operations to convince companies to delete Palestine-related content from platforms such as YouTube.

Among U.S. allies, Israel had the highest percentage of requests resulting in takedowns on Google platforms, with a nearly 90 percent takedown rate, according to Google’s data since 2011. This rate outpaces countries like France, Germany, the United Kingdom, and Google’s home country, the United States. Absent from Google’s public reports, however, are takedown requests made by individual users, a route often weaponized by the Israeli cyber unit and internally by pro-Israel employees .

The scale of content deleted specifically due to U.S. sanctions is also difficult to quantify since such decisions happen without transparency. A recent investigation by The Intercept revealed that YouTube quietly deleted the accounts of three prominent Palestinian human rights organizations due to the Trump administration’s sanctions against the groups for assisting the International Criminal Court’s war crimes case against Israeli officials. The terminated pages accounted for at least 700 videos erased, many of which spotlighted alleged human rights abuses by the Israeli government.

Dia Kayyali, a technology and human rights consultant, said that in the past several years, as Big Tech platforms have relied more on automated systems that are fed U.S. sanction and terror lists, rights groups have seen an increase in the number of journalists within the Middle East and North Africa region who have had their content related to Palestine removed from YouTube, even when the content they post does not violate the company’s policies. The same could have happened with Inlakesh’s account, Kayyali said.

“And that’s part of the problem with automation — because it just does a really bad job of parsing content — content that could be graphic, anything that has any reference to Hamas,” Kayyali said. Hamas is included within the U.S. foreign terror organization list and Iran remains one of the most sanctioned countries by the U.S. government.

Google and other Big Tech platforms rely heavily on U.S. sanction lists in part to avoid potential liability from the State Department. But such caution is not always warranted, said Mohsen Farshneshani, principal attorney at the Washington, D.C.-based Sanctions Law Center.

Multinational corporations like Google tend to lean toward “overcompliance” with sanction regulations, often deleting content even when it legally is not required to do so, harming journalists and human rights groups, said Farshneshani.

Under U.S. law, in the Berman Amendment to the International Emergency Economic Powers Act, informational materials — in this case, reporting and journalism — are exempt from being subject to sanctions.

“Deleting an entire account is far from what the statutes or the regulations ask of U.S. entities.”

Such a carveout should have protected Inlakesh’s page from being deleted, Farshneshani said. Google likely could have taken down specific videos that raised concern, or demonetized specific videos or the entire account, he said. (Inlakesh said that years before terminating his videos and account, YouTube had demonetized some of his content depicting Israeli military violence.)

“Deleting an entire account is far from what the statutes or the regulations ask of U.S. entities,” Farshneshani said. “The exemption is meant for situations like this. And if these companies are to uphold their part of the bargain as brokers of information for the greater global community, they would do the extra leg work to make sure the stuff stays up.”

While YouTube and Google have not stated whether Inlakesh’s history with Press TV played a factor in the deletion, the Iranian state-funded outlet has long been under Google’s scrutiny. In 2013, Google temporarily deleted Press TV’s YouTube account before permanently deleting the channel in 2019 along with its Gmail account amid the first Trump administration’s sanctions campaign against Iran. The Biden administration in 2021 seized and censored dozens of websites tied to Iran, and in 2023 placed sanctions on Press TV due to Iran’s violent crackdown on anti-government protesters after the in-custody death of Mahsa Amini .

Press TV also has been accused by rights groups and journalists for filming and airing propaganda videos in which individuals detained by Iran are coerced to “ confess ” to alleged crimes in recorded interviews, as a part of the government’s attempts to justify their imprisonment or execution.

Press TV did not respond to The Intercept’s request for comment.

Out of the many videos on his YouTube account, Inlakesh recalled only two being associated with his work for Press TV: a documentary critical of the 2020 Trump deal on Israel–Palestine and a short clip about Republicans’ Islamophobic attacks on Rep. Ilhan Omar, D-Minn., in 2019. The rest either predate or postdate his stint at Press TV.

Press TV’s U.K. YouTube channel at times appears listed as an “associated channel” in archival versions of Inlakesh’s personal YouTube page. A YouTube spokesperson stated that YouTube uses “various signals to determine the relationship between channels linked by ownership for enforcement purposes,” but did not clarify what the specific signals were.

Inlakesh maintained that he had editorial independence while at Press TV and was never directed to post to his personal YouTube page.

Jillian York, the director for international freedom of expression at the Electronic Frontier Foundation, said she understood Google’s need to moderate content, but questioned why it deleted Inlakesh’s account rather than using its policy of labeling state-sponsored content, a system that itself has been plagued with problems . “More labels, more warnings, less censorship,” York said.

“The political climate around Palestine has made it such that a lot of the Silicon Valley-based social media platforms don’t seem particularly willing to ensure that Palestinian content can stay up,” she said.

Killing the Narrative

Inlakesh said he lost several documentaries about Israel and Palestine that were hosted exclusively on YouTube. However, what he lamented most was the loss of footage of his independent coverage from the West Bank, including livestreams that document alleged Israeli military abuses and were not backed up elsewhere.

One such video, he said, was a livestream from a protest at the major Israeli settlement of Beit El on February 11, 2020, against President Donald Trump’s lopsided annexation plan for Israel and Palestine .

Through the haze of tear gas, Inlakesh filmed Israeli soldiers camped out at a nearby hill, aiming their guns at the crowd of mostly children throwing rocks.

“And then you see the children drop,” Inlakesh recalled, followed by the bang of a gunshot. Paramedics rushed over to retrieve the children as Inlakesh followed behind. In all, Inlakesh said he filmed Israeli military gunfire hit three Palestinian children, a likely war crime violation , leaving them with wounds to the arms, legs and torso.

“You’re killing part of the narrative,” Inlakesh said. “You’re actively taking away the public’s ability to assess what happened at a critical moment during the history of the conflict.”

What's the State of Jobseeking/Gigseeking/Roleseeking (Dec 2025)?

Lobsters
lobste.rs
2025-12-07 10:41:48
About a hear ago, I left an employer of a decade so that I could upend my life and move to a new country, polish old skills and build new ones, focus on personal projects, make some art—to just shake things up a bit, because I've never been too comfortable with getting too comfortable. 14 months lat...
Original Article

About a hear ago, I left an employer of a decade so that I could upend my life and move to a new country, polish old skills and build new ones, focus on personal projects, make some art—to just shake things up a bit, because I've never been too comfortable with getting too comfortable. 14 months later, her I am: new time zone, new accomplishments, and reinvigorated curiosity, interests, and enthusiasm. Toned (slightly less sedentary), tanned (pasty), and ready to go.

In the interim, it seems like jobseeking has changed dramatically. How are people looking for gigs today? Specifically people with acute LinkedIn allergies, wanting to work with smaller, skills-concentrated teams? In a leadership capacity in my prior position, I caught just the glimmerings of what I'm now told is now a slop-slurry stew making everyone's—seeker 's or hirer's—a lot less pleasant. How close to the truth is this? We nerds historically have a knack for eventually routing around misery in our own ways, so my hope is that stories have been at least somewhat exaggerated.

So, where does one with an overabundance of curiosity, who just wants to make things with a small-to-medium-sized team of really smart, eclectic weirdos—part time, full time, contract, whatever—look for opportunities in late 2025? What strategies are you all using? I'm personally interested in remote opportunities in the EU and US, but I don't want to make this all about me. I'm sure others in different locales wonder similarly. Any and all pointers appreciated.

A Struct Sockaddr Sequel

Hacker News
lwn.net
2025-12-07 09:06:39
Comments...
Original Article

One of the many objectives of the Linux Kernel Self-Protection Project (KSPP) , which just completed ten years of work , is to ensure that all array references can be bounds-checked, even in the case of flexible array members, the size of which is not known at compile time. One of the most challenging flexible array members in the kernel is not even declared as such. Almost exactly one year ago, LWN looked at the effort to increase safety around the networking subsystem's heavily used

sockaddr

structure. One year later, Kees Cook is still looking for a way to bring this work to a close.

In short, the problem is that struct sockaddr is traditionally defined as: 

    struct sockaddr {
        short sa_family;
	char sa_data[14];
    };

The sa_data field was more than large enough to hold a network address in the early 1980s when this structure was first defined for BSD Unix, but it is not large enough now. As a result, a great deal of code, in both the kernel and user space, passes around struct sockaddr pointers that, in truth, point to different structures with more space for the addresses they need to hold. In other words, sa_data is being treated as a flexible array member, even though it is not declared as one. The prevalence of struct sockaddr has thrown a spanner into the works of many attempts to better check the uses of array members in structures.

$ sudo subscribe today

Subscribe today and elevate your LWN privileges. You’ll have access to all of LWN’s high-quality articles as soon as they’re published, and help support LWN in the process. Act now and you can start with a free trial subscription.

At the end of last year's episode, much of the kernel had been changed to use struct sockaddr_storage (actually implemented as struct __kernel_sockaddr_storage ), which has a data array large enough to hold any known network address. An attempt was made to change the definition struct sockaddr to make its sa_data field into an explicit flexible array member, but that work ran into a minor snag. There are many places in the kernel where struct sockaddr is embedded within another structure. In most of these cases, sa_data is not treated as a flexible array member, so developers have freely embedded struct sockaddr anywhere within the containing structure, often not at the end.

If sa_data is redefined as a flexible array member, the compiler no longer knows how large the structure will actually be. That, in turn, means that the compiler does not know how to lay out a structure containing struct sockaddr , so it guesses and emits a warning. Or, in the case of a kernel build, tens of thousands of warnings. Kernel developers, as it turns out, would rather face the prospect of an array overflow than a warning flood of that magnitude, so this work came to a halt.

One possible solution would be to replace embedded struct sockaddr fields with struct sockaddr_storage , eliminating the flexible array member. But that would bloat the containing structures with memory that is not needed, so that approach is not popular either.

Instead, Cook is working on a patch series that introduces yet another struct sockaddr variant: 

    struct sockaddr_unsized {
	__kernel_sa_family_t	sa_family;	/* address family, AF_xxx */
	char			sa_data[];	/* flexible address data */
    };

Its purpose is to be used in internal network-subsystem interfaces where the size of sa_data needs to be flexible, but where its actual size is also known. For example, the bind() method in struct proto_ops is defined as: 

    int	(*bind) (struct socket *sock,
		 struct sockaddr *myaddr,
		 int sockaddr_len);

The type of myaddr can be changed to struct sockaddr_unsized * since sockaddr_len gives the real size of the sa_data array. Cook's patch series does many such replacements, eliminating the use of variably sized sockaddr structures in the networking subsystem. With that done, there are no more uses of struct sockaddr that read beyond the 14-byte sa_data array. As a result, struct sockaddr can be reverted to its classic, non-flexible definition, and array bounds checking can be applied to code using that structure.

That change is enough to make all of those warnings go away, so many would likely see it as a good stopping point. There is still, though, the matter of all those sockaddr_unsized structures, any of which might be the source of a catastrophic overflow at some point. So, once the dust settles from this work, we are likely to see some attention paid to implementing bounds checking for those structures. One possible approach mentioned in the patch set is to eventually add an sa_data_len field, so that the structure would contain the length of its sa_data array. That would make it easy to document the relationship between the fields with the counted_by() annotation , enabling the compiler to insert bounds checks.

While the ability to write new code in Rust holds promise for reducing the number of memory-safety bugs introduced into the kernel, the simple fact is that the kernel contains a huge amount of C code that will not be going away anytime soon. Anything that can be done to make that code safer is thus welcome. The many variations of struct sockaddr that have made the rounds may seem silly to some, but they are a part of the process of bringing a bit of safety to an API that was defined over 40 years ago. Ten years of KSPP have made the kernel safer, but the job is far from done.

Index entries for this article
Kernel Flexible array members
Kernel Networking/struct sockaddr
Kernel Releases/6.19

Structural inheritance doesn't work where you expect it to

Lobsters
trynova.dev
2025-12-07 08:37:38
Comments...
Original Article

Okay, okay, I know what you're thinking: what's this data-oriented design zealot doing, opening their mouth about object-oriented programming? Don't they know that OOP is alive, well, and keeps spawning more successful projects than any other programming paradigm ever did, like Shub-Niggurath the Black Goat of the Woods with a Thousand Young spawns progeny? And don't they know that basically everything in programming is actually OOP, including their favourite ML and DOD features?

So, let's start off with some clarifications:

  1. I'm not announcing the death of object-oriented programming, I just liked the title. Yes, it's effectively click-bait and I don't apologise because I like it and because it is an explicit reference.
  2. I don't think object-oriented programming is the worst thing in the world, and I use things like classes regularly in the TypeScript I write as a means of putting food on the table.
  3. I am about to complain about one particular programming language feature that, according to the Internet commentariat, is purely an implementation detail and has nothing to do with object-oriented programming. So, if you see yourself or your favourite programming language in the picture being drawn by this blog post, remember then that this is purely a coincidence and no object-oriented languages are being harmed by this blog post – only implementation details.

The thing we're talking about is of course structural inheritance. "What is structural inheritance and how is it different from class inheritance in this programming language?", I hear you cry. Let's define the term by counter-example first: interface inheritance or typeclassing is a form of inheritance where a "subclass" only inherits a set of behaviours from the "superclass"; structural inheritance is then a form of inheritance where both behaviours and the form of a superclass are both inherited.

Because we're talking about an implementation detail, let's try to make this very concrete: structural inheritance (as I am talking about it) means inheriting the memory layout of the superclass, such that a pointer to a subclass instance can be directly used as a pointer to a superclass instance: reading inbounds offsets from such a pointer will read exactly the same (kind of) data regardless of if the pointer points to a subclass or superclass instance.

This form of inheritance is what I will rail against today. Now, you've been warned but I'll reiterate for safety: if you see yourself or your favourite programming language in this definition, it is merely a coincidence. In particular, languages that are not at all pointed to here include JavaScript 1 and C++ 2 . Now, let's grind that ax!

Protego!

Structural inheritance isn't all bad: it is exceedingly convenient, easy to understand, and works great for many things. A commonly cited example is GUIs and how excellent inheritance is for modeling them, with structural inheritance being implied by this praise.

A (hopefully uncontroversial) case for structural inheritance's superiority in a GUI programming setting might come from defining the position of an element in a GUI. The element needs to be drawn somewhere , so left , top , width , and height properties of one form or another are simply required for all elements. It then makes sense to put these properties in the base class of the GUI library. 

class BaseElement {
  left: u32;
  top: u32;
  width: u32;
  height: u32;
}

class ButtonElement extends BaseElement {
  toggled: boolean;
}

class TextElement extends BaseElement {
  value: string;
}

In terms of memory layout, this sort of structural inheritance hierarchy means that classes look something like this in memory: 

const BaseLayout = [u32, u32, u32, u32];
const ButtonLayout = [...BaseLayout, boolean];
const TextLayout = [...BaseLayout, string];

The base class fields are placed at the start of the subclass' memory layout; note that the fields of the subclass are at the very end, despite probably being the most important fields of the class. It isn't a given, but it can be slower to load far-off fields.

This all looks pretty fine. Let's keep going then.

Reducto!

If we think about a text element in a GUI, it's fairly common that it will only take up the amount of space that it requires, up to some maximum at which point it gets cut off with maybe an ellipsis added at the end. Now, how would we do that in our GUI here?

The first obvious step is to introduce an extra wrapper around a TextElement : that wrapping element defines the maximum space that the text can take, after which ellipsis appears. Our TextElement itself then probably still has a user-definable position somehow, but its width and height values become dependent on the actual text value. Depending on the details of the system, at this point the actual width and height properties might become unused as it may be cheap enough to simply calculate the resultant size of the text from the string dynamically. Even if that is not the case, the properties at the very least become merely caches of calculated values: they are not the source of truth that they once were.

Letting that simmer, what if we introduce a flexible box or grid layout element into the GUI? The properties of our base element class might become entirely irrelevant when placed in such a layout, yet they remain an integral part of the memory layout of each instance. That is memory being used to store no information whatsoever, just pure waste.

This isn't a deal-breaker of course: we can introduce a further base class (maybe call it a Node ?) that doesn't have these properties at all and can then inherit from that to avoid the properties when we know they're not needed. This is all just engineering and trade-offs, nothing fundamental that couldn't be fixed. But at least we found an issue in the way we built our inheritance hierarchy, if nothing else.

The Curse of Structural Inheritance

Reusing the original GUI example, let's say our GUI gets an BinaryElement extends BaseElement subclass that displays 32 bits in 0s and 1s using a monospace font: calculating the width of this element is trivial, simply multiply the monospace font's character width by 32. Now, it is clear that the width property from our BaseElement is going to be unnecessary: we really don't need to store the result of this calculation in memory, and we'd much rather reuse that width property's memory to store the 32 bits that the BinaryElement is displaying. Yet, this cannot be done.

Our BinaryElement is clearly an element in terms of interface inheritance: it has ways to display some data in some location on the screen. But it is also better defined, or has a more limited use-case, than a general element is: its contents is known to be a binary number of 32 bits, and its size is known to be a static value (modulo the monospace font). Yet, no matter how hard we try, we cannot make the BinaryElement smaller than a BaseElement is; rather it must be bigger to fit the 32 bits it displays.

Now, it would be fair to say that the inheritance hierarchy here is badly designed, wrong, doesn't actually make sense, and has nothing to do with object-oriented programming in the first place anyway. Also, wasting the 8 bytes of the width and height properties is meaningless in the grand scheme of things and thus the larger waste of time is thinking about this whole issue at a all. And maybe it is, but that is not a universal truth: this is engineering, and it is all about trade-offs. In a real-world inheritance hierarchy you'll probably find that much more memory is being wasted, and the effect of it is not insigificant. At some point you'll find that the cost of not thinking about this issue is too high to bear.

This is the Curse of Structural Inheritance that I now offer for you to understand. Whenever you look at a structural inheritance hierarchy, you will see the effects of the curse: you will see the subclasses carrying around all of the superclasses data fields at the start of their memory layout, while subclass fields are relegated to the very end of the instance memory. You'll see the subclass often barely even touching the superclass fields, as many of its behaviours have been overridden in such a way as to make the generic superclass fields' data redundant. You will see all this, yet be unable to do anything to help it. It is already too late to help it.

Structural inheritance in action

Okay, maybe structural inheritance has a downside when modeling behaviourally related but otherwise unconnected objects. But what if the data is very clearly linked, surely then structural inheritance is great? This is the question I was asked recently: specifically, I was asked why I had not written two classes, GraphReadWrite and GraphReadOnly , using inheritance but had instead opted to create two entirely separate classes that largely shared the same internal field structure and copy-pasted a good bit of methods on top! Surely this is a case for inheritance?

The read-write class I wrote is used to create a graph step by step, and once the creation is done it is "frozen" to produce a read-only instance: these classes could alternatively be called a GraphBuilder and Graph , but our GraphReadWrite still internally contains a graph. So while the graph inside of GraphReadWrite may be incomplete, it is still a graph with nodes and edges and methods to read them just like the final GraphReadOnly has, not a builder with just add methods and a final build step that does all the work. So surely it is stupid to duplicate code across the two classes? I had to pause to really consider this question, but I did indeed find that I had an answer. Let's approach it by trying to write out the inheritance hierarchy.

Oh, and do stop me if you know the answer already: it seems that trying to model read-write and read-only class variants through inheritance has already been tried in enough places so as to see that it is a fool's errand. I of course had not found this out yet and so had to find my own answer.

So let's start by doing the logical thing and inheriting GraphReadOnly extends GraphReadWrite . This seems clear and simple, we have a superclass that includes all the fields and methods for storing a graph and making changes to it, and our subclass simply makes those fields readonly and overrides all the methods of the superclass to throw exceptions if called. Except that the subclass cannot re-interpret the superclass fields, not even by changing them from read-write to read-only: it can only pretend they are read-only but it cannot actually change their functionality. Furthermore, a subclass is a valid instance of a superclass, so any superclass methods can be called on a subclass instance: this means that the read-write methods of GraphReadWrite can still be called on a GraphReadOnly . The read-only instance we have is not actually all that read-only, we're just trying to pretend it is.

And I'm sure you saw the effects of the Curse of Structural Inheritance as well: our superclass has fields "for storing a graph and making changes to it ". There is extra data in the GraphReadWrite superclass that is unnecessary when we know that the graph is read-only (specifically, this was a couple of hash maps for efficient insertion). The GraphReadOnly is paying the cost of read-write features it never wants to use, while also having no actual guarantees that it truly is read-only. Obviously this is no way to build a reliable system, so we cannot go this route.

So, what we have to do is the opposite: GraphReadWrite extends GraphReadOnly . Now everything works wonderfully, the GraphReadOnly superclass only includes those fields that are needed to store a graph and none of the fields needed to mutate it efficiently, while the subclass gets to introduce those extra fields as needed (never mind that it still only has to add them at the end of the memory layout despite them probably being the first thing accessed when a write API is called). GraphReadOnly also gets to define its fields as read-only while GraphReadWrite then mutates them inside its own methods. ... Wait, what?! Oh right! Now the subclass has to break the read-only invariants of the superclass to do its work: depending on the language this might appear as const-casting or // @ts-expect-error comments. Any methods defined on the superclass that rely on the read-only nature of these fields for correctness are liable to bug out if any re-entrant code paths appear with subclass instances. The language fights the implementation, and possibly the best way to resolve the Gordian Knot is to simply give up on defining the fields as read-only in the first place. At that point you are left with a GraphReadOnly class that is read-only in name, but with no actual guarantees given. The entire point of the class has arguably been lost in our attempt at using inheritance to model read-only and read-write variants of the same data.

A third option would be to define a common base class between GraphReadOnly and GraphReadWrite and have both classes inherit from that: depending on the language this could remove some issues but I at cannot directly recall a feature of inheriting a superclass as read-only. In effect, the common base class would still need to be defined as read-write and GraphReadOnly would again have to live with being read-only in name only.

At the end of the day, structural inheritance is not a tool that I like to use all that much. When in doubt, I prefer interface inheritance and when I need structural sharing, I tend to opt for either structural composition (including a struct in your own definition; this doesn't exist in JavaScript), indirect composition (including a pointer to a struct; this is equivalent to storing an object inside an object in JavaScript), or a manual mix of the two.

Maybe consider doing the same the next time you see the Curse of Structural Inheritance lingering in your code base?

  1. While JavaScript's prototypal inheritance is more akin to interface inheritance, the class keyword and field definitions especially but also traditional constructor functions that assign properties to this are a form of structural inheritance. ↩

  2. The grand old man of structural inheritance and object-oriented programming as we mostly know and love it today! My expertise isn't here, so I should shut up but I'll just point out that non-abstract C++ classes are explicitly a form of structural inheritance. ↩

The Wild West of Post-POSIX IO Interfaces

Lobsters
youtu.be
2025-12-07 08:26:53
Deep dive into pervasiveness of ring buffer pattern across modern hw and VMs Comments...
Original Article

Sinterklaas Likes Playing On The Game Boy

Lobsters
brainbaking.com
2025-12-07 07:33:28
Comments...
Original Article

Today marks the yearly departure of Sinterklaas who, together with his faithful friend Zwarte Piet , makes his way back to sunny Spain—by horse and steamboat, of course. The festivities on the sixth of December are not to celebrate his departure but to celebrate the name day of Saint Nicholas, patron saint of the children, on which Sinterklaas is based.

For those of you outside of Europe thinking “Hey, Sinterklaas sounds a lot like our Santa Claus”, well guess what: Sinterklaas was here first and your Santa is just a poor imitation.

In The Netherlands and Belgium, Sinterklaas is a very popular tradition, where during the run up to today, even from half of November, children can put an empty shoe somewhere near the mantelpiece in the hope of the Sint (“the saint”) and Piet visiting the house (via the roof and said mantelpiece) to drop some candy in the children’s shoes. This is usually a combination of marzipan, onze-lieve-vrouw guimauves (harder marshmellows shaped like Mary), nic-nac letterkoekjes , speculaas ( spiced cookies ), and of course various chocolate figures.

The popularity of Sinterklaas inevitably also means TV shows, live shows, specialized pop-up shops, school parties, and more. In the early nineties, the then Belgische Radio- en Televisieomroep (BRT) broadcaster hosted two seasons of the Dag Sinterklaas show featuring Jan Decleir as the Sint, Frans Van der Aa as Zwarte Piet, and Bart Peeters as the innocent visitor asking nosy questions on how the duo operates.

Like many Flemish eighties/nineties kids, Dag Sinterklaas is permanently burned into my brain as part of my youth. The episode called Speelgoed (toys) from the second season is especially memorable for me, as we catch the Goedheiligen Man (The Good Saint) playing… on a Game Boy!

Jan Decleir as Sinterklaas, trying to figure out a shoot-em-up on the Game Boy. Copyright BRT, 1993.

In the episode, Sinterklaas is annoyed by the beeps and boops coming from Zwarte Piet’s Game Boy. Piet is usually portrayed as a (too) playful character that likes to fool around instead of doing the serious stuff such as reading the Spanish newspapers and updating the Dakenkaart (rooftop chart) needed to navigate the rooftops when dropping off presents. While Bart visits, Sinterklaas showcases that “simple toys” are much more enjoyable. He encourages them to play with dusty old dolls and a toll. Eventually, Piet and Bart make it outside whilst playing horse, only to catch the Sint grabbing Piet’s Game Boy to figure out for himself what these so-called compinuter spelletjes (computer games) are about. Hilarious.

Of course, that was the perfect advert for Nintendo’s handheld, especially considering the upcoming Christmas holiday period. In 1993, lots of amazing Game Boy games were released, including Link’s Awakening , Kirby’s Pinball Land , Duck Tales 2 , Turtles III: Radical Rescue , and Tetris 2 . It would be next to impossible to go after the Flemish sales data of the machine to try and prove a correlation, but if the Sint likes playing on the Game Boy, and the Sint is thé person that gets to decide what kids can play with, then why bother getting your kid a Game Gear, right? Sorry, Sega.

Perhaps I even got a Game Boy game thrown down the chimney, I can’t remember. All I can remember is the chocolate, marzipan, and VHS tapes of Disney movies.

I have searched high and low for a Dutch Club Nintendo Magazine that contains a message from the Sint and came up empty, but Volume 2 Issue 6 in 1990 contained a lovely letter from Santa Mario:

A partial of a Christmas letter from Mario in the Dutch Club Nintendo Magazine, 1990. Copyright Nintendo.

Replace the goofy Christmas hat with the mijter (mitra) of Sinterklaas, add a staff, and we’re there.

Dag Sinterklaas is undeniably a local cult hit. The DVDs are nowhere to be found, and the few copies surfacing the local second hand )e)markets go for outrageous prices. Cherishing our copy, this year is the first year we watched the episodes together with our daughter. She doesn’t have the patience to sit through some of the longer ones but it’s a giant nostalgic injection seeing Jan and Frans back in action.

BRT—now VRT; Flemish instead of Belgian—aired the series every single year until 2018. In 2019, because of the ageing image quality (and probably the emerging woke culture), twenty new episodes were produced. However, in my view, Wim Opbrouck never managed to truly capture the Sint’s spirit like Jan did, and Jonas Van Thielen as Zwarte Piet is just not as funny as Frans.

So we’ll be stuck in Dag Sinterklaas 1992-1993 mode for the next eight or so year, until our kids realize the big ruse. And even then. I will be keeping up the tradition.

retro gameboy sinterklaas

What I learnt about making websites by reading two thousand web pages

Lobsters
alexwlchan.net
2025-12-07 07:28:03
Comments...
Original Article

Over the past year, I built a web archive of over two thousand web pages – my own copy of everything I’ve bookmarked in the last fifteen years. I saved each one by hand, reading and editing the HTML to build a self-contained, standalone copy of each web page.

These web pages were made by other people, many using tools and techniques I didn’t recognise. What started as an exercise in preservation became an unexpected lesson in coding: I was getting a crash course in how the web is made. Reading somebody else’s code is a great way to learn, and I was reading a lot of somebody else’s code.

In this post, I’ll show you some of what I learnt about making websites: how to write thoughtful HTML, new-to-me features of CSS, and some quirks and relics of the web.

This article is the third in a four part bookmarking mini-series:

  1. Creating a static site for all my bookmarks – why I bookmark, why I use a static site, and how it works.
  2. Building a personal archive of the web, the slow way – how I built a web archive by hand, the tradeoffs between manual and automated archiving, and what I learnt about preserving the web.
  3. Learning how to make websites by reading two thousand web pages (this article)
  4. My favourite websites from my bookmark collection – websites that change randomly, that mirror the real world, or even follow the moon and the sun, plus my all-time favourite website design.

I know I’ve read a list of HTML tags in reference documents and blog posts, but there are some tags I’d forgotten, misunderstood, or never seen used in the wild. Reading thousands of real-world pages gave me a better sense of how these tags are actually used, and when they’re useful.

The <aside> element

MDN describes <aside> as “a portion of a document whose content is only indirectly related to the document’s main content”. That’s vague enough that I was never quite sure when to use it.

In the web pages I read, I saw <aside> used in the middle of larger articles, for things like ads, newsletter sign ups, pull quotes, or links to related articles. I don’t have any of those elements on my site, but now I have a stronger mental model of where to use <aside> . I find concrete examples more useful than abstract definitions.

I also saw a couple of sites using the <ins> (inserted text) element for ads, but I think <aside> is a better semantic fit.

The <mark> element

The <mark> element highlights text, typically with a yellow background . It’s useful for drawing visual attention to a phrase, and I suspect it’s helpful for screen readers and parsing tools as well.

I saw it used in Medium to show reader highlights, and I’ve started using it in code samples when I want to call out specific lines.

The <section> element

The <section> tag is a useful way to group content on a page – more meaningful than a generic <div> . I’d forgotten about it, although I use similar tags like <article> and <main> . Seeing it used across different sites reminded me it exists, and I’ve since added it to a few projects.

The <hgroup> (heading group) element

The <hgroup> tag is for grouping a heading with related metadata, like a title and a publication date: 

<hgroup>
  <h1>All about web bookmarking</h1>
  <p>Posted 16 March 2025</p>
</hgroup>

This is another tag I’d forgotten, which I’ve started using for the headings on this site.

The <video> element

The <video> tag is used to embed videos in a web page. It’s a tag I’ve known about for a long time – I still remember reading Kroc Camen’s article Video is for Everybody in 2010, back when Flash was being replaced as dominant way to watch video on the web.

While building my web archive, I replaced a lot of custom video players with <video> elements and local copies of the videos. This was my first time using the tag in anger, not just in examples.

One mistake I kept making was forgetting to close the tag, or trying to make it self-closing: 

<!-- this is wrong -->
<video controls src="videos/Big_Buck_Bunny.mp4"/>

It looks like <img> , which is self-closing, but <video> can have child elements, so you have to explicitly close it with </video> .

The <progress> indicator element

The <progress> element shows a progress indicator. I saw it on a number of sites that publish longer articles – they used a progress bar to show you how far you’d read. 

<label for="file">Progress:</label>
<progress id="file" max="100" value="70">70%</progress>

70%

I don’t have a use for it right now, but I like the idea of getting OS-native progress bars in HTML – no custom JavaScript or CSS required.

The <base> element

The <base> element specifies the base URL to use for any relative URLs in a document. For example, in this document: 

<base href="https://example.com/">

<img src="/pictures/cat.jpg">

the image will be loaded from https://example.com/pictures/cat.jpg .

It’s still not clear to me when you should use <base> , or what the benefits are (aside from making your URLs a bit shorter), but it’s something I’ll keep an eye out for in future projects.

Clever corners of CSS

The CSS @import rule

CSS has @import rules , which allow one stylesheet to load another: 

@import "fonts.css";

I’ve used @import in Sass, but I only just realised it’s now a feature of vanilla CSS – and one that’s widely used. The CSS for this website is small enough that I bundle it into a single file for serving over HTTP (a mere 13KB), but I’ve started using @import for static websites I load from my local filesystem, and I can imagine it being useful for larger projects.

One feature I’d find useful is conditional imports based on selectors. You can already do conditional imports based on a media query (“only load these styles on a narrow screen”) and something similar for selectors would be useful too (for example, “only load these styles if a particular class is visible”). I have some longer rules that aren’t needed on every page, like styles for syntax highlighting, and it would be nice to load them only when required.

[attr$=value] is a CSS selector for suffix values

While reading Home Sweet Homepage , I found a CSS selector I didn’t understand: 

img[src$="page01/image2.png"] {
  left: 713px;
  top:  902px;
}

This $= syntax is a bit of CSS that selects elements whose src attribute ends with page01/image2.png . It’s one of a several attribute selectors that I hadn’t seen before – you can also match exact values, prefixes, or words in space-separated lists. You can also control whether you want case-sensitive or -insensitive matching.

You can create inner box shadows with inset

Here’s a link style from an old copy of the Entertainment Weekly website: 

a { box-shadow: inset 0 -6px 0 #b0e3fb; }

A link on EW.com

The inset keyword was new to me: it draws the shadow inside the box, rather than outside. In this case, they’re setting offset-x=0 , offset-y=-6px and blur-radius=0 to create a solid stripe that appears behind the link text – like a highlighter running underneath it.

If you want something that looks more shadow-like, here are two boxes that show the inner/outer shadow with a blur radius:

I don’t have an immediate use for this, but I like the effect, and the subtle sense of depth it creates. The contents of the box with inner-shadow looks like it’s below the page, while the box with outer-shadow floats above it.

For images that get bigger, cursor: zoom-in can show a magnifying glass

On gallery websites, I often saw this CSS rule used for images that link to a larger version: 

cursor: zoom-in;

Instead of using cursor: pointer; (the typical hand icon for links), this shows a magnifying glass icon – a subtle cue that clicking will zoom or expand the image.

Here’s a quick comparison:

I knew about the cursor property , but I’d never thought to use it that way. It’s a nice touch, and I want to use it the next time I build a gallery.

Writing thoughtful HTML

The order of elements

My web pages have a simple one column design: a header at the top, content in the middle, a footer at the bottom. I mirror that order in my HTML, because it feels a natural structure.

I’d never thought about how to order the HTML elements in more complex layouts, when there isn’t such a clear direction. For example, many websites have a sidebar that sits alongside the main content. Which comes first in the HTML?

I don’t have a firm answers, but reading how other people structure their HTML got me thinking. I noticed several pages that put the sidebar at the very end of the HTML, then used CSS to position it visually alongside the content. That way, the main content appears earlier in the HTML file, which means it can load and become readable sooner.

It’s something I want to consider next time I’m building a more complex page.

I saw a lot of websites (mostly WordPress) that used HTML comments to mark the end of containers with a lot of content. For example: 

<div id="primary">
  <main id="main">
    …
  </main><!-- #main -->
</div><!-- #primary -->

These comments made the HTML much easier to read – I could see exactly where each component started and ended.

I like this idea, and I’m tempted to use it in my more complex projects. I can imagine this being especially helpful in template files, where HTML is mixed with template markup in a way that might confuse code folding , or make the structure harder to follow.

The data-href attribute in <style> tags

Here’s a similar idea: I saw a number of sites set a data-href attribute on their <style> tags, as a way to indicate the source of the CSS. Something like: 

<style data-href="https://example.com/style.css">

I imagine this could be useful for developers working on that page, to help them find where they need to make changes to that <style> tag.

Translated pages with <link rel="alternate"> and hreflang

I saw a few web pages with translated versions, and they used <link> tags with rel="alternate" and an hreflang attribute to point to those translations. Here’s an example from a Panic article , which is available in both US English and Japanese: 

<link rel="alternate" hreflang="en-us" href="https://blog.panic.com/firewatch-demo-day-at-gdc/">
<link rel="alternate" hreflang="ja"    href="https://blog.panic.com/ja/firewatch-demo-day-at-gdc-j/">

This seems to be for the benefit of search engines and other automated tools, not web browsers. If your web browser is configured to prefer Japanese, you’d see a link to the Japanese version in search results – but if you open the English URL directly, you won’t be redirected.

This makes sense to me – translations can differ in content, and some information might only be available in one language. It would be annoying if you couldn’t choose which version you wanted.

Panic’s article includes a third <link rel="alternate"> tag: 

<link rel="alternate" hreflang="x-default" href="https://blog.panic.com/firewatch-demo-day-at-gdc/">

This x-default value is a fallback, used when there’s no better match for the user’s language. For example, if you used a French search engine, you’d be directed to this URL because there isn’t a French translation.

Almost every website I’ve worked has been English-only, so internationalisation is a part of the web I know very little about.

Fetching resources faster with <link rel="preload">

I saw a lot of websites that with <link rel="preload"> tags in their <head> . This tells the browser about resources that will be needed soon, so it should start fetching them immediately.

Here’s an example from this site: 

<link rel="preload" href="https://alexwlchan.net/theme/white-waves-transparent.png" as="image" type="image/png"/>

That image is used as a background texture in my CSS file. Normally, the browser would have to download and parse the CSS before it even knows about the image – which means a delay before it starts loading it. By preloading the image, the browser can begin downloading the image in parallel with the CSS file, so it’s already in progress when the browser reads the CSS.

The difference is probably imperceptible on a fast connection, but it is a performance improvement – and as long as you scope the preloads correctly, there’s little downside. (Scoping means ensuring you don’t preload resources that aren’t used).

I saw some sites use DNS prefetching , which is a similar idea. The rel="dns-prefetch" attribute tells the browser about domains it’ll fetch resources from soon, so it should begin DNS resolution early. The most common example was websites using Google Fonts: 

<link rel="dns-prefetch" href="https://fonts.googleapis.com/" />

I only added preload tags to my site a few weeks ago . I’d seen them in other web pages, but I didn’t appreciate the value until I wrote one of my own.

Quirks and relics

There are still lots of <!--[if IE]> comments

Old versions of Internet Explorer supported conditional comments , which allowed developers to add IE-specific behaviour to their pages. Internet Explorer would render the contents of the comment as HTML, while other browsers ignored it. This was a common workaround for deficiencies in IE, when pages needed specific markup or styles to render correctly.

Here’s an example, where the developer adds an IE-specific style to fix a layout issue: 

<!--[if IE]>
  <style>
    /* old IE unsupported flexbox fixes */
    .greedy-nav .site-title {
      padding-right: 3em;
    }
  </style>
<![endif]-->

Developers could also target specific versions of IE: 

<!--[if lte IE 7]><link rel="stylesheet" href="/css/ie.css"><![endif]-->

Some websites even used conditional comments to display warnings and encourage users to upgrade, like this message which that’s still present on the RedMonk website today: 

<!--[if IE]>
  <div class="alert alert-warning">
    You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.
  </div>
<![endif]-->

This syntax was already disappearing by the time I started building websites – support for conditional comments was removed in Internet Explorer 10, released in 2012, the same year that Google Chrome became the most-used browser worldwide. I never wrote one of these comments, but I saw lots of them in archived web pages.

These comments are a relic of an earlier web. Most websites have removed them, but they live on in web archives, and in the memories of web developers who remember the bad old days of IE6.

Templates in <script> tags with a non-standard type attribute

I came across a few pages using <script> tags with a type attribute that I didn’t recognise. Here’s a simple example: 

<script type="text/x-handlebars-template" id="loading_animation">
  <div class="loading_animation pulsing <%= extra_class %> "><div></div></div>
</script>

Browsers ignore <script> tags with an unrecognised type – they don’t run them, and they don’t render their contents. Developers have used this as a way to include HTML templates in their pages, which JavaScript could extract and use later.

This trick was so widespread that HTML introduced a dedicated <template> tag element for the same purpose. It’s been in all the major browsers for years, but there are still instances of this old technique floating around the web.

Browsers won’t load external file:// resources from file:// pages

Because my static archives are saved as plain HTML files on disk, I often open them directly using the file:// protocol, rather than serving them over HTTP. This mostly works fine – but I ran into a few cases where pages behave differently depending on how they’re loaded.

One example is the SVG <use> element . Some sites I saved use SVG sprite sheets for social media icons, with markup like: 

<use href="sprite.svg#logo-icon"></use>

This works over http:// , but when loaded via file:// , it silently fails – the icons don’t show up.

This turns out to be a security restriction. When a file:// page tries to load another file:// resource, modern browsers treat it as a cross-origin request and block it. This is to prevent a malicious downloaded HTML file from snooping around your local disk .

It took me a while to figure this out. At first, all I got was a missing icon. I could see an error in my browser console, but it was a bit vague – it just said I couldn’t load the file for “security reasons”.

Then I dropped this snippet into my dev tools console: 

fetch("sprite.svg")
  .then(response => console.log("Fetch succeeded:", response))
  .catch(error => console.error("Fetch failed:", error));

It gave me a different error message, one that explicitly mentioned cross-origin requesting sharing: “CORS request not http” . This gave me something I could look up, and led me to the answer.

This is easy to work around – if I spin up a local web server (like Python’s http.server ), I can open the page over HTTP and everything loads correctly.

What does GPT stand for in attributes?

Thanks to the meteoric rise of ChatGPT, I’ve come to associate the acronym “GPT” with large language models (LLMs) – it stands for Generative Pre-trained Transformer .

That means I was quite surprised to see “GPT” crop up on web pages that predate the widespread use of generative AI. It showed up in HTML attributes like this: 

<div id="div-gpt-ad-1481124643331-2">

I discovered that “GPT” also stands for Google Publisher Tag , part of Google’s ad infrastructure. I’m not sure exactly what these tags were doing – and since I stripped all the ads out of my web archive, they’re not doing anything now – but it was clearly ad-related.

What’s the instapaper_ignore class?

I found some pages that use the instapaper_ignore CSS class to hide certain content. Here’s an example from an Atlantic article I saved in 2017: 

<aside class="pullquote instapaper_ignore">
  Somewhere at Google there is a database containing 25 million books and nobody is allowed to read them.
</aside>

Instapaper is a “read later” service – you save an article that looks interesting, and later you can read it in the Instapaper app. Part of the app is a text parser that tries to extract the article’s text, stripping away junk or clutter.

The instapaper_ignore class is a way for publishers to control what that parser includes. From a blog post in 2010 :

Additionally, the Instapaper text parser will support some standard CSS class names to instruct it:

  • instapaper_body : This element is the body container.
  • instapaper_ignore : These elements, when inside the body container, should be removed from the text parser’s output.

In this example, the element is a pull quote – a repeated line from the article, styled to stand out. On the full web page, it works. But in the unstyled Instapaper view, it would just look like a duplicate sentence. It makes sense that the Atlantic wouldn’t want it to appear in that context.

Only a handful of pages I’ve saved ever used instapaper_ignore , and even fewer are still using it today. I don’t even know if Instapaper’s parser still looks for it.

This stood out to me because I was an avid Instapaper user for a long time. I deleted my account years ago, and I don’t hear much about “read later” apps these days – but then I stumble across a quiet little relic like this, buried in the HTML.

I found a bug in the WebKit developer tools

Safari is my regular browser, and I was using it to preview pages as I saved them to my archive. While I was archiving one of Jeffrey Zeldman’s posts , I was struggling to understand how some of his CSS worked. I could see the rule in my developer tools, but I couldn’t figure out why it was behaving the way it was.

Eventually, I discovered the problem: a bug in WebKit’s developer tools was introducing whitespace that changed the meaning of the CSS.

For example, suppose the server sends this minifed CSS rule: 

body>*:not(.black){color:green;}

WebKit’s dev tools prettify it like this: 

body > * :not(.black) {
    color: green;
}

But these aren’t equivalent!

  • The original rule matches direct children of <body> that don’t have the black class.
  • The prettified version matches any descendant of <body> that doesn’t have the black class and that isn’t a direct child.

The CSS renders correctly on the page, but the bug means the Web Inspector can show something subtly wrong. It’s a formatting bug that sent me on a proper wild goose chase.

This bug remains unfixed – but interestingly, a year later, that particular CSS rule has disappeared from Zeldman’s site. I wonder if it caused any other problems?

Closing thoughts

The web is big and messy and bloated, and there are lots of reasons to be pessimistic about the state of modern web development – but there are also lots of people doing cool and interesting stuff with it. As I was reading this mass of HTML and CSS, I had so many moments where I thought “ooh, that’s clever!” or “neat!” or “I wonder how that works?”. I hope that as you’ve read this post, you’ve learnt something too.

I’ve always believed in the spirit of “view source”, the idea that you can look at the source code of any web page and see how it works. Although that’s become harder as more of the web is created by frameworks and machines, this exercise shows that it’s clinging on. We can still learn from reading other people’s source code.

When I set out to redo my bookmarks, I was only trying to get my personal data under control. Learning more about front-end web development has been a nice bonus. My knowledge is still a tiny tip of an iceberg, but now it’s a little bit bigger.

I know this post has been particularly dry and technical, so next week I’ll end this series on a lighter note. I’ll show you some of my favourite websites from my bookmarks – the fun, the whimsical, the joyous – the people who use the web as a creative canvas, and who inspire me to make my web presence better.

Don’t use ‘admin’: UK’s top 20 most-used passwords revealed as scams soar

Guardian
www.theguardian.com
2025-12-07 07:00:15
Easy-to-guess words and figures still dominate, alarming cysbersecurity experts and delighting hackers It is a hacker’s dream. Even in the face of repeated warnings to protect online accounts, a new study reveals that “admin” is the most commonly used password in the UK. The second most popular, “12...
Original Article

It is a hacker’s dream. Even in the face of repeated warnings to protect online accounts, a new study reveals that “admin” is the most commonly used password in the UK.

The second most popular, “123456”, is also unlikely to keep hackers at bay.

The annual review of the top 200 most common passwords by the tech company NordPass makes depressing reading for security experts, the police and anti-fraud bodies.

Although cybersecurity experts keep repeating that simple passwords are extremely easy to guess, these warnings are going unheeded.

In the UK, words, number combinations, and common keyboard patterns dominate the top 20. Different variations of the word “password” take up as many as five of these spots, with simple numeric combinations, including “12345678” and then “123456789” using another five. So far, so easy to hack.

Use a password management tool to help with more complicated secure passwords.
Use a password management tool to help with more complicated secure passwords. Photograph: Koshiro K/Alamy

It’s not just a problem here – Australians, Americans and Germans also use “admin” more than any other password when accessing websites, apps and logging in to their computers. Around the world, “123456” emerges as the most popular.

“Despite all efforts in cybersecurity education and digital awareness over the years, data reveals only minor improvements in password hygiene,” says Karolis Arbaciauskas of NordPass, a password manager that aims to keep details secure.

“About 80% of data breaches are caused by compromised, weak, and reused passwords, and criminals will intensify their attacks as much as they can until they reach an obstacle they can’t overcome.”

What the scam looks like

At a time when many of us grapple with a growing number of passwords, it seems people are picking the easy option. Criminals are well aware of this and will use the obvious options during a systematic attack on someone’s accounts.

“The problem with easy-to-remember passwords is that most of them can be cracked or guessed in seconds using a technique called a ‘dictionary attack’ – a systematic method of guessing a password by trying many common words and their simple variations,” Arbaciauskas says.

Woman using a laptop
Hackers use a ‘dictionary attack’, a method of trying common words and numbers and their variations. Photograph: Dominic Lipinski/PA

“Another problem is that people tend to reuse them quite often. Users cite having too many accounts to create, and remember, unique passwords for all of them. That is terrible. People who use weak passwords, or reuse them, risk their digital lives and their identities.”

Recent research from Virgin Media O2 suggests four out of every five people use the same, or nearly identical, passwords on online accounts, giving an almost open door for hackers to compromise log-ins.

You might be alerted to an attack by a message advising that you have been trying to change your email address, or other details, connected to an account.

What to do

Make your passwords long and strong. This could be by combining three random words (eg, applepenbiro) or mixing numbers, letters and special characters.

Don’t reuse the same password. The rule of thumb is that each account should have a unique password because if one account gets broken into, hackers can use the same credentials for other accounts.

Change any passwords that are variations on the same word now, starting with the important sets of accounts: banks, email, work and mobile.

Use password managers – these are often integrated into web browsers. Apple has iCloud Keychain, while Android phones have Google Password Manager, both of which can generate and save complicated passwords.

Two-factor authentication (2FA) is something you can set up for your email, and other important online accounts, to add an extra layer of security. It involves providing something that only you can access – for example, a code sent to you by text message. You should turn 2FA on for every service that offers it.

The Deep Card Conundrum

Lobsters
frontendmasters.com
2025-12-07 05:35:11
Comments...
Original Article

In the world of web design, we often talk about “cards”. Those neat little rectangles that group information together are the bread and butter of modern UI. But usually, these cards are as flat as the screens they live on. Maybe they have a subtle drop shadow to hint at elevation, but that’s where the illusion ends.

But what if a card wasn’t just a surface? What if it was a window ?

Enter the Deep Card .

Imagine a card that isn’t just a 2D plane, but a container with actual volume. A card that holds a miniature 3D world inside it. When you rotate this card, you don’t just see it skew, you see the elements inside it shift in perspective, revealing their depth. It’s like holding a glass box filled with floating objects.

The effect is mesmerizing. It transforms a static element into something tactile and alive. It invites interaction. Whether it’s for a digital trading card game, a premium product showcase, or just a portfolio piece that screams “look at me,” the Deep Card adds a layer of polish and “wow” factor that flat design simply can’t match.

But as I quickly discovered, building this illusion, especially one that feels right and performs smoothly, is a bit more of a puzzle than it first appears.

The CSS Trap

There are plenty of JavaScript libraries out there that can handle this, but I’m a bit of a CSS purist (read: stubborn). I’ve spent years pushing stylesheets to their absolute limits, and I was convinced that a clean, performant, pure CSS solution was hiding in plain sight.

On paper, the logic seems flawless. If you’ve dabbled in 3D CSS before, you know the drill:

  1. Set the Stage : Take a container element and give it some perspective .
  2. Build the World : Position the child elements in 3D space ( translateZ , rotateX , etc.).
  3. Preserve the Illusion : Apply transform-style: preserve-3d so all those children share the same 3D space.

Simple, right?

But here’s the catch. For a true “card” effect, you need the content to stay inside the card boundaries. If a 3D star floats “up” towards the viewer, you don’t want it to break the frame, you want it to be clipped by the card’s edges, reinforcing the idea that it’s inside a container.

So, naturally, you add overflow: clip (or hidden ) to the card. And that is the exact moment everything falls apart.

Comparison of overflow properties in CSS: left shows 'overflow: visible;' with layered rectangles, right shows 'overflow: clip;' with clipped edges.

The Spec Says No

Suddenly, your beautiful 3D scene flattens out. The depth vanishes. The magic is gone.

Why? Because according to the CSS Transforms Module Level 2 specification , applying any “grouping property” like overflow (with any value other than visible ), opacity less than 1, or filter , forces the element to flatten.

The sad reality: A value of preserve-3d for transform-style is ignored if the element has any grouping property values.

In other words: you can have a 3D container, or you can clip its content. You cannot do both on the same element.

For a long time, this felt like a dead end. How do you keep the 3D depth while keeping the elements contained?!

Faking It

If the spec says we can’t have both perspective and clipping, maybe we can cheat. If we can’t use real 3D depth, perhaps we can fake it.

Faking perspective is a time-honored tradition in 2D graphics. You can simulate depth by manipulating the size and position of elements based on their “distance” from the viewer. In CSS terms, this means using scale() to make things smaller as they get “further away” and translate() to move them relative to the card’s angle. 

.card {
  /* --mouse-x and --mouse-y values ranage from -1 to 1 */
  --tilt-x: calc(var(--mouse-y, 0.1) * -120deg); 
  --tilt-y: calc(var(--mouse-x, 0.1) * 120deg); 
  transform: rotateX(var(--tilt-x)) rotateY(var(--tilt-y));
}

.card-layer {
  /* Fake perspective with scale and translate */
  scale: calc(1 - var(--i) * 0.02);
  translate:
    calc(var(--mouse-x) * (var(--i)) * -20%)
    calc(var(--mouse-y) * (var(--i)) * -20%);
}Code language: CSS (css)

This technique can work wonders. There are some brilliant examples out there, like this one by Jhey , that pull off the effect beautifully without using a single line of perspective or preserve-3d .

It’s a solid approach. It’s performant, it works across browsers, and for subtle effects, it’s often indistinguishable from the real thing.

But it has a ceiling.

The illusion holds up well within a limited range of motion. But the moment you push it too far, say, by rotating the card to a sharp angle or trying to flip it 180 degrees, the math starts to show its cracks. The perspective flattens out, and the movement stops feeling natural.

As you can see, when the card turns, the inner elements lose their spatial relationship. The magic evaporates. So while this is a great tool for the toolbox, it wasn’t the complete solution I was looking for. I wanted the real deal. Full 3D, full rotation, full clipping.

Road to a Nowhere

I spent years (on and off, I’m not that obsessed) trying to crack this. I was convinced there had to be a way to have my cake and eat it too.

Theoretically, there is a way. If you can’t clip the container, you have to clip the children.

Imagine using clip-path on every single layer inside the card. You would need to calculate, in real-time, exactly where the edges of the card are relative to the viewer, and then apply a dynamic clipping mask to each child element so that it cuts off exactly at those boundaries.

This involves a lot of math, even for me. We’re talking about projecting 3D coordinates onto a 2D plane, calculating intersections, and handling the trigonometry of the user’s perspective.

A textured blackboard covered with various mathematical equations, diagrams, and symbols, creating a complex academic backdrop.

I was almost ready to give up and accept that maybe, just maybe, this was one of those things CSS just wasn’t meant to do. And then, I got a message from Cubiq .

The Breakthrough

This wasn’t the first time someone had asked me about this topic. As someone who’s known for pushing CSS 3D to its limits, I get this question a lot. People assume I have the answer. But, well… I didn’t.

So when Cubiq messaged me, showing me a GIF of a fully rotating card with deep 3D elements and asking how to achieve it, I went into auto-pilot. I gave him the standard explanation on why the spec forbids it, why overflow flattens the context, and how he could try to fake it with scale and translate .

I thought that was the end of it, but then, he surprised me.

A screenshot of a messaging app conversation featuring a user's message expressing excitement about a discovery.

My Personal Blind Spot

I’ve tried many tricks over the years, but one property I religiously avoided was perspective-origin .

If you really dig into how CSS calculates perspective, you realize that perspective-origin doesn’t just shift your point of view. It fundamentally skews the entire viewport. It creates this weird, unnatural distortion that usually looks terrible.

I cover this at length in my talk 3D in CSS, and the True Meaning of Perspective , if you’re into that sort of thing.

Cubiq, however, didn’t have my baggage. He looked at the problem with fresh eyes and realized something brilliant: just as perspective-origin can be used to create distortion, it can also be used to correct it.

Alternate blog post title idea: Finally, we found one good use for perspective-origin ! 🤣

The Solution

Here is the magic formula that Cubiq came up with: 

.card-container {
  transform: rotateX(var(--tilt-x)) rotateY(var(--tilt-y));
}

.card {
  perspective: calc(
    cos(var(--tilt-x)) * cos(var(--tilt-y)) * var(--perspective)
  );
  perspective-origin: 
    calc(cos(var(--tilt-x)) * sin(var(--tilt-y)) * var(--perspective) * -1 + 50%)
    calc(sin(var(--tilt-x)) * var(--perspective) + 50%);
  overflow: clip;
}Code language: CSS (css)

It looks a bit scary at first glance, but the logic is actually quite elegant.

Since we are using overflow: clip , the 3D context is flattened. This means the browser treats the card as a flat surface and renders its children onto that surface. Normally, this flattening would kill the 3D effect of the children. They would look like a flat painting on a rotating wall.

But here is the trick: We use perspective and perspective-origin to counter-act the rotation.

By dynamically calculating the perspective-origin based on the card’s tilt, we are essentially telling the browser: “Hey, I know you flattened this element, but I want you to render the perspective of its children as if the viewer is looking at them from this specific angle.”

We are effectively projection-mapping the 3D scene onto the 2D surface of the card. The math ensures that the projection aligns perfectly with the card’s physical rotation, creating the illusion of a deep, 3D space inside a container that the browser considers “flat.”

It’s not about moving the world inside of the card, it’s about tricking the flat projection to look 3D by aligning the viewer’s perspective with the card’s rotation.

The Lesson

I love this solution not just because it works (and it works beautifully), but because it taught me a humbling lesson.

I had written off perspective-origin as a “bad” property. I had a mental block against it because I only saw it as a source of distortion. I was so focused on the “right” way to do 3D, that I blinded myself to the tools that could actually solve the problem.

Cubiq didn’t have that bias. He saw a math problem: “I need the projection to look like X when the rotation is Y.” And he found the property that controls projection.

Breaking It Down

Now that we know it’s possible, let’s break down exactly what’s happening here, step by step, and look at some examples of what you can do with it. Let’s start with the basics.

The HTML

At its core, the structure is simple. We have a .card-container that holds the .card , which in turn contains the .card-content , that is the ‘front’ of the card and where all the inner layers live. and the card-back for the back face. 

<div class="outer-container">
  <div class="card">
    <div class="card-content">
      <!-- Inner layers go here -->
    </div>
    <div class="card-back">
      <!-- Back face content -->
    </div>
  </div>
</div>Code language: HTML, XML (xml)

Inside the .card-content , we can now add .card-layers with multiple layers in it. Here I’m setting a --i custom property on each layer to later control its depth. 

<div class="card-layers">
  <div class="card-layer" style="--i: 0"></div>
  <div class="card-layer" style="--i: 1"></div>
  <div class="card-layer" style="--i: 2"></div>
  <div class="card-layer" style="--i: 3"></div>
  <!-- more layers as needed -->
</div>Code language: HTML, XML (xml)

Now we can fill each layer with content, images, text, or whatever we want.

The Movement

To create the rotation effect, we need to track the mouse position and convert it into tilt angles for the card. So the first thing we need to do is to map the mouse position into two CSS variables, --mouse-x and --mouse-y .

This is done with few simple lines of JavaScript: 

const cardContainer = document.querySelector('.card-container');

window.addEventListener('mousemove', (e) => {
  const rect = cardContainer.getBoundingClientRect();
  const x = (e.clientX - rect.left) / rect.width * 2 - 1;
  const y = (e.clientY - rect.top) / rect.height * 2 - 1;
  cardContainer.style.setProperty('--mouse-x', x);
  cardContainer.style.setProperty('--mouse-y', y);
});Code language: JavaScript (javascript)

This gives us normalized values between -1 and 1 on each axis, so we can use them regardless of the card size or aspect ratio.

We convert these values to --tilt-x and --tilt-y in CSS, by multiplying them by the number of degrees we want the card to rotate: 

--tilt-x: calc(var(--mouse-y, 0.1) * -120deg);
--tilt-y: calc(var(--mouse-x, 0.1) * 120deg);Code language: CSS (css)

The higher the degree value, the more dramatic the rotation. 20–30 degrees will give us a subtle effect, while 180 degrees will spin the card all the way around.

Notice that --mouse-x affects --tilt-y , because movement of the mouse along the X axis should actually rotate the card around the Y axis, and vice versa. Also, we multiply --mouse-y by a negative number, because the Y axis on the screen is inverted compared to the mathematical Y axis.

Now that we have --tilt-x and --tilt-y , we can start using them. And first, we apply them to the card container to rotate it in 3D space: 

.card {
  transform: rotateX(var(--tilt-x)) rotateY(var(--tilt-y));
}Code language: CSS (css)

This gives us the basic rotation effect. The card will now tilt and spin based on the mouse position.

The Perspective

We need to remember that we need to set two different perspectives: one for the card’s container (to create the 3D effect), and one for the card’s content (to maintain the depth of the inner elements).

on the .card-container we set a standard perspective: 

.card-container {
  perspective: var(--perspective);
}Code language: CSS (css)

You can set --perspective to any value you like, but a good starting point is around 800px . Lower values will create a more dramatic perspective, while higher values will make it more subtle.

To preserve the 3D space and making sure all the inner elements share the same 3D context, we set transform-style: preserve-3d . I’m using the universal selector here to apply it to all children elements: 

* {
  transform-style: preserve-3d;
}Code language: CSS (css)

To deal with the inner perspective, we set up the perspective and perspective-origin on the .card-content element, which holds all the inner layers: 

.card-content {
  perspective: calc(
    cos(var(--tilt-x)) * cos(var(--tilt-y)) * var(--perspective)
  );
  perspective-origin: 
    calc(cos(var(--tilt-x)) * sin(var(--tilt-y)) * var(--perspective) * -1 + 50%)
    calc(sin(var(--tilt-x)) * var(--perspective) + 50%);
  overflow: clip;
}Code language: CSS (css)

Note that we added overflow: clip to the .card-content to ensure that the inner elements are clipped by the card boundaries. This combination of perspective , perspective-origin , and overflow: clip is what allows us to maintain the 3D depth of the inner elements while keeping them contained within the card.

The Depth

Now that we have the rotation and perspective set up, we can start adding depth to the inner layers. Each layer will be positioned in 3D space using translateZ , based on its --i value. 

.card-layer {
  position: absolute;
  transform: translateZ(calc(var(--i) * 1rem));
}Code language: CSS (css)

This will space out the layers along the Z axis, creating the illusion of depth. You can adjust the multiplier (here 1rem ) to control how far apart the layers are.

Putting It All Together

Using the techniques outlined above, we can create a fully functional Deep Card that responds to mouse movement, maintains 3D depth, and clips its content appropriately.

Here is a complete ‘boilerplate’ example:

You can customize it to your needs, set the number of layers, their depth, and add content within each layer to create a wide variety of Deep Card effects.

Getting Deeper

To improve the Deep Card effect and further enhance the perception of depth, we can add shadows and darkening effects to the layers.

One way to achieve darker colors is just using darker colors. We can calculate the brightness of each layer based on its depth, making deeper layers darker to simulate light falloff. 

.card-layer {
  color: hsl(0 0% calc(100% - var(--i) * 9%));
}Code language: CSS (css)

Another technique is to add semi-transparent background to each layer. This way each layer is like screen that slightly darkens the layers behind it, enhancing the depth effect. 

.card-layer {
  background-color: #2224;
}Code language: CSS (css)

Here is an example of a two cards with different effects: The first card uses darker colors for deeper layers, while the second card uses semi-transparent overlays to create a more pronounced depth effect.

Choose the one that fits your design best, or combine both techniques for an even richer depth experience.

The z-index Effect

You might notice that I’m placing all the layers inside a container ( .card-layers ) rather than making them direct children of .card-content . The reason is that since we’re moving the layers along the Z axis, we don’t want them to be direct children of an element with overflow: clip; (like .card-content ).

As mentioned earlier, once you set overflow: clip; on .card-content , its transform-style becomes flat , which means all of its direct children are rendered on a single plane. Their stacking order is determined by z-index , not by their position along the Z axis. By wrapping the layers in a container, we preserve their 3D positioning and allow the depth effect to work as intended.

The Twist

Now that we understand this limitation, let’s turn it to our advantage and see what kinds of effects we can create with it.

Here are the exact same two cards as in the previous example, but this time without a .card-layers container. The layers are direct children of .card-content :

Adding Interaction

We often use cards that need to display extra information. One of my favorite things to do in these cases is to rotate the card 180 degrees and reveal the additional content on the back side. Now, we can do exactly that, and build an entire 3D world inside the card.

In this example, we have a front face ( .card-content ) and a back face ( .card-back ). When the user clicks the card, we toggle a checkbox that rotates the card 180 degrees, revealing the back face. 

<label class="card-container">
  <input type="checkbox">
  <div class="card">
    <div class="card-content">
      <!-- front face content -->
    </div>
    <div class="card-back">
      <!-- back face content -->
    </div>
  </div>
</label>Code language: HTML, XML (xml)
.card-container {
  cursor: pointer;
    
  &:has(input:checked) .card {
    rotate: y 180deg;
  }
  
  input[type="checkbox"] {
    display: none;
  }
}Code language: CSS (css)

You can also use a button or any other interactive element to toggle the rotation, depending on your use case, and use any animation technique you like to make the rotation smooth.

Inner Movement

Of course, we can also use any animation on the inner layers to create dynamic effects. It can be wild and complex, or subtle and elegant. The key is that since the layers are in 3D space, any movement along the Z axis will enhance the depth effect.

Here a simple example with parallax layers. each layer animates it’s background position on the X axis, and to enhance the depth effect, I’m animating the layers at different speeds based on their depth: 

.card-layer {
  animation: layer calc(var(--i) * 8s) infinite linear;
}Code language: CSS (css)

And the result:

Deep Text Animation

This technique works beautifully with the concept of layered text, opening up a world of creative possibilities. There’s so much you can do with it, from subtle depth effects to wild, animated 3D lettering.

I actually wrote an entire article about this , featuring 20+ examples, and every single one of them looks fantastic inside a Deep Card. Here’s one of the examples from that article, now living inside a card:

Going Full 360

up until now, we’ve mostly focused on layering our inner content and using the Z axis to create depth. But we can definitely take it a step further, break out of the layering concept, and build a fully 3D object that you can spin around in all directions.

From here, the possibilities are truly endless. You can keep experimenting—add more interactions, more layers, or even create effects on both sides of the card to build two complete worlds, one on each face. Or, go all in and design an effect that dives deep into the card itself. The only real limit is your imagination.

Conclusion

The Deep Card is now a solved problem. We can have our cake (3D depth), eat it (clipping), and even spin it around 360 degrees without breaking the illusion.

So, the next time you hit a wall with CSS, and you’re sure you’ve tried everything, maybe take a second look at those properties you swore you’d never use. You might just find your answer hiding in the documentation you skipped.

Now, go build something deep.

Vanilla CSS is all you need

Lobsters
www.zolkos.com
2025-12-07 04:39:56
Comments...
Original Article

Back in April 2024, Jason Zimdars from 37signals published a post about modern CSS patterns in Campfire . He explained how their team builds sophisticated web applications using nothing but vanilla CSS. No Sass. No PostCSS. No build tools.

The post stuck with me. Over the past year and a half, 37signals has released two more products (Writebook and Fizzy) built on the same nobuild philosophy. I wanted to know if these patterns held up. Had they evolved?

I cracked open the source code for Campfire, Writebook, and Fizzy and traced the evolution of their CSS architecture. What started as curiosity became genuine surprise. These are not just consistent patterns. They are improving patterns. Each release builds on the last, adopting progressively more modern CSS features while maintaining the same nobuild philosophy.

These are not hobby projects. Campfire is a real-time chat application. Writebook is a publishing platform. Fizzy is a full-featured project management tool with kanban boards, drag-and-drop, and complex state management. Combined, they represent nearly 14,000 lines of CSS across 105 files.

Not a single line touches a build tool.

The Tailwind Question

Let me be clear: there is nothing wrong with Tailwind . It is a fantastic tool that helps developers ship products faster. The utility-first approach is pragmatic, especially for teams that struggle with CSS architecture decisions.

But somewhere along the way, utility-first became the only answer. CSS has evolved dramatically. The language that once required preprocessors for variables and nesting now has:

37signals looked at this landscape and made a bet: modern CSS is powerful enough. No build step required.

Three products later, that bet is paying off.

The Architecture: Embarrassingly Simple

Open any of these three codebases and you find the same flat structure: 

app/assets/stylesheets/
├── _reset.css
├── base.css
├── colors.css
├── utilities.css
├── buttons.css
├── inputs.css
├── [component].css
└── ...

That is it. No subdirectories. No partials. No complex import trees. One file per concept, named exactly what it does.

Zero configuration. Zero build time. Zero waiting.

I would love to see something like this ship with new Rails applications. A simple starting structure with _reset.css , base.css , colors.css , and utilities.css already in place. I suspect many developers reach for Tailwind not because they prefer utility classes, but because vanilla CSS offers no starting point. No buckets. No conventions. Maybe CSS needs its own omakase.

The Color System: Consistent Foundation, Evolving Capabilities

Jason’s original post explained OKLCH well. It is the perceptually uniform color space all three apps use. The short version: unlike RGB or HSL, OKLCH’s lightness value actually corresponds to perceived brightness. A 50% lightness blue looks as bright as a 50% lightness yellow.

What is worth noting is how this foundation remains identical across all three apps: 

:root {
  /* Raw LCH values: Lightness, Chroma, Hue */
  --lch-blue: 54% 0.15 255;
  --lch-red: 51% 0.2 31;
  --lch-green: 65% 0.23 142;

  /* Semantic colors built on primitives */
  --color-link: oklch(var(--lch-blue));
  --color-negative: oklch(var(--lch-red));
  --color-positive: oklch(var(--lch-green));
}

Dark mode becomes trivial: 

@media (prefers-color-scheme: dark) {
  :root {
    --lch-blue: 72% 0.16 248;   /* Lighter, slightly desaturated */
    --lch-red: 74% 0.18 29;
    --lch-green: 75% 0.20 145;
  }
}

Every color that references these primitives automatically updates. No duplication. No separate dark theme file. One media query, and the entire application transforms.

Fizzy takes this further with color-mix() : 

.card {
  --card-color: oklch(var(--lch-blue-dark));

  /* Derive an entire color palette from one variable */
  --card-bg: color-mix(in srgb, var(--card-color) 4%, var(--color-canvas));
  --card-text: color-mix(in srgb, var(--card-color) 30%, var(--color-ink));
  --card-border: color-mix(in srgb, var(--card-color) 33%, transparent);
}

One color in, four harmonious colors out. Change the card color via JavaScript ( element.style.setProperty('--card-color', '...') ), and the entire card theme updates automatically. No class swapping. No style recalculation. Just CSS doing what CSS does best.

The Spacing System: Characters, Not Pixels

Here is a pattern I did not expect: all three applications use ch units for horizontal spacing. 

:root {
  --inline-space: 1ch;      /* Horizontal: one character width */
  --block-space: 1rem;      /* Vertical: one root em */
}

.component {
  padding-inline: var(--inline-space);
  margin-block: var(--block-space);
}

Why characters? Because spacing should relate to content. A 1ch gap between words feels natural because it is literally the width of a character. As font size scales, spacing scales proportionally.

This also makes their responsive breakpoints unexpectedly elegant: 

@media (min-width: 100ch) {
  /* Desktop: content is wide enough for sidebar */
}

Instead of asking “is this a tablet?”, they are asking “is there room for 100 characters of content?” It is semantic. It is content-driven. It works.

Utility Classes: Yes, They Still Exist

Let me address the elephant in the room. These applications absolutely use utility classes: 

/* From utilities.css */
.flex { display: flex; }
.gap { gap: var(--inline-space); }
.pad { padding: var(--block-space) var(--inline-space); }
.txt-large { font-size: var(--text-large); }
.hide { display: none; }

The difference? These utilities are additive , not foundational. The core styling lives in semantic component classes. Utilities handle the exceptions: the one-off layout adjustment, the conditional visibility toggle.

Compare to a typical Tailwind component: 

<!-- Tailwind approach -->
<button class="inline-flex items-center gap-2 px-4 py-2 rounded-full
               border border-gray-300 bg-white text-gray-900
               hover:bg-gray-50 focus:ring-2 focus:ring-blue-500">
  Save
</button>

And the 37signals equivalent: 

<!-- Semantic approach -->
<button class="btn">Save</button>

.btn {
  --btn-padding: 0.5em 1.1em;
  --btn-border-radius: 2em;

  display: inline-flex;
  align-items: center;
  gap: 0.5em;
  padding: var(--btn-padding);
  border-radius: var(--btn-border-radius);
  border: 1px solid var(--color-border);
  background: var(--btn-background, var(--color-canvas));
  color: var(--btn-color, var(--color-ink));
  transition: filter 100ms ease;
}

.btn:hover {
  filter: brightness(0.95);
}

.btn--negative {
  --btn-background: var(--color-negative);
  --btn-color: white;
}

Yes, it is more CSS. But consider what you gain:

  1. HTML stays readable. class="btn btn--negative" tells you what something is, not how it looks.
  2. Changes cascade. Update --btn-padding once, every button updates.
  3. Variants compose. Add .btn--circle without redefining every property.
  4. Media queries live with components. Dark mode, hover states, and responsive behavior are co-located with the component they affect.

The :has() Revolution

If there is one CSS feature that changes everything, it is :has() . For decades, you needed JavaScript to style parents based on children. No more.

Writebook uses it for a sidebar toggle with no JavaScript: 

/* When the hidden checkbox is checked, show the sidebar */
:has(#sidebar-toggle:checked) #sidebar {
  margin-inline-start: 0;
}

Fizzy uses it for kanban column layouts: 

.card-columns {
  grid-template-columns: 1fr var(--column-width) 1fr;
}

/* When any column is expanded, adjust the grid */
.card-columns:has(.cards:not(.is-collapsed)) {
  grid-template-columns: auto var(--column-width) auto;
}

Campfire uses it for intelligent button styling: 

/* Circle buttons when containing only icon + screen reader text */
.btn:where(:has(.for-screen-reader):has(img)) {
  --btn-border-radius: 50%;
  aspect-ratio: 1;
}

/* Highlight when internal checkbox is checked */
.btn:has(input:checked) {
  --btn-background: var(--color-ink);
  --btn-color: var(--color-ink-reversed);
}

This is CSS doing what you used to need JavaScript for. State management. Conditional rendering. Parent selection. All declarative. All in stylesheets.

Progression

What fascinated me most was watching the architecture evolve across releases.

Campfire (first release) established the foundation:

  • OKLCH colors
  • Custom properties for everything
  • Character-based spacing
  • Flat file organization
  • View Transitions API for smooth page changes

Writebook (second release) added modern capabilities:

  • Container queries for component-level responsiveness
  • @starting-style for entrance animations

Fizzy (third release) went all-in on modern CSS:

  • CSS Layers ( @layer ) for managing specificity
  • color-mix() for dynamic color derivation
  • Complex :has() chains replacing JavaScript state

You can see a team learning, experimenting, and shipping progressively more sophisticated CSS with each product. By Fizzy, they are using features many developers do not even know exist. 

/* Fizzy's layer architecture */
@layer reset, base, components, modules, utilities;

@layer components {
  .btn { /* Always lower specificity than utilities */ }
}

@layer utilities {
  .hide { /* Always wins over components */ }
}

CSS Layers solve the specificity wars that have plagued CSS since the beginning. It does not matter what order your files load. It does not matter how many classes you chain. Layers determine the winner, period.

The Loading Spinner

One technique appears in all three applications that deserves special attention. Their loading spinners use no images, no SVGs, no JavaScript. Just CSS masks.

Here is the actual implementation from Fizzy’s spinners.css : 

@layer components {
  .spinner {
    position: relative;

    &::before {
      --mask: no-repeat radial-gradient(#000 68%, #0000 71%);
      --dot-size: 1.25em;

      -webkit-mask: var(--mask), var(--mask), var(--mask);
      -webkit-mask-size: 28% 45%;
      animation: submitting 1.3s infinite linear;
      aspect-ratio: 8/5;
      background: currentColor;
      content: "";
      inline-size: var(--dot-size);
      inset: 50% 0.25em;
      margin-block: calc((var(--dot-size) / 3) * -1);
      margin-inline: calc((var(--dot-size) / 2) * -1);
      position: absolute;
    }
  }
}

The keyframes live in a separate animation.css file: 

@keyframes submitting {
  0%    { -webkit-mask-position: 0% 0%,   50% 0%,   100% 0% }
  12.5% { -webkit-mask-position: 0% 50%,  50% 0%,   100% 0% }
  25%   { -webkit-mask-position: 0% 100%, 50% 50%,  100% 0% }
  37.5% { -webkit-mask-position: 0% 100%, 50% 100%, 100% 50% }
  50%   { -webkit-mask-position: 0% 100%, 50% 100%, 100% 100% }
  62.5% { -webkit-mask-position: 0% 50%,  50% 100%, 100% 100% }
  75%   { -webkit-mask-position: 0% 0%,   50% 50%,  100% 100% }
  87.5% { -webkit-mask-position: 0% 0%,   50% 0%,   100% 50% }
  100%  { -webkit-mask-position: 0% 0%,   50% 0%,   100% 0% }
}

Three dots, bouncing in sequence:

The background: currentColor means it automatically inherits the text color. Works in any context, any theme, any color scheme. Zero additional assets. Pure CSS creativity.

A Better <mark>

The default browser <mark> element renders as a yellow highlighter. It works, but it is not particularly elegant. Fizzy takes a different approach for search result highlighting: drawing a hand-drawn circle around matched terms.

Fizzy search results showing circled text highlighting

Here is the implementation from circled-text.css : 

@layer components {
  .circled-text {
    --circled-color: oklch(var(--lch-blue-dark));
    --circled-padding: -0.5ch;

    background: none;
    color: var(--circled-color);
    position: relative;
    white-space: nowrap;

    span {
      opacity: 0.5;
      mix-blend-mode: multiply;

      @media (prefers-color-scheme: dark) {
        mix-blend-mode: screen;
      }
    }

    span::before,
    span::after {
      border: 2px solid var(--circled-color);
      content: "";
      inset: var(--circled-padding);
      position: absolute;
    }

    span::before {
      border-inline-end: none;
      border-radius: 100% 0 0 75% / 50% 0 0 50%;
      inset-block-start: calc(var(--circled-padding) / 2);
      inset-inline-end: 50%;
    }

    span::after {
      border-inline-start: none;
      border-radius: 0 100% 75% 0 / 0 50% 50% 0;
      inset-inline-start: 30%;
    }
  }
}

The HTML structure is <mark class="circled-text"><span></span>webhook</mark> . The empty span exists solely to provide two pseudo-elements ( ::before and ::after ) that draw the left and right halves of the circle.

The technique uses asymmetric border-radius values to create an organic, hand-drawn appearance. The mix-blend-mode: multiply makes the circle semi-transparent against the background, switching to screen in dark mode for proper blending.

Search results for: webhook

No images. No SVGs. Just borders and border-radius creating the illusion of a hand-drawn circle.

Dialog Animations: The New Way

Fizzy and Writebook both animate HTML <dialog> elements. This was notoriously difficult before. The secret is @starting-style .

Here is the actual implementation from Fizzy’s dialog.css : 

@layer components {
  :is(.dialog) {
    border: 0;
    opacity: 0;
    transform: scale(0.2);
    transform-origin: top center;
    transition: var(--dialog-duration) allow-discrete;
    transition-property: display, opacity, overlay, transform;

    &::backdrop {
      background-color: var(--color-black);
      opacity: 0;
      transform: scale(1);
      transition: var(--dialog-duration) allow-discrete;
      transition-property: display, opacity, overlay;
    }

    &[open] {
      opacity: 1;
      transform: scale(1);

      &::backdrop {
        opacity: 0.5;
      }
    }

    @starting-style {
      &[open] {
        opacity: 0;
        transform: scale(0.2);
      }

      &[open]::backdrop {
        opacity: 0;
      }
    }
  }
}

The --dialog-duration variable is defined globally as 150ms .

This dialog animates in and out using pure CSS.

The @starting-style rule defines where the animation starts from when an element appears. Combined with allow-discrete , you can now transition between display: none and display: block . The modal smoothly scales and fades in. The backdrop fades independently. No JavaScript animation libraries. No manually toggling classes. The browser handles it.

What This Means for You

I am not suggesting you abandon your build tools tomorrow. But I am suggesting you reconsider your assumptions.

You might not need Sass or PostCSS. Native CSS has variables, nesting, and color-mix() . The features that needed polyfills are now baseline across browsers.

You might not need Tailwind for every project. Especially if your team understands CSS well enough to build a small design system.

While the industry sprints toward increasingly complex toolchains, 37signals is walking calmly in the other direction. Is this approach right for everyone? No. Large teams with varying CSS skill levels might benefit from Tailwind’s guardrails. But for many projects, their approach is a reminder that simpler can be better.

Thanks to Jason Zimdars and the 37signals team for sharing their approach openly. All code examples in this post are taken from the Campfire, Writebook, and Fizzy source code. For Jason’s original deep-dive into Campfire’s CSS patterns, see Modern CSS Patterns and Techniques in Campfire . If you want to learn modern CSS, these three codebases are an exceptional classroom.

Package Manager Design Tradeoffs

Lobsters
nesbitt.io
2025-12-07 04:39:04
Comments...
Original Article

Package managers make dozens of design decisions with no right answer. Each choice has real costs and benefits, and choosing one side often forecloses other options. This is a survey of those tradeoffs.

Full index replication vs on-demand queries

apt downloads complete package indexes with apt update . Resolution happens locally against this full index. npm and PyPI serve metadata per-package through API queries.

Full replication means fast resolution once synced and works offline. But initial sync is slow, takes disk space, and stale data requires re-syncing. On-demand queries mean smaller bandwidth and always-current data, but resolution requires network access and many round trips. Cargo’s sparse indexes try to get benefits of both, fetching only metadata for crates you actually need.

One version vs many versions retained

Homebrew keeps one version of each formula, when a new version is released the old one disappears. Most language package managers keep every published version available indefinitely.

One version simplifies everything, no version resolution needed, no storage growth, the ecosystem moves together. But breakage propagates immediately, you can’t pin while waiting for a fix, and everyone must upgrade in lockstep. Many versions give flexibility and let projects move at different speeds. But old versions accumulate vulnerabilities, maintainers face pressure to support multiple releases, and you need resolution logic to pick among them.

Source distribution vs binary distribution

Cargo and Go distribute source code; installs involve compilation. PyPI wheels, Maven jars, and NuGet packages are prebuilt binaries.

Source distribution means one artifact works on any platform, users can audit exactly what they’re running, and reproducible builds are possible if the toolchain is deterministic. Binary distribution means fast installs, no compiler toolchain needed on the client, and maintainers control the build environment. The cost is building for every supported platform and trusting that the binary matches the source.

Single artifact vs platform matrix

Cargo publishes one crate per version. PyPI wheels have separate artifacts per Python version, ABI, and platform ( cp39-manylinux_x86_64 , cp310-macosx_arm64 , etc.).

Single artifact is simple, one thing to publish, one thing to verify, no matrix explosion. But it only works when packages are platform-independent or when you push compilation to install time. Platform matrices give fast installs for native code without requiring build tools. The cost is build infrastructure for every supported platform, larger registry storage, and client-side logic to pick the right artifact.

Single registry vs multiple registries

RubyGems and Cargo have a single canonical registry by convention. Maven routinely uses multiple repositories with priority ordering. pip users juggle PyPI plus internal indexes.

Single registry means simpler configuration, no ambiguity about where a package comes from, and easier security reasoning. Multiple registries let organizations run private packages, mirror public packages for reliability, and control what enters their dependency graph. But fallback ordering creates confusion about which version you’re getting. Dependency confusion is a real attack vector: publish a malicious package to a public registry with the same name as a private one, and misconfigured clients fetch the attacker’s version instead.

Maximal vs minimal version selection

Most package managers pick the newest version satisfying constraints. Go modules use minimal version selection, picking the oldest version that works.

Maximal selection gives you bug fixes and security patches automatically. You’re running versions closer to what maintainers tested. But you’re always one bad publish away from breakage, and builds change over time as new versions appear. Minimal selection is deterministic without a lockfile since the algorithm itself produces stable results. It’s also forwards-compatible: when a library adds a new dependency, downstream consumers’ resolved versions don’t change unless they also add that dependency. But you might get bugs fixed in newer versions, and maintainers must test their minimum bounds carefully because users will actually get those minimums.

Fail on conflicts vs allow multiple versions

When two packages want incompatible versions of a dependency, what happens? pip fails resolution. npm dedupes where possible but nests conflicting versions so each package gets what it asked for. Nix allows multiple versions via content-addressed storage.

Failing keeps the ecosystem coherent, if your dependencies can’t agree you find out immediately. But it means you sometimes can’t use two packages together at all. Nesting conflicting versions avoids resolution failures but bloats installs and causes problems when types or state cross version boundaries. Nix sidesteps the problem entirely by giving each package its own isolated dependency tree, stored by content hash so identical versions are shared. But this requires a different storage model and breaks assumptions about where packages live on disk.

Open publishing vs gated review

npm, PyPI, and crates.io let anyone publish immediately with no review. Debian requires packages to be sponsored and reviewed before entering the archive. Homebrew reviews pull requests before formulas are merged.

Open publishing grows ecosystems fast, anyone can contribute, iteration is quick, and there’s no bottleneck. But it invites typosquatting, malware, and low-quality packages. Gated review catches problems before they reach users and maintains quality standards. But it creates delays, requires reviewer time, and limits who can participate. The review bottleneck can also become a governance chokepoint.

Flat vs scoped vs hierarchical namespaces

RubyGems has a flat namespace: rails , rake , nokogiri . npm added scopes: @babel/core , @types/node . Maven uses reverse-domain hierarchical naming: org.apache.commons:commons-lang3 .

Flat namespaces are simple, names are short and memorable. But popular names get claimed early, squatting is easy, and name collisions require awkward workarounds. Scopes add organizational structure and make collisions rarer, but they require governance for who owns scopes. Maven’s hierarchical approach ties names to domain ownership, which provides clear authority but creates verbose identifiers and assumes stable domain ownership.

Central registry vs external identifier

npm controls who gets what name on npmjs.com. Go modules use URLs as package names; github.com/user/repo derives from domain and repository ownership.

Central authority enables dispute resolution, curation, and clean namespaces, the registry can transfer names, reserve important ones, and handle conflicts. But it concentrates power and creates a single point of control. External identifiers remove the bottleneck, no one needs permission to publish. But names become tied to infrastructure that changes, domains expire, repositories move, organizations rename. A name that made sense in 2019 might point somewhere dangerous in 2025. And when a source host goes offline, the package becomes unfetchable with no migration path.

Explicit publish step vs pull from source

npm, Cargo, and RubyGems require maintainers to run a publish command. Go modules pull directly from git tags.

Explicit publishing creates an intentional gate, maintainers decide what’s a release, you can publish a subset of the repo keeping packages small, and the registry can validate at publish time. But published code can diverge from the repo. The xz Utils backdoor exploited this gap, with malicious code in tarballs that wasn’t in the repository. Pull-from-source means the repo is the source of truth, what you audit is what you run, release is just git tagging. But you get everything in the repo including test fixtures, and you can’t easily unpublish since tags persist in forks. And pull-from-source doesn’t prevent all supply chain attacks, just the class that relies on tarball divergence. Malicious code committed to the repo still flows through.

Yanking vs full deletion

When something bad gets published, Cargo and RubyGems let you yank, marking a version as unavailable for new resolves while keeping it accessible for existing lockfiles. npm allows deletion but with time limits.

Yanking preserves reproducibility, existing projects keep working. But the bad version remains accessible, which matters if the problem is a security vulnerability or malicious code. Full deletion actually removes the problem but breaks reproducibility, projects with that version locked suddenly can’t build.

Build hooks allowed vs forbidden

npm’s postinstall runs arbitrary code during installation. Cargo’s build.rs can do the same, though by convention it’s limited to build configuration and native compilation. Go deliberately has no build hooks.

Hooks enable native compilation, downloading platform-specific binaries, and environment-specific setup, esbuild uses postinstall to fetch the right binary for your platform. Cargo’s build.rs output is cached and only re-runs when inputs change, reducing repeated execution. But hooks are a massive attack surface, a compromised dependency can run anything during install. pnpm disables scripts by default. No hooks means predictable builds and a smaller attack surface, Go pays for this by making native code integration painful.

Semver format vs arbitrary strings vs enforced semantics

Cargo, npm, and Hex require versions in semver format (x.y.z) but trust maintainers to follow the compatibility conventions. apt and pacman allow arbitrary version strings. Elm actually enforces semver semantics by diffing package APIs and rejecting publishes that break compatibility without a major bump.

Semver format lets tooling assume structure and provide smart defaults for version ranges. But format alone doesn’t guarantee meaning, and maintainers often get compatibility wrong. Arbitrary strings offer flexibility for upstream projects that don’t follow semver, but resolvers can’t infer compatibility. Enforced semantics catch mistakes but only work when the type system is expressive enough to capture API compatibility. Elm can do this; Python couldn’t.

System-wide vs per-project installation

apt installs packages into shared system directories, one version of OpenSSL serves every application. Bundler and Cargo install per-project, isolating dependencies completely.

System-wide installation saves disk space and means security patches apply everywhere at once. When Debian pushes a fix for libssl, every application gets it on the next upgrade. But you can’t run two applications that need different versions of the same library. Per-project installation allows conflicting requirements to coexist but duplicates storage and means each project must be updated separately when vulnerabilities appear.

Coordinated releases vs rolling updates

Debian stable freezes a set of packages tested together. Arch updates packages continuously as upstream releases them.

Frozen releases give stability, you know that every package in Debian 12 works with every other package in Debian 12 because someone tested those combinations. But software is often years out of date. Rolling releases give freshness and quick security updates but packages might not work together at any given moment, an update to one package might break another before the fix propagates.

Registry-managed signing vs author-controlled signing

npm signs packages at the registry level. Debian requires GPG signatures from maintainers. PyPI supports Sigstore, tying signatures to identity providers rather than long-lived keys.

Registry-managed signing is transparent to publishers but means you’re trusting the registry, not the author. Author-controlled signing (GPG) proves authorship but requires key management, which maintainers often get wrong - keys expire, get lost, or lack rotation. Keyless signing through identity providers (Sigstore) removes key management but ties identity to external services.

These tradeoffs interact. Pull-from-source publishing means you can’t enforce build-time validation. Allowing multiple versions simultaneously makes conflict handling moot but version boundaries create new problems. Deterministic resolution without lockfiles requires minimal version selection. You can’t have fast, small, fully secure, and perfectly reproducible builds all at once, every package manager picks which constraints to prioritize.

Scientists link sugar substitute sorbitol to liver disease in zebrafish

Hacker News
scitechdaily.com
2025-12-07 04:10:11
Comments...
Original Article

Discovering the indieweb with calm tech

Lobsters
alexsci.com
2025-12-07 03:23:43
Comments...
Original Article
Blog Home

When social media first entered my life, it came with a promise of connection. Facebook connected college-aged adults in a way that was previously impossible, helping to shape our digital generation. Social media was our super-power and we wielded it to great effect.

Yet social media today is a noisy, needy, mental health hazard. They push distracting notifications, constantly begging us to “like and subscribe”, and trying to trap us in endless scrolling. They have become sirens that lure us into their ad-infested shores with their saccharine promise of dopamine.

The Siren (1888) by Edward Armitage with the text 'Connect with friends and the world around you on Facebook' added.

Beware the siren's call

How can we defeat these monsters that have invaded deep into our world, while still staying connected?

StreetPass for Mastodon

A couple weeks ago I stumbled into a great browser extension, StreetPass for Mastodon . The creator, tvler , built it to help people find each other on Mastodon. StreetPass autodiscovers Mastodon verification links as you browse the web, building a collection of Mastodon accounts from the blogs and personal websites you’ve encountered.

StreetPass is a beautiful example of calm technology . When StreetPass finds Mastodon profiles it doesn’t draw your attention with a notification, it quietly adds the profile to a list, knowing you’ll check in when you’re ready.

A screenshot showing the StreetPass extension's popup window open. It lists several Mastodon profiles and the timestamps they were discovered

StreetPass recognizes that there’s no need for an immediate call to action. Instead it allows the user to focus on their browsing, enriching their experience in the background. The user engages with StreetPass when they are ready, and on their own terms.

StreetPass is open source and available for Firefox , Chrome , and Safari .

Inspired by StreetPass, I applied this technique to RSS feed discovery.

Blog Quest

Blog Quest is a web browser extension that helps you discover and subscribe to blogs. Blog Quest checks each page for auto-discoverable RSS and Atom feeds (using rel="alternate" links) and quietly collects them in the background. When you’re ready to explore the collected feeds, open the extension’s drop-down window.

A browser extension popup showing several feeds it discovered

The extension integrates with several feed readers, making subscription management nearly effortless.

Blog Quest is available for both Firefox and Chrome . The project is open source and I encourage you to build your own variants.

I reject the dead Internet theory: I see a vibrant Internet full of humans sharing their experiences and seeking connection. Degradation of the engagement-driven web is well underway, accelerated by AI slop. But the independent web works on a different incentive structure and is resistant to this effect. Humans inherently create, connect, and share: we always have and we always will. If you choose software that works in your interest you’ll find that it’s possible to make meaningful online connections without mental hazard.

Check out StreetPass and Blog Quest to discover a decentralized, independent Internet that puts you in control.

You can't drown out the noise of social media by shouting louder, you've got to whisper.

Image credits

Hello! I'm Robert Alexander, a DevSecOps consultant available for contract work . This blog features some of my work and thoughts on software, the cloud, and security. You can subscribe to my posts with your favorite RSS client .

Statements are my own and do not represent the positions or opinions of my employer.

Z2 – Lithographically fabricated IC in a garage fab

Hacker News
sam.zeloof.xyz
2025-12-07 03:03:09
Comments...
Original Article

Homemade 1000+ transistor array chip

In 2018 I made the first lithographically fabricated integrated circuits in my garage fab. I was a senior in high school when I made the Z1 amplifier, and now I’m a senior in college so there are some long overdue improvements to the amateur silicon process.
DSC_9414ano
The Z1 had 6 transistors and was a great test chip to develop all the processes and equipment. The Z2 has 100 transistors on a 10µm polysilicon gate process – same technology as Intel’s first processor . My chip is a simple 10×10 array of transistors to test, characterize, and tweak the process but this is a huge step closer to more advanced DIY computer chips. The Intel 4004 has 2,200 transistors and I’ve now made 1,200 on the same piece of silicon.

Screen Shot 2021-08-12 at 4.28.35 PM

Only half joking
Only half joking

Previously, I made chips with a metal gate process . The aluminum gate has a large work function difference with the silicon channel beneath it which results in a high threshold voltage (>10V). I used these metal gate transistors in a few fun projects like a guitar distortion pedal and a ring oscillator LED blinker but both of these required one or two 9V batteries to run the circuit due to high Vth. By switching to a polysilicon gate process, I get a ton of performance benefits (self aligned gate means lower overlap capacitances) including a much lower Vth which makes these chips compatible with 2.5V and 3.3V logic levels. The new FETs have excellent characteristics: 

NMOS Electrical Properties:
Vth             = 1.1 V
Vgs MAX         = 8 V
Cgs             = <0.9 pF
Rise/fall time  = <10 ns
On/off ratio    = 4.3e6
Leakage current = 932 pA (Vds=2.5V)

I was particularly surprised by the super low leakage current. This value goes up about 100x in ambient room lighting.

Now we know that it’s possible to make really good transistors with impure chemicals, no cleanroom, and homemade equipment. Of course, yield and process repeatability are diminished. I’ll do more testing to collect data on the statistics and variability of FET properties but it’s looking good!

DSC_9419

The chip is small, about one quarter the die area of my previous ICs (2.4mm^2) which makes it hard to probe. There’s a simple 10×10 array of N-channel FETs on each chip which will give me a lot of characterization data. Since it’s such a simple design, I was able to lay it out using Photoshop. Columns of 10 transistors share a common gate connection and each row is strung together in series with adjacent transistors sharing a source/drain terminal. It’s similar to NAND flash but I only did this to keep the metal pads large enough so I can reasonably probe them, if every FET had 3 pads for itself they would be too small.

It’s hard to convey the excitement of seeing a good FET curve displayed on the curve tracer after dipping a shard of rock into chemicals all day.

A single 10µm NMOS transistor can be see below, with slight misalignment in the metal layer (part of the left contact is uncovered). Red outline is polycrystalline silicon, blue is the source/drain.

So far I’ve made an opamp (Z1) and a memory-like array (Z2). More interesting circuits are definitely possible even with this low transistor density. The process needs some tweaking but now that I’m able to consistently make good quality transistors I should be able to design more complex digital and analog circuits. Testing each chip is very tedious so I am trying to automate the process and I’ll post more data then. I’ve made 15 chips (1,500 transistors) and know there’s at least one completely functional chip and at least two “mostly functional”, meaning ~80% of the transistors work instead of 100%. No proper yield data yet. The most common defect is a drain or source shorted to the bulk silicon channel, not a leaky or shorted gate like on my Z1 process.

Profilometer scan of gate
Profilometer scan of gate layer (y axis in angstrom, x axis is micron)

I said before that the gate used to be made out of aluminum and now it’s silicon which makes the chips work a lot better. Silicon comes in three varieties that we care about: amorphous, polycrystalline, and monocrystalline. From left to right, these become more electrically conductive but also much harder to deposit. In fact, monocrystalline Si can’t be deposited, you can only grow it in contact with another mono-Si layer as a seed (epitaxy). Since the gate must be deposited on top of an insulating dielectric, poly is the best we can do. We can heavily dope the polysilicon anyway to make it more conductive.

A typical self-aligned polysilicon gate process requires silane, a toxic and explosive gas, to deposit polycrystalline silicon layers. It may also be possible by sputtering or evaporating amorphous silicon and annealing with a laser . A major theme of this DIY silicon process is to circumvent expensive, difficult, or dangerous steps. So, I came up with a modified process flow. It’s a variation on the standard self-aligned methods to allow doping via high temperature diffusion rather than ion implantation. The effect is that I’m able to buy a silicon wafer with the polysilicon already deposited on it from the factory and pattern it to make transistors instead of putting my own polysilicon down halfway through the process. This is a nice short term workaround but it would be best to design a polysilicon deposition process using the laser anneal method mentioned above.

Wafers are available with all kinds of materials deposited on them already, so I just had to find one with a thin layer of SiO2 (gate oxide, ~10nm) followed by a thicker polysilicon (300nm). I found a lot of 25 200mm (EPI, prime, [1-0-0], p-type) wafers on eBay for $45 which is essentially a lifetime supply, so email me if you want one. The gate oxide is the most fragile layer and requires the most care during fabrication. Since I bought the wafer with a nice high quality oxide on it already that was capped off and kept clean by the thick polysilicon layer, I was able to eliminate all the aggressive cleaning chemicals (sulfuric acid, etc) from the process and still make great transistors. Minimal process chemicals and tools are listed below. 

Chemicals used in home poly-gate process:
-Water
-Alcohol
-Acetone
-Phosphoric acid
-Photoresist
-Developer (2% KOH)
-N type dopant (filmtronics P509)
-HF (1%) or CF4/CHF3 RIE
-HNO3 for poly etch or SF6 RIE
Equipment used in home poly-gate process:
-Hotplate
-Tube furnace
-Lithography apparatus
-Microscope
-Vacuum chamber to deposit metal

Z2 “gate first” process (similar to standard self-aligned process but without a field oxide):

I snapped one of the test chips in half (functional Z2 but with bad layer alignment and thin metal, about 300nm) and put it in my SEM for a cross section:

Find the dust particle in the red circle below, use that to get oriented in the coming cross section views.

xsecloc

Xsection (1)

NMOS cross section
NMOS cross section

Because I bought the wafer already with gate oxide and polysilicon on it, I can’t grow a field oxide. These thick oxide layers are typically used to mask dopants and require a long high temperature step which would oxidize all of my poly and there would be none remaining. So, my modified process uses an additional masking step (the “gate” mask is typically not found in a self-aligned process) that allows me to use the polysilicon itself as a dopant mask and hard-baked photoresist as the field dielectric. This alternative processing results in the stepped structure you can see in the orange region on the NMOS cross section above. This process subtlety is mentioned here, read this twitter thread .

Gate length measurement
Gate length measurement

This process isn’t ideal and I want to make some changes so it’s CMOS compatible but it simplifies fabrication and makes it possible with a minimal set of tools. The 1µm dielectric layer (orange) would ideally be CVD SiO2 (it’s possible to build a TEOS oxide reactor at home) but I used a photoresist instead. Most photoresists can be baked around 250°C to form a hard permanent dielectric layer that is an easy alternative to CVD or PECVD oxide. A spin-on-glass/sol-gel could also be used here. SiO2 etching is done with a buffered HF solution made from rust stain remover or RIE.

Huge composite stitched die image:

0001_stitch

Thanks for following my work and feel free to contact me with your thoughts!

Oblast: a better Blasto game for the Commodore 64

Lobsters
oldvcr.blogspot.com
2025-12-07 01:45:35
Comments...
Original Article

Way back (well, six months ago, anyway), when I was wiring up a Gremlin Blasto arcade board , we talked at length about this 1978 arcade game's history and its sole official home computer port by Milton Bradley to the Texas Instruments 99/4A. In the single player mode you run around in a maze and try to blow up all the mines, which can set off sometimes impressive chain reactions, all the while making sure you yourself don't go up in flames in the process.

The TI-99/4A version was the Blasto I originally remember playing as I never did play Blasto in the arcades. (Also, for the record, we're not talking about Sony's unrelated Blasto for the PlayStation which, other than having the voice talents of the late and lamented Phil Hartman, was apparently a traumatic slog both for its developers and the few people who actually played it.) To the credit of its three composite authors, it is a competent and accurate conversion that also adds configurable options, colour graphics and music; in fact, TI's Blasto is probably my favourite game on the /4A, more so than any other cartridge. On the other hand, because it's an accurate conversion, it also inherits all of the original's weaknesses, which admittedly hail from the limited CPU and ROM capacity of the arcade hardware.

So, in that article, I mentioned two future Blasto projects . One is to save my pennies for a custom arcade cabinet to put the board in, though I just spent a cool grand plus on tires which used up a lot of those pennies and I've also got Christmas presents to buy. But the second was to write my own take on TI Blasto and soup it up. This project is the second one from my bucket list that I've completed. It took a couple years of work on it off and on, but it's finally done, with faster action and animation, a massive number of procedurally generated screens, and fully configurable gameplay.

I've christened it Oblast, and it's free to play on your real Commodore 64 or emulator. Let's talk about what's the same and what's different.

The antediluvian 1978 Blasto ran on Hustle hardware, which was derived from Gremlin's original (and mercilessly copied) Blockade game as designed by Lane Hauck. Programmer Bill Blewitt managed to squeeze Blasto's entire game code, not counting character graphics, into just 2K of ROM. This code had to generate and draw the maze, handle one or two player inputs, handle their projectiles, check for collisions and trigger the audio and "boom" circuits, all while simultaneously setting off explosions that could trigger other explosions and other collisions. In the upright version it also had free game logic. Given its hardware and software size constraints the arcade game's gameplay limitations, which we'll discuss in a moment, were understandable.

When Milton Bradley picked up the license (from Gremlin's new owner Sega) as a developer for the new TI 99/4, they kept the gameplay and basic rules in their home computer port almost identical.

Instead, the programmers added music tracks, a colour display, and multiple configuration options. You could set not only the game's speed (I always played Full Tilt) ...

... but also how the maze was drawn, including whether trails existed (areas of the map pre-cleared for motion) and how dense the mines were.

Likely as a way to emphasize the TMS9918(A)'s colour capabilities, the MB programmers changed the setting of the game to a green earth-bound landscape with blue (?) mines and reworked the spaceships into tanks. The density option probably had an even greater impact on gameplay than the speed setting because a maze with a lot of mines made for a zippier, more explosive game. You could rig some big bangs this way, though these were sadly were let down by the TMS9919/SN76489's relatively weak noise output. The program also vainly tried to play a simple tune during the game but this was inevitably interrupted and forced to start over by any sound effect (i.e., a tank shooting, mines exploding).

As with the original, you have infinite lives but finite time. If you trip on an explosion, or the other player shoots you in two-player mode, you lose valuable seconds until you respawn. However, you respawn at your original spawn point as if you were teleported there, a conceivable failure mode for a fanciful spaceship but an extremely unlikely one for a terrestrial tank, which makes a good segue into some of its other idiosyncrasies:

  • Each player can only have a single projectile in motion at any time. However, as soon as that projectile impacts, you can immediately fire another one. This is clearly motivated by the limited memory in the original game, but I don't know of any artillery shell in real life that works like that!
  • As a related phenomenon, although you can move while an explosion or chain reaction is occurring (with a slight reduction in frame rate), you can't shoot — at least not until the explosions stop, at which point you can once again shoot immediately. This also seems to be a concession to limited available memory as the game can't track multiple chain reactions at once.
  • Tanks absolutely can't go over mines or obstacles; they act as completely impassible barriers. I guess that might make sense with spaceships, but it seems like a rather wussy sort of tank.

Also, there's only one screen. If you shoot all the mines before time runs out, the arcade Blasto would give you a free game in the single player mode, if you were playing on the upright version and if you were in a jurisdiction where free games weren't considered illegal gambling, as they were at the time in some areas (pinball also suffered from this). But that was meaningless on the no-coins-needed TI port, where shooting all the mines would win you a free ... game over screen, the same prize you'd get for losing.

Now, I want to point out that despite those things, I loved TI Blasto and played quite a bit of it. But we can improve on what is already an awful lot of fun.

It took a while to get the fundamentals laid down, and it was immediately obvious that the most important mechanic in the game had to be the chain reaction since everybody likes to blow %@#$ up. Consequently, the code that handles the explosions was the first part of the game I wrote, as I reasoned the game wouldn't be worth completing if I couldn't get it fast or frantic enough. This very early draft was a simple proof of concept using PETSCII graphic characters to model the algorithm; character graphics were a must because doing this on the high-resolution screen would have played like molasses.

The game doesn't track explosions anywhere else but the screen itself: everything it needs to determine the next frame of animation is by looking at what's set on the characters present. It scans the entire playfield each time to do this which necessarily locks the animation to a constant frame rate — even if the whole screen were alive with multiple explosions, it would take nearly exactly as much time as if only one single bomb were being detonated, keeping gameplay speed consistent. I did a lot of code unrolling to make this work as quick as possible and the final draft of the original "screen test" is what's in Oblast now.

The black area is because I already knew I'd be using sprites for the tank and I didn't want to mess around with having to manage the high bit for X coordinates greater than 255, so I reserved the right-hand portion of the screen for an information pane. This also had the nice side effect of reducing how much of the screen must be scanned.

For Oblast, I've concentrated exclusively on the single-player mode in which I played Blasto most, which also simultaneously solved some technical issues. (I may make a two-player version in the future if I figure out good solutions to them.) Although I've kept the spirit of TI Blasto's configurability, I made it extend not just to maze generation but even to the game's core rule set. The configuration portion is largely written as a BASIC stub with some 6502 assembly language helpers for speed, with the bulk of the remainder and the entirety of the main game loop also in assembly language.

There are four preset games, the parameters for which I selected after tweaking them during repeated playtesting. The first is the one I consider "typical" for most players to start with (and the one you'll see the computer attempt to play during Oblast's attract mode), the second has an increased number of bombs, the third adds trails and more Blasto-like rules for more classic gameplay, and the fourth is a completely gonzo game mode which has become my personal favourite after a rough day at work.

If you don't like those presets, or want to tweak them further, there is a full game configuration screen letting you set game modes and the starting level/screen. The game supports up to 384 procedurally generated screens and you can start on any of them from 0 to 255. The screens are generated from constant seed data (in this case the 64's BASIC ROM) and thus designed to generate the same screen with the same parameters, facilitating muscle memory for longer play if you get good.

Like the two versions of Blasto, Oblast has mines (bombs) and obstacles (trees). You can very precisely control the densities of both. You can also have the game generate Blasto-style trails horizontally, vertically or both, you can set how quickly your tank's fuel is exhausted (i.e., your time limit, the only option which cannot be zero), and you can indicate if your tank is invulnerable to explosions and how quickly to cycle your shells. I'll talk about how that works in a moment. If you press one of the preset function keys in the configuration screen, then its settings are loaded as a starting point for you to modify.

For the presets, where a new player wouldn't know exactly the game conditions they trigger, I pondered various in-game ways of informing them and hit on an easy-to-implement "dot matrix printout" motif where the BASIC stub scrolls a "briefing" before starting play, making asynchronous "printer" noises based on the bit pattern of each line's ASCII codes. This same motif is used for the basic built-in help since I had some extra space available.

Once you've got the settings the way you want, or you just want to keep playing the same preset, after a game ends you can bypass the presets and game configuration screens and jump right into a new game with the same settings by pressing the fire button.

Here's two examples of the procedural screen generation at work, both level 0. The top screen is what you'd start at if you chose the "Regular Duty" (F1) preset; the second is "More Like Classic Blasto" (F5). Both have the same seed pointer, and you can see some commonalities in bomb and tree positions, but the second screen has a slightly lower bomb density and a slightly higher tree density plus trails going both horizontally and vertically. Each collection of settings will always generate the same screens on your machine. The game code manually counts the number of bombs and trees at the end of map generation since they may be taken away by trails or in the process of ensuring the tank has a cleared starting position.

Although we're using a custom character set for speed, I still wanted the colour flexibility of high resolution where you can have different text and background colours. To do so Oblast is something of a love letter to one of the VIC-II's more underutilized display modes, extended background colour mode (ECM). ECM supports up to four background colours on the same screen and the main game uses two additional colours besides the green background, the most obvious being the black background of the information pane, but also a yellow background as part of animating explosions. The price you pay for this flexibility is that only 64 characters of the standard 256-entry character set can be used; the two high bits instead become a bit pair to select the background colour.

That meant making a lot of careful decisions about what I was going to actually display and getting those shapes into the first 64 character glyphs, shown here in Ultrafont+ . You'll notice that I've replaced some of the letters and typographic characters with graphic shapes because I knew I would never actually need to display those letters or symbols. Everything you see on the screen except for the tank and the shells is a character in this custom font. On the bright side, this character limit also means we can reduce the memory needed by the game font by 75 percent.

By looking for the bit set for the black background of the (impervious) information pane, as well as the wall character that also has this bit set, the game knows not to propagate explosions into that area. The yellow background comes in for knowing what needs to detonate next frame: the routine uses that bit as a deferred marker so that as it sweeps sequentially through the screen it doesn't update the same bomb twice in the same frame and prematurely snuff it out. Since setting that bit will also cause a different background colour to be used, we use yellow to make the explosion visually interesting as another side effect.

Parenthetically, the TMS9918 and TMS9918A also have a feature like this which TI Blasto itself appears to use: each 32 character block of its 256-character fonts can have its own colours. Unlike the VIC-II's ECM which must be specially enabled, this is a standard feature of the 32x24 text mode (which TI calls "Graphic I"), but the character shapes remain unchanged in each block which may require making duplicates (whereas with ECM they are always drawn from the first 64 glyphs).

If there are a lot of bombs on screen, as is the case in the fourth preset and my favourite gameplay mode, nearly the entire screen will be consumed with the explosion which animates around you as you shoot other things. This wasn't possible in either of the original Blastos. Also, instead of trying to play music during game play, all three SID voices are used for noise generation (with a low-pass filter and some resonance smeared on for a woofer-like effect). Voice 1 is triggered when you fire your gun and voice 2 is always running as the tank's engine, with its frequency varying with motion and tank rotation. Voice 3 is used specifically for explosions because it's the only SID voice where you can directly sample both the oscillator waveform output and the current amplitude of the audio envelope. We take these samples, scale them to the activity on screen, and feed the result into the VIC-II's screen fine X scroll. Lots of explosions cause lots of shaking, yet the shaking is always in sync with the audio.

Besides the character graphics, the other major screen component are the sprites. The tank is a composite of three sprites: an animated set for the tank tread, the main tank body, and an accent layer. This is sharper than using multicolour sprites where your horizontal resolution is halved. These three sprites move together and the build system automatically precalculates frames to rotate them off a template, which are played back on screen when you turn. Unlike both versions of Blasto where the tank is limited to integral character positions, the tank in Oblast is larger than the bombs and trees and can move in single pixels, though I still limited movement to 90 degree angles so I didn't have to do expensive trig computations to figure out a shell's trajectory.

One sprite being used as the fuel gauge needle left four sprites for the shells. I had earlier considered using character graphics for them too, but animating shells that way would be slower and less smooth than moving sprites. On the other hand, then, without resorting to tricks there can only be four shells onscreen at once which also didn't seem very tank-like. After some thought I came up with a game mechanic to explain it. In the information pane in these two shots, you see the level number, the fuel gauge which acts as your timer, and then four blue shell indicators. Three of these indicators are dark blue, indicating they are reloading (the fourth is a bright cyan, indicating ready). We'll simply define the reloading time for any shell chamber as the maximum length of time it takes a shell to get across the screen in any direction. Thus, no matter how much you fire, you can only ever have four on-screen because the reloading time will lock you out. (Blasto-style fire control where shells recycle immediately as they hit something is preserved for that game mode, or if you turn on "fast cycl[ing] shells" from the configuration screen.)

While propagating explosions is approximately constant-time, other operations in the game may not be, and there's no reason to walk the screen if nothing's marked as needing it. That means we need a timebase to keep frame rates stable. For this purpose I used the Kernal jiffy clock, which on both PAL and NTSC systems is triggered by the Timer A interrupt to tick about every 1/60 second. The game loop locks to this and uses it to know when to move game objects and trigger screen updates. Still, even this isn't fast enough for moving very speedy things like the shells you fire and the game felt too slow. So ... we make the Timer A interrupt even faster, flogging it at 240Hz instead of 60Hz (the game has prescaler values for both PAL and NTSC), making jiffies 1/240 of a second instead and moving objects at that rate.

This does have interesting interactions when the VIC-II is still drawing your screen at either 50 or 60Hz even as you update it four times as quickly, and most of these interactions have to do with collisions because you can move objects faster than the VIC-II can detect they intersect. The bombs are as big as they are because that gives lots of opportunities to detect a shell striking one, but tank collisions remained unreliable with smaller graphics like trees. Fortunately, however, we've already declared we didn't like the fact that trees and bombs (i.e., obstacles and mines) were impassible objects, so we can make this deficiency into a virtue. The game keeps running track of where the tank last was and if a collision is detected immediately moves it back to that position. However, because collisions are detected inconsistently at this rate of motion and the game updates the tank's coordinates faster than the VIC will draw them, it ends up manifesting onscreen as the tank simply slowing down when it has to cross an obstacle. I like that better than just coming to a dead halt.

Explosions, however, are nice big targets and we have no problem detecting when the tank gets nailed by one of those. In the game modes where your tank is vulnerable, we throw your tank into a temporary tailspin, start flashing the border and the information pane (which is just a matter of setting its colour register), turn on voice 1 and voice 3 at the same time for an even bigger boom, and take the envelope and waveform from voice 3 and put it into the fine Y scroll register as well as the X to really throw the screen around. My favourite game mode allows you to blow up the entire playfield with impunity, of course.

I also decided to overhaul the scoring with special bonuses silently awarded after completing a screen and detailed cumulatively at the end when your score is added up (total bombs exploded plus total bonuses earned). Don't cheat and look at the source code, but the descriptions of the bonuses should give you a clue as to how you win them. Note that some bonuses are mutually exclusive, and some are explicitly disabled ("n/a") in certain game configurations that make them impossible or unavoidable.

Should you beat the default high score, you'll see another use of extended background colour mode for the champion medal (you'll just have to beat it fair and square, no spoiler screenshots). This segment uses FLD to scroll the medal into view and then cycles the ECM registers for a masked multiple colour bar effect without having to split the screen horizontally. It's a simple effect that I threw together in an afternoon but I think it looks nice. While the game configuration screen looks like it might use ECM for the top title, it actually doesn't because I needed lowercase letters, so I employ a much simpler trick for that screen which shouldn't take you long to figure out.

A key goal was to get the entire game in memory at once without additional loading or disk access, meaning you can even run it from cassette tape if you want to. In memory everything is arranged around the two character sets, the bank of sprites and the two hi-res title screens which are in fixed locations to deal with the VIC-II's more constrained view of memory (one of the hi-res screens is slotted under the BASIC ROM so I could free up 8K for something else). I then redistributed the various machine language subroutines and the three music tracks around those assets while also ensuring the BASIC menu stub had enough free memory to maintain its variables. After the core game was done I added two more extras on, the attract mode (which required some reworking to fit) and a really silly credits sequence, which implements a double-buffered screen scroller and takes advantage of the fact that the main music track sounds pretty neat slowed down. The entire mess is then single-parted using my custom cross-linker and optionally compressed.

Oblast is freeware and open source on Github . You can build it with Perl 5, the

xa65 cross assembler

and optionally the

pucrunch compressor

. The Perl tools to generate the sprites, the tokenized BASIC code and the uncompressed runnable linked version are all included. Say that you want to change the presets to your own preferred settings: just change the corresponding

DATA

statement in the BASIC code, do a

make

and instantly have your modified binary. All I ask is that modified binaries that you provide to others should use a different name so they aren't confused with the original, and note that this game and any derivative works based on it or its components are under the Floodgap Free Software License .

If you just want to play it, the Github releases tab provides compressed (for actual floppy disks or tape or other media with limited space) and uncompressed (for fast DMA cartridges and emulators) builds as .prg files you can run directly. You'll need a joystick or equivalent in port 2, and the game should run on any PAL or NTSC Commodore 64. This is hardly the last game, let alone project, on my bucketlist , but it's good to knock another one off it. Also, please don't blow up trees in real life.

If you've enjoyed playing, buy me a coffee Pibb .

Eurydice: a Rust to C compiler (yes)

Hacker News
jonathan.protzenko.fr
2025-12-07 01:41:33
Comments...
Original Article

Perhaps the greatest surprise of the last two years was, for me, the realization that people not only care about compiling C to Rust (for obvious reasons, such as, ahem, memory safety) – they also care about compiling Rust to C! Wait, what?

I wrote about this briefly a couple years ago, but the level of interest for the project, I must say, took me somewhat by surprise. So let’s talk about compiling Rust to C a little more today.

Barriers to Rust adoption

Rust is making big progress in terms of adoption, and represents a great value proposition, especially for new code. Both my former employer and my new employer , like pretty much everyone else these days, have big projects that are written in pure Rust or can have Rust components. Even Windows kernel drivers can be written in Rust now. Amazing stuff.

However, if your project is, say, an open-source library that gets compiled on a wonderfully diverse set of target architectures, OSes, distributions and toolchains, well, chances are… one of these is not going to support Rust. Think of a crypto library: there will be people out there with an obscure compiler for a weird embedded target, and they really want to compile your library, because they’ve been told not to roll out their own crypto. Or perhaps you have a format library ridden with memory errors and you want to port it to Rust. Or maybe your company has an in-house analysis that only runs on C code. Regardless of the scenario, there will always be that one legacy use-case that prevents you from switching to Rust until it’s 2035, all those LTS versions (looking at you RHEL) are finally retired, and you yourself are too close to retirement to even care anymore.

That is, unless you’re willing to use a Rust to C compiler.

Why?

Having a backwards-compat scenario where Rust can be compiled to C serves several purposes.

  1. It allows for a gradual transition. The codebase can be ported to Rust, and refactored / cleaned up / rewritten to use all the nice Rust things (data types, pattern-matching, polymorphism, memory safety), thus making you and your developers much, much happier. Meanwhile, the C version co-exists so that you don’t alienate your userbase.
  2. It only requires maintaining a single version. The Rust code is authoritative; the C code is derived from it automatically, either on CI, or at least with a CI job that checks that the two are in sync.
  3. It allows for a census of problematic scenarios. By making the Rust version the default (and putting the fallback C behind a --write-us-an-email flag), there is finally a way to enumerate those mythical users who cannot switch to Rust just yet.

If that sounds appealing, meet Eurydice.

Eurydice is a compiler from Rust to C that aims to produce readable C code. Of course, readability is subjective; also, seeing that Rust relies on whole-program monomorphization, the C code is bound to be more verbose than the Rust code. But you can judge for yourself: here’s the result of compiling libcrux to C .

The output of the test suite is under version control, and there are a lot more tests to peruse. See for instance this bit , compared to the Rust original .

The design of Eurydice

Eurydice plugs in directly at the MIR level, using Charon to avoid reimplementing the wheel and paying the price of interacting with the guts of rustc . Our paper on Charon says more about its architecture.

The advantage of plugging in at the MIR level is that i) we do not have to interpret syntactic sugar, which means our translation is more faithful to the Rust semantics, and ii) we have way fewer constructs that need compiling to C. Even then, it’s no easy feat to translate Rust to C.

There is naturally, the need to perform whole-program monomorphization, over types and const-generic arguments; the compilation of pattern matches into tagged unions; recognizing instances of iterators that can be compiled to native C for -loops. Then, there are more subtle things, such as compiling array repeat expressions sensibly – zero-initializers when possible, initializer lists otherwise, unless it generates too much code, in which case for -loops are preferable. And finally, there are all the rules about visibility, static , inline , etc. that are very C-specific and depend on how you want to lay out your C files.

The translation is complicated by the constraint that the generated code ought to be readable: for instance, we compile Rust structs to C structs, including DST s, by relying on flexible array members . We also work hard to avoid using the fully-generic tagged union pattern when possible, instead eliminating the tag when e.g. the Rust enum only has a single case. Additionally, we rely on Charon to reconstruct control-flow, rather than compile the MIR CFG to C code ridden with goto s; again, this is for code quality.

At a low-level, there were many interesting tidbits.

  • Because arrays in Rust are values, we wrap them within C structs to give them value semantics in C, too; concretely, [u32; 8] becomes struct { uint32_t data[8]; } . (A previous version of Eurydice would emit uint32_t * , and rely on various memcpy s to implement value semantics, but this produced a translation that was not type-generic, and there were plenty of finicky corner cases. We revamped the compilation scheme recently.)
  • The notion of lvalue in C means we need to insert more variable declarations than in Rust – for instance, you can’t trivially compile &[0u32; 1] without naming the array.
  • The fact that the evaluation order is so loosely defined in C means that intermediary computations need to be stored in intermediary variables to enforce the evaluation order.
  • Rust relies on whole-program monomorphization; this means that the C code is inevitably going to contains multiple copies of the same types and functions, but for different choices of type and const generic argumnets. This is currently done with a builtin phase in Eurydice (for historical reasons), but in the long run, we want to rely on Charon’s support for monomorphization.
  • There are plenty of peephole optimizations that are required for good code quality, such as recognizing array::from_fn and generating sensible code that initializes the array in-place (instead of relying on the fully-general compilation scheme for closures), or recognizing instances of the Eq trait that deserve dedicated treatment (such as using memcmp for arrays and slices of flat data).

A final design choice is that for now, Eurydice may define more behaviors than Rust – for instance, Rust panics on integer overflow, but Eurydice-compiled code does not. This is because we assume the input code is verified, and therefore has been shown to be free of panics. This design choice can be easily changed, though.

In practice, as soon as you use traits, the C code becomes more voluminous than the Rust code. We rely on a configuration file mechanism to control the placement of monomorphized instances of a given function, rather than put everything in one big C file. This currently requires a lot of manual intervention to give good results on large projects.

Implementing of Eurydice

Eurydice starts by compiling the MIR AST obtained out of Charon into KaRaMeL ’s internal AST. This is ~3000 lines of OCaml code, so that’s already pretty involved. A lot of the work revolves around trait methods and their monomorphization, given Rust’s expressive trait system.

Then, about 30 nanopasses simplify the KaRaMeL AST until it becomes eligible for compilation to C. Of those, a handful were originally written for KaRaMeL and were somewhat reusable; this includes compilation of data types, as well as monomorphization. The rest was written from scratch for Eurydice, and totals about ~5000 lines of OCaml code.

A particularly gnarly phase was eliminating MIR’s variable assignments as much as possible: in MIR, every variable starts out uninitialized at the beginning of the function; then, in lieu of the variable declaration, we have an assignment with the initial value. Naturally, having a variable declaration in the right spot is better for code quality, so an initial phase tries to reconstruct these assignments. That’s a drawback of using MIR, but we still firmly believe that sticking to something that has clear semantics is ultimately better.

Fun fact: because there are so many peephole optimizations, I got tired of maintaining enormous pattern-matches that would try to catch every flavor of Rust iterator that can be compiled to a C for-loop. Instead, a custom OCaml syntax extension allows writing concrete syntax for the internal KaRaMeL language in OCaml patterns. Those magic patterns then get compiled at compile-time to OCaml AST nodes for an actual OCaml pattern that matches the (deeply-embedded) syntax of KaRaMeL’s AST. This relies on a ppx that lexes, parses and compiles the concrete syntax.

Deploying Eurydice-generated code

Eurydice-generated code expects some hand-written glue that contains macros and static inline functions; sometimes, it’s simply more convenient to write a single macro that uses a type, rather than have Eurydice generate N copies of a polymorphic function that gets specialized each time. A typical example is compiling the Eq trait for arrays: it’s nicer to emit Eurydice_array_eq(a1, a2, len, t) , which macro-expands to !(memcmp(a1, a2, len*sizeof(t))) , rather than have N such functions, each containing a for-loop specialized for different values of t .

Eurydice generates code that is either (C11 and C++20-compatible) or (C++-17 compatible, but not C-compatible). The reason for this is that Rust allows enum values (e.g. Foo { bar: baz } ) in any expression position. For simplicity, Eurydice emits a compound initializer (Foo) { .tag = bar, .value = { .case_Foo = { .bar = baz }}} , or a C++20 aggregate that uses designated initializers, relying on a macro (not shown here) to hide the syntax differences between the two. But C++17 does not have designated initializers, so there is an option for Eurydice to emit different code that relies on member pointers to achieve sensibly the same effect .

Limitations of Eurydice

Naturally, there are many limitations to this approach. Here are the main ones that come to mind:

  • we cannot guarantee that the layout of objects will be the same in C as in Rust; conceivably, one could parse the layout information from MIR, then emit compiler-specific alignment directives to keep the two identical, but this is not done currently;
  • the generated code violates strict aliasing , because creating a user-defined DST involves casting one pointer type (a struct containing an array) to another (a struct with a flexible array member instead); I’m not sure what the best fix is, so for now, please compile your code with -fno-strict-aliasing ;
  • the code that Eurydice sees is MIR after applying cfg tweaks; this means that for code that is intended to be multi-platform, some tricks need to be applied, otherwise, Eurydice will only “see” one version of the code (AVX2, or ARM64, or something else)
  • because monorphization is so pervasive, the configuration language needs to express things such as “types that reference __m256i , an AVX2-only type, need to go into a separate file to be compiled with -mavx2 ”; this can get tedious real fast but I’m not sure I know how to do better.

What’s next?

There is ongoing work to integrate Eurydice-generated code for both Microsoft and Google ’s respective crypto libraries.

The community grew recently, with wonderful contributions by GitHub users @ssyram and @lin23299. There are more in the pipeline, and I look forward to seeing the supported subset of Rust grow even more. Next on the horizon is support for dyn traits via vtables, and relying on Charon’s monomorphization to get MIR exactly as the Rust compiler would monomorphize it, intead of relying on a custom procedure in Eurydice.

An ambitious goal is for the whole standard library of Rust to be extractable via Eurydice in 2026. This is non-trivial, but I believe this achievement is within reach. Stay tuned.

PS: Why the name?

People keep asking about the name; because the project shares a large amount of infrastructure with Aeneas and Charon , I had to follow the Greek mythology theme. Specifically, the myth of Eurydice resonated with me: I thought I was saved from the hell of generating C code , and was going to go back to the world of the living, but alas, no.

Eurydice: a Rust to C compiler (yes)

Lobsters
jonathan.protzenko.fr
2025-12-07 01:39:25
Comments...
Original Article

Perhaps the greatest surprise of the last two years was, for me, the realization that people not only care about compiling C to Rust (for obvious reasons, such as, ahem, memory safety) – they also care about compiling Rust to C! Wait, what?

I wrote about this briefly a couple years ago, but the level of interest for the project, I must say, took me somewhat by surprise. So let’s talk about compiling Rust to C a little more today.

Barriers to Rust adoption

Rust is making big progress in terms of adoption, and represents a great value proposition, especially for new code. Both my former employer and my new employer , like pretty much everyone else these days, have big projects that are written in pure Rust or can have Rust components. Even Windows kernel drivers can be written in Rust now. Amazing stuff.

However, if your project is, say, an open-source library that gets compiled on a wonderfully diverse set of target architectures, OSes, distributions and toolchains, well, chances are… one of these is not going to support Rust. Think of a crypto library: there will be people out there with an obscure compiler for a weird embedded target, and they really want to compile your library, because they’ve been told not to roll out their own crypto. Or perhaps you have a format library ridden with memory errors and you want to port it to Rust. Or maybe your company has an in-house analysis that only runs on C code. Regardless of the scenario, there will always be that one legacy use-case that prevents you from switching to Rust until it’s 2035, all those LTS versions (looking at you RHEL) are finally retired, and you yourself are too close to retirement to even care anymore.

That is, unless you’re willing to use a Rust to C compiler.

Why?

Having a backwards-compat scenario where Rust can be compiled to C serves several purposes.

  1. It allows for a gradual transition. The codebase can be ported to Rust, and refactored / cleaned up / rewritten to use all the nice Rust things (data types, pattern-matching, polymorphism, memory safety), thus making you and your developers much, much happier. Meanwhile, the C version co-exists so that you don’t alienate your userbase.
  2. It only requires maintaining a single version. The Rust code is authoritative; the C code is derived from it automatically, either on CI, or at least with a CI job that checks that the two are in sync.
  3. It allows for a census of problematic scenarios. By making the Rust version the default (and putting the fallback C behind a --write-us-an-email flag), there is finally a way to enumerate those mythical users who cannot switch to Rust just yet.

If that sounds appealing, meet Eurydice.

Eurydice is a compiler from Rust to C that aims to produce readable C code. Of course, readability is subjective; also, seeing that Rust relies on whole-program monomorphization, the C code is bound to be more verbose than the Rust code. But you can judge for yourself: here’s the result of compiling libcrux to C .

The output of the test suite is under version control, and there are a lot more tests to peruse. See for instance this bit , compared to the Rust original .

The design of Eurydice

Eurydice plugs in directly at the MIR level, using Charon to avoid reimplementing the wheel and paying the price of interacting with the guts of rustc . Our paper on Charon says more about its architecture.

The advantage of plugging in at the MIR level is that i) we do not have to interpret syntactic sugar, which means our translation is more faithful to the Rust semantics, and ii) we have way fewer constructs that need compiling to C. Even then, it’s no easy feat to translate Rust to C.

There is naturally, the need to perform whole-program monomorphization, over types and const-generic arguments; the compilation of pattern matches into tagged unions; recognizing instances of iterators that can be compiled to native C for -loops. Then, there are more subtle things, such as compiling array repeat expressions sensibly – zero-initializers when possible, initializer lists otherwise, unless it generates too much code, in which case for -loops are preferable. And finally, there are all the rules about visibility, static , inline , etc. that are very C-specific and depend on how you want to lay out your C files.

The translation is complicated by the constraint that the generated code ought to be readable: for instance, we compile Rust structs to C structs, including DST s, by relying on flexible array members . We also work hard to avoid using the fully-generic tagged union pattern when possible, instead eliminating the tag when e.g. the Rust enum only has a single case. Additionally, we rely on Charon to reconstruct control-flow, rather than compile the MIR CFG to C code ridden with goto s; again, this is for code quality.

At a low-level, there were many interesting tidbits.

  • Because arrays in Rust are values, we wrap them within C structs to give them value semantics in C, too; concretely, [u32; 8] becomes struct { uint32_t data[8]; } . (A previous version of Eurydice would emit uint32_t * , and rely on various memcpy s to implement value semantics, but this produced a translation that was not type-generic, and there were plenty of finicky corner cases. We revamped the compilation scheme recently.)
  • The notion of lvalue in C means we need to insert more variable declarations than in Rust – for instance, you can’t trivially compile &[0u32; 1] without naming the array.
  • The fact that the evaluation order is so loosely defined in C means that intermediary computations need to be stored in intermediary variables to enforce the evaluation order.
  • Rust relies on whole-program monomorphization; this means that the C code is inevitably going to contains multiple copies of the same types and functions, but for different choices of type and const generic argumnets. This is currently done with a builtin phase in Eurydice (for historical reasons), but in the long run, we want to rely on Charon’s support for monomorphization.
  • There are plenty of peephole optimizations that are required for good code quality, such as recognizing array::from_fn and generating sensible code that initializes the array in-place (instead of relying on the fully-general compilation scheme for closures), or recognizing instances of the Eq trait that deserve dedicated treatment (such as using memcmp for arrays and slices of flat data).

A final design choice is that for now, Eurydice may define more behaviors than Rust – for instance, Rust panics on integer overflow, but Eurydice-compiled code does not. This is because we assume the input code is verified, and therefore has been shown to be free of panics. This design choice can be easily changed, though.

In practice, as soon as you use traits, the C code becomes more voluminous than the Rust code. We rely on a configuration file mechanism to control the placement of monomorphized instances of a given function, rather than put everything in one big C file. This currently requires a lot of manual intervention to give good results on large projects.

Implementing of Eurydice

Eurydice starts by compiling the MIR AST obtained out of Charon into KaRaMeL ’s internal AST. This is ~3000 lines of OCaml code, so that’s already pretty involved. A lot of the work revolves around trait methods and their monomorphization, given Rust’s expressive trait system.

Then, about 30 nanopasses simplify the KaRaMeL AST until it becomes eligible for compilation to C. Of those, a handful were originally written for KaRaMeL and were somewhat reusable; this includes compilation of data types, as well as monomorphization. The rest was written from scratch for Eurydice, and totals about ~5000 lines of OCaml code.

A particularly gnarly phase was eliminating MIR’s variable assignments as much as possible: in MIR, every variable starts out uninitialized at the beginning of the function; then, in lieu of the variable declaration, we have an assignment with the initial value. Naturally, having a variable declaration in the right spot is better for code quality, so an initial phase tries to reconstruct these assignments. That’s a drawback of using MIR, but we still firmly believe that sticking to something that has clear semantics is ultimately better.

Fun fact: because there are so many peephole optimizations, I got tired of maintaining enormous pattern-matches that would try to catch every flavor of Rust iterator that can be compiled to a C for-loop. Instead, a custom OCaml syntax extension allows writing concrete syntax for the internal KaRaMeL language in OCaml patterns. Those magic patterns then get compiled at compile-time to OCaml AST nodes for an actual OCaml pattern that matches the (deeply-embedded) syntax of KaRaMeL’s AST. This relies on a ppx that lexes, parses and compiles the concrete syntax.

Deploying Eurydice-generated code

Eurydice-generated code expects some hand-written glue that contains macros and static inline functions; sometimes, it’s simply more convenient to write a single macro that uses a type, rather than have Eurydice generate N copies of a polymorphic function that gets specialized each time. A typical example is compiling the Eq trait for arrays: it’s nicer to emit Eurydice_array_eq(a1, a2, len, t) , which macro-expands to !(memcmp(a1, a2, len*sizeof(t))) , rather than have N such functions, each containing a for-loop specialized for different values of t .

Eurydice generates code that is either (C11 and C++20-compatible) or (C++-17 compatible, but not C-compatible). The reason for this is that Rust allows enum values (e.g. Foo { bar: baz } ) in any expression position. For simplicity, Eurydice emits a compound initializer (Foo) { .tag = bar, .value = { .case_Foo = { .bar = baz }}} , or a C++20 aggregate that uses designated initializers, relying on a macro (not shown here) to hide the syntax differences between the two. But C++17 does not have designated initializers, so there is an option for Eurydice to emit different code that relies on member pointers to achieve sensibly the same effect .

Limitations of Eurydice

Naturally, there are many limitations to this approach. Here are the main ones that come to mind:

  • we cannot guarantee that the layout of objects will be the same in C as in Rust; conceivably, one could parse the layout information from MIR, then emit compiler-specific alignment directives to keep the two identical, but this is not done currently;
  • the generated code violates strict aliasing , because creating a user-defined DST involves casting one pointer type (a struct containing an array) to another (a struct with a flexible array member instead); I’m not sure what the best fix is, so for now, please compile your code with -fno-strict-aliasing ;
  • the code that Eurydice sees is MIR after applying cfg tweaks; this means that for code that is intended to be multi-platform, some tricks need to be applied, otherwise, Eurydice will only “see” one version of the code (AVX2, or ARM64, or something else)
  • because monorphization is so pervasive, the configuration language needs to express things such as “types that reference __m256i , an AVX2-only type, need to go into a separate file to be compiled with -mavx2 ”; this can get tedious real fast but I’m not sure I know how to do better.

What’s next?

There is ongoing work to integrate Eurydice-generated code for both Microsoft and Google ’s respective crypto libraries.

The community grew recently, with wonderful contributions by GitHub users @ssyram and @lin23299. There are more in the pipeline, and I look forward to seeing the supported subset of Rust grow even more. Next on the horizon is support for dyn traits via vtables, and relying on Charon’s monomorphization to get MIR exactly as the Rust compiler would monomorphize it, intead of relying on a custom procedure in Eurydice.

An ambitious goal is for the whole standard library of Rust to be extractable via Eurydice in 2026. This is non-trivial, but I believe this achievement is within reach. Stay tuned.

PS: Why the name?

People keep asking about the name; because the project shares a large amount of infrastructure with Aeneas and Charon , I had to follow the Greek mythology theme. Specifically, the myth of Eurydice resonated with me: I thought I was saved from the hell of generating C code , and was going to go back to the world of the living, but alas, no.

Using LLMs at Oxide

Hacker News
rfd.shared.oxide.computer
2025-12-07 01:17:40
Comments...
Original Article

LLM use varies widely, and the ramifications of those uses vary accordingly; it’s worth taking apart several of the (many) uses for LLMs.

LLMs as readers

LLMs are superlative at reading comprehension, able to process and meaningfully comprehend documents effectively instantly. This can be extraordinarily powerful for summarizing documents — or of answering more specific questions of a large document like a datasheet or specification. (Ironically, LLMs are especially good at evaluating documents to assess the degree that an LLM assisted their creation!)

While use of LLMs to assist comprehension has little downside, it does come with an important caveat: when uploading a document to a hosted LLM (ChatGPT, Claude, Gemini, etc.), there must be assurance of data privacy — and specifically, assurance that the model will not use the document to train future iterations of itself. Note that this may be opt-out (that is, by default, a model may reserve the right to train on uploaded documents), but can generally be controlled via preferences — albeit occasionally via euphemism. (OpenAI shamelessly calls this checked-by-default setting "Improve the model for everyone", making anyone who doesn’t wish the model to train on their data feel as if they suffer from a kind of reactionary avarice.)

A final cautionary note: using LLMs to assist comprehension should not substitute for actually reading a document where such reading is socially expected. More concretely: while LLMs can be a useful tool to assist in the evaluating of candidate materials per [rfd3] , their use should be restricted to be as a tool, not as a substitute for human eyes (and brain!).

LLMs as editors

LLMs can be excellent editors. Engaging an LLM late in the creative process (that is, with a document already written and broadly polished), allows for LLMs to provide helpful feedback on structure, phrasing, etc. — all without danger of losing one’s own voice. A cautionary note here: LLMs are infamous pleasers — and you may find that the breathless praise from an LLM is in fact more sycophancy than analysis. This becomes more perilous the earlier one uses an LLM in the writing process: the less polish a document already has, the more likely it is that an LLM will steer to something wholly different — at once praising your groundbreaking genius while offering to rewrite it for you.

LLMs as writers

While LLMs are adept at reading and can be terrific at editing, their writing is much more mixed. At best, writing from LLMs is hackneyed and cliché-ridden; at worst, it brims with tells that reveal that the prose is in fact automatically generated.

What’s so bad about this? First, to those who can recognize an LLM’s reveals (an expanding demographic!), it’s just embarrassing — it’s as if the writer is walking around with their intellectual fly open . But there are deeper problems: LLM-generated writing undermines the authenticity of not just one’s writing but of the thinking behind it as well. If the prose is automatically generated, might the ideas be too? The reader can’t be sure — and increasingly, the hallmarks of LLM generation cause readers to turn off (or worse).

Finally, LLM-generated prose undermines a social contract of sorts: absent LLMs, it is presumed that of the reader and the writer, it is the writer that has undertaken the greater intellectual exertion. (That is, it is more work to write than to read!) For the reader, this is important: should they struggle with an idea, they can reasonably assume that the writer themselves understands it — and it is the least a reader can do to labor to make sense of it.

If, however, prose is LLM-generated, this social contract becomes ripped up: a reader cannot assume that the writer understands their ideas because they might not so much have read the product of the LLM that they tasked to write it. If one is lucky, these are LLM hallucinations: obviously wrong and quickly discarded. If one is unlucky, however, it will be a kind of LLM-induced cognitive dissonance: a puzzle in which pieces don’t fit because there is in fact no puzzle at all. This can leave a reader frustrated: why should they spend more time reading prose than the writer spent writing it?

This can be navigated, of course, but it is truly perilous: our writing is an important vessel for building trust — and that trust can be quickly eroded if we are not speaking with our own voice. For us at Oxide, there is a more mechanical reason to be jaundiced about using LLMs to write: because our hiring process very much selects for writers, we know that everyone at Oxide can write — and we have the luxury of demanding of ourselves the kind of writing that we know that we are all capable of.

So our guideline is to generally not use LLMs to write, but this shouldn’t be thought of as an absolute — and it doesn’t mean that an LLM can’t be used as part of the writing process. Just please: consider your responsibility to yourself, to your own ideas — and to the reader.

LLMs as code reviewers

As with reading comprehension and editing, LLMs can make for good code reviewers. But they can also make nonsense suggestions or otherwise miss larger issues. LLMs should be used for review (and can be very helpful when targeted to look for a particular kind of issue), but that review should not be accepted as a human substitute.

LLMs as debuggers

LLMs can be surprisingly helpful debugging problems, but perhaps only because our expectations for them would be so low. While LLMs shouldn’t be relied upon (clearly?) to debug a problem, they can serve as a kind of animatronic rubber duck , helping to inspire the next questions to ask. (And they can be surprising: LLMs have been known to debug I2C issues from the screenshot of a scope capture!) When debugging a vexing problem one has little to lose by using an LLM — but perhaps also little to gain.

LLMs as programmers

LLMs are amazingly good at writing code — so much so that there is borderline mass hysteria about LLMs entirely eliminating software engineering as a craft. As with using an LLM to write prose, there is obvious peril here! Unlike prose, however (which really should be handed in a polished form to an LLM to maximize the LLM’s efficacy), LLMs can be quite effective writing code de novo . This is especially valuable for code that is experimental or auxiliary or otherwise throwaway. The closer code is to the system that we ship, the greater care needs to be shown when using LLMs. Even with something that seems natural for LLM contribution (e.g., writing tests), one should still be careful: it’s easy for LLMs to spiral into nonsense on even simple tasks. Still, they can be extraordinarily useful — and can help to provide an entire spectrum of utility in writing software; they shouldn’t be dismissed out of hand.

Wherever LLM-generated code is used, it becomes the responsibility of the engineer. As part of this process of taking responsibility, self-review becomes essential: LLM-generated code should not be reviewed by others if the responsible engineer has not themselves reviewed it. Moreover, once in the loop of peer review, generation should more or less be removed: if code review comments are addressed by wholesale re-generation, iterative review becomes impossible.

In short, where LLMs are used to generate code, responsibility, rigor, empathy and teamwork must remain top of mind.

Trains cancelled over fake bridge collapse image

Hacker News
www.bbc.com
2025-12-07 00:37:15
Comments...
Original Article

Zoe Toase , North West and

Laura O'Neill , North West

BBC/Network Rail A side-by-side photo showing a damaged bridge on the right. A section of the barriers that run along the top of the bridge appears to have collapsed and a pile of rubble can be seen underneath. A large hole can be seen in front of the bridge. The left is a photo of the bridge taken today showing it is undamaged. BBC/Network Rail

A photo taken by a BBC North West Tonight reporter showed the bridge is undamaged

Trains were halted after a suspected AI-generated picture that seemed to show major damage to a bridge appeared on social media following an earthquake.

The tremor, which struck on Wednesday night , was felt across Lancashire and the southern Lake District.

Network Rail said it was made aware of the image which appeared to show major damage to Carlisle Bridge in Lancaster at 00:30 GMT and stopped rail services across the bridge while safety inspections were carried out.

A BBC journalist ran the image through an AI chatbot which identified key spots that may have been manipulated.

Network Rail A photo showing damage to a bridge. A section of the barriers that run along the top of the bridge appears to have collapsed and a pile of rubble can be seen underneath. A large hole can be seen in front of the bridge Network Rail

Network Rail said it was made aware that the image was on social media

Network Rail said the railway line was fully reopened at around 02:00 GMT and it has urged people to "think about the serious impact it could have" before creating or sharing hoax images.

"The disruption caused by the creation and sharing of hoax images and videos like this creates a completely unnecessary delay to passengers at a cost to the taxpayer," a spokesperson said.

"It adds to the high workload of our frontline teams, who work extremely hard to keep the railway running smoothly," the spokesperson said.

"The safety of rail passengers and staff is our number one priority and we will always take any safety concerns seriously."

The British Transport Police said it was "made aware" of the situation but there was no ongoing investigation into the incident.

Network Rail said 32 services including passenger and freight trains were delayed because of hoax.

A spokesperson for the rail provider said a mix of passenger and freight train would have been impacted.

They said some of them would have been directly stopped or slowed while it checked the lines, but a lot of the trains were delayed as a result of earlier services still being in their path.

The spokesperson said many of them would have been local but because of the length of the West Coast Main Line some trains were delayed as far north as Scotland.

A photo showing the bridge is undamaged

A BBC North West reporter visited the bridge today and confirmed it was undamaged

Railway expert Tony Miles said due to the timing of the incident, very few passengers will have been impacted by the hoax as the services passing through at that time were primarily freight and sleeper trains.

"They generally go slow so as not to disturb the passengers trying to sleep - this means they have a bit of leeway to go faster and make up time if they encounter a delay," he said.

"It's more the fact that Network Rail will have had to mobilise a team to go and check the bridge which could impact their work for days."

He urged people to consider hoaxes like this could have on real people.

"If they actually did delay a train it could have impacted someone who had to get to a medical appointment, or a flight or a funeral.

"It may seem like a game, but anyone who's thinking of doing this should consider how it will impact real people."

How to make a macOS screen saver

Lobsters
wadetregaskis.com
2025-12-07 00:23:35
Comments...
Original Article

Magnitude-7.0 earthquake hits in remote wilderness along Alaska-Canada border

Hacker News
apnews.com
2025-12-07 00:11:47
Comments...
Original Article

JUNEAU, Alaska (AP) — A powerful, magnitude-7.0 earthquake struck in a remote area near the border between Alaska and the Canadian territory of Yukon on Saturday. There was no tsunami warning, and officials said there were no immediate reports of damage or injury.

The U.S. Geological Survey said it struck about 230 miles (370 kilometers) northwest of Juneau, Alaska, and 155 miles (250 kilometers) west of Whitehorse, Yukon.

In Whitehorse, Royal Canadian Mounted Police Sgt. Calista MacLeod said the detachment received two 911 calls about the earthquake.

“It definitely was felt,” MacLeod said. “There are a lot of people on social media, people felt it.”

Alison Bird, a seismologist with Natural Resources Canada, said the part of Yukon most affected by the temblor is mountainous and has few people.

“Mostly people have reported things falling off shelves and walls,” Bird said. “It doesn’t seem like we’ve seen anything in terms of structural damage.”

The Canadian community nearest to the epicenter is Haines Junction, Bird said, about 80 miles (130 kilometers) away. The Yukon Bureau of Statistics lists its population count for 2022 as 1,018.

The quake was also about 56 miles (91 kilometers) from Yakutat, Alaska, which the USGS said has 662 residents.

It struck at a depth of about 6 miles (10 kilometers) and was followed by multiple smaller aftershocks.

Struggling Towards an Algebraic Theory of Music

Lobsters
reasonablypolymorphic.com
2025-12-07 00:05:16
Comments...
Original Article

For the last few months, I’ve been trying to come up with a nice, denotational basis for what music is. But I’m running out of steam on the project, so I thought I’d write what I’ve figured out, and what I’ve tried but doesn’t work. Hopefully this will inspire someone to come tell me what I’m being stupid about and help get the whole process unstuck.

What Music Is Not

It’s tempting to gesticulate wildly, saying that music is merely a function from time to wave amplitudes, eg something of the form: 

μ Music = Time -> Amplitude

While I think it’s fair to say that this is indeed the underlying denotation of sound, this is clearly not the denotation of music. For example, we can transpose a song up a semitone without changing the speed—something that’s very challenging without a great deal of in the waveform representation. And we can play a musical phrase backwards, which is probably impossible in a waveform for any timbral envelope.

Since we have now two examples of “reasonable to want to do” with musical objects, which cannot be expressed in terms of a function Time -> Amplitude , we must conceed that waveforms-over-time cannot be the denotation of music.

What Music Might Be

Music is obviously temporal, so keeping the “function from time” part seems relevant. But a function from time to what? As a first attempt: 

data Note = Note Pitch Timbre Volume

μ Music = Time -> Set (Duration, Note)

which, for a given time, returns a set of notes starting at that time, and how long they ought to be played for. An immediate improvement would be to parameterize the above over notes: 

μ (Music a) = Time -> Set (Duration, a)

It’s tempting to try to eliminate more of the structure here with our parametricity, but I was unable to do so. In contrapuntal music, we will want to be able to express two notes starting at the same moment, but ending at different times.

One alluring path here could to write monophonic voices, and combine them together for polyphony: 

μ (Voice a) = {- something like -} Time -> (Duration, a)
μ (Music a) = Time -> Set (Voice a)

Such an encoding has many unfavorable traits. First, it just feels yucky. Why are there two layers of Time ? Second, now I-as-a-composer need to make a choice of which voice I put each note in, despite the fact that this is merely an encoding quirk. So no, I don’t think this is a viable path forward.

So let’s return to our best contender: 

μ (Music a) = Time -> Set (Duration, a)

This definition is trivially a monoid, pointwise over the time structure: 

μ (Music a <> Music b) = Music (μ a <> μ b)
μ mempty = Music mempty

If we think about abstract sets here, rather than Data.Set.Set , such an object is clearly a functor. There are many possible applicatives here, but the pointwise zipper seems most compelling to me. Pictorally: 

pure a
=
  |----- a ----forever...


liftA2 f
  |---- a ----|-- b --|
  |-- x --|---- y ----|
=
  |- fax -|fay|- fby -|

Such an applicative structure is quite nice! It would allow us to “stamp” a rhythm on top of a pure representation of a melody.

However, the desirability of this instance is a point against μ (Music a) = Time -> Set (Duration, a) , since by Conal Elliott’s typeclass morphism rule , the meaning of the applicative here ought to be the applicative of the meaning. Nevertheless, any other applicative structure would be effecitvely useless, since it would require the notes on one side to begin at the same time as the notes on the other. To sketch: 

-- bad instance!
liftA2 f (Music a) (Music b) =
    Music (liftA2 (\(d1, m) (d2, n) -> (d1 <> d2, f m n)) a b)

Good luck finding a musically meaningful pure for such a thing!

Ok, so let’s say we commit to the pointwise zippy instance as our applicative instance. Is there a corresponding monad? Such a thing would substitute notes with more music. My first idea of what to do with such a thing would be to replace chords with texture. For example, we could replace chords with broken chords, or with basslines that target the same notes.

Anyway, the answer is yes, there is such a monad. But it’s musically kinda troublesome. Assume we have the following function: 

notes :: Music a -> [(Maybe Interval, a)]

which will convert a Music a into its notes and an optional temporal interval (optional because pure goes on forever.) Then, we can write our bind as: 

m >>= f = flip foldMap (notes m) \case
  (Nothing, a) -> f a
  (Just (start, duration), a) ->
    offset start $ _ duration $ f a

where offset changes when a piece of music occurs. We are left with a hole of type: 

Duration -> Music a -> Music a

whose semantics sure better be that it forces the given Music to fit in the alotted time. There are two reasonable candidates here: 

scaleTo  :: Duration -> Music a -> Music a
truncate :: Duration -> Music a -> Music a

where scaleTo changes the local interpretation of time such that the entire musical argument is played within the given duration, and truncate just takes the first Duration ’s worth of time. Truncate is too obviously unhelpful here, since the >>= continuation doesn’t know how much time it’s been given, and thus most binds will drop almost all of their resulting music.

Therefore we will go with scaleTo . Which satisfies all of the algebraic (monad) laws, but results in some truly mystifying tunes. The problem here is that this is not an operation which respects musical meter. Each subsequent bind results in a correspondingly smaller share of the pie. Thus by using only bind and mconcat, it’s easy to get a bar full of quarter notes, followed by a bar of sixty-fourth notes, followed by two bars full of of 13-tuplets. If you want to get a steady rhythm out of the whole thing, you need a global view on how many binds deep you’re ever going to go, and you need to ensure locally that you only produce a small powers-of-two number of notes, or else you will accidentally introduce tuplets.

It’s a mess. But algebraically it’s fine.

What Music Seems Like It Should Be

The above foray into monads seems tentatively promising for amateur would-be algorithmic composers (read: people like me.) But I have been reading several books on musical composition lately, and my big takeaway from them is just how damn contextual notes are.

So maybe this means we want more of a comonadic interface. One in which you can extend every note, by taking into account all of the notes in its local vicinity. This feels just as right as the monadic approach does, albeit in a completely different way. Being able to give a comonad instance for Music would require us to somehow reckon with having only a single a at any given time. Which appeals to my functional programmer soul, but again, I don’t know how to do it.

But imagine if we did have a comonadic instance. We could perform voice leading by inspecting what the next note was, and by futzing around with our pitch. We could do some sort of reharmonization by shifting notes around according to what else is happening.

But maybe all of this is just folly.

Music as it’s actually practiced doesn’t seem to have much of the functionaly-compositional properties we like—ie, that we can abstract and encapsulate. But music doesn’t appear to be like that! Instead, a happy melody takes a different character when played on major vs minor chords. Adding a dissonant interval can completely reconceptualize other notes.

It feels like a bit of a bummer to end like this, but I don’t really know where to go from here. I’ve worked something like six completely-different approaches over the last few months, and what’s documented here is the most promising bits and pieces. My next thought is that maybe music actually forms a sheaf , which is to say that it is a global solution that respects many local constraints.

All of this research into music has given me much more thoughts about music qua music which I will try to articulate the next time I have an evening to myself. Until then.

←

Show HN: FuseCells – a handcrafted logic puzzle game with 2,500 levels

Hacker News
apps.apple.com
2025-12-06 23:51:08
Comments...
Original Article

Strategic Thinking Deduction

Free · In‑App Purchases · Designed for iPad

FuseCells blends Sudoku deduction, Minesweeper logic, and Nonogram-style reasoning into a clean cosmic puzzle experience. Every puzzle is handcrafted and fully logical. FuseCells is a clean, no ads, satisfying logic puzzle that blends the deduction of Sudoku, the neighbor logic of Minesweeper, and the pattern reasoning of Nonogram into one fresh cosmic experience. Every puzzle is handcrafted and fully solvable using pure logic, no guessing, no randomness. How It Works: Fill the grid with symbols (Planet, Star, Moon) using number hints. Each hint shows how many neighbors must share the same symbol, in either 4-way or 8-way mode. Green means correct, yellow means possible, red means impossible — follow the logic and the solution emerges naturally. What’s Inside: • Three sectors with unique atmosphere: Solar System, Milky Way, Deep Space • 2500 handcrafted puzzles with progressive difficulty • Crystal rating system that rewards perfect logic • Daily Challenges with bonus rewards • Smart hint system for difficult moments • Beautiful cosmic visuals and a relaxing, minimalist design Why Players Love It: • Pure logical deduction — never guess • No ads in the free version • Works fully offline • Clean, color-coded feedback • Perfect for short sessions or long logic runs • Inspired by classic puzzles but completely original in design Free vs Pro Free: 50 levels per sector, Daily Challenges, all game modes Pro: All 2500 levels, future sectors Start your cosmic logic journey today and see why so many players enjoy this unique puzzle experience. The grid is waiting.

Winter Style Added! • Fresh winter-themed UI design • Bug fixes and improvements • Enhanced performance Stay cozy and enjoy puzzling!

  • Data Not Linked to You

    The following data may be collected but it is not linked to your identity:

    • Purchases
    • Identifiers

Information

  • Seller

    • Igor Cuiumju

  • Size

    • 28.2 MB

  • Category

    • Puzzle

  • Compatibility
    Requires iOS 15.6 or later.

    • iPhone
      Requires iOS 15.6 or later.

    • iPad
      Requires iPadOS 15.6 or later.

    • iPod touch
      Requires iOS 15.6 or later.

    • Mac
      Requires macOS 12.5 or later and a Mac with Apple M1 chip or later.

    • Apple Vision
      Requires visionOS 1.0 or later.

  • Languages
    English and 9 more

    • English, French, German, Italian, Japanese, Korean, Portuguese, Russian, Simplified Chinese, Spanish

  • Age Rating
    4+

    • 4+

  • In-App Purchases
    Yes

    • FuseCells 10 Hints Pack $0.99

    • FuseCells 30 Hints Pack $1.99

    • FuseCells 100 Hints Pack $4.99

    • FuseCells PRO Lifetime $3.99

  • Copyright

    • © Igor Cuiumju

Supports

A fork of Calibre called Clbre, because the AI is stripped out

Hacker News
github.com
2025-12-06 23:50:01
Comments...
Original Article

clbre

clbre is a fork of calibre with the aim of stripping out the AI integration.

I hope to keep it up to date, but for now this is for my own purposes.

All copyrights, trademarks, etc. belong to their owners, and I do not make any claim to them.

calibre

calibre is an e-book manager. It can view, convert, edit and catalog e-books in all of the major e-book formats. It can also talk to e-book reader devices. It can go out to the internet and fetch metadata for your books. It can download newspapers and convert them into e-books for convenient reading. It is cross platform, running on Linux, Windows and macOS.

For more information, see the calibre About page .

Build Status

Screenshots

Screenshots page

Usage

See the User Manual .

Development

Setting up a development environment for calibre .

A tarball of the source code for the current calibre release.

Bugs

Bug reports and feature requests should be made in the calibre bug tracker at Launchpad . GitHub is only used for code hosting and pull requests.

Support calibre

calibre is a result of the efforts of many volunteers from all over the world. If you find it useful, please consider contributing to support its development. Donate to support calibre development .

Building calibre binaries

See Build instructions for instructions on how to build the calibre binaries and installers for all the platforms calibre supports.

calibre package versions in various repositories

Packaging Status

OpenTelemetry Distribution Builder

Hacker News
github.com
2025-12-06 23:41:18
Comments...
Original Article

🚀 OpenTelemetry Distribution Builder

GitHub Release Apache 2.0 License

Build custom OpenTelemetry Collector Distributions from manifest files with a local build utility, Docker, Google Cloud Build, or a GitHub Action.

Quick Start • Documentation • Examples

🤔 Why OpenTelemetry Distribution Builder?

Built on top of the OpenTelemetry Collector Builder (OCB) , it uses a manifest.yaml to define the components you need, then automates packaging for multiple platforms and manages version releases via GitHub.

While OCB (OpenTelemetry Collector Builder) focuses on building single collector binaries, the OpenTelemetry Distribution Builder provides a complete distribution management solution:

  • 🔨 Builds multi-platform binaries using OCB under the hood
  • 📦 Generates installation packages following OTel community best practices
  • 🚀 Automates versioned releases through GitHub Actions
  • 🔄 Simplifies updates through manifest-based configuration

It handles all the complex aspects of managing your own distribution that have historically made building custom collectors challenging. With the OpenTelemetry Distribution Builder, you can focus on defining your components while the tooling takes care of the rest.

✨ Features

  • 🎯 Custom Component Selection : Build distributions with exactly the components you need
  • 🌐 Multi-Platform Support : Build for multiple architectures (amd64, arm64)
  • 📦 Multiple Package Formats : Generate APK, DEB, RPM, and TAR.GZ packages
  • 🔄 GitHub Actions Integration : Seamless CI/CD integration
  • 🚀 Automated Releases : Streamlined versioning and release process
  • 🔍 Platform-Specific Builds : Optimize for your target environment

🚀 Quick Start

  1. Create a new repository

  2. Add your manifest file ( manifest.yaml ): 

    dist:
  name: my-otelcol
  description: My Custom OpenTelemetry Collector Distro
  # ...
extensions:
  -  # ...
exporters:
  -  # ...
processors:
  -  # ...
receivers:
  -  # ...
connectors:
  -  # ...
providers:
  -  # ...

  3. Set up GitHub Actions ( .github/workflows/build.yml ): 

name: OpenTelemetry Distribution Build

on:
   push:
     tags:
       - "v*"
   workflow_dispatch:

  permissions:
    contents: write # This is required for creating/modifying releases

  jobs:
    build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Build the OpenTelemetry distribution using this custom action
      - uses: observiq/otel-distro-builder@v1
        with:
          manifest: "./manifest.yaml"

      # Create a GitHub Release and attach the build artifacts
      # This makes the artifacts available for download from the Releases page
      - name: Create Release
        uses: softprops/action-gh-release@v2
        with:
          files: ${{ github.workspace }}/artifacts/*

  1. Trigger a build : 

    git tag v1.0.0 && git push --tags

  2. (Optional) Build with Docker : 

    docker pull ghcr.io/observiq/otel-distro-builder:main
docker run --rm -v $(pwd):/workspace -v $(pwd)/build:/build ghcr.io/observiq/otel-distro-builder:main \
  --manifest /workspace/manifest.yaml

📚 Documentation

To view detailed guides, see the docs directory.

GitHub Action Configuration

Inputs

Input Description Default
manifest Path to manifest file ./manifest.yaml
platforms Target platforms linux/amd64

Outputs

All generated packages and binaries are available in the ${{ github.workspace }}/artifacts/* folder.

Docker Usage 

# Pull the latest version
docker pull ghcr.io/observiq/otel-distro-builder:latest

# Pull specific version
docker pull ghcr.io/observiq/otel-distro-builder:v1.0.5

# Run a build
docker run --rm -v $(pwd):/workspace -v $(pwd)/build:/build ghcr.io/observiq/otel-distro-builder:main \
  --manifest /workspace/manifest.yaml \
  # Optional
  --artifacts /workspace/artifacts \
  --goos linux \
  --goarch amd64 \
  --ocb-version 0.121.0 \
  --go-version 1.22.1 \ 
  --supervisor-version 0.122.0

🛠️ Development

Prerequisites

  • Python 3
  • Docker
  • Make

Available Commands 

# Show all commands
make help

# Setup development environment
make setup

# Run tests
make test

# Run linting
make lint

Build Scripts

run_cloud_build.sh

Triggers a build using Google Cloud Build: 

./scripts/run_cloud_build.sh -m manifest.yaml -p project_id -b artifact_bucket

Options:

  • -m : Path to manifest file (required)
  • -p : Google Cloud project ID
  • -b : Artifact bucket name

run_local_build.sh

This script is used to build a custom OpenTelemetry Collector distribution using a local Docker container: 

./scripts/run_local_build.sh -m manifest.yaml [-o output_dir] [-v ocb_version] [-g go_version]

# Optionally, run it with
make build-local # to get the latest version of the otelcol and ocb
# Or
make build output_dir=./artifacts ocb_version=0.121.0 go_version=1.22.1 supervisor_version=0.122.0

Options:

  • -m : Path to manifest file (required)
  • -o : Directory to store build artifacts (default: ./artifacts)
  • -v : OpenTelemetry Collector Builder version (default: auto-detected from manifest)
  • -g : Go version to use for building (default: 1.24.1)

The artifacts will be saved to the specified output directory (default: ./artifacts ).

📁 Project Structure 

otel-distro-builder/
├── builder/                # Builder application
│   ├── src/               # Core builder code
│   ├── templates/         # Build templates
│   ├── tests/            # Test suite
│   └── Dockerfile        # Builder image definition
├── action/                # GitHub Action
├── scripts/              # Build scripts
└── Makefile              # Development commands

🔧 Build Process

  1. Builder Image Preparation : Build and push to registry
  2. Manifest Processing : Upload and validate manifest configuration
  3. Build Execution :
    • Download OpenTelemetry Collector Builder (OCB)
    • Generate Go source files
    • Build platform-specific packages
    • Create SBOMs and checksums
  4. Artifact Management : Upload to GitHub, Google Cloud Storage, or save locally

📦 Build Artifacts

The builder produces:

  • 📦 Binary packages (APK, DEB, RPM)
  • 📚 Source tarball
  • 🔧 Raw binary
  • 📋 SBOM files
  • 🔍 Checksums
  • 📝 Build metadata

Storage Locations

  • Local builds : ./artifacts directory
  • Cloud builds : gs://<bucket>/<build_id>/

🔢 Versioning

We follow semantic versioning. The builder is available in several forms:

  • GitHub Action: Use @v1 for latest 1.x version, or @v1.0.5 for specific versions
  • Docker Image: Use main for latest, or version tags like v1.0.5
  • Container Registry: ghcr.io/observiq/otel-distro-builder:main or ghcr.io/observiq/otel-distro-builder:v1.0.5

📚 Examples

Check out our example workflows in .github/workflows/examples/ for common use cases:

  • Multi-platform builds
  • Container publishing
  • Custom package configurations

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

📄 License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Made with ❤️ by the Bindplane team

Kilauea erupts, destroying webcam [video]

Hacker News
www.youtube.com
2025-12-06 23:39:02
Comments...
Original Article

PatchworkOS: An OS for x86_64, built from scratch in C and assembly

Hacker News
github.com
2025-12-06 23:33:23
Comments...
Original Article

PatchworkOS

Build and Test
PatchworkOS is currently in a very early stage of development, and may have both known and unknown bugs.

Desktop Screenshot

PatchworkOS is a modular non-POSIX operating system for the x86_64 architecture that rigorously follows an "everything is a file" philosophy, in the style of Plan9. Built from scratch in C and assembly, its intended to be an educational and experimental operating system.

In the end this is a project made for fun, but the goal is to make a "real" operating system, one that runs on real hardware and has the performance one would expect from a modern operating system without jumping ahead to user space features, a floppy disk driver and a round-robin scheduler is not enough.

Also, this is not a UNIX clone, its intended to be a (hopefully) interesting experiment in operating system design by attempting to use unique algorithms and designs over tried and tested ones. Sometimes this leads to bad results, and sometimes, with a bit of luck, good ones.

Finally, despite its experimental nature and scale, the project aims to remain approachable and educational, something that can work as a middle ground between fully educational operating systems like xv6 and production operating system like Linux.

Will this project ever reach its goals? Probably not, but thats not the point.

Features

Kernel

File System

  • Unix-style VFS with mountpoints, hardlinks, per-process namespaces, etc.
  • Custom Framebuffer BitMaP (.fbmp) image format, allows for faster loading by removing the need for parsing.
  • Custom Grayscale Raster Font (.grf) font format, allows for antialiasing and kerning without complex vector graphics.

User Space

  • Custom C standard library and system libraries.
  • Highly modular shared memory based desktop environment.
  • Theming via config files .
  • Note that currently a heavy focus has been placed on the kernel and low-level stuff, so user space is quite small... for now.

And much more...

Notable Differences with POSIX

  • Replaced fork(), exec() with spawn() .
  • No "user" concept.
  • Non-POSIX standard library.
  • Even heavier focus on "everything is a file".
  • File flags instead of file modes/permissions.

Limitations

  • Currently limited to RAM disks only (Waiting for USB support).
  • Only support for x86_64.

Notable Future Plans

  • Fully Asynchronous I/O and syscalls (io_uring?).
  • USB support (The holy grail).

Doxygen Documentation

As one of the main goals of PatchworkOS is to be educational, I have tried to document the codebase as much as possible along with providing citations to any sources used. Currently, this is still a work in progress, but as old code is refactored and new code is added, I try to add documentation.

If you are interested in knowing more, then you can check out the Doxygen generated documentation . For an overview check the topics section in the sidebar.

Modules

PatchworkOS uses a "modular" kernel design, meaning that instead of having one big kernel binary, the kernel is split into several smaller "modules" that can be loaded and unloaded at runtime. In effect, the kernel can rewrite itself by adding and removing functionality as needed.

This is highly convenient for development but it also has practical advantages, for example, there is no need to load a driver for a device that is not attached to the system, saving memory.

Make your own Module

The process of making a module is intended to be as straightforward as possible. For the sake of demonstration, we will create a simple "Hello, World!" module.

First, we create a new directory in src/kernel/modules/ named hello , and inside that directory we create a hello.c file to which we write the following code: 

#include <kernel/module/module.h>
#include <kernel/log/log.h>

#include <stdint.h>

uint64_t _module_procedure(const module_event_t* event)
{
    switch (event->type)
    {
    case MODULE_EVENT_LOAD:
        LOG_INFO("Hello, World!\n");
        break;
    default:
        break;
    }

    return 0;
}

MODULE_INFO("Hello", "<author>", "A simple hello world module", "1.0", "MIT", "BOOT_ALWAYS");

An explanation of the code will be provided later.

Now we need to add the module to the build system. To do this, just copy a existing module's .mk file without making any modifications. For example, we can copy src/modules/drivers/ps2/ps2.mk to src/modules/hello/hello.mk . The build system will handle the rest, including copying the module to the final image.

Now, we can build and run PatchworkOS using make all run , or we could use make all and then flash the generated bin/PatchworkOS.img file to a USB drive.

Now to validate that the module is working, you can either watch the boot log and spot the Hello, World! message, or you could use grep on the /dev/klog file in the terminal program like so: 

cat /dev/klog | grep "Hello, World!"

This should output something like: 

[   0.747-00-I] Hello, World!

Thats all, if this did not work, make sure you followed all the steps correctly, if there is still issues, feel free to open an issue.

What can I do now?

Whatever you want. You can include any kernel header, or even headers from other modules, create your own modules and include their headers or anything else. There is no need to worry about linking, dependencies or exporting/importing symbols, the kernels module loader will handle all of it for you. Go nuts.

Code Explanation

This code in the hello.c file does a few things. First, it includes the relevant kernel headers.

Second, it defines a _module_procedure() function. This function serves as the entry point for the module and will be called by the kernel to notify the module of events, for example the module being loaded or a device attached. On the load event, it will print using the kernels logging system "Hello, World!" , resulting in the message being readable from /dev/klog .

Finally, it defines the modules information. This information is, in order, the name of the module, the author of the module (thats you), a short description of the module, the module version, the licence of the module, and finally a list of "device types", in this case just BOOT_ALWAYS , but more could be added by separating them with a semicolon ( ; ).

The list of device types is what causes the kernel to actually load the module. I will avoid going into to much detail (you can check the documentation for that), but I will explain it briefly.

The module loader itself has no idea what these type strings actually are, but subsytems within the kernel can specify that "a device of the type represented by this string is now available", the module loader can then load either one or all modules that have specified in their list of device types that it can handle the specified type. This means that any new subsystem, ACPI, USB, PCI, etc, can implement dynamic module loading using whatever types they want.

So what is BOOT_ALWAYS ? It is the type of a special device that the kernel will pretend to "attach" during boot. In this case, it simply causes our hello module to be loaded during boot.

For more information, check the Module Doxygen Documentation .

ACPI (WIP)

PatchworkOS features a from-scratch ACPI implementation and AML parser, with the goal of being, atleast by ACPI standards, easy to understand and educational. It is tested on the Tested Configurations below and against ACPICA's runtime test suite, but remains a work in progress (and probably always will be).

See ACPI Doxygen Documentation for a progress checklist.

See ACPI specification Version 6.6 as the main reference.

What is ACPI?

ACPI or Advanced Configuration and Power Interface is used for alot of things in modern systems but mainly power management and device enumeration/configuration. Its not possible to go over everything here, instead a brief overview of the parts most likely to cause confusion while reading the code will be provided.

It consists of two main parts, the ACPI tables and AML bytecode. If you have completed a basic operating systems tutorial, you have probably seen the ACPI tables before, for example the RSDP, FADT, MADT, etc. These tables are static in memory data structures storing information about the system, they are very easy to parse but are limited in what they can express.

AML or ACPI Machine Language is a turning complete "mini language", and the source of mutch frustration, that is used to express more complex data, primarily device configuration. This is needed as its impossible for any specification to account for every possible hardware configuration that exists currently, much less that may exist in the future. So instead of trying to design that, what if we could just had a small program generate whatever data we wanted dynamically? Well thats more or less what AML is.

Device Configuration

To demonstrate how ACPI is used for device configuration, we will use the PS/2 driver as an example.

If you have followed a basic operating systems tutorial, you have probably implemented a PS/2 keyboard driver at some point, and most likely you hardcoded the I/O ports 0x60 and 0x64 for data and commands respectively, and IRQ 1 for keyboard interrupts.

Using this hardcoded approach will work for the vast majority of systems, but, perhaps surprisingly, there is no standard that guarantees that these ports and IRQs will actually be used for PS/2 devices. Its just a silent agreement that pretty much all systems adhere to for legacy reasons.

But this is where the device configuration from AML comes in, it lets us query the system for the actual resources used by the PS/2 keyboard, so we dont have to rely on hardcoded values.

If you where to decompile the AML bytecode into its original ASL (ACPI Source Language), you might find something like this: 

Device (KBD)
{
    Name (_HID, EisaId ("PNP0303") /* IBM Enhanced Keyboard (101/102-key, PS/2 Mouse) */)  // _HID: Hardware ID
    Name (_STA, 0x0F)  // _STA: Status
    Name (_CRS, ResourceTemplate ()  // _CRS: Current Resource Settings
    {
        IO (Decode16,
            0x0060,             // Range Minimum
            0x0060,             // Range Maximum
            0x01,               // Alignment
            0x01,               // Length
            )
        IO (Decode16,
            0x0064,             // Range Minimum
            0x0064,             // Range Maximum
            0x01,               // Alignment
            0x01,               // Length
            )
        IRQNoFlags ()
            {1}
    })
}

Note that just like C compiles to assembly, ASL compiles to AML bytecode, which is what the OS actually parses.

In the example ASL, we se a Device object representing a PS/2 keyboard. It has a hardware ID ( _HID ), which we can cross reference with a online database to confirm that it is indeed a PS/2 keyboard, a status ( _STA ), which is just a bitfield indicating if the device is present, enabled, etc, and finally the current resource settings ( _CRS ), which is the thing we are really after.

The _CRS might look a bit complicated but focus on the IO and IRQNoFlags entries. Notice how they are specifying the I/O ports and IRQ used by the keyboard? Which in this case are indeed 0x60 , 0x64 and 1 respectively. So in this case the standard held true.

So how is this information used? Durring boot, the _CRS information of each device is parsed by the ACPI subsystem, it then queries the kernel for the needed resources, assigned them to each device and makes the final configuration available to drivers.

Then when the PS/2 driver is loaded, it gets told "you are handling a device with the name \_SB_.PCI0.SF8_.KBD_ (which is just the full path to the device object in the ACPI namespace) and the type PNP0303 ", it can then query the ACPI subsystem for the resources assigned to that device, and use them instead of hardcoded values.

Having access to this information for all devices also allows us to avoid resource conflicts, making sure two devices are not trying to use the same IRQ(s) or I/O port(s).

Of course, it gets way, way worse than this, but hopefully this clarifies why the PS/2 driver and other drivers, might look a bit different than what you might be used to.

Everything is a File

PatchworkOS strictly follows the "everything is a file" philosophy in a way similar to Plan9, this can often result in unorthodox APIs or could just straight up seem overly complicated, but it has its advantages. We will use sockets to demonstrate the kinds of APIs this produces.

Sockets

In order to create a local seqpacket socket, you open the /net/local/seqpacket file. The opened file can be read to return the ID of your created socket. We provide several helper functions to make this easier, first, without any helpers, you would do 

fd_t fd = open("/net/local/seqpacket");
if (fd == ERR) 
{
    /// ... handle error ...
}
char id[32] = {0};
if (read(fd, id, 31) == ERR) 
{
    /// ... handle error ...
}
close(fd);

Using the sread() helper that reads the entire contents of a file descriptor to simplify this to 

fd_t fd = open("/net/local/seqpacket");
if (fd == ERR) 
{
    /// ... handle error ...
}
char* id = sread(fd); 
if (id == NULL) 
{
    /// ... handle error ...
}
close(fd);
// ... do stuff ...
free(id);

Finally, you can use the sreadfile() helper that reads the entire contents of a file from its path to simplify this even further to 

char* id = sreadfile("/net/local/seqpacket"); 
if (id == NULL) 
{
    /// ... handle error ...
}
// ... do stuff ...
free(id);

Note that the socket will persist until the process that created it and all its children have exited. Additionally, for error handling, all functions will return either NULL or ERR on failure, depending on if they return a pointer or an integer type respectively. The per-thread errno variable is used to indicate the specific error that occurred, both in user space and kernel space.

The ID that we have retrieved is the name of a directory in the /net/local directory, in which are three files that we use to interact with the socket. These files are:

  • data : used to send and retrieve data
  • ctl : Used to send commands
  • accept : Used to accept incoming connections

So, for example, the sockets data file is located at /net/local/[id]/data .

Say we want to make our socket into a server, we would then use the bind and listen commands with the ctl file, Once again, we provide several helper functions to make this easier. First, without any helpers, you would do 

char ctlPath[MAX_PATH] = {0};
if (snprintf(ctlPath, MAX_PATH, "/net/local/%s/ctl", id) < 0) 
{
    /// ... handle error ...
}
fd_t ctl = open(ctlPath);
if (ctl == ERR) 
{
    /// ... handle error ...
}
if (write(ctl, "bind myserver && listen") == ERR) // We use && to chain commands.
{
    /// ... handle error ...
}
close(ctl);

Using the F() macro which allocates formatted strings on the stack and the swrite() helper that writes a null-terminated string to a file descriptor, we can simplify this to 

fd_t ctl = open(F("/net/local/%s/ctl", id));
if (ctl == ERR) 
{
    /// ... handle error ...
}
if (swrite(ctl, "bind myserver && listen") == ERR)
{
    /// ... handle error ...
}
close(ctl);

Finally, using the swritefile() helper that writes a null-terminated string to a file from its path, we can simplify this even further to 

if (swritefile(F("/net/local/%s/ctl", id), "bind myserver && listen") == ERR)
{
    /// ... handle error ...
}

Note that we name our server myserver .

If we wanted to accept a connection using our newly created server, we just open its accept file by writing 

fd_t fd = open(F("/net/local/%s/accept", id));
if (fd == ERR) 
{
    /// ... handle error ...
}
/// ... do stuff ...
close(fd);

The file descriptor returned when the accept file is opened can be used to send and receive data, just like when calling accept() in for example Linux or other POSIX operating systems. The entire socket API attempts to mimic the POSIX socket API, apart from using these weird files everything (should) work as expected.

For the sake of completeness, if we wanted to connect to this server, we can do 

char* id = sreadfile("/net/local/seqpacket"); // Create new socket and get its ID.
if (id == NULL) 
{
    /// ... handle error ...
}
if (swritefile(F("/net/local/%s/ctl", id), "connect myserver") == ERR) // Connect to the server named "myserver".
{
    /// ... handle error ...
}
/// ... do stuff ...
free(id);

Doxygen Documentation

Namespaces

Namespaces are a set of mountpoints that is unique per process, which allows each process a unique view of the file system and is utilized for access control.

Think of it like this, in the common case, you can mount a drive to /mnt/mydrive and all processes can then open the /mnt/mydrive path and see the contents of that drive. However, for security reasons we might not want every process to be able to see that drive, this is what namespaces enable, allowing mounted file systems or directories to only be visible to a subset of processes.

As an example, the "id" directories mentioned in the socket example are a separate "sysfs" instance mounted in the namespace of the creating process, meaning that only that process and its children can see their contents.

To control which processes can see a newly mounted or bound file system or directory, we use a propegation system, where a the newly created mountpoint can be made visible to either just the creating process, the creating process and its children, or the creating process, its children and its parents. Additionally, its possible to specify the behaviour of mountpoint inheritance when a new process is spawned.

Doxygen Documentation

Namespace Sharing

In cases where the propagation system is not sufficient, it's possible for two processes to voluntarily share a mountpoint in their namespaces using bind() in combination with two new system calls share() and claim() .

For example, if process A wants to share its /net/local/5 directory from the socket example with process B, they can do 

// In process A
fd_t dir = open("/net/local/5:directory");

// Create a "key" for the file descriptor, this is a unique one time use randomly generated token that can be used to retrieve the file descriptor in another process.
key_t key;
share(&key, dir, CLOCKS_PER_SEC * 60); // Key valid for 60 seconds (CLOCKS_NEVER is also allowed)

// In process B
// The key is somehow communicated to B via IPC, for example a pipe, socket, argv, etc.
key_t key = ...;

// Use the key to open a file descriptor to the directory, this will invalidate the key.
fd_t dir = claim(key);
// Will error here if the original file descriptor in process A has been closed, process A exited, or the key expired.

// Make "dir" ("/net/local/5" in A) available in B's namespace at "/any/path/it/wants". In practice it might be best to bind it to the same path as in A to avoid confusion.
bind(dir, "/any/path/it/wants");

// Its also possible to just open paths in the shared directory without polluting the namespace using openat().
fd_t somePath = openat(dir, "data");

Note that error checking is ommited for brevity.

This system guarantees consent between processes, and can be used to implement more complex access control systems.

An interesting detail is that when process A opens the net/local/5 directory, the dentry underlying the file descriptor is the root of the mounted file system, if process B were to try to open this directory, it would still succeed as the directory itself is visible, however process B would instead retrieve the dentry of the directory in the parent superblock, and would instead see the content of that directory in the parent superblock. If this means nothing to you, don't worry about it.

Doxygen Documentation

File Flags?

You may have noticed that in the above section sections, the open() function does not take in a flags argument. This is because flags are part of the file path directly so if you wanted to create a non-blocking socket, you can write 

open("/net/local/seqpacket:nonblock");

Multiple flags are allowed, just separate them with the : character, this means flags can be easily appended to a path using the F() macro. Each flag also has a short hand version for which the : character is ommited, for example to open a file as create and exclusive, you can do 

open("/some/path:create:exclusive");

or

For a full list of available flags, check the Doxygen documentation .

Permissions?

Permissions are also specified using file paths there are three possible permissions, read, write and execute. For example to open a file as read and write, you can do 

open("/some/path:read:write");

or

Permissions are inherited, you cant use a file with lower permissions to get a file with higher permissions. Consider the namespace section, if a directory was opened using only read permissions and that same directory was bound, then it would be impossible to open any files within that directory with any permissions other than read.

For a full list of available permissions, check the Doxygen documentation .

But why?

Im sure you have heard many an argument for and against the "everything is a file" philosophy. So I wont go over everything, but the primary reason for using it in PatchworkOS is "emergent behavior" or "composability" which ever term you prefer.

Take the namespace sharing example, notice how there isent any actually dedicated "namespace sharing" system? There are instead a series of small, simple building blocks that when added together form a more complex whole. That is emergent behavior, by keeping things simple and most importantly composable, we can create very complex behaviour without needing to explicitly design it.

Lets take another example, say you wanted to wait on multiple processes with a waitpid() syscall. Well, thats not possible. So now we suddenly need a new system call. Meanwhile, in a "everything is a file system" we just have a pollable /proc/[pid]/wait file that blocks untill the process dies and returns the exit status, now any behaviour that can be implemented with poll() can be used while waiting on processes, including waiting on multiple processes at once, waiting on a keyboard and a process, waiting with a timeout, or any weird combination you can think of.

Plus its fun.

Benchmarks

All benchmarks were run on real hardware using a Lenovo ThinkPad E495. For comparison, I've decided to use the Linux kernel, specifically Fedora since It's what I normally use.

Note that Fedora will obviously have a lot more background processes running and security features that might impact performance, so these benchmarks are not exactly apples to apples, but they should still give a good baseline for how PatchworkOS performs.

All code for benchmarks can be found in the benchmark program , all tests were run using the optimization flag -O3 .

Memory Allocation/Mapping

The test maps and unmaps memory in varying page amounts for a set amount of iterations using generic mmap and munmap functions. Below is the results from PatchworkOS as of commit 4b00a88 and Fedora 40, kernel version 6.14.5-100.fc40.x86_64 . 

xychart-beta
title "Blue: PatchworkOS, Green: Linux (Fedora), Lower is Better"
x-axis "Page Amount (in 50s)" [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]
y-axis "Time (ms)" 0 --> 40000
line [157, 275, 420, 519, 622, 740, 838, 955, 1068, 1175, 1251, 1392, 1478, 1601, 1782, 1938, 2069, 2277, 2552, 2938, 3158, 3473, 3832, 4344, 4944, 5467, 6010, 6554, 7114, 7486]
line [1138, 2226, 3275, 4337, 5453, 6537, 7627, 8757, 9921, 11106, 12358, 13535, 14751, 16081, 17065, 18308, 20254, 21247, 22653, 23754, 25056, 26210, 27459, 28110, 29682, 31096, 33547, 34840, 36455, 37660]
Loading

We see that PatchworkOS performs better across the board, and the performance difference increases as we increase the page count.

There are a few potential reasons for this, one is that PatchworkOS does not use a separate structure to manage virtual memory, instead it embeds metadata directly into the page tables, and since accessing a page table is just walking some pointers, its highly efficient, additionally it provides better caching since the page tables are likely already in the CPU cache.

In the end we end up with a $O(1)$ complexity per page operation, or technically, since the algorithm for finding unmapped memory sections is $O(r)$ in the worst case where $r$ is the size of the address region to check in pages, having more memory allocated would potentially actually improve performance but only by a very small amount. We do of course get $O(n)$ complexity per allocation/mapping operation where $n$ is the number of pages.

Note that as the number of pages increases we start to see less and less linear performance, this is most likely due to CPU cache saturation.

For fun, we can throw the results into desmos to se that around $800$ to $900$ pages there is a "knee" in the curve. Saying that $x$ is the number of pages per iteration and $y$ is the time in milliseconds let us split the data into two sets. We can now perform linear regression which gives us

$$y = \begin{cases} 2.25874x+53.95918 & \text{if } x \leq 850, \quad R^2=0.9987,\\\ 8.68659x-5762.6044 & \text{if } x > 850, \quad R^2=0.9904. \end{cases}$$

Performing quadratic regression on the same data gives us

$$y = \begin{cases} 0.000237618x^{2}+2.06855x+77.77119 & \text{if } x \leq 850, \quad R^2=0.9979,\\\ 0.00626813x^{2}-6.04352x+2636.69231 & \text{if } x > 850, \quad R^2=0.9973. \end{cases}$$

From this we see that for $x \le 850$ the linear regression has a slightly better fit while for $x &gt; 850$ the quadratic regression has a slightly better fit, this is most likely due to the CPU or TLB caches starting to get saturated. All in all this did not tell us much more than we already knew, so it was kinda pointless, but perhaps it is interesting to someone.

Of course, there are limitations to this approach, for example, it is in no way portable (which isn't a concern in our case), each address space can only contain $2^8 - 1$ unique shared memory regions, and copy-on-write would not be easy to implement (however, the need for this is reduced due to PatchworkOS using a spawn() instead of a fork() ).

All in all, this algorithm would not be a viable replacement for existing algorithms, but for PatchworkOS, it serves its purpose very efficiently.

VMM Doxygen Documentation

Paging Doxygen Documentation

Shell Utilities

PatchworkOS includes its own shell utilities designed around its file flags system, when file flags are used we also demonstrate the short form. Included is a brief overview with some usage examples. For convenience the shell utilities are named after their POSIX counterparts, however they are not drop-in replacements.

touch

Opens a file path and then immediately closes it. 

# Create the file.txt file only if it does not exist.
touch file.txt:create:exclusive
touch file.txt:ce

# Create the mydir directory.
touch mydir:create:directory
touch mydir:cd

cat

Reads from stdin or provided files and outputs to stdout. 

# Read the contents of file1.txt and file2.txt.
cat file1.txt file2.txt

# Read process exit status (blocks until process exits)
cat /proc/1234/wait

# Copy contents of file.txt to dest.txt and create it.
cat < file.txt > dest.txt:create
cat < file.txt > dest.txt:c

echo

Writes to stdout. 

# Write to file.txt.
echo "..." > file.txt

# Append to file.txt, makes ">>" unneeded.
echo "..." > file.txt:append
echo "..." > file.txt:a

ls

Reads the contents of a directory to stdout. 

# Prints the contents of mydir.
ls mydir

# Recursively print the contents of mydir.
ls mydir:recursive
ls mydir:R

rm

Removes a file or directory. 

# Remove file.txt.
rm file.txt

# Recursively remove mydir and its contents.
rm mydir:directory:recursive
rm mydir:dR

There are other utils available that work as expected, for example stat and link .

Setup

Requirements

Requirement Details
OS Linux (WSL might work, but I make no guarantees)
Tools GCC, make, NASM, mtools, QEMU (optional)

Build and Run 

# Clone this repository, you can also use the green Code button at the top of the Github.
git clone https://github.com/KaiNorberg/PatchworkOS
cd PatchworkOS

# Build (creates PatchworkOS.img in bin/)
make all

# Run using QEMU
make run

Additional commands 

# Clean build files
make clean

# Build with debug mode enabled
make all DEBUG=1

# Build with debug mode enabled and testing enabled (you will need to have iasl installed)
make all DEBUG=1 TESTING=1

# Debug using qemu with one cpu and GDB
make all run DEBUG=1 QEMU_CPUS=1 GDB=1

# Debug using qemu and exit on panic
make all run DEBUG=1 QEMU_EXIT_ON_PANIC=1

# Generate doxygen documentation
make doxygen

# Create compile commands file
make compile_commands

Repo Structure

Source code can be found in the src/ directory, with public API headers in the include/ directory, private API headers are located alongside their respective source files. 

.
├── meta              // Meta files including screenshots, doxygen, etc.
├── lib               // Third party files, for example doomgeneric.
├── root              // Files to copy to the root of the generated image.
└── <src|include>     // Source code and public API headers.
    ├── kernel        // The kernel and its core subsystems.
    ├── modules       // Kernel modules, drivers, filesystems, etc.
    ├── programs      // User space programs.
    ├── libstd        // The C standard library.
    └── libpatchwork  // The PatchworkOS system library, gui, etc.

Grub Loopback

For frequent testing, it might be inconvenient to frequently flash to a USB. You can instead set up the .img file as a loopback device in GRUB.

Add this entry to the /etc/grub.d/40_custom file: 

menuentry "Patchwork OS" {
        set root="[The grub identifer for the drive. Can be retrived using: sudo grub2-probe --target=drive /boot]"
        loopback loop0 /PatchworkOS.img # Might need to be modified based on your setup.
        set root=(loop0)
        chainloader /efi/boot/bootx64.efi
}

Regenerate grub configuration using sudo grub2-mkconfig -o /boot/grub2/grub.cfg .

Finally copy the generated .img file to your /boot directory, this can also be done with make grub_loopback .

You should now see a new entry in your GRUB boot menu allowing you to boot into the OS, like dual booting, but without the need to create a partition.

Troubleshooting

  • QEMU boot failure : Check if you are using QEMU version 10.0.0, as that version has previously caused issues. These issues appear to be fixed currently however consider using version 9.2.3
  • Any other errors? : If an error not listed here occurs or is not resolvable, please open an issue in the GitHub repository.

Testing

Testing uses a GitHub action that compiles the project and runs it for some amount of time using QEMU with DEBUG=1 , TESTING=1 and QEMU_EXIT_ON_PANIC=1 set. This will run some additional tests in the kernel (for example it will clone ACPICA and run all its runtime tests), and if QEMU has not crashed by the end of the allotted time, it is considered a success.

Note that the QEMU_EXIT_ON_PANIC flag will cause any failed test, assert or panic in the kernel to exit QEMU using their "-device isa-debug-exit" feature with a non-zero exit code, thus causing the GitHub action to fail.

Tested Configurations

  • QEMU emulator version 9.2.3 (qemu-9.2.3-1.fc42)
  • Lenovo ThinkPad E495
  • Ryzen 5 3600X | 32GB 3200MHZ Corsair Vengeance

Currently untested on Intel hardware (broke student, no access to hardware). Let me know if you have different hardware, and it runs (or doesn't) for you!

Contributing

Contributions are welcome! Anything from bug reports/fixes, performance improvements, new features, or even just fixing typos or adding documentation.

If you are unsure where to start, check the Todo List .

Check out the contribution guidelines to get started.

Nostalgia

The first Reddit post and image of PatchworkOS from back when getting to user space was a massive milestone and the kernel was supposed to be a UNIX-like microkernel.

Saving Japan's exceptionally rare 'snow monsters'

Hacker News
www.bbc.com
2025-12-06 23:06:53
Comments...
Original Article

Getty Images Dozens of snow-covered trees appear like figures, with the tree in the foreground appearing particularly like a monster. A blue sky is seen behind them (Credit: Getty Images) Getty Images

A unique natural wonder is being eroded. Can Japan bring its breathtaking "juhyo" back from the brink?

Each winter, the upper slopes of Mount Zao in northern Japan – one of the country's best-known ski areas – are transformed. Fir trees coated in thick frost and snow swell into ghostly figures known as "juhyo" or "snow monsters".

Juhyo form only under exceptionally rare atmospheric conditions, emerging when strong, persistent winter winds carry supercooled water droplets that freeze on contact with the local evergreen Aomori todomatsu trees, gradually layering into rime ice .

At Mount Zao, these formations occur during sustained westerly winds of up to 26m per second (85ft per second), with surface air temperatures between -6.3C to -0.1C (21-31F) and unusually high cloud liquid water content. Under these precise conditions, the rime thickens on the windward side of trees into overlapping ridges known as "shrimp tails", the distinctive shapes that cluster together to form the towering juhyo figures.

"Because such precise meteorological and ecological conditions align in very few places, Zao's snow monsters are a phenomenon almost unique to northern Japan," says Fumitaka Yanagisawa, an emeritus professor of geochemistry who studies the juhyo at Yamagata University.

The snow monsters are the biggest winter draw of the Zao area, a mountain range which lies between Japan's Yamagata and Miyagi prefectures and attracts tens of thousands of visitors annually.

But recent research indicates that the monsters are becoming slimmer.

Getty Images Mount Zao's snow monsters are a phenomenon almost unique to northern Japan (Credit: Getty Images) Getty Images

Mount Zao's snow monsters are a phenomenon almost unique to northern Japan (Credit: Getty Images)

In August 2025 , a research team led by Yanagisawa announced findings that quantified what locals have long observed . By analysing identical-angle photographs of Zao's summit taken since 1933, the team measured the thickness of the figures on a six-point scale. The findings (which have not yet been published in a scientific journal) indicate a widespread shrinking of the juhyo.

"In the 1930s, we saw juhyo five to six metres [16-20ft] across," Yanagisawa says. "By the postwar decades, they were often two to three metres [7-10ft]. Since 2019, many are half a metre [1.6ft] or less. Some are barely columns."

The cause is twofold, says Yanagisawa: a warming climate and a forest under attack. The host tree, Aomori todomatsu, suffered a moth outbreak in 2013 that stripped its needles . Bark beetles followed in 2015, boring into weakened trunks. Yamagata officials report that around 23,000 firs, about a fifth of the prefectural side's stands, have died. With fewer branches and leaves, there is little surface for snow and ice to cling to.

Another 2019 study found that in nearby Yamagata City, average temperatures from December to March have risen by about 2C (3.6F) over the past 120 years . The lower altitude limit of juhyo formation has shifted upward in step with this warming, it found, while the juhyo also last for fewer days of the year.

"Unique landscapes are already being lost to climate change," says Akihiko Ito, an ecologist who specialises in forests and climate change at the University of Tokyo.

Yanagisawa Fumitaka Snow monsters forming on the same tree on the Jizōdake summit of Mount Zao in 2008, 2019 and 2022 (Credit: Yanagisawa Fumitaka) Yanagisawa Fumitaka

Snow monsters forming on the same tree on the Jizōdake summit of Mount Zao in 2008, 2019 and 2022 (Credit: Yanagisawa Fumitaka)

Research shows that Japan's warming climate and extreme weather are already damaging many of its high mountain forests . "Seasonal shifts in spring and autumn can harm leaves, and insect outbreaks are expanding. These stresses may reduce forest growth and density," Ito says.

Across Japan's alpine zones, temperatures have been rising faster than the global average since the 1980s. "In scenarios where climate change continues to advance significantly by the end of this century, it is possible that in warmer-than-usual winters, juhyo may no longer form at all," Ito says.

The threat has prompted action across Yamagata. In March 2023, the prefecture launched the Juhyo Revival Conference – a permanent council bringing together researchers, officials, local businesses and residents to coordinate long-term efforts to restore the fir forests and preserve Mount Zao's snow monsters.

Juhyo are not only a natural spectacle but also a pillar of the local economy. "The influx of tourists supports hotels, restaurants and souvenir shops throughout the area," says Genji Akiba, deputy director of the Zao Onsen Tourism Association. "If the juhyo disappear, it would be a huge blow."

"Revival is a strong wish of our citizens," says Yoko Honma, a conservation specialist at Yamagata Prefecture's nature division. Since 2019, the local forest office has transplanted more than 190 naturally regenerated saplings from lower slopes to the summit zone near the ropeway station. "Because it takes 50 to 70 years for these firs to mature, the key is sustaining conservation across generations," says Honma. "We need patience and continuity."

Yanagisawa Fumitaka/ Tohoku Regional Forest Office An area of Mount Zao's Jizōdake summit in 2010 (top left), 2013 (top right), 2020 (lower left) and 2025 (lower right) (Credit: Yanagisawa Fumitaka/ Tohoku Regional Forest Office) Yanagisawa Fumitaka/ Tohoku Regional Forest Office

An area of Mount Zao's Jizōdake summit in 2010 (top left), 2013 (top right), 2020 (lower left) and 2025 (lower right) (Credit: Yanagisawa Fumitaka/ Tohoku Regional Forest Office)

In Murayama, about 20km (12 miles) north-west of Zao, students from a forestry and environmental science course at Murayama Technical High School have also taken up the challenge of reviving the firs.

More like this:

• The overlooked benefits of real Christmas trees

• The secrets of the Amazon's most mysterious river

• The mysterious black fungus from Chernobyl that may eat radiation

Since 2022, the students have been planting Aomori todomatsu trees and studying how to propagate and protect the species. Together with staff from the Yamagata Forest Office, they visit Mount Zao to collect young fir saplings and bring them back to their school for research. There, they cultivate stems through cuttings and experiment with methods for artificially propagating and efficiently producing seedlings.

"It's been challenging," says Rin Oizumi, a second-year student in the course. "When the seeds we sowed in heavy rain finally sprouted, I felt both relief and excitement. But it was heartbreaking to find that some plots had been damaged by field mice, which had eaten the young shoots." The students have also conducted preliminary experiments using branches of a related fir species, which have shown successful germination.

Harada Takato Efforts are being made to transplant of Aomori-todomatsu saplings to Mount Zao in an attempt to revive the forests (Credit: Harada Takato) Harada Takato

Efforts are being made to transplant of Aomori-todomatsu saplings to Mount Zao in an attempt to revive the forests (Credit: Harada Takato)

Kanon Taniai, Oizumi's classmate, recalls seeing more and more fallen or dead trees as she and other students neared the summit one day in July 2024. "It made me feel really sad," she says. "Growing seedlings is hard work, but we want to do what we can to help bring Mount Zao back to life."

For Taniai, protecting the Juhyo means passing their legacy to the next generation. "They are called snow monsters because nothing else looks like them," she says. "I want the world to see them, and to feel how special Japan's nature is."

--

For essential climate news and hopeful developments to your inbox, sign up to the Future Earth newsletter, while The Essential List delivers a handpicked selection of features and insights twice a week.

For more science, technology, environment and health stories from the BBC, follow us on Facebook and Instagram .

Wave of (Open Street Map) Vandalism in South Korea

Hacker News
www.openstreetmap.org
2025-12-06 22:34:56
Comments...
Original Article

About a week ago, vandals began defacing the map in South Korea. Over the course of that week, I rolled back hundreds of changes, and with the help of the site’s moderators, I banned over 50 malicious accounts.

In fact, the problem arose not even a week ago, but about a month ago. Back then, South Korean media reported that one account had allegedly leaked the locations of all the country’s military bases to the public on Openstreetmap. ( link )

Even then, this false media claim struck me as disgusting, but it didn’t cause any major problems. Almost simultaneously, another Korean media report about Openstreetmap appeared: allegedly, an error in the domestic mapping services NaverMap and KakaoMap displaying a river in North Korea was linked to Openstreetmap’s activities: ( link )

It’s hard to say whether this is a deliberate attack on OpenStreetMap or whether the Korean journalists simply lack the basic journalistic training to understand the issue. I’m leaning toward the latter, as if there was a deliberate intent, they would have acted more intelligently. But the result is what matters. This news quickly found its way into the ultra-conservative circles of the Korean right, who are obsessed with conspiracy theories, set up their accounts under the hashtag #YoonAgain, and believe that OpenStreetMap is a creation of Chinese communists and Russian Putinists. They say that yesterday, Russia bombed Ukraine with OpenStreetMap, and tomorrow, North Korea will start bombing the South. So they raided OpenStreetMap with the goal of once again saving (and embarrassing) their country.

But what’s funniest about this whole situation for me is that it wasn’t even Korean military bases that were vandalized, but power plants. Apparently, some employee at one of them (or even at a ministry) was such a crazy person that he managed to convince his superiors that there was data somewhere online that needed to be urgently deleted. He didn’t send an official request, didn’t write on the forum, and refuses to engage in dialogue (I tried). He and at least a few other people simply persistently create accounts on OpenStreetMap and try to delete things that I can restore with a single click.

That’s how it goes :)

United States Antarctic Program Field Manual (2024) [pdf]

Hacker News
www.usap.gov
2025-12-06 22:26:26
Comments...
Original Article
No preview for link for known binary extension (.pdf), Link: https://www.usap.gov/usapgov/travelAndDeployment/documents/Continental-Field-Manual-2024.pdf.

Simon Josefsson: Reproducible Guix Container Images

PlanetDebian
blog.josefsson.org
2025-12-06 22:22:41
Around a year ago I wrote about Guix Container Images for GitLab CI/CD and these images have served the community well. Besides continous use in CI/CD, these Guix container images are used to confirm reproducibility of the source tarball artifacts in the releases of Libtasn1 v4.20, InetUtils v2.6, L...
Original Article

Catala – Law to Code

Hacker News
catala-lang.org
2025-12-06 22:11:33
Comments...
Original Article

How the 'hypnagogic state' of drowsiness could enhance your creativity

Hacker News
theconversation.com
2025-12-06 21:55:29
Comments...
Original Article

The Beatles’ song Yesterday was written in what psychologists refer to as the “hypnagogic state”. This is the twilight zone between sleep and wakefulness, when we drowsily linger in a semi-conscious state, experiencing vivid mental images and sounds.

Waking up one morning in early 1965, Paul McCartney became aware of a long complex melody playing inside his head. He jumped straight out of bed, sat down at his piano and picked out the melody on the keys. He quickly found the chords to go with the melody and created some holding phrases (as songwriters call them, before they write proper lyrics) to fit the melody.

Finding it difficult to believe that such a beautiful melody could emerge spontaneously, McCartney suspected that he was subconsciously plagiarising another composition. As he recalled : “For about a month I went round to people in the music business and asked them whether they had ever heard it before … I thought if no one claimed after a few weeks, then I could have it.” But it turned out to be original.

Many great discoveries and inventions have emerged from the hypnagogic state. The physicist Niels Bohr effectively won the Noble prize while semi-conscious. Drifting off to sleep, he dreamt he saw the nucleus of the atom, with the electrons spinning around it, just like the solar system with the sun and planets – and in this way he “discovered” the structure of the atom.

The sweet spot

Research has shown that the hypnagogic state is a creative “sweet spot.” For example, in a 2021 study , participants in a hypnagogic state were three times more likely to discover the “hidden rule” that could solve a mathematical problem.

Psychologists associate creativity with qualities such as openness to experience and cognitive flexibility . Others have suggested that creativity arises from co-ordination between the cognitive control network of the brain (which deals with planning and problem solving) and the default mode network (which is associated with daydreaming and mind-wandering).

However, in my view, one of the most important theories of creativity is one of the oldest, put forward by the early British psychologist Frederic Myers in 1881. According to Myers, ideas and insights come as a sudden “uprush” from a subliminal mind.

As Myers saw it, our conscious mind is just a small segment of our overall mind, including not only what Sigmund Freud called the unconscious, but also wider and higher levels of consciousness. Ideas may gestate unconsciously for a long time before they emerge into conscious awareness.

Woman making a square with her fingers

Creativity often comes from beyond consciousness. oneinchpunch/Shutterstock

This is why it often feels as if ideas come from beyond the mind, as if they are gifted to us. They can come from beyond our conscious mind.

The importance of relaxation

The hypnagogic state is so creative because, as we hover between sleep and wakefulness, the conscious mind is barely active. For a brief period, our mental boundaries are permeable, and there is a chance creative insights and ideas will flow through from the subliminal mind.

In a more general sense, this is why creativity is often associated with relaxation and idleness. When we relax, our conscious minds are usually less active. Often, when we are busy, our minds are full of chattering thoughts, so there is no space for creative insights to flow through.

This is also why meditation is strongly associated with creativity . Research shows that meditation promotes general creative qualities such as openness to experience and cognitive flexibility.

But perhaps even more importantly, meditation quietens and softens the conscious mind, so that we’re more liable to receive inspiration from beyond it. As I point out in my book The Leap , this is why there is a strong connection between spiritual awakening and creativity.

Nurturing the hypnogogic state

Research has found that around 80% of people have experienced the hypnagogic state, and that around a quarter of the population experience it regularly. It is slightly more common in women than men.

It is most likely to occur at the onset of sleep, but can also occur on waking up, or during the day if we become drowsy and zone out of normal consciousness.

Can we use the hypnagogic state to enhance our creativity? It’s certainly possible to linger in the hypnagogic state, as you probably know from Sunday morning lie-ins.

However, one of the difficulties is capturing the ideas that arise. In our drowsiness, we may not feel the impulse to record of our ideas. It’s tempting to tell ourselves before falling back to sleep, “This is such a good idea that it will definitely stick in my mind.” But when we wake up some time later, the idea is gone forever.

However, through mental training, there is no reason why we can’t build up a habit of recording our hypnagogic ideas. The best practice is to keep a pen and paper right on a bedside table. Or for a more contemporary variant, keep your phone beside the bed, with the recording app open.

In fact, this is a practice that Paul McCartney has always followed. He even trained himself to write in the dark for this purpose.

We can also use a technique of “ conscious napping ” to generate ideas. Whenever the great inventor Thomas Edison was stuck for a solution or new idea, he would allow himself to drift into unconsciousness, while holding a metal ball. As he fell asleep, the ball would clatter to the ground and wake him, when he would often find that a new insight had emerged.

More generally, we should use idleness as a way of cultivating creativity. Don’t think of napping or relaxing as a waste of time. Far from being unproductive, they may lead to the most inspired ideas and insights of our lives.

This article features references to books that have been included for editorial reasons, and may contain links to bookshop.org. If you click on one of the links and go on to buy something from bookshop.org The Conversation UK may earn a commission.

Screenshots from developers: 2002 vs. 2015 (2015)

Hacker News
anders.unix.se
2025-12-06 21:55:09
Comments...
Original Article

In 2002 I asked a number of developers/Unix people for screenshots of their desktops. I recently republished them , and, seeing the interest this generated, I thought it’d be fun to ask the same people* again 13 years later. To my delight I managed to reach many of them.

* Sans Dennis Ritchie and itojun, who are no longer with us.

So, without further ado:

July 2002

my desktop is pretty boring, since it consists of xterm windows to whatever unix system i am using at the moment. the machine itself is likely to be running some x-window server like exceed on some flavor of windows, though for many years i just used an x terminal.

October 2015

If you thought it was boring last time, check this out!

2002:

I don’t know how to make a screenshot, because I normally use my computer in text-mode. I have X and GNOME installed, but I use them only occasionally.

2015:

Under X, I use the standard environment of Trisquel, but mostly I type at Emacs in a console.

September 2002

Well, my desktop is quite boring. I mostly work with four xterms and a few Netscape windows. The KDE bar hides automatically, you can only see a thin grey line at the bottom.

November 2015

Here is the new one. You'll see that, like before, I have lots of xterms where I work on Vim, Zimbu and email. Now using the Chrome browser, showing off the Zimbu homepage. But clearly everything has become bigger!

September 2002

Linux (2.4.20-pre5), Gnome2, vim, Pine.

October 2015

Not that much has changed in 13 years. Still using Linux. Still just a browser window and a ton of terminals hiding behind them. The main change is that switched from Pine to Thunderbird for email at some point. The OS on my laptop here is Ubuntu with Unity although there are a lot of Debian packages installed so it is a bit of a hybrid at this point. Oh, and yes, my son Carl is a lot older now.

August 2002

Ah, my desktop is pretty boring, I used fvwm 1.24 as my window manager and I try to have no more than 1 or 2 windows open per virtual desktop. I use FreeBSD 4-STABLE as my operating system. I first came across Unix when I got an account on a Pyramid 90x running OSx. This had a dual-universe setup: both AT&T and BSD-style environments, chosen by an environment variable. Initially I was given the AT&T environment, but my friends convinced me to ``come over” to BSD. Since then I’ve been a BSD afficionado.

After OSx, SunOS 3.5 and later SunOS releases, until 386BSD 0.1 came out and I started to run BSD at home. Then when 386BSD transmogrified to FreeBSD, I went with FreeBSD.

In terms of desktop, I’m a command-line guy, always will be. My favourite editor is vi, my favourite shell is tcsh (but kudos to rc for elegance). So I don’t really feel the need for GUI things like Gnome or KDE :-)

October 2015

How things have (and have not changed). I'm still a command-line junkie with at least two xterm windows open. I'm still using a 3x3 virtual desktop. However, instead of fvwm, it is now LXDE. I've also switched from FreeBSD to Linux and I'm running Lubuntu as my distribution.

There are a lot of indispensable GUI tools that I use. These include Firefox, lyx, Gimp, KeepassX, Shutter, viking, dia, Wireshark, calibre, audacity, Handbrake and VLC. But where possible I still prefer to script things. My main development languages are still shell, Perl and C.

My shell is now bash. The vi keystrokes are burned into my fingertips and, as long as vim can be ported to new systems, that will be my text editor until I pass on. My mail client is now mutt (definitely not a web client) and my mail is stored locally, not on someone else's server.

The only issue I have is that, since a job change, I now have to deal with Windoze things. Thus, I have VirtualBox, libreoffice and Wine to help me do that.

I started with Unix on a Pyramid 90x. I now have a smart phone that blows the 90x out of the water on performance, RAM and storage. But I'm so very happy that, somewhere down underneath, there is still a Bourne shell and an operating system that does open(), close(), read(), write(), fork() and exec()!

Jordan Hubbard (FreeBSD co-founder, later Director of UNIX Technology at Apple; now CTO of iXsystems):

July 2002
November 2015

You’ll probably be sad (or perhaps not) to hear that my desktop hasn’t really changed much at all - still OS X, though because OS X has virtual desktops now I have multiple “desktops” (6 of them) where Mail.app runs on one, Safari on another, Calendar, Slack, etc - all on separate desktops. This makes it a bit boring, but here’s the one I probably spend the most time in - the terminal window desktop. :)

Discussion: Hacker News ; reddit: /r/programming , /r/linux

Copy-Item is 27% slower than File Explorer

Hacker News
til.andrew-quinn.me
2025-12-06 21:55:08
Comments...
Original Article
1
2
3
4
5

File Explorer drag & drop     ########## (112 MBps)
Copy-Item                     #######    (82 MBps)
Built in SFTP client          ######     (70 MBps)
Built in robocopy             ##         (25 MBps)
WSL 2 rsync                   #          (13 MBps)

In table form:

Tool Speed (MBps) Difference
Drag and drop ~112 —
Copy-Item ~82 -27%
sftp ~70 -37%
robocopy ~25 MBps -78%
rsync (WSL 2) ~13 MBps -88%

I feel like I’m losing my mind.

It all started innocently enough with a misconfigured NAS. I’ve been making some investments into the homelab recently, both for professional reasons and personal curiosity; by now I consider a couple of long CAT 7 cables and some unmanaged 1 Gbps Ethernet switches to be some of the best money I’ve spent this year. Every device in our home has a cozy data autobahn to rush down to talk to every other device in our home, and what’s more, I’ve actually seen rsync copy files from my NAS to my Debian daily driver at 100+ MBps quite regularly.

But even after I un-mis-configured the NAS, I was surprised to find that rsync on Windows Subsystem for Linux pulled in data at a paltry 9 MBps, for an ideal use case (one large file). Removing the -z compression flag moved us up by 30-50%, which is commendable, but not the order of magnitude increase I was doubtfully hoping for.

Was something wrong with the connection? Getting up from the strandmon would have required physical movement, so instead I just took that same exact file via the File Explorer network share, and just Ctrl-C Ctrl-V’d it onto my desktop - and I was immediately hitting 111-112 MBps again! There goes that theory. The calls are coming from inside the house.

I don’t have much occasion to use these skills these days, but I actually became a somewhat proficient PowerShell user and Windows admin back in my first job. My curiosity was fully piqued by this point so I busted out Old Reliable - robocopy , rsync before rsync. It’s been built in since Windows 7 and handles an absurd number of weird edge cases due to its longevity, so you would expect it to be, well, fast , right. Like this is one of those Laws of Lindy Software that everyone supposedly knows; software that was created to be demanding on system resources 30 years ago is blazing fast to the point of instantaneous today, that’s why everyone loves Vim and Emacs over VS Code, etc.

Not so, apparently! robocopy doesn’t print its progress in seconds out of the box, but it can restart things from where they were last left off, so a few date && robocopy and Ctrl+C’s later, I had a pretty reliable estimate of… 25 MBps.

My jaw just about hit the floor, reader. It was here I began to realize that something was rotten in the state of Denmark. You probably shouldn’t be launching 10- or 100-hour long copy jobs between Windows machines in the first place if you have any other options, but on the other hand, civilization advances when we extend the number of important operations which we can perform without thinking about them, and I would sure like to know that the venerable old tool I would script such enormous jobs around wouldn’t do it at one-fifth the speed of doing it by hand. No, I didn’t investigate robocopy ’s myriad other options in detail, but in this situation I do not think I have to - we are comparing default behaviors against default behaviors! This is also why I didn’t go super deep into the WSL question; I should be able to assume that WSL comes shipped with sane defaults that don’t decimate my network connection because I didn’t set the MTU to 1337 by hand or what have you.

These days Windows also comes with a very nice ssh and sftp client out of the box, and this is where I could feel my heart beating a little faster, because in my day to day work now sftp is actually my preferred method of shuttling files to and from my little corporate laptop. Here, at long last, we crossed the 50 MBps mark - get same-file.dat got a semi-respectable 70 MBps down from the NAS. Only a 37% reduction compared to drag and drop! And here, at least, we start to get some kind of explanation for why this reduction might happen - SFTP is an encrypted protocol, so maybe those CPU cycles add up to a lot of extra work over time or slowdown. That… shouldn’t feel convincing to anyone who gives it more than 15 seconds of thought, but we all live with our eyes wide shut at times.

I had one last thing to test, PowerShell’s good old fashioned Copy-Item . Those of us with a Unix background more or less assume, without looking into it too deeply, that good old fashioned cp is about as fast as you can get for things like this. You’re edging up to the limits of what the actual hardware can do. You assume cp as a binary is laser focused on, well, making copies as fast as possible. Surely Copy-Item would work the same way and be close to parity with File Explorer, right?

Alas. While it was the fastest of the bunch otherwise, Copy-Item still topped out at an average 82 MBps when it came to copying the file from the NAS’s network drive to my local filesystem. I have to say, this was the most disheartening of the bunch in a way; despite all Microsoft’s efforts to the contrary, reaching all the way back to the dawn of PowerShell and the Azure corporate revolution, here they are, still making poor bedraggled sysadmins pay a speed tax for trying to do their work in a minimally sane way.

There are, to my knowledge, no scripting options at all on Windows that come close to the fearsome power of File Explorer’s copy and paste out of the box. I hope you never have to use this knowledge, but if you do, may your 100-hour copy job not accidentally be a 130-, 140, 500-, or 1100-hour one instead just because you chose to try to be clever about things.

(Also hyvää itsenäisyyspäivää , for those who celebrate!)

The past was not that cute

Hacker News
juliawise.net
2025-12-06 21:53:35
Comments...
Original Article

I was excited when cottagecore became a thing. Maybe my interest in retro clothes and handicrafts would be less embarrassing now!

Cottagecore, Pinterest 2025

I still enjoy it. But in spaces focused on old-fashioned vibes, you encounter a lot of people who believe that the past was actually this charming.

1879s farmers through the eyes of the 1970s.
Actual farmer, Texas 1937 .

Laura Ingalls Wilder ‘s Little House on the Prairie books are problematic, and also I will always love them. She wrote about the beauty of family and hard work, but she wrote them because she spent her whole life supporting disabled family members. She and her daughter beautified her “pioneer girl” history to make good books. Her daughter describes the reality:  “It took seven successive years of complete crop failure, with work, weather and sickness that wrecked [my father’s] health permanently, and interest rates of 36 percent on money borrowed to buy food, to dislodge us from that land.”

My own version of this mistake was thinking that people’s personalities were different in the past. I grew up listening to folk music and imagining a past where nice boys would admire a nice quiet girl like me, and I wouldn’t have to figure out dating because everything would just unfold, probably on a May morning. My mother pointed out that a lot of the songs along the lines of “my own true love proved false to me” were about unplanned pregnancies.

I also assumed the bonny lasses in these songs would be wholesome and nice. But were popular girls of the past nicer people than they are now?

Some of my picture came from growing up in the Anglo-American folk dance and music community: it had a lot of aging hippies with graduate degrees. So I came away imagining a past with a lot of the kind of people who become engineers and English teachers. A more accurate picture would have been “Imagine a small town where the same 19 kids form your entire group of peers and potential partners.”

Bookish girls like Belle didn’t really go to live in enchanted castles with huge libraries. They stayed in villages where everyone thought they were weird and their best option was Gaston.

Maybe my favorite podcast episode ever is Rachel Laudan on food history : “I did have the extraordinary good fortune to grow up eating what I think the romantic movement dreams of. We had milk fresh from the cow; I never had pasteurized milk until I went to school. We had fish from the river, pheasant from the farm. The food was extremely good. . . . everything was fresh from the garden. So, I do romanticize—some of that because the taste was often extraordinary. And then I tweak myself and I say, ‘Look, Rachel, your mother spent all day, every day gardening or cooking.’ Essentially. As well as doing other chores. And she said to you, ‘Rachel, it’s servitude. I want you to have a life I didn’t have.’ “

I love living in a time and place where we get to choose aesthetics. I have bread rising in my kitchen right now, and I’m looking forward to baking it in an electric oven that doesn’t require me stacking wood or putting smoke into my house.

So I’ll continue to enjoy retro vibes, and draw on the past for lessons on how to be a human. (For example, making music together is one of life’s great experiences, and it’s a mistake to entirely substitute recorded music for that.) But I’ll enjoy doing so with indoor plumbing, dental care, and a desk job.

Jonathan Dowland: thesis

PlanetDebian
jmtd.net
2025-12-06 21:41:13
It's done! It's over! I've graduated, I have the scroll, I'm staring at the eye-watering prices for the official photographer snap, I'm adjusting to post-thesis life. My PhD thesis revisions have been accepted and my thesis is now available from Newcastle University Library's eThesis repository. A...
Original Article

It's done! It's over! I've graduated, I have the scroll, I'm staring at the eye-watering prices for the official photographer snap, I'm adjusting to post-thesis life.

My PhD thesis revisions have been accepted and my thesis is now available from Newcastle University Library's eThesis repository .

As part of submitting my corrections, I wrote a brief report detailing the changes I made from my thesis at the time of the viva. I also produced a latexdiff marked-up copy of the thesis to visualise the exact changes. In order to shed some light on the post-viva corrections process, at least at my institution, and in the hope that they are some use to someone, I'm sharing those documents:

Coffee link to slower biological aging in those w mental illness–to point

Hacker News
www.kcl.ac.uk
2025-12-06 21:33:03
Comments...
Original Article

New research from King’s College London finds that coffee consumption within the NHS recommended limit is linked to longer telomere lengths – a marker of biological ageing – among people with bipolar disorder and schizophrenia. The effect is comparable to roughly five years younger biological age.

Telomeres are structures that protect DNA. As people get older, their telomeres shorten as part of the natural human ageing process. This process has been shown to be accelerated among people with severe mental illness, such as bipolar disorder and schizophrenia, who have an average life expectancy 15 years shorter than the general population.

Previous research shows that coffee possesses health benefits. It may reduce oxidative stress in the general population, helping slow biological ageing processes like telomere shortening. The new study, published in BMJ Mental Health , explores whether coffee consumption could slow this ageing process among those with severe mental illness.

Researchers at the Institute of Psychiatry, Psychology & Neuroscience measured the effects of coffee consumption on telomere length among 436 participants aged 18 to 65 with schizophrenia, bipolar disorder or major depressive disorder with psychosis.

They found that coffee consumption of up to four cups per day was linked to longer telomeres, comparable to a biological age five years younger than non-coffee drinkers.

The longest telomeres were seen among those who consumed three to four cups per day. Too much coffee reduced this positive effect, with participants who consumed more than four cups having shorter telomeres than those who consumed between three and four cups.

V Mlakar et all 2025 figure: As coffee consumption (X axis) increases up to 3-4 cups, telomere length (Y axis) increases.

Figure from Vid Mlakar et al. 2025: As coffee consumption increases up to 3-4 cups, telomere length increases. At 5+ cups, telomere length begins to shorten again.

These effects remained after accounting for variations in age, sex, ethnicity, medication and tobacco use.

We know that coffee can help slow biological ageing in the general population, but little is known about its effect on people with severe mental illness – a population whose lifespan is already shortened, in part due to age-related diseases. Our study shows that up to four cups of coffee per day is linked to longer telomeres among people with bipolar disorder and schizophrenia. This is comparable to a biological age of five years younger than non-coffee drinkers.

Vid Mlakar, PhD student at King’s College London and first author of the study

Coffee is a beverage that many people consume daily. On one hand, we know that excessive coffee consumption can have negative effects on health, such as reducing sleep quality. However, our new study suggests that coffee consumption up to a certain point may have benefits for biological ageing. Many of the factors that are known to affect biological ageing, such as genetics and negative stressful life experiences, are beyond our control. Lifestyle factors like coffee consumption are something we can actively modify, making research like this particularly valuable.

Dr Monica Aas, MRC Research Fellow at King’s College London and senior author of the study

Dr Aas added: "Studies such as this also support the idea that we should move away from viewing coffee as simply “good or bad”, and instead consider a more balanced view. Still, these results need to be confirmed in other independent studies and longitudinal research before we can determine if this is a causal effect."

Data were from the Norwegian TOP study, collected between 2007 and 2018. The researchers included participants who had available data on mental health diagnosis (assessed using the Structured Clinical Interview for DSM-IV), telomere length (measured by extracting DNA from blood samples) and self-reported coffee consumption.

The researchers note that the study did not have information on the type of coffee consumed (instant versus filter) or the caffeine concentration of each cup. The NHS advises limiting caffeine intake to 400 mg/day (approximately four cups of coffee).

The study was funded by the Research Council of Norway, the KG Jebsen Stiftelsen and an Medical Research Council Fellowship. The team has recently received funding from the British Medical Association’s Margaret Temple grant to investigate telomere shortening in a longitudinal cohort of patients with psychosis. This project will allow them to explore further how several lifestyle factors, as well as stress, influence the rate of telomere shortening over time.

"Coffee intake is associated with telomere length in severe mental disorders" (Vid Mlakar et al.) was published in BMJ Mental Health. DOI: 10.1136/bmjment-2025-301700

For more information, please contact Milly Remmington (School of Mental Health & Psychological Sciences Communications Manager).

Show HN: Tascli, a command line based (human) task and record manager

Hacker News
github.com
2025-12-06 20:56:38
Comments...
Original Article

tascli

Crates.io tests benchmark Downloads

A simple, fast, local CLI tool for tracking tasks and records from unix terminal.

Installation: 

cargo install tascli
# or use brew
brew tap Aperocky/tascli
brew install tascli

tascli demo

Basic Usage

Tasks and records are stored in ~/.local/share/tascli/tascli.db (configurable) with rusqlite .

Tasks

Create tasks with deadlines: 

# Basic tasks
tascli task "Create readme" today
tascli task "Publish package" tomorrow
tascli task "Do taxes" 4/15

# With category
tascli task -c work "Read emails" week

Create recurring tasks: 

tascli task "write diary" daily
tascli task "mortgage payment" "monthly 17th"

List tasks: 

# List active tasks
$ tascli list task

output: 

Task List:
----------------------------------------------------------------------------------------------
| Index  | Category            | Content                               | Deadline            |
----------------------------------------------------------------------------------------------
| 1      | life (recurring)    | write diary                           | Today               |
----------------------------------------------------------------------------------------------
| 2      | tascli              | Add pagination capability for tascli  | Sunday              |
|        |                     | list actions                          |                     |
----------------------------------------------------------------------------------------------
| 3      | tascli              | Add readme section on timestring      | Sunday              |
|        |                     | format                                |                     |
----------------------------------------------------------------------------------------------
| 4      | life                | Do state taxes                        | Sunday              |
----------------------------------------------------------------------------------------------
| 5      | tascli              | Sort list output by time instead of   | Sunday              |
|        |                     | internal id                           |                     |
----------------------------------------------------------------------------------------------
| 6      | tascli              | Fix length issue for unicode chars    | Sunday              |
----------------------------------------------------------------------------------------------
| 7      | life                | Two month pictures - follow the lead  | 4/23                |
|        |                     | from the previous one month pictures  |                     |
----------------------------------------------------------------------------------------------

Complete tasks: 

# Mark index 1 as done
tascli done 1

Completing a task or a recurring tasks will generate a corresponding record.

Search tasks: 

tascli list task --search "rust"

List all tasks in tascli category (including completed) 

tascli list task -s all -c tascli

Example output: 

Task List:
----------------------------------------------------------------------------------------------
| Index  | Category            | Content                               | Deadline            |
----------------------------------------------------------------------------------------------
| 1      | baby (Recurring)    | Mix egg yolk milk for Rowan           | Daily (fulfilled)   |
----------------------------------------------------------------------------------------------
| 2      | tascli              | Fix addition and modification commands| Today (completed)   |
|        |                     | output to have N/A for index          |                     |
----------------------------------------------------------------------------------------------
| 3      | tascli              | Insert guardrail against accidental   | Today (completed)   |
|        |                     | valid syntax like 'task list' that is |                     |
|        |                     | mistakenly made                       |                     |
----------------------------------------------------------------------------------------------
| 4      | tascli              | Create a gif for readme               | Today (completed)   |
----------------------------------------------------------------------------------------------
| 5      | tascli              | Add pagination capability for tascli  | Sunday              |
|        |                     | list actions                          |                     |
----------------------------------------------------------------------------------------------
| 6      | tascli              | Add readme section on timestring      | Sunday              |
|        |                     | format                                |                     |
----------------------------------------------------------------------------------------------

Records

Create records (for tracking events): 

# With current time
tascli record -c feeding "100ML"

# With specific time
tascli record -c feeding -t 11:20AM "100ML"

List records: 

# -d 1 stand for only get last 1 day of record
tascli list record -d 1

Search records: 

tascli list record --search "secret"

Example output: 

Records List:
----------------------------------------------------------------------------------------------
| Index  | Category            | Content                               | Created At          |
----------------------------------------------------------------------------------------------
| 1      | feeding             | 110ML                                 | Today 1:00AM        |
----------------------------------------------------------------------------------------------
| 2      | feeding             | breastfeeding                         | Today 4:10AM        |
----------------------------------------------------------------------------------------------
| 3      | feeding             | 100ML                                 | Today 7:30AM        |
----------------------------------------------------------------------------------------------
| 3      | life (Recurring)    | write diary                           | Today 10:30AM       |
----------------------------------------------------------------------------------------------
| 4      | feeding             | 110ML                                 | Today 11:20AM       |
----------------------------------------------------------------------------------------------

Time Format

This application accepts flexible time strings in various formats:

  • Simple dates : today , tomorrow , yesterday , friday , eom (end of month), eoy (end of year)
  • Date formats : YYYY-MM-DD , MM/DD/YYYY , MM/DD (current year)
  • Time formats : HH:MM , 3:00PM , 3PM
  • Combined : 2025-03-24 15:30 , tomorrow 3PM

When only a date is provided, the time defaults to end of day (23:59:59). When only a time is provided, the date defaults to today.

Recurring Formats (schedules) are applicable to tasks:

  • Recurring Formats : daily , daily 9PM , weekly , weekly Friday 9AM , weekly mon-fri , monthly 1st
  • Recurring Formats (II) : every day , every 9PM , every monday , every 9th of the month , every 2/14

Configuration

If storing the db file in location other than ~/.local/share/tascli/tascli.db is preferred, create a config file: 

{
    "data_dir": "/where/you/want/it"
}

at ~/.config/tascli/config.json to adjust the location of the stored file. Note, if you already have existing tasks, you may want to move/copy the db file there first.

Help

tascli uses clap for argument parsing, use --help to get help on all levels of this cli: 

aperocky@~$ tascli -h
Usage: tascli <COMMAND>

Commands:
  task    add task with end time
  record  add record
  done    Finish tasks
  update  Update tasks or records wording/deadlines
  delete  Delete Records or Tasks
  list    list tasks or records
  help    Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version
aperocky@~$ tascli task -h
add task with end time

Usage: tascli task [OPTIONS] <CONTENT> [TIMESTR]

Arguments:
  <CONTENT>  Description of the task
  [TIMESTR]  Time the task is due, default to EOD

Options:
  -c, --category <CATEGORY>  Category of the task
  -h, --help                 Print help

Show HN: TapeHead – A CLI tool for stateful random access of file streams

Hacker News
github.com
2025-12-06 20:23:56
Comments...
Original Article

TapeHead

A command-line utility for random access of file streams.

Preview 

File: "test.txt" (67 bytes) [RW]

[pos:0]> seek 10
[pos:10]> read . 5
ello!
[in:5, pos:15]> read 9 5
hello
[in:5, pos:14]> quit
Screen recording demo TapeHead demo

The tool runs a REPL session in which you can run commands to read, write, and seek to any position within the file stream. It's quite useful for tasks like debugging a driver, or anything that has to do with file streams.

Installation 

cargo install --git https://github.com/emamoah/tapehead.git

Motivation

I wrote a shabby version of this when I was trying to debug my scull driver , which controls a simple virtual character device. I was amazed I couldn't find a tool that allowed me to statefully seek, read and write to a file, so I just improvised some code. After that, I thought it a good idea to rewrite it properly and publish it, in case someone else one day might need it too.

The initial name I thought to give it was "seeker" (since seeking was the most important aspect) but there was already a crate with that name, so I came up with "TapeHead" which also characterises the tool's behaviour quite well.

Running

Interface 

TapeHead v0.1.0

Author: Emmanuel Amoah (https://emamoah.com/)

Enter "help" for more information.



File: "test.txt" (67 bytes) [RW]

[pos:0]>

After the prologue is a line with the following details about the opened file: 

File: "test.txt" (67 bytes) [RW]
          ^          ^       ^
      filepath   filesize   permissions

The permissions can be one of these three, detected automatically when opening the file:

  • [RW] - Readable & Writable
  • [RO] - Read-only
  • [WO] - Write-only

Prompt

The prompt contains a combination of the following segments:

  • pos:<number> - Current position of the file pointer, always shown. If the stream is not seekable (e.g., a Unix FIFO), it displays a * instead of a number.
  • in:<number> - Number of bytes read from the file after executing the previous command. Not shown if nothing was read.
  • out:<number> - Number of bytes written to the file after executing the previous command. Not shown if nothing was written.

Usage

Commands

The following are the supported commands in the REPL, also accessible through the help command.

  • read <seek> [count]

    • Read count number of bytes from the position specified by seek . If count is omitted, read to the end of the file.

  • readb <seek> [count]

    • Same as read , but prints the contents as a hex dump. Useful for examining raw bytes.

  • write <seek> <contents>

    • Write the given text in contents to the file from the position specified by seek . contents can contain whitespace only after the first non-whitespace character.

  • writeb <seek> <hex bytes>

    • Write the given raw bytes to the file. Bytes are written as space-separated hex values and are case-insensitive. e.g., 6C 6f 6C .

  • seek <seek>

    • Move the file pointer to the position specified by seek .

  • help

    • View this help menu.

  • quit

    • Quit the program.

Seek

The following syntaxes are allowed for commands with a seek argument.

  • .

    • Keep file pointer at its current position.

  • number (e.g. 9 )

    • Move to the number th position from the beginning of the file.

  • +number (e.g. +10 )

    • Move forward number bytes from the current position. It is possible to seek beyond the end of a file.

  • -number (e.g. -2 )

    • Move backwards number bytes from the current position. Seeking to a position before byte 0 is an error.

  • [number]< (e.g. 40< , < )

    • Move to the number 'th byte from the end of the file. If number is omitted, move to the end of the file.

Example commands

  • read . - Read the rest of the file from the current position.
  • write < hello - Seek to the end of the file and write the text "hello".
  • read 0 10 - Read the first 10 bytes of the file.
  • seek 5< - Seek to the 5th-to-last byte of the file.
  • write -5 hello - Move backwards 5 bytes and write "hello".
  • writeb 0 74 61 70 65 68 65 61 64 0a - Write "tapehead" followed by a newline at the beginning of the file.

What Lobsters Says - a browser extension to help you find Lobste.rs discussions

Lobsters
github.com
2025-12-06 20:18:25
I've used a "What HN Says" browser extension (https://github.com/pinoceniccola/what-hn-says-webext) for years and wanted a version for Lobste.rs so I built this small extension using ClojureScript (so you will need a Java runtime installed to build it as of now). It basically let's you see if the cu...
Original Article

What Lobsters Says

A browser extension that shows you if the page you're viewing has been discussed on Lobsters . Built with ClojureScript.

Inspired by What HN Says by pinoceniccola.

Building

Requires Java (for ClojureScript compiler) and either npm or bun. 

# Install dependencies
bun install  # or: npm install

# Development (watch mode with hot reload)
bun run dev  # or: npm run dev

# Production build
bun run build  # or: npm run build

Installing in Browser

Chrome/Edge/Brave:

  1. Go to chrome://extensions/
  2. Enable "Developer mode"
  3. Click "Load unpacked" and select the public/ folder

How It Works

Click the extension icon on any page. It searches Lobsters for discussions about that URL - first checking domain-specific history, then falling back to recent hottest/newest/active stories. Shows up to 4 matching discussions with scores, comment counts, and direct links.

License

MIT

Zebra-Llama: Towards Efficient Hybrid Models

Hacker News
arxiv.org
2025-12-06 20:15:54
Comments...
Original Article

View PDF HTML (experimental)

Abstract: With the growing demand for deploying large language models (LLMs) across diverse applications, improving their inference efficiency is crucial for sustainable and democratized access. However, retraining LLMs to meet new user-specific requirements is prohibitively expensive and environmentally unsustainable. In this work, we propose a practical and scalable alternative: composing efficient hybrid language models from existing pre-trained models. Our approach, Zebra-Llama, introduces a family of 1B, 3B, and 8B hybrid models by combining State Space Models (SSMs) and Multi-head Latent Attention (MLA) layers, using a refined initialization and post-training pipeline to efficiently transfer knowledge from pre-trained Transformers. Zebra-Llama achieves Transformer-level accuracy with near-SSM efficiency using only 7-11B training tokens (compared to trillions of tokens required for pre-training) and an 8B teacher. Moreover, Zebra-Llama dramatically reduces KV cache size -down to 3.9%, 2%, and 2.73% of the original for the 1B, 3B, and 8B variants, respectively-while preserving 100%, 100%, and >97% of average zero-shot performance on LM Harness tasks. Compared to models like MambaInLLaMA, X-EcoMLA, Minitron, and Llamba, Zebra-Llama consistently delivers competitive or superior accuracy while using significantly fewer tokens, smaller teachers, and vastly reduced KV cache memory. Notably, Zebra-Llama-8B surpasses Minitron-8B in few-shot accuracy by 7% while using 8x fewer training tokens, over 12x smaller KV cache, and a smaller teacher (8B vs. 15B). It also achieves 2.6x-3.8x higher throughput (tokens/s) than MambaInLlama up to a 32k context length. We will release code and model checkpoints upon acceptance.

Submission history

From: Mehdi Rezagholizadeh [ view email ]
[v1] Thu, 22 May 2025 20:39:57 UTC (12,646 KB)

PocketMage Is an E Ink PDA For the Modern Era

Lobsters
www.hackster.io
2025-12-06 20:01:51
Comments...
Original Article

Gather ‘round kids, while Uncle Cameron tells you a story. Back in the far away times before smartphones, people had these things called PDAs (Personal Digital Assistants). They were kind of like smartphones, except they weren’t very smart and they lacked most connectivity options. But they were very popular among Business™ types. Now Ashtf is bringing the PDA back for the modern era in the form of the E Ink display-equipped PocketMage .

This is still a work in progress, but it already looks great. PocketMage has a folding clamshell design with a tactile keyboard and two screens. The primary screen is an E Ink display, which has great contrast and consumes very little power. The second, smaller screen is an OLED above the keyboard and it provides auxiliary information. That’s good for content that needs a faster refresh rate than the E Ink display can provide.

Rather than an SoC (System-on-Chip) running a full operating system, Ashtf chose to build PocketMage around an ESP32. That runs his own custom OS written in C++. Like the PDAs of yore, that is really just a launcher for a handful of different apps. Right now, those include a text editor, a file manager, a task tracker, a calendar, a journal, a dice-roller, a USB transfer utility, and a dictionary.

That set of apps actually puts PocketMage ahead of a lot of the PDAs from a few decades ago, which is impressive. Users can also develop their own apps.

Once Ashtf completes the PocketMage, he will release everything as an open-source project so people can build their own PDAs. He will also sell kits.

That should happen soon, as he has already posted a build guide to YouTube.

CATL Expects Oceanic Electric Ships in 3 Years

Hacker News
cleantechnica.com
2025-12-06 20:00:17
Comments...
Original Article

Qinggang Tug 1 CATL battery-powered tugboat, courtesy of CATL.

Support CleanTechnica's work through a Substack subscription or on Stripe .

News in batteries and electrification has been coming fast and furious lately. Recently, Su Yi, the head of CATL’s marine division, stated, “CATL’s marine business already covers inland rivers, lakes, and coastal waters, and is now advancing toward ocean-going applications.”

“In the near future — perhaps within the next three years — we will achieve pure-electric vessels navigating the open seas.”

CATL is determined to provide zero-carbon marine transportation. CATL has not been content to merely make batteries, but has dedicated efforts towards application in several sectors, including grid storage , passenger vehicles, and ships. CATL’s marine division has been in operation since 2017 , expanding efforts in shipping. In 2023, it introduced a comprehensive battery replenishment solution including battery swapping, charging, and a cloud-based system providing shared mobile application of containerized power for optimal efficiency and economics. Those efforts resulted in several major products , containerized mobile power, high-voltage high-power charging systems, and the cloud information platform. This comprehensive system provides a seamless solution for electric ships.

CATL marine battery system, courtesy of CATL.

Sharp-eyed readers may have noticed CATL’s recent discussion with shipping giant Maersk and that it has been busy collaborating with shipping partners. CATL has supplied batteries for over 900 vessels, including the Yangtze River Three Gorges 1 cruise ship , the world’s first pure electric ocean-going passenger ship, and the Qinggang Tug 1 tugboat .

CATL battery-powered cruise ship, courtesy of CATL.
Yujian 77 CATL battery-powered ferry boat, courtesy of CATL.
Qinggang Tug 1 CATL battery-powered tugboat, courtesy of CATL.

Recent battery price drops indicate that possibilities for long range electric shipping are improving. The expected timeline for electric shipping dovetails with the expected timeline for sodium-ion battery (SIB) volume production and resulting cost reductions . The material costs of SIBs are expected to lower costs significantly, opening up applications and speeding up electrification. While passenger transport has been successfully electrified, with EVs surpassing ICE parity with battery costs well below $100/kWh enabling widespread adoption, ships can increasingly take advantage of lower-cost batteries for expanded electrification. Studies have shown that long-distance electric ships with up to 5,000 km of range can be successfully utilized using today’s battery capabilities, without significant weight and volume. CATL appears to be aware of this. Marine division head Su Yi notes CATL’s “ full-spectrum growth ” strategy, with goals to electrify maritime and aviation sectors. Sodium-ion technology may remove the last barrier to widespread maritime electrification.

Photo by Christopher Arcus

Sign up for CleanTechnica's Weekly Substack for Zach and Scott's in-depth analyses and high level summaries , sign up for our daily newsletter , and follow us on Google News !

Advertisement

Have a tip for CleanTechnica? Want to advertise? Want to suggest a guest for our CleanTech Talk podcast? Contact us here .

Sign up for our daily newsletter for 15 new cleantech stories a day . Or sign up for our weekly one on top stories of the week if daily is too frequent.

CleanTechnica uses affiliate links. See our policy here .

CleanTechnica's Comment Policy

Use Python for Scripting

Lobsters
hypirion.com
2025-12-06 19:50:43
Comments...
Original Article

We all like to use the simplest tool that solves our problem, but it’s not always optimal in the long run, nor when you want to run it on multiple machines.

Say, for example, that you have a shell script that builds your project. You want to ensure you can call the script from different folders, so the first thing you want to do is to find the root of the project and make that the working directory: 

SCRIPT_PATH="$(readlink -f "$0")"
PROJECT_ROOT="$(dirname "${SCRIPT_PATH}")"
cd "${PROJECT_ROOT}"

After that, because you have a bug with some build artefacts being stale and cached, you want to remove all the artefacts you’ve built before. That ensures we don’t have any old crap left from an earlier build: 

find build gen -type f \( -name '*.o' -o -name '*.a' \) -print0 \
  | xargs -0 -r rm

Of course, we want a build signature for the release, so we put the build date and git commit into a version file: 

BUILD_DATE="$(date -d 'now' +%F)"

cp version.template build/version.txt

sed -i "s/@VERSION@/${COMMIT_TAG:-dev}/" build/version.txt
sed -i "s/@BUILD_DATE@/${BUILD_DATE}/" build/version.txt

Finally, we build the project with whatever build command(s) the project relies on.

There are several issues here. For me, this script will work great. But for a Mac user, all these steps will fail! The calls to readlink , find , xargs , date and sed all depend on functionality that’s provided by the GNU versions (Linux) of these programs, and they don’t exist or don’t work the same way in the BSD versions (Mac). Similarly, there are a bunch of arguments that work on Mac and not on Linux.

There are ways around all of these problems, but they aren’t elegant. And if you’re only familiar with either Mac or Linux, you’ll often be surprised when the tool doesn’t work on the other OS. This is probably most annoying for Mac users who want to modify a script running on a CI machine, but I can confirm that it’s not fun to start on a new project where your first task is to fix the development script to make it work on Linux.

Python to the Rescue

As the title suggests, Python 3 (hereafter just Python) is a very good alternative to complicated shell scripts. There are four reasons for that:

  1. Python is installed on pretty much every machine
  2. Many developers are familiar with Python (to some extent)
  3. Python has a big and standardised standard library
  4. Python code is easier to read after the fact

Well-known and Already Installed

The most important argument in my eyes is that Python 3 is installed on basically every machine out there.

In addition, we all “know” Python. Maybe we haven’t used it in big projects, but we have all used it at some point – or something that looks like it. That makes it much easier to get into.

Standardised Standard Library

As long as you stick to the standard library, Python will work the same on all the machines you run your script on. The reference implementation, CPython, is the one that 99.8% of developers have installed and are using. The other 0.2% know what they’re doing, but I’d still be surprised if they experience differences: All the implementations I know of work hard to have semantically equivalent behaviour to the reference implementation.

The only thing you need to be aware of is which version of Python you’re relying on. In practice, only relatively new packages may cause issues; if some machines are using an older Python version, some methods or packages may not be available. For common usage in scripts, that doesn’t happen often.

And backwards compatibility is something Python takes seriously – as all big languages should. PEP 387 gives guidance on how breaking changes are introduced, and that seldom happens. But if something’s not deprecated, your code will run just fine for at least 5 years without any issues.

Of course, you want to know ahead of time if you’re using code that is deprecated and may go away in the future. If you add 

import warnings
warnings.simplefilter("default", DeprecationWarning)

at the top of your script, you’ll get deprecation warnings for functions you run in your code, and usually some good tips on how to migrate away.

As an example, datetime.utcnow() was deprecated in version 3.12, and the warning you get if you turn on DeprecationWarning is that you should replace it with datetime.now(timezone.utc) instead. Note that this is likely a “soft deprecation” that will never disappear from the standard library. But if the maintainers do decide to remove it, then PEP 387 states it’ll (most likely) happen in 2028 at the earliest.

Big Standard Library

Together with the fact that the standard library is there and won’t go away, the biggest reason to use Python is that it is big. You can do all the actions I mentioned above with the standard library, and a ton of other things as well.

Need to read/write JSON or XML? No problem. Need to fetch some data from the Internet? There’s an HTTP client built in 1 . You have access to good time types, good data structures, even a sqlite3 package if you need it.

Aside from specific build tools like protoc , openapi or the compiler for the languages/frameworks you use, it’s hard to think of something you can easily do in a shell script that you can’t also do easily in Python.

Easier to Read

I don’t like people saying “language X is easier to read than Y”, especially if the languages have roughly the same semantics. Those arguments/comments are usually a long-winded way of saying “I can read X better because I have used it more often”.

But there’s a bit more weight behind claiming that Python is easier to read than shell scripts. First and foremost, if you don’t use either of them that often, it’s much harder to recall and understand the data types and string operations in a shell script than in Python. For example, here’s the way you capitalise a list of strings in Bash: 

morning_greetings=('hi' 'hello' 'good morning')
energetic_morning_greetings=()

for s in "${morning_greetings[@]}"; do
  energetic_morning_greetings+=( "${s^^}!" )
done

If you don’t use Bash often, then "${morning_greetings[@]}" won’t tell you much, and it’s not obvious why it’s written that way. When you attempt to write it, it’s easy to forget the exact incantation, and you will silently introduce bugs if you forget some of the parts. If you forget the [@] part, you’ll only get back the first element. And if you forget the double quotes, then “good morning” is turned into two elements, not one. And what is ${s^^} really doing?

By the way, don’t add a comma after the elements in the list, because 

morning_greetings=('hi', 'hello', 'good morning')

is the list with the items 'hi,' , 'hello,' and 'good morning' .

Finally, it should be noted that this doesn’t work in ZSH, and if you’re using ZSH, you probably are a bit more of a terminal enthusiast than the average person. And if the enthusiasts can’t use this in their favourite shell, it’s less likely that they will use it in the scripts they make.

As you can see, there are tons of mistakes you’re likely to make if you’re not using Bash often.

The equivalent in Python is this: 

morning_greetings = ['hi', 'hello', 'good morning']
energetic_morning_greetings = \
   [s.upper() + '!' for s in morning_greetings]

This isn’t necessarily easier to grasp the first time you see it: The list comprehension [x for y in z] isn’t immediately understandable. But when you know how it works, then s.upper() is easier to understand than ${s^^} .

That’s one of the reasons that makes it more accurate to say that Python is easier to read: Methods have human-readable names in Python. And while there’s a pattern for Bash’s string operations, it’s still easier for a rusty developer to grasp what s.removesuffix('.com') is doing, compared to ${s%.com} … or what len(morning_greetings) is doing compared to ${#morning_greetings[@]} .

The other thing that gives greater weight to this argument is that Python has a larger vocabulary, which means you don’t have to make as many custom functions or use as many weird workarounds. Sure, morning_greetings.pop(1) isn’t very easy to understand at first glance (it removes the element at index 1), but after that, it makes sense. But since Bash doesn’t have a similar method, you’re forced to do things like this: 

unset 'morning_greetings[1]'
arr=( "${morning_greetings[@]}" )

and that is certainly not easy to understand, and even worse, there’s no docstring to look up to grok what it does.

Don’t Throw the Baby Out with the Bathwater

All of these reasons to use Python come up when your script grows to a certain size, or you’re doing things that are hard or unreadable in Bash. And from experience, when the script’s already in Bash, you won’t even consider rewriting it in Python.

As with all things, there’s obviously no need to port a straightforward 10-20 line Bash script. But if you struggle to comprehend what the heck’s going on next time you’re editing the script, don’t forget that Python’s a good alternative.

The general who refused to crush Tiananmen's protesters

Hacker News
www.economist.com
2025-12-06 19:47:13
Comments...
Original Article

How Kentucky Gov. Andy Beshear Thinks Democrats Can Win Rural America

Portside
portside.org
2025-12-06 19:27:30
How Kentucky Gov. Andy Beshear Thinks Democrats Can Win Rural America Dave Sat, 12/06/2025 - 14:27 ...
Original Article

On Thursday, I asked Beshear about rural America, Democratic messaging, and whether he’s running for president in 2028. This conversation has been lightly edited for length and clarity.

Monica Potts: When I saw the headline, I thought you were going to write about farmers. I didn’t see that, and I was pleasantly surprised. What you did write about is the loss of hospitals, where a lot of rural Americans work now. I’m wondering if you feel that that’s something that people misunderstand about rural America. People still tend to think of it as a largely farming area and largely affected by agricultural policy. Do you think that this is something people miss about the new reality of living in rural America?

Andy Beshear: I think people miss that rural America is more complicated, and they might not think that there are more industries, that there need to be more services, and that there are serious implications of policies like the one that the Trump administration has pushed forward. Our fastest growing industry in rural Kentucky is health care. And so the idea that the “ big, ugly bill ” would gut rural health care means it’s not only reducing options to get health care in rural America, but it’s attacking a foundation of the economy. Every rural hospital we have is the number one payroll in its community and the number two employer behind the public school.

If you remove that business, if you remove those employees who live and spend money in that community, you don’t just close the rural hospital. You may close the local bank, the local coffee shop, the local restaurant, the local insurance company. Not thinking about health care as essential to rural America is not understanding how the economy works.

M.P.: As an example in your piece, you wrote about helping to bring a green paper plant into a rural county in Kentucky. And I read that, and I thought, well, that sounds nice, but we can’t do that everywhere. What are some of the other kinds of ideas about the real investment it would take to help sagging economies in rural counties now?

A.B.: Well, first off, I think we have to be intentional where we can be to locate new jobs in rural Kentucky and in rural America, and I see a greater hunger for it out of the private sector than at any point in my lifetime. We landed the Pratt paper mill , 300-plus new jobs at $40 an hour outside of Henderson, Kentucky, former coal mining town. We put two giant battery plants , two of the biggest in the world, next to a town called Glendale, a very small town in Hardin County, but outside of Elizabethtown. So we’ve created as many rural jobs as we have urban jobs by making sure that we’re putting the opportunity in front of those businesses.

But to make that possible everywhere in rural America, it takes a real investment in infrastructure. It takes making the upfront investment that says to rural Kentucky and rural Americans that we care about you and we want you to be able to compete for those next great jobs. Look at Appalachia, the topography creates big challenges. We are four-laning the Mountain Parkway, which is basically our own interstate-like road to the heart of Appalachia. Why? Because if you want to put a new manufacturing facility, they’re going to want four lanes so that they can ship their products across the United States. It means we also have to invest in water and wastewater, which, you know, there are many parts of rural America that still don’t have clean drinking water, which should be a basic human right, but then you have to have the amount of water necessary to bring in that next opportunity. So in Kentucky, we have programs like what we call our Product Development Initiative , where we put state dollars into improving infrastructure at sites.

M.P.: The plant that you wrote about is an environmentally conscious paper plant. When I see a lot of people talking about Democrats winning back voters, they’re talking about moderating on some issues, including things like the environment, and some cultural issues. I’m wondering if you think that to win voters in rural America, Democrats need to moderate on anything?

A.B.: I think when it comes to jobs, they’re not Democrat or Republican, they’re not left or right. A green job to someone is a job that pays them enough to support their family. I remember that paper plant and the groundbreaking, and we’re in this former coal town, and the owner comes on through a Zoom on a massive screen and says, We’re bringing 350—and then he said the phrase “green jobs”—to Henderson. And everyone stood up and applauded, because they are great jobs where you can support a family. I believe that communities are ready.

And I also believe that sustainability isn’t primarily being driven by government policy. It’s being driven by the demands of the private sector. Every company that comes to Kentucky with their power wants affordability, wants reliability, and then wants sustainability. And so for me, being pragmatic, I’ve got to deliver all three, which means we need greener, more sustainable power production. We need greener jobs, because that’s what the private sector and ultimately consumers are demanding. So no, I don’t think that we have to back away from beliefs about climate change, but I do think within those beliefs, we have to deliver a better life for our people. That means, if you can bring in good, paying green jobs, people of all political ideologies will work in them because it makes life better and easier for their family.

M.P.: One of the things that I thought that President Joe Biden was underappreciated about was that he did make a big effort to bring new plants, especially to red states, and to reform American industrial policy through the Inflation Reduction Act. He did talk a lot about the day-to-day economic concerns that people had. He walked with unions, and he tried to reach out to workers. Why do you feel like that message wasn’t convincing, even when Vice President Kamala Harris took it up in her race in 2024?

A.B.: Well, I think two things. First, as Democrats, we got to get dirt on our boots, and we’ve got to show up in the areas where our policies are creating new jobs, new opportunities, more accessible health care, safer infrastructure, better schools. The signing in the Rose Garden isn’t real anymore. A signing of a bill in Frankfort [Kentucky’s capital] doesn’t directly impact people on that day. So we’ve got to be there at the announcement, at the groundbreaking. And you know, people make fun of it, the most important one is the ribbon cutting. Why? Because the jobs are there, because the future is better for families. We’ve got to make sure that people in rural America see Democrats and see the results of the policies that we’re pushing for.

The second piece, though, is we’ve got to do things faster. The Biden administration passed a lot of good legislation that has spurred a lot of economic development in my state, but the Democrats need to admit that there are times when we are over-regulated, and we’ve created so many rules that some programs that we believe are essential for the American people simply take too long. American people don’t see and feel now the Internet for All program . It’s been three years, and we don’t have a single inch of fiber in the ground. So if you’re a Democrat or a Republican and you believe that the internet is essential, then we should be able to develop a program that gets it out much, much faster.

M.P.: What are some of the regulations that you feel like could be maybe waived or used to speed up the process?

A.B.: What we’ve seen in Kentucky is even a permitting process doesn’t have to be adversarial. If you were talking to the companies and groups that you’re working with, we get most of our factories up and running three to six months faster than most states, and we abide by every environmental and workplace safety rule. What we do is work with and communicate with groups that are doing these projects. They know the expectations. If there are ways to find a solution, move something one direction or another, you impact fewer streams, you invoke fewer rules. In the Internet for All, it wasn’t that they were going to provide the money, set the rules, and then audit us to make sure that we followed them. It was that we had developed every piece of a plan we had to contract and subcontract before we could even submit the plan to potentially be approved. It was set up as a multiyear process before the construction ever started. And again, it was meant to be transformational. But if you want to actually transform in a way that helps people’s everyday lives, you’ve got to be a little bit impatient. You’ve got to understand that people are hurting now and need help now. But if it takes five years to put a program in place, you may have lost an entire generation that needed that help, that needed that assistance, or that deserves that infrastructure.

M.P.: Speaking of losing a generation, I know that in Kentucky there were some really bad river floods a few years ago, and some early decisions made by the Trump administration in a second term may have delayed some of the money going out for recovering from those floods. There are other issues like that going on now, where we’re losing funding for science, for education, for all kinds of things. And I’m wondering how people in the near term kind of survive, or think about the future.

A.B.: Decisions by the Trump administration are making life a lot harder for our American families. Start with tariffs that are raising the price on everything. That young couple in rural or urban America that can’t buy their first house, even though they’re older than their parents were when they could buy it; it’s only become harder for them with tariffs on lumber and upholstery and cabinets, virtually everything that goes into a house has been made more expensive by the tariffs.

Then move to the big, ugly bill that’s going to make it harder to get health care in your own community. Many will lose coverage. People have to drive two hours just to give birth. And what does that mean near the end of a pregnancy? Does it mean staying in a hotel where your husband or spouse is hours away and might not be there?

The Trump administration says it’s going to [change how it assesses] natural disasters, saying large snowstorms might not be included in the future. Well, those have significant costs, and people’s lives are on the line every time that there is a large snowstorm. Look at the amount that’s being shifted in the SNAP program. You know, almost $66 million of new administrative costs in Kentucky, and what that’s going to mean to food availability. All of these policy positions by the Trump administration make life harder for for Americans, but make life a lot harder for rural Americans.

Trump pushed for a tax cut for the wealthiest of Americans, who primarily live in big urban cities, but won’t push for an extension of a tax credit to help people who get health care through the ACA that primarily live in rural America.

M.P.: In some ways, do you think that the actions the Trump administration has taken make the job for Democrats easier in 2026 and 2028 because they can say, “Look what we can offer as a change from this, if you don’t like what he’s done”?

A.B.: I think it puts Democrats in a better position if the Democratic Party remains laser-focused on people’s everyday needs and then provides them the roadmap to a better life. That’s why I talk about simply saying “affordability” isn’t enough. We need to be talking about it a lot. It needs to be talking about the American dream, where it’s not just that you can pay your grocery bill at the end of the month, but you can actually get ahead. The young couple can get that new house. You can take your family on the same vacation you went on as a kid. You believe if you show up and work hard at your job that you can be a little bit better off, and that your kids can be much better off. Yes, I think that’s a compelling message for all of America, and it’s probably more compelling, sadly, because of the pain that Trump is causing and will continue to cause.

M.P.: We’ve avoided talking about agricultural policy a little bit, but it does shape a lot of how rural America is funded. And I’m wondering if there’s anything you think should be revised or reformed in agricultural policy.

A.B.: Well, you look at what the tariff policy is doing to soybean farmers that may lose the Chinese market, the largest market, potentially forever, to Brazil and Argentina, at a time when the U.S. is trying to send billions of dollars to Argentina. These are hard-working farmers that when they’re not growing soybean, they’re growing corn, and Donald Trump’s tariffs and his attacks on the sovereignty of Canada have impacted our bourbon industry, which is a huge purchaser of the corn. You look at the elimination of USAID; you look at the elimination of the Farm-to-Cafeteria programs, and our farmers have been getting hit every way possible, losing multiple markets all at the same time. Now I’m starting to see them speak out. Certainly, our cattle farmers are speaking out, and that’s important because we all care about our families more than we care about any political party, and we need to make sure that simply being a Democrat or Republican isn’t as important as being an American with an economy that can work for all of us.

M.P.: Are you going to run for president?

A.B.: Well, this weekend, I’m going to become head of the Democratic Governors Association. What you’re going to see out of me in 2026 is working to elect Democrats all over the country.

I think you’re going to see us win in rural America. You know, we’ve got a very strong candidate in Iowa, [gubernatorial candidate] Rob Sand; I’m excited to see his campaign. And if we do our work, we’ll change the map for 2028, where Democrats won’t just be battling in five states with zero margin of error. We’ll have an expanded map to where whoever our candidate is can compete in more places and get their message out to more Americans.

Germany votes to bring in voluntary military service programme for 18-year-olds

Hacker News
www.bbc.com
2025-12-06 19:19:44
Comments...
Original Article

Germany's parliament, the Bundestag, has voted to introduce voluntary military service, in a move aimed at boosting national defences after Russia's full-scale invasion of Ukraine.

It marks a significant shift in Germany's approach to its military and follows Chancellor Friedrich Merz's push to create Europe's strongest conventional army.

The change means that all 18-year-olds in Germany will be sent a questionnaire from January 2026 asking if they are interested and willing to join the armed forces. The form will be mandatory for men and voluntary for women.

Students at schools across Germany have said they will join strikes in as many as 90 cities on Friday to protest against the move.

Many young Germans either oppose the new law or are sceptical.

"We don't want to spend half a year of our lives locked up in barracks, being trained in drill and obedience and learning to kill," the organisers of the protests wrote in a statement posted on social media.

"War offers no prospects for the future and destroys our livelihoods."

In Hamburg alone, about 1,500 people were expected to join the protests, and school head teachers warned parents not to take their children out of school for the day.

German MPs voted by 323 votes to 272 to back the change, making their country the latest European country to launch some form of revised military service.

Last month, France said it was introducing 10 months of voluntary military training for 18- and 19-year-olds.

The government says military service will be voluntary for as long as possible, but from July 2027, all 18-year-old men will have to take a medical exam to assess their fitness for possible military service.

Universal medical examinations were necessary, Defence Minister Boris Pistorius said, so that in the event of an attack, Germany would not waste time determining "who is operationally capable as a homeland protector and who is not".

Germany's armed forces, the Bundeswehr, currently have around 182,000 service personnel, and Pistorius wants to increase that number by 20,000 over the next year.

The long-term aim is to raise the number by the early 2030s to 260,000, supplemented by approximately 200,000 reservists, to meet new Nato force targets and strengthen Germany's defences.

While the plan is for voluntary service, if the security situation worsens or if too few volunteers came forward, a form of compulsory military service could be considered by the Bundestag.

If war were to break out, the military would be able to draw on the questionnaires and medical exams for potential recruits.

Like other European countries, Germany ran down its armed forces during the peacetime years of the 1990s. During the Cold War it had an army of almost half a million.

Compulsory military service in Germany was ended in 2011 under former chancellor Angela Merkel.

But now, in the face of perceived threats from Russia and heavy pressure from Germany's traditional ally, the US, Friedrich Merz has pledged to rebuild the Bundeswehr into Europe's strongest conventional army.

Nato countries have come under pressure from US President Donald Trump's White House to increase their spending on defence.

Incentives for voluntary service are relatively high, with a promised salary of about €2,600 a month. In France, volunteers will be paid at least €800 (£700) a month.

The Bundestag was also set to vote on Friday on a contentious pensions reform bill, which will keep the state pension at current levels until 2031.

The bill is a key pillar of the coalition deal between Merz's conservatives and his centre-left partner, the Social Democrats, who have just a slender governing majority of just 12 votes.

However, there had been doubts about whether it would pass the parliamentary vote, as younger members of Merz's conservatives threatened to rebel. They say the plan is financially unsustainable and will leave younger generations shouldering the burden.

But Germany's opposition far-left Left party said it would abstain from voting, which means the coalition needs fewer votes to pass it and so won't have to worry about potential rebels from its own ranks.

A government crisis may narrowly have been averted.

OMSCS Open Courseware

Hacker News
sites.gatech.edu
2025-12-06 19:14:35
Comments...
Original Article

Skip to content

Georgia Tech’s Online Master of Science in Computer Science (OMSCS) program is proud to make the course content* for many of its courses publicly available through Ed Lessons. Select a course below to view the public content for that course.

Note that students enrolled in OMSCS should access their course content through Canvas, as the for-credit versions of these courses may include graded components or recent content updates not available through OMSCS Open Courseware.

* Course content typically includes things such as lecture videos and exercises; it will not include things like homeworks, projects quizzes, exams, or other graded assignments.

React2Shell flaw exploited to breach 30 orgs, 77k IP addresses vulnerable

Bleeping Computer
www.bleepingcomputer.com
2025-12-06 19:07:33
Over 77,000 Internet-exposed IP addresses are vulnerable to the critical React2Shell remote code execution flaw (CVE-2025-55182), with researchers now confirming that attackers have already compromised over 30 organizations across multiple sectors. [...]...
Original Article

React

Over 77,000 Internet-exposed IP addresses are vulnerable to the critical React2Shell remote code execution flaw (CVE-2025-55182), with researchers now confirming that attackers have already compromised over 30 organizations across multiple sectors.

React2Shell is an unauthenticated remote code execution vulnerability that can be exploited via a single HTTP request and affects all frameworks that implement React Server Components, including Next.js, which uses the same deserialization logic.

React disclosed the vulnerability on December 3, explaining that unsafe deserialization of client-controlled data inside React Server Components enables attackers to trigger remote, unauthenticated execution of arbitrary commands.

Developers are required to update React to the latest version, rebuild their applications, and then redeploy to fix the vulnerability.

On December 4, security researcher Maple3142 published a working proof-of-concept demonstrating remote command execution against unpatched servers. Soon after, scanning for the flaw accelerated as attackers and researchers began using the public exploit with automated tools.

Over 77,000 vulnerable IP addresses

Shadowserver Internet watchdog group now reports that it has detected 77,664 IP addresses vulnerable to the React2Shell flaw, with approximately 23,700 in the United States.

Geographic distribution of vulnerable IP addresses
Geographic distribution of vulnerable IP addresses
Source: ShadowServer

The researchers determined that IP addresses were vulnerable using a detection technique developed by Searchlight Cyber/Assetnote, where an HTTP request was sent to servers to exploit the flaw, and a specific response was checked to confirm whether a device was vulnerable.

GreyNoise also recorded 181 distinct IP addresses attempting to exploit the flaw over the past 24 hours, with most of the traffic appearing automated. The researchers say the scans are primarily originating from the Netherlands, China, the United States, Hong Kong, and a small number of other countries.

Unique IP addresses observed scanning for React2Shell
Unique IP addresses observed scanning for React2Shell
Source: Greynoise

Palo Alto Networks reports that more than 30 organizations have already been compromised through the React2Shell flaw, with attackers exploiting the vulnerability to run commands, conduct reconnaissance, and attempt to steal AWS configuration and credential files.

These compromises include intrusions linked to known state-associated Chinese threat actors.

Widespread exploitation of React2Shell

Since its disclosure, researchers and threat intelligence companies have observed widespread exploitation of the CVE-2025-55182 flaw .

GreyNoise reports that attackers frequently begin with PowerShell commands that perform a basic math function to confirm the device is vulnerable to the remote code execution flaw.

These tests return predictable results while leaving minimal signs of exploitation: 

powershell -c "40138*41979"
powershell -c "40320*43488"

Once remote code execution was confirmed, attackers were seen executing base64-encoded PowerShell commands that download additional scripts directly into memory. 

powershell -enc <base64>

One observed command executes a second-stage PowerShell script from the external site (23[.]235[.]188[.]3), which is used to disable AMSI to bypass endpoint security and deploy additional payloads.

According to VirusTotal , the PowerShell script observed by GreyNoise installs a Cobalt Strike beacon on the targeted device, giving threat actors a foothold on the network.

Amazon AWS threat intelligence teams also saw rapid exploitation hours after the disclosure of the React CVE-2025-55182 flaw, with infrastructure associated with China-linked APT hacking groups known as Earth Lamia and Jackpot Panda.

In this exploitation, the threat actors perform reconnaissance on vulnerable servers by using commands such as whoami and id , attempting to write files, and reading /etc/passwd.

Palo Alto Networks also observed similar exploitation, attributing some of it to UNC5174 , a Chinese state-sponsored threat actor believed to be tied to the Chinese Ministry of State Security.

"Unit 42 observed threat activity we assess with high confidence is consistent with CL-STA-1015 (aka UNC5174), a group suspected to be an initial access broker with ties to the Chinese Ministry of State Security," Justin Moore, Senior Manager at Palo Alto Networks Unit 42, told BleepingComputer via email.

"In this activity, we observed the deployment of Snowlight and Vshell malware, both highly consistent with Unit 42 knowledge of CL-STA-1015 (also known as UNC5174)."

The deployed malware in these attacks is:

  • Snowlight : A malware dropper that allows remote attackers to drop additional payloads on breached devices.
  • Vshell : A backdoor commonly used by Chinese hacking groups for remote access, post-exploitation activity, and to move laterally through a compromised network.

The rush to patch

Due to the severity of the React flaw, companies worldwide have rushed to install the patch and apply mitigations.

Yesterday, Cloudflare rolled out emergency detections and mitigations for the React flaw in its Web Application Firewall (WAF) due to its widespread exploitation and severity.

However, the update inadvertently caused an outage affecting numerous websites before the rules were corrected.

CISA has also added CVE-2025-55182 to the Known Exploited Vulnerabilities (KEV) catalog, requiring federal agencies to apply patches by December 26, 2025, under Binding Operational Directive 22-01.

Organizations using React Server Components or frameworks built on top of them are advised to apply updates immediately, rebuild and redeploy their applications, and review logs for signs of PowerShell or shell command execution.

tines

Break down IAM silos like Bitpanda, KnowBe4, and PathAI

Broken IAM isn't just an IT problem - the impact ripples across your whole business.

This practical guide covers why traditional IAM practices fail to keep up with modern demands, examples of what "good" IAM looks like, and a simple checklist for building a scalable strategy.

20 Million Americans’ Work Can Be Replaced With Today’s AI

Portside
portside.org
2025-12-06 19:01:50
20 Million Americans’ Work Can Be Replaced With Today’s AI Dave Sat, 12/06/2025 - 14:01 ...
Original Article
20 Million Americans’ Work Can Be Replaced With Today’s AI Published

Getty Images / karetoria

In the midst of a soggy job market, there’s been a lengthy debate over whether contemporary AI is actually replacing workers — or just providing bosses with an excuse to lay off certain employees and offload their responsibilities onto the ones who remain.

The answer isn’t clear, but a new study out of the Massachusetts Institute of Technology is sure to add fuel to the fire. Analyzing 151 million American workers, the researchers calculated that today’s AI systems are already mature enough to automate the tasks of more than 20 million American workers, or 11.7 percent of the entire labor force, if they were fully deployed across the country.

The researchers examined how many occupational tasks could be automated by AI, along with the dollar value of each task. The total vulnerable tasks — representing an astonishing $1.2 trillion in wage value — give a rough idea of the potential scale of disruption to the $9.4 trillion American labor market, and the huge windfall for any companies that could actually deliver all that automation.

The study’s authors deployed the Iceberg Index , a tool co-created by MIT and the Oak Ridge National Laboratory (ORNL) that simulates how AI could impact an American workforce of 151 million people in every state and not just coastal tech centers.

“Basically, we are creating a digital twin for the U.S. labor market,” ORNL director and co-author of the study Prasanna Balaprakash told CNBC .

The tool simulated each worker and treated them “as autonomous agents executing over 32,000 skills across 3,000 counties and interacting with thousands of AI tools,” the paper explained. The researcher also tracked skills that could be vulnerable to today’s AI systems and measured the “wage value of skills AI systems can perform within each occupation.”

Right now, disruptions from AI amounts to just 2.2 percent, or around $211 billion in wage value; that “represents the tip of the iceberg,” the research team wrote, winking at the possibility that AI is poised to drastically upend the lives of countless workers.

The researchers said the study and index are meant for policy makers to measure the impact of AI and any related legislation and programs on their constituents. That kind of information will be crucial going forward for elected government officials as they balance advanced technologies and the needs of their electorate, the researchers said.

Naturally, people on social media groused angrily about the study while waving around metaphorical pitchforks, saying that the American worker will miss out on what they criticized as unequal prosperity.

“The new American Dream: your company hits all-time highs the same quarter it axes your team,” one online commenter wrote .

“People will be on streets if they don’t wake up to this AI bullsh*t,” another wrote . “Start boycotting!”

Medicare for All Is Popular

Portside
portside.org
2025-12-06 18:56:12
Medicare for All Is Popular Dave Sat, 12/06/2025 - 13:56 ...
Original Article

Americans are consistently saying that the cost of living is their top concern, and health care prices, in particular, are set to soar in 2026 — with employers expected to face the largest price increase in more than a decade. This comes as Americans already spend far more than any other country on health care, despite having some of the worst rates of life expectancy and infant mortality among wealthy nations.

Medicare for All, a policy popularized by Senator Bernie Sanders, would provide all Americans with health care that is free at the point of service, paid for by tax increases.

In a new survey , Data for Progress finds that 65% of voters support a Medicare for All system — described as a “national health insurance program…that would cover all Americans and replace most private health insurance plans.” This includes majorities of Democrats (78%) and Independents (71%), and a plurality of Republicans (49%).

Next, respondents were provided with more details about what a Medicare for All system would entail: that it would “eliminate most private insurance plans and replace premiums with higher taxes, while guaranteeing health coverage for everyone and eliminating most out-of-pocket costs like copays and deductibles.”

After receiving this information, a similar majority of voters (63%) support Medicare for All — including 64% of Independents and a slight plurality of Republicans.

Finally, respondents were provided with arguments from both sides on Medicare for All, with supporters arguing that the policy would ensure everyone can receive the care they need and save families money, and opponents arguing that it would raise taxes and give the government too much control over health care.

After receiving these messages, a majority of voters (58%) say they still support Medicare for All.

Broadly, these findings demonstrate that voters support Medicare for All, even after being informed that it would raise taxes and eliminate most private insurance plans.

Survey Methodology

From November 14 to 17, 2025, Data for Progress conducted a survey of 1,207 U.S. likely voters nationally using web panel respondents. The sample was weighted to be representative of likely voters by age, gender, education, race, geography, and recalled presidential vote. The survey was conducted in English. The margin of error associated with the sample size is ±3 percentage points. Results for subgroups of the sample are subject to increased margins of error. Partisanship reflected in tabulations is based on self-identified party affiliation, not partisan registration. For more information please visit https://dataforprogress.org/our-methodology .

Fixed Points and Strike Mandates (2012)

Lobsters
www.pvk.ca
2025-12-06 18:55:16
Comments...
Original Article

Many tasks in compilation and program analysis (in symbolic computation in general, I suppose) amount to finding solutions to systems of the form \(x = f(x)\). However, when asked to define algorithms to find such fixed points, we rarely stop and ask “which fixed point are we looking for?”

In practice, we tend to be interested in fixed points of monotone functions: given a partial order \((\prec)\), we have \(a \prec b \Rightarrow f(a)\prec f(b)\). Now, in addition to being a fairly reasonable hypothesis, this condition usually lets us exploit Tarski’s fixed point theorem . If the domain of \(f\) (with \(\prec\)) forms a complete lattice , so does the set of fixpoints of \(f\) ! As a corollary, there then exists exactly one least and one greatest fixed point under \(\prec\).

This is extremely useful, because we can usually define useful meet and join operations, and enjoy a complete lattice. For example, for a domain that’s the power set of a given set, we can use \(\subset\) as the order relation, \(\cup\) as join, and \(\cap\) as meet. However, what I find interesting to note is that, when we don’t pay attention to which fixpoint we wish to find, humans seem to consistently develop algorithms that converge to the least or greatest one, depending on the problem. It’s as though we all have a common blind spot covering one of the extreme fixed points.

A simple example is dead value (useless variable) elimination. When I ask people how they’d identify such variables in a program, the naïve solutions tend to be very similar. They exploit the observation that a value is useless if it’s only used to compute values that are themselves useless. The routines start out with every value live (used), and prune away useless values, until there’s nothing left to remove.

These algorithms converge to solutions that are correct, but suboptimal (except for cycle-free code). We wish to identify as many useless values as possible, to eliminate as many computations as possible. Yet, if we start by assuming that all values are live, our algorithm will fail to identify some obviously-useless values, like x in: 

for (...)
  x = x

We could keep adding more special cases. However, the correct (simplest) solution is to try and identify live values, rather than dead ones. A value is live if it’s used to compute a live value. Moreover, return values and writes to memory are always live. Our routine now starts out by assuming that only the latter values are live, and adjoins live values as it finds them, until there’s nothing left to add.

In this case, the intuitive solution converges to the greatest fixed point, but we’re looking for the least fixed point. Setting the right initial value ensures convergence to the right fixed point.

Other common instances of this pattern are reference counting instead of marking , or performing type propagation by initially assigning the top type to all values (like SBCL).

# I recently found a use for fixed point computations outside of math and computer science.

Most university or CEGEP student unions in Québec will vote (or already have voted) on strike mandates to help organize protests against rising university tuition fees this winter and spring. There are hundreds of such unions across the province representing, in total, around four hundred thousand students. The vast majority of these unions comprise a couple hundred (or fewer) students, and many feel it would be counter-productive for only a tiny number of students to be on strike. Thus, strike mandates commonly include conditions regarding the minimal number of other students who also hold strike mandates, along with additional lower bounds on the number of unions and universities or colleges involved. As far as I know, all the mandates adopted so far are monotone: if they are satisfied by a set striking unions, they are also satisfied by all of its supersets.

Tarski’s theorem applies (again, with \((\subset, \cup, \cap)\) on the power set of the set of student unions). Which fixed point are we looking for?

It’s clear to me that we’re looking for the fixed point with the largest set of striking unions. In some situations, the least fixed point could trivially be the empty set (or all unions that did not adopt any lower bound). Moreover, the mandates are usually presented with an explanation to the effect that, if unions representing at least \(n_0\) students adopt the same mandate, then all unions that have adopted the mandate will go on strike simultaneously.

I asked fellow graduate students in computer science to sketch an algorithm to determine which unions should go on strike given their mandates; they started with the set of student unions currently on strike, and adjoined unions for which all the conditions were met. Such algorithms will converge toward the least fixed point. For example, there could be two unions, each comprising 5 000 students, with the same strike floor of 10 000 students, and these algorithms would have both unions deadlocked, waiting for the other to go on strike.

Instead, we should start by assuming that all the unions (with a strike mandate) are on strike, and iteratively remove unions whose conditions are not all met, until we hit the greatest fixed point. I’m fairly sure this will end up being a purely theoretical concern, but it’s a pretty neat case of abstract mathematics helping us interpret a real-world situation.

This pattern of intuitively converging toward a suboptimal solution seems to come up a lot when computing fixed points. It’s not necessarily a bad choice: conservative initial values tend to lead to faster convergence, and often have the property that intermediate solutions are always correct (feasible). When we need quick results, it may make sense to settle for suboptimal solutions. However, it ought to be a deliberate choice, rather than a consequence of failing to consider other possibilities.

Running Claude Code in a loop to mirror human development practices

Hacker News
anandchowdhary.com
2025-12-06 18:53:57
Comments...
Original Article

This all started because I was contractually obligated to write unit tests for a codebase with hundreds of thousands of lines of code and go from 0% to 80%+ coverage in the next few weeks - seems like something Claude should do. So I built Continuous Claude , a CLI tool to run Claude Code in a loop that maintains a persistent context across multiple iterations.

Current AI coding tools tend to halt after completing a task once they think the job is done and they don’t really have an opportunity for self-criticism or further improvement. And this one-shot pattern then makes it difficult to tackle larger projects. So in contrast to running Claude Code “as is” (which provides help in isolated bursts), what you want is to run Claude code for a long period of time without exhausting the context window.

Turns out, it’s as simple as just running Claude Code in a continuous loop - but drawing inspiration from CI/CD practices and persistent agents - you can take it a step further by running it on a schedule or through triggers and connecting it to your GitHub pull requests workflow. And by persisting relevant context and results from one iteration to the next, this process ensures that knowledge gained in earlier steps is not lost, which is currently not possible in stateless AI queries and something you have to slap on top by setting up markdown files to store progress and context engineer accordingly.

While + git + persistence

The first version of this idea was a simple while loop: 

while true; do
  claude --dangerously-skip-permissions "Increase test coverage [...] write notes for the next developer in TASKS.md, [etc.]"
  sleep 1
done

to which my friend Namanyay of Giga AI said “genius and hilarious”. I spent all of Saturday building the rest of the tooling. Now, the Bash script acts as the conductor, repeatedly invoking Claude Code with the appropriate prompts and handling the surrounding tooling. For each iteration, the script:

  1. Creates a new branch and runs Claude Code to generate a commit
  2. Pushes changes and creates a pull request using GitHub’s CLI
  3. Monitors CI checks and reviews via gh pr checks
  4. Merges on success or discards on failure
  5. Pulls the updated main branch, cleans up, and repeats

When an iteration fails, it closes the PR and discards the work. This is wasteful, but with knowledge of test failures, the next attempt can try something different. Because it piggybacks on GitHub’s existing workflows, you get code review and preview environments without additional work - if your repo requires code owner approval or specific CI checks, it will respect those constraints.

Context continuity

A shared markdown file serves as external memory where Claude records what it has done and what should be done next. Without specific prompting instructions, it would create verbose logs that harm more than help - the intent is to keep notes as a clean handoff package between runs. So the key instruction to the model is: “This is part of a continuous development loop… you don’t need to complete the entire goal in one iteration, just make meaningful progress on one thing, then leave clear notes for the next iteration… think of it as a relay race where you’re passing the baton.”

Here’s an actual production example: the previous iteration ended with “Note: tried adding tests to X but failed on edge case, need to handle null input in function Y” and the very next Claude invocation saw that and prioritized addressing it. A single small file reduces context drift, where it might forget earlier reasoning and go in circles.

What’s fascinating is how the markdown file enables self-improvement. A simple “increase coverage” from the user becomes “run coverage, find files with low coverage, do one at a time” as the system teaches itself through iteration and keeps track of its own progress.

Continuous AI

My friends at GitHub Next have been exploring this idea in their project Continuous AI and I shared Continuous Claude with them.

One compelling idea from the team was running specialized agents simultaneously - one for development, another for tests, a third for refactoring. While this could divide and conquer complex tasks more efficiently, it possibly introduces coordination challenges. I’m trying a similar approach for adding tests in different parts of a monorepository at the same time.

The agentics project combines an explicit research phase with pre-build steps to ensure the software is restored before agentic work begins. “The fault-tolerance of Agent in a Loop is really important. If things go wrong it just hits the resource limits and tries again. Or the user just throws the generated PR away if it’s not helpful. It’s so much better than having a frustrated user trying to guide an agent that’s gone down a wrong path,” said GitHub Next Principal Researcher Don Syme .

It reminded me of a concept in economics/mathematics called “radiation of probabilities” (I know, pretty far afield, but bear with me) and here, each agent run is like a random particle - not analyzed individually, but the general direction emerges from the distribution. Each run can even be thought of as idempotent: if GitHub Actions kills the process after six hours, you only lose some dirty files that the next agent will pick up anyway. All you care about is that it’s moving in the right direction in general, for example increasing test coverage, rather than what an individual agent does. This wasteful-but-effective approach becomes viable as token costs approach zero, similar to Cursor’s multiple agents.

Dependabot on steroids

Tools like Dependabot handle dependency updates, but Continuous Claude can also fix post-update breaking changes using release notes. You could run a GitHub Actions workflow every morning that checks for updates and continuously fixes issues until all tests pass.

Large refactoring tasks become manageable: breaking a monolith into modules, modernizing callbacks to async/await, or updating to new style guidelines. It could perform a series of 20 pull requests over a weekend, each doing part of the refactor with full CI validation. There’s a whole class of tasks that are too mundane for humans but still require attention to avoid breaking the build.

The model mirrors human development practices. Claude Code handles the grunt work, but humans remain in the loop through familiar mechanisms like PR reviews. Download the CLI from GitHub to get started: AnandChowdhary/continuous-claude

The Unexpected Effectiveness of One-Shot Decompilation with Claude

Simon Willison
simonwillison.net
2025-12-06 18:30:56
The Unexpected Effectiveness of One-Shot Decompilation with Claude Chris Lewis decompiles N64 games. He wrote about this previously in Using Coding Agents to Decompile Nintendo 64 Games, describing his efforts to decompile Snowboard Kids 2 (released in 1999) using a "matching" process: The matching...
Original Article

The Unexpected Effectiveness of One-Shot Decompilation with Claude ( via ) Chris Lewis decompiles N64 games. He wrote about this previously in Using Coding Agents to Decompile Nintendo 64 Games , describing his efforts to decompile Snowboard Kids 2 ( released in 1999 ) using a "matching" process:

The matching decompilation process involves analysing the MIPS assembly, inferring its behaviour, and writing C that, when compiled with the same toolchain and settings, reproduces the exact code: same registers, delay slots, and instruction order. [...]

A good match is more than just C code that compiles to the right bytes. It should look like something an N64-era developer would plausibly have written: simple, idiomatic C control flow and sensible data structures.

Chris was getting some useful results from coding agents earlier on, but this new post describes how a switching to a new processing Claude Opus 4.5 and Claude Code has massively accelerated the project - as demonstrated started by this chart on the decomp.dev page for his project:

Chart showing progress in matching code for Snowboard Kids 2. It slowly climbs from 20% to 25% from 3rd September to 17th November, then rises quickly to 45% by 2nd December

Here's the prompt he was using .

The big productivity boost was unlocked by switching to use Claude Code in non-interactive mode and having it tackle the less complicated functions (aka the lowest hanging fruit) first. Here's the relevant code from the driving Bash script : 

simplest_func=$(python3 tools/score_functions.py asm/nonmatchings/ 2>&1)
# ...
output=$(claude -p "decompile the function $simplest_func" 2>&1 | tee -a tools/vacuum.log)

score_functions.py uses some heuristics to decide which of the remaining un-matched functions look to be the least complex.

Show HN: I made a tool to make PDFs look scanned because bureaucracy

Hacker News
github.com
2025-12-06 17:42:57
Comments...
Original Article

Scanify

Transform PDF documents to look like scanned documents.

Examples

Original

Default Scan

All Effects 

scanify --aggressive --bent --dusty input.pdf

Installation

Homebrew 

brew tap Francium-Tech/tap
brew install scanify

From Source 

git clone https://github.com/Francium-Tech/scanify.git
cd scanify
swift build -c release
cp .build/release/scanify /usr/local/bin/

Usage 

# Basic usage - creates input_scanned.pdf
scanify document.pdf

# Specify output path
scanify document.pdf scanned_output.pdf

# Aggressive mode - more noise, rotation, artifacts
scanify --aggressive document.pdf

# Bent paper effect - shadow band like curved paper
scanify --bent document.pdf

# Dusty scanner - random dust specks and particles
scanify --dusty document.pdf

# Combine options
scanify --aggressive --bent --dusty document.pdf

Options

Option Description
--aggressive Apply stronger scan effects (more noise, rotation, artifacts)
--bent Add paper bend shadow effect (like curved paper under a scanner)
--dusty Add random dust specks and hair particles (like a dirty scanner glass)
--version Show version
--help Show help

Effects Applied

Base Effects (always applied)

Effect Description
Paper darkening Whites become slightly gray (scanned paper is never pure white)
Edge shadows Vignette effect from scanner lid
Top shadow Gradient shadow at top edge
Uneven lighting Slightly off-center lighting variation
Noise/grain Paper texture simulation
Slight blur Optical imperfection
Random rotation Paper feed misalignment
Saturation reduction Scanner color limitations

Optional Effects

--aggressive

Amplifies all base effects for a more dramatic scan appearance:

  • Stronger rotation (up to 1.5°)
  • More noise and grain
  • Heavier paper darkening
  • More pronounced edge shadows

--bent

Adds a horizontal shadow band across the page to simulate paper that isn't perfectly flat on the scanner glass.

--dusty

Adds random artifacts to simulate a dirty scanner:

  • 15-40 dust specks of varying sizes
  • 0-3 thin hair/fiber lines
  • Randomly scattered (not uniform)

Requirements

  • macOS 12.0 or later

License

MIT License - see LICENSE

Perl's Decline Was Cultural

Hacker News
www.beatworm.co.uk
2025-12-06 17:42:07
Comments...
Original Article

According to the Discourse, somebody killed perl

There's been a flurry of discussion on Hacker News and other tech forums about what killed Perl. I wrote a lot of Perl in the mid 90s and subsequently worked on some of the most trafficked sites on the web in mod_perl in the early 2000s, so I have some thoughts. My take: it was mostly baked into the culture. Perl grew amongst a reactionary community with conservative values, which prevented it from evolving into a mature general purpose language ecosystem. Everything else filled the gap.

I remember Perl

Something to keep in mind, is that although this is my personal take, and therefore entirely an opinion piece, I was there at the time. I stopped doing Perl properly when I left Amazon, I think this would have been around 2005. It's based on the first hand impressions of somebody who was very deeply involved in Perl in its heyday, and moved on. I have a lot of experience, from both inside and outside the tent.

Perl's roots are sysadmin

What culture? Perl always had a significant amount of what you might call "BOFH" culture, which came from its old UNIX sysadmin roots. All of those passive aggressive idioms and in jokes like "RTFM" , "lusers" , "wizards" , "asking for help the wrong way" etc. None of this is literally serious, but it does encode and inform social norms that are essentially tribal and introverted. There implicitly is a privileged population, with a cost of entry to join. Dues must be paid. Cultural conservatism as a first principle.

This stems from the old locked-down data centre command culture. When computer resource was expensive, centralised, fragile, and manually operated, it was rigidly maintained by gatekeepers, defending against inappropriate use. I started my career as an apprentice programmer at the very end of this era, (late 80s) pre-web, and before microcomputers had made much inroads, and this really was the prevailing view from inside the fort. (This is a drawback about fort-building. Once you live in a fort, it's slightly too easy to develop a siege mentality). Computers are special, users are inconvenient, disruption is the main enemy.

An unfortunate feedback loop in this kind of "perilous" environment is that it easily turns prideful. It's difficult to thrive here, if you survive and do well you are skilled; you've performed feats; you should mark your rites of passage. This can become a dangerous culture trap. If you're not careful about it, you may start to think of the hazards and difficulties, the "foot guns", as necessary features - they teach you those essential survival skills that mark you out. More unkindly, they keep the stupid folk out, and help preserve the high status of those who survived long enough to be assimilated. Uh-oh, now you've invented class politics.

The problem with this thinking is that it's self-reinforcing. Working hard to master system complexities was genuinely rewarding - you really were doing difficult things and doing them well. This is actually the same mechanism behind what eventually became known as 'meritocracy' 1 , but the core point is simpler - if difficulty itself becomes a badge of honour, you've created a trap: anything that makes the system more approachable starts to feel like it's cheapening what you achieved. You become invested in preserving the barriers you overcame.

(This is the same mentality that built leetcode interview pipelines BTW, but let's leave that sidebar alone for now)

So the UNIX operator culture tended to operate as a tribal meritocracy (as opposed to the UNIX implementer culture, which fell out of a different set of cultural norms, quite an interesting side bar itself 2 ), a cultural priesthood, somewhat self-regarding, rewarding of cleverness and knowledge hoarding, prone to feats of bravado, full of lore, with a defensive mentality of keeping the flame aloft, keeping the plebs happy and fed, and warding off the barbarians. As we entered the 90s it was already gently in decline, because centralised computing was giving way to the rise of the microcomputer, but the sudden explosive growth of the WWW pulled internet / Unix culture suddenly back into the mainstream with an enormous and public opportunity vacuum. Everyone suddenly has an urgent need to write programs that push text off UNIX file-systems (and databases) and into web pages, and Perl is uniquely positioned to have a strong first-mover advantage in this suddenly vital, novel ecosystem. But it's culture and values are very much pulled across from this previous era.

(Springing out of this, Perl had an, at best grudging, tolerance for 'difficult genius' types, alongside this baseline culture. Unfortunately, this kind of toxic personality tends to thrive in the type of culture I've described, and they do set to help the tone. I'm not here to call out people specifically, because I'm trying to make a point rather than feed a culture war, or dig up gossip, but there were several significant examples, you can probably find lore if you like. I think the kindest way I can describe the compounding effect of this is that there was a strong cultural norm along the lines of "It's OK to be rude, as long as it's for a good cause".)

A fort within a fort

I remember this tension as always being tangibly there. Perl IRC and mailing lists were quite cliquey and full of venerated experts and in-jokes, rough on naivety, keen on robust, verbose debate, and a little suspicious of newcomers. And very cult-like. The " TIMTOWTDI " rule, although ostensibly liberal, literally means 'there is more than one way to do it in Perl ' - and you can perhaps infer from that that there's little to no reason to do it using anything else. Elevating extreme flexibility like this is paradoxically also an engine of conservatism. If Perl can already do anything, flexibly, in multiple ways, then the language itself doesn't need to change - 'we already have one of those here, we don't need new things'. This attitude determined how Perl intended to handle evolution: the core language would remain stable (a fort inside a fort, only accessible to high level wizards), while innovation was pushed outward to CPAN. You could add features outside of core by writing and consuming third party libraries, you could bend language behaviour with pragmas without modifying Perl itself. The very best CPAN modules could theoretically be promoted into core, allowing the language to evolve conservatively from proven, widely-used features.

On paper, this sounds reasonable. In practice, I think it encoded a fundamental conflict of interest into the community early on, and set the stage for many of the later growth problems. I'm not going to pretend that Perl invented dependency hell, but I think it turned out to be another one of those profound misfeatures that their cultural philosophy lead them to mistake for virtue, and embrace.

An interesting thing I think has been missed discussing the context of the original blog piece, about whether Perl 6 significantly impacted Perl growth, is the fact that Perl 6 itself manifested out of ongoing arguments. Perl 6 is a schism. Here's a oft-cited note from Larry Wall himself about the incident that sparked Perl 6, at YAPC OSCON 2000

We spent the first hour gabbing about all sorts of political and organizational issues of a fairly boring and mundane nature. Partway through, Jon Orwant comes in, and stands there for a few minutes listening, and then he very calmly walks over to the coffee service table in the corner, and there were about 20 of us in the room, and he picks up a coffee mug and throws it against the other wall and he keeps throwing coffee mugs against the other wall, and he says "we are f-ed unless we can come up with something that will excite the community, because everyone's getting bored and going off and doing other things".

(Pause a second and ask yourself about the sort of social culture that both allows this kind of behaviour at public events, and then chooses to embrace it as a key piece of cultural lore)

The impact of Perl 6

Perl 6 was really a schism . Perl was already under a great amount of strain trying to accommodate the modernising influx of post dot-com mainstream web application building, alongside the entrenched conservatism of the core maintainers, and the maintenance burden of a few years exponential growth of third-party libraries, starting to build a fractal mess of slightly differentiating, incompatible approaches of those multiple ways to do things that were effectively now table-stakes language features, as the deployment landscape started to tiptoe towards a more modern, ubiquitous WWW 3 .

So, while I agree that it's wrong to generalise that 'Perl 6 killed Perl', I would say that Perl 6 was a symptom of the irreconcilable internal forces that killed Perl. Although, I also intend to go on to point out that Perl isn't dead, nothing has actually killed Perl. Killed Perl is a very stupid way to frame the discussion, but here we are.

So... Perl 6 is created as a valve to offset that pressure, and it kind of works. Up to a point. Unfortunately I think the side effect really is that the two branches of the culture, in the process of forking, double down on their encoded norms. Perl 5.x beds down as the practical, already solved way to do all the same things, with little need to change. Any requirements for more modern application patterns that are emerging in the broader web development environment, like idk, Unicode, REST clients, strict data structures, asynchronous I/O, whatever? That can either wait for Perl6 or you can pull things together using the CPAN if you want to move right now. Perl 6 leans the other way - they don't need to ship immediately, we have Perl 5 already here for doing things, Perl 6 is going to innovate on everything , and spend it's time getting there, designing up-front. 4 They spend at least two years writing high level requirement specs. They even spin out a side-project trying to build a universal virtual machine to run all dynamic programming languages that never delivers 5

This is the landscape where Perl's central dominance of 'back end' web programming continues to slip. Unfortunately, alongside the now principled bias toward cultural conservatism, Perl 5 has an explicit excuse for it. The future is over there, and exciting, and meanwhile we're working usefully, and getting paid, and getting stuff done. Kind of OK from inside the fort. Some day we'll move to the newer fort, but right now this is fine . Not very attractive to newcomers though, really. And this is also sort of OK, because Perl doesn't really want those sort of newcomers, does it? The kind that turns up on IRC or forums and asks basic questions about Perl 6 and sadly often gets treated with open contempt.

Meanwhile, over there

Ruby has sprouted "Ruby on Rails", and it's taken the dynamic web building world by storm. Rails is a second generation web framework, that's proudly an 'opinionated web framework'. Given that the web application architecture is starting to stabilise into a kind of three-tier system , with a client as a web browser, a middle tier as a monolithic application server, and a persistence layer as a relational database , and a split server architecture serving static and dynamic content from different routes, here is just one way to do that, with hugely developer friendly tooling turning this into a cookie-cutter solution for the 80% core, and a plugin and client-side decoration approach that allows for the necessary per-site customisation.

Ruby is interesting as well. Ruby is kind of a Perl6 really. More accurately it's a parallel universe Perl5 Ruby comes from Japan, and has developed as an attempt to build something similar to Perl, but it's developed much later, by programming language enthusiasts, and for the first ten years or so, it's mostly only used in Japan. To my line of thinking this is probably important. Ruby does not spring from decades of sysadmin or sysop culture. Ruby is a language for programmers, and is at this point an sensible candidate for building something like Rails with - a relatively blank canvas for dynamic programming, with many of the same qualities as Perl, with less legacy cruft, and more modern niceties, like an integrated object system, exceptions, straightforward data structures. Ruby also has adopted 'friendliness' as a core value, and the culture over there adopts a principled approach to aggressively welcoming newcomers, promoting easiness, and programmer happiness and convenience as strong first class principles.

Rails is a huge hit. At this point, which is around about the time I stopped significantly using Perl (2004-2005) (because I quit my job, not out of any core animosity toward it, in fact, in my day, I was really quite a Perl fan ), Rails is the most appealing place to start as a new web programmer. Adoption rate is high, community is great, velocity of development is well-paced, and there's a lovely , well-lit, onboarding pipeline for how to start. You don't even really need to know ruby. It has a one-shot install tool, and generates working websites from templates, almost out of the box. It's an obvious starting point.

Perl being Perl, develops several analogue frameworks to Rails, all of them interdependently compatible and incompatible with each other and each other's dependencies, all of them designed to be as customisable and as user configurable as they possibly can be 6

PHP

There are also the other obvious contenders. PHP has been there all along, and it's almost coming up from entirely the opposite cultural background of Perl. PHP is a users language . It's built to be deployed by copying script files to your home directory, with minimal server side impact or privileges. It's barely designed at all, but it encounters explosive growth all the way through the first (and through into the second) web era, almost entirely because it makes the barrier to onboarding so low as to be non-existent. PHP gets a couple of extra free shots in the arm

  1. Because it's architecture is so amenable to shared-server hosting, it is adopted as the primary implementation language of the blogging boom. An entire generation of web developers is born of installing and customising WordPress and text-pattern et. al by installing it directly into your home directory on a rented CPanel host account. It's the go-to answer for 'I'm not a programmer really but how do I get a personal web site' 7 This zero gate-keeping approach keeps the PHP stack firmly on the table of 'basic' web programmers all through the history of the web up to the current day.
  2. Because of these initially lightweight deployment targets, PHP scales like little else, mostly because it's execution model leans strongly towards idempotent execution, with each web request tearing up and tearing down the whole environment. In a sense, this is slower than keeping hot state around, but it does lend itself extremely well to shared-nothing horizontal scaling, which as the web user base increases gigantically throughout the 2000s era, is the simplest route to scaling out. Facebook famously, is built in PHP at this point in time.

Python

There is of course one other big horse in the race in this era, and it's a particularly interesting one in many ways, certainly when contrasted with Perl. This is of course, Python. Python is a close contemporary of Perl's but once again, it's roots are somewhere very different. Python doesn't come from UNIX culture either. Python comes from academia, and programming language culture. It's kind of a forgotten footnote, but Python was originally built for the Amoeba operating system , and it's intention was to be a straightforward programming language for scripting this 8 . The idea was to build a language that could be the 'second programming language' for programmers. Given that this is the 1980s, early 1990s, the programmers would be expected to be mostly using C / C++ ,perhaps Pascal. Python was intended to allow faster development for lighter weight programs or scripting tasks. I suppose the idea was to take something that you might want to build in a shell script, but provide enough high level structured support that you could cleanly build the kind of things that quickly become a problem in shell scripts. So, it emphasises data structures, and scoped variables, and modules, and prioritises making it possible to extend the language with modules. Typical things that experienced programmers would want to use. The language was also designed to be portable between the different platforms programmers would use, running on the desktops of the day, but also on the server. As a consequence, it had a broad standard library of common portable abstractions around standard system features - file-systems, concurrency, time, FFI. For quite a long time, one of python's standard mottoes was 'batteries included'.

Python never set the world on fire at any particular moment, but it remained committed to a clear evolutionary incremental development, and clean engineering principles. Again, I think the key element here is cultural tone. Python is kind of boring, not trying to be anyone's best language, or even a universal language. Python was always a little fussy, maybe snobby, slightly abstracted away from the real world. It's almost as old as Perl and it just kept incrementally evolving, picking up users, picking up features, slowly broadening the standard library. The first time I saw Python pick up an undeniable mainstream advantage would also have been around the early 2000s, when Google publicly adopted it as one of their house standard languages. Never radical, just calmly evolving in it's environs.

Nature abhors a vacuum

When I sketch out this landscape, I remain firmly convinced that most of Perl's impedance to continued growth were cultural. Perl's huge moment of relevance in the 90s was because it cross-pollinated two diverging user cultures. Traditional UNIX / database / data-centre maintenance and admin users, and enthusiastic early web builders and scalers. It had a cultural shock phase from extremely rapid growth, the centre couldn't hold, and things slowly fell apart.

Circling back though, it's time to address the real elephant in the room. Perl manifestly did not die. It's here right now. It's installed I think by default, on almost every single computer I own and operate, without me doing a single thing to make that happen. It's still used every day by millions of people on millions of systems (even if that isn't deliberate). It's still used by many people entirely deliberately for building software, whether that's because they know it and like it and it works, or because they're interfacing with or working on legacy Perl systems (of which there are still many), or maybe they're using it still in it's original intentional role - A capable POSIX-native scripting language, with much better performance and a broader feature-set than any shell or awk. I still occasionally break it out myself, for small scripts I would like to use more than once, or as parts of CLI pipelines.

What I don't do any more is reach for Perl first to make anything new. In my case, it's just because I typically am spoilt for options that are a better fit for most tasks, depending on whatever it is I'm trying to achieve. By the time I came to Perl, (1998-ish), I was already on my third career phase, I had a strong UNIX background, and had already built real things in lisp, java, pascal, visual basic and C++. My attitude to languages was already informed by picking a tool to fit the task at hand. Boy did I love Perl for a few years. The product/market-fit for those early web days was just beautiful. The culture did have too much of the negative tropes I've been pointing at, but that wasn't really a problem personally for me, I'd grown up amongst the BOFHs inside the data centres already, it wasn't too hard for me to assimilate, nor pick up the core principles. I did occasionally bounce off a couple of abrasive characters in the community, but mostly this just kept me loosely coupled, I enjoyed how the language solved the problems I needed solving quickly, I enjoyed the flexibility, and I also enjoyed the way that it made me feel smart, and en-route to my wizard's robes and hat, when i used it to solve harder problems in creative ways, or designed ways around bugs and gremlins. For a good 3-4 years I would have immediately picked it as my favourite language.

So as I say, I didn't fall out of it with any sense of pique, I just naturally moved to different domains, and picked up tools that best fit. After Amazon, I spent t a lot of time concentrating on OS X and audio programming, and that involved a lot of objective C, C++. The scripting tools in that domain were often in ruby, sometimes python. For personal hacking, I picked up lisp again 9 (which I'd always enjoyed in school). I dipped in and out of Perl here and there for occasional contract work, but I tended to gravitate more towards larger database stuff, where I typically found C, java and python. The next time I was building web things, it was all Rails and ruby, and then moving towards the web services / REST / cloud era, the natural fits were go, and of course node and JavaScript or Typescript. I've always been a polyglot, and I've always been pretty comfortable moving between programming languages. The truth of the matter is, that the majority of programming work is broadly similar, and the specific implementation details of the language you use don't matter all that much, if it's a good fit for the circumstances.

I can't imagine Perl disappearing entirely in my lifetime. I can remember entire programming environments and languages that are much, much deader than I can ever see Perl becoming.

  • Pascal used to be huge for teaching and also for desktop development in the 8/16 bit era
  • Objective C - only really useful inside the Apple ecosystem, and they're hell bent on phasing it out.
  • Before I got into the Internet, I used to build application software for 16 bit Windows (3.11) which was a vast market, in a mixture of database 4GLs (like PowerBuilder, Gupta/Centura SQLWindows) and Win16 C APIs. This entire universe basically no longer exists, and is fully obsolete. There must be many similar cases.
  • I mean who the hell realistically uses common lisp any more outside of legacy or enthusiast markets? Less people than Perl I'm sure.

Perl also got to be if not first, then certainly early to dominate a new market paradigm. Plenty of things never manage that. It's hard to see Perl as anything other than an enormous success on these terms. Perl innovated and influenced languages that came after in some truly significant ways.

  • Tightly embedding regular expressions and extending regular expressions (the most commonly used regular expression dialect in other tools is Perl)
  • CPAN, for package/library distribution via the internet, with dependency resolution - and including important concepts like supply chain verification with strong package signatures
  • A huge emphasis on testing, automated test harnesses, and CI. Perl test format (TAP) is also widely found in other CI/harness systems
  • Blending the gap between shell / scripting / and system programming in a single tool. I suppose this is debatable, but the way Perl basically integrated all the fundamental POSIX/libc as native built-ins with broadly the same semantics, but with managed memory and shell conventions was really revolutionary. Before this, most languages I had ever seen broadly tended to sit in one box, afterwards, most languages tended to span across several.
  • Amazing integrated documentation, online, in-tool and also man pages. POD is maybe the most successful ever implementation of literate programming ideas (although most of the real docs don't intertwingle the documentation very much iirc)

Just these points, and I'm sure there are many others that could be made, are enough of a legacy to be proud of.

Counterfactuals are stupid (but also fun). If I squint, I can imagine that a Perl with a less reactionary culture, and a healthier acceptance of other ideas and environmental change might have been able to evolve alongside the other tools in the web paradigm shift, and still occupy a more central position in today's development landscape. That's not the Perl we have though, and that didn't happen. And I'm very confident that without the Perl we did have, the whole of modern software practice would be differently shaped. I do think Perl now lives in a legacy role, with a declining influence, but that's really nothing to feel shame or regret for. Nobody is going to forcibly take Perl away as long as POSIX exists, and so far as I can see, that means forever. In 2025 too, I can see the invisible hand creeping up on some of these other systems I've mentioned. Rust is slowly absorbing C and C++. Ruby (and of course Rails) is clearly in decline, in a way that probably consigns it to become a similar legacy state. From a certain angle, it looks a lot like Typescript is slowly supplanting Python. I won't be entirely surprised if that happens, although at my age I kind of doubt I'll live to see the day.

Footnotes

1 : Meritocracy is a fun word. It was originally coined as a pejorative term to describe a dystopian mechanism by which modern i.e. Western / British society entrenches and justifies an unfair and unequal distribution of privilege

2 : The UNIX implementer culture, is scientific/academic and fell out of Bell Labs. I guess you could extend this school of thought as a cultural sweep towards building abstracted cloud operations, toward plan 9/ Inferno / go

3 : Web 2.0 was first defined in 1999 by Darcy DiNucci in a print article , the term didn't become mainstream until it was picked up and promoted by Tim O'Reilly (then owner/operator of perl.com, trivia fans), an astute inside observer of the forces driving web development

4 : Another unfortunate bit of luck here. Right at the point of time that 'agile' starts getting some traction as a more natural way to embrace software development - i.e. iterating in small increments against a changing environment and requirements, Perl 6 decides to do perhaps the most waterfall open source development process ever attempted. . It is fifteen years before Perl 6 ships something resembling a usable programming language.

5 : The Parrot VM , a lovely quixotic idea, which sadly fizzled out, after even Perl 6 stopped trying to target it. Interestingly enough, both python and ruby both made relatively high profile ports to the JVM that were useful enough to be used for production deploys in certain niches.

6 : A side effect of this degree of abstraction, is that as well as being very hard to get started, it's easy to fall foul of performance overhead.

7 : This ubituitious ecosystem of small footprint wordpress custom installs gives birth to the web agency model of commercial website building / small ecommerce sites, which thrives and is suprisingly healthy today. Recent, and slighly optimistic surveys have pitched WordPress as powering over 40% of all websites today. Now this is certainly inflated, but even if the realistic number is half of that, that's still pretty damn healthy.

8 : It's often repeated that Python was designed as a teaching language, but as far as I know, that's not actually the case. The designer of Python, Guido Van Rossum was previously working on a project that was a intended as training language, called ABC, and many of ABC's syntax and structural features influenced or made their way into Python.

9 : Common lisp is a better answer to an infinitely flexible 'everything' chainsaw language than perl, IMHO

posted by cms on

tagged as

GitHub Actions Has a Package Manager, and It Might Be the Worst

Lobsters
nesbitt.io
2025-12-06 17:40:01
Comments...
Original Article

After putting together ecosyste-ms/package-manager-resolvers , I started wondering what dependency resolution algorithm GitHub Actions uses. When you write uses: actions/checkout@v4 in a workflow file, you’re declaring a dependency. GitHub resolves it, downloads it, and executes it. That’s package management. So I went spelunking into the runner codebase to see how it works. What I found was concerning.

Package managers are a critical part of software supply chain security. The industry has spent years hardening them after incidents like left-pad, event-stream, and countless others. Lockfiles, integrity hashes, and dependency visibility aren’t optional extras. They’re the baseline. GitHub Actions ignores all of it.

Compared to mature package ecosystems:

Feature npm Cargo NuGet Bundler Go Actions
Lockfile ✓ ✓ ✓ ✓ ✓ ✗
Transitive pinning ✓ ✓ ✓ ✓ ✓ ✗
Integrity hashes ✓ ✓ ✓ ✓ ✓ ✗
Dependency tree visibility ✓ ✓ ✓ ✓ ✓ ✗
Resolution specification ✓ ✓ ✓ ✓ ✓ ✗

The core problem is the lack of a lockfile. Every other package manager figured this out decades ago: you declare loose constraints in a manifest, the resolver picks specific versions, and the lockfile records exactly what was chosen. GitHub Actions has no equivalent. Every run re-resolves from your workflow file, and the results can change without any modification to your code.

Research from USENIX Security 2022 analyzed over 200,000 repositories and found that 99.7% execute externally developed Actions, 97% use Actions from unverified creators, and 18% run Actions with missing security updates. The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to secrets. GitHub Actions fails to provide adequate tooling for any of them. A follow-up study using static taint analysis found code injection vulnerabilities in over 4,300 workflows across 2.7 million analyzed. Nearly every GitHub Actions user is running third-party code with no verification, no lockfile, and no visibility into what that code depends on.

Mutable versions. When you pin to actions/checkout@v4 , that tag can move. The maintainer can push a new commit and retag. Your workflow changes silently. A lockfile would record the SHA that @v4 resolved to, giving you reproducibility while keeping version tags readable. Instead, you have to choose: readable tags with no stability, or unreadable SHAs with no automated update path.

GitHub has added mitigations. Immutable releases lock a release’s git tag after publication. Organizations can enforce SHA pinning as a policy. You can limit workflows to actions from verified creators. These help, but they only address the top-level dependency. They do nothing for transitive dependencies, which is the primary attack vector.

Invisible transitive dependencies. SHA pinning doesn’t solve this. Composite actions resolve their own dependencies, but you can’t see or control what they pull in. When you pin an action to a SHA, you only lock the outer file. If it internally pulls some-helper@v1 with a mutable tag, your workflow is still vulnerable. You have zero visibility into this. A lockfile would record the entire resolved tree, making transitive dependencies visible and pinnable. Research on JavaScript Actions found that 54% contain at least one security weakness, with most vulnerabilities coming from indirect dependencies. The tj-actions/changed-files incident showed how this plays out in practice: a compromised action updated its transitive dependencies to exfiltrate secrets. With a lockfile, the unexpected transitive change would have been visible in a diff.

No integrity verification. npm records integrity hashes in the lockfile. Cargo records checksums in Cargo.lock . When you install, the package manager verifies the download matches what was recorded. Actions has nothing. You trust GitHub to give you the right code for a SHA. A lockfile with integrity hashes would let you verify that what you’re running matches what you resolved.

Re-runs aren’t reproducible. GitHub staff have confirmed this explicitly : “if the workflow uses some actions at a version, if that version was force pushed/updated, we will be fetching the latest version there.” A failed job re-run can silently get different code than the original run. Cache interaction makes it worse: caches only save on successful jobs, so a re-run after a force-push gets different code and has to rebuild the cache. Two sources of non-determinism compounding. A lockfile would make re-runs deterministic: same lockfile, same code, every time.

No dependency tree visibility. npm has npm ls . Cargo has cargo tree . You can inspect your full dependency graph, find duplicates, trace how a transitive dependency got pulled in. Actions gives you nothing. You can’t see what your workflow actually depends on without manually reading every composite action’s source. A lockfile would be a complete manifest of your dependency tree.

Undocumented resolution semantics. Every package manager documents how dependency resolution works. npm has a spec. Cargo has a spec. Actions resolution is undocumented. The runner source is public , and the entire “resolution algorithm” is in ActionManager.cs . Here’s a simplified version of what it does: 

// Simplified from actions/runner ActionManager.cs
async Task PrepareActionsAsync(steps) {
    // Start fresh every time - no caching
    DeleteDirectory("_work/_actions");

    await PrepareActionsRecursiveAsync(steps, depth: 0);
}

async Task PrepareActionsRecursiveAsync(actions, depth) {
    if (depth > 10)
        throw new Exception("Composite action depth exceeded max depth 10");

    foreach (var action in actions) {
        // Resolution happens on GitHub's server - opaque to us
        var downloadInfo = await GetDownloadInfoFromGitHub(action.Reference);

        // Download and extract - no integrity verification
        var tarball = await Download(downloadInfo.TarballUrl);
        Extract(tarball, $"_actions/{action.Owner}/{action.Repo}/{downloadInfo.Sha}");

        // If composite, recurse into its dependencies
        var actionYml = Parse($"_actions/{action.Owner}/{action.Repo}/{downloadInfo.Sha}/action.yml");
        if (actionYml.Type == "composite") {
            // These nested actions may use mutable tags - we have no control
            await PrepareActionsRecursiveAsync(actionYml.Steps, depth + 1);
        }
    }
}

That’s it. No version constraints, no deduplication (the same action referenced twice gets downloaded twice), no integrity checks. The tarball URL comes from GitHub’s API, and you trust them to return the right content for the SHA. A lockfile wouldn’t fix the missing spec, but it would at least give you a concrete record of what resolution produced.

Even setting lockfiles aside, Actions has other issues that proper package managers solved long ago.

No registry. Actions live in git repositories. There’s no central index, no security scanning, no malware detection, no typosquatting prevention. A real registry can flag malicious packages, store immutable copies independent of the source, and provide a single point for security response. The Marketplace exists but it’s a thin layer over repository search. Without a registry, there’s nowhere for immutable metadata to live. If an action’s source repository disappears or gets compromised, there’s no fallback.

Shared mutable environment. Actions aren’t sandboxed from each other. Two actions calling setup-node with different versions mutate the same $PATH . The outcome depends on execution order, not any deterministic resolution.

No offline support. Actions are pulled from GitHub on every run. There’s no offline installation mode, no vendoring mechanism, no way to run without network access. Other package managers let you vendor dependencies or set up private mirrors. With Actions, if GitHub is down, your CI is down.

The namespace is GitHub usernames. Anyone who creates a GitHub account owns that namespace for actions. Account takeovers and typosquatting are possible. When a popular action maintainer’s account gets compromised, attackers can push malicious code and retag. A lockfile with integrity hashes wouldn’t prevent account takeovers, but it would detect when the code changes unexpectedly. The hash mismatch would fail the build instead of silently running attacker-controlled code. Another option would be something like Go’s checksum database, a transparent log of known-good hashes that catches when the same version suddenly has different contents.

How Did We Get Here?

The Actions runner is forked from Azure DevOps, designed for enterprises with controlled internal task libraries where you trust your pipeline tasks. GitHub bolted a public marketplace onto that foundation without rethinking the trust model. The addition of composite actions and reusable workflows created a dependency system, but the implementation ignored lessons from package management: lockfiles, integrity verification, transitive pinning, dependency visibility.

This matters beyond CI/CD. Trusted publishing is being rolled out across package registries: PyPI, npm, RubyGems, and others now let you publish packages directly from GitHub Actions using OIDC tokens instead of long-lived secrets. OIDC removes one class of attacks (stolen credentials) but amplifies another: the supply chain security of these registries now depends entirely on GitHub Actions, a system that lacks the lockfile and integrity controls these registries themselves require. A compromise in your workflow’s action dependencies can lead to malicious packages on registries with better security practices than the system they’re trusting to publish.

Other CI systems have done better. GitLab CI added an integrity keyword in version 17.9 that lets you specify a SHA256 hash for remote includes. If the hash doesn’t match, the pipeline fails. Their documentation explicitly warns that including remote configs “is similar to pulling a third-party dependency” and recommends pinning to full commit SHAs. GitLab recognized the problem and shipped integrity verification. GitHub closed the feature request.

GitHub’s design choices don’t just affect GitHub users. Forgejo Actions maintains compatibility with GitHub Actions, which means projects migrating to Codeberg for ethical reasons inherit the same broken CI architecture. The Forgejo maintainers openly acknowledge the problems , with contributors calling GitHub Actions’ ecosystem “terribly designed and executed.” But they’re stuck maintaining compatibility with it. Codeberg mirrors common actions to reduce GitHub dependency, but the fundamental issues are baked into the model itself. GitHub’s design flaws are spreading to the alternatives.

GitHub issue #2195 requested lockfile support. It was closed as “not planned” in 2022. Palo Alto’s “Unpinnable Actions” research documented how even SHA-pinned actions can have unpinnable transitive dependencies.

Dependabot can update action versions, which helps. Some teams vendor actions into their own repos. zizmor is excellent at scanning workflows and finding security issues. But these are workarounds for a system that lacks the basics.

The fix is a lockfile. Record resolved SHAs for every action reference, including transitives. Add integrity hashes. Make the dependency tree inspectable. GitHub closed the request three years ago and hasn’t revisited it.

Further reading:

FreeBSD 15: Why You’ll Want It

Lobsters
freebsdfoundation.org
2025-12-06 17:34:38
Comments...
Original Article
December 4, 2025

FreeBSD 15.0 landed earlier this week, and we read through the release notes with a fine-toothed comb to highlight some of the key improvements. Here are the standouts.

The BIG One: “pkgbase”

After roughly a decade of work, the base system can now be managed using pkg. Now that it’s here, what does pkgbase mean for everyday users? Drawing from a talk by Baptiste Daroussin, here are the key benefits:

  • Allows users to do fine grained installations (no toolchain, no debug symbols, etc.)
  • Offers more precise merging of configuration files.
  • Developers can easily ship packages for testing.
  • Permits simpler binary upgrades, including smoother tracking of STABLE and CURRENT.

This gives administrators much more flexibility in keeping systems minimal, consistent, and up to date.

Improved Desktop & Laptop Support

Several updates directly benefit laptop and desktop users:

Wi-Fi enhancements

  • The rtwn(4) driver now supports 802.11ac (VHT) for supported Realtek chipsets (RTL8812A and RTL8821A).
  • The new iwx(4) driver, FreeBSD’s native driver for newer Intel wireless chipsets, appears in this release as an alternative to iwlwifi(4) .

Audio & device handling

  • Asynchronous device detach is now supported. This improves hot-plug behavior (e.g., USB headsets) and eases use of PulseAudio in cases that require operating system sleep/wake.

AMD GPU stability

  • Fixes landed for gradual slowdowns and freezes affecting certain AMD GPUs when using the amdgpu DRM driver from the drm-kmod ports package.

Offline Help for New Users

Brand new FreeBSD users often need guidance right after install, especially before their system is online.

To help:

  • The existing freebsd-base man page remains an invaluable starting point.
  • A new networking man page provides quick, offline guidance for troubleshooting early network setup.

This is extremely handy if a freshly installed system isn’t connecting to the network.

Major Improvements on Amazon Web Services

Running FreeBSD in AWS? You’ll notice some meaningful improvements:

  • FreeBSD “base” EC2 images now boot up to 76% faster than corresponding 14.0-RELEASE images, with the largest improvements found on arm64 (“Graviton”) instances.
  • The FreeBSD project now publishes “small” EC2 images; these are the “base” images minus debug symbols, tests, 32-bit libraries, the LLDB debugger, the Amazon SSM Agent, and the AWS CLI. This reduces the amount of disk space in use when the EC2 instance finishes booting from ~5 GB to ~1 GB . (wow!)
  • The FreeBSD project now publishes “builder” EC2 images; these boot into a memory disk and extract a clean “base” image onto the root disk (mounted at /mnt) to be customized before creating an AMI. 584265890303 (Sponsored by Amazon)

Other Noteworthy Updates

A few additional items that deserve attention:

  • Possibly bigger news than it appears – bhyve(8) and vmm(4) now support arm64 and riscv platforms.
  • FreeBSD introduces a native mechanism for controlled privilege escalation via mdo(1) and mac_do(4) . This provides a built-in alternative to installing tools like sudo or doas when users need limited administrative capabilities.

But Wait! There’s More!

These highlights only scratch the surface. If you’re planning to upgrade —or just want a deeper look—reading the full FreeBSD 15.0 release notes is absolutely worthwhile.

Unaggregating Cloud Watch Metrics

Lobsters
tomlarkworthy.github.io
2025-12-06 17:08:18
Comments...
Original Article

Infisical (YC W23) Is Hiring Engineers to Build the Modern OSS Security Stack

Hacker News
www.ycombinator.com
2025-12-06 17:01:53
Comments...
Original Article

Unified platform for secrets, certs, and privileged access management

Senior Full Stack Engineer (US & Canada)

$160K - $250K • 0.05% - 0.25% • San Francisco, CA, US / Remote (US)

Role

Engineering, Full stack

Apply to

Infisical

and hundreds of other fast-growing YC startups with a single profile.

Apply to role ›

About the role

Infisical is looking to hire exceptional talent to join our teams in building the open source security infrastructure stack for the AI era.

We're building a generational company with a world-class engineering team. This isn’t a place to coast — but if you want to grow fast, take ownership, and solve tough problems, you’ll be challenged like nowhere else.

What We’re Looking For

We’re looking for an exceptional Full Stack Engineer to help us build, optimize, and expand the foundation of the platform.

We’ve kept our hiring standards exceptionally high since we expect engineers to tackle a broad range of challenges on a day-to-day basis. Examples of past engineering initiatives include developing strategies for secret rotation and dynamic secrets, a gateway to provide secure access to private resources, protocols like EST and KMIP , integrations for syncing secrets across cloud providers, and entire new product lines such as Infisical PKI and Infisical SSH.

You’ll be working closely with our CTO and the rest of the engineering team to:

  • Develop and maintain features whilst communicating directly with enterprise customers.
  • Expand our newer Infisical PKI , Infisical SSH , and Infisical KMS product lines.
  • Experiment with novel approaches for applying AI to secrets management and more broadly security infrastructure.

Requirements

  • Deep technical mastery of the JavaScript ecosystem, particularly React.js, Node.js, and TypeScript (3+).
  • Exceptional attention to detail and eager to learn.
  • A bias toward action—able to make decisions with incomplete information, iterate quickly, and take calculated risks.
  • Based in the United States.

Bonus

  • Expertise in Go.
  • Has some understanding of devops/developer tools.
  • Previous founder or startup experience.
  • Previous experience building in open source or developer tools in either a professional or personal setting.
  • Excellent written and oral communication skills to interact with customers directly.

How You’ll Grow

In this role, you’ll play a pivotal part in shaping Infisical’s future—making key technical decisions, establishing foundational processes, and tackling complex scalability challenges. As you gain experience and the team expands, you'll have the opportunity to take full ownership of specific areas of our platform, driving them end-to-end with autonomy and impact.

Overall, you’ll be one of the defining pieces of our team as we scale to thousands of customers over the next 18 months.

Team, Values & Benefits

Our team brings experience from companies like Figma, AWS, and Red Hat. We operate primarily as a remote team but maintain a strong presence in San Francisco, where we have an office. We also get together in person throughout the year for off-sites, conferences, and team gatherings.

At Infisical, we offer competitive compensation, including both salary and equity options. Additional benefits, such as a lunch stipend and a work setup budget, are available with more details to be found on our careers page .

About Us

Infisical is the open source security infrastructure platform that engineers use for secrets management, internal PKI, key management, and SSH workflow orchestration. We help developers and organizations securely manage over 1.5 billion secrets each month including application configuration, database credentials, certificates, and more.

We’ve raised $19M from Y Combinator, Google, and Elad Gil, and our customers include Hugging Face, Lucid, and LG.

Join us on a mission to make security easier for all developers — starting with secrets management.

About Infisical

Infisical is the #1 open source secret management platform – used by tens of thousands of developers.

We raised $3M from Y Combinator, Gradient Ventures (Google's VC fund), and awesome angel investors like Elad Gil, Arash Ferdowsi (founder/ex-CTO of Dropbox), Paul Copplestone (founder/CEO of Supabase), James Hawkins (founder/CEO of PostHog), Andrew Miklas (founder/ex-CTO of PagerDuty), Diana Hu (GP at Y Combinator), and more.

We are default alive, and have signed many customers ranging from fastest growing startups to post-IPO enterprises.

Infisical

Founded: 2022

Batch: W23

Team Size: 30

Status: Active

Location: San Francisco

Founders

Ireland's Inability to Defend Itself

Hacker News
www.irishpoliticsnewsletter.ie
2025-12-06 16:55:13
Comments...
Original Article

Ukrainian President Volodymyr Zelensky arrived in Ireland this week for his first official state visit. Extensive security measures were put in place across Dublin and the surrounding areas ahead of the high-profile visit. It has since emerged that four military drones breached a no-fly zone and flew toward Zelensky’s plane shortly before it landed at Dublin Airport on Monday night. They were drones, but not the sort you buzz the neighbour’s cat with. It is being reported that these were serious, military-grade pieces of hardware, and for two unhurried hours, they did lazy, insolent loops in the sky above an Irish naval vessel, their navigation lights brazenly glowing in the dark. It was less an intrusion than a slow, deliberate loitering, a military mechanical staring contest.

No one is officially putting a name to the culprits. Russian? Probably. Nuisance-happy hobbyists with suspiciously deep pockets? Your guess is as good as MI6’s. But the Irish security services have a theory. With Zelensky due to touch down, drones that fly with their lights on aren’t trying to be subtle; they’re trying to be seen . The aim wasn’t surveillance so much as theatre, a deliberate flex to rattle the cage and complicate the diplomatic proceedings. It’s the sort of provocation that’s too deniable to be an act of war, but too pointed to be anything but hostile. In the parlance of our times, it’s what you’d call a spot of ‘hybrid warfare’: cost-effective, electronically delivered bullying, leaving a sovereign nation to scratch its head and wonder why anyone would be annoyed at us.

Zelensky’s aircraft touched down slightly ahead of schedule at 11 p.m., a fortunate detail, as the drones would have been directly in President Zelensky’s flight path if it had arrived on time. Some security analysts are calling it an assassination attempt on Zelensky.

We all really know who was behind this. The Russian quislings in the Irish Parliament will talk about NATO and provocation. But Ireland is like a small child in geopolitical terms, unable to defend itself and not a member of NATO, but we are the vulnerable backdoor into Europe for Russia, and we are sitting on unprotected digital gold. Roughly three out of every four undersea data cables in the entire northern hemisphere are obliged to meander through Ireland’s vast liquid backyard, which is 10 times our land mass. Every cat TikTok from New York, every terse Brussels communiqué to Washington, every billion-dollar London stock trade, most of it takes a compulsory, deep-sea detour through this one nation’s jurisdiction. All of it unprotected, successive governments goaded by an opposition into neglecting our security on the principle of laissez-faire neutrality.

Ireland likes to think of itself as a small, heroic little Republic in a dangerous world, a plucky emerald bauble bobbing bravely against the choppy seas of transatlantic geopolitics, humming ballads about defeating imperialism. At the same time, the hundreds of Russian ghost ships glide past the fibre‑optic cables that pulse under the waves like the nervous system of some enormous, comatose digital god. In hard geopolitical security terms, Ireland is closer to a pampered child at the end of a well‑policed cul‑de‑sac. The big houses in the local neighbourhood pay for the private security, the floodlights, the patrols that keep the wolves from the manicured lawns. The neighbours also keep installing new locks, new cameras, new alarms, while Ireland props the front door open with a copy of its non-existent neutrality proclamation and grandly calls it a principle. Naively thinking that hostile fascist powers respect principles.

The miracle here is that the Irish child is rich. Not just “grand, can afford the round” rich, but a statistical hallucination of prosperity, GDP per head swollen by corporate tech and pharma behemoths taking advantage of our favourable tax laws. Multinational money is sluicing through the country like rainwater off a tin roof. On paper, this is one of the wealthiest corners of the globe; in practice, when you look for the instruments by which serious states insist on their continued existence, you find a handful of ships, some overstretched soldiers and sailors, and a nonexistent ability to defend ourselves. The richest family on the street, no locks on the door, but very strong opinions about the ethics of locksmithing.

Every cul‑de‑sac has one of these families. They vaguely understand that an underlying danger exists; they have seen the footage from Ukraine, the shards of tower blocks, the sudden hot light of artillery in the distance, but they cannot permit the thought that any of this belongs to the same world as their own. For them, war is not a material process but a sort of moral failing, something that happens in regrettable countries to people who did not read the right poetry. The Irish version is called neutrality. It pads around the house in its socks, murmuring about the triple locks and the UN, telling anyone who will listen that it does not “do” sides, while its internet traffic crosses cables that are explicitly defended by other countries’ navies and its skies protected by its former colonial masters in the UK. An Irish solution to an Irish problem.

This is what sovereignty looks like after a lifetime of outsourcing. We like to outsource in Ireland. For decades, we’ve outsourced our health and housing problems to “charities”. Very rarely has this outsourcing worked. For longer still, we’ve outsourced our security to the UK, USA and NATO. How does that make us neutral? Our laissez-faire approach to neutrality has, in fact, made us a target for bad state actors.

Irish airspace is like a ring doorbell pointed at the sky, but the ring doorbell is in the UK: Ireland gets the notification if something really nasty shows up, and we rely on calling someone else to deal with it. An unidentified aircraft wanders in from the North Atlantic, and somewhere in Lincolnshire, a jet spools up; the child peers out from behind the curtains and congratulates itself on having maintained a principled distance from the whole affair. The seas around Ireland bristle with infrastructure – gas pipes, data lines, the capillaries of a continent, and decades of Irish security policy was to essentially hope that no one noticed.

Naturally, everyone has noticed. Military planners in London, Washington and Brussels talk about Ireland in the same way an exhausted parent talks about childproofing the house. If you wanted to test the defences of the European order without starting World War Three, you would not go straight for Berlin or Paris; you would look for something softer, somewhere the guardians feel responsible for, but the legal paperwork is hazy, a place full of crucial systems and sentimental stories and very little actual steel. You would look at the island that insists it is neutral while flying, economically, digitally and politically, in tight formation with the same system it refuses to help defend.

Ireland’s self‑image, understandably, does not enjoy this description. Ireland narrates itself in the heroic register: small but principled, a nation that shook off an empire and now wanders the earth in a blue helmet, mopping up after other people’s catastrophes, radiating moral authority from behind a modest smile. In this story, neutrality is a kind of sainthood. The state remains untainted by alliance politics, unsullied by the grubby calculus of deterrence; it arrives, eventually, with medics, engineers and U.N. peacekeepers, like a conscience dropping by after the damage has been done.

Look closely at this sainthood, and you see a cheaper arrangement. Neutrality is not written in stars or stone; it is an improvisation. The country does not abstain from war so much as it abstains from the price of preparing for one. We have a talent for moralising the lack of certain capacities: no jets becomes a principled suspicion of air power; a threadbare navy becomes a deep spiritual commitment to not being a “militarised state.” Even the constitutional palaver over sending troops abroad functions mainly as a ritual scene in which a certain kind of political class re‑enacts its reluctance to possess a force able to defend itself. We will go to the countries less fortunate than ourselves, yes, but do not worry, we are very conflicted about it.

Meanwhile, the new multipolar world obligingly refuses to play this game. War has returned to Europe, not as archival footage but as live‑streamed carnage; cables have been cut, pipelines mysteriously sabotaged, and energy systems flickered in the dark. The things that make Ireland rich – the tax arrangements, the data centres humming gently in business parks, the invisible streams of numbers crossing the continental shelf are now recognised as potential targets. The Atlantic is no longer a protective moat; it’s a tangle of cables and pipes, each one a fragile strand that could easily be broken as it’s largely undefended. The consequences would be dire if they were broken.

In this setting, the spectacle of a state that spends lavishly to attract the world’s tech giants but baulks at the cost of defending the physical infrastructure that makes us rich becomes less a charming idiosyncrasy and more a structural absurdity. It is like watching someone mortgage their house to buy a collection of antiques and then refusing, on ethical grounds, to have a front door. When neighbours point out that this might not be wise, Ireland patiently explains that front doors are escalatory. If you start with a door, next thing you know, you are buying a lock, and ring doorbell, and that is how an arms races begin.

To be fair, the child has begun to notice the brick through the window in the next street. Budgets rise; white papers flutter down from government departments, solemn and heavy as overfed pigeons. There are promises of radar and sonar, of patrol aircraft, of more people in uniform who know which way to point a rifle. Committees declare that previous levels of ambition were insufficient and that a new level, still modest but at least numerically larger, will now be pursued. The political language shifts from complacent neutrality to “shared European responsibilities” and “hybrid threats.” The cul‑de‑sac is reluctantly considering the possibility of at least putting a latch on the door.

But even here, a particular kind of Irish incantion persists: the ability to turn the bare minimum into a grand gesture. Doubling almost nothing still leaves you with very little. The government promises a transformation, an unprecedented leap, that will leave the country spending still less than most of its peers and possessing, at the end of the process, something that could generously be described as the outline of a modest, modern defence. It is like a teenager agreeing to start doing chores around the house and announcing this as a revolutionary break with centuries of oppression. The security, the insurance, the responsibility for the street itself – these things can remain someone else’s problem.

What makes all this more than just a domestic eccentricity is the way Ireland’s self‑flattering story depends on the very order it refuses to help maintain. The island’s prosperity is not some natural efflorescence of limestone and rain; it is the product of a global system of trade, law and power, a system held together, ultimately, by the possibility of force. American carrier groups, satellite constellations, and nuclear submarines are as much a part of the Irish economic miracle as low corporation tax. The Irish state lives inside an architecture built largely by the same military multinational forces it professes to oppose, now suitably redecorated and multilateralised so that the old trauma can be polished into myth while the financial benefits to the state roll in.

This is the real indecency at the heart of the “heroic neutral” pose. It is not that Ireland refuses to buy enough fighter jets to impress the sort of people who use fighter jets to protect themselves. It is that it demands to be treated as a fully sovereign moral subject while leaving the unromantic work of ensuring its continued existence to others. It is the demeanour of a young adult that insists it has moved out while its laundry still appears in the wash basket, mysteriously cleaned and folded, at the end of the bed. The Atlantic cables hum, guarded by navies whose flags never fly in Dublin’s parades, but the Irish Republic lectures them about multilateralism.

There is a perfectly respectable case for staying out of NATO, for maintaining some distance from American geopolitical clusterfucks, for preferring the blue helmet to the alliance badge. But that case only holds if neutrality is backed by something other than naivety. To be truly neutral in this world is to have the means to make invasion, sabotage or coercion so expensive that no one bothers, to build a hard shell around your moral core. Anything less is not neutrality; it is a politically bankrupt superstition.

So the choice before Ireland is disarmingly simple. We can grow up: accept that the stories we tell ourselves also incur obligations, that sovereignty is not a vibe but an infrastructure, and that a state rich enough to host the data of half the world has no excuse for treating its own defence as an afterthought. Or it can continue as the child in the cul‑de‑sac, tugging at its parents’ sleeves, insisting it is “neutral” while other people’s aircraft circle overhead and other people’s ships comb the seabed, a ward of a new world order it pretends to float above. The dangerous thing about refusing to grow up is not that the world will punish you for your innocence. It is possible that eventually, the adults may decide to stop paying for your innocence, and you’ll be punished by others for it anyway.

The Everyday Repercussions to Ireland if Transatlantic Cables Are Cut

In an increasingly interconnected world, the seamless flow of information across continents is vital to the functioning of modern economies and societies. For Ireland, a small but globally connected nation, transatlantic cables are the lifelines that enable communication, commerce, and connectivity with the rest of the world. These undersea cables, which carry vast amounts of data between North America and Europe, are critical to Ireland’s digital infrastructure. But what would happen if these cables were cut? The repercussions would be far-reaching, affecting everything from business operations to everyday life for Irish citizens.

Diffie-Hellman Key Exchange Visualizer

Lobsters
sidney.sh
2025-12-06 16:26:34
Comments...
Original Article

AI Energy Score v2: Refreshed Leaderboard, now with Reasoning

Lobsters
huggingface.co
2025-12-06 16:10:35
Comments...
Original Article

Back to Articles

AIEnergyScore_LightBG

Today, we’re excited to launch a refreshed AI Energy Score leaderboard , featuring a new cohort of text generation models and the introduction of reasoning as a newly benchmarked task . We’ve also improved the benchmarking code and submission process, enabling a more streamlined evaluation workflow. With the increased interest of the community towards measuring and comparing the energy use of AI models, our benchmarking efforts are more important than ever to inform sustainably-minded AI development and policymaking.

AI Energy Score

Background 📜

The AI Energy Score project combines insights from prior benchmarking efforts to deliver a unified framework for evaluating the energy efficiency of AI models. Initially launched in February 2025, the leaderboard compares the energy efficiency of AI models across 10 tasks and multiple modalities (text, image, audio) using a standardized approach built on custom datasets and the latest generation of GPUs. The launch was covered in media outlets like The Economist , NPR , Newsweek , Nature and many more, was recognized at the Paris Peace Forum in the context of the Paris AI Summit, and was recently highlighted during Sasha's New York Climate Week TED Talk .

In the months since the launch, the push for a standardized benchmark has only accelerated, with regulatory efforts like the EU AI Act Code of Practice (signed by key industry players like OpenAI, Microsoft, and Google) now explicitly calling for an inference energy benchmark, and efforts from organizations like the IEEE and the Green Software Foundation working towards a standardized way for measuring energy use and carbon emissions. Recent environmental disclosures from companies such as Google and Mistral have shed light on these impacts, but since they do not use the same methodology, it’s hard to compare these disclosures. Each of these efforts is valuable, but without a standard they’re like comparing apples-to-bananas-to-pineapples. That’s where the AI Energy Score comes in: if all providers used this approach, we could finally compare models on a level playing field.

V2 of the AI Energy Score Leaderboard 🏆

For the second version of the leaderboard, we partnered up Scott Chamberlin from Neuralwatt to streamline the benchmarking approach and to add the ability to test reasoning models. Under the hood, we are still using Code Carbon and the same datasets that we initially developed for the first version of the leaderboard (see the documentation for more details). To streamline the approach, we created a new open-source package, AI Energy Benchmarks , which we hope will become the underlying coding package that enables energy benchmarking on a variety of hardware and software configurations.

Key Findings 🔎︎

Reasoning comes at a cost 🧠

This year has seen a rise in the popularity of reasoning models, which use an internal monologue to “reason” through questions, with the intent of increasing performance. Many of the most recent LLMs now include reasoning modes, either through a simple switch that turns the feature on or off, such as in Microsoft’s Phi 4, or through multiple levels of reasoning, as seen in OpenAI’s GPT-OSS models, which offer low, medium, and high modes.

According to our analysis, reasoning models use, on average, 30 times more energy than models with no reasoning capabilities (or with reasoning turned off). Honing in on specific models with and without the reasoning functionality enabled, we can see a huge difference: between 150 and 700 times more energy used by the same model with reasoning enabled compared to without:

Model name Params Reasoning GPU energy (Wh) per 1k queries Energy Increase due to Reasoning
DeepSeek-R1-Distill-Llama-70B 70B Off 49.53 154
DeepSeek-R1-Distill-Llama-70B 70B On 7,626.53
Phi-4-reasoning-plus 15B Off 18.42 514
Phi-4-reasoning-plus 15B On 9,461.61
SmolLM3-3B 3B Off 18.35 697
SmolLM3-3B 3B On 12,791.22

This can be explained in large part by the number of output tokens generated by the models themselves (for “reasoning” through their answers) – models with reasoning enabled use between 300 and 800 times more tokens than their base equivalents. This adds up as reasoning models are used more and more in consumer-facing tools and applications, since they will tend to output longer responses (which has also been found in recent research ).

Additionally, the energy use of reasoning models is less predictable than that of standard LLMs. The impact of LLMs has traditionally shown a “ strong correlation between a model’s size and its footprint ”. However, each reasoning model may produce its reasoning traces in different ways and degrees of verbosity, making this approximation difficult. This matters because many people still assume that smaller models are always better, but the intensity of the reasoning process now has to be considered as well. This is yet another reason we need standardized and transparent benchmarks in this area.

GPT-OSS Energy comparison chart

Models with multiple levels of reasoning, like the GPT-OSS series, offer useful insight into the dynamics between model size and reasoning intensity. The 20B class shows a 4.8x difference between its high and low reasoning modes, while the 120B class has a much smaller 1.6x swing. Comparing the reasoning modes across the two classes reveals a 4.7x difference on the low setting, with much smaller deltas for the medium and high modes at roughly 1.6x (See here for another GPT-OSS energy analysis) .

Are newer models more efficient? Results are mixed 🤷

For this update, we’ve added 39 new models, including 21 in the text generation task: 11 Class A models that fit on a single consumer GPU, 3 Class B models that require a cloud GPU, and 7 Class C models that require multiple GPUs.

Comparing the energy use of models from this cohort to the February 2025 cohort can help us understand potential efficiency progress in AI over the last nine months. To compare fairly, we selected models with no reasoning (or with reasoning turned off), no mixture-of-experts architecture (since this was more rare earlier this year) and compared their energy usage to a reference model of similar size (in terms of active parameters) from the previous leaderboard. The results were mixed:

GPT-OSS Energy comparison chart

Of the 15 models that met these criteria, the majority (9) had greater or equal energy use compared to models of a similar size from February. The range was large, with some models using only 3% of the energy while others used 4x more!

This runs counter to the prevailing narrative that AI is becoming more efficient and underscores the need for users and developers to choose the right model for each task . Selecting appropriately helps avoid wasting compute resources on queries for which a simpler and more efficient model would work well. Approaches such as routers , which can choose the most appropriate model for incoming queries, will be increasingly useful in these scenarios, and energy data from AI Energy Score can be used alongside performance-based metrics to route user queries to the right model at the right time.

Salesforce Model Card Blog

Adoption in practice 🚀

One story we’re excited to share: Salesforce has integrated the AI Energy Score into its internal model benchmarking suite. Model Cards are now automatically generated with energy transparency included, and Salesforce has committed to publishing this information for all production models going forward. This shows how easy it is to integrate our Docker-based process, and serves as an example for other organizations.

We’ve also seen the AI Energy Score project featured by the Coalition for Sustainable AI as an example of best practices in benchmarking AI, and have presented it at events such as the ITU AI for Good conference and the IEA Forum on Energy and AI as examples of concrete initiatives that can be adopted by policymakers and developers alike to quantify and reduce the environmental impacts of AI.

Sustainable AI Coalition July Newsletter

What’s next for AI Energy Score 🔮

AI Energy Score is a dynamic project that will continue to evolve as new models and tasks are added and as the AI community grows. In the future, we hope to add additional modalities such as video generation , which has been shown to be very energy intensive even compared to image generation, as well as agentic tasks including computer use, coding, and tool use. We also aim to build more interest and buy-in from companies developing AI models, and we hope that more proprietary models will be tested and benchmarked alongside the current models, which are primarily open weights.

The future of AI Energy Score depends on community support. Together, we can build the transparency foundation the industry needs to align AI innovation with our planetary boundaries. If you’re interested in contributing, integrating the Score into your own systems, or exploring other forms of collaboration, feel free to start a discussion here .

I built a tiny RSS generator for my Advent of Code solutions

Lobsters
hamatti.org
2025-12-06 15:53:49
Comments...
Original Article

RSS/Atom feeds are one of the great technologies in the open web . They allow me to follow other people and them to follow me.

This week, I started solving Advent of Code problems and this time I’m publishing my explanations as part of my Digital Garden and that doesn’t support separated RSS feeds for a subset of notes.

Two IndieWeb principles that I love are Make what you need and Use what you make . In that spirit, I built a tiny tool today afterwork to enable people to follow my Advent of Code explanations via RSS.

A bit of background: my personal website at https://hamatti.org is built with Eleventy (and a bunch of custom scripts) and my digital garden at https://notes.hamatti.org is powered by notes in Obsidian and Quartz that turns them to a website.

To build this tiny RSS generator, I combined a Node.js script with Eleventy’s Global Data Files and templating . 

const fs = require("fs");

function addEntry(entryUrl) {
  // Read existing JSON in
  const data = JSON.parse(fs.readFileSync("_data/aoc2025.json", "utf-8"));

  // Extract Advent of Code puzzle day from URL
  // URL looks like this:
  // https://notes.hamatti.org/technology/advent-of-code/2025/day-1
  // so capture the number from `day-1` part.
  const day = entryUrl.match(/day-(\d+)/)[1];

  const today = new Date();

  // Add new feed entry
  data.entries.push({
    url: entryUrl,
    title: `Advent of Code 2025, day ${day}, explanation and solution.`,
    created: today,
  });

  // Update metadata
  data.updated = today;

  // Write new data to JSON file
  fs.writeFileSync("_data/aoc2025.json", JSON.stringify(data));
}

if (process.argv.length < 3) {
  console.log("Usage: npm run aocfeed [url-to-note]");
  process.exit(1);
}

addEntry(process.argv[2]);

It’s a tiny script that reads existing JSON, adds a new entry and writes it back.

Unfortunately I don’t yet have an easy way to add the full content since it’s not tied to my digital garden publishing pipeline but that can be an exercise for a day when I have more time.

The data it writes looks like this, at _data/aoc2025.json : 

{
  "entries": [
    {
      "url": "https://notes.hamatti.org/technology/advent-of-code/2025/day-1",
      "title": "Advent of Code 2025, day 1, explanation and solution.",
      "created": "2025-12-02T15:17:43.981Z"
    },
    {
      "url": "https://notes.hamatti.org/technology/advent-of-code/2025/day-2",
      "title": "Advent of Code 2025, day 2, explanation and solution.",
      "created": "2025-12-02T15:17:58.315Z"
    }
  ],
  "updated": "2025-12-02T15:17:58.315Z"
}

and I have a feed template like this 

---
permalink: feed/aoc2025.xml
excludeFromSitemap: true
---
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Advent of Code 2025 - Solutions and explanations by Juhis</title>
  <link href="https://hamatti.org/feed/aoc2025.xml" rel="self" />
  <link href="https://notes.hamatti.org/technology/advent-of-code/2025/advent-of-code-2025-landing-page" />
  <updated>{{aoc2025.updated}}</updated>
  <id>https://hamatti.org/feed/aoc2025.xml</id>
  <author>
    <name>{{ metadata.author.name }}</name>
    <email>{{ metadata.author.email }}</email>
  </author>
  {%- for entry in aoc2025.entries | reverse | limit(30) %}
  <entry>
    <title>{{entry.title}}</title>
    <link href="{{ entry.url }}" />
    <updated>{{ entry.created }}</updated>
    <id>{{entry.url}}</id>
    <content type="html">
      {{entry.title}}
    </content>
  </entry>
  {%- endfor %}
</feed>

Which reads data from the aoc2025.json file and populates a feed whenever I call Eleventy to create a new build.

Once I’ve published a new note, I run 

node _scripts/updateAdventOfCodeFeed.js [url-to-note]

and push changes to my website.

Once that deploy is done, users can follow my Advent of Code via https://hamatti.org/feed/aoc2025.xml .

If something above resonated with you, let's start a discussion about it! Email me at juhamattisantala at gmail dot com and share your thoughts . In 2025, I want to have more deeper discussions with people from around the world and I'd love if you'd be part of that.

Encryption and Feminism: We’re Briging The Conversation Online

Internet Exchange
internet.exchangepoint.tech
2025-12-04 15:25:51
A follow-up to our Mozilla Festival session on Encryption and Feminism: Reimagining Child Safety Without Surveillance....
Original Article
privacy and security

A follow-up to our Mozilla Festival session on Encryption and Feminism: Reimagining Child Safety Without Surveillance.

Encryption and Feminism: We’re Briging The Conversation Online
Gerda Binder, Hera Hussain, Georgia Bullen, Audrey Hingle, Lucy Purdon, and Mallory Knodel in our MozFest session.

By Audrey Hingle

Our MozFest session on Encryption and Feminism: Reimagining Child Safety Without Surveillance was bigger than a one-hour festival slot could contain. The room filled fast, people were turned away at the door, and the Q&A could have gone on twice as long. Many attendees told us afterwards that this is the conversation they’ve been waiting to have. That feminist perspectives on encryption aren’t just welcome, they’re needed. So we’re opening the circle wider and taking it online so more people can join in.

In the room, we heard reflections that reminded us why this work matters. In feedback forms, attendees told us encryption isn’t only a security feature, it’s “part of upholding the rights of kids and survivors too, now let’s prove that to the rest of the world!” Another participant said they left ready to “be a champion of encryption to protect all.” Someone else named what many feel: “More feminist spaces are needed!”

It quickly became clear that this work is collective. It’s about shifting assumptions, building new narratives, and demanding technology that does not treat privacy as optional or as something only privacy hardliners or cryptography experts care about. Privacy is safety, dignity, and a precondition for seeking help. It is necessary to explore identity, form relationships, and grow up. Privacy is a human right.

We also heard calls for clarity and practicality: to reduce jargon, show people what encryption actually does, and push for privacy-preserving features more generally like screenshot protection and sender-controlled forwarding.

Participants also reminded us that encryption must account for disparity and intersectionality. Surveillance is not experienced equally. Some communities never get to “opt in” or consent at all. Feminist principles for encryption must reflect that reality.

And importantly, we heard gratitude for the tone of the session: open, candid, grounded, and not afraid to ask hard questions. “Normalize the ability to have tricky conversations in movement spaces,” someone wrote. We agree. These conversations shouldn’t only happen at conferences, they should live inside policy rooms, product roadmaps, activist communities, parenting forums, classrooms.

So let’s keep going.

New Virtual Session: Encryption and Feminism: Reimagining Child Safety Without Surveillance

🗓️ Feb 10, 4PM GMT, Online

Whether you joined us at MozFest, could't make it to Barcelona, or were one of the many who could not get into the room, this session is for you. We are running the event again online so more people can experience the conversation in full. We will revisit the discussion, share insights from the panel, and walk through emerging Feminist Encryption Principles, including the ideas and questions raised by participants.

Speakers will include Chayn’s Hera Hussain , Superbloom’s Georgia Bullen , Courage Everywhere’s Lucy Purdon , UNICEF’s Gerda Binder , and IX’s Mallory Knodel, Ramma Shahid Cheema and Audrey Hingle.

Help us grow this conversation. Share it with friends and colleagues who imagine a future where children are protected without surveillance and where privacy is not a privilege, but a right.

We hope you’ll join us!

Related : If you care about privacy-preserving messaging apps, Phoenix R&D is inviting feedback through a short survey asking for input on what features matter most for those in at-risk contexts.

Hidden Influences: How algorithmic recommenders shape our lives by Dr. Luca Belli

New book from IX client Dr. Luca Belli looks at how recommender systems function, how they are measured, and why accountability remains difficult. Luca draws on his experience co-founding Twitter’s ML Ethics, Transparency and Accountability work, contributing to standards at NIST, and advising the European Commission on recommender transparency.

Now available via MEAP on Manning. Readers can access draft chapters as they are released, share feedback directly, and receive the final version when complete. Suitable for researchers, policy teams, engineers, and anyone involved in governance or evaluation of large-scale recommendation systems. It is also written for general readers, with no advanced technical knowledge required, so when you're done with it, hand it to a curious family member who wants to understand how algorithms decide what they see.

Support the Internet Exchange

If you find our emails useful, consider becoming a paid subscriber! You'll get access to our members-only Signal community where we share ideas, discuss upcoming topics, and exchange links. Paid subscribers can also leave comments on posts and enjoy a warm, fuzzy feeling.

Not ready for a long-term commitment? You can always leave us a tip .

Become A Paid Subscriber

Internet Governance

Digital Rights

Technology for Society

Privacy and Security

Upcoming Events

Careers and Funding Opportunities

United States

Global

What did we miss? Please send us a reply or write to editor@exchangepoint.tech .

New wave of VPN login attempts targets Palo Alto GlobalProtect portals

Bleeping Computer
www.bleepingcomputer.com
2025-12-06 15:18:19
A campaign has been observed targeting Palo Alto GlobalProtect portals with login attempts and launching scanning activity against SonicWall SonicOS API endpoints. [...]...
Original Article

Palo Alto Networks

A campaign has been observed targeting Palo Alto GlobalProtect portals with login attempts and launching scanning activity against SonicWall SonicOS API endpoints.

The activity started on December 2nd and originated from more than 7,000 IP addresses from infrastructure operated by the German IT company 3xK GmbH, which runs its own BGP network (AS200373) and operates as a hosting provider.

Initially, the actor targeted GlobalProtect portals with bruteforce and login attempts, then pivoted to scanning SonicWall API endpoints, threat intelligence company GreyNoise says in a report this week.

GlobalProtect is the VPN and remote access component of Palo Alto Networks’ firewall platform, used by large enterprises, government agencies, and service providers.

Number of IP addresses driving the attacks
Number of IP addresses driving the attacks
Source: GreyNoise

According to GreyNoise, the GlobalProtect login attempts targeted two profiles in the company's sensor network for passive capture of scanning and exploitation activity.

The researchers say that the surge used three client fingerprints previously observed in scanning attempts recorded between late September and mid-October.

This past activity originated from four ASNs with no history of malicious activity, generating over 9 million non-spoofable HTTP sessions, mostly targeting GlobalProtect portals.

In mid-November, GreyNoise also observed activity from 3xK Tech GmbH's infrastructure probing GlobalProtect VPN portals with 2.3 million scan sessions. Most of the attacking IPs (62%) were located in Germany, and used the same TCP/JA4t fingerprints.

Based on the analyzed indicators, the company confidently attributes both activities to the same actor.

On December 3, the same three fingerprints were seen in scanning activity targeting SonicWall SonicOS API.

SonicWall scanning activity
SonicWall scanning activity
Source: GreyNoise

SonicOS is the operating system running on SonicWall firewalls, exposing API endpoints for configuration, remote management, and monitoring.

Malicious scanning targeting these endpoints is typically done to identify vulnerabilities and misconfigurations. GreyNoise has previously noted that these scans may also help discover exposed infrastructure in preparation for potential exploitation of upcoming flaws.

For this reason, defenders are advised to monitor for IPs associated with this type of activity and block them.

It is also recommended to monitor authentication surfaces for abnormal velocity/repeated failures, track recurring client fingerprints, and use dynamic, context-aware blocking instead of static reputation lists.

BleepingComputer has contacted Palo Alto Networks and SonicWall about this activity.

Palo Alto Networks said that it detected increased scanning aimed at GlobalProtect interfaces, and confirmed that it "represents credential-based attacks, not an exploit of a software vulnerability."

"Furthermore, our internal telemetry and Cortex XSIAM protection confirm this activity does not constitute a compromise of our products or services," the company told BleepingComputer.

Palo Alto Networks recommends customers enforce Multi-Factor Authentication (MFA) to protect against credential abuse.

tines

Break down IAM silos like Bitpanda, KnowBe4, and PathAI

Broken IAM isn't just an IT problem - the impact ripples across your whole business.

This practical guide covers why traditional IAM practices fail to keep up with modern demands, examples of what "good" IAM looks like, and a simple checklist for building a scalable strategy.

Artificial intelligence research has a slop problem, academics say: ‘It’s a mess’

Guardian
www.theguardian.com
2025-12-06 15:00:55
AI research in question as author claims to have written over 100 papers on AI that one expert calls a ‘disaster’ A single person claims to have authored 113 academic papers on artificial intelligence this year, 89 of which will be presented this week at one of the world’s leading conference on AI a...
Original Article

A single person claims to have authored 113 academic papers on artificial intelligence this year, 89 of which will be presented this week at one of the world’s leading conference on AI and machine learning, which has raised questions among computer scientists about the state of AI research.

The author, Kevin Zhu, recently finished a bachelor’s degree in computer science at the University of California, Berkeley, and now runs Algoverse, an AI research and mentoring company for high schoolers – many of whom are his co-authors on the papers. Zhu himself graduated from high school in 2018.

Papers he has put out in the past two years cover subjects like using AI to locate nomadic pastoralists in sub-Saharan Africa, to evaluate skin lesions , and to translate Indonesian dialects. On his LinkedIn, he touts publishing “100+ top conference papers in the past year”, which have been “cited by OpenAI, Microsoft, Google, Stanford, MIT, Oxford and more”.

Zhu’s papers are a “disaster”, said Hany Farid, a professor of computer science at Berkeley, in an interview. “I’m fairly convinced that the whole thing, top to bottom, is just vibe coding,” he said, referring to the practice of using AI to create software.

Farid called attention to Zhu’s prolific publications in a recent LinkedIn post , which provoked discussion of other, similar cases among AI researchers, who said their newly popular discipline faces a deluge of low-quality research papers, fueled by academic pressures and, in some cases, AI tools.

In response to a query from the Guardian, Zhu said that he had supervised the 131 papers, which were “team endeavors” run by his company, Algoverse. The company charges $3,325 to high-school students and undergraduates for a selective 12-week online mentoring experience – which involves help submitting work to conferences.

“At a minimum, I help review methodology and experimental design in proposals, and I read and comment on full paper drafts before submission,” he said, adding that projects on subjects such as linguistics, healthcare or education involved “principal investigators or mentors with relevant expertise”.

The teams used “standard productivity tools such as reference managers, spellcheck, and sometimes language models for copy-editing or improving clarity”, he said in response to a query about whether the papers were written with AI.

Bot watchers in turmoil

The review standards for AI research differ from most other scientific fields. Most work in AI and machine learning does not go undergo the stringent peer-review processes of fields such as chemistry and biology – instead, papers are often presented less formally at major conferences such as NeurIPS , one of the world’s top machine learning and AI gatherings, where Zhu is slated to present.

Zhu’s case points at a larger issue in AI research, said Farid. Conferences including NeurIPS are being overwhelmed with increasing numbers of submissions: NeurIPS fielded 21,575 papers this year, up from under 10,000 in 2020. Another top AI conference, the International Conference on Learning Representations (ICLR), reported a 70% increase in its yearly submissions for 2026’s conference, nearly 20,000 papers, up from just over 11,000 for the 2025 conference.

“Reviewers are complaining about the poor quality of the papers, even suspecting that some are AI-generated. Why has this academic feast lost its flavor?” asked the Chinese tech blog 36Kr in a November post about ICLR, noting that the average score reviewers had awarded papers had declined year-over-year.

Meanwhile, students and academics are facing mounting pressure to rack up publications and keep up with their peers. It is uncommon to produce a double-digit number – much less triple – of high quality academic computer science papers in a year, academics said. Farid says that at times, his students have “vibe coded” papers to up their publication counts.

“So many young people want to get into AI. There’s a frenzy right now,” said Farid.

NeurIPS reviews papers submitted to it, but its process is far quicker and less thorough than standard scientific peer review, said Jeffrey Walling, an associate professor at Virginia Tech. This year, the conference has used large numbers of PhD students to vet papers, which a NeurIPS area chair said compromised the process.

“The reality is that often times conference referees must review dozens of papers in a short period of time, and there is usually little to no revision,” said Walling.

Walling agreed with Farid that too many papers are being published right now, saying he’d encountered other authors with over 100 publications in a year. “Academics are rewarded for publication volume more than quality … Everyone loves the myth of super productivity,” he said.

On Zhu’s Algoverse’s FAQ page, answers discusses how the company’s program can help applicants’ future college or career prospects, saying: “The skills, accomplishments, and publications you achieve here are highly regarded in academic circles and can indeed strengthen your college application or résumé. This is especially true if your research is admitted to a top conference – a prestigious feat even for professional researchers.”

Farid says that he now counsels students to not go into AI research, because of the “frenzy” in the field and the large volume of low-quality work being put out by people hoping to better their career prospects.

skip past newsletter promotion

“It’s just a mess. You can’t keep up, you can’t publish, you can’t do good work, you can’t be thoughtful,” he said.

Slop flood

Much excellent work has still come out of this process. Famously, Google’s paper on transformers, Attention Is All You Need – the theoretical basis for the advances in AI that led to ChatGPT – was presented at NeurIPS in 2017.

NeurIPS organisers agree the conference is under pressure. In a comment to the Guardian, a spokesperson said that the growth of AI as a field had brought “a significant increase in paper submissions and heightened value placed on peer-reviewed acceptance at NeurIPS”, putting “considerable strain on our review system”.

Zhu’s submissions were largely to workshops within NeurIPS, which have a different selection process than the main conference and are often where early-career work gets presented, said NeurIPS organisers. Farid said he did not find this a substantive explanation for one person to put his name on more than 100 papers.

“I don’t find this a compelling argument for putting your name on 100 papers that you could not have possibly meaningfully contributed to,” said Farid.

The problem is bigger than a flood of papers at NeurIPS. ICLR used AI to review a large volume submissions – resulting in apparently hallucinated citations and feedback that was “very verbose with lots of bullet points”, according to a recent article in Nature.

The feeling of decline is so widespread that finding a solution to the crisis has become the subject of papers itself. A May 2025 position paper – an academic, evidence-based version of a newspaper op-ed – authored by three South Korean computer scientists that proposed a solution to the “unprecedented challenges with the surge of paper submissions, accompanied by growing concerns over review quality and reviewer responsibility”, won an award for outstanding work at the 2025 International Conference on Machine Learning.

Meanwhile, says Farid, major tech companies and small AI safety organisations now dump their work on arXiv, a site once reserved for little-viewed preprints of math and physics papers, flooding the internet with work that is presented as science – but is not subject to review standards.

The cost of this, says Farid, is that it is almost impossible to know what’s actually going on in AI – for journalists, the public, and even experts in the field: “You have no chance, no chance as an average reader to try to understand what is going on in the scientific literature. Your signal-to-noise ratio is basically one. I can barely go to these conferences and figure out what the hell is going on.”

“What I tell students is that, if what you’re trying to optimize publishing papers, you know, it’s actually honestly not that hard to do. Just do really crappy low-quality work and bomb conferences with it. But if you want to do really thoughtful, careful work, you’re at a disadvantage because you’re effectively unilaterally disarmed,” he said.

HTML as an Accessible Format for Papers

Hacker News
info.arxiv.org
2025-12-06 14:59:52
Comments...
Original Article

HTML as an accessible format for papers

Accessibility barriers in research are not new, but they are urgent. The message we have heard from our community is that arXiv can have the most impact in the shortest time by offering HTML papers alongside the existing PDF.

arXiv has successfully launched papers in HTML format. We are gradually backfilling HTML for arXiv's corpus of over 2 million papers over time. Not every paper can be successfully converted, so a small percentage of papers will not have an HTML version. We will work to improve conversion over time.

The link to the HTML format will appear on abstract pages below the existing PDF download link. Authors will have the opportunity to preview their paper’s HTML as a part of the submission process.

The beta rollout is just the beginning. We have a long way to go to improve HTML papers and will continue to solicit feedback from authors, readers, and the entire arXiv community to improve conversions from LaTeX.

Why "experimental" HTML?

Did you know that 90% of submissions to arXiv are in TeX format, mostly LaTeX? That poses a unique accessibility challenge: to accurately convert from TeX—a very extensible language used in myriad unique ways by authors—to HTML, a language that is much more accessible to screen readers and text-to-speech software, screen magnifiers, and mobile devices. In addition to the technical challenges, the conversion must be both rapid and automated in order to maintain arXiv’s core service of free and fast dissemination.

Because of these challenges we know there will be some conversion and rendering issues. We have decided to launch in beta with “experimental” HTML because:

  1. Accessible papers are needed now. We have talked to the arXiv community, especially researchers with accessibility needs, and they overwhelmingly asked us not to wait.
  2. We need your help. The obvious work is done. Reports from the community will help us identify issues we can track back to specific LaTeX packages that are not converting correctly.

Error messages you may see in HTML papers

HTML papers on arXiv.org are a work in progress and will sometimes display errors. As we work to improve accessibility we share with you the causes of these errors and what authors can do to help minimize them. Learn more about error messages you may see in HTML papers

Ways to help

1) Read HTML papers and report issues

We encourage the community to try out HTML papers in your field:

Report an issue

  • Go to the abstract page for a paper you are interested in reading.
  • Look in the section where you find the link to the PDF download, and click the new link for HTML.
  • Report issues by either a) clicking on the Open Issue button b) selecting text and clicking on the Open Issue for Selection button or c) use Ctrl+? on your keyboard. If you are using a screen reader, use Alt+y to toggle accessible reporting buttons per paragraph.

Please do not create reports that the HTML paper doesn't look exactly like the PDF paper

Our primary goal for this project is to make papers more accessible, so the focus during the beta phase will value function over form. HTML layouts that are incorrect or are illegible are important to report. But we do expect the HTML papers to present differently than the same paper rendered in PDF. Line breaks will occur in different places and there is likely to be more white space. In general, the HTML paper won't present as compactly. Intricate typographic layouts will not be rendered so intricately. This is by design.

HTML is a different medium and brings its own advantages versus PDF. In addition to being much more compatible with assistive technologies, HTML does a far better job adapting to the characteristics of the device you are reading on, including mobile devices.

2) Help improve the conversion from LaTeX

If you are an author you can help us improve conversions to HTML by following our guide to LaTeX Markup Best Practices for Successful HTML Papers .

If you are a developer and have free development cycles, help us improve conversions! Our collaborators at LaTeXML maintain a list of issues and welcome feedback and developer contributions.

If you are a publisher, member of a society, or conference organizer you can help us improve conversions to HTML by reviewing the .cls files your organization recommends to authors for unsupported packages. Providing .cls files that use supported packages is an easy way to support and sow accessibility in the scientific community.

Thank you to our collaborators

First, we want to share a special thank you to all the scientists with disabilities who have generously shared their insights, expertise, and guidance throughout this project.

We want to thank two organizations without which HTML papers on arXiv would not be possible: The LaTeX Project , and the LaTeXML team from NIST. We deeply thank each member of these teams for their knowledge, incredible work, and commitment to accessibility.

Trump’s War on Venezuelans: From Operation Aurora to Operation Southern Spear

Portside
portside.org
2025-12-06 14:56:09
Trump’s War on Venezuelans: From Operation Aurora to Operation Southern Spear Kurt Stand Sat, 12/06/2025 - 09:56 ...
Original Article

On October 11, 2024, Donald Trump stood before a rally in the Colorado suburb of Aurora, flanked by mug shots he labeled “Illegal immigrant gang members from Venezuela.” Following an introduction by his current senior advisor, Stephen Miller, who gestured to the posters and asked the crowd if “these are the neighbors you want,” Trump took the stage to announce a campaign promise he dubbed “Operation Aurora.” Under the plan, he pledged to invoke the Alien Enemies Act, a 1798 wartime law that authorizes deporting noncitizens from nations at war with the United States, to conduct mass deportations if re-elected.

The choice to name his mass deportation agenda after the Colorado suburb was a direct nod to a pervasive right-wing narrative that the Venezuelan gang Tren de Aragua had taken over Aurora apartment complexes housing primarily Venezuelan migrants. The claim, sparked by a viral video , was immediately refuted by tenants and local authorities , who instead pointed to CBZ Management—a multi-million-dollar property firm—as the criminal culprit that had circulated the false narrative to deflect blame for its own neglect.  Nevertheless, a year later, the Aurora myth continues to play a key role in a larger feedback loop in which local police bulletins, uncritically disseminated by federal agencies and right-wing media, create boogie men to serve as the ideological scaffolding for an ethnonationalist agenda.

As promised, on March 15, Trump invoked the Alien Enemies Act, claiming that the Tren de Aragua is operating “in conjunction” with Venezuelan President Nicolás Maduro to “invade” our borders in a “plot against America.” Though unsupported by evidence—and rejected by the United States’ own intelligence community—the claim has now become the legal pretext for the sweeping suspension of due process rights and the criminalization of Venezuelans at home and abroad. This reality was made manifest in the detention, deportation, and eventual torture of 238 Venezuelan migrants sent to El Salvador’s Centro de Confinamiento del Terrorismo (CECOT) mega prison.

Since then, the same security paradigm has been deployed to Venezuela’s Caribbean waters as the political justification of what U.S. Secretary of War Pete Hegseth recently dubbed “Operation Southern Spear.” The operation, pushed atop thinly veiled accusations of drug trafficking, has led to the deployment of the largest concentration of U.S. naval and air assets in the region in 20 years and extrajudicial strikes against alleged drug traffickers that have killed at least 83 people across the Caribbean and Eastern Pacific.

Situated within this same security paradigm, the case of Aurora exposes how the architecture of U.S. imperialism seeps into the fabric of domestic life and reveals the contradictory nature of U.S. expansionism. Venezuelan migrants—caught at the nexus of the War on Terror, the War on Drugs, and the War on Migrants—are both created and criminalized by U.S. policy, while media hysteria dictates who belongs, who is dangerous, and who can be disappeared without notice.

A Slumlord’s Scapegoat

The narrative of a Venezuelan gang takeover in Aurora was seeded prior to the viral videos that fueled it. On July 28, 2024, then Aurora City Council member Danielle Jurinsky falsely claimed that a public Venezuelan election watch party, held at a local Target parking lot, ended in “assaults, theft,” and a “shot up police car.” Although the Aurora Police Department immediately released a statement debunking her claims, Mayor Mike Coffman amplified the rhetoric, stating that “residents were owed an apology from the Venezuelan immigrant community.”

As this political firestorm ignited, a quieter crisis was culminating for New York-based CBZ Management. The corporate landlord, which owns three Aurora complexes, was facing legal repercussions from the city over rampant code violations, including mold, pests, broken utilities, and unsafe conditions documented as far back as 2020 . By mid-2024, facing mounting fines and possible criminal penalties, the company was on the brink of legal reckoning.

In reality, while tenants had suffered from violent crime and unsafe conditions, there was no evidence that the complexes were “taken” by gangs—let alone the Venezuelan group.

On July 29th, just one day after the election watch party, city attorneys emailed CBZ’s lawyers a proposed settlement that included a 60-day jail sentence for company owner Zev Baumgarten, citing chronic neglect. Within a week, CBZ retained a PR firm that spun the narrative by pushing a sensational story to local news: their properties had been “taken over by Tren de Aragua.”

In reality, while tenants had suffered from violent crime and unsafe conditions, there was no evidence that the complexes were “taken” by gangs—let alone the Venezuelan group. There was one telling link in the story, however: every viral claim of a Tren de Aragua “takeover” that emerged during this period could be traced back to CBZ-owned properties. “You see a similar pattern emerging here where they own multiple properties within 10 minutes of each other,” said Nate Kassa, a tenant organizer who assisted the residents placed at the center of the Tren de Aragua takeover story. “All of them are essentially slums.”

It was the tenants that ultimately had to answer for their slumlord’s misdeeds. In August 2024, as many as 200 residents were evicted from one CBZ property in Aurora with as little as 24-hours notice, while tenants at the two other properties—The Edge at Lowry and Whispering Pines apartments—were offered a $1,200 buyout to leave under threat of eviction. In a pattern that would be repeated across the country, those that choose to stay, citing the financial difficulties of relocation, saw armed federal agents essentially carry out the work of eviction in Aurora.

Aurora ICE Raids and Community Resistance

In the pre-dawn hours of February 6, 2025, a militarized convoy of dozens of ICE, DHS, ATF and DEA vehicles descended on residences across Denver and Aurora, including the Edge at Lowry and Whispering Pines. The operation, launched two weeks after Trump’s inauguration, was designed for flashy spectacle. Federal agents armed with rifles and battering rams descended on the apartment complexes, accompanied by an embedded Fox News crew that aired the raids in real-time on social media. Using flash grenades and rubber bullets, officials claimed they were targeting over 100 alleged members of the Tren de Aragua gang.

Even still, for those swept up in the crackdown, the consequences were severe. Some residents, branded as “gang members” based upon flimsy evidence that largely conflated their Venezuelan nationality with criminality, were ultimately arrested and deported to CECOT.

Yet according to community organizers, on the ground a different story unfolded. With the Trump administration publicly promising mass raids at the apartments for months, a robust organizing network of activists with the Colorado Rapid Response Network (CORRN) were already on high alert. At the Edge at Lowry, activists slept in their cars. “We knew ICE was targeting that building, so we got to know people that lived there and came up with an organizing strategy” explained Sophia Morrow*, a teacher, parent, and member of CORRN for five years. As federal agents went door-to-door, protesters with bullhorns fanned out, shouted “Know Your Rights” information in English and Spanish, and informed residents they were not obligated to open their doors without a judicial warrant.

The result was a dramatic failure . Although ICE was reluctant to share any official arrest figures, FOX later reported that only 30 individuals were detained across the entirety of the massive operation.  Speaking in front of the White House, border czar Tom Homan falsely blamed the failure on media leaks and threatened the activists, saying “they may find themselves in a pair of handcuffs very soon.” The truth, however, was much simpler: a coordinated community defense network proved stronger than the force of hundreds of armed federal officers.

Even still, for those swept up in the crackdown, the consequences were severe. Some residents , branded as “gang members” based upon flimsy evidence that largely conflated their Venezuelan nationality with criminality, were ultimately arrested and deported to CECOT .

The Raids Continue

While the February raids—and the resistance against them—caused a media firestorm, the machinery of immigration enforcement has continued. It has done so in ways both more mundane and more insidious.

For Jorge*, a young Venezuelan granted Temporary Protected Status (TPS) in 2023, this reality struck when a routine delivery to drop off a television landed him at the Fort Carson Army Base in Colorado Springs. Though his heart pounded as he approached the military installation, he had a valid work permit and driver’s license while his asylum case was pending.

At the entrance, a guard took his ID, handed him a visitor form, and told him to wait while the paperwork was processed. Jorge did as he was told, but was surrounded by eleven armed soldiers soon after sitting down. Confused, he asked if he was being detained. “We just need to revise your paperwork,” an officer assured him in Spanish. Moments later, Jorge was handcuffed, placed in an unmarked vehicle, and taken to the Aurora GEO ICE facility to face removal proceedings—all while the soldier who had ordered the television arrived to casually collect it. The transaction continued around this unfolding nightmare.

Speaking with Jorge at the GEO ICE facility through the wall-mounted phones of our no contact visit, he struggled to make sense of what had transpired. Sophia later explained his detention. “It is a misconception that ICE needs a reason to detain people,” she said. Indeed, section 237 (a)(4) of the 1952 Immigration and Nationality Act gives ICE the authority to arrest and detain noncitizens pending immigration hearings, even if they have committed no crime.

Jorge’s community has fallen into a state of suspended terror. Within a month, another friend was detained at a mandatory ICE check-in, leaving his roommates—including one who cares for a young child—prisoners in their own home, terrified that a trip to the grocery store could be their last moment of freedom.

The Imperial Playbook

Venezuelans are punished at home by U.S. policies that cripple their nation with bombs and sanctions, punished for fleeing the crisis under U.S. narratives of security threats, and punished on U.S. soil as scapegoats for deteriorating living conditions.

Jorge’s story encapsulates the vicious, circular logic of U.S. imperialism. Fleeing Venezuela’s economic and political turmoil —a crisis exacerbated by years of U.S. foreign policy and sanctions—he was encouraged to move north under the promised shield of TPS. Told the city “would be a safe place,” he finally settled in Denver as one of some 40,000 Venezuelan migrants who arrived in Colorado between 2022 and 2024. Yet for many of these migrants, TPS proved to be a Faustian bargain.

This precarious situation was shattered on November 7 , when TPS was formally eliminated by the Trump administration for an estimated 250,000 Venezuelans , aided in part by a recent Supreme Court ruling . Overnight, people who had been protected were now reclassified as “illegal” and targeted accordingly. Now they are being deported back to their home country, which faces renewed threats of U.S. military intervention—a move that brings with it the threat of a new cycle of mass human displacement.

This pattern of suffering is not random nor new, but has been erected through bipartisan consensus and forged by a coalition of U.S. hawks and right-wing Venezuelans. While studies show U.S. sanctions are a root cause of the Venezuelan migration crisis, this reality is routinely omitted from media reports. As a result, the exodus itself is weaponized to criminalize the mobility of migrants and undermine Venezuela’s political sovereignty.

Trapped in this never-ending feedback loop of imperial violence, Venezuelans are punished at home by U.S. policies that cripple their nation with bombs and sanctions, punished for fleeing the crisis under U.S. narratives of security threats, and punished on U.S. soil as scapegoats for deteriorating living conditions.

Editor’s note: the names of Jorge and Sophia have been changed to protect their identities.

Isabel María Villalón is an independent scholar and Latin Americanist currently pursuing a master’s degree in Latin American Studies at the Universidad Nacional Autónoma de México (UNAM). Her recent work focuses on the intersections of neoliberalism, labor, and migration in Central America.

T he North American Congress on Latin America (NACLA) is an independent, nonprofit organization founded in 1966 to examine and critique U.S. imperialism and political, economic, and military intervention in the Western hemisphere. In an evolving political and media landscape, we continue to work toward a world in which the nations and peoples of Latin America and the Caribbean are free from oppression, injustice, and economic and political subordination.

Carlo is no longer maintained

Hacker News
github.com
2025-12-06 14:55:03
Comments...
Original Article

Carlo - headful Node app framework

❗Carlo is no longer maintained .

Carlo provides Node applications with Google Chrome rendering capabilities, communicates with the locally-installed browser instance using the Puppeteer project, and implements a remote call infrastructure for communication between Node and the browser.

API | FAQ | Contributing

image

What can I do?

With Carlo, users can create hybrid applications that use Web stack for rendering and Node for capabilities:

  • For Node applications, the web rendering stack lets users visualize the dynamic state of the app.
  • For Web applications, additional system capabilities are accessible from Node.
  • The application can be bundled into a single executable using pkg .

How does it work?

  • Carlo locates Google Chrome installed locally.
  • Launches Chrome and establishes a connection over the process pipe.
  • Exposes a high-level API for rendering in Chrome with the Node environment.

Usage

Install Carlo

npm 

npm i carlo
# yarn add carlo

Carlo requires at least Node v7.6.0.

Example - Display local environment

Save file as example.js 

const carlo = require('carlo');

(async () => {
  // Launch the browser.
  const app = await carlo.launch();

  // Terminate Node.js process on app window closing.
  app.on('exit', () => process.exit());

  // Tell carlo where your web files are located.
  app.serveFolder(__dirname);

  // Expose 'env' function in the web environment.
  await app.exposeFunction('env', _ => process.env);

  // Navigate to the main page of your app.
  await app.load('example.html');
})();

Save file as example.html 

<script>
async function run() {
  // Call the function that was exposed in Node.
  const data = await env();
  for (const type in data) {
    const div = document.createElement('div');
    div.textContent = `${type}: ${data[type]}`;
    document.body.appendChild(div);
  }
}
</script>
<body onload="run()">

Run your application:

Check out systeminfo and terminal examples with richer UI and RPC-based communication between the Web and Node in the examples folder.

API

Check out the API to get familiar with Carlo.

Testing

Carlo uses Puppeteer project for testing. Carlo application and all Carlo windows have corresponding Puppeteer objects exposed for testing. Please refer to the API and the systeminfo project for more details.

Contributing to Carlo

Look at the contributing guide to get an overview of Carlo's development.

FAQ

Q: What was the motivation behind this project when we already have Electron and NW.js? How does this project differ from these platforms, how does it achieve something that is not possible/harder with Electron or NW.js?

  • One of the motivations of this project is to demonstrate how browsers that are installed locally can be used with Node out of the box.
  • Node v8 and Chrome v8 engines are decoupled in Carlo, providing a maintainable model with the ability to independently update underlying components. Carlo gives the user control over bundling and is more about productivity than branding.

Q: Can a Node app using Carlo be packaged as a Desktop app?

The pkg project can be used to package a Node app as a Desktop app. Carlo does not provide branding configurability such as application icons or customizable menus, instead, Carlo focuses on productivity and Web/Node interoperability. Check out the systeminfo example and call pkg package.json to see how it works.

Q: What happens if the user does not have Chrome installed?

Carlo prints an error message when Chrome can not be located.

Q: What is the minimum Chrome version that Carlo supports?

Chrome Stable channel, versions 70.* are supported.

Drones to Diplomas: How Russia’s Largest Private University is Linked to a $25M Essay Mill

Krebs
krebsonsecurity.com
2025-12-06 14:45:03
A sprawling academic cheating network turbocharged by Google Ads that has generated nearly $25 million in revenue has curious connections to a Kremlin-connected oligarch whose Russian university builds drones for Russia's war against Ukraine....
Original Article

A sprawling academic cheating network turbocharged by Google Ads that has generated nearly $25 million in revenue has curious ties to a Kremlin-connected oligarch whose Russian university builds drones for Russia’s war against Ukraine.

The Nerdify homepage.

The link between essay mills and Russian attack drones might seem improbable, but understanding it begins with a simple question: How does a human-intensive academic cheating service stay relevant in an era when students can simply ask AI to write their term papers? The answer – recasting the business as an AI company – is just the latest chapter in a story of many rebrands that link the operation to Russia’s largest private university.

Search in Google for any terms related to academic cheating services — e.g., “help with exam online” or “term paper online” — and you’re likely to encounter websites with the words “nerd” or “geek” in them, such as thenerdify[.]com and geekly-hub[.]com . With a simple request sent via text message, you can hire their tutors to help with any assignment.

These nerdy and geeky-branded websites frequently cite their “honor code,” which emphasizes they do not condone academic cheating, will not write your term papers for you, and will only offer support and advice for customers. But according to This Isn’t Fine , a Substack blog about contract cheating and essay mills, the Nerdify brand of websites will happily ignore that mantra.

“We tested the quick SMS for a price quote,” wrote This Isn’t Fine author Joseph Thibault . “The honor code references and platitudes apparently stop at the website. Within three minutes, we confirmed that a full three-page, plagiarism- and AI-free MLA formatted Argumentative essay could be ours for the low price of $141.”

A screenshot from Joseph Thibault’s Substack post shows him purchasing a 3-page paper with the Nerdify service.

Google prohibits ads that “enable dishonest behavior.” Yet, a sprawling global essay and homework cheating network run under the Nerdy brands has quietly bought its way to the top of Google searches – booking revenues of almost $25 million through a maze of companies in Cyprus, Malta and Hong Kong, while pitching “tutoring” that delivers finished work that students can turn in.

When one Nerdy-related Google Ads account got shut down, the group behind the company would form a new entity with a front-person (typically a young Ukrainian woman), start a new ads account along with a new website and domain name (usually with “nerdy” in the brand), and resume running Google ads for the same set of keywords.

UK companies belonging to the group that have been shut down by Google Ads since Jan 2025 include:

– Proglobal Solutions LTD (advertised nerdifyit[.]com);
– AW Tech Limited (advertised thenerdify[.]com);
– Geekly Solutions Ltd (advertised geekly-hub[.]com).

Currently active Google Ads accounts for the Nerdify brands include:

-OK Marketing LTD (advertising geekly-hub[.]net⁩), formed in the name of the Ukrainian national Alexander (Oleksandr) Korsukov ;
– Two Sigma Solutions LTD (advertising litero[.]ai), formed in the name of Olekszij Pokatilo .

Google’s Ads Transparency page for current Nerdify advertiser OK Marketing LTD.

Messrs. Korsukov and Pokatilo have been in the essay-writing business since at least 2009, operating a paper-mill enterprise called Livingston Research . According to a lengthy account from a former employee , Livingston Research mainly farmed its writing tasks out to low-cost workers from Kenya, Philippines, Pakistan, Russia and Ukraine.

In 2011, the two men set up a Cyprus corporation called VLS Research Ltd , which would later change its name to CLS Research Ltd . Pokatilo moved from Ukraine to the United Kingdom in Sept. 2015 and co-founded a company called Awesome Technologies , which pitched itself as a way for people to outsource tasks by sending a text message to the service’s assistants.

The other co-founder of Awesome Technologies is 36-year-old Filip Perkon , a Swedish man living in London who touts himself as a serial entrepreneur and investor. Years before starting Awesome together, Perkon and Pokatilo co-founded a student group called Russian Business Week while the two were classmates at the London School of Economics. According to the Bulgarian investigative journalist Christo Grozev , Perkon’s birth certificate was issued by the Soviet Embassy in Sweden.

Alexey Pokatilo (left) and Filip Perkon at a Facebook event for startups in San Francisco in mid-2015.

Around the time Perkon and Pokatilo launched Awesome Technologies, Perkon was building a social media propaganda tool called the Russian Diplomatic Online Club , which Perkon said would “turbo-charge” Russian messaging online. The club’s newsletter urged subscribers to install in their Twitter accounts a third-party app called Tweetsquad that would retweet Kremlin messaging on the social media platform.

Perkon was praised by the Russian Embassy in London for his efforts: D uring the contentious Brexit vote that ultimately led to the United Kingdom leaving the European Union, the Russian embassy in London used this spam tweeting tool to auto-retweet the Russian ambassador’s posts from supporters’ accounts .

Neither Mr. Perkon nor Mr. Pokatilo replied to requests for comment.

A review of corporations tied to Mr. Perkon as indexed by the business research service North Data finds he holds or held director positions in several U.K. subsidiaries of Synergy , Russia’s largest private education provider. Synergy has more than 35,000 students, and sells T-shirts with patriotic slogans such as “Crimea is Ours,” and The Russian Empire — Reloaded.”

The president of Synergy is Vadim Lobov , a Kremlin insider whose headquarters on the outskirts of Moscow reportedly features a wall-sized portrait of Russian President Vladimir Putin in the pop-art style of Andy Warhol. For a number of years, Lobov and Perkon co-produced a cross-cultural event in the U.K. called Russian Film Week .

Synergy President Vadim Lobov and Filip Perkon, speaking at a press conference for Russian Film Week, a cross-cultural event in the U.K. co-produced by both men.

Mr. Lobov was one of 11 individuals reportedly hand-picked by the convicted Russian spy Marina Butina to attend the 2017 National Prayer Breakfast held in Washington D.C. just two weeks after President Trump’s first inauguration.

While Synergy University promotes itself as Russia’s largest private educational institution, hundreds of international students tell a different story. Online reviews from students paint a picture of unkept promises: Prospective students from Nigeria, Kenya, Ghana, and other nations paying thousands in advance fees for promised study visas to Russia, only to have their applications denied with no refunds offered.

“My experience with Synergy University has been nothing short of heartbreaking,” reads one such account. “When I first discovered the school, their representative was extremely responsive and eager to assist. He communicated frequently and made me believe I was in safe hands. However, after paying my hard-earned tuition fees, my visa was denied. It’s been over 9 months since that denial, and despite their promises, I have received no refund whatsoever. My messages are now ignored, and the same representative who once replied instantly no longer responds at all. Synergy University, how can an institution in Europe feel comfortable exploiting the hopes of Africans who trust you with their life savings? This is not just unethical — it’s predatory.”

This pattern repeats across reviews by multilingual students from Pakistan, Nepal, India, and various African nations — all describing the same scheme: Attractive online marketing, promises of easy visa approval, upfront payment requirements, and then silence after visa denials.

Reddit discussions in r/Moscow and r/AskARussian are filled with warnings. “It’s a scam, a diploma mill,” writes one user. “They literally sell exams. There was an investigation on Rossiya-1 television showing students paying to pass tests.”

The Nerdify website’s “About Us” page says the company was co-founded by Pokatilo and an American named Brian Mellor . The latter identity seems to have been fabricated, or at least there is no evidence that a person with this name ever worked at Nerdify.

Rather, it appears that the SMS assistance company co-founded by Messrs. Pokatilo and Perkon (Awesome Technologies) fizzled out shortly after its creation, and that Nerdify soon adopted the process of accepting assignment requests via text message and routing them to freelance writers.

A closer look at an early “About Us” page for Nerdify in The Wayback Machine suggests that Mr. Perkon was the real co-founder of the company: The photo at the top of the page shows four people wearing Nerdify T-shirts seated around a table on a rooftop deck in San Francisco, and the man facing the camera is Perkon.

Filip Perkon, top right, is pictured wearing a Nerdify T-shirt in an archived copy of the company’s About Us page. Image: archive.org.

Where are they now? Pokatilo is currently running a startup called Litero.Ai , which appears to be an AI-based essay writing service. In July 2025, Mr. Pokatilo received pre-seed funding of $800,000 for Litero from an investment program backed by the venture capital firms AltaIR Capital, Yellow Rocks, Smart Partnership Capital, and I2BF Global Ventures.

Meanwhile, Filip Perkon is busy setting up toy rubber duck stores in Miami and in at least three locations in the United Kingdom . These “Duck World” shops market themselves as “the world’s largest duck store.”

This past week, Mr. Lobov was in India with Putin’s entourage on a charm tour with India’s Prime Minister Narendra Modi. Although Synergy is billed as an educational institution, a review of the company’s sprawling corporate footprint (via DNS) shows it also is assisting the Russian government in its war against Ukraine.

Synergy University President Vadim Lobov (right) pictured this week in India next to Natalia Popova, a Russian TV presenter known for her close ties to Putin’s family, particularly Putin’s daughter, who works with Popova at the education and culture-focused Innopraktika Foundation.

The website bpla.s ynergy[.]bot , for instance, says the company is involved in developing combat drones to aid Russian forces and to evade international sanctions on the supply and re-export of high-tech products.

A screenshot from the website of synergy,bot shows the company is actively engaged in building armed drones for the war in Ukraine.

KrebsOnSecurity would like to thank the anonymous researcher NatInfoSec for their assistance in this investigation.

Kidney Recipient Dies After Transplant from Organ Donor Who Had Rabies

Hacker News
www.nytimes.com
2025-12-06 14:40:51
Comments...
Original Article

Please enable JS and disable any ad blocker

Quoting Daniel Lemire

Simon Willison
simonwillison.net
2025-12-06 14:40:46
If you work slowly, you will be more likely to stick with your slightly obsolete work. You know that professor who spent seven years preparing lecture notes twenty years ago? He is not going to throw them away and start again, as that would be a new seven-year project. So he will keep teaching using...
Original Article

If you work slowly, you will be more likely to stick with your slightly obsolete work. You know that professor who spent seven years preparing lecture notes twenty years ago? He is not going to throw them away and start again, as that would be a new seven-year project. So he will keep teaching using aging lecture notes until he retires and someone finally updates the course.

— Daniel Lemire , Why speed matters

Tiny Core Linux: a 23 MB Linux distro with graphical desktop

Hacker News
www.tinycorelinux.net
2025-12-06 14:18:42
Comments...
Original Article

Welcome to The Core Project - Tiny Core Linux

The Core Project is a highly modular based system with community build extensions.

It starts with a recent Linux kernel, vmlinuz, and our root filesystem and start-up scripts packaged with a basic set of kernel modules in core.gz. Core (11MB) is simply the kernel + core.gz - this is the foundation for user created desktops, servers, or appliances. TinyCore is Core + Xvesa.tcz + Xprogs.tcz + aterm.tcz + fltk-1.3.tcz + flwm.tcz + wbar.tcz

TinyCore becomes simply an example of what the Core Project can produce, an 16MB FLTK/FLWM desktop.

CorePlus ofers a simple way to get started using the Core philosophy with its included community packaged extensions enabling easy embedded frugal or pendrive installation of the user's choice of supported desktop, while maintaining the Core principal of mounted extensions with full package management.

It is not a complete desktop nor is all hardware completely supported. It represents only the core needed to boot into a very minimal X desktop typically with wired internet access.

The user has complete control over which applications and/or additional hardware to have supported, be it for a desktop, a netbook, an appliance, or server, selectable by the user by installing additional applications from online repositories, or easily compiling most anything you desire using tools provided.

The latest version: 16.2

News

About Our Project

Our goal is the creation of a nomadic ultra small graphical desktop operating system capable of booting from cdrom, pendrive, or frugally from a hard drive. The desktop boots extremely fast and is able to support additional applications and hardware of the users choice. While Tiny Core always resides in ram, additional applications extensions can either reside in ram, mounted from a persistent storage device, or installed into a persistent storage device.

We invite interested users and developers to explore Tiny Core. Within our forums we have an open developement model. We encourage shared knowledge. We promote community involvement and community built application extensions. Anyone can contribute to our project by packaging their favorite application or hardware support to run in Tiny Core. The Tiny Core Linux Team currently consists of eight members who peruse the forums to assist from answering questions to helping package new extensions.

Join us here and on IRC Freenode #tinycorelinux .

Learn. Share. Grow your knowledge of Linux.

Robert Shingledecker, December 01, 2008

Ecosocialism or Extinction: Defending Life, Building Free Territories and Ecosocialism From and for the Peoples

Portside
portside.org
2025-12-06 14:10:24
Ecosocialism or Extinction: Defending Life, Building Free Territories and Ecosocialism From and for the Peoples Kurt Stand Sat, 12/06/2025 - 09:10 ...
Original Article

We don’t sell our land because it is like our mother. Our territory is our body. And we don’t sell our body. We don’t sell our mother. We wouldn’t sell it, because it is sacred.
And we start suffering pressures of invasion, pressure from mining, from agribusiness, which has expanded a lot, pressure from logging companies, which are deforesting our territories. And we have been resisting.
—Auricelia Arapiun, Coordinator of Indigenous Organizations of the Brazilian Amazon (COIAB).

We gather at moment of profound capitalist attacks on life, within the framework of the actions organized by the peoples in response to COP30. This meeting has allowed us, once again, to reaffirm that both the rise of the far right and the false solutions proposed by governments that call themselves progressive (yet do not hesitate to privatize the commons or facilitate attacks against peoples and leaders who face daily the consequences of the logic of infinite capital growth in their territories) push us to struggle for a world in which living systems are at the center of all our political constructions, and to forcefully reject any attempt at intimidation.

We have seen an example of what happens when, instead of strengthening the struggles of peoples who defend their territories at the risk of their own lives, the defenders of progressive neoliberalism place themselves at the service of capital and predatory extractivism. The political threats suffered by our Indigenous comrade Auricelia Arapiun during her intervention in our roundtable on the current conjuncture clearly reveal a sector acting within communities to sow fear and fragmentation. Yet we — just as Auricelia expressed in her response to the threat — neither remain silent nor compromise.

The offensive of the far right also manifests in our territories through attempts to violate our sovereignty, reproducing the same logics of subjugation and domination that existed in the past and persist today. Against this imperialist offensive, we, ecosocialists, defend a united front to resist and protect ourselves.

Ecosocialism, as a tool to build another world, has become necessary and urgent. The accelerating destruction of ecosystems’ capacity for reproduction and the neocolonial and imperialist character of the supposed alternatives proposed by the very system that created the current climate emergency represent a threat to our continuity as a species, leading us toward a point of no return.

Faced with this challenge, the only possible path is the coordinated organization of our struggles in order to surpass the capitalist system. The organized struggle of peoples, their resistance to systems of domination, and their progress in building other worlds founded on solidarity, complementarity, and reciprocity — respecting the knowledge and cosmovisions of different peoples as well as their legitimate rights to self-defense and self-determination — form the fundamental basis of our strategy.

These days of debate brought together representatives of peoples in struggle from different regions of Abya Yala [an Indigenous name for the Americas, meaning “Continent of life” —ed.] and other continents, who have raised their voices globally to denounce that capitalist and imperialist extractivisms are causing environmental and human destruction in many territories. It is necessary to strengthen the alliances among peoples in resistance in order to combat this destruction, while consolidating forms of life-production historically developed by the peoples and today threatened by the contamination and appropriation of water, land, and air by transnational corporations and governments.

The voices of Indigenous peoples were central in this gathering, identifying a shared context of colonialism, invasion, dispossession, extractivism, and false solutions — accompanied by policies of annihilation and genocide, which not only kill but also render these peoples invisible through criminalization and persecution. At this stage, we see the relationship between body and territory as a fabric where structural violence resides, but also the struggle for life. This struggle manifests in alternative forms of resistance, through the valorization and articulation of knowledge and cosmologies in which ancestry and nature are inseparable; and through self-defense, self-determination, community life, and the importance of hope and unity across territories.

These struggles for life also appear in ecofeminisms, highlighting the struggles of women and feminized bodies across different territories of Abya Yala as they confront the close and historical relationship between capitalism and the violence inflicted on the land, the territories, and women.

From the various forms of extractivism emerges a violence expressed through the contamination and destruction of land; the predation and theft of our commons; the fragmentation of cultural perspectives; and upon the feminized, impoverished, and racialized bodies of thousands of women of the Global South.

This analysis, in addition to identifying capitalism as the structural origin of all territorial violence, also proposes solutions capable of overcoming these contradictions — such as community water management, food autonomy, self-government, community justice, and a subversive conception of care. This vision of care arises from a structural critique of the neoliberalization of the care discourse, which continues to support the logic of capital. In contrast, we position ourselves in favor of collective and community care for radical transformation.

Eco-unionism is a fundamental component of the ecosocialist struggle. The fight for more and better working conditions, combined with the awareness that the exploitation of the working class and the dispossession of our commons serve the interests of capital and mutually reinforce each other, creates the conditions needed to mobilize and advance the structural causes of the oppressions we suffer under capitalism. In this sense, rejecting fracking in Colombia, Latin America, the Caribbean, and worldwide is a task we assume with responsibility to contribute to building free territories. We know that this will only be possible if trade unions articulate with social, popular, Indigenous, and peasant movements in each country, while maintaining their autonomy in defending territories, life, and its reproduction. Through internationalist solidarity, we commit to promoting spaces that denounce violations of labor, human, and natural rights.

From within this shared fabric, we unanimously cry out: Free Palestine, from the river to the sea; ceasefire in Gaza; and condemnation of the genocidal State of Israel for the massacre of the Palestinian people. A people who resist, who sow, who maintain the conviction to stand tall — and whom we embrace through internationalist solidarity, multiplying global actions of support such as BDS and the Flotilla, examples of grassroots resistance that the State of Israel considers threats.

We also demand that governments in the region break their relations with Israel, as in the case of agreements with Mekorot, Israel’s national water company, which has become an instrument of colonial domination. Water is a common good and, in Palestine, it is used as a political and economic weapon: Israel controls water sources, prevents Palestinians from drilling wells, collecting rainwater, or maintaining cisterns, thereby creating total dependence and a system of water apartheid. Palestine is a laboratory of domination whose techniques spread to other territories, and resistance and solidarity with the Palestinian people must be global. We, ecosocialists of the world, stand with and build active solidarity with the Palestinian people and their right to exist.

Days before the start of COP30, we once again observe that this space is incapable of responding to the needs of territories; on the contrary, it presents itself as a mechanism for the financialization of nature. This is why we reaffirm our denunciation and rejection of the payment of odious and illegitimate debts, and call for the dismantling of the international mechanisms that drive and legitimize them. These mechanisms mortgage our future in exchange for the delivery of strategic goods that capital needs for its unlimited reproduction. It is essential to dismantle the debt system, which subordinates and limits the capacity for a planned exit from the system.

We expect nothing from these spaces that propose projects such as carbon credits, which — just like TFFF — embrace the narrative that the problem is that the commons are not yet fully commodified and that there exists a “market failure” to overcome. We also denounce governments complicit in ecocidal projects, such as the Brazilian government which, only days before COP30 in Belém — an Amazonian territory — approved offshore oil exploitation at the mouth of the Amazon, and which, during COP30, approved the registration of 30 new pesticides.

We reaffirm agroecology as one of the paths that build our ecosocialist strategy. The production of agroecological food, rooted in peasant and Indigenous traditions, is not only an alternative to the dominant agro-food system — whose main actors are agribusiness and commodity production — but also a way to restore and rebuild ecosystems, and to break the alienation between countryside and city, making it fundamental in the fight against climate change. It is crucial to understand that agroecology cannot exist within green capitalism, as it involves, as a political practice, a structural transformation of current relations of production and life.

Recognizing that ecosocialism has for years worked to build manifestos and programs defining this strategy, we discussed the next steps and concluded that there can be no ecosocialism without free territories. We are certain that eco-territorial struggles and the construction of a livable world are the path we must follow, strengthening our initiatives in solidarity, and creating spaces where we can advance the construction of ecosocialism from and for the peoples.

To reach this goal, it is necessary to accumulate victories that show us the way. Carrying out mobilizations and campaigns among the different collectives engaged in building this ecosocialist project is essential to consolidate an integrated and internationalist process of coordinated resistance and shared strategy.

The continuation of this struggle and the construction of the ecosocialist program we need, along with the internationalization of the ecosocialist movement, are tasks we began ten years ago in these gatherings, and which were consolidated with the formation of the Internationalist Network of Ecosocialist Encounters in 2024, following the meeting in Buenos Aires.

Among new initiatives, we announce the Seventh Internationalist Ecosocialist Gathering, to be held in Belgium in May 2026; the International Ecosocialist Seminar, to be held in Brazil as part of the First International Anti-Fascist Conference; and the Third Latin American and Caribbean Ecosocialist Gathering, in 2027, in Colombia. We are convinced that these gatherings must transcend borders and generate common actions of struggle capable of striking simultaneously at the concentrated powers of capitalist extractivism in each territory where we are present.

However, Ecosocialist Gatherings alone are not enough to advance the construction of a program truly rooted in concrete struggles. For this reason, we propose the creation of joint actions and campaigns on Palestine, fossil fuels, mining, debt, and free trade agreements; the defense of water; the struggle against agribusiness; and forest restoration. We also propose mapping which companies are aligned with ecocidal projects in Latin American and Caribbean countries, in order to issue joint denunciations and communiqués. Additionally, we propose organizing territorial ecosocialist meetings prior to the one in Colombia, so that the debates reflect eco-territorialized formulations and proposals.

Finally, we want our space of construction to be living and diverse, capable of generating deep debates among its collectives, in order to think about and question our understanding of ecosocialism — reaffirming that ecosocialism is not a green-tinted socialism, but a proposal for a profound transformation of our relationships, both among ourselves and with nature. It is another way of doing politics, capable of building a new world, dignified and beautiful to live in, for human beings and all other living beings.

LINKS - International Journal of Socialist Renewal is a journal for a post-Cold War left. It is a journal that rejects the Stalinist distortion of the socialist project; takes into account ecological questions; is committed to taking steps to bring together the forces for socialism in the world today; a journal that aspires to unite Marxists from different political traditions because it discusses openly and constructively.

‘Everyone will miss the socialising – but it’s also a relief’: five young teens on Australia’s social media ban

Guardian
www.theguardian.com
2025-12-06 14:00:55
As the under-16s social media ban looms, Guardian Australia speaks to five 13 to 15-year-olds about what they will miss, and what government should be doing instead Australia’s world-first social media ban for under-16s will begin in just a few days. Malaysia, Denmark and Norway are to follow suit a...
Original Article

A ustralia’s world-first social media ban for under-16s will begin in just a few days. Malaysia, Denmark and Norway are to follow suit and the European Union last week passed a resolution to adopt similar restrictions. As the world watches on, millions of Australian adolescents and their parents are wondering just what will actually change come 10 December.

Concerns around the negative impact social media use can have on the wellbeing of young people have been around since the quaint days of Myspace – long before those to be affected by the ban were even born.

Supporters of the social media “delay” policy believe that restricting under-16s access will reduce mental health risks, exposure to harmful content and the social malaise associated with being chronically online. Those opposed worry that the legislation is ill-conceived, will push children into murkier corners of the web, impinge on their human rights, exacerbate some mental health conditions or simply prove futile.

As the world waits to see just how this grand experiment will play out, the Guardian spoke to teenagers about their feelings as the ban approaches. All five are on the youth advisory board of the Australian Theatre for Young People, which will next month host a production of The Censor, which canvasses issues of teen social media use, literacy and adult censorship of their lives.

From indifference to frustration or simply being confounded by the choices being made by the adults in the room, the perennial teen question remains: “What’s the big deal?”

‘It feels deeply unfair the government is punishing an entire age group ’

Sarai Ades says ‘removing social media removes a major avenue for self-discovery and belonging’.
Sarai Ades says ‘removing social media removes a major avenue for self-discovery and belonging’. Photograph: Joel Pratley/The Guardian

Sarai Ades, 14

I’ve been using Snapchat, Facebook Messenger as well as Instagram and TikTok since I got a phone in 2023.

I find the “delay” very frustrating because it targets young people instead of addressing the issues at the root of online harms; things like algorithmic amplification of bigotry, the spread of misinformation and the lack of accountability for the companies and creators who profit from harmful or discriminatory content.

Teens are not the primary drivers of harmful content. Much of it comes from high-influence adult creators, political commentators and extremist groups, and it feels deeply unfair that the government is punishing an entire age group by removing access rather than regulating platforms.

I think media literacy is very important, especially for our generation. We need to be taught the right methods to deal with the online world in ways that don’t become harmful – and that includes the bigotry that is often disguised as “opinions” on social media. Eliminating [access] without doing anything about building our media literacy is the equivalent of the government banning books until we are 16 and expecting us to magically start reading critically.

Right now, media literacy doesn’t figure highly in our syllabus, but it should. It’s taught in a very boring way and it isn’t up to date; we might learn about cyberbullying but not echo chambers, for example. And with the rise of AI, there is a whole new world to understand that could harm us, but we aren’t taught anything about that. I think the government could definitely be focusing its efforts there and on specific aspects of social media, rather than just establishing a whole ban.

eSafety commissioner questioned on Roblox and social media ban after Guardian investigation – video

Having less exposure to this kind of harmful content might be good for our mental health in some ways, but I think the negatives outweigh the positives.

I grew up on another continent and social media is a primary way that I feel connected with these childhood friends. Australia promotes itself as a multicultural, globally connected country, but this ban will cut teens off from their cultural communities and international friendships. And for multicultural teens that loss is significant, especially living in an Anglo-dominant country like Australia.

My generation uses social media as a tool for identity formation; we explore other cultures, our creative expression, neurodivergence, gender identity, political beliefs. Removing social media removes a major avenue for self-discovery and belonging.

My daily screen time average on socials is about two hours, shared between Pinterest, Instagram and TikTok equally. Developing a healthy relationship with your screen time has become quite trendy. Me and most of my friends already use apps that help us manage our time online, we all use “do not disturb” and only check our phones when we have time.

Whatever time I do gain will be used to invest in my goals, but the irony is that I probably wouldn’t have half those goals without social media. The ban doesn’t just take away an app, it takes away inspiration and opportunities. For kids like myself in the performing arts, taking away our ability to share our skills online also limits our exposure and could put us behind on the world stage.

Even if they aren’t trying to develop a presence though, the ban will drive teens to less savoury corners of the internet – shady apps, private browsers or unsafe websites, instead of regulated platforms. Cutting off access to socials won’t eliminate the need for online connection, it will just drive it underground.

‘I don’t think I’ll be affected that much’

Pia Monti says she has had ‘a really positive experience on the social media I use, so I don’t see why it should be banned for everyone’.
Pia Monti says she has had ‘a really positive experience on the social media I use, so I don’t see why it should be banned for everyone’. Photograph: Joel Pratley/The Guardian

Pia Monte, 13

I only have accounts on WhatsApp and Pinterest so I won’t be directly affected by the social media ban, but I still don’t like it. I know people whose main support networks are on social media, so if that’s being taken away they will be left with nothing. I’ve had a really positive experience on the social media I use, so I don’t see why it should be banned for everyone.

I don’t keep my phone in my room overnight, I just check it quickly in the morning for notifications, but I don’t really use it before school. I use it mostly to take a break from study, my daily screentime average this week was 49 minutes.

I mostly use social media to look at cooking videos on YouTube or Pinterest; I can still use YouTube for that without an account and Pinterest isn’t being banned. Neither is WhatsApp and that’s what I use mostly to keep up with friends – I don’t know that many people my age who spend lots of time on the apps that are being banned anyway – so I don’t think I’ll be affected that much. I might get accounts for things like Instagram when I’m older, but there’s no rush.

skip past newsletter promotion

‘I don’t think two years is a long time to wait to use it again’

Grace Guo says she is ‘quite indifferent to the ban’. She says ‘it won’t affect me that much because I mostly use messages and WhatsApp to chat with friends’.
Grace Guo says she is ‘quite indifferent to the ban’. She says ‘it won’t affect me that much because I mostly use messages and WhatsApp to chat with friends’. Photograph: Joel Pratley/The Guardian

Grace Guo, 14

I’m quite indifferent to the ban, it won’t affect me that much because I mostly use messages and WhatsApp to chat with friends. I do post on YouTube sometimes, but I won’t care if I can’t. Sometimes the comments aren’t very nice so I won’t miss those. And anyway, you can still access the content without an account. When I’m using YouTube to research new skills or school work, sometimes irrelevant or inappropriate things come up, but that doesn’t change if you have an account or not.

I’ve only been using Instagram since the start of this year, it’s just another way to communicate with friends and I don’t think two years is a long time to wait to use it again. It’s not a big deal to me, I’ll just use other messaging apps more.

I only looked at my phone to check the time this morning. I first checked it for messages after school on my way home on the train. My screen time average is about an hour or two a day; most of that is chatting with friends across all the apps. I feel like our generation is quite dependent on it and I have friends who use their phones lots more than me.

Overall, for kids my age, I think things will be pretty much the same – they will get around it if they want, use the banned apps without accounts or just wait. Maybe for younger kids the ban will delay some harmful experiences. But also it might be a lot to deal with at 16 if you haven’t been exposed before.

‘At the moment I only use YouTube and Discord’: Ewan Buchanan-Constable
‘ At the moment I only use YouTube and Discord’: Ewan Buchanan-Constable. Photograph: Joel Pratley/The Guardian

Ewan Buchanan-Constable, 15

I’m a bit disappointed with the ban because I was introduced to a lot of my current creative interests through YouTube. At the moment I only use YouTube and Discord. I have a couple of videos up, but it’s not a regular thing. I use Discord to talk to friends and to engage with communities, mostly about movies and video games. Most of my friends use the apps that are being banned but I don’t feel excluded because the content that’s on there doesn’t really interest me.

I use my phone as an alarm but I don’t start using it in bed. Once I’ve eaten and showered I might put something on YouTube in the background while I get ready, but it’s not my main focus. My daily average is about two hours a day. On the weekends though, my screen time goes up to more like five hours because I often have podcasts, music or videos playing on YouTube while I study or do chores.

Adults are viewing social media as something that hijacks our whole life, but really it’s just something we use in downtime and it doesn’t stop us from reaching out to each other and arranging to leave the house and hang out or anything like that. I also read, write and draw a lot. I play Dungeons and Dragons and video games.

The way I look at it is that the goals the government has about mental health and limiting the harm of inappropriate content could be achieved through regulation and education rather than a ban.

Making that education start earlier would be important, being able to navigate online spaces safely is something kids need to know about much younger now. And I think it might be dangerous to forget that and just let them loose at 16. They’re going to encounter lots of weird stuff online before then anyway, so they should be taught about it earlier.

I’m not sure if I will ever start using social media, unless it’s for my career. I want to be an actor so for me it would be a professional tool rather than a source of entertainment.

Emma Williams says the government ‘could spend their money better on educating rather than restricting’.
Emma Williams says the government ‘could spend their money better on educating rather than restricting’. Photograph: Joel Pratley/The Guardian

Emma Williamson, 15

I turn 16 in February, so it will just be the holidays without Instagram for me really. I opened my account a couple of years ago and then I changed schools so it has been a good way to stay in touch with those people I don’t see every day any more.

I think everyone will miss the socialising part. But it’s also a relief to not have to do that on a platform designed to lure you in and waste your time, no one is going to miss scrolling. I get bored of watching reels quickly anyway, and AI slop is the worst, it takes so long to keep blocking that content and more keeps coming. I only want to see real content about my interests, like old audition videos of actors.

My daily average for Instagram is about half an hour. I use an app called Fitlock that gives me access to certain apps I choose based on how many steps I do. It’s easy to turn off, but I try to stick to what I’ve earned. And funnily enough, most of what I’ve learned about how and why to keep social media use down has been from the apps themselves. There’s lots of content from people a bit older than me encouraging people to get off their phones and go and live their lives in the real world.

My friends and I have group chats on Instagram but also on WhatsApp, so we will just spend more time there. Saying that, I find that Instagram content can be a fun conversation starter, so maybe we won’t chat as much without that to share.

At school they teach about cyberbullying and harmful content but never about how to use the apps in healthy ways. I think the government could spend their money better on educating rather than restricting. Right now though what they teach us is really boring, they’d need to work on that.

  • The Censor will play at ATYP’s Rebel theatre as part of Sydney festival, 22 to 25 January 2026

An Asteroid Threatening Earth Is Teeming With Ingredients for Life, Scientists Discover

403 Media
www.404media.co
2025-12-06 14:00:38
Scientists found sugars that are essential for life on asteroid Bennu, which has a 1 in 2,700 chance of hitting Earth in 2182....
Original Article

Welcome back to the Abstract! Here are the studies this week that fought for their food, took one for the team, passed the extraterrestrial sugar, and got lost in an ancient haze.

First, a story about the spiciest meatball in the animal kingdom. Then: ants are being interesting again, a new discovery about an old rock, and a walk in an ancient sulfur rainstorm.

As always, for more of my work, check out my book First Contact: The Story of Our Obsession with Aliens or subscribe to my personal newsletter the BeX Files .

Pond frog versus murder hornet

Sugiura, Shinji. “Pond frog as a predator of hornet workers: High tolerance to venomous stings.” Ecosphere.

Most animals don’t eat hornets, because dinner is just not as fun if it comes with a side of deadly venom and stab wounds. But a scientist has now observed an incredible exception to the rule with the humble black-spotted pond frog ( Pelophylax nigromaculatus ), which will straight-up house a hornet and ask for seconds.

Hornets have occasionally been found in the bellies of pond frogs, suggesting that the amphibians can tolerate their intense stings, but not much else is known about this unusual predator-prey relationship. To remedy the situation, Shinji Sugiura of Kobe University went out to the prefecture of Hyogo in Central Japan and netted a bunch of hornets from grasslands and forests—including the infamous “murder hornet” Vespa mandarinia , the largest in the world. He then captured pond frogs from wetlands with paddy fields and ponds in Hyogo and Shimane prefectures. Then, he let them duke it out in the lab in the world’s gnarliest series of cage matches.

“When a frog opened its mouth and its tongue made contact with a hornet, the action was classified as an attack on the hornet,” Sugiura said in the study. “If the frog did not stop the attack, spit out, or regurgitate the hornet, it was considered to have successfully consumed the hornet.”

The results revealed that most frogs made short work of the hornets (Videos S2 ) even though their meals were actively stinging them in their faces, eyes, tongues, palates, or throats of the frogs during attacks (Figure 3c,d ).

“None of the frogs regurgitated the hornets after swallowing them,” Sugiura noted. “All frogs that swallowed hornets excreted the undigested body parts of the hornets as feces 2–4 days after ingestion.”

Lets just sit with that mental image of poopy undigested hornets for a second. What a nightmare. But what’s truly wild about this study is that the insects are known to inject lethal doses of venom into much larger animals, like mice, so the frogs clearly have some unknown defense against their attacks.

“Although many frogs were stung repeatedly by [hornets] in this study…none of the frogs died, and all individuals resumed normal behavior shortly after being stung,” Suguira said. “Moreover, despite repeated stings, most of the frogs ultimately consumed hornet workers…indicating a high level of predation success even against the largest hornet species.”

We humans are so lucky that when we sit down to dinner, our food generally does not try to kill us with repeated venomous needlepoint impalements. Count your blessings!

In other news…

Meet the ant-y Christs

Dawson, Erika H. “Altruistic disease signalling in ant colonies.” Nature Communications.

We’ll move now from death by frog munchies to death by team spirit. Scientists have discovered that ant pupae (baby ants) will sacrifice themselves if they are sick, lest they risk the health of the entire colony.

“Here we show…that sick ant pupae instead actively emit a chemical signal that in itself is sufficient to trigger their own destruction by colony members,” said researchers led by Erika H. Dawson of the Institute of Science and Technology Austria. “Our data suggest the evolution of a finely-tuned signalling system…that triggers pupal signalling for sacrifice. This demonstrates a balanced interplay between individual and social immunity that efficiently achieves whole-colony health.”

In other words, if an ant gets bitten by a zombie in a movie, it would immediately let everyone know and offer its life for the good of the group. Do what you will with this information.

Do you take sugar in your asteroid?

Furukawa, Yoshihiro et al. “Bio-essential sugars in samples from asteroid Bennu.” Nature Geoscience.

Scientists have found bio-essential sugars, including ribose and glucose, in samples of an asteroid called Bennu that were brought to Earth by NASA’s OSIRIS-REx mission in 2023 . The discovery marks the first time key sugars have been found in any extraterrestrial sample. Ribose is an essential ingredient of RNA (ribonucleic acid), making it a particularly significant find in the quest to understand how life arose on Earth, and if it exists elsewhere.

https://youtu.be/9LyH6jTefU8?si=CU3rVmPEzTbPf5oK

“All five of the canonical nucleobases in DNA and RNA, and phosphate, were previously found in Bennu samples,” said researchers led by Yoshihiro Furukawa of Tohoku University. “Our detection of ribose means that all the components of RNA are present in Bennu.”

“Our confident detection in Bennu of abundant glucose—the hexose molecule that is life’s common energy source—and other hexoses indicates that they were present in the early solar system,” the team added. “Thus, all three crucial building blocks of life”— bio-essential sugars, nucleobases, and protein-building amino acids—”would have reached the prebiotic Earth and other potentially habitable planets.”

While Bennu bears the stuff of life, it may also be an omen of death: It has a 1 in 2,700 chance of hitting Earth on September 24, 2182. These are very low odds, but the risk is high enough to classify Bennu as potentially hazardous. So while visions of sugar plums may dance in your head this season, beware the nightmares about sugar-asteroids.

It’s raining sulfur—hallelujah!

Reed, Nathan W. “An Archean atmosphere rich in sulfur biomolecules.” Proceedings of the National Academy of Sciences.

I’ve made you walk through many valleys of death in this newsletter, but we’ll close with some unadulterated life. Scientists have discovered that many of the sulfur molecules that help make up all modern organisms may have rained down from the hazy skies of the Archean period four billion years ago.

Assuming the results are confirmed in future research, it would mean that these sulfur molecules could have predated life, upending a leading hypothesis that they were a product of life and thus emerged later.

The work challenges “the assumption that life must have ‘invented’ sulfur biomolecules during evolution…by demonstrating the production of a variety of sulfur biomolecules, including cysteine, in laboratory experiments mimicking the atmospheric chemistry of the early Earth,” said researchers led by Nathan Reed of NASA, who conducted the work while at the University of Colorado, Boulder.

“The results presented here imply that an atmospheric organic haze is a potential powerhouse in providing a diversity of essential biomolecules in sufficient quantities for a budding global biosphere,” the team concluded.

Taken together with the Bennu study, it looks as if early Earth was positively marinating in life juices from multiple sources, including the sky and extraterrestrial impactors. Though this still doesn’t explain how living things sprang up from the prebiotic stew, it provides further confirmation that the ingredients of life as we know it are spread far and wide here in our solar system, and beyond.

Thanks for reading! See you next week.

GrapheneOS is the only Android OS providing full security patches

Hacker News
grapheneos.social
2025-12-06 13:58:31
Comments...
Original Article

How I discovered a hidden microphone on a Chinese NanoKVM

Hacker News
telefoncek.si
2025-12-06 13:54:59
Comments...
Original Article

NanoKVM is a hardware KVM switch developed by the Chinese company Sipeed. Released last year, it enables remote control of a computer or server using a virtual keyboard, mouse, and monitor. Thanks to its compact size and low price, it quickly gained attention online, especially when the company promised to release its code as open-source. However, as we’ll see, the device has some serious security issues. But first, let’s start with the basics.

How Does the Device Work?

As mentioned, NanoKVM is a KVM switch designed for remotely controlling and managing computers or servers. It features an HDMI port, three USB-C ports, an Ethernet port for network connectivity, and a special serial interface. The package also includes a small accessory for managing the power of an external computer.

Using it is quite simple. First, you connect the device to the internet via an Ethernet cable. Once online, you can access it through a standard web browser (though JavaScript JIT must be enabled). The device supports Tailscale VPN, but with some effort (read: hacking), it can also be configured to work with your own VPN, such as WireGuard or OpenVPN server. Once set up, you can control it from anywhere in the world via your browser.

NanoKVM

NanoKVM

The device could be connected to the target computer using an HDMI cable, capturing the video output that would normally be displayed on a monitor. This allows you to view the computer’s screen directly in your browser, essentially acting as a virtual monitor.

Through the USB connection, NanoKVM can also emulate a keyboard, mouse, CD-ROM, USB drive, and even a USB network adapter. This means you can remotely control the computer as if you were physically sitting in front of it - but all through a web interface.

While it functions similarly to remote management tools like RDP or VNC, it has one key difference: there’s no need to install any software on the target computer. Simply plug in the device, and you’re ready to manage it remotely. NanoKVM even allows you to enter the BIOS, and with the additional accessory for power management, you can remotely turn the computer on, off, or reset it.

This makes it incredibly useful - you can power on a machine, access the BIOS, change settings, mount a virtual bootable CD, and install an operating system from scratch, just as if you were physically there. Even if the computer is on the other side of the world.

NanoKVM is also quite affordable. The fully-featured version, which includes all ports, a built-in mini screen, and a case, costs just over €60, while the stripped-down version is around €30. By comparison, a similar RaspberryPi-based device, PiKVM, costs around €400. However, PiKVM is significantly more powerful and reliable and, with a KVM splitter, can manage multiple devices simultaneously.

As mentioned earlier, the announcement of the device caused quite a stir online - not just because of its low price, but also due to its compact size and minimal power consumption. In fact, it can be powered directly from the target computer via a USB cable, which it also uses to simulate a keyboard, mouse, and other USB devices. So you have only one USB cable - in one direction it powers NanoKVM, on the other it helps it to simulate keyboard mouse and other devices on a computer you want to manage.

The device is built on the open-source RISC-V processor architecture, and the manufacturer eventually did release the device’s software under an open-source license at the end of last year. (To be fair, one part of the code remains closed, but the community has already found a suitable open-source replacement, and the manufacturer has promised to open this portion soon.)

However, the real issue is security.

Understandably, the company was eager to release the device as soon as possible. In fact, an early version had a minor hardware design flaw - due to an incorrect circuit cable, the device sometimes failed to detect incoming HDMI signals. As a result, the company recalled and replaced all affected units free of charge. Software development also progressed rapidly, but in such cases, the primary focus is typically on getting basic functionality working, with security taking a backseat.

So, it’s not surprising that the developers made some serious missteps - rushed development often leads to stupid mistakes. But some of the security flaws I discovered in my quick (and by no means exhaustive) review are genuinely concerning.

One of the first security analysis revealed numerous vulnerabilities - and some rather bizarre discoveries. For instance, a security researcher even found an image of a cat embedded in the firmware. While the Sipeed developers acknowledged these issues and relatively quickly fixed at least some of them, many remain unresolved.

NanoKVM

NanoKVM

After purchasing the device myself, I ran a quick security audit and found several alarming flaws. The device initially came with a default password, and SSH access was enabled using this preset password. I reported this to the manufacturer, and to their credit, they fixed it relatively quickly. However, many other issues persist.

The user interface is riddled with security flaws - there’s no CSRF protection, no way to invalidate sessions, and more. Worse yet, the encryption key used for password protection (when logging in via a browser) is hardcoded and identical across all devices. This is a major security oversight, as it allows an attacker to easily decrypt passwords. More problematic, this needed to be explained to the developers. Multiple times.

Another concern is the device’s reliance on Chinese DNS servers. And configuring your own (custom) DNS settings is quite complicated. Additionally, the device communicates with Sipeed’s servers in China - downloading not only updates but also the closed-source component mentioned earlier. For this closed source component it needs to verify an identification key, which is stored on the device in plain text. Alarmingly, the device does not verify the integrity of software updates, includes a strange version of the WireGuard VPN application (which does not work on some networks), and runs a heavily stripped-down version of Linux that lacks systemd and apt . And these are just a few of the issues.

Were these problems simply oversights? Possibly. But what additionally raised red flags was the presence of tcpdump and aircrack - tools commonly used for network packet analysis and wireless security testing. While these are useful for debugging and development, they are also hacking tools that can be dangerously exploited. I can understand why developers might use them during testing, but they have absolutely no place on a production version of the device.

A Hidden Microphone

And then I discovered something even more alarming - a tiny built-in microphone that isn’t clearly mentioned in the official documentation . It’s a miniature SMD component, measuring just 2 x 1 mm, yet capable of recording surprisingly high-quality audio.

What’s even more concerning is that all the necessary recording tools are already installed on the device! By simply connecting via SSH (remember, the device initially used default passwords!), I was able to start recording audio using the amixer and arecord tools. Once recorded, the audio file could be easily copied to another computer. With a little extra effort, it would even be possible to stream the audio over a network, allowing an attacker to eavesdrop in real time.

Hidden Microphone in NanoKVM

Hidden Microphone in NanoKVM

Physically removing the microphone is possible, but it’s not exactly straightforward. As seen in the image, disassembling the device is tricky, and due to the microphone’s tiny size, you’d need a microscope or magnifying glass to properly desolder it.

To summarize : the device is riddled with security flaws, originally shipped with default passwords, communicates with servers in China, comes preinstalled with hacking tools, and even includes a built-in microphone - fully equipped for recording audio - without clear mention of it in the documentation. Could it get any worse?

I am pretty sure these issues stem from extreme negligence and rushed development rather than malicious intent. However, that doesn’t make them any less concerning.

That said, these findings don’t mean the device is entirely unusable.

Since the device is open-source, it’s entirely possible to install custom software on it. In fact, one user has already begun porting his own Linux distribution - starting with Debian and later switching to Ubuntu. With a bit of luck, this work could soon lead to official Ubuntu Linux support for the device.

This custom Linux version already runs the manufacturer’s modified KVM code, and within a few months, we’ll likely have a fully independent and significantly more secure software alternative. The only minor inconvenience is that installing it requires physically opening the device, removing the built-in SD card, and flashing the new software onto it. However, in reality, this process isn’t too complicated.

And while you’re at it, you might also want to remove the microphone… or, if you prefer, connect a speaker. In my test, I used an 8-ohm, 0.5W speaker, which produced surprisingly good sound - essentially turning the NanoKVM into a tiny music player. Actually, the idea is not so bad, because PiKVM also included 2-way audio support for their devices end of last year .

Basic board with speaker

Basic board with speaker

Final Thoughts

All this of course raises an interesting question: How many similar devices with hidden functionalities might be lurking in your home, just waiting to be discovered? And not just those of Chinese origin. Are you absolutely sure none of them have built-in miniature microphones or cameras?

You can start with your iPhone - last year Apple has agreed to pay $95 million to settle a lawsuit alleging that its voice assistant Siri recorded private conversations . They shared the data with third parties and used them for targeted ads. “Unintentionally”, of course! Yes, that Apple, that cares about your privacy so much.

And Google is doing the same. They are facing a similar lawsuit over their voice assistant, but the litigation likely won’t be settled until this fall. So no, small Chinese startup companies are not the only problem. And if you are worried about Chinese companies obligations towards Chinese government, let’s not forget that U.S. companies also have obligations to cooperate with U.S. government. While Apple is publicly claiming they do not cooperate with FBI and other U. S. agencies (because thy care about your privacy so much), some media revealed that Apple was holding a series secretive Global Police Summit at its Cupertino headquarters where they taught police how to use their products for surveillance and policing work . And as one of the police officers pointed out - he has “ never been part of an engagement that was so collaborative .”. Yep.

P.S. How to Record Audio on NanoKVM

If you want to test the built-in microphone yourself, simply connect to the device via SSH and run the following two commands:

  • amixer -Dhw:0 cset name='ADC Capture Volume 20' ( this sets microphone sensitivity to high )
  • arecord -Dhw:0,0 -d 3 -r 48000 -f S16_LE -t wav test.wav & > /dev/null & ( this will capture the sound to a file named test.wav )

Now, speak or sing (perhaps the Chinese national anthem?) near the device, then press Ctrl + C , copy the test.wav file to your computer, and listen to the recording.

Free Buses Can Be a Reality — Just Look at Maryland

Portside
portside.org
2025-12-06 13:28:10
Free Buses Can Be a Reality — Just Look at Maryland Kurt Stand Sat, 12/06/2025 - 08:28 ...
Original Article

During the COVID-19 pandemic and its associated economic downturn, many people couldn’t pay their transportation costs, and often didn’t. In New York City in 2021, some 21 percent of bus riders did not pay the fare, a figure that grew to 48 percent in 2024 . Some local governments, including New York City, responded with reduced or free fare programs. From 2023 to 2024, New York’s Metropolitan Transit Authority (MTA) ran a zero-fare bus pilot that served around 43,000 riders . The pilot , championed by then-assembly member Zohran Mamdani, offered free trips on one bus in each borough.

To expand this small pilot to universal zero-fare buses throughout New York City is a tall task, with a total 2024 bus ridership of 409 million and 6,300 buses . As mayor-elect Mamdani and his administration look to grow zero-fare buses in New York, they have a stellar example just a few hours south of New York, in Maryland.

The largest free bus program in Maryland by ridership is in Montgomery County, a suburb north of Washington, D.C.

Montgomery County first made its “Ride On” buses free to all riders under 18 in 2019 . Then on June 29, 2025, it made all of its buses fare-free for all passengers. The system has a fleet of nearly 400 buses, 80 routes, and provided 19.2 million rides in the 2025 fiscal year. In the three months since free fares were instituted, ridership has increased by 5.4 percent . Phil McLaughlin, General Manager of Transit Services for Montgomery County Department of Transportation (MCDOT), said an estimated 1 percent to 2 percent of that ridership increase came from instituting zero fares.

One reason Montgomery Country’s City Council adopted the zero-fare program was financial. Montgomery County reduced fares during the pandemic from $2 to $1 and saw lower fare revenues (dropping from $10 million to roughly $1.6 million ) overall. When faced with the need to upgrade its fare collection systems to tap-to-pay in order to align its fare boxes with Washington D.C.’s Metro system, it became clear that replacing the fare boxes would cost more than they would recoup: The cost was estimated at $22 million and would take approximately eight years to begin turning a profit . In addition, it cost the county $557,000 annually to collect fares. Montgomery County was faced with either paying police to enforce fares or going fare-free. County Executive Marc Elrich proposed a fully fare-free bus system in his 2026 fiscal year proposed operating budget , and the county council adopted it soon after.

Montgomery County Councilmember Evan Glass, chair of the Council’s Transportation & Environment Committee, told Truthout that fare-free buses are “bringing real impacts for Montgomery County residents” and the increased ridership shows that “cost barriers make a real difference. In a county with a median income of $115,000, compared to $35,000 for the average bus rider, fare-free transit is fundamentally about equity.”

The oldest free bus program in Maryland is likely Baltimore’s Charm City Circulator (CCC), first established in 2010 by then-mayor Sheila Dixon and funded through state transportation department grants. In 2024, the CCC provided 1.43 million rides, according to the Baltimore City Department of Transportation. The CCC has over 100 stops along five routes , with buses arriving every 13-20 minutes, and operates on an $8 million budget. Its impact has been overwhelmingly positive: a 2022 rider survey showed that over 40 percent of people who use the CCC would have used single-occupancy vehicles to make their trips, if not for this free service, thus reducing overall traffic congestion and vehicle emissions.

These programs, while crucial, still need to be built upon. Previously, local residents have pointed out that the bus service has a history of disproportionately serving majority-white neighborhoods and tourist mainstays , and has connected primarily to the city’s largest predominantly white institutions such as Johns Hopkins and the University of Maryland, while not offering the same level of service to majority-Black neighborhoods or Baltimore’s historically Black colleges and universities, such as Morgan State University or Coppin State. More public investment in the city-wide transit programs can help bridge the gap. In one promising improvement, Baltimore mayor Brandon Scott recently announced an expansion of the CCC to three East Baltimore neighborhoods beginning on December 7. The Baltimore City Department of Transportation told Truthout that the expansion is aimed at “improving equity of city services” and that it will add fast, free transit options to “communities that have relatively low vehicle ownership.”

Dr. Lawrence Brown, a research scientist in the Center for Urban Health Equity at Morgan State University, told Truthout that while “there is still work to be done,” the changes to the CCC since 2020 — including the Cherry Hill line addition in 2024 (which serves a neighborhood that is 90 percent Black and historically redlined) — “represents a change in the right direction.” Dr. Brown elaborated that “The Green line changes are definitely an improvement from an equity perspective as it extends into historically redlined central East Baltimore,” but “the Brooklyn/Curtis Bay community is still being left out although it also has tremendous transit needs.”

In an interview, Minister Glenn Isaac Smith, co-founder and president of the Baltimore Transit Equity Coalition, told Truthout , “We had long advocated for the extension of the circulator” including its expansion into Cherry Hill, and called the November changes “a welcome addition.” He continued, “The East Baltimore extension will service people who need transportation and a lot of times cannot afford it, especially if they are on a fixed income and need to travel to doctor’s appointments, or need to travel to buy groceries since it’s a food desert.”

West of Baltimore, the more rural but quickly-growing Frederick County made all of its mass transit free at the start of the COVID-19 pandemic and never went back. Transit Services of Frederick County, which includes buses and a paratransit service called Transit Plus, served a little shy of 1 million riders in fiscal year 2025 [928,650 riders], a record ridership for the county (up from over 540,000 trips in FY2022). It operates on a $9.3 million budget and has 15 routes and a fleet of 48 buses, four of which are all-electric and powered by a solar array at the Frederick County landfill, according to an emailed statement from Mary Dennis, Communications Manager for Transit Services of Frederick County. Its buses also serve many of the local food banks in Frederick, a fact advertised on Frederick Transit’s homepage .

Frederick resident and Transit Plus rider Jose Gabriel Coronado Flores, who uses a wheelchair, told Truthout that “I’m glad price is no longer a concern.” Coronado Flores pointed out that Transit Plus’ scheduled hours interfere with civic participation for the elderly and disabled people who rely on it, as many Frederick County government meetings happen later in the evening, when Transit Plus no longer schedules trips. This “keeps out many disabled people from participating in the processes which have to do with them,” Coronado Flores said.

Christian Benford, a candidate for Frederick County Council, told Truthout , “It is fantastic to have a program that provides alternative travel, reduces carbon emissions, and is at no-cost to residents.” Benford also pointed out improvements could be made to accessibility in the county overall, adding, “Frederick, as a whole, is not very walkable. So, while there are stops at key locations, it still requires crossing dangerous intersections or moving on uneven or poorly maintained public infrastructure.”

Like New York, Maryland faced reduced fare collections on buses in the state during the pandemic. Rather than increasing enforcement, Montgomery County and Frederick County, implemented temporary reduced or zero-fare measures. Seeing the financial and public benefits, both counties went fully zero-fare permanently. Zero-fare buses address a key equity gap noted in a 2024 report by Montgomery County called “ Ride On Reimagined ,” which found that 27 percent of Montgomery County residents in Maryland make less than $20,000 a year, showing the need for free public transit to connect residents to jobs and education. Maryland’s zero-fare bus programs provide an important model to the incoming Mamdani administration and others who want increased equity in their cities.

More can and should be done in Maryland, as these advancements in buses are coming amid a $ 2.1 billion statewide shortfall in public transit funding, leading to the elimination of many new road and transit projects. In Baltimore, a light rail extension called the Red Line was cancelled in 2015, and “set transportation in Baltimore back ten years,” said Smith. Maryland Gov. Wes Moore has made getting the Red Line built one of his platform promises , and Smith said, “We look forward to further expansions into the east-west corridor” of Baltimore.

Like in New York, there is much coordination that needs to take place between state, city, and county governing bodies to make public transit deliver better for all its residents. But Benford, at least, sees opportunity in what’s left to be done: “This is a great instance to foster better partnership between the county and city councils to engage with constituents. We may have different governing bodies but should always look for meaningful ways to work together.”

Alexis Goldstein is a former Wall Street professional who now works on financial policy. She is a proud rank-and-file member of the National Treasury Employees Union, Chapter 335.

Truthout is a nonprofit news organization dedicated to providing independent reporting and commentary on a diverse range of social justice issues. Since our founding in 2001, we have anchored our work in principles of accuracy, transparency, and independence from the influence of corporate and political forces. Truthout is an indispensable resource for activists, movement leaders and workers everywhere. Please make this work possible with a quick donation .

Why Speed Matters

Hacker News
lemire.me
2025-12-06 12:46:42
Comments...
Original Article

The one constant that I have observed in my professional life is that people underestimate the need to move fast.

Of course, doing good work takes time. I once spent six months writing a URL parser. But the fact that it took so long is not a feature, it is not a positive, it is a negative.

If everything is slow-moving around you, it is likely not going to be good. To fully make use of your brain, you need to move as close as possible to the speed of your thought.

If I give you two PhD students, one who completed their thesis in two years and one who took eight years… you can be almost certain that the two-year thesis will be much better.

Moving fast does not mean that you complete your projects quickly. Projects have many parts, and getting everything right may take a long time.

Nevertheless, you should move as fast as you can.

For multiple reasons:

1. A common mistake is to spend a lot of time—too much time—on a component of your project that does not matter. I once spent a lot of time building a podcast-like version of a course… only to find out later that students had no interest in the podcast format.

2. You learn by making mistakes. The faster you make mistakes, the faster you learn.

3. Your work degrades, becomes less relevant with time. And if you work slowly, you will be more likely to stick with your slightly obsolete work. You know that professor who spent seven years preparing lecture notes twenty years ago? He is not going to throw them away and start again, as that would be a new seven-year project. So he will keep teaching using aging lecture notes until he retires and someone finally updates the course.

What if you are doing open-heart surgery? Don’t you want someone who spends days preparing and who works slowly? No. You almost surely want the surgeon who does many, many open-heart surgeries. They are very likely to be the best one.

Now stop being so slow. Move!

Touching the Elephant – TPUs

Hacker News
considerthebulldog.com
2025-12-06 12:29:28
Comments...
Original Article

Understanding the Tensor Processing Unit

Reed

latency hiding

Something New

There is mythological reverence for Google’s Tensor Processing Unit. While the world presently watches NVIDIA’s gravity drag more companies into its orbit, there sits Google, imperial and singular. Lots of companies participate in the “Cambrian-style explosion of new-interesting accelerators” [14] – Groq, Amazon, and Tenstorrent come to mind – but the TPU is the original existence proof. NVIDIA should take credit for the reemergence of deep learning, but the GPU wasn’t designed with deep learning in mind. What’s strange is that the TPU isn’t a secret. This research is indebted to Google’s public chest-thumping, but the devices themselves have long been exclusive to Google’s datacenters. That is over a decade of work on a hardware system sequestered behind their walls. That the TPU is so well documented yet without a true counterpart creates a strange asymmetry. Google is well positioned in the AI race because of their decision over a decade ago to build a hardware accelerator. It is because of the TPU.

On the back of DistBelief Google had gotten neural networks running at scale. In 2013 however they realized that they would need to double their datacenter capacity to meet the growing demand for these new services. “Even if this was economically reasonable, it would still take significant time, as it would involve pouring concrete, striking arrangements for windmill farm contracts, ordering and installing lots of computers, etc.” [14] The race against the clock began, and 15 months later the TPU was born. Fast forward to April of this year when Sundar Pichai announced the 7th generation TPU, Ironwood, at Google Cloud Next. The headline figures were eye-popping. 9,216 chips in a pod, 42.5 Exaflops, 10 MW [21] . In 12 years the TPU went from a research project to a goliath rack-scale system.

Perhaps reverence is warranted. The development of the TPU is set against the backdrop of a changing hardware scaling landscape. It used to be that to get better programs you just had to wait. With each new generation of chip Moore’s Law and Dennard Scaling brought enormous tailwinds in transistor density, power efficiency, and wall clock improvements. But in the aughts and 2010s there was no more sitting and no more waiting. The advancements in chip physics were not producing exponential returns as they once had, and workload demands continued growing.

Casting this as mythology however obscures the details and risks making the TPU seem like magic. The development of the TPU is the story of trade-offs and constraints and co-design. It touches hardware, software, algorithms, systems, network topology, and everything in between. It did not happen by accident, but through the deliberate process of design and iteration. When thinking about the TPU it’s natural to ask:

How did we get here?

Slowing Down

For decades the industry relied on Moore’s Law to pack more transistors into a smaller area and on Dennard Scaling to get more energy efficiency from those transistors. This netted out to smaller, faster, and more efficient devices. You didn’t need to change your software or architecture to realize significant gains, regardless of the domain. CPU performance doubled every 1.5 years from 1985-2003, and every 2 years from 2003-2010. The doubling speed since is closer to every 20 years [14] . The AlexNet moment in 2012 charted a course to the current renaissance in neural networks. Different hardware suddenly opened the door for new questions to be asked. The range of problems that neural networks were suited to solve, along with their appetite for bigger data and bigger models, meant that this algorithmic paradigm was taking off as our scaling paradigms began to languish.

Scaling Post Moore’s Law Degradation in the reliability of chip performance scaling under different regimes [14]

The TPU falls into the broad classification of hardware accelerators, of which the marquee distinction is that it is specialized for certain computational domains, hence the name Domain Specific Accelerator. Whereas general purpose devices are designed to accommodate the maximum number of program shapes, specialized designs are defined as much by what they can do as what they can’t. They trade off generality for performance. If we can’t rely on Moore’s Law and Dennard Scaling, and there are new workloads demanding attention, the goal is to optimize for the characteristics of those workloads and to discard everything else. Specialization asks what the optimal way to spend a fixed transistor and energy budget is to squeeze out performance.

Linear algebra is ripe for specialization because a relatively small set of parallelizable operations dominate neural networks. For the TPU that meant a monastic focus on those primitives. Neural networks are simple compositions of Matrix-Vector, Matrix-Matrix, and Elementwise computations over large tensors. Consider that matrix multiplication has cubic complexity. While computationally expensive, this one class of operations is the spine for a large fraction of what is required for a neural network. This narrows the window of optimizations that need to be baked into silicon. Matrix multiplies have the property that as the size of inputs grow, the ratio of compute, O(n^3), to data access, O(n^2), improves [15] . If you can dedicate hardware to speeding up arithmetic and coordinating data movement you can exploit this, and the arithmetic properties are complemented by the runtime properties. Neural networks can be fully specified ahead of time. With clever planning a program can be entirely mapped out before an instruction is issued. There was rarely a need before to design, tape out, and deploy custom silicon. Free performance gains made the economics of simply waiting versus the cost of designing an ASIC a non-starter. The decline of hardware scaling made exploring these realities attractive.

Energy Per Operation of Common Operations Horowitz Energy per Operation [11]

This opportunity is best exploited in the power budget. Compare the relative cost of arithmetic to control, memory access, and data movement. Horowitz [11] notes that over 50% of processor die energy is dissipated in caches and register files. These inefficiencies exist to mitigate the even greater inefficiency of large memory accesses. In [12] they cite that the energy to fetch and interpret instructions is 10-4000x more expensive than to perform simple operations. Moving and accessing data costs significantly more power, and what is required of deep learning is more arithmetic per unit control. Finding ways to circumvent relative power inefficiencies with specialization means rearchitecting chips to remove that waste.

The Inference Chip

TPU Block Diagram Block diagram of TPUv1 [1]

Datacenter expansions plans are a hell of a drug. To stem the tide of models devouring datacenter capacity, the first ASIC needed to focus on inference. Inference only needs a forward pass through the neural network. A simple neural network layer might look like this:

$$ ReLU( (X \cdot W) + b ) $$

Where X and W are input data and model weights, ReLU is a non-linear activation function, and b is a bias term. A matrix multiply followed by some elementwise addition and an elementwise maximum function. Imagine that chaining a handful of these layers together forms the totality of an inference. This simplified view on early model architectures gives us the general template for designing TPUv1. Matrix multiply, some activation looking functions on that result, feed the results to storage, repeat. To meet the initial deadlines the TPU design exploited this loop-like behavior.

TPUv1 is a single-threaded co-processor connected over PCIe with a 24MiB software-controlled Unified Buffer, an 8-bit integer systolic array, and 8GiB DDR3 DRAM. The device runtime lays out tensors, plans memory transfers with a programmable DMA controller between the host and the Unified Buffer (on-chip SRAM), and tiles compute operands. The host sends 12-bit CISC instructions to the device’s instruction buffer which the in-order sequencer consumes to move data to DRAM and issue MXU ops. The datapath consumes ~2/3 of the die area of the chip [1] . Take care to notice what it is not. It is not a multi-level cache hierarchy. There is no multi-threading or branch prediction or prefetching or TLB. The systolic array executes arithmetic and the runtime eliminates control overhead. TPUv1 is a spartan device aimed at making inference fast.

The heart of the device is the Matrix Multiplication Unit (MXU). It is a 256x256, 2D weight-stationary systolic array of processing elements, in this case MACs. The MXU targets dense GEMMs to maximize arithmetic intensity. The TPU is designed to keep the MXU busy. You can find nice animated demonstrations of data moving through the systolic array here or here .

MXU Cycle Timing MXU Cycle Timing

We’ll start with a simplified 4x4 systolic array. Although there are design variations of systolic execution [18] [36] , we are concerned with the 2D weight-stationary variant. The weights are pre-loaded into the array from the right hand side (the top in this diagram), and the inputs stream in from the left hand side (conveniently on the left). Once the weights are loaded they sit resident in the MACs, one weight per MAC. As the inputs flow from left to right, the MACs compute the product of the resident weight and the streamed input each cycle. The result of that computation is passed downward to the next processing element. If a MAC has one of these partial sums, it adds it to the result of the weight/input product and passes that new sum downward. At the bottom edge of the array there are no more computations and the result is passed to a 4096 row x 256-element bank of 32-bit accumulators.

MXU Double Buffering MXU Double Buffering

Notice that weight pre-loading doesn’t happen all at once. It would waste cycles to wait for each MAC to have a resident weight before streaming in inputs. Weight pre-loading instead happens diagonally, with the left-most part of the systolic array receiving weights first. When the left column of processing elements has weights, the inputs begin streaming diagonally top to bottom. This imposes significant timing coordination for such a simple component. Much of the rest of the chips’ design can be thought of as accommodating these timing needs, and a particular instantiation of that is the liberal use of double buffering.

MXUs can perform immense amounts of arithmetic, but data movement/control stops at the edges of the systolic array. Between processing elements there is only result-passing with chains of two-input adders. If either weight or input data is not where it needs to be, stalls burn cycles that hurt MXU utilization. Spelling it out:

  • The MXU holds two 64KiB tiles of weights with one reserved for double buffering
  • Four 64KiB weight tiles act as a FIFO queue to decouple memory accesses and weight loads between DRAM and the MXU
  • The Unified Buffer stores intermediate results from the accumulators and prepares new data to feed to the systolic array
  • The bank of accumulators logically splits 4096 rows into two chunks of 2048 rows, one to feed outputs and one to drain them
Sizing the MXU The number of processing elements that touch data before it reaches the accumulators grows quadratically with the array size which affects the speed of the computation. For a 256x256 array that is 65,536 MACs vs. 262,144 MACs in the 512x512 configuration. During fill/drain you pay an O(num_edges) cost to populate the buffers. Fewer edges better amortize this overhead. As arrays shrink they are penalized by wiring constraints. They perform less compute per data access and require running many wires between components. Sizing this device is a delicate balance between compute intensity and layout constraints, which we will see again in later generations.

The runtime knows how long each operation it issues should take, so it can intelligently overlap them with one another. During matrix multiplications the UB prepares the next batch of inputs, the fixed activation units operate on the results in the accumulators, and the Weight FIFO banks more weights. Matrix multiplies are relatively long latency, which leaves lots of cycles between when work starts and when work ends. The runtime schedules memory accesses, data movement and computation deterministically to minimize stop-the-world pauses rather than make coordination dependent on the MXU. Hiding latency with overlapping improves parallelism, improves data reuse, and conserves energy otherwise wasted in control flow.

The headline figures from their paper are anachronistic by now, but they help contextualize the accomplishment of the first gen chip. 25x as many MACs and 3.5x the on-chip memory of the K80 GPU. 15-30x the inference speed and 30-80x the perf/W of the K80 and the Haswell CPU [1] . The fixed-latency, software-managed design created a hardware accelerator that eschewed prevailing designs that spent energy in cache hierarchies and control overhead. Maniacal focus on mitigating inference bottlenecks with large SRAM and coordinated data movement proved that TPUv1 worked.

The Training Chip

Neural networks need to be trained before they can be used for inference, and TPUv1 was not designed for training. Requirements include backpropagation to modify weights during execution, gradients with higher precision than int8, and support for diverse activation functions. This costs orders of magnitude more FLOPs [2] , and those FLOPs must be distributed over multiple devices while maintaining deterministic execution. TPUv1’s fixed activation units were not flexible enough for experimenting with new algorithms. The memory subsystem was not flexible enough to coordinate work between multiple devices. The UB was not flexible enough to tuck more Matrix-Vector work in behind the MXU. The whole device was too tightly coupled. Adding that flexibility, without reverting to a general-purpose processor, needed a radically different datapath.

TPUv2 Block Diagram TPUv2 Block Diagram [2]

TPUv2 was animated from the bones of TPUv1, but only the MXU feels familiar. TPUv2 is a dual-core chip. Each core pairs a scalar controller with programmable vector units, local SRAM, a 128x128 MXU, and HBM. It adds inter-core interconnects (ICI) to communicate between the memory systems of each core and across chips. Two 128x128 MXUs combine to total the same 256x256 array from TPUv1 but simplify the circuit design. Unequal logic, wire, and SRAM scaling on smaller process nodes made arithmetic improvements comparatively free, enabling the chip designers to focus on the laggard scaling axes [2] . For the second generation MXUs that meant two efficiencies over their predecessor: BrainFloat16 and wire routing.

BF16 floating point format BF16 floating point format [3]

Dynamic range matters more than precision for neural network training. Gradients represented as integers don’t produce adequate convergence behavior; you need floating point numbers to make fine-grained weight updates. Accessing higher precision numerics however means sacrificing die area. Logic circuits need more adders to handle mantissa bits. Floating point adder arrays scale as (M+1) * (M+1), where M is the size of the mantissa, – 576 adders for fp32 and 121 adders for fp16 [14] – totalling more die area and more energy spent on arithmetic. Notice that although bf16 is the same number of bits as fp16, the proportion of exponent bits to mantissa bits is higher. bf16 only requires 64 adders in the MAC circuitry, and less circuitry means more MACs in the same package and power budget [2] [14] .

MXU Sizing Considerations MXU Sizing Considerations [32]

Chip geometry considerations extend beyond individual processing elements. Big cores need long, global wires routed to/from functional units, FIFOs, and control units. Though wire diameters shrink on improved process nodes, their resistance and capacitance scale unevenly. Long wires are chunked into shorter segments connected with repeaters, but this induces signal delay making circuit timings more complex [5] . MXU configurations with multiple smaller cores shorten average wire lengths but need wires routed all over the chip. The trade off is between compute bandwidth and array utilization. Compute utilization scales down quadratically with the array area, but smaller arrays use more energy-efficient wires. Splitting the die into two cores and running fewer, shorter wires to the vector and control units balances wiring scaling with utilization.

TPU Scalar Unit TPU Scalar Unit [32]

All those wires have to lead to somewhere. To drive the new datapath, TPUv2 introduces the scalar unit. When a user submits a program, the XLA compiler performs static analysis, lowering the program into 322-bit VLIW instruction bundles. XLA schedules DMAs, vector ops, and MXU work in a deterministic stream. The complexity of organizing program control flow is absorbed by software, keeping the scalar unit relatively simple. It is single-threaded and contains 4KB of scratchpad SRAM (SMEM), small instruction memory (IMEM), and a 32 element, 32-bit register file (SReg) connected to a dual-issue ALU. Sync registers flag when arithmetic and memory blocks are busy to explicitly synchronize execution. The host sends instructions over PCIe to HBM, where they are DMA’d into the Scalar Unit’s IMEM as overlays. Scalar instruction slots execute locally, and the vector/matrix slots are decoded and dispatched to the VPU/MXU [3] . There is no dynamic runtime scheduling, just instruction fetch, decode, and forward.

Two programmable vector processing units (VPU) consolidate the fixed function blocks from TPUv1. The VPU is a 2D SIMD processor designed to increase the ratio of vector operations to matrix operations. Each VPU has 128 vector lanes with 8 sublanes. Each sublane is connected to 32 dual-issue ALUs with lane-local register files (Vregs). The VPU is backed by 16MiB on-chip Vector Memory (VMEM) that mediates data movement to the MXU with pushes/pops onto a Result FIFO [3] . Each core’s VMEM has local access to half of the chip’s HBM, and DMAs to VMEM are strided to fetch contiguous tiles of data rather than issuing many small DMAs. The VPU accesses VMEM with explicit loads/stores to Vregs which remove the need for a cache hierarchy.

The simplicity of describing the rearchitected datapath belies the complexity that the subsystems represent. Whereas general purpose devices use branch predictors, TLBs, Out of Order execution, and a bevy of techniques to shuttle data and instructions, the TPU routes around a cache-centric design with software-managed execution. The aforementioned general purpose mechanisms alleviate runtime dependencies at the expense of more hardware and more energy. Control and caches consume massive amounts of the limited energy budget, so redesigning this subsystem is the difference between an economic chip and a renegotiated contract with power providers. When you know what operations you need, the order you need them in, and the operational characteristics of the hardware, you can move control flow to compile time. The VPU and Scalar Units are co-designed to leverage this operating paradigm, moving program orchestration to software.

VLIW Instruction Bundles Sample VLIW Instructions

VLIW instructions expose this complexity. They contain slots for 2 scalar, 4 vector, 2 matrix, 1 miscellaneous, and 6 immediate instructions [3] . Slots map to scalar/vector/matrix arithmetic, loads and stores, DMAs, synchronization flags, and data literals. Though innocuously named, the miscellaneous slot controls heaven and earth. It is reserved for kernel launches, DMAs, and synchronization guards which we can think of as WAITs. Data dependencies must be carefully sequenced to ensure operation A finishes before operation B uses its results. XLA utilizes the misc slot to keep subsystems working while guarding against illegal instruction sequences. Operational latencies are known constants at compile time, and XLA can use those values to place WAIT instructions at exactly the right point in the VLIW stream to minimize stalls.

Simplified TPU Program Simplified TPU Instruction Overlay

Subsystems operate with different latencies: scalar arithmetic might take single digit cycles, vector arithmetic 10s, and matrix multiplies 100s. DMAs, VMEM loads/stores, FIFO buffer fill/drain, etc. all must be coordinated with precise timing. The MXU might be busy executing a matrix multiply for 128 cycles, meanwhile the VPU is preparing the next tile of weights for the Result FIFO. While DMAs prepare new data for VMEM a DMA_OVERLAY instruction gets inserted to fetch new instructions for IMEM. When the MXU finishes a tile, the hardware sends a signal to clear the MXU_BUSY bit in the scalar unit’s sync registers. When the scalar unit evaluates a WAIT_MXU instruction it sees that the bit is unset and hops to the next instruction for decoding. The scalar unit JUMPs to the new VLIW bundle region and the program continues. Seamlessly overlapping the work of an arbitrary DAG requires extraordinary co-design between the device and the software.

Decoupling the hardware gave software the capacity to drive massive data and instruction level parallelism. VLIW slots can launch 8 operations per cycle. That is 2048 vector ALUs and two 128x128 systolic arrays with minimal control overhead. HBM, VMEM, Vregs, and the MXU all remain busy with the same pipelining and overlap philosophy from TPUv1, only now massively scaled up. XLA wrests power away from control and back into the arithmetic units with coordinated, deterministic execution. Determinism across devices requires explicit communication between chips.

ICI forms the backbone of the training pods. It creates a coherent communication fabric that lets chips operate locally while composing into a mesh of devices acting as one large core. Two on-chip ICI links route data between the HBM and VMEM of each core. Four 496Gbit/s bidirectional off-chip links connect a TPU to its neighbors in the rack with OSFP passive copper. RDMAs over this fabric let chips treat remote HBM as explicitly addressable endpoints. Racks arrange 256 chips as a 16x16 2D torus over ICI to form the full supercomputer pod. ICI removes frequent host communication, skipping the cost of network cards, switches, and communication delays. All this sacrifices 13% of the die area for gains in distributing computations [3] .

1D Torus One dimensional torus wraparound

Let’s imagine that we’re playing a game of telephone. You and 8 friends are arranged in a 3x3 grid, and you can only communicate with your adjacent neighbors. Your goal is to send a message from the person at (0,0) to the person at (2,2) in the fewest messages. Many paths achieve this, but the shortest one is always four. Now imagine that the people on the left edge of the grid can wrap messages around to people on the right edge of the grid. This is logically like mirroring you and all your friends over that wraparound axis. These 3 new connections make our shortest path 3 instead of 4.

2D Torus Logical mirroring in two dimensional torus wraparound, adapted from [5]

ICI plays this game of telephone in two dimensions. During backpropagation and optimizer state updates intermediate values accumulate across different partitions of the model located on different chips. Results must be broadcast to all the chips participating in the computation for synchronization. Whereas on-chip work is explicitly synchronized with hardware flags, work across chips is implicitly synchronized with MPI-style collectives (All-to-All, AllReduce, etc.). Torus topologies improve communication bandwidth and increase access to different communication patterns during synchronization.

32 wraparound links at 496Gbit/s enable 15.9Tbit/s of bisection bandwidth [3] , which tells us how much data can move through the network. In a 4x4 array, a cut down the middle would sever 4 connections. That same cut down the middle of a 2D torus severs 8 connections. Even if each connection carries the same amount of data, there are more paths for data to move through which helps reduce congestion. XLA absorbs the complexity of cross-device scheduling. Software can trust that RDMAs will reach their intended stacks of HBM traveling along the ICI interface.

The same DNA ostensibly runs through TPUv1, yet the chips look and feel utterly different. The microarchitecture, software, and networking each became independently sophisticated parts of a larger system. Subsystems decoupled from one another yet still composed neatly. Where TPUv1 tightly choreographed everything, TPUv2 divided components into independent, asynchronously operating units communicating through explicit queues and synchronization points. TPUv3 was a minor revision in comparison. It has two MXUs per core, an increased clock, double the HBM capacity with 30% higher bus speeds, higher ICI bandwidth, and scales up to a 1024 node liquid-cooled rack. The dies only increased 6% relative to TPUv2 because engineers learned how to better lay the chip out [3] . Scaling the system to meet the continued growth of neural networks pushed future designs into new territory.

Scaling Up

As the footprint of the system grew, so too did the complexity of operating it. Our focus up to now has emphasized chip-local comparisons, e.g. How expensive are these operations relative to one another? How does the memory hierarchy work? How do subsystems A and B communicate on-device? While the TPUs remain the atom of the supercomputer, as we zoom out we observe the crystalline structure of the racks and pods. The fourth generation TPU is better examined thinking about memory as one unified domain. Specialization forces care in the microarchitecture, but the questions change. Where are collectives slow? How are larger tensors handled? Can we scale the racks further? Viewing the world from low altitude we find that TPUv4’s design emphasizes system scaling and energy management.

Peeking behind the accounting curtain for a moment, they note that “most OpEx cost is for provisioning power and not for electricity use, so saving power already provisioned doesn’t improve TCO as much as one might hope” [5] . Total Cost of Ownership (TCO) tries to consider the all in cost of the pods. On the back of a napkin we break this out into CapEx (equipment, installation, etc.) and OpEx (personnel, maintenance, power, etc.). Initially CapEx might dominate ASIC design, but as the platform matures, thinking through operational requirements produces different sets of optimizations. The need for fast, power efficient devices remains but extends out into the unknowable future. As model demands increase, better economics need compositional scalability in an efficient power envelope.

A brief note: TPUv4 is the training design and TPUv4i is the inference design. The impetus was to keep training and inference chips nearly identical so that there weren’t two separate designs awkwardly diverging into separate projects [5] . The relevant change is that the inference chip has one core while the training chip is dual-core.

4th Gen MXU Simplified Model of TPUv4 Systolic Execution

Fourth generation chips keep TPUv3’s MXU footprint, totaling 4 MXUs per core. In previous MXU designs partial sums moved downwards each cycle through a series of N two-input adders, where N is the size of the array, before reaching the output accumulators. TPUv4 batches groups of four products before passing them to custom 4-input adders. Batching products reduces the length of the adder chain from N to N/4, quartering the operational latency. Above we see 12 PEs bank four multiplies to reduce hops from 12 to 3. The specific implementation of these circuits isn’t clear from the paper, but this should provide enough motivation to understand the change. This circuit design decreases die area 40% and reduces peak power 12% [5] .

CMEM Speed Ups CMEM Speed Ups [4]

Accessing DRAM is still expensive, and inference workloads underutilize chips. TPUv4 adds 128MiB shared CMEM that is like an L3 cache but with the niceties of software-managed programmability. CMEM helps to keep all 4 MXUs busy with computations at the cost of 28% of the TPUv4 die area. On the 7nm process node, SRAM memory accesses are 20x more energy-efficient than DRAM accesses [5] . CMEM’s memory bandwidth sits in between HBM and VMEM, but unlike HBM it can both read and write data. Expanding the memory hierarchy and keeping data closer to the arithmetic units allows XLA to cut out expensive trips to DRAM. During inference, prefetching model weights into SRAM for multi-tenancy drives higher utilization of chip resources that may otherwise be sitting idle. The ability to swap weights out from SRAM rather than DRAM makes paying the context switching cost feasible. All that die area and upfront CapEx gets amortized over the life of the chip in TCO so long as XLA can effectively leverage it.

SparseCores

SparseCore Block Diagram SparseCore Block Diagram [4]

Contrary to the prevailing LLMs-everywhere paradigm, ad serving and recommendation models (DLRMs) run the world. SparseCores (SC) are built to accelerate these models at the cost of 5% die area and power [4] . The key features of these models are their usage of embeddings. Embeddings map data into enormous sparse matrices. Efficiently handling these sparse matrices requires clever strategies to shard tensors across devices and to make looking up the correct slice of data fast. Unstructured sparsity suffers massive memory traffic and imbalances between compute, communication, and data-dependent execution. The MXU is ill-suited to make progress on sparse workloads because they waste cycles on empty computations and don’t directly manage communication.

SparseCores address this class of models with a “Sea of Cores” architecture designed to accelerate collectives and memory accesses [4] . SCs are segmented into 16 individual compute elements (tiles) near DRAM that support multiple outstanding memory accesses [4] . Tiles communicate over a data crossbar with one another and over the on-chip fabric to the rest of the device. A stream of CISC instructions enabling data-dependent communication gets issued by the processor’s core sequencer. The Fetch unit (8-wide SIMD vector processor, scVPU) reads data from HBM into 2.5MiB of sparse memory (Spmem), and the Flush Unit writes data out to HBM. Five on-board cross channel units (XPU) perform embedding specific operations. When embeddings are distributed across remote devices SCs leverage the existing ICI bandwidth to access remote memory. The dataflow looks as follows:

  • HBM DMA issued and read by Fetch Unit to Spmem
  • scVPU and XPUs operate on data
  • Flush unit writes data out to HBM (or remote HBM via RDMAs)

SCs alleviate the need for the MXU to handle computation and memory traffic on sparse data. They remove the CPU/DRAM bottleneck and shift sparse phases off the MXU path. The cores issue many RDMAs across the global address space of the TPUv4 pods, speeding up embeddings based models 30.1x versus CPUs [4] . Dedicating a small amount of die area to the gather/scatter intensive DLRMs allows the device to be flexible and efficient under multiple algorithmic regimes.

Cores are getting crowded: MXUs, SparseCores, VPUs, HBM, and ICI routers. We see this component management pressure in the VLIW bundles. Driving the additional MXUs and CMEM required the VLIW bundle size to expand ~25% [5] . Adding new subsystems to the microarchitecture adds efficiencies that bubble up to system level performance, but lurking behind each of these changes is the specter of wiring. Fitting more efficient work onto the package with point-to-point connections became too great a tax. Training racks need to be close to one another in the datacenter to amortize the cost of cooling infrastructure, and this physical constraint forces the usage of optical fiber. ICI cabling in TPUv2/v3 coupled rack deployments so that a supercomputer couldn’t go into operation until the full pod was deployed [5] . To realize the TCO and energy wins of the microarchitecture system scaling needed to decouple and compose.

TPUv4i Floorplan TPUv4i Floorplan [5]

The ICI needed to breathe. Previous revisions of ICI handled both on-chip communication and off-chip communication. More wires needed to be routed to/from the ICI interface as the number of components grew. This circuit layout pressure was complemented by the equally frustrating reality that handling on-chip and off-chip communication increased contention for ICI bandwidth. TPUv4 separates these concerns by adding a dedicated on-chip interconnect (OCI) fabric. The OCI interface handles data movement on-chip so that ICI can solely route traffic across chips. Notice in the fourth generation floorplan how much die area is reserved for OCI [5] . Shorter wires run between components and OCI rather than point-to-point. The OCI interface acts as the mailman. The Scalar Unit drops a message off at the OCI to submit a DMA to DRAM, and the OCI routes it to the memory controller. It tucks subsystem communication behind a unified data exchange interface that shortens wire routes and opens a path to flexible scaling in future designs.

Arbitrating memory accesses between HBM, VMEM, IMEM, SMEM and now CMEM meant maintaining too many sets of independent lanes. OCI uses 512B-wide native data paths segmented into four, 128B-wide groups across the memory hierarchy. Each group serves a quarter of the total HBM bandwidth (153GB/s) so that independent transfers don’t serialize behind one another [5] . Transferring small IMEM overlays shouldn’t have to wait on the completion of a long-latency tensor DMA. This partitioning strategy gives software more flexibility when scheduling work across a device. The full HBM bandwidth is available to each group, but software can schedule multiple concurrent transfers instead of funneling everything through one contested path. XLA plans large transfers to CMEM, CMEM feeds the arithmetic units, OCI handles message passing, and ICI routes and manages RDMAs. OCI and CMEM jointly help to improve spatial locality and reduce trips to HBM.

4D Tensor (R)DMAs

TPUv2/v3 used two-dimensional, relaxed order DMAs to stride along two axes when moving data. This forced XLA to decompose complex tensor reshapes into multiple DMA operations. TPUv4(i) uses four-dimensional DMAs that stride along three axes moving 512-byte chunks [5] . Operations that previously required multiple round-trips to memory now happen in a single DMA. The architecture distributes DMA engines throughout the chip rather than centralizing them. Each engine acts as a co-processor that can decode and execute tensor operations independently. The unified design works identically for on-chip transfers, cross-chip transfers, and host transfers. XLA inserts explicit synchronization, but in exchange it gets predictable performance and the freedom to schedule data movement aggressively. The compiler knows the latency and pipelines around it.

TPUv3 had already resorted to optical fiber across racks to enable the full 2D torus, but the 1024 node supercomputer could not expand its physical footprint. Rigid ICI wiring constraints meant individual racks couldn’t be used until each pod was deployed, and the system topology was fixed as configured unless a technician recabled the pod. Rack maintenance brought the whole pod offline with it. Optical Circuit Switching (OCS) infrastructure was the cure. Even though optical solutions are expensive, OCS optical components represent less than five percent of both system and power costs [4] [10] . Centralizing cross-rack communications inserted massive programmability into the system. Substituting the cross-rack links with a programmable OCS provided massive gains in “scale, availability, utilization, modularity, deployment, security, power, and performance” [4] , unlocking a new scaling paradigm.

OCS Logical Diagram OCS Logical Diagram

Each rack in TPUv4 is a 4x4x4 cube, where this cube configuration is chosen to optimize all-to-all communications. Previous pod sizes (16x16 in v2, up to 128x32 in v3) were topology-limited. Devices could communicate between racks over ICI, but the system topology was statically programmed by the cabling. OCS removed these hard limits by centralizing cross-rack communication over an optical switching fiber. OCS offloads link establishment to an array of MEMS mirrors that dynamically configure links between devices in milliseconds [4] . New system topologies can be programmed on the fly by software, placing workloads on idle, non-contiguous machines. Dynamically reconfiguring the OCS improves system availability, tolerating outages in 0.1% - 1.0% of the CPU hosts [6] . TPUv4 pods scale up to 8x8 racks totaling a 4096 node cluster connected over OCS.

The OCSes isolate scaling complexity. Each rack contains 64 chips laid out logically as a cube. With 6 cube faces (+/- X/Y/Z), and 16 (4x4) chips per face, 96 optical links go to the OCS per rack. In the full 64 (8x8) rack pod, that is 6,144 uplinks to the OCS. This requires 48 OCSes that have 128 active ports to connect all the uplinks [4] . Moving cross-rack interconnects to a dedicated optical panel at this scale enabled programmable topologies, eased deployment by decoupling racks, and allowed software to effectively use OCS as a “plugboard” to route around node and link failures.

Mirror, Mirror on the Wall

MEMS Mirrors MEMS Mirrors [10]

OCSes use micro-electro-mechanical systems (MEMS) mirrors that tilt in three dimensions to steer optical beams. Each OCS contains two arrays of 136 mirrors. Each mirror has four voltage-controlled actuators that rotate it along two axes, steering light from any input port to any output port with sub-degree accuracy. Rather than monitoring each of the 136 mirrors with a dedicated photodetector, OCS uses a single camera per array with an 850nm monitoring laser. Image processing algorithms optimize the high-voltage driver signals to minimize insertion loss across the entire array. Once positioned, each mirror draws 10s of milliwatts to maintain alignment [10] .

Circulators Circulators [10]

Circulators double the OCS’s effective capacity by enabling bidirectional communication. A circulator is a three-port optical device. Light entering port 1 exits port 2, light entering port 2 exits port 3. This cyclic property means a single fiber and a single OCS port can carry traffic in both directions simultaneously halving the required fiber count and OCS ports [10] .

Full connectivity of the OCS across the pods meant that the torus topologies of the previous generations could now add a third wraparound dimension. The distance between racks was no longer a constraint, and since the OCS can program chip-to-chip connections on the fly a path to new topologies emerged. Not only could the connections between racks wrap around the z-dimension, they could twist.

Example Twisted Tori Sample 1D Twisted Tori

We’ll make one modification to our previous wraparound topology diagram. Instead of wraparounds connecting only to the other side of their respective row/column, OCS programmability means that these connections can be offset. Adding twists to the wraparounds is an option not a requirement. Having the option to twist the network topology allows for new questions, e.g. given the communication pattern of this model, how should data be sent between participating chips? Twists make algorithmic experimentation and optimization two independently tractable targets and broadens the horizon of available efficiencies. Even without twisted topologies a third wraparound dimension adds bisection bandwidth to the network. The bisection bandwidth of 2D tori scales with the side length of the interconnects, N^(1/2). Adding the additional wraparound dimension scales bisection bandwidth with the area of the interconnects, N^(2/3). More paths in the topology shorten hops between participating nodes and alleviate system congestion along busy routes during synchronization. OCS better utilizes available devices and diversifies achievable topologies.

TPUv4(i) requires our thinking to broaden. We shouldn’t forget the impacts that microarchitecture improvements drive, but we need to consider the economics of the system holistically. Building warehouse scale solutions requires thinking about power provisioning, rack availability, interconnects, network topology, and accounting. Energy efficiency is still the overarching principle, but at datacenter scale. The message is simple: Target TCO over CapEx [5] . Adding CMEM is more expensive now but less expensive over time. Optical interconnects are expensive now but cost <3% of the fully operational system [4] . The duration of the design trade-offs became smeared into the future. All the same apparitions motivating TPUv1 go bump in the night, but they cast shorter shadows. TCO implies a system that requires operation, and the software that keeps the system available is an equal part of TPU’s development.

Island Hopping

Up to now we have enjoyed the quiet refuge of spreadsheet analysis, but the world is imperfect. Hardware dies, electricity spikes, and networks suffer congestion. The triumph of composing the system into decoupled, single responsibility units is not trivial, but infrastructure needs to serve real users. A cast of supporting software must keep chips available. Rock solid hardware relies on software to rationalize TCO obsession. The software is as much a part of the TPU story as the hardware.

We want to train a model. We decide which devices we need, pay rent, and start gawking at loss curves. When we submit our job for execution we don’t worry about the thousands of eager folks just like us. This mass of users vying for a fixed number of TPUs in sporadic intervals presents a problem. As the infrastructure provider what matters is that users don’t experience downtime. Components regularly fail and workloads are hard to predict. Once power has been provisioned every second of idle chip time or suboptimal workload allocation works against your best TCO approximations. Whether by underutilization or oversubscription, wasted resources are the enemy. Outer loop software that manages TPUs coordinates with XLA to find available nodes, check resource health, and configure ICI/OCS [6] . XLA needs to know which TPUs the computation will run on as well as the requested network topology because device placement is part of the program. Optimizing the system for high availability means dealing with the constraints imposed by ahead of time scheduling.

TPU Resource Fragmentation TPU Fragmentation [6]

Slices, Single Program Multiple Data (SPMD), and gang scheduling undergird TPU execution. Most workloads don’t consume an entire pod. Slices are declarations in code that allow developers to request an <X,Y,Z> device mesh which XLA uses to partition and shard models. This abstraction squirrels away both topology size and communication patterns. Pipeline parallelism may want a 2x2x1024 slice while data parallelism wants a 16x16x16 slice. The topology choice optimizes which communications are fast and which are slow. Mapping communications to a slice topology gives developers the freedom to experiment with parallelism strategies.

ICI coupling in TPUv3 meant the scheduler needed to find contiguous, healthy chips for workload placement. OCS lifted that restriction in TPUv4, but in both generations once a set of devices is selected the topology remains static for the duration of the program. A program owns the devices that it runs on until the program exits [8] . Concurrent users submitting unknowable slice sizes makes assigning devices like Tetris. The scheduler must place new jobs onto devices as old jobs pop in and out of existence. It needs mechanisms to rebalance suboptimal device allocations.

A single executable distributed to each participating device runs an identical program. SPMD encapsulates this many devices, single program framework. Developers write models as if they are running on one giant device, and the complexity of managing device-level data placement disappears from view. XLA’s partitioner rewrites every operation in the model to work on local tensor shards, inserting an AllReduce where gradients need to sync, scattering data where it needs to spread, and gathering results where they need to combine [7] . The single logical program becomes thousands of coordinated physical programs each operating on its local slice of data. Control is synchronized explicitly on-device with VLIW barriers and implicitly between devices by collectives. Gang scheduled execution means that each device launches the program all at once, trading off runtime resilience for performance. When a fault crops up during execution the job must be checkpointed and relocated [8] . The hardware stays simple, the software stays deterministic, but the orchestration layer must handle outages, link failures, and maintenance.

TPU Job Lifecycle TPU Job Lifecycle [6]

Software must anticipate failures to juggle pre-allocated workloads. In [6] they note “To train a model, all TPU processes must be simultaneously up to synchronously update their weights via ICI collectives. A single failed, or interrupted process will interrupt the whole training process.” When a user submits a job, the cluster management client Borg queues it. If resources are fragmented or a job fails, Borg can preempt running workloads to shuffle them to different devices. When a job is ready to be scheduled, Borg selects a subset of devices and publishes an xconnect to the Pod Manager. The PM discovers pending xconnects and sends commands to the appropriate OCSes to connect the requested ICI channels. Once ICI connections stabilize, libtpunet configures the device’s ICI and programs its forwarding tables. XLA consumes the topology built by libtpunet to shard the model. Once execution begins, each device has its compiled program in local memory, knows its neighbors via ICI routing tables, and has its slice of the model weights in HBM. Thousands of devices execute in lockstep, synchronizing through collectives, without a single global runtime controller. The user does not see any of this background orchestration.

Fault Tolerant Routing

ICI Interface ICI Interface [6]

Packets hop through a path of ICI switches and optical fibers to arbitrary pairs of TPUs determined by libtpunet once during setup. xconnects initiate mirror configuration in the OCS, triggering on-chip device managers to initialize physical connections between ICIs. When libtpunet issues an ICI session start it clears and rightsizes the ICI buffers in the data layer for new RDMAs. Routing is handled by forwarding tables that provide a simple abstraction to locate destination TPUs. XLA emits sets of RDMA operations called transactions for collective communications. On-chip DMA engines read data from HBM and send the data to the ICI’s transaction layer to send over the network [6] . All the required hardware for training drags down MTBF [6] , so the system needs to be resilient to outages without bringing everything down.

TPU Fault Tolerance TPU Fault Tolerance [6]

The system manages faulty links with fault tolerant routing. An offline integer linear program simulates link outages and frames the route selection as a max flow problem, using an all-to-all collective as the canonical use case. The results from the ILP are cached and accessible by libtpunet. Fault tolerant routing uses Wild First Routing as its heuristic. Packets can take a wild hop around faulty links before reverting to fault free routing. Though using fault tolerant routing may induce network congestion, TPU availability benefits [6] .

Getting the whole system to cooperate at scale needs clear boundaries and hand-offs. Borg, PM, and libtpunet bless the configuration of the workload before triggering execution. When TCO skews towards operation, getting these pieces right is as important as systolic arrays and memory hierarchies. But this presentation of how the software works is also subject to the constant evolution of the TPU. Cores communicate over OCI. Chips communicate over ICI. Racks connect remote ICI links over OCS. That leaves us with one final communication frontier: the datacenter network.

Mixture of Experts Routing Mixture of Experts Routing [38]

SPMD assumes every device can communicate over ICI with predictable latency, which constrains developers to slice sizes that fit on a single pod. Islands of accelerators [8] leave idle capacity stranded across pods, and under contention, jobs struggle to get the right-shaped device allocation. Individual pods also constrain algorithmic flexibility. Unlike traditional transformers, Mixture-of-Experts models include runtime data dependencies. The gating mechanism in MoEs introduces non-deterministic routing during execution. The SPMD model has to be stretched to express the fine-grained, data-dependent control flow these models need. If you want to shard experts across pods there is no natural way to do so. Without the DCN there is no dynamic routing, resource sharing, or use of idle chips across pods.

The datacenter network (DCN) connects islands using Google’s Jupiter fabric [9] . From the TPU’s point of view it is the communication that doesn’t occur over ICI. Extending the many cores, one logical system scaling approach gets complicated by varying latency and bandwidth characteristics. Two solutions emerged from these limitations. Multislice extends SPMD across pod boundaries. It is a conservative but compatible approach with existing code. Pathways abandoned synchronous execution for asynchronous dataflow. It is more complex but necessary for true heterogeneity.

Multislice over DCN logical Diagram Logical diagram of Multislice over DCN [26]

Multislice extends existing SPMD code across pod boundaries with minimal changes. Pod boundaries are treated as just another level in the communication hierarchy. SPMD still uses gang-scheduled execution, but XLA understands that some collectives happen over ICI and others happen over slower DCN. The familiar declarative slice syntax adds a parameter to select devices across islands. The compiler optimizes collective placement to minimize cross-pod traffic. Multislice expands the number of devices available for training by providing access to resources across pods [26] .

Pathways System Overview Pathways System Overview [8]

Pathways is a plug-in replacement for JAX’s backend that virtualizes the datacenter [8] . Instead of one giant SPMD program running in lockstep, it models execution as a DAG of compiled functions distributed across islands. Gang scheduling still happens within each island, but between islands coordination is asynchronous. There’s no single global runtime controller for the whole job. Mixture-of-Experts models can route activations dynamically to experts on different pods, and pipeline parallel stages can span multiple islands connected over DCN. Multiple programs can time-multiplex accelerators without context-switching overhead. Users request devices and the client compiles programs into a device-agnostic Pathways IR. XLA analyzes the program, the resource manager assigns physical TPUs, and the system inserts data movement operations between shards. Orchestration is complete by the time execution begins. Each device knows its program, its neighbors, and its slice of model weights.

Pathways uses a sharded dataflow model built on Plaque [8] . Each node represents a compiled function executing across thousands of TPU shards. The system uses parallel asynchronous dispatch. Pathways pipelines host side work in parallel instead of waiting for computation A to finish before preparing computation B. A control-plane scheduler per island enforces gang scheduling across programs. Between islands, Pathways uses centralized dispatch to coordinate placement and data movement. Data moves directly between accelerators over ICI within islands and DCN between islands. Pathways matches multi-controller performance by front-loading coordination work, even though cross-island dispatch is mediated by the control plane rather than issued independently by each host. This execution model performs as well as JAX and lifts restrictions on algorithmic expressibility [8] .

A dedicated upstart could reproduce the hardware design philosophy, but the software co-design makes the TPU a mammoth. Borg allocates resources and preempts jobs. The Pod Manager configures optical switches. libtpunet knows every ICI routing edge case and manages fault tolerance. XLA compiles with full knowledge of topology and latencies. SPMD partitions models while maintaining the illusion of one giant device. Multislice extends that illusion across pods. Pathways rethinks distributed execution and virtualizes the datacenter as one programmable pool. Schedulers, compilers, and coordination systems all play one long song. Building a TPU competitor needs generations of hard earned experience points. Each new design reconsiders which approaches were dead ends. Admitting you were wrong and doubling back is the game. Thinking about the TPU is thinking about Everything Else.

Ceci n’est pas une TPU

After TPUv4 the well of detailed microarchitecture papers runs dry. You can still find information scattered across the internet, but not in the same succinct, curated way. Maybe more papers will be released publicly and we’ll be able to study these designs in greater detail, but until then we have to cobble together an understanding of our own. TPUv4 and v4i are followed by TPUv5p (performance) and v5e (efficiency), Trillium (v6e), and Ironwood (v7). We know that the inference (e) optimized designs retain a single-core architecture and use 2D tori instead of 3D tori. We know the interconnect and HBM performance numbers for the fifth, sixth, and seventh generation chips. We know that Trillium and Ironwood revert to 256x256 systolic arrays. We know that Ironwood scales up to 9,216 chips for training and 256 for inference with 1.77PB HBM that delivers 42.5 FP8 ExaFlops (6x Perf/W improvement over TPUv4) with a chiplet design for next generation reasoning and MoE workloads [16] [20] [21] [23] [24] .

And I know that all of this fails to capture the totality of the enhancements since TPUv4. But a spec sheet like the one here or a primer like the one here could have told us that. The subsequent papers have focused on the system, but discussions of the system hide the simple origins of the device behind a hodgepodge of specs and new thundering heights. The essence of the thing becomes a folklorish amalgam of TPU lore. Myths are about meaning. Moore’s Law was never free in the literal sense. It required diligent engineering and enduring frustration, but decade after decade the compounding continued. The idea of Moore’s Law cast a spell that actualized its reality.

By nature the TPU is what it is not. The thrust and posturing of papers, talks, slides, and internet chatter focus on the technical minutiae, but the seams that hold this constellation of facts and figures together are the ordinary and the human. They are long emails and oscilloscopes in equal measure. How many of these choices go unseen? Hand-wringing about the system internals helps us to glimpse the creative act, but we mistake the painting for the paint chemistry. In this new world where nothing is free, every decision comes at an intentional, excruciating cost. The weight of the space of possibilities grows heavier knowing that each decision may foreclose another. Each choice is an act of reinvention in the face of a future that folds onto itself.

The TPU is an artifact born out of the quiet solace of steady hands doing careful engineering. AI DSAs are unlikely to be self-fulfilling in the same infinite feeling way as Moore’s Law. They will be five hundred ordinary decisions that compose into something greater. Can we make it smaller? Can we make it bigger? Can we make it easier to use? When we skim specs like the ones strewn above we notice the changes and feel the weight of what they represent. As new pressures get applied new entities emerge. For a moment we sense each decision branching into some unknown. Our new AI-obsessed world brings with it the demands of new ways of thinking. It is a reminder that the future is always at hand, and that if we participate in the myth-making we find that there are dragons after all.

References:

[1]: In-Datacenter Performance Analysis of a Tensor Processing Unit​

[2]: The Design Process for Google’s Training Chips: TPUv2 and TPUv3

[3]: A Domain-Specific Supercomputer for Training Deep Neural Networks

[4]: TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

[5]: Ten Lessons From Three Generations Shaped Google’s TPUv4i

[6]: Resiliency at Scale: Managing Google’s TPUv4 Machine Learning Supercomputer

[7]: GSPMD: General and Scalable Parallelization for ML Computation Graphs

[8]: PATHWAYS: ASYNCHRONOUS DISTRIBUTED DATAFLOW FOR ML

[9]: Jupiter Evolving: Transforming Google’s Datacenter Network via Optical Circuit Switches and Software-Defined Networking

[10]: Mission Apollo: Landing Optical Circuit Switching at Datacenter Scale

[11]: Computing’s Energy Problem

[12]: Domain-Specific Hardware Accelerators

[13]: The Accelerator Wall: Limits of Chip Specialization

[14]: The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design

[15]: Domain specific architectures for AI inference

[16]: How to Think About TPUs – Chapter 2

[17]: TPU Deep Dive

[18]: Understanding Matrix Multiplication on a Weight-Stationary Systolic Architecture

[19]: First in-depth look at Google’s TPU Architecture

[20]: WITH “IRONWOOD” TPU, GOOGLE PUSHES THE AI ACCELERATOR TO THE FLOOR

[21]: Ironwood: The first Google TPU for the age of inference

[22]: TPU Architecture – Google Documentation

[23]: Google Ironwood TPU Swings for Reasoning Model Leadership at Hot Chips 2025

[24]: Announcing Trillium, the sixth generation of Google Cloud TPU

[25]: A deep dive into SparseCore for Large Embedding Models

[26]: How to scale AI training to up to tens of thousands of Cloud TPU chips with Multislice

[27]: A Machine Learning Supercomputer With An Optically Reconfigurable Interconnect and Embeddings Support – HotChips Slides

[28]: Challenges in large scale training of Giant Transformers on Google TPU machines – HotChips Slides

[29]: Exploring Limits of ML Training on Google TPUs – HotChips Slides

[30]: Cloud TPU: Codesigning Architecture and Infrastructure – HotChips Slides

[31]: A DOMAIN-SPECIFIC TPU SUPERCOMPUTER FOR TRAINING DEEP NEURAL NETWORKS – Slides

[32]: Google’s Training Chips Revealed: TPUv2 and TPUv3 – Slides

[33]: A Decade of Machine Learning Accelerators: Lessons Learned and Carbon Footprint – MLSys Slides

[34]: Sparse-TPU: Adapting Systolic Arrays for Sparse Matrices

[35]: Systolic Array For VLSi

[36]: Why Systolic Architectures?

[37]: Doubly Twisted Torus Networks for VLSI Processor Arrays

[38]: Mixture of Experts Explained